Skip to content

Ops: migrate Railway Postgres + collab from chasehuh → Sume (recreate + pg_restore) #94

Open
Open
Ops: migrate Railway Postgres + collab from chasehuh → Sume (recreate + pg_restore)#94

Description

@chasehuh
chasehuh
opened on Aug 9, 2026

Summary

Move agentnote's Railway-hosted Postgres + agentnote-collab off the personal chasehuh Railway workspace onto the Sume Railway workspace by recreating the stack and restoring data with pg_dump/pg_restore. Do not use Railway project transfer. Keep the Vercel app on team cwhuh / project memo (https://www.agentnote.dev) for this issue — only Railway-side ownership + env cutover.

Why This Matters

  • chasehuh Railway hit a trial expiry outage (2026-08-09): Postgres Offline, collab Failed, Vercel saw ECONNRESET / Connection terminated unexpectedly (Next.js digest 2206690639). Data survived on volume (~120 MB) after Hobby reactivation, but personal-trial billing is an unacceptable SPOF for a daily notes app.
  • Sume workspace already hosts other paid infra; putting agentnote DB + collab there aligns billing, backups, and ops with the rest of Chase's stack.
  • Dataset is small (~12 MB DB, tens of notes) — recreate+restore is cheaper and safer than workspace transfer edge cases (volumes, TCP proxies, env refs).

Conversation Context

Decisions locked in chat (2026-08-09, agent transcript f550bb91-8b0e-4029-944c-abdeb28e8fcf):

  1. Chosen approach: recreate under Sume Railway (new Postgres + collab), pg_dump → pg_restore, flip Vercel envs, then decommission chasehuh. Rejected: Railway project transfer.
  2. Scope preference: Railway-first. GitHub repo transfer and Vercel team move are optional later, not required for this issue.
  3. Immediate prod recovery already done (do not redo as "migration"):
    • Hobby plan restored chasehuh Postgres Online; data intact.
    • Sticky serverless failure after outage: warm Vercel isolates kept a rejected ensureSchema promise → every hit resurfaced digest 2206690639 even though DB was healthy. Fixed in PR #93 (lib/db.ts: clear sticky promise, recreate pool, retry once; serverless-friendly pool settings). Production redeploy flushed warm isolates.
  4. Operator keeps multiple agentnote.dev tabs open by default — cutover must not leave split-brain CRDT (two DBs / two collab URLs).
  5. Prior note-body wipe RCAs (fix: stop 409-rebased stale buffers from silently clobbering newer note bodies (0804 wipe RCA) #73, fix: stop row-stamped peer drafts from laundering stale buffers into a valid save token (post-#73 multi-tab clobber) #74) are orthogonal; migration must preserve note_revisions and CRDT tables.

Current Behavior

Ownership map (source = chasehuh)

Layer Today
GitHub chasehuh/agentnote
Vercel team cwhuh, project memo, prod https://www.agentnote.dev (also agentnote.dev, legacy memo.chasehuh.com redirects)
Railway workspace chasehuh (personal), project chasehuh, id 075539a8-061e-4223-9c7b-e0eb9cebe200, env production / 911f18c1-8dc5-40b5-8f1c-563eb080fad5
Postgres Online, image postgres-ssl:18, region sfo, volume postgres-volume ~120 MB / 500 MB, volume id cd5bb48b-661d-4f99-9233-8e69b669a408, service id 1e6b049b-87bb-4d04-8795-35481d5593d7
Public DB proxy tokaido.proxy.rlwy.net:21292 → app DB railway
Private DB postgres.railway.internal:5432 (collab should use this)
Collab service agentnote-collab, id 3c62fd5e-d13a-42fd-b9c9-f5bebb3d1555, https://agentnote-collab-production.up.railway.app, /health → {"ok":true,"service":"agentnote-collab"}
Clerk existing production Clerk app (keys on Vercel + collab) — unchanged by this migration

Vercel production env keys (values secret — do not paste into issues)

  • DATABASE_URL → public TCP proxy host tokaido.proxy.rlwy.net:21292 / db railway
  • NEXT_PUBLIC_AGENTNOTE_CRDT=1
  • NEXT_PUBLIC_AGENTNOTE_COLLAB_URL=wss://agentnote-collab-production.up.railway.app
  • CLERK_SECRET_KEY, NEXT_PUBLIC_CLERK_*, CRON_SECRET

Live data snapshot (2026-08-09, chasehuh)

Table Rows (approx)
notes 57
note_revisions 851
note_aliases 4
note_doc_snapshots 20
note_doc_updates 0
DB size ~12 MB

Target Railway workspace

  • Railway workspace name: Sume (workspace_id 286434e5-8392-454e-b4d5-3458e919dbae).
  • No agentnote project there yet. Existing Sume projects are unrelated (sume-com, sume_so, sume-com-gha-runners, train_chasehuh_com, agent-sandbox, sume-hermes-cloud-poc, abgcmo_com).

Naming: "Sume" vs "sumelabs" (read this before touching a console)

Three different vendors use three different names for the same org. Confusing the Railway workspace with the Vercel team is the single easiest way to provision into the wrong account.

Thing Exact name to look for Id Not called
Railway workspace — target Sume 286434e5-8392-454e-b4d5-3458e919dbae not sumelabs, not Sume Labs
Railway workspace — source chasehuh (personal) c8e45e5e-9fc4-4d92-a062-77004d068145 —
Vercel team owning this app cwhuh → project memo → www.agentnote.dev — not sumelabs; out of scope to move
Vercel team for other Sume work sumelabs — unrelated to this issue
GitHub repo chasehuh/agentnote — out of scope to transfer

Needs verification (raised by read-only audit, 2026-08-09): the Railway API reports both workspaces as type: personal, team_id: none. So Sume is a personal Railway workspace, not a Railway Team/org. Moving there separates billing account and blast radius from the trial-expired chasehuh account, which is the stated goal — but it does not by itself remove the "one personal account is a SPOF" class of risk (no seat-based recovery, no org billing contact). Decide before cutover:

  • (a) proceed as-is onto Sume personal workspace (accepted, documented risk), or
  • (b) first upgrade/convert Sume to a Railway Team so billing and access survive one person.
    This does not change the runbook below — only which workspace step 1 provisions into.

Needs verification: which card/plan pays for Sume (must be a non-trial paid plan before cutover), and preferred region (source is sfo — match unless Sume standard is elsewhere).

App code that depends on this infra

  • lib/db.ts — Pool + ensureSchema migrations (post-fix: recover DB pool after Railway disconnects #93 recovery)
  • lib/notes.ts, lib/crdt/* — note CRUD + CRDT persistence
  • services/collab/ — Hocuspocus server; env: DATABASE_URL, CLERK_SECRET_KEY, optional AGENTNOTE_ALLOWED_ORIGINS, PORT
  • components/agentnote-app.tsx — reads NEXT_PUBLIC_AGENTNOTE_CRDT / NEXT_PUBLIC_AGENTNOTE_COLLAB_URL
  • README.md — CRDT + Railway ops docs (update URLs after cutover)
  • vercel.json crons — /api/cron/purge-archived 09:00 UTC, /api/cron/purge-note-revisions 09:15 UTC, /api/cron/compact-note-docs 09:30 UTC, all CRON_SECRET-gated and all hitting DATABASE_URL

Related issues

Desired Behavior

  1. New Railway project under Sume workspace, e.g. agentnote (name TBD), region preferably sfo, with:
    • Postgres 18 (or current Railway Postgres template) + volume
    • Daily volume backups + PITR enabled on day one
    • TCP proxy for Vercel DATABASE_URL (public)
    • agentnote-collab service deployed from services/collab with private DATABASE_URL (*.railway.internal), same Clerk secret as prod, AGENTNOTE_ALLOWED_ORIGINS pinned to production origins
  2. Logical dump of chasehuh railway DB restored into the new DB with row-count + digest equality checks.
  3. Controlled cutover behind a hard write freeze on the source database — no dual-write, no "close your tabs and hope".
  4. Post-cutover: /health ok, signed-in notes list loads, open a CRDT note, multi-tab sync works, revision count matches.
  5. chasehuh Railway project left frozen read-only and scheduled for delete after a 7-day soak, not deleted on cutover day.
  6. README + ops notes updated with new hosts (no secrets).

Source Of Truth

Internal

  • README.md — CRDT, collab env table, Railway backup steps, production collab URL
  • services/collab/railway.json — build/start/healthcheck (see caveat in Implementation Notes — this file is not what the live service runs)
  • services/collab/src/env.ts — required env
  • lib/db.ts — schema bootstrap + pool behavior (fix: recover DB pool after Railway disconnects #93)
  • .env.example — env names

External

Proposed API / Schema

No application schema change. Migration is infra + data only.

Preserve public tables, indexes, FKs and sequence positions:

  • notes (id PK TEXT, user_id, title, body, created_at, updated_at, is_public, public_id, published_at, author_handle, deleted_at, parent_id)
  • note_aliases (alias PK, note_id FK)
  • note_revisions (id BIGSERIAL, note_id FK, user_id, title, body, created_at)
  • note_doc_updates (seq BIGSERIAL, note_id FK, user_id, update_bin BYTEA, created_at)
  • note_doc_snapshots (note_id PK/FK, user_id, state_bin BYTEA, state_vector BYTEA, through_seq BIGINT, updated_at)

Restore with pg_dump --format=custom --no-owner --no-acl into an empty database so pg_restore recreates schema, data, indexes, FKs and setvals the BIGSERIAL sequences. ensureSchema then runs as an idempotent no-op on first app hit.

Do not restore --data-only into an app-created schema. note_revisions.id and note_doc_updates.seq are BIGSERIAL; a data-only load into fresh tables can leave the sequences at 1 and the next CRDT append will collide on the PK.

Validation Rules

  • Cutover is atomic from the app's POV: never leave prod Vercel pointing at the new DB while any collab writes the old DB (or vice versa).
  • Do not rotate Clerk keys as part of this issue.
  • Do not change note ids / public_ids.
  • Do not paste production passwords, Clerk secrets, full DATABASE_URLs, or note bodies into GitHub. Digests and row counts only.

Implementation Notes

Cutover runbook

Ordering is load-bearing. Steps 0–4 are reversible and can be done days ahead; the maintenance window is only steps 5–8 (target < 20 minutes, realistically ~5 for a 12 MB dump).

Schedule constraint: avoid 09:00–09:35 UTC — all three Vercel crons fire in that window against DATABASE_URL and would run mid-flip.

Preconditions: psql/pg_dump/pg_restore ≥ 18 on the operator machine (pg_dump --version; brew install libpq on macOS — a v16 client cannot dump an 18 server), Railway CLI logged in, Vercel CLI logged in to team cwhuh.

0. Inventory freeze (source, read-only)

Capture the source baseline. Keep the output locally; only counts/digests go in the issue.

railway link            # select workspace chasehuh → project chasehuh → env production
railway connect Postgres --tunnel-only     # leave running; prints local host/port/password
-- baseline.sql — run against SOURCE
SELECT current_setting('server_version') AS pg_version,
       pg_size_pretty(pg_database_size(current_database())) AS db_size;

SELECT 'notes'              AS t, count(*) FROM notes
UNION ALL SELECT 'note_aliases',       count(*) FROM note_aliases
UNION ALL SELECT 'note_revisions',     count(*) FROM note_revisions
UNION ALL SELECT 'note_doc_snapshots', count(*) FROM note_doc_snapshots
UNION ALL SELECT 'note_doc_updates',   count(*) FROM note_doc_updates
ORDER BY 1;

SELECT sequencename, last_value FROM pg_sequences
WHERE schemaname = 'public' ORDER BY 1;

Also record: the collab service's build settings (see caveat below), the TCP proxy host/port, and the AGENTNOTE_ALLOWED_ORIGINS value.

1. Provision the Sume project (no data yet)

  1. Create project agentnote in Railway workspace Sume (286434e5-…) — re-read the naming table above before clicking.
  2. Add Postgres (match source major version 18), region sfo unless Sume standard differs.
  3. Backups tab → enable Daily (retained 6 days) and, recommended, Weekly (1 month).
  4. Backups tab → Enable PITR. Do this now: the restore window starts at the first post-enable base backup, so enabling it after cutover leaves the migration day itself uncovered.
  5. Postgres service → Settings → Networking → TCP Proxy on port 5432. Copy DATABASE_PUBLIC_URL from the service variables — do not assume the old tokaido.… hostname or port carry over; recreate always mints a new host, port and password.
  6. Confirm the Sume plan is a paid, non-trial plan before continuing.

2. Deploy agentnote-collab to Sume (still pointed at an empty DB)

Connect the service to GitHub chasehuh/agentnote (the Sume workspace needs its own GitHub authorization) and set:

Setting Value
Root directory repo root (build runs pnpm from the workspace root)
Build command pnpm install --frozen-lockfile && pnpm --filter @agentnote/collab build
Start command node services/collab/dist/index.mjs
Healthcheck path /health
Restart policy on failure, max 10

Caveat found in the read-only audit (2026-08-09): the live chasehuh service reports builder RAILPACK, while services/collab/railway.json declares NIXPACKS. That file is not being applied (it sits at services/collab/, not the service root). The current build config lives in dashboard settings, so recreating from the repo file alone will not reproduce it — enter the settings above by hand, or set the service's config-as-code path to services/collab/railway.json deliberately and accept the builder change. Pick one and record which.

Variables on the new collab service:

  • DATABASE_URL → reference variable ${{Postgres.DATABASE_URL}} (private *.railway.internal), never the public proxy URL. A reference survives password rotation; a pasted URL does not.
  • CLERK_SECRET_KEY → same production Clerk secret as today (not rotated by this issue).
  • AGENTNOTE_ALLOWED_ORIGINS → https://www.agentnote.dev,https://agentnote.dev,https://memo.chasehuh.com
  • PORT → injected by Railway; do not set.

Verify: curl -sS https://<new-collab-host>/health → {"ok":true,"service":"agentnote-collab"}.

⚠️ /health proves almost nothing. onRequest answers /health before any DB call, and readCollabEnv() only checks that DATABASE_URL is set. A collab service with a wrong or unreachable DATABASE_URL will still report Online and pass the Railway healthcheck. Real DB verification only happens when a client opens a room (step 8).

✅ That same property is what makes this step safe to do before the restore: nothing touches Postgres until a room opens, so the target DB stays empty. Do not point any browser at the new collab URL before step 6 — one authenticated room open runs ensureSchema, creates the tables, and the pg_restore in step 7 will then fail on an already-populated schema.

3. Rehearsal restore (proves the dump, touches nothing live)

# terminal A: tunnel to SOURCE
railway connect Postgres --tunnel-only

# terminal B
pg_dump "postgresql://postgres:<pw>@localhost:<port>/railway" \
  --format=custom --no-owner --no-acl \
  --file="agentnote-rehearsal-$(date -u +%Y%m%dT%H%M%SZ).dump"

Restore into a scratch database on the Sume server, leaving railway empty:

psql "$SUME_DATABASE_PUBLIC_URL" -c 'CREATE DATABASE restore_drill;'
time pg_restore --dbname="${SUME_DATABASE_PUBLIC_URL%/railway}/restore_drill" \
  --no-owner --no-acl --exit-on-error \
  agentnote-rehearsal-*.dump

Run the verification SQL (§ Verification SQL) against restore_drill, then:

psql "$SUME_DATABASE_PUBLIC_URL" -c 'DROP DATABASE restore_drill;'

Record how long the restore took — that number sizes the maintenance window. Then confirm railway is still pristine:

SELECT count(*) AS public_tables FROM pg_tables WHERE schemaname = 'public';  -- must be 0

4. Stage the Vercel env change (do not apply yet)

Have the new values ready to paste, and note the current production deployment id for Instant Rollback:

vercel --scope cwhuh ls memo --prod        # record the current prod deployment URL/id

New values to stage: DATABASE_URL = Sume DATABASE_PUBLIC_URL, NEXT_PUBLIC_AGENTNOTE_COLLAB_URL = wss://<new-collab-host>.

Optional but recommended, and cheapest to do now: put a stable custom domain in front of collab (collab.agentnote.dev CNAME → the Railway service domain) so NEXT_PUBLIC_AGENTNOTE_COLLAB_URL never has to change again — every future change to it costs a full production rebuild. Needs verification: where agentnote.dev DNS is managed.

🔒 Maintenance window starts here

5. Freeze the source (this is the anti-split-brain primitive)

Closing tabs is not a freeze. Make the source database physically unable to accept writes:

-- against SOURCE
ALTER DATABASE railway SET default_transaction_read_only = on;

-- kick existing sessions so they reconnect under the new default
SELECT pg_terminate_backend(pid) FROM pg_stat_activity
WHERE datname = 'railway' AND pid <> pg_backend_pid();

Then scale the source collab to 0 replicas (Railway → agentnote-collab → Settings → Replicas), so no Hocuspocus process can hold a room open.

Expect, and accept, that the source app now errors rather than degrading to read-only: ensureSchema issues CREATE TABLE IF NOT EXISTS / ALTER TABLE on cold isolates, which fail under default_transaction_read_only. Warm isolates may keep serving reads. Treat www.agentnote.dev as down for the length of the window; that is the point — nothing can write to the old DB behind your back.

Confirm the freeze:

SELECT count(*) AS writers FROM pg_stat_activity
WHERE datname = 'railway' AND state <> 'idle' AND pid <> pg_backend_pid();

6. Final dump

pg_dump "postgresql://postgres:<pw>@localhost:<port>/railway" \
  --format=custom --no-owner --no-acl \
  --file="agentnote-final-$(date -u +%Y%m%dT%H%M%SZ).dump"

pg_dump has no incremental mode — this is a fresh full dump, and at ~12 MB it takes seconds. (The earlier draft's "final incremental dump" is not a thing; ignore it.) Store the file somewhere private: it contains every note body. Not in git, not in a chat message.

7. Restore into the empty Sume railway database

Guard first, then restore:

psql "$SUME_DATABASE_PUBLIC_URL" -c \
  "SELECT count(*) AS public_tables FROM pg_tables WHERE schemaname='public';"   # must be 0

pg_restore --dbname="$SUME_DATABASE_PUBLIC_URL" \
  --no-owner --no-acl --exit-on-error \
  agentnote-final-*.dump

Now run the full verification SQL against both source and target. Because the source is frozen, the digests must be exactly equal. If any row differs, stop and roll back (§ Rollback) — do not flip.

8. Flip Vercel and rebuild

vercel --scope cwhuh env rm  DATABASE_URL production
vercel --scope cwhuh env add DATABASE_URL production          # paste Sume DATABASE_PUBLIC_URL
vercel --scope cwhuh env rm  NEXT_PUBLIC_AGENTNOTE_COLLAB_URL production
vercel --scope cwhuh env add NEXT_PUBLIC_AGENTNOTE_COLLAB_URL production   # wss://<new-collab-host>
vercel --scope cwhuh --prod --force                            # NEW BUILD, not a promote

NEXT_PUBLIC_* is inlined at build time. A new build is mandatory. Promoting or rolling back to an existing deployment reuses the old bundle and would ship the old collab URL against the new database — the exact split-brain this window exists to prevent. (Instant Rollback is the right tool for rollback, and the wrong tool for cutover.)

DATABASE_URL is read at runtime, but the old deployment's warm isolates hold pooled sockets to the old DB until they drain. #93 hardened the pool against this; the source freeze is what actually makes it safe. Vercel keeps serving the old deployment until the new build is ready — a few minutes of frozen-source errors is the expected user-visible cost.

Leave the source frozen. Do not un-freeze on success — a frozen source is the guarantee that no write can land somewhere that will be thrown away.

🔓 Maintenance window ends here

9. Smoke (immediately after the build goes live)

  1. curl -sS https://<new-collab-host>/health → ok.
  2. Load https://www.agentnote.dev signed in → notes list renders with the expected count.
  3. Open a recent note; confirm the body matches (this is the first real proof that collab's private DATABASE_URL works).
  4. Open the same note in a second tab; type in one, confirm it appears in the other in well under a second (WebSocket path, not the 1.5 s poll).
  5. Hard-refresh both tabs; confirm no duplicated text (would indicate two document histories).
  6. Confirm note_revisions gained a row for the edited note.
  7. Vercel runtime logs: no Connection terminated unexpectedly, no digest 2206690639 storms.
  8. Railway → new Postgres → confirm the Daily backup schedule and PITR are still enabled and healthy after the write load.

10. Soak (7 days), then decommission

  • Keep the chasehuh project Online but frozen (default_transaction_read_only = on, collab at 0 replicas) for 7 days. Frozen-not-deleted is what makes rollback possible.
  • Watch the first cron cycle (09:00–09:35 UTC next day) succeed against the new DB.
  • Before deleting anything, archive one final pg_dump of the source offsite. Railway docs: wiping a volume deletes all of its backups — volume backups do not survive project deletion, only logical dumps do.
  • Record the decommission date in an issue comment, then delete the chasehuh project.

Verification SQL

Run identically against source and target. Output is counts, lengths, and digests — no note content, so results are safe to paste into the issue.

-- 1. row counts
SELECT 'notes'              AS t, count(*) FROM notes
UNION ALL SELECT 'note_aliases',       count(*) FROM note_aliases
UNION ALL SELECT 'note_revisions',     count(*) FROM note_revisions
UNION ALL SELECT 'note_doc_snapshots', count(*) FROM note_doc_snapshots
UNION ALL SELECT 'note_doc_updates',   count(*) FROM note_doc_updates
ORDER BY 1;

-- 2. full-content digests (must match exactly while the source is frozen)
SELECT 'notes' AS t,
       md5(string_agg(n::text, '|' ORDER BY n.id)) AS digest FROM notes n
UNION ALL
SELECT 'note_aliases',
       md5(string_agg(a::text, '|' ORDER BY a.alias)) FROM note_aliases a
UNION ALL
SELECT 'note_revisions',
       md5(string_agg(r::text, '|' ORDER BY r.id)) FROM note_revisions r
UNION ALL
SELECT 'note_doc_snapshots',
       md5(string_agg(s::text, '|' ORDER BY s.note_id)) FROM note_doc_snapshots s
UNION ALL
SELECT 'note_doc_updates',
       md5(string_agg(u::text, '|' ORDER BY u.seq)) FROM note_doc_updates u
ORDER BY 1;

-- 3. CRDT payload integrity (byte lengths, not bytes)
SELECT note_id, through_seq,
       octet_length(state_bin)    AS state_bytes,
       octet_length(state_vector) AS sv_bytes
FROM note_doc_snapshots ORDER BY note_id;

-- 4. sequences — the data-only-restore footgun
SELECT sequencename, last_value FROM pg_sequences
WHERE schemaname = 'public' ORDER BY 1;

-- 5. referential integrity after restore (all must be 0)
SELECT
  (SELECT count(*) FROM note_aliases a       LEFT JOIN notes n ON n.id = a.note_id       WHERE n.id IS NULL) AS orphan_aliases,
  (SELECT count(*) FROM note_revisions r     LEFT JOIN notes n ON n.id = r.note_id       WHERE n.id IS NULL) AS orphan_revisions,
  (SELECT count(*) FROM note_doc_snapshots s LEFT JOIN notes n ON n.id = s.note_id       WHERE n.id IS NULL) AS orphan_snapshots,
  (SELECT count(*) FROM note_doc_updates u   LEFT JOIN notes n ON n.id = u.note_id       WHERE n.id IS NULL) AS orphan_updates,
  (SELECT count(*) FROM notes WHERE user_id IS NULL)                                                          AS null_user_ids;

-- 6. structure parity (target only, after first app hit; ensureSchema must be a no-op)
SELECT table_name, column_name, data_type, is_nullable
FROM information_schema.columns
WHERE table_schema = 'public' ORDER BY table_name, ordinal_position;

Rollback

Pick the row matching how far you got. The decision hinges on whether any write has landed on the Sume DB.

Failure point Action Data loss
Steps 0–4 (provisioning, rehearsal) Nothing to undo. Leave Sume resources or delete them. Source was never frozen. none
Step 5–7 — freeze applied, restore failed or digests mismatch ALTER DATABASE railway RESET default_transaction_read_only; on source, terminate sessions again so it takes effect, scale source collab back to 1 replica. Drop/recreate the Sume railway DB before retrying. Vercel untouched. none
Step 8 — env flipped, build failed or smoke fails, no writes on Sume yet Vercel Instant Rollback to the pre-cutover deployment id from step 4 (its bundle still has the old collab URL inlined — correct here). Then revert both env vars so the next build is consistent. Unfreeze the source as above. none
After smoke passed and users have written to Sume Do not roll back envs naively — the old bundle writes to the old DB and you get two divergent histories. Fix forward if at all possible. If reverting is unavoidable: freeze the Sume DB the same way, pg_dump it, restore into a fresh empty database on the source server, repoint DATABASE_URL at that new DB, unfreeze, rebuild. none if done as a reverse migration; unbounded if envs are flipped without one

Always verify after any rollback: source unfrozen (SHOW default_transaction_read_only; → off), source collab back to 1 replica, and the two-tab CRDT smoke test passing.

Likely files to modify (docs only)

  • README.md — production collab URL (line ~203), Railway workspace name in the ops/backups section (~234–240), and a pointer to PITR alongside the existing daily-backup steps
  • Optionally docs/ops-railway-sume.md if README grows too long — Needs verification whether that split is wanted

Tests

  • No unit test required for infra.
  • The verification SQL above is the test; paste counts + digests into the issue as the evidence trail.
  • Optional scripts/verify-note-db-counts.ts — Needs verification / non-goal unless cheap.

Edge Cases And Risks

  • Split-brain CRDT — two writers, one document. Two collab processes (old + new) or two databases holding the same note doc produce divergent note_doc_updates sequences. This does not heal: snapshots are compacted per-database with their own through_seq, so a naive union of the two histories surfaces as duplicated body text, not a merge. Prevention is the step-5 freeze plus scaling the source collab to 0 — not tab hygiene. README already warns that two room keys for one note means two in-memory documents reconciling only through Postgres; two Postgres instances is the same bug, worse.
  • Client IndexedDB survives the cutover. Tabs mirror each note into agentnote.note.{userId}.{noteId} (y-indexeddb) and push anything the server has never seen on reconnect. After the flip, stale tabs re-push local edits into the new server — self-healing for the CRDT body, but not for HTTP-path writes (title, parent_id, publish, archive, note_revisions), which simply go to whichever DB that tab's deployment points at. A note archived during the window can also be resurrected by a stale tab's replay. The source freeze is what bounds this.
  • /health is a liveness check, not a DB check (verified in services/collab/src/index.ts): it returns before touching Postgres, and readCollabEnv() only asserts DATABASE_URL is non-empty. A misconfigured collab shows Online and green healthchecks. Only step 9.3 proves the database wiring.
  • Booting the app against the target too early poisons the restore. ensureSchema runs on the first room open / first request and creates the full schema. pg_restore --exit-on-error into that non-empty schema aborts; without --exit-on-error it half-loads and leaves sequences wrong. Hence the public_tables = 0 guard in step 7.
  • Sticky serverless pools: hardened in fix: recover DB pool after Railway disconnects #93 (rejected ensureSchema promise cleared, pool recreated, one retry, max: 1 / allowExitOnIdle), but still require a new production build — and the build is mandatory anyway for NEXT_PUBLIC_AGENTNOTE_COLLAB_URL.
  • Promote ≠ deploy. Vercel Instant Rollback/promote reuses a built bundle with NEXT_PUBLIC_* already inlined. Correct for rollback, catastrophic as a cutover shortcut.
  • Proxy host/port/password churn: recreate always mints a new TCP proxy hostname, port and Postgres password. Copy DATABASE_PUBLIC_URL fresh; never assume tokaido.proxy.rlwy.net:21292 carries over.
  • Use a reference variable for collab's DATABASE_URL (${{Postgres.DATABASE_URL}}). A hardcoded private URL breaks silently on password rotation; the public proxy adds latency, egress billing, and idle disconnects.
  • Private networking is IPv6-only between Railway services. pg handles it, but if the new collab is ever moved outside the project the private host stops resolving.
  • Volume backups do not protect against deletion. Railway: wiping a volume deletes all of its backups, and backups restore only into the same project + environment. The offsite pg_dump is the only artifact that survives deleting chasehuh.
  • PITR window starts at enable time. Enabling it after cutover leaves migration day uncovered — hence step 1.4.
  • Backup retention is shorter than intuition: Daily = 6 days, Weekly = 1 month, Monthly = 3 months. A 7-day soak outlives the daily retention by a day.
  • Manual backups are capped at 50% of volume size — fine at 120 MB / 500 MB, worth knowing before the volume grows.
  • Cron collision: the three vercel.json crons fire 09:00/09:15/09:30 UTC against DATABASE_URL. Cutting over inside that window means a purge or compaction job running against a half-flipped stack.
  • pg_dump client version must be ≥ server (18). An older client refuses the dump outright.
  • Dump files contain every note body — treat as production-sensitive; never attach to the issue.
  • Sume is a personal-type Railway workspace (see naming section) — resolves the trial-expiry SPOF, not the single-account SPOF.
  • Region mismatch if Sume defaults away from sfo — added latency Vercel ↔ DB on every request.
  • Trial/billing on Sume: confirm the paid plan before cutover so a fresh volume is not stranded mid-migration.

Non-Goals

  • Transferring the Railway project between workspaces
  • Moving GitHub repo org or the Vercel project to sumelabs
  • Changing Clerk application / OAuth setup
  • Schema redesign, CRDT protocol changes, or multi-region HA
  • Deleting chasehuh on cutover day
  • Migrating local/dev databases

Acceptance Criteria

  • Sume Railway project has Online Postgres + Online agentnote-collab, and collab's DB wiring is proven by opening a real note (not by /health)
  • Daily volume backups and PITR enabled on Sume Postgres before the cutover window
  • Rehearsal restore completed into a scratch DB and its duration recorded
  • Source database frozen (default_transaction_read_only = on) and source collab scaled to 0 for the whole window
  • Row counts and md5 digests match exactly, source vs target, for all five tables
  • pg_sequences.last_value for note_revisions and note_doc_updates matches source
  • Referential-integrity query returns all zeros on the target
  • Vercel Production DATABASE_URL and NEXT_PUBLIC_AGENTNOTE_COLLAB_URL point at Sume endpoints
  • A new production build (not a promote) completed after the env change
  • Signed-in smoke passes: list + open note + two-tab CRDT edit + refresh with no duplicated text + note_revisions row added
  • First post-cutover cron cycle (09:00–09:35 UTC) succeeded against the new DB
  • Pre-cutover deployment id recorded for Instant Rollback; rollback path rehearsed on paper
  • README updated with new collab URL / workspace name / PITR pointer (no secrets)
  • Offsite final dump archived before decommission; chasehuh decommission date recorded in an issue comment and the source left frozen until then

QA Plan

  1. Pre-flight: pg_dump --version ≥ 18; Sume plan is paid; PITR + Daily backups green; target railway has public_tables = 0.
  2. Rehearsal: restore into restore_drill, run all six verification queries, drop it, re-confirm public_tables = 0.
  3. Frozen parity: with the source frozen, run the count + digest queries on both sides; require exact equality before flipping.
  4. Post-flip functional: /health; load https://www.agentnote.dev; open a recent note; two-tab live edit; hard refresh both tabs and confirm no duplicated text; confirm a note_revisions row appeared.
  5. Log check: Vercel runtime logs clean of Connection terminated / digest 2206690639; Railway collab logs clean of collab auth rejected bursts and collab store skipped.
  6. Cron: confirm the next 09:00–09:35 UTC cycle returns 200s (CRON_SECRET unchanged).
  7. Backup drill on the new host: take one manual backup and confirm it lands.
  8. Soak: 7 days with the source frozen, then archive the final dump and decommission.

Suggested PR Scope

  • S — docs-only PR after infra cutover (README.md, optional docs/ops-railway-sume.md)
  • Infra work is Railway dashboard + CLI + Vercel env — likely zero app code
  • Do not bundle unrelated editor features

Runbook provenance

Expanded 2026-08-09 by an Opus pass over the live infrastructure using read-only Railway APIs and the repo at fix/db-schema-ready-reset. No resources were created, no env changed, no data dumped or restored. Findings folded in above: Railway Sume reports type: personal; the live collab service builds with RAILPACK while services/collab/railway.json declares NIXPACKS (the file is not in effect); /health never touches Postgres; backup retention is Daily 6d / Weekly 1mo / Monthly 3mo and PITR must be enabled ahead of time; pg_dump has no incremental mode, so the earlier "final incremental dump" step was replaced with a single frozen-source full dump.

Open Needs verification items for a human operator: (1) Sume personal workspace vs. converting to a Railway Team; (2) who pays for Sume and on which plan; (3) preferred region if not sfo; (4) where agentnote.dev DNS is managed, if the collab.agentnote.dev stable-CNAME option is taken; (5) whether to keep collab build config in the dashboard or start honouring services/collab/railway.json; (6) whether ops docs live in README or a new docs/ file.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions