Skip to content

dump-databases.sh: add a logical CouchDB dump for Obsidian LiveSync - #87

Merged
mathewcsims merged 1 commit into
mainfrom
couchdb-logical-dump
Aug 14, 2026
Merged

dump-databases.sh: add a logical CouchDB dump for Obsidian LiveSync#87
mathewcsims merged 1 commit into
mainfrom
couchdb-logical-dump

Conversation

@mathewcsims

Copy link
Copy Markdown
Owner

CouchDB was the only database in this repo with no consistent dump. The reasoning for leaving it out — .couch files are append-only with MVCC, so a copy caught mid-write recovers to the last valid header — was reasoned from the file format and never tested, and it only promises a consistent state, not that the file is readable by whatever CouchDB version you restore onto years later.

What it does

Dumps _all_docs?include_docs=true over the container's own HTTP API. What that buys over the file copy is a plain-JSON artifact independent of CouchDB's on-disk format, its version, and its server UUID. It restores into any CouchDB — or into nothing at all; the documents can be read with jq.

Credentials come from the container's own environment and are fed to curl via --config on stdin, so the password appears in no argv, on the host or inside the container — same hygiene as MYSQL_PWD in dump_mysql. curl -f so an HTTP error is a non-zero exit run_dump catches, rather than an error body written out as though it were a dump.

Tested by restoring it, not just by running it

The dump was fed back through _bulk_docs with new_edits=false into a scratch database:

  • 864 documents in, 864 out, zero _bulk_docs errors
  • First five document ids and their revisions identical to the source
  • Scratch database dropped afterwards; _all_dbs confirmed back to ["_replicator","_users","obsidian"]

The restore command is in the function header. new_edits=false is the part that matters — it preserves original _rev values, so a restored database is recognisable to already-configured clients rather than a fresh one they must re-sync from scratch.

Two things this would have broken silently

Both fixed here:

  • Rotation. The cleanup glob strips -<stamp>.(sql|db).gz to derive a label. A .json.gz matched nothing, so the whole filename became the label, the glob found no files, and these dumps would have accumulated forever instead of keeping $KEEP. json added, with a comment explaining why that list is load-bearing.
  • Health check. Added obsidian. It returns 401 unauthenticated (require_valid_user), which the existing rule already counts as healthy — a served 401 is the auth gate working.

Recorded honestly

It does not compress. The contents are LiveSync ciphertext, and encrypted bytes are incompressible, so the .gz is roughly raw size (5.3 MB tonight) and Kopia will dedupe little between nights. Fine at a text vault's scale; worth revisiting if the vault ever grows attachments.

_local/ documents are excluded — replication checkpoints and LiveSync's device registry, i.e. per-device sync state regenerated on reconnect, not user data.

Full script run clean: all 10 dumps ok, all 11 health checks healthy.

🤖 Generated with Claude Code

The only database in this repo with no consistent dump. The reasoning for
leaving it out was that .couch files are append-only with MVCC, so a copy
caught mid-write recovers to the last valid header — reasoned from the file
format, never tested, and it only promises *a* consistent state, not that the
file is readable by whatever CouchDB version you restore onto later.

Dumps `_all_docs?include_docs=true` over the container's own HTTP API. What
that buys over the file copy is a plain-JSON artifact independent of
CouchDB's on-disk format, its version, and its server UUID.

Credentials come from the container's own environment and are fed to curl via
--config on STDIN, so the password appears in no argv, on the host or inside
the container — the same hygiene as MYSQL_PWD in dump_mysql. curl -f so an
HTTP error is a non-zero exit run_dump catches, rather than an error body
written out as though it were a dump.

TESTED BY RESTORING IT, not just by running it. The dump was fed back through
_bulk_docs with new_edits=false into a scratch database: 864 documents in, 864
out, zero errors, the first five document ids AND their revisions identical to
the source. Scratch database dropped afterwards. The restore command is in the
function header.

Two things this change would have broken silently, both fixed here:

* The rotation glob strips `-<stamp>.(sql|db).gz` to derive a label. A
  .json.gz matched nothing, so the whole filename became the label, the glob
  found no files, and these dumps would have accumulated forever instead of
  keeping $KEEP. `json` added, with a comment saying why the list matters.
* The post-dump health check gained `obsidian`. It returns 401 unauthenticated
  (require_valid_user), which the existing rule already counts as healthy — a
  served 401 is the auth gate working.

Recorded honestly in the header: it does not compress. The contents are
LiveSync ciphertext, and encrypted bytes are incompressible, so the .gz is
roughly raw size (5.3 MB tonight) and Kopia will dedupe little between nights.
Fine at a text vault's scale; revisit if the vault grows attachments.

_local/ documents are excluded, being replication checkpoints and LiveSync's
device registry — per-device sync state, regenerated on reconnect, not data.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@mathewcsims
mathewcsims merged commit 133bff8 into main Aug 14, 2026
7 checks passed
@mathewcsims
mathewcsims deleted the couchdb-logical-dump branch August 14, 2026 11:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant