dump-databases.sh: add a logical CouchDB dump for Obsidian LiveSync - #87
Merged
Conversation
The only database in this repo with no consistent dump. The reasoning for leaving it out was that .couch files are append-only with MVCC, so a copy caught mid-write recovers to the last valid header — reasoned from the file format, never tested, and it only promises *a* consistent state, not that the file is readable by whatever CouchDB version you restore onto later. Dumps `_all_docs?include_docs=true` over the container's own HTTP API. What that buys over the file copy is a plain-JSON artifact independent of CouchDB's on-disk format, its version, and its server UUID. Credentials come from the container's own environment and are fed to curl via --config on STDIN, so the password appears in no argv, on the host or inside the container — the same hygiene as MYSQL_PWD in dump_mysql. curl -f so an HTTP error is a non-zero exit run_dump catches, rather than an error body written out as though it were a dump. TESTED BY RESTORING IT, not just by running it. The dump was fed back through _bulk_docs with new_edits=false into a scratch database: 864 documents in, 864 out, zero errors, the first five document ids AND their revisions identical to the source. Scratch database dropped afterwards. The restore command is in the function header. Two things this change would have broken silently, both fixed here: * The rotation glob strips `-<stamp>.(sql|db).gz` to derive a label. A .json.gz matched nothing, so the whole filename became the label, the glob found no files, and these dumps would have accumulated forever instead of keeping $KEEP. `json` added, with a comment saying why the list matters. * The post-dump health check gained `obsidian`. It returns 401 unauthenticated (require_valid_user), which the existing rule already counts as healthy — a served 401 is the auth gate working. Recorded honestly in the header: it does not compress. The contents are LiveSync ciphertext, and encrypted bytes are incompressible, so the .gz is roughly raw size (5.3 MB tonight) and Kopia will dedupe little between nights. Fine at a text vault's scale; revisit if the vault grows attachments. _local/ documents are excluded, being replication checkpoints and LiveSync's device registry — per-device sync state, regenerated on reconnect, not data. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CouchDB was the only database in this repo with no consistent dump. The reasoning for leaving it out —
.couchfiles are append-only with MVCC, so a copy caught mid-write recovers to the last valid header — was reasoned from the file format and never tested, and it only promises a consistent state, not that the file is readable by whatever CouchDB version you restore onto years later.What it does
Dumps
_all_docs?include_docs=trueover the container's own HTTP API. What that buys over the file copy is a plain-JSON artifact independent of CouchDB's on-disk format, its version, and its server UUID. It restores into any CouchDB — or into nothing at all; the documents can be read withjq.Credentials come from the container's own environment and are fed to curl via
--configon stdin, so the password appears in no argv, on the host or inside the container — same hygiene asMYSQL_PWDindump_mysql.curl -fso an HTTP error is a non-zero exitrun_dumpcatches, rather than an error body written out as though it were a dump.Tested by restoring it, not just by running it
The dump was fed back through
_bulk_docswithnew_edits=falseinto a scratch database:_bulk_docserrors_all_dbsconfirmed back to["_replicator","_users","obsidian"]The restore command is in the function header.
new_edits=falseis the part that matters — it preserves original_revvalues, so a restored database is recognisable to already-configured clients rather than a fresh one they must re-sync from scratch.Two things this would have broken silently
Both fixed here:
-<stamp>.(sql|db).gzto derive a label. A.json.gzmatched nothing, so the whole filename became the label, the glob found no files, and these dumps would have accumulated forever instead of keeping$KEEP.jsonadded, with a comment explaining why that list is load-bearing.obsidian. It returns 401 unauthenticated (require_valid_user), which the existing rule already counts as healthy — a served 401 is the auth gate working.Recorded honestly
It does not compress. The contents are LiveSync ciphertext, and encrypted bytes are incompressible, so the
.gzis roughly raw size (5.3 MB tonight) and Kopia will dedupe little between nights. Fine at a text vault's scale; worth revisiting if the vault ever grows attachments._local/documents are excluded — replication checkpoints and LiveSync's device registry, i.e. per-device sync state regenerated on reconnect, not user data.Full script run clean: all 10 dumps ok, all 11 health checks healthy.
🤖 Generated with Claude Code