feat: pre-announcement hardening across kernel, adapters, lenses, CLI, and supply chain - #43
Merged
Conversation
- refuse non-LibreDB files with a WAL magic header; never truncate foreign bytes (headerless v0.1.x files keep opening via a legacy read path) - classify recovery failures: torn tails truncate (reported through the new onRecovery option), mid-log corruption refuses to open - validate record payload structure during replay - reject async transact() callbacks before commit - latch the database after a failed append/fsync (fsyncgate) - copy keys and values at the transaction boundary (no aliasing) - snapshot getRange so delete-while-scanning visits every entry - name close()-during-transaction instead of surfacing raw EBADF - exclusive open lock via the FileSystem seam: pid/host/nonce lock file with stale-lock reclaim; CLI --force verifies holder liveness - node-fs: fd-based positional reads, parent-directory fsync on create, fsync after recovery truncation - opfs: loop reads so a short read cannot masquerade as a torn tail - recovery treats an incomplete read as an IO fault, never truncation - typed LibreDbError with stable codes on every kernel failure
… profiles - reject ':' and empty collection/table names (namespace isolation) - reject lone-surrogate keys, ids, names, and kv values (UTF-8 round-trip) - reject NaN/Infinity in relational number columns - doc() refuses relational tables; table() refuses document collections - find()/where() reject explicitly-undefined predicate fields eagerly - CLI get/scan escape control characters by default; --raw opts out - SimFS gains armAppendError/armFsyncError; DST covers partial-append latch, fsync-fault latch, and multi-cycle crash-recover-write runs - seeded binary fuzz: full byte alphabet, empty/large payloads, and the sorted invariant asserted on recovered entries
- npm publish with provenance (OIDC id-token); release tag must match package.json version; JSR CLI pinned to an exact version - Dependabot for actions, docker, and npm devDependencies - Node 22 smoke job exercises the built package (open, lenses, lock, reopen) so declared Node support is off the honor system - Docker runtime switches to distroless :nonroot - README/RELIABILITY state the durability contract precisely: the clean-crash guarantee plus every handled failure mode (IO-error latch, exclusive lock, foreign-file refusal, corruption refusal, short reads); status line updated to early beta 0.1.x - performance envelope and bulk-load guidance documented - CLI docs cover the new lock semantics, output escaping, --raw, and the backup/restore procedure - OPFS flush()'s weaker power-loss guarantee documented in the adapter and BROWSER.md - changeset for the whole hardening wave (minor)
There was a problem hiding this comment.
Pull request overview
This PR hardens LibreDB ahead of a public announcement by tightening the kernel durability/format contract, improving adapter correctness (including locking), adding lens-boundary input validation, and expanding test/CI/supply-chain assurances so behavior, docs, and release workflows align.
Changes:
- Kernel: introduce LRDB magic + format version header, typed
LibreDbErrorcodes, corruption vs torn-tail recovery classification, asynctransactrejection, IO-failure latch (“fsyncgate”), buffer copy contract, getRange snapshot semantics, and an optional filesystem lock seam. - Adapters/CLI: implement exclusive
<path>.locklocking innode-fs, looped reads in OPFS/node-fs, safer--forceunlocking, and default terminal-output sanitization (--rawopt-out). - Tests/CI/docs: add extensive hardening + fuzz/DST suites, Node 22 smoke job, Dependabot config, pinned JSR CLI, npm provenance, distroless nonroot Docker runtime, and updated reliability/performance/CLI docs.
Reviewed changes
Copilot reviewed 36 out of 36 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| src/core.ts | WAL header + recovery classification, typed errors, async transact rejection, fsync-failure latch, buffer copying, getRange snapshotting, lock seam |
| src/index.ts | Re-export typed error surface (LibreDbError, codes/types) at Node entry |
| src/browser.ts | Re-export typed error surface at browser entry |
| src/adapter/node-fs.ts | FD-based IO, directory fsync-on-create, <path>.lock exclusive lock + --force unlock helpers |
| src/adapter/node-fs.test.ts | Tests for node-fs IO, directory fsync, lock behavior, and --force semantics |
| src/adapter/opfs.ts | Loop OPFS reads to avoid legal short-read data loss; document flush limitations |
| src/cli/run.ts | Use kernel lock for writes, add --force unlock path, sanitize get/scan output by default (--raw opt-out) |
| src/cli/run.test.ts | CLI tests updated for new locking semantics and output sanitization |
| src/cli/readonly-fs.ts | Make fsync a no-op to support recovery’s fsync-after-truncate behavior |
| src/cli/readonly-fs.test.ts | Update tests to match readonly fsync no-op |
| src/cli/lock.ts | Removed legacy CLI-only advisory lock implementation |
| src/cli/lock.test.ts | Removed tests for legacy CLI-only advisory lock |
| src/core.hardening.test.ts | New kernel hardening suite (foreign-file refusal, corruption policy, latch, locking, copying, async rejection, etc.) |
| src/core.recovery.test.ts | Adjust crash simulation to account for lock behavior |
| src/lens/catalog.ts | Add well-formed UTF-16 checks, stricter namespace validation, catalog kind routing helpers |
| src/lens/catalog.test.ts | Update tests for stricter namespace-name rejection |
| src/lens/document.ts | Enforce catalog kind routing (doc vs table), reject lone surrogates in ids, reject undefined predicate values |
| src/lens/relational.ts | Reject NaN/Infinity, undefined predicate fields in where, use unguarded collection handle for schema lens |
| src/lens/relational.test.ts | Update to use collectionHandle since doc() now blocks relational names |
| src/lens/kv.ts | Reject lone-surrogate keys/values via assertWellFormedText |
| src/lens/hardening.test.ts | New tests pinning lens-boundary rejection rules (names, surrogates, NaN/Infinity, doc/table routing, undefined predicates) |
| src/sim/simfs.ts | Add deterministic append/fsync fault injection for DST |
| src/sim/dst.ts | Export torn-tail injector for composing multi-cycle crash schedules |
| src/sim/dst.test.ts | Expand DST tests for corruption classification, short-read behavior, and IO-failure latch scenarios |
| src/sim/fuzz.test.ts | New seeded binary fuzz for byte-alphabet keys/values + sortedness invariant |
| scripts/node-smoke.mjs | Node runtime smoke test over built dist/ (lenses + locking + reopen) |
| .github/workflows/ci.yml | Add Node 22 smoke job; update size-budget messaging |
| .github/workflows/publish.yml | Add tag/package.json version check, npm provenance, pinned jsr CLI, OIDC permissions |
| .github/dependabot.yml | Add weekly Dependabot updates for actions/docker/npm |
| Dockerfile | Switch runtime to distroless :nonroot pinned digest |
| .size-limit.json | Increase bundle budget to 5 kB and ignore node:crypto |
| README.md | Update durability/performance envelope/status claims to match new contract |
| docs/RELIABILITY.md | Precisely document durability contract + handled failure modes |
| docs/CLI.md | Document new locking/--force, output escaping, and backup/restore procedure |
| docs/BROWSER.md | Document OPFS flush durability caveat more explicitly |
| .changeset/pre-announcement-hardening.md | Minor changeset describing the coordinated hardening release |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…ng wave Review of PR #43 by a four-lens adversarial workflow confirmed gaps in the new guarantees; each is closed with a pinning test: - an all-zeros file parsed as an empty legacy record and was adopted (truncated + appended to); isLegacyLog now requires a non-empty first record, matching the kernel's own write invariant - the record length field was outside the checksum, so mid-log damage to a length masqueraded as a torn tail and silently truncated acknowledged commits; v1 record headers now checksum their own length field (refuse instead of truncate), and legacy files keep the legacy framing on later appends - a torn-header prefix check now compares against the full 8 expected header bytes, so a short foreign file sharing only the LRDB magic is refused instead of truncated - stale-lock reclaim and forceUnlock were check-then-delete; both now claim the lock atomically by renaming it aside, so two racers can never both acquire and force can never delete a fresh writer's lock - release() verifies ownership on the claimed bytes and restores a lock it does not own - a refused open no longer leaks the WAL file descriptor; close() releases the lock even when the underlying file close throws - getRange snapshots entry references and copies per yield, so an early-exited scan no longer pays for the whole range - transact() rejects async callbacks at compile time as well as runtime - doc()'s relational-name guard runs lazily inside each operation's transaction, so handles can be built inside transact() again - readonlyFileSystem and nodeFileSystem are exported: the supported way to inspect a database a live writer holds locked - node-smoke exercises the CLI entry as the workflow comment claims - docs: downgrade warning (v1 file opened by <=0.1.3 is truncated), legacy torn-first-record refusal, Docker nonroot bind-mount guidance
Member
Author
|
Adversarial review round 1 (4 lenses, 31 agents) found and this push fixes: all-zeros foreign-file adoption, unprotected record length field (v1 record headers now self-checksummed), non-atomic stale-lock reclaim (rename-aside claims), fd leak on refused opens, lock release on throwing close, getRange early-exit cost, compile-time async rejection, doc() handle construction inside transactions, and a supported readonly open. Docs updated with the downgrade warning and Docker nonroot guidance. Gate green, 336 tests, 100% coverage. |
- files shorter than the 8-byte header are refused untouched, even when they share magic-prefix bytes; the torn-first-header auto-repair is removed (destroying an ambiguous file is never the answer; nothing in a sub-header file was ever acknowledged) - commit IO failures are typed: transact() throws LibreDbError FAILED with the adapter error as cause instead of the raw ENOSPC/EIO - lock sentinel is an exact first-line match, so a file merely starting with the sentinel text is foreign - automatic stale-lock reclaim now requires a VERIFIABLY dead holder; anonymous locks (empty, or the sentinel-only 0.1.x format) carry no liveness info — they may even be a concurrent lock() between its exclusive create and its sentinel write — and need --force - isStaleLock treats only ENOENT as vanished; other read errors are real problems, not staleness - tryCreateLock loops writeSync so a partial write cannot leave a malformed lock that reads as an anonymous stray - well-formedness validation no longer depends on String.prototype.isWellFormed (absent on older browser engines the browser entry may reach); a hand-rolled surrogate scan replaces it - recover() comments clarified: the version bytes gate the v1 path, and a foreign file matching the entire 8-byte header is byte-for-byte indistinguishable from a real empty database - size budget 5 kB -> 6 kB for the public entry (lock protocol growth)
- doc()'s relational-kind guard memoizes only the settled state (name cataloged as document); an uncataloged name keeps re-checking, so a handle built before table() cataloged the name can no longer write around schema validation - parseLock validates the pid field: a sentineled-but-mangled lock downgrades to anonymous (never auto-stale) instead of probing pid=NaN as a dead holder and stealing a live writer's lock - the CI job summary reads the bundle budget from .size-limit.json instead of hardcoding a stale number
A 10000-seed soak takes ~6s and tripped bun's default 5s per-test timeout, reporting an invariant failure that was actually the clock. The invariant itself holds across all 10000 seeds (verified standalone, both assertions). The timeout now scales with the seed count so soaks fail only on a broken invariant.
…tion and cleanup - doc get/delete (and the relational paths through them) validate the id like put already did: a lone-surrogate id encodes to the replacement character's bytes and would silently read — or delete — a document legitimately stored under that id - a throwing onRecovery callback no longer leaks the WAL file handle: the callback runs inside the same guard that closes the file on a refused open - CLI inspect escapes control characters in namespace names (the lens validator rejects ':' and surrogates but control bytes are legal name characters); --raw opts out, matching get/scan
- the control-regex suppression in sanitize() targets oxlint by name (only oxlint's rule fires there; the eslint-disable form was an unused directive for eslint while still being consumed by oxlint) - types.test.ts builds comparison arrays with Array.from().concat() instead of map-spread, clearing the last standing lint warning - the node-smoke CI job also compiles the standalone binary and runs a set/get round-trip: previously a compile regression would first surface during publish, after npm publish had already succeeded
The comment claimed an early-exited scan pays only for consumed entries; in fact the reference walk over the matching range is eager at first next() — only the per-yield byte copies are lazy. Say so.
… and stale claims - CLI import enforces the kv lens's well-formed string invariant: a JSON file whose \uD800-style escapes decode to lone surrogates was previously written through the kernel directly, where two distinct malformed keys collide on the same UTF-8 bytes - BrowserOpenOptions carries onRecovery, so the browser type surface matches the runtime (RecoveryInfo was exported but the option was not expressible); pinned by a compile-level test - README line-count claim updated honestly: the kernel is one file of under a thousand lines, roughly half explanatory prose (measured: 449 code / 456 comment); DESIGN.md gains a dated addendum with the same numbers so the decision log tracks reality - the redundant reclaim continue-branch in lock() is folded away (removed and gone both retry the exclusive create), restoring true 100% line coverage across all files (node-fs was at 99.31%)
The one uncovered line in the repo was lock()'s refusal branch for the reclaim race: the holder looks dead at the isStaleLock pre-check but alive when the claimed bytes are re-judged. Desyncing the two liveness probes (spyOn process.kill) simulates the race exactly, pinning both the LOCKED refusal and the rename-back. All files are back at true 100% line/function/statement coverage.
This was referenced Jul 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the full pre-announcement audit program (docs/pre-announcement-notes.md): every Wave 1 launch blocker and every Wave 2 hardening item, in one coordinated change so the durability contract, the code, the tests, and the docs land consistent with each other.
Kernel (guarded core)
LRDB+ u16 version):open()on a non-LibreDB file now throwsNOT_A_DATABASEand leaves the file byte-for-byte untouched — previously a typo'd path silently truncated the file to zero. Headerless v0.1.x files keep opening through a legacy read path (verified against the real Studio demo file).onRecoveryoption); mid-log corruption throwsCORRUPT_WALand truncates nothing; record payloads are bounds-validated during replay. One documented limitation: a corrupted length field on the final record is indistinguishable from a torn tail in format v1.transact()throwsFAILEDuntil reopen. Without this, later acknowledged commits sat behind the torn record and were silently destroyed by the next recovery — the finding that falsified the README's headline durability claim.transact(async tx => ...)now throwsASYNC_TRANSACTIONbefore committing (writes after the first await never reached the WAL).FileSystem.lock()seam: a secondopen()on the same file — same process or another — throwsLOCKED; locks from verifiably dead holders are reclaimed automatically.LibreDbErrorwith a stablecode(exported withErrorCode,RecoveryInfo).close()inside a transaction throwsCLOSE_IN_TRANSACTIONinstead of raw EBADF.Adapters
readFileSyncper call).INCOMPLETE_READ, never as a torn tail; the weakerflush()power-loss guarantee is documented.Lenses
:(namespace isolation); lone-surrogate strings rejected wherever they become keys/ids/names/values;NaN/Infinityrejected by number columns;doc()refuses relational tables andtable()refuses document collections;find()/where()reject explicitly-undefined predicate fields.CLI
--forceverifies holder liveness and never deletes non-libredb files;get/scanescape control characters by default (--rawopts out).Tests
armAppendError/armFsyncError; DST covers the failure latch, fsync faults, and multi-cycle crash-recover-write schedules; a seeded binary fuzz drives the full byte alphabet (empty keys, multi-KB values) through crash cycles and asserts the recovered store is exactly the model and strictly sorted. Coverage stays 100% line/function/statement; 331 tests.Supply chain / CI
--provenance(OIDC); release tag verified against package.json; JSR CLI pinned; Dependabot for actions/docker/npm; a Node 22 smoke job exercises the built package; Docker runtime switches to distroless:nonroot.Docs
Notes for review
Closes #7, #13, #14, #15, #16, #17, #18, #19, #20, #21, #22, #23, #24, #25, #26, #27, #28, #29, #30, #31, #32, #33, #34, #35, #36, #37, #38, #40, #41, #42