feat: pre-announcement hardening across kernel, adapters, lenses, CLI, and supply chain

.changeset/pre-announcement-hardening.md

@@ -0,0 +1,43 @@
+---
+"@libredb/libredb": minor
+---
+
+Durability, safety, and API-contract hardening across the kernel, adapters, lenses, and CLI (the pre-announcement audit wave).
+
+On-disk format: new databases now begin with an 8-byte `LRDB` magic/version header, and each record header carries a checksum of its own length field. Files written by earlier releases (headerless) keep opening through a legacy read path, and keep their legacy record framing on later appends. The header is what lets `open()` refuse a file that is not a LibreDB database with a clear error instead of destroying it; the record-header checksum is what lets recovery refuse a damaged length field instead of mistaking it for a torn tail.
+
+DOWNGRADE WARNING: a file written by this release must never be opened by 0.1.3 or older — the old recovery cannot parse the header, classifies the whole file as a torn tail, and silently truncates it to zero bytes. Back up before any downgrade. Three smaller legacy-behavior changes: a headerless file whose only record is torn/incomplete now refuses to open as `NOT_A_DATABASE` (0.1.3 recovered it to an empty database; refusing is the safe reading, since such a file is indistinguishable from a foreign one); any file shorter than the 8-byte header is likewise refused untouched (a crash inside the first bytes of a brand-new database's first-ever commit therefore needs a manual delete — nothing in it was acknowledged); and a legacy length-field corruption still reads as a torn tail (the legacy format has no header checksum — the v1 format exists to close exactly that gap).
+
+Kernel:
+
+- `open({ path })` on a non-LibreDB file throws `NOT_A_DATABASE` and leaves the file byte-for-byte untouched (previously the file was silently truncated to zero).
+- Recovery classifies failures: a torn tail truncates (reported through the new `onRecovery` open option), while mid-log corruption throws `CORRUPT_WAL` and truncates nothing. Record payloads are structurally validated during replay.
+- A failed append/fsync latches the database: every later `transact()` throws `FAILED` until reopen, so an IO error can never lead recovery to silently drop later acknowledged commits.
+- `transact()` rejects async callbacks (`ASYNC_TRANSACTION`): writes after an `await` could never reach the log.
+- Keys and values are copied at the transaction boundary in both directions — caller buffer reuse and mutation of returned buffers can no longer corrupt the store.
+- `getRange` snapshots at first iteration, so delete-while-scanning visits every entry exactly once.
+- `close()` inside a transaction throws `CLOSE_IN_TRANSACTION` instead of surfacing a raw file error.
+- `open()` takes an exclusive per-file lock (`<path>.lock`, pid/host/nonce): a second writer throws `LOCKED` instead of silently diverging; locks from verifiably dead holders are reclaimed automatically. `FileSystem` gains an optional `lock()` seam method.
+- All kernel failures are now `LibreDbError` instances carrying a stable `code` (exported, with the `ErrorCode` and `RecoveryInfo` types).
+
+Adapters:
+
+- node-fs: creating a database fsyncs the parent directory (a fresh database can no longer vanish wholesale on power loss); a directory-fsync failure that is not a platform limitation (e.g. EIO) now surfaces as an error instead of being silently ignored; recovery truncation is fsync'd; reads are positional on the WAL's own file descriptor instead of re-reading the whole file per call.
+- OPFS: reads loop until filled, so a legal short read can no longer masquerade as a torn tail; recovery treats an incomplete read as an IO fault (`INCOMPLETE_READ`), never as license to truncate.
+
+Lenses:
+
+- Collection/table names may not be empty or contain `:` (both broke namespace isolation); ids keep full freedom.
+- Strings that are not well-formed UTF-16 (lone surrogates) are rejected wherever they would become keys, ids, names, or kv values — distinct strings can no longer silently collide on one key.
+- Relational `number` columns reject `NaN` and the infinities (JSON would store them as `null`).
+- `doc()` refuses a name cataloged as a relational table (it would bypass schema validation); `table()` refuses a document collection's name.
+- `find()`/`where()` reject a predicate field explicitly set to `undefined`, which previously matched documents *missing* the field.
+
+CLI:
+
+- Write commands rely on the kernel's exclusive lock; `--force` removes a lock only when its holder is not verifiably alive, and never deletes a file that is not a libredb lock. Automatic reclaim is stricter still: only a lock whose holder is VERIFIABLY dead (same host, pid gone) is reclaimed without `--force` — anonymous locks (empty, or the sentinel-only 0.1.x format) carry no liveness information and now require `--force`.
+- `get`/`scan` escape control characters (including tab and newline) by default so untrusted values cannot inject terminal escape sequences — scripts that consumed values verbatim should pass `--raw`.
+
+New exports: `LibreDbError`, `ErrorCode`, `RecoveryInfo`, `nodeFileSystem`, and `readonlyFileSystem` (open a database for inspection with no lock and no writes — the supported way to read a file a live writer holds).
+
+Docker image now runs as a non-root user (distroless `:nonroot`, uid 65532): bind-mounted directories must be writable by that uid, or pass `--user "$(id -u):$(id -g)"`.

.github/PULL_REQUEST_TEMPLATE.md

@@ -1,6 +1,7 @@
 <!--
 Thank you for contributing to LibreDB. Keep the PR title in Conventional Commits form
-(e.g. "feat(lens): add prefix scan") — PRs are squash-merged and the title becomes the changelog entry.
+(e.g. "feat(lens): add prefix scan") — PRs are squash-merged and the title becomes the commit message.
+The user-facing changelog comes from changesets, not the PR title.
 -->
 
 ## What and why

.github/dependabot.yml

@@ -0,0 +1,25 @@
+# Dependabot keeps the SHA-pinned actions, the digest-pinned Docker bases, and
+# the npm devDependencies from going stale: pinning without an update loop
+# inverts over time (CVE fixes never arrive unless someone remembers to bump
+# digests by hand). Weekly PRs preserve the pinning discipline — every bump is
+# still a reviewed, pinned change.
+version: 2
+updates:
+  - package-ecosystem: "github-actions"
+    directory: "/"
+    schedule:
+      interval: "weekly"
+  - package-ecosystem: "docker"
+    directory: "/"
+    schedule:
+      interval: "weekly"
+  - package-ecosystem: "npm"
+    directory: "/"
+    schedule:
+      interval: "weekly"
+    # Zero runtime dependencies is a design fact (license tripwire enforces
+    # it), so everything here is devDependencies; group the noise.
+    groups:
+      dev-dependencies:
+        patterns:
+          - "*"

.github/workflows/ci.yml

@@ -49,6 +49,7 @@ jobs:
         run: |
           BYTES=$(bun run size --json 2>/dev/null | jq -r '.[0].size // empty' || true)
           if [ -n "$BYTES" ]; then KB=$(awk "BEGIN{printf \"%.2f\", $BYTES/1024}"); else KB="n/a"; fi
+          BUDGET=$(jq -r '.[0].limit' .size-limit.json)
           {
             echo "## LibreDB CI"
             echo ""
@@ -60,5 +61,36 @@ jobs:
             echo "| License tripwire (runtime deps) | ${{ steps.license.outcome }} |"
             echo ""
             echo "- Coverage: 100% line/function/statement (enforced by bunfig.toml)"
-            echo "- Bundle: ${KB} kB (min+brotli) / 4 kB budget"
+            echo "- Bundle: ${KB} kB (min+brotli) / ${BUDGET} budget"
           } >> "$GITHUB_STEP_SUMMARY"
+
+  node-smoke:
+    name: node 22 smoke
+    # package.json declares engines.node >= 22 and the docs advertise `npx
+    # libredb` and Node embedding, but the gate runs on Bun. This job keeps the
+    # Node half of the declared runtime surface off the honor system: build the
+    # real dist/, then open, write, lock, reopen, and read a file-backed
+    # database under Node, plus one pass of the CLI entry.
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+      - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0
+        with:
+          bun-version-file: .bun-version
+      - uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
+        with:
+          node-version: "22"
+      - run: bun install --frozen-lockfile
+      - run: bun run build
+      - name: Smoke-test the built package under Node
+        run: node scripts/node-smoke.mjs
+      # The standalone binary is otherwise only compiled during publish (and
+      # only after npm publish succeeded), so a compile regression would first
+      # surface at release time. One cheap compile + round-trip here keeps it
+      # honest on every PR.
+      - name: Smoke-test the compiled standalone binary
+        run: |
+          bun build --compile src/cli/main.ts --outfile /tmp/libredb-smoke
+          /tmp/libredb-smoke set /tmp/smoke.libredb greeting hello
+          test "$(/tmp/libredb-smoke get /tmp/smoke.libredb greeting)" = "hello"
+          /tmp/libredb-smoke stats /tmp/smoke.libredb

.github/workflows/publish.yml

@@ -29,8 +29,23 @@ jobs:
     # `latest` channel or Docker `:latest`. Every job carries the same guard.
     if: ${{ !github.event.release.prerelease }}
     runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      id-token: write # OIDC: signs the npm provenance attestation
     steps:
       - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+      # The release tag must name the version the tree actually carries: a tag
+      # created before `changeset:version` (or a typo'd tag) must fail loudly
+      # here, not ship binaries and images labeled with the wrong semver.
+      - name: Verify release tag matches package.json
+        env:
+          TAG: ${{ github.event.release.tag_name }}
+        run: |
+          PKG="v$(node -p 'require("./package.json").version')"
+          if [ "$TAG" != "$PKG" ]; then
+            echo "Release tag $TAG does not match package.json $PKG" >&2
+            exit 1
+          fi
       - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0
         with:
           bun-version-file: .bun-version
@@ -45,9 +60,9 @@ jobs:
           node-version: "22"
           registry-url: "https://registry.npmjs.org"
       # npm publish runs prepublishOnly (build + attw + publint) automatically.
-      # Provenance is intentionally omitted while the repo is private; once it is
-      # public, add `--provenance` here and `id-token: write` to permissions.
-      - run: npm publish --access public
+      # --provenance attaches a signed attestation binding the tarball to this
+      # repo, commit, and workflow (verifiable via `npm audit signatures`).
+      - run: npm publish --access public --provenance
         env:
           NODE_AUTH_TOKEN: ${{ secrets.NPMJS_TOKEN }}
       # Run summary on the run page (no emoji, per repo convention).
@@ -81,7 +96,11 @@ jobs:
       - uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
         with:
           node-version: "22"
-      - run: npx --yes jsr publish
+      # Pinned: an unpinned `jsr@latest` would hand the publish-capable OIDC
+      # token to whatever the registry serves at run time (supply-chain risk).
+      # Bump the version deliberately, like every other pin in this workflow
+      # (Dependabot does not track npx invocations).
+      - run: npx --yes jsr@0.14.3 publish
       - name: Job summary
         if: success()
         run: |

.size-limit.json

@@ -2,12 +2,17 @@
   {
     "name": "public entry (min+brotli)",
     "path": "dist/index.js",
-    "ignore": ["node:fs", "node:os", "node:path"],
-    "limit": "4 kB"
+    "ignore": [
+      "node:crypto",
+      "node:fs",
+      "node:os",
+      "node:path"
+    ],
+    "limit": "6 kB"
   },
   {
     "name": "browser entry (min+brotli)",
     "path": "dist/browser.js",
-    "limit": "4 kB"
+    "limit": "5 kB"
   }
 ]

ARCHITECTURE.md

@@ -169,7 +169,11 @@ interface Database {
   close(): void;
 }
 
-const open: (options?: { path?: string; fs?: FileSystem }) => Database;
+const open: (options?: {
+  path?: string;
+  fs?: FileSystem;
+  onRecovery?: (info: RecoveryInfo) => void;  // reports a truncated torn tail
+}) => Database;
 ```
 
 With a `path`, the database is file-backed and durable. Without one, it is purely
@@ -340,8 +344,17 @@ time comes, without the three-box machinery becoming mandatory.
 Each committed transaction is one length-framed, checksummed record:
 
 ```
-  record = [ u32 payloadLength ] [ u32 crc32(payload) ] [ payload ]
-                  4 bytes              4 bytes
+  file   = [ 4-byte magic "LRDB" ] [ u16 formatVersion ] [ u16 reserved ]   8-byte file header
+           ...followed by records, back to back
+
+  record = [ u32 payloadLength ] [ u32 crc32(length bytes) ] [ u32 crc32(payload) ] [ payload ]
+                  4 bytes               4 bytes                     4 bytes
+
+  The record header carries a checksum of itself (the length field), so
+  recovery can tell a trustworthy length from a damaged one. Files written by
+  v0.1.x predate the file header and use 8-byte record headers
+  [ u32 payloadLength ][ u32 crc32(payload) ]; they still open, and keep that
+  legacy framing on appends (a file's format cannot change mid-file).
 
   payload = one or more ops, back to back:
       set    = [ u8 1 ] [ u32 keyLen ] [ key ] [ u32 valLen ] [ value ]
@@ -354,8 +367,10 @@ A real record from a live file, decoded (the first write of a relational table -
 its schema, before any row):
 
 ```
+  4c 52 44 42 00 01 00 00             file header: magic "LRDB", format version 1
   00 00 00 9d                         payloadLength = 157
-  ea 44 78 48                         crc32
+  af fa 30 e5                         crc32 of the header's own length bytes
+  ea 44 78 48                         crc32 of the payload
   01                                  op = SET
   00 00 00 16                         keyLen = 22
   00 6c 69 62 72 65 64 62 3a ...      key = "\x00libredb:catalog:users"
@@ -375,9 +390,13 @@ The fsync happens *before* the in-memory commit becomes visible:
    on disk, survives a crash         now visible in memory
 ```
 
-So when `transact()` returns, the change is on disk. If the append fails, the code
-throws with memory and disk still agreeing on the *prior* state. A read-only
-transaction writes nothing to the log.
+So when `transact()` returns, the change is on disk. If the append or fsync
+fails, `transact()` throws a typed error (code FAILED) with memory keeping the
+prior state -- and the database latches: every later `transact()` throws FAILED
+until it is closed and reopened. Appending past a possibly-torn tail could let
+the next recovery silently discard later, acknowledged commits; refusing
+further writes is what keeps "a returned transact() is durable" true. A
+read-only transaction writes nothing to the log.
 
 ### Recovery and torn writes
 
@@ -387,10 +406,13 @@ Because the log is append-only and fsynced, a crash can only ever damage the
 ```
   recover(file):
       replay each intact record in order, rebuilding the sorted array
-      stop at the first record that is:
-          - torn   (header promises more bytes than exist), or
-          - corrupt (crc32 of payload does not match)
-      truncate the file at that point   # next append starts from a clean boundary
+      a TORN TAIL -- a header promising more bytes than exist, or the FINAL
+      record's payload failing its crc32 -- is truncated away and fsynced,
+      and reported through the open option onRecovery({ truncatedBytes })
+      a payload crc32 failure with intact data AFTER it, or a v1 record
+      header failing its own checksum, is CORRUPTION: recovery throws
+      CORRUPT_WAL and leaves the file untouched -- it never truncates
+      committed records to get past damage
 ```
 
 ```
@@ -610,7 +632,10 @@ The kernel never calls `node:fs` directly. Every byte to disk goes through one
 small interface:
 
 ```ts
-interface FileSystem { open(path: string): WalFile; }
+interface FileSystem {
+  open(path: string): WalFile;
+  lock?(path: string): () => void;  // optional exclusive lock; a second open throws LOCKED
+}
 
 interface WalFile {
   size(): number;
@@ -662,7 +687,7 @@ what reopening each decision would entail, not a committed roadmap.
 | Working set         | the whole store lives in memory      | bounded by RAM, not disk                 |
 | Log growth          | append-only, no compaction           | the file grows with write *history*      |
 | Multi-key atomicity | lenses auto-commit per operation     | for atomic multi-writes use `transact`   |
-| Durability edge     | no directory fsync on first create   | see 10.3 -- a known hardening gap        |
+| Durability edge     | browser OPFS `flush()` is weaker than POSIX fsync | power-loss durability in the browser is engine-dependent |
 
 None of these are hidden. The cost is concentrated where it is cheapest to reason
 about, and the throughline is that **every one of them could be addressed above the
@@ -717,16 +742,17 @@ cleanly into two kinds of work.
 These close real correctness gaps on the existing design. They are tracked as
 known limitations, not new directions:
 
-- **Directory fsync on first file creation.** Creating a file durably requires
-  fsyncing the *directory*, not just the file -- otherwise a power loss can lose
-  the directory entry for a freshly created database. Currently not done.
+- **Directory fsync on first file creation** -- done. The `node:fs` adapter
+  fsyncs the parent directory when it creates the database file, so a power
+  loss can no longer lose the directory entry of a freshly created database.
 - **WAL compaction / checkpointing.** Today the log only grows (section 5). A
   checkpoint -- fold the committed state into a compact snapshot, then trim the
   log -- bounds file size and speeds recovery. This is the single most important
   hardening item for any long-lived database.
-- **Short-read recovery robustness.** A note carried out of the crash-recovery
-  work: make sure a partial read at the tail is always treated as a torn record,
-  never as data.
+- **Short-read recovery robustness** -- done, and stricter than first sketched:
+  a read that returns fewer bytes than the file holds throws a typed
+  INCOMPLETE_READ error instead of being treated as data or as a torn tail, so
+  a transient IO fault can never cause recovery to truncate committed records.
 
 **Scaling features (each reopens a locked decision).**
 These are not on the v1 path and would each force a deliberate decision to be
@@ -769,7 +795,11 @@ by staying small and correct, not by absorbing every feature.
 | `lens/document.ts`    | lens         | JSON documents, by-id CRUD, scan and find            |
 | `lens/relational.ts`  | lens         | schema-validated tables, where/select/join           |
 | `lens/catalog.ts`     | edge         | reserved namespace, registry, validate-on-reopen     |
-| `index.ts`            | public        | the npm export surface                               |
+| `adapter/node-fs.ts`  | edge         | the real `node:fs` WAL adapter (fd reads, directory fsync, lock file) |
+| `adapter/opfs.ts`     | edge         | the browser OPFS WAL adapter                         |
+| `cli/`                | tooling      | the libredb CLI (inspect, stats, get, scan, set, delete, import) and the read-only filesystem |
+| `index.ts`            | public       | the Node npm export surface                          |
+| `browser.ts`          | public       | the browser export surface (no Node built-ins)       |
 | `sim/`                | test harness | simulated filesystem and crash-recovery oracle (DST) |
 
 ---
@@ -793,7 +823,7 @@ Putting it together -- what happens when you insert a row into a file-backed tab
                 append + fsync                    # durable here
                 committed = working               # visible here
 
-  on disk (demo.libredb), now two records:
+  on disk (demo.libredb), now an 8-byte file header followed by two records:
       [ \x00libredb:catalog:users -> {relational, schema} ]
       [ users:1 -> {"id":"1","name":"Ada","age":36,"active":true} ]