From f16a099aa0f42e79113ca83b2640c449045ea837 Mon Sep 17 00:00:00 2001 From: Fable Date: Sat, 4 Jul 2026 16:06:53 -0500 Subject: [PATCH 1/2] =?UTF-8?q?docs(sdk):=20package=20docs=20fan-out=20?= =?UTF-8?q?=E2=80=94=20OVERVIEW,=20DETAILS,=20AGENT=5FMAP,=20topology=20pe?= =?UTF-8?q?r=20package?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 3-4 of the Moot-SDK documentation initiative. Voice: Turabian at 8th grade per the LatticeLib pilot exemplar. Front-matter pins per-repo authored_commit and per-source git blob hashes for docs-scrub. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01Pzkqg3bpbbhfuWRsYomJ3o --- .../kits/ConvergenceKit/docs/AGENT_MAP.md | 140 ++ packages/kits/ConvergenceKit/docs/DETAILS.md | 458 ++++++ .../ConvergenceKit/docs/INTERFACE_DOCTRINE.md | 115 -- packages/kits/ConvergenceKit/docs/OVERVIEW.md | 163 ++ .../kits/ConvergenceKit/docs/topology.svg | 113 ++ .../kits/PersistenceKit/docs/AGENT_MAP.md | 363 +++++ packages/kits/PersistenceKit/docs/DETAILS.md | 1361 +++++++++++++++++ .../PersistenceKit/docs/INTERFACE_DOCTRINE.md | 273 ---- packages/kits/PersistenceKit/docs/OVERVIEW.md | 279 ++++ .../kits/PersistenceKit/docs/topology.svg | 164 ++ packages/kits/QueueKit/docs/AGENT_MAP.md | 179 +++ packages/kits/QueueKit/docs/DETAILS.md | 567 +++++++ packages/kits/QueueKit/docs/OVERVIEW.md | 172 +++ packages/kits/QueueKit/docs/topology.svg | 128 ++ packages/libs/LoopbackHTTP/docs/AGENT_MAP.md | 79 + packages/libs/LoopbackHTTP/docs/DETAILS.md | 250 +++ packages/libs/LoopbackHTTP/docs/OVERVIEW.md | 110 ++ packages/libs/ObserverSink/docs/AGENT_MAP.md | 104 ++ packages/libs/ObserverSink/docs/DETAILS.md | 341 +++++ packages/libs/ObserverSink/docs/OVERVIEW.md | 129 ++ packages/libs/ObserverSink/docs/topology.svg | 104 ++ 21 files changed, 5204 insertions(+), 388 deletions(-) create mode 100644 packages/kits/ConvergenceKit/docs/AGENT_MAP.md create mode 100644 packages/kits/ConvergenceKit/docs/DETAILS.md delete mode 100644 packages/kits/ConvergenceKit/docs/INTERFACE_DOCTRINE.md create mode 100644 packages/kits/ConvergenceKit/docs/OVERVIEW.md create mode 100644 packages/kits/ConvergenceKit/docs/topology.svg create mode 100644 packages/kits/PersistenceKit/docs/AGENT_MAP.md create mode 100644 packages/kits/PersistenceKit/docs/DETAILS.md delete mode 100644 packages/kits/PersistenceKit/docs/INTERFACE_DOCTRINE.md create mode 100644 packages/kits/PersistenceKit/docs/OVERVIEW.md create mode 100644 packages/kits/PersistenceKit/docs/topology.svg create mode 100644 packages/kits/QueueKit/docs/AGENT_MAP.md create mode 100644 packages/kits/QueueKit/docs/DETAILS.md create mode 100644 packages/kits/QueueKit/docs/OVERVIEW.md create mode 100644 packages/kits/QueueKit/docs/topology.svg create mode 100644 packages/libs/LoopbackHTTP/docs/AGENT_MAP.md create mode 100644 packages/libs/LoopbackHTTP/docs/DETAILS.md create mode 100644 packages/libs/LoopbackHTTP/docs/OVERVIEW.md create mode 100644 packages/libs/ObserverSink/docs/AGENT_MAP.md create mode 100644 packages/libs/ObserverSink/docs/DETAILS.md create mode 100644 packages/libs/ObserverSink/docs/OVERVIEW.md create mode 100644 packages/libs/ObserverSink/docs/topology.svg diff --git a/packages/kits/ConvergenceKit/docs/AGENT_MAP.md b/packages/kits/ConvergenceKit/docs/AGENT_MAP.md new file mode 100644 index 0000000..a65ee40 --- /dev/null +++ b/packages/kits/ConvergenceKit/docs/AGENT_MAP.md @@ -0,0 +1,140 @@ +--- +doc: AGENT_MAP +package: ConvergenceKit +repo: moot-system +authored_commit: 909513d0a8ecb1e9e903af9f4d25b3e4f2528242 +authored_date: 2026-07-04 +sources: + - path: Sources/ConvergenceKit/SyncEngine.swift + blob: 32e69e0ea5002d8b699866183371527f675023cd + - path: Sources/ConvergenceKit/SyncRecord.swift + blob: 8c29f63ea9f965f1ea75d3bbff71632f76699233 + - path: Sources/ConvergenceKit/SyncTypes.swift + blob: 52ed3cb07f0483d0e541b86a8da080791402d132 + - path: Sources/ConvergenceKitCloudKit/CKRecordMapping.swift + blob: 9b61903296533f8044bee5a4f93ca2dc1b76b653 + - path: Sources/ConvergenceKitCloudKit/CloudKitSyncEngine.swift + blob: 7dd43ccf5454739a3d1ae1cda0870cf507c479c6 + - path: Sources/ConvergenceKitFederation/FederationIdentity.swift + blob: b64358fe36fd241049e9f5bdbbb335f1f3d29464 + - path: Sources/ConvergenceKitFederation/FederationSyncEngine.swift + blob: 9bf722f8e7fc24ae12ac72d571c135da2b40af62 + - path: Sources/ConvergenceKitFederation/HyperplaneFamilyExchange.swift + blob: 4880e847918c253f3d99a02fdd65c15f172d32e1 + - path: Sources/ConvergenceKitNone/ConvergenceKitNone.swift + blob: 5d3556a08c8bc80ec6b824ceb4e0838db9c95ecc +--- + +# AGENT_MAP — ConvergenceKit + +PURPOSE: replicates PersistenceKit rows across device/estate boundaries. One protocol (SyncEngine), three interchangeable backends (None/CloudKit/Federation). App writes only to PersistenceKit; backend observes via StorageObserver, ships via SyncRecord wire format, applies inbound through PersistenceKit's own write path (RowStore.upsert/insert/delete), which fires StorageObserver again on the receiving side. + +DEPS: core target imports SubstrateTypes (HLC, Fingerprint256), PersistenceKit (Storage, StorageObserver, TableChange, TypedValue). ConvergenceKitCloudKit adds CloudKit, os. ConvergenceKitFederation adds Crypto (swift-crypto, Ed25519), os. ConvergenceKitNone adds nothing beyond core. Imported by: NONE within the SDK at this commit — application-layer composition only; QueueKit's Package.swift explicitly excludes it as a dependency (spec §11, DECISION_KIT_GRAPH_REFACTOR_2026-05-19.md). Rust port in rust/ mirrors core types + wire format + None/Federation backends; CloudKit has no Rust port (Apple-only, by design). + +ENTRY POINTS (most callers need only these): +- SyncEngine.swift:26 `SyncEngine.enable(manifest:storage:) async throws` — call once before push/pull/subscribe +- SyncEngine.swift:34/:38 `push() async throws -> SyncReceipt` / `pull() async throws -> SyncReceipt` — one-shot cycles +- SyncEngine.swift:41 `subscribe() -> AsyncStream` — live feed for UI +- FederationSyncEngine.swift:80 `FederationSyncEngine.pair(with:via:family:)` — Federation-only, establishes peer relationship before push/pull do anything + +## Symbol Table + +### Protocol — SyncEngine.swift +- :21 `protocol SyncEngine: Sendable` — 4 methods + 1 async property, every backend conforms +- :26 `enable(manifest:storage:) async throws` — starts local-write observation; establishes remote subscriptions if any +- :30 `disable() async throws` — idempotent teardown +- :34 `push()` / :38 `pull()` — one-shot, return SyncReceipt +- :41 `subscribe() -> AsyncStream` +- :44 `state: SyncState { get async }` + +### Core types — SyncTypes.swift +- :22 `enum SyncDirection: String` — bidirectional | pushOnly | pullOnly +- :29 `enum ConflictPolicy: String` — lastWriterWinsByHLC (default) | appendOnly | localWins | remoteWins +- :42 `struct SyncedTable` — name/direction/primaryKeyColumn/conflictPolicy; :54 init defaults direction=.bidirectional, conflictPolicy=.lastWriterWinsByHLC +- :70 `struct SyncManifest` — kitID/schemaVersion/zoneIdentifier/tables; :94 `table(named:) -> SyncedTable?` +- :100 `struct SyncReceipt` — pushed/pulled/conflicts/timestamp; :113 `.empty` shared zero-value +- :117 `enum SyncEvent` — .remoteChangesApplied(count:) / .pushCompleted(receipt:) / .peerConnected(identity:) / .peerDisconnected(identity:reason:) / .error(SyncError) +- :126 `enum SyncState` — .disabled | .enabled(zone:lastPushAt:lastPullAt:) | .syncing(direction:) | .error(_:retryAt:) +- :134 `enum SyncError: Error, Equatable` — notEnabled/alreadyEnabled/schemaMismatch/kitMismatch/transportFailure/decodingFailure/encodingFailure/peerUnreachable/authenticationFailed/unsupportedTable/corruptRemoteIdentity(recordName:) — last case: NEVER fabricate a UUID for an unparseable remote recordName, see CKRecordMapping.decode + +### Wire format — SyncRecord.swift +- :28 `struct SyncRecord: Codable` — table/event/rowKey/values/hlc/schemaVersion/kitID; explicit CodingKeys — Rust serde renames match verbatim, do not let these drift +- :63 `enum SyncEventKind: String, Codable` — insert|update|delete; :68 `init(from: StorageEvent)`, :76 `.asStorageEvent` — Codable mirror, kept separate from PersistenceKit.StorageEvent deliberately +- :87 `struct PackedHLC: Codable, Hashable` — physicalTime/logicalCount/nodeID; Codable wrapper of SubstrateTypes.HLC +- :112 `struct SyncValueMap: Codable` — wraps [String:TypedValue] as [String:SyncValueBox] +- :144 `struct SyncValueBox` — kind + payload (13 TypedValue cases); adjacently-tagged JSON matching Rust `#[serde(tag="kind",content="payload")]` +- :206 `SyncValueBox: Codable` extension — :211 encode omits payload key for .null (matches Rust unit-variant omission); :227 timestamp encode guards `interval.isFinite` + Int64 range BEFORE `Int64(interval)` cast — Int64(_:) traps on NaN/±inf/overflow, corrupt inbound data can produce these +- :296 `struct FingerprintWire: Codable, Hashable` — block0..3 UInt64, Codable wrapper of SubstrateTypes.Fingerprint256 + +### None backend — ConvergenceKitNone.swift +- :28 `final class NoSyncEngine: SyncEngine` — passthrough; push/pull return .empty once enabled, throw .notEnabled before +- :53 `subscribe()` — stream that NEVER emits; closes only on caller task cancellation +- :64 `actor StateActor` — enable throws .alreadyEnabled on repeat; tracks manifest only for state reporting + +### CloudKit mapping — CKRecordMapping.swift +- :26 `enum CKRecordMapping` — generic row↔CKRecord translator, driven by table name + manifest, no per-table hardcoding +- :31 `recordType(kitID:table:) -> String` — "\(kitID)_\(table)" +- :36 `recordID(rowKey:zone:) -> CKRecord.ID` +- :43 `record(from:table:rowKey:hlc:schemaVersion:kitID:zone:) throws -> CKRecord` — adds reserved _syncHLC/_syncSchemaVersion/_syncKitID fields +- :65 `decode(_:) throws -> DecodedRecord` — reverse; throws .decodingFailure if _syncHLC/_syncSchemaVersion missing; recovers table from "kitID_table" split; **throws .corruptRemoteIdentity if recordName is not a valid UUID — NEVER fabricate a fresh UUID here** +- :103 `assign(value:to:forKey:)` (private) — TypedValue→CKRecord field; .array throws .encodingFailure (unsupported); .fingerprint packs 4×UInt64 LE into 32 bytes +- :147 `typedValue(from:)` (private) — CKRecord field→TypedValue; NSNumber objCType sniffed for bool/float/int disambiguation +- :172 `packed(_:) -> Int64` / :179 `unpacked(_:) -> HLC` — HLC↔Int64, PINNED bit layout: 48 bits physical | 12 bits logical | 4 bits node — changing widths breaks previously-stored _syncHLC values +- :192 `struct SyncMeta` — hlc/schemaVersion/kitID extracted from _sync* fields +- :198 `struct DecodedRecord` — table/rowKey/values (clean, no _sync* keys)/syncMeta + convenience accessors + +### CloudKit engine — CloudKitSyncEngine.swift +- :34 `final class CloudKitSyncEngine: SyncEngine` — :43 init(containerIdentifier:) defers CKContainer resolution to enable() so tests can construct without iCloud entitlement +- :78 `actor CloudKitStateActor` — owns container/manifest/storage/pendingOutbound/serverChangeToken/hlcGenerator +- :112 `enable(manifest:storage:)` — creates CK zone (tolerates already-exists); subscribes storage.observer.observe per push-eligible table → recordOutbound +- :106 `hlcGenerator = HLCGenerator(nodeID: Int32.random(in: 1...0x0F))` — mints HLC only when an observed change carries none +- :175 `push()` — drains pendingOutbound; builds CKRecords via CKRecordMapping.record; one modifyRecords(saving:deleting:) call; HLC: prefer change.hlc, else hlcGenerator.send(now:) (NOT currentTime() — send advances the logical counter, avoiding same-millisecond collisions) +- :254 `pull()` — recordZoneChanges(inZoneWith:since:); per-record gate: kitID match → schemaVersion match → table declared & not pushOnly; any failure caught+logged+counted as conflict, does NOT abort the batch; deletions are bare CKRecord.ID (no table info) — attempted against every non-pushOnly manifest table +- :339 `applyInbound(_:syncedTable:storage:)` (internal, not private — LWW tests call via @testable) — dispatches 4 ConflictPolicy arms; .lastWriterWinsByHLC reads existing row's _syncHLC as EITHER .hlc (InMemory) OR .int (SQLite/Postgres, raw packed value) — must handle both + +### Federation identity — FederationIdentity.swift +- :24 `struct PeerIdentity: Hashable` — 32-byte Ed25519 public key +- :32 `struct LocalIdentity` — :36 init() generates fresh Curve25519.Signing.PrivateKey per engine instance; :42 init(privateKeyBytes:) restores from caller bytes — module does NOT persist identity itself +- :48 `sign(_:) throws -> Data` +- :53 `enum FederationSignature` — :54 `verify(_:of:by:) -> Bool` — returns false (not throw) for both malformed pubkey bytes and bad signature + +### Federation pairing shapes — HyperplaneFamilyExchange.swift +- :27 `struct HyperplaneFamilySpec: Codable, Hashable` — seed + dimension (default 256); seed alone reproduces the family deterministically on both sides +- :42 `struct PairingProposal: Codable` — proposerPublicKey/proposedFamily/nonce +- :54 `struct PairingAcceptance: Codable` — accepterPublicKey/acceptedFamily/signatureOfProposal +- NOTE: these types are defined but NOT YET wired into FederationSyncEngine.pair — v1.0 pair() takes an already-agreed HyperplaneFamilySpec directly, does not negotiate/sign Proposal/Acceptance + +### Federation engine — FederationSyncEngine.swift +- :39 `final class FederationSyncEngine: SyncEngine` +- :80 `pair(with:via:family:) async throws` — exchanges pubkeys + family spec; in-process only at v1.0 +- :84 `identity: LocalIdentity { get async }` +- :98 `enum PayloadKind: UInt8, Codable` — .syncRecordBatch = 0x01 (only v1.0 variant); 0x02 RESERVED for fieldWriteEventBatch — never reassign +- :128 `envelopeSigningBytes(senderPublicKey:payloadKind:payload:hlc:) -> Data` — PINNED byte layout (all LE): pubkey(32) | kind(1) | payload_len(4) | payload(N) | hlc.physicalTime(8) | hlc.logicalCount(4) | hlc.nodeID(4); byte-identical to Rust `envelope_signing_bytes` in federation.rs — signature covers THIS, not raw payload (closes relabel/replay seam) +- :174 `struct SignedEnvelope: Codable` — senderPublicKey/payloadKind/payload/signature/hlc; batch-level hlc minted AFTER all record HLCs in the batch (strictly later) +- :211 `protocol Relay: Sendable` — send(to:message:) / drain(for:) — transport extension point; hosted relay is a drop-in conformer +- :220 `final class FederationRelay: Relay` — in-process only at v1.0, NSLock-protected [pubkey: [SignedEnvelope]] inbox dict +- :243 `actor FederationStateActor` — :244 `localIdentity = LocalIdentity()` generated once per actor +- :263 `enable(manifest:storage:)` — same observer-wiring pattern as CloudKitStateActor +- :279 `disable()` — cancels observer tasks THEN AWAITS each task's completion before clearing state (deliberately different from a cancel-only teardown — closes a race where a buffered change lands after disable() returns) +- :318 `pair(with:via:family:)` — registers peer locally, calls peerActor.acceptPeering symmetrically — one call registers both sides +- :331 `push()` — builds SyncRecord batch from pendingOutbound (HLC: change.hlc ?? hlcGenerator.send(now:), same rule as CloudKit), JSON-encodes, mints batch HLC, signs via envelopeSigningBytes, sends SignedEnvelope to every peer's relay +- :422 `pull()` — per envelope, in order: (1) senderPublicKey MUST equal the specific paired peer's key (ADR-013 — valid self-signature alone does not prove pairing authorization), (2) payloadKind MUST be .syncRecordBatch, (3) signature MUST verify over envelopeSigningBytes, (4) only then JSON-decode; each surviving SyncRecord re-checked for kitID/schemaVersion/table-declared before applyInbound +- :526 `applyInbound(_:syncedTable:storage:)` (internal, not private — LWW tests) — 4 ConflictPolicy arms × insert/update vs delete (delete gets full policy dispatch here, unlike CloudKit which receives bare record IDs with no policy routing): .appendOnly rejects remote deletes outright; .lastWriterWinsByHLC gates delete on stale-HLC check same as upsert path; .remoteWins deletes unconditionally; .localWins rejects all remote deletes + +## INVARIANTS / GOTCHAS + +- Application code NEVER calls a backend's push/pull for ordinary use — it writes to PersistenceKit; the enabled backend observes via StorageObserver and does the rest. Inbound changes apply through PersistenceKit's normal RowStore path (upsert/insert/delete), which is what wakes downstream StorageObserver watchers on the receiving side — do not special-case "just-synced" rows. +- CKRecordMapping.decode: an unparseable `recordID.recordName` MUST throw `.corruptRemoteIdentity`, never fabricate a fresh UUID. A fabricated identity creates a phantom row that desyncs forever (never matches the real remote row on later rounds). +- HLC bit layout in CKRecordMapping.packed/unpacked is PINNED: 48/12/4 bits (physical/logical/node). Changing widths invalidates every previously stored `_syncHLC` value. +- envelopeSigningBytes layout is PINNED and cross-port: pubkey(32)|kind(1)|payload_len(4 LE)|payload|hlc.physicalTime(8 LE)|hlc.logicalCount(4 LE)|hlc.nodeID(4 LE). Must stay byte-identical to Rust's `envelope_signing_bytes` in `rust/src/federation.rs`, or cross-port signatures fail verification. +- SyncRecord / SyncValueBox / PackedHLC / FingerprintWire CodingKeys are an explicit cross-port JSON contract (Rust serde field renames match these strings verbatim). Do not let automatic Codable key derivation replace these without checking the Rust side. +- PayloadKind 0x02 is RESERVED for `fieldWriteEventBatch` — never assign it to anything else; a receiver on an older build would silently misinterpret a reused byte value. +- Federation pull() enforces sender==paired-peer-key BEFORE trusting a valid signature (ADR-013): a syntactically valid self-signed envelope from a non-paired sender must be rejected, not merely a badly-signed one. +- `_syncHLC` on a stored row can arrive as TypedValue `.hlc` (InMemory backend, preserves the type) OR `.int` (SQLite/Postgres, raw packed integer, because those schemas don't declare the column as `.hlc`). Both `CloudKitStateActor.applyInbound` and `FederationStateActor.applyInbound` must handle both cases — do not assume one representation. +- lastWriterWinsByHLC gates BOTH upserts and deletes on HLC comparison in Federation's applyInbound; CloudKit's applyInbound only gates upserts this way because CloudKit deletes arrive as bare record IDs with no policy dispatch at all (deletes just happen, scoped only by "table is in the manifest and not pushOnly"). +- FederationStateActor.disable() cancels observer tasks THEN awaits their completion before clearing state — do not reduce this to cancel-only; a buffered change could otherwise land in the outbox after disable() returns. +- HLC minting for locally-observed changes with no HLC of their own always uses `HLCGenerator.send(now:)`, never a read-only clock snapshot (`currentTime()` equivalent) — `send` advances the logical counter so two changes minted in the same millisecond do not collide. Same rule in both CloudKitStateActor.push and FederationStateActor.push. +- Timestamp encoding in SyncValueBox guards `interval.isFinite` and Int64-range BEFORE casting to Int64 — `Int64(_:)` traps (crashes) on NaN/±infinity/out-of-range Double. Corrupt inbound sync data can produce such a value; the guard converts a potential crash into a catchable EncodingError. +- Public API stability per README: adding a case to SyncDirection, ConflictPolicy, SyncEvent, SyncState, or SyncError is a breaking change requiring a major version bump and a decision record. Backend additions (a fourth SyncEngine conformer) are additive, not breaking. +- CloudKit backend has NO Rust port by design (Apple-only; Swift side always handles iCloud transport). Do not expect rust/ to mirror ConvergenceKitCloudKit. +- ConvergenceKit is intentionally NOT a dependency of other in-SDK kits at this commit (e.g., QueueKit's Package.swift excludes it per spec §11) — it is wired in at the application-composition layer, not the kit-graph layer. diff --git a/packages/kits/ConvergenceKit/docs/DETAILS.md b/packages/kits/ConvergenceKit/docs/DETAILS.md new file mode 100644 index 0000000..6050a4a --- /dev/null +++ b/packages/kits/ConvergenceKit/docs/DETAILS.md @@ -0,0 +1,458 @@ +--- +doc: DETAILS +package: ConvergenceKit +repo: moot-system +authored_commit: 909513d0a8ecb1e9e903af9f4d25b3e4f2528242 +authored_date: 2026-07-04 +sources: + - path: Sources/ConvergenceKit/SyncEngine.swift + blob: 32e69e0ea5002d8b699866183371527f675023cd + - path: Sources/ConvergenceKit/SyncRecord.swift + blob: 8c29f63ea9f965f1ea75d3bbff71632f76699233 + - path: Sources/ConvergenceKit/SyncTypes.swift + blob: 52ed3cb07f0483d0e541b86a8da080791402d132 + - path: Sources/ConvergenceKitCloudKit/CKRecordMapping.swift + blob: 9b61903296533f8044bee5a4f93ca2dc1b76b653 + - path: Sources/ConvergenceKitCloudKit/CloudKitSyncEngine.swift + blob: 7dd43ccf5454739a3d1ae1cda0870cf507c479c6 + - path: Sources/ConvergenceKitFederation/FederationIdentity.swift + blob: b64358fe36fd241049e9f5bdbbb335f1f3d29464 + - path: Sources/ConvergenceKitFederation/FederationSyncEngine.swift + blob: 9bf722f8e7fc24ae12ac72d571c135da2b40af62 + - path: Sources/ConvergenceKitFederation/HyperplaneFamilyExchange.swift + blob: 4880e847918c253f3d99a02fdd65c15f172d32e1 + - path: Sources/ConvergenceKitNone/ConvergenceKitNone.swift + blob: 5d3556a08c8bc80ec6b824ceb4e0838db9c95ecc +--- + +# ConvergenceKit Details + +This document walks through every source file in the package. Read +`OVERVIEW.md` first for the big picture. Files appear here in pipeline +order: the shared types and protocol that every backend depends on, the +wire format, then the three backends from simplest to most complex. + +## SyncTypes.swift + +This file provides the core enumerations and value types shared by every +backend: `SyncDirection`, `ConflictPolicy`, `SyncedTable`, `SyncManifest`, +`SyncReceipt`, `SyncEvent`, `SyncState`, and `SyncError`. + +`SyncDirection` states which way one table replicates: `bidirectional`, +`pushOnly`, or `pullOnly`. Declaring direction per table matters because not +every table should travel both ways; a table that only a phone should ever +write, for example, can be marked `pushOnly` from the phone and `pullOnly` +everywhere else, and a backend that receives such a table simply skips it. + +`ConflictPolicy` states how a receiver resolves a row that changed on both +sides. `lastWriterWinsByHLC` is the default: it compares the Hybrid Logical +Clock, described in `OVERVIEW.md`, on the incoming record against the clock +on the stored row, and the more recent write survives. `appendOnly` treats +the table as a write-once audit log, where the row's own key already makes +duplicate delivery harmless, so incoming rows are always upserted rather +than compared. `localWins` and `remoteWins` are the two absolute policies, +useful when one side of a sync relationship is known in advance to be more +authoritative than the other. + +`SyncedTable.init(name:direction:primaryKeyColumn:conflictPolicy:)` bundles +one table's replication rules; `direction` and `conflictPolicy` default to +`bidirectional` and `lastWriterWinsByHLC` so a simple table declaration only +needs a name and a primary key column. `SyncManifest` is the full +declaration for one sync session: a `kitID` and `schemaVersion` that +identify the application and its data version, a `zoneIdentifier` naming +where remote data lives (a CloudKit zone name or a federation-facing label), +and the list of `SyncedTable` entries. `SyncManifest.table(named:)` looks up +one table's rules by name; every backend calls this before deciding whether +and how to apply an incoming or outgoing change. + +`SyncReceipt` summarizes one push or pull cycle: how many rows moved in each +direction, how many conflicts were detected, and when the cycle finished. +`SyncReceipt.empty` is the shared zero-value every backend returns when +there is nothing to do, so callers can treat "nothing happened" and "an +empty batch happened" identically. + +`SyncEvent` is what a live subscription emits: remote changes applied, a +push completed, a peer connected or disconnected, or an error. `SyncState` +is the coarser summary a settings screen would bind to: disabled, enabled +with a zone and last-activity timestamps, actively syncing in a direction, +or stuck in an error with an optional retry time. + +`SyncError` covers every failure a backend can report: not enabled yet, +already enabled, a schema or kit mismatch on an incoming record, a transport +failure, an encoding or decoding failure, an unreachable peer, a failed +authentication, an unsupported table, and `corruptRemoteIdentity`. That last +case exists for a specific reason explained where it is thrown, in +`CKRecordMapping.swift`: a remote record whose identifier cannot be read as +a valid row key must never be given a fabricated one, because a fabricated +identity would create a phantom row that never matches the real one on +later sync rounds. `SyncError` conforms to `Equatable` so tests can assert +exactly which error a call produced. + +## SyncEngine.swift + +This file provides `SyncEngine`, the protocol every backend conforms to. It +is the seam that lets application code, and the other two documents in this +package, describe "a backend" once instead of three times. + +The protocol has four methods and one property. `enable(manifest:storage:)` +must be called once before any other method; it is where a backend +establishes remote subscriptions, if any, and starts observing local +writes. `disable()` tears the same things down and must be safe to call more +than once. `push()` and `pull()` are the one-shot operations that move data +in each direction and return a `SyncReceipt`. `subscribe()` returns an +`AsyncStream` for callers that want a live feed rather than +polling. `state` is a computed, asynchronously readable property for +user-interface bindings that only need the coarse `SyncState` snapshot. + +Declaring `enable`/`push`/`pull` as `async throws` rather than synchronous +matters because every real backend performs network or disk work; declaring +the protocol this way once means every conforming backend, and every caller, +handles that uniformly. + +## SyncRecord.swift + +This file provides the wire format for one replicated row change: +`SyncRecord`, plus the smaller types it is built from. + +`SyncRecord` carries a table name, an event kind, a row key, the changed +column values (or `nil` for a delete), an HLC, a schema version, and a kit +identifier. It declares its `CodingKeys` explicitly rather than relying on +Swift's automatic key derivation, because the SDK's Rust port decodes the +exact same JSON, using serde field renames that must match these strings +character for character. An automatic default would still work today, but +would silently break the two ports' agreement the moment either side's +naming convention changed without the other noticing. + +`SyncEventKind` is a Codable, string-backed mirror of PersistenceKit's +`StorageEvent` (`insert`, `update`, `delete`). `SyncEventKind.init(from:)` +and `.asStorageEvent` convert both ways. A record's event kind is +duplicated here, rather than reusing `StorageEvent` directly, because +`StorageEvent` belongs to PersistenceKit and is not guaranteed to be a +stable wire format; `SyncEventKind` is ConvergenceKit's own promise about +what a byte on the wire means. + +`PackedHLC` is a Codable wrapper around `SubstrateTypes.HLC`, exposing the +same three integer fields (`physicalTime`, `logicalCount`, `nodeID`) +directly, because the raw `HLC` type is not itself built for a stable wire +encoding. + +`SyncValueMap` wraps `[String: TypedValue]`, PersistenceKit's row-value +type, as `[String: SyncValueBox]`. `TypedValue` covers thirteen kinds of +column value, several of which (an HLC, a fingerprint, a nested array) do +not map onto a single native JSON type, so `SyncValueBox` gives each value +an explicit `kind` string tag alongside its `payload`. This is an +adjacently-tagged encoding, matching Rust serde's +`#[serde(tag = "kind", content = "payload")]` attribute, chosen specifically +so a Rust decoder and a Swift decoder agree on the same JSON shape without +either language's default enum encoding leaking through. + +`SyncValueBox.encode(to:)` and `.init(from:)` implement that tagging by +hand. Two cases deserve attention. The `.null` case omits the `payload` key +entirely on encode, matching how Rust serde omits content for a unit +variant; a decoder that expected a `payload` key here would fail against a +Rust-produced null. The `.timestamp` case converts a `Date` to whole epoch +seconds as an `Int64`, and guards the conversion first: `Int64(_:)` traps +the program on a `Double` that is not finite or is outside `Int64`'s range, +and a corrupt or maliciously crafted inbound record could carry such a +value. The guard turns a potential crash into a normal, catchable +`EncodingError` instead. + +`FingerprintWire` is a Codable wrapper around `SubstrateTypes.Fingerprint256` +exposing its four 64-bit blocks directly, for the same reason `PackedHLC` +exists: the underlying substrate type is not itself declared Codable in a +form guaranteed stable across releases, so the wire format owns its own +copy of the shape. + +## ConvergenceKitNone.swift + +This file provides `NoSyncEngine`, the passthrough backend, and its private +`StateActor`. + +`NoSyncEngine` conforms to `SyncEngine` with the smallest possible bodies: +`enable(manifest:storage:)` records that sync is on and remembers the +manifest, only for the sake of reporting a `SyncState`; `push()` and +`pull()` each check that `enable` was called and then return +`SyncReceipt.empty`; `subscribe()` returns a stream that never emits +anything, closed only when the caller cancels its own task. There is +nothing here to synchronize against, so the state lives in `StateActor`, a +Swift actor, purely so that `enable`/`disable`/`isEnabled` reads and writes +are safe to call from concurrent tasks without ConvergenceKit's own locking +code. + +This backend exists so that development builds, unit tests, and genuinely +single-device deployments can enable the same `SyncManifest`-driven code +path as a production backend, without paying for a real remote connection +or writing separate no-sync branches throughout the application. + +## CKRecordMapping.swift + +This file provides `CKRecordMapping`, the generic translator between a +PersistenceKit row and Apple's `CKRecord`, and the two result types it +produces. + +The mapping is generic rather than hand-written per table because a +CloudKit record type and its fields are derived entirely from data already +known at runtime: the table's name and the manifest's `kitID`. +`recordType(kitID:table:)` builds the CloudKit record type string as +`"\(kitID)_\(table)"`. `recordID(rowKey:zone:)` builds a `CKRecord.ID` from +the row's UUID and the zone. Because both functions are pure string and +identifier arithmetic, adding a new synced table to a manifest never +requires new mapping code in this file. + +`record(from:table:rowKey:hlc:schemaVersion:kitID:zone:)` converts a row's +values into a `CKRecord`, then adds three reserved fields — +`_syncHLC`, `_syncSchemaVersion`, `_syncKitID` — that carry ConvergenceKit's +own metadata inside an otherwise ordinary CloudKit record. The private +`assign(value:to:forKey:)` helper switches over every `TypedValue` case; the +`.fingerprint` case packs the four 64-bit blocks into 32 bytes of +little-endian `Data`, and the `.array` case throws +`SyncError.encodingFailure`, because CKRecord has no native representation +for a nested list of typed values at this version. + +`decode(_:)` reverses the process. It reads the three reserved fields back +out, and rejects a record missing `_syncHLC` or `_syncSchemaVersion` as a +decoding failure, because a record without sync metadata cannot be +attributed to any schema version or conflict-resolved correctly. It then +recovers the original table name from the `kitID_table` record type by +splitting on the first underscore. The `rowKey` recovery is the one place +this file enforces a hard rule explained in its own comment: if +`record.recordID.recordName` does not parse as a `UUID`, the function throws +`SyncError.corruptRemoteIdentity` rather than inventing a fresh UUID. A +fabricated identity would silently create a new local row every sync round, +because the fabricated UUID could never match the real one; refusing to +guess turns a slow, invisible data leak into a single visible error the +caller can log and skip. + +`packed(_:)` and `unpacked(_:)` convert an `HLC` to and from a single +`Int64` for storage in a CloudKit `NSNumber` field, using a fixed bit +layout: 48 bits of physical time, 12 bits of logical count, 4 bits of node +identifier. The layout is a contract, not an implementation detail — any +change to the bit widths would make previously stored `_syncHLC` values +unreadable. + +`SyncMeta` and `DecodedRecord` are the two small result types `decode(_:)` +returns: `SyncMeta` isolates the three reserved fields, and `DecodedRecord` +pairs them with the table name, row key, and clean application values (no +`_sync*` keys). Keeping the reserved fields out of `values` matters because +`values` round-trips directly into PersistenceKit's own row-value type, +which should never see ConvergenceKit's internal bookkeeping fields mixed +in with application data. + +## CloudKitSyncEngine.swift + +This file provides `CloudKitSyncEngine`, the CloudKit-backed `SyncEngine` +conformance, and its private `CloudKitStateActor`. + +`CloudKitSyncEngine.init(containerIdentifier:)` accepts an optional CloudKit +container identifier, or `nil` to resolve `CKContainer.default()` later. +Resolution is deliberately deferred to `enable()` rather than done in +`init`, specifically so a test can construct the engine without an iCloud +entitlement configured; only calling `enable()`, `push()`, or `pull()` +actually touches CloudKit. + +`CloudKitStateActor.enable(manifest:storage:)` creates the estate's CloudKit +zone (tolerating a "zone already exists" failure as expected, not an error) +and then, for every manifest table that is not `pullOnly`, subscribes to +`storage.observer.observe(table:events:)` and forwards every observed +`TableChange` into `pendingOutbound`. This is the mechanism, described in +`OVERVIEW.md`, that turns an ordinary PersistenceKit write into a queued +outbound sync record without the application calling anything sync-specific. + +`push()` drains `pendingOutbound`, and for each change looks up the table's +`SyncedTable` in the manifest, skipping tables the manifest does not +declare or has marked `pullOnly`. For an insert or update it builds a +`CKRecord` via `CKRecordMapping.record`; for a delete it builds only the +`CKRecord.ID` to delete. The HLC used per record prefers one the change +already carries; if the observation carried none — true of the InMemory and +SQLite storage observers at this version — `push()` mints one from an +`HLCGenerator` seeded with a random node ID in `1...0x0F`, using +`send(now:)` rather than reading the clock without advancing it, so two +changes minted in the same push never collide on an identical timestamp. +The accumulated saves and deletes are sent together in one +`modifyRecords(saving:deleting:savePolicy:atomically:)` call. + +`pull()` fetches everything new since the last stored `serverChangeToken` +via `recordZoneChanges(inZoneWith:since:)`, then applies each pulled record +through `applyInbound`. Three checks gate every record before it is +applied: the decoded `kitID` must match the manifest's, the decoded +`schemaVersion` must match, and the table must be declared in the manifest +and not marked `pushOnly`. Any of the three failing, or a decode error, is +caught, logged, and counted as a conflict rather than aborting the whole +pull — one bad record must not stop the rest of the batch from applying. +Deletions arrive as bare `CKRecord.ID` values with no record type attached, +so `pull()` cannot know which table a deleted record belonged to; it +attempts the delete against every manifest table that is not `pushOnly`, +using the manifest itself as the only scope guard. + +`applyInbound(_:syncedTable:storage:)` is the method that actually enforces +`ConflictPolicy`. `.appendOnly` always upserts, relying on the row's own key +for idempotency. `.lastWriterWinsByHLC` reads back any existing row's stored +`_syncHLC` — which arrives as `.hlc` under the InMemory backend or as a raw +packed `.int` under SQLite and Postgres, because those schemas do not +declare the column's `TypedValue` kind — and silently discards the incoming +change if its HLC is older. A change that survives the comparison is +upserted with its own `_syncHLC`, `_syncSchemaVersion`, and `_syncKitID` +merged back into the row, which is what lets the *next* inbound write read +a comparison point back out. `.remoteWins` upserts unconditionally. +`.localWins` inserts only if no row with that primary key exists yet. The +method is declared with internal, not `private`, access specifically so LWW +(last-writer-wins) conflict tests can call it directly through +`@testable import`, bypassing the real CloudKit network stack. + +## FederationIdentity.swift + +This file provides the identity types federation pairing and signing are +built from: `PeerIdentity`, `LocalIdentity`, and `FederationSignature`. + +`PeerIdentity` is a thin wrapper around a 32-byte Ed25519 public key, +carried for a paired peer. `LocalIdentity.init()` generates a fresh Ed25519 +keypair in memory using `Curve25519.Signing.PrivateKey()`; every +`FederationSyncEngine` instance calls this once, at construction, so each +estate has its own persistent-for-the-session signing identity. +`LocalIdentity.init(privateKeyBytes:)` restores an identity from bytes a +caller already has, for a host application that wants to persist and reload +the estate's key across launches; this module deliberately does not persist +identity itself, leaving that decision to the host. `sign(_:)` produces an +Ed25519 signature over arbitrary data using the stored private key. + +`FederationSignature.verify(_:of:by:)` checks a signature against a raw +32-byte public key. It returns `false`, rather than throwing, both when the +public key bytes do not form a valid Curve25519 key and when the signature +itself does not match — a receiver treats a malformed peer key exactly like +a bad signature, because both mean the message cannot be trusted. + +## HyperplaneFamilyExchange.swift + +This file provides three Codable value types for the pairing handshake: +`HyperplaneFamilySpec`, `PairingProposal`, and `PairingAcceptance`. + +`HyperplaneFamilySpec` carries a `seed` and a `dimension` (default 256). +Two estates that share the same seed can each independently reproduce the +same hyperplane family — the substrate machinery, defined elsewhere, that +makes their 256-bit fingerprints directly comparable. Sharing only a seed, +rather than the derived family itself, keeps the pairing message small and +keeps the two sides' math guaranteed identical, because both derive it with +the same deterministic procedure from the same seed. + +`PairingProposal` and `PairingAcceptance` model a proposer offering a family +spec plus a nonce (a value used once, to prevent a replay of an old +proposal), and an accepter countersigning that proposal. This file defines +only the shapes; as `FederationSyncEngine.swift` documents at its own +`pair(with:via:family:)` method, the shipped v1.0 pairing path does not yet +negotiate or sign these two structs — the caller supplies an +already-agreed `HyperplaneFamilySpec` directly. These types exist ahead of +that wiring so the wire shape is fixed and ready. + +## FederationSyncEngine.swift + +This file provides `FederationSyncEngine`, the Ed25519-authenticated +peer-to-peer `SyncEngine` backend; the envelope format its messages travel +in; the `Relay` transport abstraction and its in-process implementation; +and `FederationStateActor`, where the actual protocol logic lives. + +`PayloadKind` is a one-byte discriminator identifying what an envelope +carries. `.syncRecordBatch` (`0x01`) is the only kind that exists at v1.0; +`.fieldWriteEventBatch` (`0x02`) is reserved in a code comment for a future +payload and must never be reassigned to anything else, because a receiver +on an older version would silently misinterpret a reused byte value. + +`envelopeSigningBytes(senderPublicKey:payloadKind:payload:hlc:)` builds the +exact byte sequence a `SignedEnvelope`'s signature covers: the sender's +32-byte public key, the one-byte payload kind, a four-byte little-endian +payload length, the payload itself, and the HLC's three fields, each in a +fixed little-endian layout. Signing this constructed sequence, rather than +the raw JSON payload, closes what the code calls the relabel/replay seam: if +the signature only covered the payload bytes, an attacker who obtained a +valid signed payload could re-wrap it with a different sender key or +payload kind and have it re-verified as something it never was signed to +mean. The exact byte layout matters because the Rust port's +`envelope_signing_bytes` function must reproduce these bytes exactly, or a +signature made on one port fails verification on the other. + +`SignedEnvelope` is the wire message this signature protects: the sender's +public key, the payload kind, the opaque payload, the signature, and a +batch-level HLC. The batch HLC is minted after every record's own HLC, in +`push()`, so it is strictly later — a reader can trust the envelope's HLC as +a batch watermark distinct from any single record's HLC inside it. + +`Relay` is a two-method protocol — `send(to:message:)` and +`drain(for:)` — that abstracts the transport a `SignedEnvelope` travels +over. `FederationRelay` is the only conformer shipped at v1.0: an in-process, +lock-protected dictionary from a recipient's public key to a list of pending +envelopes. It exists so the full pairing, signing, and conflict-resolution +protocol can be exercised in tests without a real network, and so a future +hosted relay (a `SyncServer`) can conform to the same protocol as a drop-in +replacement without touching `FederationSyncEngine` itself. + +`FederationStateActor` is where enable, pairing, push, and pull are +implemented, as a Swift actor so its mutable state — peers, pending +outbound changes, subscribers — is safe under concurrent access without +manual locking. `enable(manifest:storage:)` mirrors +`CloudKitStateActor.enable`: it subscribes to `storage.observer.observe` for +every push-eligible table and appends observed changes to +`pendingOutbound`. `disable()` differs from the CloudKit actor in one +deliberate way: it cancels every observer task and then *awaits* each one's +completion before clearing state, rather than only cancelling. The code +comment explains why — without awaiting, a change already in flight inside +a cancelled task could still land in the outbox after `disable()` returned, +which would violate the guarantee that disabling sync stops capturing +writes immediately. + +`pair(with:via:family:)` registers the peer's public key, relay, and family +spec locally, then calls `acceptPeering` on the peer's own actor so the +relationship is recorded symmetrically on both sides from one call — a test +or a caller need not call `pair` twice. Pairing here is a local bookkeeping +step, not yet the signed `PairingProposal`/`PairingAcceptance` handshake +`HyperplaneFamilyExchange.swift` defines the shapes for. + +`push()` converts every pending `TableChange` into a `SyncRecord` (minting +an HLC the same way `CloudKitStateActor.push()` does, and for the same +reason), JSON-encodes the batch, builds and signs an envelope, and hands it +to every paired peer's relay. `pull()` drains each peer's relay inbox and, +for every envelope, runs four checks in order before trusting its contents: +the sender's public key must match the specific peer it arrived from (not +merely any known peer — the code comment cites ADR-013, since a valid +self-signature alone does not prove the pairing handshake was completed); +the payload kind must be a known one; the signature must verify over the +reconstructed canonical bytes; and only then is the JSON payload decoded. +A record that passes all four still individually re-checks its `kitID` and +`schemaVersion` against the manifest and its table against the manifest's +declared tables, exactly like `CloudKitSyncEngine.pull()`, before reaching +`applyInbound`. + +`applyInbound(_:syncedTable:storage:)` implements the same four +`ConflictPolicy` arms as `CloudKitStateActor.applyInbound`, extended to +cover delete events explicitly (`CloudKitSyncEngine` receives deletes as +bare CloudKit record IDs with no policy dispatch; federation's `SyncRecord` +always carries an explicit event kind, so its delete path can apply a +policy the same way inserts and updates do). Under +`.lastWriterWinsByHLC`, a stale delete — one whose HLC is older than the +row's stored `_syncHLC` — is silently rejected rather than removing a row a +peer has since updated; `.appendOnly` rejects every remote delete +outright, because an append-only table is write-once by definition; +`.remoteWins` deletes unconditionally; `.localWins` rejects every remote +delete, leaving local state authoritative. The method is internal, not +`private`, for the same testing reason as its CloudKit counterpart. + +## Rust Port and Conformance + +The `rust/` directory contains a second implementation of most of this +package: the core types, the `SyncRecord` wire format, the `SyncEngine` +trait, and both the None and Federation backends, in `rust/src/types.rs`, +`record.rs`, `engine.rs`, `none.rs`, `federation.rs`, and `pairing.rs`. +CloudKit is Apple-only and has no Rust equivalent by design; the Swift +package alone handles that transport, as `rust/README.md` and +`rust/src/lib.rs` both state explicitly. + +The Rust and Swift Federation backends are gated by shared behavior rather +than a byte-fixture file the way some other SDK packages are: both ports +implement observer-driven outbox population, the same four `ConflictPolicy` +arms in `apply_record`/`applyInbound`, and identical Ed25519 envelope +signing bytes in `envelope_signing_bytes`/`envelopeSigningBytes`. The Rust +test suite (`rust/tests/`) exercises wire-format round trips, last-writer- +wins ordering, and inbound event routing per conflict policy, each +described in `rust/README.md` as mirroring a named Swift test file. When you +change conflict resolution, the envelope signing byte layout, or a +`SyncRecord` field name in either port, update both and re-run both test +suites — the JSON field names and the signing byte layout are the contract +between them. diff --git a/packages/kits/ConvergenceKit/docs/INTERFACE_DOCTRINE.md b/packages/kits/ConvergenceKit/docs/INTERFACE_DOCTRINE.md deleted file mode 100644 index 0a3d9e0..0000000 --- a/packages/kits/ConvergenceKit/docs/INTERFACE_DOCTRINE.md +++ /dev/null @@ -1,115 +0,0 @@ -# ConvergenceKit Interface Doctrine - -For coding agents implementing kits that enable sync on their PersistenceKit instance. ConvergenceKit's contract is with PersistenceKit; downstream kits do not call ConvergenceKit directly except to declare and enable it. - -## 1. Sync is enabled at the application layer, not the kit - -The kit does not own its sync configuration. The application that composes the kit graph decides whether to sync, which backend, and which zone identifier. Kits declare the manifest of what *would* sync if sync were enabled; the application picks a SyncEngine and calls `enable`. - -```swift -// In the kit -public extension LocusKit { - public static func syncManifest(estateID: UUID) -> SyncManifest { - SyncManifest( - kitID: "LocusKit", - schemaVersion: 1, - zoneIdentifier: "LocusKit-\(estateID.uuidString)", - tables: [ - SyncedTable(name: "drawers", primaryKeyColumn: "row_id"), - SyncedTable(name: "tunnels", primaryKeyColumn: "tunnel_id"), - SyncedTable(name: "audit_events", primaryKeyColumn: "event_id", - conflictPolicy: .appendOnly) - ] - ) - } -} - -// In the application -let locusKit = LocusKit(storage: storage) -let sync = CloudKitSyncEngine() -try await sync.enable(manifest: LocusKit.syncManifest(estateID: estate.id), - storage: storage) -``` - -The kit never wires its own sync. The application does. - -## 2. Choose ConflictPolicy per table, not per kit - -Different tables have different conflict semantics. The audit log is append-only (`.appendOnly`); substrate noun tables project from the audit log and tolerate `.lastWriterWinsByHLC`. Queue jobs claim by HLC; settings might be `.localWins`. Per-table choice is the kit author's responsibility. - -```swift -SyncedTable(name: "audit_events", primaryKeyColumn: "event_id", - conflictPolicy: .appendOnly) -SyncedTable(name: "settings", primaryKeyColumn: "key", - conflictPolicy: .localWins) -SyncedTable(name: "jobs", primaryKeyColumn: "job_id", - conflictPolicy: .lastWriterWinsByHLC) -``` - -## 3. Sync direction is declarative - -Most tables are `.bidirectional`. Push-only (`.pushOnly`) is for tables a device emits but never receives (logs, telemetry). Pull-only (`.pullOnly`) is for tables a device consumes but never writes (read-only state from a central source). The kit declares; the application doesn't override per-call. - -## 4. Schema version must match across peers - -If two devices run different schema versions of the same kit, sync between them rejects with `SyncError.schemaMismatch`. The receiver queues the record (or drops it, depending on backend) until both sides upgrade. Document this for the application that orchestrates updates. - -Schema version is bumped whenever PersistenceKit's `SchemaDeclaration.version` is bumped. The two numbers must agree. - -## 5. The audit log uses appendOnly - -The audit log table that GeniusLocusKit owns has `(event_id, hlc)` as its compound primary key. ConvergenceKit's `.appendOnly` policy maps to an idempotent upsert on this key. Duplicate appends from sync replay are no-ops at the storage layer; the CRDT property holds. - -Other kits with append-only logs (queue audit, federation handshake history) use the same policy. - -## 6. The application catches SyncError - -ConvergenceKit operations throw `SyncError` for backend-attributable failures. The application decides retry policy. The kit does not see these errors unless the application surfaces them. - -```swift -do { - _ = try await sync.push() -} catch SyncError.transportFailure(let detail) { - // queue for retry; show offline indicator -} catch SyncError.schemaMismatch { - // prompt user to update the app -} -``` - -## 7. Subscribe for live updates, push/pull for one-shots - -`subscribe()` returns a long-running stream that fires events as sync activity happens. Use it for UI bindings ("syncing now," "last synced 30 seconds ago") and for waking up watchers that need notification when remote work arrives. - -`push()` and `pull()` are one-shots. Use them for explicit refresh ("pull on app foreground," "push before close") or in test paths where determinism matters. - -The two patterns coexist; subscribe stays open while push/pull run. - -## 8. CloudKit and Federation can run side by side - -A single PersistenceKit instance can have both a ConvergenceKit-CloudKit engine and a ConvergenceKit-Federation engine enabled simultaneously. The manifests pick different zones / peer sets; the engines observe the same StorageObserver independently. Multi-backend deployments (an app syncs its entities via CloudKit between a user's devices AND federates a GeniusLocus estate with a partner via ConvergenceKit-Federation) are natively supported. - -The cost is two observer subscriptions per table. Document the resource use in the application. - -## 9. Federation pairing is out of band - -Pairing two estates over Federation requires exchanging public keys plus the HyperplaneFamilySpec. At v1.0 this is in-process (`engineA.pair(with: engineB, via: relay, family:)`). Cross-machine pairing is a v1.x concern; expect QR code, NFC, or AirDrop as the eventual out-of-band channel. - -The kit does not initiate pairing. The application orchestrates it (pairing flow in the UI), and the kit's sync manifest just declares what flows once pairing is in place. - -## 10. Test against ConvergenceKitNone in CI - -CI test suites for kits with sync declarations use ConvergenceKitNone unless the test specifically exercises sync semantics. ConvergenceKitNone validates the manifest, succeeds on enable/disable, and returns empty receipts on push/pull. Fast and deterministic. - -For sync-specific tests use ConvergenceKitFederation in-process pairing (two engines, shared relay, real round-trip). For full CloudKit integration tests, gate on `CLOUDKIT_TEST_CONTAINER` and provision a test container per the project's CI setup. - -## 11. When in doubt, file a decision record - -If you find yourself wanting to: - -- Add a case to `SyncDirection`, `ConflictPolicy`, `SyncEvent`, `SyncState`, or `SyncError` -- Add a new method to `SyncEngine` -- Bypass schema checking -- Sync something that isn't a PersistenceKit row (large blob, file, stream) -- Add a backend-specific escape hatch - -Stop. Write a decision record in `docs/decisions/` proposing the change. The closed-enum design depends on every change being deliberate. diff --git a/packages/kits/ConvergenceKit/docs/OVERVIEW.md b/packages/kits/ConvergenceKit/docs/OVERVIEW.md new file mode 100644 index 0000000..d16e2da --- /dev/null +++ b/packages/kits/ConvergenceKit/docs/OVERVIEW.md @@ -0,0 +1,163 @@ +--- +doc: OVERVIEW +package: ConvergenceKit +repo: moot-system +authored_commit: 909513d0a8ecb1e9e903af9f4d25b3e4f2528242 +authored_date: 2026-07-04 +sources: + - path: Sources/ConvergenceKit/SyncEngine.swift + blob: 32e69e0ea5002d8b699866183371527f675023cd + - path: Sources/ConvergenceKit/SyncRecord.swift + blob: 8c29f63ea9f965f1ea75d3bbff71632f76699233 + - path: Sources/ConvergenceKit/SyncTypes.swift + blob: 52ed3cb07f0483d0e541b86a8da080791402d132 + - path: Sources/ConvergenceKitCloudKit/CKRecordMapping.swift + blob: 9b61903296533f8044bee5a4f93ca2dc1b76b653 + - path: Sources/ConvergenceKitCloudKit/CloudKitSyncEngine.swift + blob: 7dd43ccf5454739a3d1ae1cda0870cf507c479c6 + - path: Sources/ConvergenceKitFederation/FederationIdentity.swift + blob: b64358fe36fd241049e9f5bdbbb335f1f3d29464 + - path: Sources/ConvergenceKitFederation/FederationSyncEngine.swift + blob: 9bf722f8e7fc24ae12ac72d571c135da2b40af62 + - path: Sources/ConvergenceKitFederation/HyperplaneFamilyExchange.swift + blob: 4880e847918c253f3d99a02fdd65c15f172d32e1 + - path: Sources/ConvergenceKitNone/ConvergenceKitNone.swift + blob: 5d3556a08c8bc80ec6b824ceb4e0838db9c95ecc +--- + +# ConvergenceKit Overview + +## What This Library Does + +ConvergenceKit copies the rows that PersistenceKit stores from one device or +one MOOTx01 estate to another. An estate is one user's complete memory store +in MOOTx01; estates can federate, which means separate estates share and +compare memories. PersistenceKit is the package that reads and writes rows on +one device. ConvergenceKit sits beside it and watches for changes, then ships +those changes somewhere else: another of the user's own devices through +Apple's CloudKit, or a different user's estate through a peer-to-peer +exchange the SDK calls federation. + +Application code never calls ConvergenceKit's sync methods directly for +ordinary reads and writes. It writes to PersistenceKit as usual. +ConvergenceKit is configured once, alongside PersistenceKit, to observe that +same storage. From that point on, replication is a side effect of normal +writes rather than a task the application must remember to perform. + +## The Problem It Solves + +A memory captured on one device is useless on another device until it +travels there. Two hard problems stand between "captured here" and +"visible there." + +The first problem is conflicting writes. If the same row changes on two +devices before they synchronize, something must decide which change wins. +ConvergenceKit answers this with a per-table conflict policy declared once, +in advance, by the application. The default policy, `lastWriterWinsByHLC`, +settles conflicts using a Hybrid Logical Clock (HLC): a timestamp that +combines a device's wall-clock reading with a small counter, so that two +devices can order their changes consistently even when their clocks are not +perfectly synchronized. Three other policies exist for cases where +last-writer-wins is wrong: `appendOnly` for audit logs that must never +overwrite an entry, `localWins` for data a device should never let a peer +overwrite, and `remoteWins` for the opposite case. + +The second problem is trust. Federated sync crosses a perimeter between two +separate people's estates, so a receiving device must be able to prove that +an incoming batch of changes really came from the peer it paired with, and +was not altered in transit. ConvergenceKit answers this with Ed25519 +signatures. Ed25519 is a public-key signature scheme: each estate generates +a private key it never shares and a public key it hands to peers during +pairing. Every batch of changes is signed with the sender's private key +before it leaves the device, and the receiver checks the signature with the +sender's public key before applying anything. + +CloudKit sync does not need this second protection, because it stays inside +one person's Apple account, authenticated by Apple's own private database. +Federated sync crosses to a stranger's device, so it needs its own +authentication independent of any platform account system. + +## How It Works + +ConvergenceKit defines one protocol, `SyncEngine`, and three backends that +conform to it: `NoSyncEngine`, `CloudKitSyncEngine`, and +`FederationSyncEngine`. An application picks exactly one backend per +PersistenceKit instance. All three backends expose the same four operations, +so application code that calls them does not need to know which backend is +active. + +`enable(manifest:storage:)` turns sync on. A `SyncManifest` is a declaration, +written once by the application, of which PersistenceKit tables to sync, in +which direction (`bidirectional`, `pushOnly`, or `pullOnly`), and under which +conflict policy. Enabling a backend also starts it watching PersistenceKit's +`StorageObserver` for local inserts, updates, and deletes on every table the +manifest lists as push-eligible. + +`push()` sends pending local changes outward and returns a `SyncReceipt` +summarizing what moved. `pull()` fetches pending remote changes and applies +them through PersistenceKit's own write path, which is what makes the +receiving side work correctly: an applied row is not a special sync +artifact, it is an ordinary row, so anything already watching PersistenceKit +notices the change the normal way. `subscribe()` returns a live stream of +`SyncEvent` values (changes applied, pushes completed, peers connecting or +disconnecting, errors) for a user interface to display. + +Every backend that crosses a real network boundary needs a wire format for a +changed row. `SyncRecord` is that format: a table name, an event kind, a row +key, the changed column values, an HLC, a schema version, and a kit +identifier. The schema version and kit identifier exist so a receiver can +reject a record it does not understand, rather than misapply it. `SyncRecord` +is Codable with explicit coding keys, because the SDK maintains a Rust port +of large parts of the system, and both ports must produce the same JSON +field names for the same data. + +## How the Pieces Fit + +Figure 1 shows the library's topology: the shared protocol and wire format +at the center, PersistenceKit on either end, and the three backends as +interchangeable implementations in between. + +![Figure 1. Topology of ConvergenceKit](topology.svg) + +*Figure 1. Topology of ConvergenceKit. Application code writes only to +PersistenceKit. Whichever backend is enabled observes those writes and ships +them outward; on the receiving side, applying an inbound change routes back +through PersistenceKit's normal write path. Dashed regions mark the three +interchangeable backends and the external systems (CloudKit's private +database, the federation relay) each one talks to.* + +`NoSyncEngine` is the simplest backend: it accepts `enable()`, then returns +empty receipts from `push()` and `pull()` forever. It exists for +single-device deployments, development, and tests, where wiring in a real +backend would add cost without adding value. + +`CloudKitSyncEngine` replicates rows through Apple's CloudKit private +database, one CloudKit zone per estate. `CKRecordMapping` is the piece that +makes this backend generic rather than hand-written per table: it converts +any PersistenceKit row into a `CKRecord` and back, driven entirely by the +table name and the manifest, so adding a new synced table never requires new +mapping code. + +`FederationSyncEngine` replicates rows to a different user's estate. Two +engines "pair" by exchanging Ed25519 public keys and a shared +`HyperplaneFamilySpec`, a parameter that lets both sides compare certain +256-bit fingerprints (short fixed-size codes computed from content) on equal +terms after pairing. Once paired, every pushed batch of `SyncRecord` values +is wrapped in a `SignedEnvelope`, signed, and handed to a `Relay`. At v1.0, +the shipped `Relay` implementation, `FederationRelay`, is in-process only, +holding pending messages in memory for peers running in the same process. +This is enough to exercise the full protocol, including signature +verification and conflict resolution, in tests; a wire transport that +reaches a peer on a different machine is future work. + +## What Ships in the Package + +The package ships four Swift targets: `ConvergenceKit` (the protocol, the +wire format, and the shared types), `ConvergenceKitNone`, +`ConvergenceKitCloudKit`, and `ConvergenceKitFederation`. A `rust/` port +mirrors the core types, the wire format, and the None and Federation +backends; CloudKit is Apple-only and has no Rust equivalent by design, so +the Swift side always handles that transport. Every field name shared +between the two ports is pinned in code comments and exercised by tests on +both sides, because a receiver on one port must decode a record written by +the other exactly. diff --git a/packages/kits/ConvergenceKit/docs/topology.svg b/packages/kits/ConvergenceKit/docs/topology.svg new file mode 100644 index 0000000..1f65269 --- /dev/null +++ b/packages/kits/ConvergenceKit/docs/topology.svg @@ -0,0 +1,113 @@ + + + + + + + + + + + + + ConvergenceKit: PersistenceKit rows in, replicated rows out + + + + Application code + writes via PersistenceKit + + + PersistenceKit (local) + Storage · RowStore · StorageObserver + + + SyncEngine + enable · push · pull · subscribe + + + Remote + CloudKit database or peer's PersistenceKit + + + + + + push + + pull + + + + apply inbound via RowStore -> wakes local watchers + + + + Backends - one enabled per PersistenceKit instance; each conforms to SyncEngine + + + ConvergenceKitNone + passthrough; empty receipts + + + ConvergenceKitCloudKit + CKRecordMapping <-> CKRecord + + + ConvergenceKitFederation + Ed25519 SignedEnvelope + + + + + + + + + External systems (v1.0 transport) + + + CloudKit private database + Apple iCloud, per-estate zone + + + FederationRelay + in-process inbox (v1.0) + + + + + + + Cross-machine relay transport and the signed pairing handshake (HyperplaneFamilyExchange) are v1.x scope. + diff --git a/packages/kits/PersistenceKit/docs/AGENT_MAP.md b/packages/kits/PersistenceKit/docs/AGENT_MAP.md new file mode 100644 index 0000000..f27e7b1 --- /dev/null +++ b/packages/kits/PersistenceKit/docs/AGENT_MAP.md @@ -0,0 +1,363 @@ +--- +doc: AGENT_MAP +package: PersistenceKit +repo: moot-system +authored_commit: 909513d0a8ecb1e9e903af9f4d25b3e4f2528242 +authored_date: 2026-07-04 +sources: + - path: Sources/PersistenceKit/AuditLog.swift + blob: ca4c0c9623056a49ad888a7a77e720a7519556bb + - path: Sources/PersistenceKit/BlobStore.swift + blob: f052c3ecc693d5414d57ad2c6da5fb4c5fe28f79 + - path: Sources/PersistenceKit/CacheInvalidator.swift + blob: 0a844b8037404dd72d08df6887ceee1e8c6014f2 + - path: Sources/PersistenceKit/CachingRowStore.swift + blob: 7edf9a86eb31dd9a17abd97b7baa9e8d5425e266 + - path: Sources/PersistenceKit/Column.swift + blob: 3cfc3ba856ebe15e8756ae65dae736cbd2c288a6 + - path: Sources/PersistenceKit/EncryptionMode.swift + blob: fdb7aa7ec63088ae09739f5e2a8e1468c386b987 + - path: Sources/PersistenceKit/ErasureLedger.swift + blob: eb2ca35f69783eead6049949bbaad68decfc9915 + - path: Sources/PersistenceKit/ErasureOverlay.swift + blob: d5a86f2baa10647134f285d6996a8e62cbe23ace + - path: Sources/PersistenceKit/EstateCacheConfig.swift + blob: 7003951cb5b2b075b6bb0c49b5fe372a97bfeb3a + - path: Sources/PersistenceKit/EstateConfiguration.swift + blob: be91569405e5c48a99859b4595540a59bd1a1994 + - path: Sources/PersistenceKit/GCPin.swift + blob: 73336dc4c693fcfa285a7eff8ed6520f27951ff8 + - path: Sources/PersistenceKit/GeneratedColumn.swift + blob: 34e466c80a0e9aac5d24303eaad132901333470b + - path: Sources/PersistenceKit/HashingRowStore.swift + blob: fe233ed2f4fda177373ad105d14fa321d325d0df + - path: Sources/PersistenceKit/NoOpObserver.swift + blob: 99f62b6889df50c5f09fce749f363637891f116c + - path: Sources/PersistenceKit/NovelTokenTaggerChoice.swift + blob: 9588d6303431883a2c8e0ee727f7ebe3e3f6d139 + - path: Sources/PersistenceKit/PersistenceKitTelemetry.swift + blob: a24e54dfb97beb86dac9422bbfc7cc3c2d959d73 + - path: Sources/PersistenceKit/Predicate.swift + blob: 522802c842dfc7573137efd7a3f36300d5201468 + - path: Sources/PersistenceKit/RowCrypto.swift + blob: 6678e426c0902cdae6d164904001526756e40916 + - path: Sources/PersistenceKit/RowStore.swift + blob: 9a96ff89528ca786d51224324cacb78cf810037d + - path: Sources/PersistenceKit/Schema.swift + blob: 5a6894c68d6f29eecfe9acc34d7c06ee2a655ac3 + - path: Sources/PersistenceKit/SnapshotRegistry.swift + blob: 88087bda544165c4d24c514a4f3e0641a620512e + - path: Sources/PersistenceKit/Storage.swift + blob: 7484a40913b28cb11a6bd2e3ea822dc8fe8eb63e + - path: Sources/PersistenceKit/StorageError.swift + blob: 743c2d1a24c7bafedb217e4c6bbf30ca20d03be8 + - path: Sources/PersistenceKit/StorageIntrospection.swift + blob: 35522b6601037246907bb88ebb2fb2eb5ea9b0e2 + - path: Sources/PersistenceKit/StorageObserver.swift + blob: f0a8b61a15344e02193137a3393be345dd51cf25 + - path: Sources/PersistenceKit/Transaction.swift + blob: d58d478618b31cc3985275be6ceee84a6fc222de + - path: Sources/PersistenceKit/TypedValue.swift + blob: ec124b7aebd86f64f67fea6656396d374666b741 + - path: Sources/PersistenceKitInMemory/InMemoryAuditLog.swift + blob: 1ba83d2408d3384d4d8e4f286fa2db743a433cc9 + - path: Sources/PersistenceKitInMemory/InMemoryBlobStore.swift + blob: e02822bde83c899b6d18ab8d82b90afba6abb5ac + - path: Sources/PersistenceKitInMemory/InMemoryObserver.swift + blob: 71b98c9da2d869a81a78265b7ef1f64a57a907af + - path: Sources/PersistenceKitInMemory/InMemoryRowStore.swift + blob: 2f7c92612c46b8a097e4211147c924f8356e47d1 + - path: Sources/PersistenceKitInMemory/InMemoryStorage.swift + blob: 9820c771bfe39ab738eea1f3b1623491fcfb1326 + - path: Sources/PersistenceKitInMemory/PredicateEvaluator.swift + blob: e8735422ff87addff6a2a3a89da85c495d07f07c + - path: Sources/PersistenceKitPostgreSQL/PostgreSQLConnection.swift + blob: 5a282a6064a69a543a5d650ccc7eeeae2c5a3e4f + - path: Sources/PersistenceKitPostgreSQL/PostgreSQLIdentifierValidator.swift + blob: 120cf0c1576a7db45d79bf6865e2f15cff09a5ac + - path: Sources/PersistenceKitPostgreSQL/PostgreSQLPool.swift + blob: eddc0c3af4135565e15d2d763b0d24240973dd6f + - path: Sources/PersistenceKitPostgreSQL/PostgreSQLPredicateCompiler.swift + blob: 6f0791e0372b09cf2605bae4cf149e48bbba2834 + - path: Sources/PersistenceKitPostgreSQL/PostgreSQLSchema.swift + blob: f165f0877b96c87e2f7de46012f8f8acb3c03cc8 + - path: Sources/PersistenceKitPostgreSQL/PostgreSQLStorage.swift + blob: 849926f9e6def788fae2f38757ad44bcd52a18a0 + - path: Sources/PersistenceKitPostgreSQL/PostgreSQLStores.swift + blob: bff90838d1324d82ae76a0895f15399537e9e343 + - path: Sources/PersistenceKitReplication/IncrementalReplicationSession.swift + blob: 2f90378146e043d75c739d400d98c7770e07af78 + - path: Sources/PersistenceKitReplication/ReplicationTypes.swift + blob: bb13c63d1febd50ad46c1814610d7f6c31a33112 + - path: Sources/PersistenceKitReplication/StorageReplicator.swift + blob: f5acc5993c8647e53c127c6871a1282c24b1c427 + - path: Sources/PersistenceKitSQLite/KeychainKeyStore.swift + blob: 0071732291a7cb6ce0777bd230a6188276fb4f32 + - path: Sources/PersistenceKitSQLite/SQLiteConnection.swift + blob: ece56dc7e25e67656bb37f5222c18c1166c750cc + - path: Sources/PersistenceKitSQLite/SQLiteIdentifierValidator.swift + blob: 713339c137d6af1cfbba5a3584e05bdda70b42c5 + - path: Sources/PersistenceKitSQLite/SQLiteObserver.swift + blob: 2cb61ed75dae5ab81a3cdfb5c9581731d25fd960 + - path: Sources/PersistenceKitSQLite/SQLitePredicateCompiler.swift + blob: 5770adb2024c54ea6921651632ca65ee84d416af + - path: Sources/PersistenceKitSQLite/SQLiteSchema.swift + blob: 4ba6fc175fe9d486b17af1088a23da64041b12ae + - path: Sources/PersistenceKitSQLite/SQLiteStorage.swift + blob: 417e63cc07b60295ac874187ebb796bf9f3b3c86 + - path: Sources/PersistenceKitSQLite/SQLiteStores.swift + blob: 76499ffb70d979f0d13e8cf9e32bc38ff28ffdb5 +--- + +# AGENT_MAP — PersistenceKit + +PURPOSE: storage abstraction layer for MOOTx01 estates. One protocol +surface (Storage = RowStore+BlobStore+AuditLog+StorageObserver), three +interchangeable backends (SQLite/PostgreSQL/InMemory), decorators +(caching, hash-on-write) and cross-cutting toolkits (row encryption, +erasure ledger/overlay, snapshot registry+GC pin, telemetry) built on the +core protocol, plus a full-estate replication module. Owns NO vector +search (VectorKit owns k-NN; PersistenceKit only guarantees the +ACCOMMODATION contract — storage round-trip for vector payloads). + +DEPS: imports SubstrateTypes (HLC, Fingerprint256, AuditEvent, +AsOfCoordinate, ContentHash — defined upstream, NOT in this package), +IntellectusLib (telemetry, zero-cost when disabled), CryptoKit +(AES-GCM), Security/Keychain (Apple key store, SQLite target only), +postgres-nio + NIOSSL (PostgreSQL target only), vendored SQLCipher C +target (SQLite target only). Imported by: every kit that persists estate +data (LocusKit and others up the graph); kits never import a backend +target directly except at estate-construction time. Rust port in +`rust/` mirrors the trait surface and backend set but is NOT +conformance-fixture-gated against the Swift side (no shared byte- +identical corpus) — the cross-language contract is the protocol shape +and wire format only. + +ENTRY POINTS (most callers need only these): +- Storage.swift:16 `protocol Storage` — conform once per backend; exposes `.rowStore`/`.blobStore`/`.auditLog`/`.observer` +- Storage.swift:29 `Storage.open(schema:)` — bring backend up to declared schema +- Storage.swift:36 `Storage.transaction(isolation:_:)` — atomic multi-op block; ext. default isolation = .readCommitted +- RowStore.swift:33 `RowStore.insert/upsert/update/delete/query/count` — core row I/O +- EstateConfiguration.swift:8 `struct EstateConfiguration` — one value opens one Storage instance +- StorageReplicator.swift:89 `StorageReplicator.replicate(from:to:schema:)` — full-snapshot estate copy + +## Symbol Table + +### Core protocols — Storage.swift, RowStore.swift, BlobStore.swift, AuditLog.swift, StorageObserver.swift, StorageIntrospection.swift, Transaction.swift +- Storage.swift:16 `protocol Storage: Sendable` — rowStore/blobStore/auditLog/observer + open/close/transaction/migrate/currentSchemaVersion(for:) +- Storage.swift:58 `extension Storage.transaction(_:)` — default isolation .readCommitted +- RowStore.swift:8 `typealias RowKey = UUID` +- RowStore.swift:10 `struct StorageRow` — [String:TypedValue] wrapper, subscript by column name +- RowStore.swift:22 `struct RowHandle: Hashable` — (table, key) +- RowStore.swift:32 `protocol RowStore: Sendable` — insert/upsert/update/delete/query/count/querySkipCorrupt/query(...columns:)/begin·commit·rollbackTransaction (all true requirements, not ext defaults — needed for dynamic dispatch through `any RowStore`) +- RowStore.swift:72 `querySkipCorrupt(...)` — best-effort corpus scan, skips+counts StorageError.corruptStoredValue rows; NEVER use for point lookups +- RowStore.swift:96 `query(...columns:)` — column projection = no-blob read path; nil columns = full read (superset, always correct) +- RowStore.swift:194/214/236 `query/querySkipCorrupt(...asOf:)` — .present passthrough; .asOf(_) throws StorageError.featureGated("asOfQuery") until NT-L4+NT-P3 both merge +- RowStore.swift:260-266 ext defaults: beginTransaction/commitTransaction/rollbackTransaction — no-op (correct for backends w/o multi-stmt txn) +- BlobStore.swift:8 `typealias BlobKey = String` +- BlobStore.swift:10 `protocol BlobStore: Sendable` — put/get/delete/exists/size/listKeys; listKeys() order UNSPECIFIED (caller sorts) +- AuditLog.swift:23 `protocol AuditLog: Sendable` — append/appendBatch (idempotent on (eventID,hlc))/iterate(after:rowID:limit:)/eventsForRow/count +- StorageObserver.swift:26 `enum StorageEvent` — insert|update|delete +- StorageObserver.swift:32 `struct TableChange` — table/event/rowKey/values/hlc +- StorageObserver.swift:60 `enum BlobEvent` — put|delete +- StorageObserver.swift:73 `struct BlobChange` — key/event/bytes (bytes non-nil only on .put; last-write-wins on same key) +- StorageObserver.swift:111 `struct DirtyChainEvent` — changedRowId/parentNodeId/grandparentNodeId/contentHash/table — hash-on-write payload (NT-P2) +- StorageObserver.swift:140 `protocol StorageObserver: Sendable` — observe(table:events:)/observeBlobs()/observeDirtyChain(); delivery AT-LEAST-ONCE, order preserved PER-TABLE only +- StorageObserver.swift:168 ext default `observeDirtyChain()` — immediately-finished stream (back-compat for pre-hash-on-write observers) +- StorageIntrospection.swift:39 `struct StorageStats` — superset of fields across 3 backends; nil = "not measured" (see file's field/backend table); capturedAt is caller-injected (NEVER Date() inside engine) +- StorageIntrospection.swift:209 `protocol StorageIntrospection` — separate from Storage (additive capability); probe via `as? StorageIntrospection` +- Transaction.swift:9 `enum IsolationLevel` — readCommitted|repeatableRead|serializable +- Transaction.swift:15 `protocol StorageTransaction: Sendable` — rowStore/blobStore/auditLog (no nested txns, no savepoints in v1.0) + +### Value/type algebra — TypedValue.swift, Column.swift, Predicate.swift, Schema.swift, GeneratedColumn.swift, StorageError.swift +- TypedValue.swift:25 `enum TypedValue: Sendable, Hashable` — CLOSED 13-case wire format (null/bool/int/bitmap/float/text/blob/uuid/timestamp/json/hlc/fingerprint/array); new case = update every backend, deliberate cost +- Column.swift:9 `struct Column: Comparable` — (table,name); ordered by table then name for stable test fixtures +- Column.swift:24 `enum ColumnType` — uuid|bitmap|text|timestamp|float|int|bool|blob|json|hlc|fingerprint +- Predicate.swift:11 `indirect enum StoragePredicate` — CLOSED; logical(and/or/not/isTrue/isFalse) + comparison(eq/neq/lt/lte/gt/gte/isNull/isNotNull/in/like) + bitmap(bitmaskAll/Any/None/bitwiseEq, Int64 cols only) +- Predicate.swift:41 `StoragePredicate.all(_:)` — AND-combine w/ short-circuit: empty→isTrue, contains isFalse→isFalse +- Predicate.swift:55 `StoragePredicate.any(_:)` — OR-combine w/ short-circuit: empty→isFalse, contains isTrue→isTrue +- Predicate.swift:74 `struct OrderClause` — column + OrderDirection (default .ascending) +- Schema.swift:9 `struct SchemaDeclaration` — kitID/version/tables/indices/migrations +- Schema.swift:31 `struct TableDeclaration` — name/columns/primaryKey/uniqueConstraints/generatedColumns/appendOnly/hashable +- Schema.swift:82 `enum ColumnRole` — createdHlc|tombstonedHlc (as-of temporal filter tagging, ADR-017 §15) +- Schema.swift:90 `struct ColumnDeclaration` — name/type/nullable/defaultValue/role +- Schema.swift:128 `struct Migration` — fromVersion/toVersion/[SchemaOperation] +- Schema.swift:140 `enum SchemaOperation` — createTable/dropTable/addColumn/dropColumn/renameColumn/addIndex/dropIndex/custom(sqlite:postgresql:) +- Schema.swift:153-209 `ColumnDeclaration` static convenience ctors (.uuid/.bitmap/.text/.timestamp/.int/.float/.bool/.blob/.json/.hlc/.createdHlc/.tombstonedHlc/.fingerprint) +- Schema.swift:214-230 `TableDeclaration.createdHlcColumn/tombstonedHlcColumn/supportsAsOfFilter` — derive as-of eligibility from column roles +- GeneratedColumn.swift:43 `struct GeneratedColumn` — name/type/expression; ALWAYS STORED (PG has no VIRTUAL) +- GeneratedColumn.swift:63 `indirect enum GeneratedExpression` — column/literal/bitAnd/bitOr/bitXor/shiftRight/shiftLeft/equal/notEqual; evaluates to Int64 (bool=0/1) +- GeneratedColumn.swift:91 `renderSQL()` — shared SQLite+PG renderer; XOR emitted as (a|b)-(a&b) (SQLite lacks native XOR op) +- GeneratedColumn.swift:122 `evaluate(_:)` — InMemory direct evaluation against row dict +- GeneratedColumn.swift:150 `integerValue(_:)` — extract Int64 from int-family TypedValue; non-integer/absent → 0 (InMemory sentinel) +- StorageError.swift:5 `enum StorageError: Error, Equatable` — CLOSED; notable cases: :22 corruptStoredValue (NEVER fabricate a default — throw), :28 invalidConfiguration (fail-closed on platform-invalid config), :37 featureGated(feature:) (as-of query gate), :44 invalidIdentifier(name:) (SQL identifier injection guard, SECFIX-WS2-PK) + +### Estate configuration — EstateConfiguration.swift, EstateCacheConfig.swift, EncryptionMode.swift, NovelTokenTaggerChoice.swift +- EstateConfiguration.swift:8 `struct EstateConfiguration` — estateID/backend/encryptionConfig(default .plaintext)/cacheConfig(default .disabled)/novelTokenTagger(default .hmm) +- EstateConfiguration.swift:45 `enum BackendConfiguration` — .sqlite(url:busyTimeout:) | .postgresql(connectionString:poolSize:connectionTimeout:idleTimeout:) | .inMemory +- EstateConfiguration.swift:108 `queueSibling(filename:)` — derives sibling EstateConfiguration for per-estate queue DB; .sqlite→sibling file `.`; .inMemory→sibling inMemory; .postgresql→ throws featureGated (deferred, ADR-021) +- EstateConfiguration.swift:166 `deriveQueueSiblingID(parentID:filename:)` (private) — deterministic XOR-fold, NO UUID() call, same parent+filename ⇒ same sibling ID always +- EstateCacheConfig.swift:23 `struct EstateCacheConfig` — enabled/ceilingBytes(clamped ≥0)/sensitivityThreshold(clamped ≤2 — Secret=3 NEVER cacheable) +- EstateCacheConfig.swift:53 `EstateCacheConfig.disabled` — zero-change default, cache off +- EncryptionMode.swift:20 `enum EncryptionMode` — .plaintext (Mode1) | .rowEncryption (Mode2, per-row AES-GCM) | .fullDatabase (Mode3, SQLCipher whole-file); Mode4 (DB+threshold/FedRAMP) DELIBERATELY ABSENT — not a build capability +- EncryptionMode.swift:44 `struct EstateEncryptionConfig` — mode/keyIdentifier/`package`-scoped key (SymmetricKey?, never public, never logged) +- EncryptionMode.swift:68 `init(_ mode:)` — mints fresh 256-bit key + UUID keyID for encrypting modes; nil/nil for .plaintext +- EncryptionMode.swift:88 `fullDatabase(key:)` — builds Mode3 config from caller-supplied 256-bit key (Keychain-sourced) +- EncryptionMode.swift:99 `usesRowCrypto` — true ONLY for .rowEncryption (Mode3 has no per-row seam — whole file is ciphertext) +- EncryptionMode.swift:106 `fullDatabaseKeyHex` — lowercase hex for `PRAGMA key`; NEVER logged +- NovelTokenTaggerChoice.swift:43 `enum NovelTokenTaggerChoice: Sendable, Hashable, Codable` — .hmm (default, cross-platform-deterministic, federation-safe) | .nlTagger (Apple-only, non-deterministic across OS versions, federation-INCOMPATIBLE w/o re-tag — enforcement deferred to v1.1); FIXED AT ESTATE CREATION, no change-after-creation in v1.0; mirrors identically-named PersistenceKit-independent enum in LatticeLib — bridge by switch, no shared import +- NovelTokenTaggerChoice.swift:74 `NovelTokenTaggerChoice.default` — .hmm + +### Row-level crypto — RowCrypto.swift +- RowCrypto.swift:61 `protocol AeadProvider` (package) — swap point for AEAD algorithm; MUST generate fresh random nonce per encrypt call, MUST return [nonce][tag][ciphertext], MUST throw (never return garbage) on auth failure +- RowCrypto.swift:81 `struct CryptoKitAeadProvider: AeadProvider` — default; AES-GCM-256; :87 nonceByteCount=12, :88 tagByteCount=16 +- RowCrypto.swift:144 `enum RowCrypto` (package) — :151 encrypt(_:key:provider:), :168 decrypt(_:key:provider:); default provider = CryptoKitAeadProvider() +- RowCrypto.swift:191 `encryptedForWrite(_:config:provider:)` (package) — no-op unless usesRowCrypto + "content" column present as .text; encrypts + stamps keyID +- RowCrypto.swift:220 `decryptedForRead(_:config:provider:)` (package) — no-op unless keyID present AND matches config.keyIdentifier (mismatched keyID ⇒ pass through as ciphertext, NEVER attempt decrypt under wrong key) +- RowCrypto.swift:249 `assertContentKeyIDInvariant(_:table:config:)` (package) — structural guard (FUP-D/E-1): throws constraintViolation if encrypting estate has .text content with empty/absent keyID (seam didn't run) + +### Caching + hash-on-write decorators — CachingRowStore.swift, CacheInvalidator.swift, HashingRowStore.swift +- CachingRowStore.swift:46 `typealias ParentChainProvider = @Sendable (String, RowKey) -> [RowHandle]` — kit-supplied Merkle ancestor lookup +- CachingRowStore.swift:52 `final class CachingRowStore: RowStore, Sendable` — LRU hot-tier decorator; TRANSPARENCY GUARANTEE: results always identical to unwrapped backing store +- CachingRowStore.swift:164 `query(...asOf:)` — cache key = (RowHandle, AsOfCoordinate); .present entries evicted on write, .asOf(hlc) entries NEVER evicted (immutable, GC-pinned) +- CachingRowStore.swift:218 `invalidate(table:key:)` — external-write invalidation hook, called by CacheInvalidator +- CachingRowStore.swift:302 `extractKey(from:)` — cache lookups ONLY feasible for single-key `.eq(_, .uuid)` predicates; all else bypasses cache +- CachingRowStore.swift:334 `actor CacheActor` — owns entries/accessCounter/totalBytes; makes outer class Sendable w/o manual locks +- CachingRowStore.swift:421 `isAdmissible(_:)` (private, in CacheActor) — sensitivity gate: reads `provenance` bitmap bits[30:35] scale-gapped (0/16/32/48→ordinal 0-3); ordinal==3(Secret) ALWAYS rejected; absent column→admit; unparseable/unrecognized→REJECT (fail-closed) +- CachingRowStore.swift:482 `evictLRU()` (private) — O(n) min-scan by accessOrder; acceptable at cache-sized N +- CacheInvalidator.swift:34 `final class CacheInvalidator: Sendable` — StorageObserver→CachingRowStore.invalidate bridge for external writers +- CacheInvalidator.swift:52 `init(cache:observer:tables:)` — subscriptions registered before first await inside Task.detached; race note: a write issued in the SAME instant as init() may precede subscription (caller should yield if strict ordering required) +- CacheInvalidator.swift:92 `cancel()` / deinit — cancels root task ⇒ propagates to all per-table child tasks +- HashingRowStore.swift:36 `typealias ContentHashProvider = @Sendable (table:rowKey:values:) -> ContentHash` — kit-injected hash fn (PersistenceKit never imports a hash impl) +- HashingRowStore.swift:49 `typealias HashParentChainProvider = @Sendable (table:rowKey:) -> (parentNodeId:grandparentNodeId:)?` +- HashingRowStore.swift:55 `struct HashOnWriteConfig` — hashableTables/hashProvider/parentChainProvider +- HashingRowStore.swift:81 `final class HashingRowStore: RowStore, @unchecked Sendable` — decorator chain position: caller → HashingRowStore → CachingRowStore → backend +- HashingRowStore.swift:295 `augmentWithHashForKnownKey(table:rowKey:mergedValues:)` (private) — UPDATE/upsert-as-update path: hashes FULL merged row (not partial SET dict) per SECFIX-WS2-PK F6; strips stale content_hash before hashing so insert vs update paths agree +- HashingRowStore.swift:330 `augmentWithHash(table:values:)` (private) — INSERT path; extracts rowKey from "id" column by convention +- HashingRowStore.swift:369 `emitDirtyChain(...)` (private) — delivers DirtyChainEvent via injected ObserverRegistryRef (nil = computed but not delivered) + +### Erasure — ErasureLedger.swift, ErasureOverlay.swift +- ErasureLedger.swift:15 `struct ErasureLedgerEntry` — drawerId + erasedHlc +- ErasureLedger.swift:28 `enum ErasureLedgerTables` — .ledger = "erasure_ledger" +- ErasureLedger.swift:35 `ErasureLedgerSchema.ledgerTable` — TableDeclaration, appendOnly:true (UPDATE/DELETE throw appendOnlyViolation at storage layer) +- ErasureLedger.swift:53 `ErasureLedgerOps.recordErasure(...)` — throws duplicateKey if drawerId already erased (erase-once contract) +- ErasureLedger.swift:68 `ErasureLedgerOps.isErased(...)` — fast point-lookup, hot path for every erasure-subject read +- ErasureLedger.swift:86 `ErasureLedgerOps.lookupErasure(...)` — full entry incl. erasedHlc +- ErasureOverlay.swift:22 `struct ErasureOverlayConfig` — extractErasureId(row)->String? (nil=not erasure-subject) + contentColumns([String]) supplied by calling kit +- ErasureOverlay.swift:64 `ErasureOverlay.apply(rows:config:rowStore:)` — TWO-PHASE FAIL-CLOSED: phase1=query already ran; phase2=per-row ledger check; ledger-check THROW ⇒ row DROPPED entirely (never shown-when-uncertain) +- ErasureOverlay.swift:96 `nullContentColumns(row:columns:)` — nulls listed columns, preserves skeleton (id/hlc/timestamps) + +### Snapshots + GC — SnapshotRegistry.swift, GCPin.swift +- SnapshotRegistry.swift:18 `struct SnapshotId: Hashable` — :24 `.mint()` UUID-backed +- SnapshotRegistry.swift:32 `struct SnapshotRecord` — snapshotId/hlc/label/createdAt +- SnapshotRegistry.swift:47 `struct SnapshotAttestation` — snapshotId/subjectKind/subjectId/merkleRoot/keyVersion?; subject semantics owned by CALLING kit, not PersistenceKit +- SnapshotRegistry.swift:73 `enum SnapshotTables` — .registry="snapshot_registry", .attestations="snapshot_attestations" +- SnapshotRegistry.swift:83/98 `SnapshotSchema.registryTable` / `.attestationsTable` — TableDeclarations +- SnapshotRegistry.swift:130 `SnapshotRegistryOps.createSnapshot(...)` — mints SnapshotId, inserts registry row + N attestation rows +- SnapshotRegistry.swift:161 `.listSnapshots(...)` — HLC ascending +- SnapshotRegistry.swift:177 `.deleteSnapshot(...)` — attestations deleted BEFORE registry row (child-first) +- SnapshotRegistry.swift:201 `.attestations(rowStore:snapshotId:)` — ordered by (subjectKind, subjectId) +- GCPin.swift:22 `GCPin.minimumRetainableHlc(rowStore:)` — MIN(hlc) across all snapshots; nil = nothing pinned = all vacuumable +- GCPin.swift:44 `GCPin.isPinned(rowStore:rowHlc:)` — rowHlc.packed >= minRetainableHlc.packed + +### Telemetry + observer stubs — PersistenceKitTelemetry.swift, NoOpObserver.swift +- PersistenceKitTelemetry.swift:81 `reportStorageStats(_:estateID:now:)` — OFF by default; single Atomic load+branch when Intellectus.isEnabled==false (~1ns, no stats() call); emits `persistence.db.*` namespace, one metric per non-nil StorageStats field; `now` ALWAYS caller-injected (never Date() inline) +- NoOpObserver.swift:10 `final class NoOpObserver: StorageObserver, Sendable` — immediately-finished streams; used directly by PostgreSQLStorage (no live PG notification channel) + +### Backend: PersistenceKitInMemory +- InMemoryStorage.swift:24 `final class InMemoryStorage: Storage, Sendable` — no persistence across process runs; always effectively-serializable isolation (full state snapshot per txn) +- InMemoryStorage.swift:78 `transaction(isolation:_:)` — mutates LIVE actor state directly (NOT a detached-copy-then-replace) — replace-on-commit would silently drop concurrent bare inserts landing between snapshot and replace (real incident: 5-10% lost QueueKit sends under burst); rollback restores pre-txn snapshot +- InMemoryStorage.swift:102/110 `beginNotificationBuffering()` / `commitNotifications()` — buffer during txn, flush only on successful commit (SECFIX-WS2-PK F2: rolled-back writes must never reach observers) +- InMemoryStorage.swift:144 `actor InMemoryStateActor` — sole owner of InMemoryState; :462 `materializeGenerated(_:_:)` static — computes GeneratedColumn values identically to SQL backends' STORED columns +- InMemoryStorage.swift:388 `queryRows(...)` — predicate/order/pagination applied to FULL row first, projection applied LAST (matches SQL ORDER-BY-on-unselected-column semantics) +- InMemoryStorage.swift:635 `HLC.packed` (extension) — canonical bit-pack: physical<<16 | logical<<4 | node +- InMemoryRowStore.swift:19 `final class InMemoryRowStore: RowStore, Sendable` — thin forward to state actor; column-projection kept for cross-backend StorageRow shape parity, not for a real transfer saving +- InMemoryBlobStore.swift:6 `final class InMemoryBlobStore: BlobStore, Sendable` — forwards to actor blob dict +- InMemoryAuditLog.swift:19 `final class InMemoryAuditLog: AuditLog, Sendable` — forwards to actor; dedup on (eventID,hlc) enforced in actor +- InMemoryObserver.swift:15 `final class ObserverRegistry: @unchecked Sendable` — NSLock-based (NOT actor) so `register` is SYNCHRONOUS — subscription recorded before observe() returns, no race window for immediately-following write; mirrors Rust ObserverHub::subscribe +- InMemoryObserver.swift:104 `notify(_:)` — matching subs snapshotted UNDER lock, yielded OUTSIDE lock (avoids deadlock vs. continuation onTermination which also locks) +- PredicateEvaluator.swift:6 `enum PredicateEvaluator` — in-memory StoragePredicate interpreter (only backend evaluating the tree directly rather than compiling to a query string) +- PredicateEvaluator.swift:61 `likeMatch(_:pattern:)` — SQL LIKE→NSRegularExpression (%→.*, _→.) +- PredicateEvaluator.swift:80 `enum TypedValueComparator` — :81 `compare(_:_:)` shared ordering; .null sorts first; .hlc compared via .packed (single total order) + +### Backend: PersistenceKitSQLite +- SQLiteStorage.swift:22 `final class SQLiteStorage: Storage, Sendable` — one connection per estate +- SQLiteStorage.swift:110 `actor SQLiteBackend` — owns connection + inTransaction flag + pendingBlobNotifications (buffered during txn, SECFIX-WS2-PK F3) +- SQLiteStorage.swift:411 `insertRow` / :461 `upsertRow` / :495 `updateRows` / :535 `deleteRows` / :559 `queryRows` — ALL validate every table/column identifier via `validateSQLIdentifier` before interpolation (SECFIX-WS2-PK F9); insertRow+queryRows run RowCrypto encrypt/decrypt seam +- SQLiteStorage.swift:656 `queryRowsSkipCorrupt(...)` — cursor-level per-row skip+log on corruptStoredValue; other errors re-thrown (systemic) +- SQLiteStorage.swift:782 `storageStats(now:)` — PRAGMA page_size/page_count/freelist_count; WAL frame count derived from `-wal` FILE SIZE (NOT PRAGMA wal_checkpoint — can SQLITE_LOCKED even same-actor) +- SQLiteStorage.swift:917 `readColumn(...)` — type-tolerant decode (valid-but-coerced value passes through) vs. parse-failure (unparseable UUID/timestamp TEXT) → THROWS corruptStoredValue, never fabricates +- SQLiteConnection.swift:24 `final class SQLiteConnection: @unchecked Sendable` — thin sqlite3 C API wrapper +- SQLiteConnection.swift:29 `init(url:busyTimeout:keyHex:)` — ORDER MATTERS: symlink-refusal check (CAND-052, lstat semantics) → sqlite3_open_v2 → PRAGMA key (Mode3, MUST be first stmt) → Apple Data Protection → WAL/synchronous/busy_timeout/foreign_keys pragmas +- SQLiteConnection.swift:162 `final class SQLiteStatement` — prepared-stmt wrapper; :180 `bind(_:at:)` TypedValue→sqlite3_bind_*; :239 `step()` +- SQLiteConnection.swift:318 `enum ISO8601` — :356 `string(from:)` clamps Date to RFC-3339 range [0001,9999] before formatting (logs warning on clamp, never emits unparseable-back string); :392 `date(from:)` tries :412 `fastParseCanonicalUTC` FIRST (allocation-free, handles the exact canonical shape this kit writes — Merkle rollup re-decode was ~80% of CPU via ICU formatter before this fast path), falls back to ISO8601DateFormatter for anything else +- SQLiteStores.swift:22 `final class SQLiteRowStore: RowStore, Sendable` — forwards to SQLiteBackend; :75 `querySkipCorrupt` overrides protocol default with real cursor-level skip +- SQLiteStores.swift:90/102/119 `SQLiteBlobStore` / `SQLiteAuditLog` / `SQLiteTransaction` — thin forwards +- SQLiteSchema.swift:9 `enum SQLiteSchema` — :12 `nativeType(_:)` ColumnType→SQLite storage class (.uuid/.timestamp→TEXT, .hlc→INTEGER packed, .fingerprint→BLOB); :29 `createTable(_:)`; :68 `appendOnlyTriggers(_:)` emits BEFORE UPDATE/DELETE RAISE(ABORT) trigger pair +- SQLiteSchema.swift:103/132/167 internal tables: `_storagekit_migrations` (kit_id PK) / `_storagekit_audit` (event_id+hlc PK, full-precision physical_time/logical_count/node_id columns alongside lossy packed hlc) / `_storagekit_blobs` (key PK) +- SQLitePredicateCompiler.swift:27 `SQLitePredicateCompiler.compile(_:)` — throws on invalidIdentifier; every column name validated before interpolation (SECFIX-WS2-PK F7) +- SQLiteIdentifierValidator.swift:28 `validateSQLIdentifier(_:)` — `[A-Za-z_][A-Za-z0-9_]*`; SINGLE seam for SQLite module (SQLiteStorage + SQLitePredicateCompiler both call this, no forked copy) +- SQLiteObserver.swift:26 `actor SQLiteObserverRegistry` — row subs via sqlite3_update_hook (op/table/rowid ONLY, no column values); blob subs fed directly by SQLiteBackend.putBlob/deleteBlob call sites (hook cannot carry bytes) +- KeychainKeyStore.swift:36 `struct KeychainKeyStore` (Apple-only, `#if canImport(Security)`) — Mode3 key source; :72 `estateAccount(for:)` SHA-256(standardized path)→per-estate Keychain account, so co-processes agree without coordination; :82 `loadOrCreateKey()` idempotent under concurrent first-callers; :111 `deleteKey()` idempotent (missing item = success); access class `kSecAttrAccessibleAfterFirstUnlockThisDeviceOnly` (background-readable post-first-unlock, device-only, no iCloud sync) + +### Backend: PersistenceKitPostgreSQL +- PostgreSQLStorage.swift:23 `final class PostgreSQLStorage: Storage, Sendable` — observer = NoOpObserver() (no live PG change-notification impl) +- PostgreSQLStorage.swift:47 estate isolation: search_path pinned to `pk_` schema per connection (PG analogue of SQLite one-file-per-estate); `public` stays on path for shared extensions +- PostgreSQLStorage.swift:120 `actor PostgreSQLBackend` — :129 `encryptionConfig` is `nonisolated let` (immutable+Sendable, read sync by row stores for the crypto seam) +- PostgreSQLStorage.swift:206/333 per-kit schema version stored in `_storagekit_meta` under composite key `"schema_version:"`; global max under plain key `"schema_version"` +- PostgreSQLStorage.swift:246 `storageStats(now:)` — pg_database_size / pg_stat_database (blks_hit,blks_read,xact_commit,xact_rollback,deadlocks) / pg_locks⋈pg_database (granted=false ⇒ lockContention) +- PostgreSQLPool.swift:12 `actor PostgreSQLPool` — fixed-size; :38 `acquire()` reuse→open-new(if under size)→CheckedContinuation-wait w/ timeout→poolExhausted +- PostgreSQLPool.swift:101 `openConnection()` — every new conn: CREATE SCHEMA IF NOT EXISTS + SET search_path, closes conn on setup failure (never hands back half-configured conn) +- PostgreSQLPool.swift:168 `parseTLSMode(host:)` — `ARIA_MCP_POSTGRES_TLS` env var: disable|require|(absent/unrecognized→prefer, incl. loopback — explicit opt-out required for plaintext) +- PostgreSQLConnection.swift:23 `PostgresConnection` ext — :25 `executeSimple` / :33 `executeParameterized` wrap postgres-nio errors as StorageError.backendError +- PostgreSQLConnection.swift:45 `makeBindings(_:)` — TypedValue→PostgresBindings; .fingerprint serialized as 32 raw bytes fixed block order +- PostgreSQLConnection.swift:103/116 `decodeRow(_:columns:)` / `decodeCell(_:type:)` — decode failure → `.null` (NOT a throw — PG wire protocol already enforces column types at a lower level than SQLite affinity) +- PostgreSQLPredicateCompiler.swift:27 `.compile(_:)` — `$1,$2,...` positional params; :89/:102 bitmask cases reference prior bindings by FINAL numeric position (bindings.count-1 / bindings.count) +- PostgreSQLSchema.swift:8 `enum PostgreSQLSchemaEmitter` — :47 `typeSQL(_:)` (.uuid→UUID native, .timestamp→TIMESTAMPTZ, .json→JSONB); :83 `appendOnlyFunctionSQL` ONE shared plpgsql trigger fn (CREATE OR REPLACE, idempotent) for ALL append-only tables; :97 `appendOnlyTriggerStatements` DROP-then-CREATE per table (PG has no CREATE TRIGGER IF NOT EXISTS) +- PostgreSQLStores.swift:25 `final class PostgreSQLRowStore: RowStore, Sendable` — :34 `withConnection` routes to txn.connection if inside a PostgreSQLTransactionContext, else pool.acquire/release +- PostgreSQLStores.swift:240 `renderPredicate(_:startIndex:bindings:)` (private) — renumbers `$N` placeholders in REVERSE order (avoid $10 clobbered by naive forward $1 replace) +- PostgreSQLStores.swift:258/352 `PostgreSQLBlobStore` / `PostgreSQLAuditLog` — lazily CREATE TABLE IF NOT EXISTS own backing tables on first use (not in caller's SchemaDeclaration) +- PostgreSQLStores.swift:509 `decodeAuditEvent(_:)` — required fields `try` (throw on failure); optional before-state fields `try?` (NULL = valid "no prior state") +- PostgreSQLIdentifierValidator.swift:26 `validatePSQLIdentifier(_:)` — identical rule to SQLite's validator; independent copy (3 total incl. Rust `validate_sql_identifier`) — SECFIX-WS2-PK F7/F9/F10, one seam PER MODULE + +### PersistenceKitReplication +- ReplicationTypes.swift:24 `struct ReplicationCursor: Equatable` — hlcWatermark/rowsWritten/auditEventsWritten/blobsWritten +- ReplicationTypes.swift:51 `enum ReplicationError` — schemaMismatch(sourceVersion:destinationVersion:sourceKitID:destinationKitID:) | storageFailure(detail:) +- StorageReplicator.swift:69 `enum StorageReplicator` — :89 `replicate(from:to:schema:)` core; :103 `flush(from:into:schema:)` / :115 `hydrate(into:from:schema:)` direction-named wrappers +- StorageReplicator.swift:125 `replicateFull(...)` — 3 steps: (1) per-kit schema-version GATE (throws schemaMismatch, NO auto-migrate), (2) snapshotSource() reads ALL rows/audit/blobs into Sendable payload BEFORE opening dest txn, (3) one `.serializable` dest transaction does upsert(conflictColumns:=table.primaryKey — NOT RowHandle.key) +- StorageReplicator.swift:216 blob-delete propagation (SECFIX-WS2-PK F5) — destination keys ABSENT from source snapshot are deleted; additive-only copy would otherwise leak orphaned blobs forever +- StorageReplicator.swift:256 `snapshotSource(...)` — generated columns FILTERED OUT of each row before staging (destination recomputes); a blob present in listKeys() but absent on get() (TOCTOU) → THROWS storageFailure, never silently dropped +- IncrementalReplicationSession.swift:56 `actor BlobDirtySet` — :61 `accumulate` last-write-wins per key; :67 `drain()` sorted-by-key, atomic clear; :79 `restore(_:)` UNION semantics (does not overwrite newer dirt) +- IncrementalReplicationSession.swift:104 `struct DirtyKey: Comparable` — (table, pkEncoded) sorted string key; ORDER IS LOAD-BEARING — deterministic replay across independent sync runs +- IncrementalReplicationSession.swift:135 `actor DirtySet` — :163 `accumulate(_:)` extracts PK cols from TableChange.values; missing PK col or nil values dict ⇒ LOG+SKIP (never crash); :195 `drain()` sorted; :213 `restore(_:)` union (Set.insert no-op if present) +- IncrementalReplicationSession.swift:242 `final class IncrementalReplicationSession: Sendable` — :283 `start(source:schema:)` subscribes one Task per schema table + one blob Task; :266 deinit cancels all tasks +- IncrementalReplicationSession.swift:361 `sync(from:to:fromCursor:)` — schema-version gate (same as full snapshot) → drain both dirty sets → re-scan each dirty row from SOURCE (never trust the original notification's payload for rows — row may have changed again) → missing row ⇒ destination DELETE, present ⇒ UPSERT → audit events filtered to HLC > fromCursor.hlcWatermark → ONE serializable dest transaction +- IncrementalReplicationSession.swift:396-406 RETRY-PRESERVATION: on ANY error after drain, `dirtySet.restore` + `blobDirtySet.restore` are AWAITED SYNCHRONOUSLY inside the catch — NEVER fire-and-forget/detached — a detached restore could race an immediate caller retry and reproduce the lost-keys bug nondeterministically +- IncrementalReplicationSession.swift:521 `snapshotDirtyRows(...)` (private) — blob ops need NO source re-read (change event already carries payload); row ops DO re-scan (predicate = exact PK match, `pkPredicate(for:table:)` at :617) + +## INVARIANTS / GOTCHAS + +- TypedValue is a CLOSED 13-case enum. Adding a case requires updating every backend (SQLite/PostgreSQL/InMemory/Rust) in the same change — this cost is deliberate, not an oversight. +- StoragePredicate is CLOSED. Backends treat it as opaque data except when compiling; never special-case another backend's predicate shape. +- SQL IDENTIFIER INJECTION GUARD (SECFIX-WS2-PK F1/F7/F9/F10): every caller-supplied table/column name reaching a dynamically-built SQL string is validated against `[A-Za-z_][A-Za-z0-9_]*` FIRST — three independent seams (SQLiteIdentifierValidator.validateSQLIdentifier, PostgreSQLIdentifierValidator.validatePSQLIdentifier, Rust validate_sql_identifier), one per module, never forked within a module. Double-quoting alone is NOT sufficient — a name containing `"` escapes the delimiter. +- corruptStoredValue is THROWN, never papered over. A stored UUID/timestamp string that fails to parse must surface as an error — substituting a fabricated UUID or epoch-0 date is a silent data-identity lie. querySkipCorrupt / queryRowsSkipCorrupt exist specifically so a CORPUS scan (not a point lookup) can skip-and-log instead of aborting wholesale. +- as-of temporal query (.asOf(hlc)) is GATED OFF by default — every default implementation throws StorageError.featureGated("asOfQuery"). Do not implement/ungate a backend override until NT-L4 (lineage-wide expunge) and NT-P3 (erasure overlay) have both merged — ungating early risks resurfacing erased content through a stale snapshot read. +- CachingRowStore TRANSPARENCY GUARANTEE: every read must return exactly what the unwrapped backing store would. Cache changes latency only, NEVER correctness. Sensitivity gate fails CLOSED — unparseable/unrecognized provenance bit patterns are rejected from the cache, never admitted "just in case." +- Sensitivity ordinal 3 (Secret) is NEVER cacheable, enforced twice: EstateCacheConfig clamps sensitivityThreshold to ≤2 at construction, AND CacheActor.isAdmissible re-checks ordinal==3 unconditionally as defense-in-depth. +- Decorator chain order matters: caller → HashingRowStore → CachingRowStore → backend. HashingRowStore must see the row BEFORE caching decides whether to admit it. +- HashingRowStore hashes the FULL committed row on UPDATE/upsert-as-update (pre-read + merge), never the partial SET-columns dict — a partial hash would diverge from what INSERT computes for identical final data (SECFIX-WS2-PK F6). +- RowCrypto: a keyID mismatch on read means "sealed under a key this estate does not hold" — pass the row through as still-ciphertext, do NOT attempt decrypt (would only surface as an AES-GCM auth failure). Mode 3 (fullDatabase) makes the per-row seam a permanent no-op (usesRowCrypto==false) — the whole file is already ciphertext. +- assertContentKeyIDInvariant is a STRUCTURAL guard, not just documentation: any encrypting-estate write path that reaches a backend with `.text` content and no/empty keyID throws constraintViolation. If you add a new content-bearing write path (a new upsert variant, a migration path), it must run the encryption seam BEFORE this guard sees it, or the guard will correctly reject the write. +- ErasureOverlay is FAIL-CLOSED: a ledger-check failure drops the row from the result entirely rather than showing possibly-erased content. This is NOT the same as failing open — silence (row absence) is the safe failure mode here. +- erasure_ledger table is appendOnly:true, enforced by the STORAGE LAYER (SQLite triggers / PostgreSQL trigger function / InMemory RowStore.update·delete throw) — not just by convention in ErasureLedgerOps. +- InMemoryStorage.transaction mutates LIVE actor state (not a detached copy replaced on commit) — this is a scar from a real lost-write incident (5-10% of QueueKit sends under burst). Do not "simplify" this back to copy-then-replace. +- Notification buffering (SECFIX-WS2-PK F2/F3) applies during EVERY transaction on EVERY backend (InMemory row+blob, SQLite blob): observers must never see a notification for a write that was later rolled back. Buffer before the block runs, flush only after commit, discard on rollback. +- IncrementalReplicationSession retry-preservation restore MUST be awaited synchronously inside the failure catch block, never dispatched to a detached Task — a detached restore races an immediate caller retry and can reproduce a lost-dirty-keys bug nondeterministically. +- Full-snapshot replication (StorageReplicator) upserts on `table.primaryKey` as the conflict column, NEVER on RowHandle.key (which is a fresh random UUID per call and would insert duplicates on every re-run). +- Replication requires exact per-kit schema-version equality between source and destination (srcVersion == dstVersion == schema.version). No auto-migration, ever — a version mismatch is ReplicationError.schemaMismatch, not a best-effort attempt. +- Generated columns are ALWAYS STORED (never VIRTUAL) — PostgreSQL has no VIRTUAL form, and this keeps SQLite/PostgreSQL/InMemory semantically identical. Generated column names are filtered OUT of any row payload staged for upsert during replication (destination recomputes them; writing a value into a GENERATED column errors on both SQL backends). +- HLC.packed truncates physicalTime to 40 bits — lossy for far-future timestamps. SQLite's `_storagekit_audit` table therefore stores physical_time/logical_count/node_id as separate full-precision columns alongside the packed value; decode from those three columns, not from unpacking the packed column, or a cold rebuild's lastHLC can silently disagree with a snapshot's. +- ISO8601 fast-path parser (`fastParseCanonicalUTC`) exists purely for CPU cost: the general ISO8601DateFormatter was measured at ~80% of total CPU during large imports (Merkle rollup re-decodes every row's timestamp on every insert). The fast path recognizes ONLY the exact canonical shape this kit writes and returns nil (triggering formatter fallback) for anything else — never weaken its exactness to "handle more cases," that reintroduces the cost it exists to avoid. +- NovelTokenTaggerChoice is FIXED AT ESTATE CREATION in v1.0 — no change-after-creation path exists yet. `.nlTagger` estates cannot safely federate with `.hmm` estates without full re-tagging; this is not yet enforced automatically (v1.1 work) — caller discipline is the only guard today. +- Pinned/deterministic constants: sensitivityThreshold clamp ≤2, Secret ordinal ==3 always excluded, SQL identifier pattern `[A-Za-z_][A-Za-z0-9_]*`, ciphertext envelope layout [12-byte nonce][16-byte tag][ciphertext], KeychainKeyStore.keyByteCount=32, RFC-3339 round-trip range [0001-01-01, 9999-12-31]. +- PersistenceKit imports NO vector-search engine and must never grow one — the ACCOMMODATION contract (vector payload round-trip, bulk hydration, count, delete via the general RowStore/BlobStore surfaces) is the full extent of this package's obligation toward VectorKit's workload (ADR-008). diff --git a/packages/kits/PersistenceKit/docs/DETAILS.md b/packages/kits/PersistenceKit/docs/DETAILS.md new file mode 100644 index 0000000..15e2861 --- /dev/null +++ b/packages/kits/PersistenceKit/docs/DETAILS.md @@ -0,0 +1,1361 @@ +--- +doc: DETAILS +package: PersistenceKit +repo: moot-system +authored_commit: 909513d0a8ecb1e9e903af9f4d25b3e4f2528242 +authored_date: 2026-07-04 +sources: + - path: Sources/PersistenceKit/AuditLog.swift + blob: ca4c0c9623056a49ad888a7a77e720a7519556bb + - path: Sources/PersistenceKit/BlobStore.swift + blob: f052c3ecc693d5414d57ad2c6da5fb4c5fe28f79 + - path: Sources/PersistenceKit/CacheInvalidator.swift + blob: 0a844b8037404dd72d08df6887ceee1e8c6014f2 + - path: Sources/PersistenceKit/CachingRowStore.swift + blob: 7edf9a86eb31dd9a17abd97b7baa9e8d5425e266 + - path: Sources/PersistenceKit/Column.swift + blob: 3cfc3ba856ebe15e8756ae65dae736cbd2c288a6 + - path: Sources/PersistenceKit/EncryptionMode.swift + blob: fdb7aa7ec63088ae09739f5e2a8e1468c386b987 + - path: Sources/PersistenceKit/ErasureLedger.swift + blob: eb2ca35f69783eead6049949bbaad68decfc9915 + - path: Sources/PersistenceKit/ErasureOverlay.swift + blob: d5a86f2baa10647134f285d6996a8e62cbe23ace + - path: Sources/PersistenceKit/EstateCacheConfig.swift + blob: 7003951cb5b2b075b6bb0c49b5fe372a97bfeb3a + - path: Sources/PersistenceKit/EstateConfiguration.swift + blob: be91569405e5c48a99859b4595540a59bd1a1994 + - path: Sources/PersistenceKit/GCPin.swift + blob: 73336dc4c693fcfa285a7eff8ed6520f27951ff8 + - path: Sources/PersistenceKit/GeneratedColumn.swift + blob: 34e466c80a0e9aac5d24303eaad132901333470b + - path: Sources/PersistenceKit/HashingRowStore.swift + blob: fe233ed2f4fda177373ad105d14fa321d325d0df + - path: Sources/PersistenceKit/NoOpObserver.swift + blob: 99f62b6889df50c5f09fce749f363637891f116c + - path: Sources/PersistenceKit/NovelTokenTaggerChoice.swift + blob: 9588d6303431883a2c8e0ee727f7ebe3e3f6d139 + - path: Sources/PersistenceKit/PersistenceKitTelemetry.swift + blob: a24e54dfb97beb86dac9422bbfc7cc3c2d959d73 + - path: Sources/PersistenceKit/Predicate.swift + blob: 522802c842dfc7573137efd7a3f36300d5201468 + - path: Sources/PersistenceKit/RowCrypto.swift + blob: 6678e426c0902cdae6d164904001526756e40916 + - path: Sources/PersistenceKit/RowStore.swift + blob: 9a96ff89528ca786d51224324cacb78cf810037d + - path: Sources/PersistenceKit/Schema.swift + blob: 5a6894c68d6f29eecfe9acc34d7c06ee2a655ac3 + - path: Sources/PersistenceKit/SnapshotRegistry.swift + blob: 88087bda544165c4d24c514a4f3e0641a620512e + - path: Sources/PersistenceKit/Storage.swift + blob: 7484a40913b28cb11a6bd2e3ea822dc8fe8eb63e + - path: Sources/PersistenceKit/StorageError.swift + blob: 743c2d1a24c7bafedb217e4c6bbf30ca20d03be8 + - path: Sources/PersistenceKit/StorageIntrospection.swift + blob: 35522b6601037246907bb88ebb2fb2eb5ea9b0e2 + - path: Sources/PersistenceKit/StorageObserver.swift + blob: f0a8b61a15344e02193137a3393be345dd51cf25 + - path: Sources/PersistenceKit/Transaction.swift + blob: d58d478618b31cc3985275be6ceee84a6fc222de + - path: Sources/PersistenceKit/TypedValue.swift + blob: ec124b7aebd86f64f67fea6656396d374666b741 + - path: Sources/PersistenceKitInMemory/InMemoryAuditLog.swift + blob: 1ba83d2408d3384d4d8e4f286fa2db743a433cc9 + - path: Sources/PersistenceKitInMemory/InMemoryBlobStore.swift + blob: e02822bde83c899b6d18ab8d82b90afba6abb5ac + - path: Sources/PersistenceKitInMemory/InMemoryObserver.swift + blob: 71b98c9da2d869a81a78265b7ef1f64a57a907af + - path: Sources/PersistenceKitInMemory/InMemoryRowStore.swift + blob: 2f7c92612c46b8a097e4211147c924f8356e47d1 + - path: Sources/PersistenceKitInMemory/InMemoryStorage.swift + blob: 9820c771bfe39ab738eea1f3b1623491fcfb1326 + - path: Sources/PersistenceKitInMemory/PredicateEvaluator.swift + blob: e8735422ff87addff6a2a3a89da85c495d07f07c + - path: Sources/PersistenceKitPostgreSQL/PostgreSQLConnection.swift + blob: 5a282a6064a69a543a5d650ccc7eeeae2c5a3e4f + - path: Sources/PersistenceKitPostgreSQL/PostgreSQLIdentifierValidator.swift + blob: 120cf0c1576a7db45d79bf6865e2f15cff09a5ac + - path: Sources/PersistenceKitPostgreSQL/PostgreSQLPool.swift + blob: eddc0c3af4135565e15d2d763b0d24240973dd6f + - path: Sources/PersistenceKitPostgreSQL/PostgreSQLPredicateCompiler.swift + blob: 6f0791e0372b09cf2605bae4cf149e48bbba2834 + - path: Sources/PersistenceKitPostgreSQL/PostgreSQLSchema.swift + blob: f165f0877b96c87e2f7de46012f8f8acb3c03cc8 + - path: Sources/PersistenceKitPostgreSQL/PostgreSQLStorage.swift + blob: 849926f9e6def788fae2f38757ad44bcd52a18a0 + - path: Sources/PersistenceKitPostgreSQL/PostgreSQLStores.swift + blob: bff90838d1324d82ae76a0895f15399537e9e343 + - path: Sources/PersistenceKitReplication/IncrementalReplicationSession.swift + blob: 2f90378146e043d75c739d400d98c7770e07af78 + - path: Sources/PersistenceKitReplication/ReplicationTypes.swift + blob: bb13c63d1febd50ad46c1814610d7f6c31a33112 + - path: Sources/PersistenceKitReplication/StorageReplicator.swift + blob: f5acc5993c8647e53c127c6871a1282c24b1c427 + - path: Sources/PersistenceKitSQLite/KeychainKeyStore.swift + blob: 0071732291a7cb6ce0777bd230a6188276fb4f32 + - path: Sources/PersistenceKitSQLite/SQLiteConnection.swift + blob: ece56dc7e25e67656bb37f5222c18c1166c750cc + - path: Sources/PersistenceKitSQLite/SQLiteIdentifierValidator.swift + blob: 713339c137d6af1cfbba5a3584e05bdda70b42c5 + - path: Sources/PersistenceKitSQLite/SQLiteObserver.swift + blob: 2cb61ed75dae5ab81a3cdfb5c9581731d25fd960 + - path: Sources/PersistenceKitSQLite/SQLitePredicateCompiler.swift + blob: 5770adb2024c54ea6921651632ca65ee84d416af + - path: Sources/PersistenceKitSQLite/SQLiteSchema.swift + blob: 4ba6fc175fe9d486b17af1088a23da64041b12ae + - path: Sources/PersistenceKitSQLite/SQLiteStorage.swift + blob: 417e63cc07b60295ac874187ebb796bf9f3b3c86 + - path: Sources/PersistenceKitSQLite/SQLiteStores.swift + blob: 76499ffb70d979f0d13e8cf9e32bc38ff28ffdb5 +--- + +# PersistenceKit Details + +This document walks through every source file in the package. Read +`OVERVIEW.md` first for the big picture. Files appear grouped by target, +in the order a reader should learn them: the core protocols and value +types first, then the decorators and cross-cutting toolkits built on top +of them, then the three backends that implement the core protocols, and +finally the replication module that operates across two backends. + +## Target: PersistenceKit (core) + +### Storage.swift + +This file provides the `Storage` protocol — the single entry point every +backend implements. A term of art first: a protocol in Swift is a contract +that a type promises to fulfill; any type that conforms to `Storage` can +be used anywhere `Storage` is expected, regardless of what physical +engine backs it. + +`Storage` bundles four sub-stores — `rowStore`, `blobStore`, `auditLog`, +`observer` — plus five life-cycle functions: `open(schema:)` creates files +or connections and brings the backend up to its declared schema version; +`close()` shuts it down cleanly and must be safe to call more than once; +`transaction(isolation:_:)` runs a block of work atomically, rolling back +every change if the block throws; and the two `currentSchemaVersion` +overloads report how far a backend's schema has migrated, either +globally or for one named kit. `migrate(to:)` advances the schema forward. +A protocol extension supplies a default isolation level of read-committed, +so most callers can write `storage.transaction { ... }` without naming a +level at all. + +### RowStore.swift + +This file provides the typed row input/output protocol, plus three small +supporting types: `RowKey` (a type alias for `UUID`), `StorageRow` (an +immutable, subscriptable wrapper around a `[String: TypedValue]` row), and +`RowHandle` (a `(table, key)` pair that identifies exactly one row). + +`RowStore` declares the operations every backend must support: `insert`, +`upsert`, `update`, `delete`, `query`, and `count`. Two further query +variants matter for performance and safety. `query(...columns:)` is the +column-projecting form: passing a specific list of column names means a +column such as a large text blob is never transferred out of storage at +all, which matters when a caller only needs a handful of small columns +from a wide table. `querySkipCorrupt(...)` is the resilience form: rather +than aborting an entire corpus scan the moment one row's stored value +fails to parse, it skips that row, counts it, and returns everything +else — the file's documentation is explicit that this method exists for +best-effort scans (like "every drawer in the estate"), never for a +single-row point lookup, where a corrupt value should always be a loud +error. + +A protocol extension supplies default implementations for all of the +above (falling back to the plain, unprojected `query`), plus a family of +as-of temporal query overloads. As-of querying means asking what a row +looked like at a past point in time, identified by an `AsOfCoordinate` +(a type defined in the `SubstrateTypes` dependency). The default +implementation of the as-of overloads answers `.present` queries normally +and throws `StorageError.featureGated` for any `.asOf(hlc)` request — the +feature is deliberately turned off until two other in-flight pieces of +work (lineage-wide expunge and the erasure overlay) both ship, because +turning it on early could let an as-of read resurface content that was +supposed to have been erased. Finally, this file declares the write- +transaction boundary — `beginTransaction`, `commitTransaction`, +`rollbackTransaction` — as protocol requirements rather than +protocol-extension defaults. This distinction matters for a subtle +Swift reason: a protocol requirement dispatches to the concrete type even +when called through an `any RowStore` existential, while a +protocol-extension default does not always do so reliably. Because +`CachingRowStore` must forward these calls to whatever it wraps, they +have to be true requirements. + +### BlobStore.swift + +This file provides the blob input/output protocol: `put`, `get`, +`delete`, `exists`, `size`, and `listKeys`. A blob is a chunk of raw +bytes identified by an arbitrary string key — typically a +content-addressed hash or a row's UUID combined with a column name. +`listKeys()` exists specifically so the replication primitive can +enumerate every blob in a backend for a full-snapshot copy; because +key order is unspecified and may differ between backends and even +between calls, any caller that needs a stable order (replication does) +sorts the returned keys itself. + +### AuditLog.swift + +This file provides the append-only history protocol. An audit log +records every change made to a row, in order, and never edits or removes +a past entry. `append` and `appendBatch` are both idempotent on the +compound key `(eventID, hlc)` — HLC stands for Hybrid Logical Clock, a +timestamp format that orders events consistently even across multiple +devices without perfectly synchronized clocks — so replaying the same +event twice (which can happen during sync) never creates a duplicate. +`iterate(after:rowID:limit:)` walks the log in HLC order and accepts a +resume cursor, and `eventsForRow(_:)` returns just one row's history for +building a point-in-time projection of that row. PersistenceKit's role +stops at durable, ordered storage of these events; enforcing the +conflict-resolution rules of a CRDT (a data structure designed so +concurrent edits always merge the same way) belongs to a higher kit. + +### StorageObserver.swift + +This file provides the live change-notification protocol, plus four +supporting event types. `StorageEvent` is the three kinds of row change +(`insert`, `update`, `delete`); `TableChange` bundles one such event with +its table, row key, values, and HLC. `BlobEvent` and `BlobChange` are the +blob-store analogues — a `put` event carries the written bytes so a +subscriber (specifically, the incremental replication session) can avoid +a second round-trip just to re-read what was written. `DirtyChainEvent` +is a third, more specialized notification: when a row in a table marked +hashable is written, the write also carries the row's new content hash +and its two nearest ancestors in a Merkle-style containment hierarchy — +the minimum information a downstream consumer needs to incrementally +recompute an integrity tree without rescanning it from scratch. + +`StorageObserver` itself declares three subscription methods — +`observe(table:events:)`, `observeBlobs()`, `observeDirtyChain()` — each +returning an `AsyncStream`, a Swift type for values that arrive over time. +Delivery is documented as at-least-once, and ordering is preserved within +one subscription but not promised across different tables. A protocol +extension supplies a default no-op stream for `observeDirtyChain()` so +older observer implementations, written before hash-on-write existed, +keep compiling without modification. + +### StorageIntrospection.swift + +This file provides an optional, separate capability protocol for +reporting backend health, plus the `StorageStats` value type that carries +the numbers. `StorageIntrospection` is deliberately not merged into +`Storage` itself — it is a distinct protocol so that adding it never +breaks an existing conformer, and a caller checks for the capability with +`storage as? StorageIntrospection` rather than assuming every backend has it. + +`StorageStats` holds a superset of fields that no single backend fully +populates: page-level statistics such as `pageSize` and `walFrameCount` +are SQLite-only; buffer cache and transaction counters such as +`cacheHitRatio` and `deadlockCount` are PostgreSQL-only; `rowCount` and +`blobCount` are InMemory-only. A backend that cannot supply a given field +sets it to `nil` rather than fabricating a zero, so a caller can tell "not +measured" apart from "measured as zero." The file's doc comment includes a +field-by-field table cross-referencing exactly which backend fills which +field — the authoritative reference for anyone adding a new statistic. + +### Transaction.swift + +This file provides the transaction protocol and the three isolation +levels a caller may request: `readCommitted`, `repeatableRead`, and +`serializable`. `StorageTransaction` exposes the same three sub-stores +`Storage` does (`rowStore`, `blobStore`, `auditLog`) so that code written +inside a transaction block looks exactly like code written outside one — +the only difference is that every operation inside the block either all +commits together or all rolls back together. The file's own comment notes +that nested transactions and savepoints are out of scope for this version +of the design. + +### StorageError.swift + +This file provides the single closed error type every PersistenceKit +operation can throw. Being a closed `enum` (as opposed to an open +protocol) means every possible failure is enumerated in one place, so a +caller's `switch` over a `StorageError` can be exhaustive. The cases cover +backend availability, schema and migration failures, constraint and +uniqueness violations, connection-pool exhaustion, transaction conflicts, +type mismatches, and three especially important safety cases: +`corruptStoredValue` (a stored value could not be parsed back to its +declared type — thrown instead of silently substituting a fabricated +default, because a fabricated UUID or an epoch-zero date would be a +quiet data-identity lie), `invalidConfiguration` (an `EstateConfiguration` +requests something impossible on the current platform, such as an +Apple-only tagger on a non-Apple host), and `invalidIdentifier` (a +caller-supplied SQL column or table name contains characters outside the +safe set, which is the guard that keeps a dynamically-built query from +becoming a SQL-injection vector even though the name is already +double-quoted). + +### Column.swift + +This file provides `Column`, a `(table, name)` pair used throughout the +predicate and schema types to reference one column, and `ColumnType`, the +closed set of value kinds a column can hold: `uuid`, `bitmap`, `text`, +`timestamp`, `float`, `int`, `bool`, `blob`, `json`, `hlc`, and +`fingerprint`. `Column` conforms to `Comparable` (ordered first by table, +then by name) purely so test fixtures and other code that sorts columns +gets a stable, repeatable order. + +### TypedValue.swift + +This file provides `TypedValue`, the tagged union that carries every +value crossing the PersistenceKit boundary — the wire format of the whole +library. Every backend pattern-matches on the case and emits its own +native representation: SQLite might store a `.uuid` as a `TEXT` column, +PostgreSQL might store it as a native `UUID` column, but callers on both +sides always see the same Swift enum. The case set is deliberately +closed; the file's own comment says adding a new case is expected to +require updating every backend, and that this cost is the right price to +pay for true backend portability — a new value kind can never partially +work on only some backends. Two of the twelve cases, `.hlc` and +`.fingerprint`, exist specifically because PersistenceKit is forbidden +from reimplementing substrate math: the `HLC` and `Fingerprint256` types +themselves come from the `SubstrateTypes` dependency, and PersistenceKit +only needs a slot to carry them. + +### Predicate.swift + +This file provides `StoragePredicate`, the closed query-condition tree +every backend compiles to its own query language, plus `OrderClause` and +`OrderDirection` for sorting. The predicate cases fall into three +families: logical (`and`, `or`, `not`, `isTrue`, `isFalse`), comparison +(`eq`, `neq`, `lt`, `lte`, `gt`, `gte`, `isNull`, `isNotNull`, `in`, +`like`), and bitmap (`bitmaskAll`, `bitmaskAny`, `bitmaskNone`, +`bitwiseEq`, all restricted to integer-family columns). PersistenceKit +treats a predicate as opaque data except when compiling it — no backend +ever needs to special-case another backend's query shape, because the +tree is the same tree everywhere. + +The two static helpers `all(_:)` and `any(_:)` build an `and`/`or` tree +from a list of predicates with useful short-circuiting: an empty list of +conditions to AND together collapses to `isTrue` (an always-true filter, +because "no restrictions" should match everything), an empty list to OR +together collapses to `isFalse`, and a list already containing a +contradictory trivial case (`isFalse` inside an `all`, `isTrue` inside an +`any`) collapses the whole expression immediately rather than building a +needlessly large tree. + +### GeneratedColumn.swift + +This file provides first-class computed columns: a column whose value is +derived from other columns in the same row by a small, structured integer +expression, rather than stored directly by a caller. `GeneratedColumn` +names the column, its result type, and its `GeneratedExpression`. +`GeneratedExpression` is a closed, recursive enum covering exactly the +bit-field algebra a bitmap-heavy schema needs: reading another column +(`.column`), a constant (`.literal`), bitwise AND/OR/XOR, left and right +shift by a fixed amount, and equality/inequality tests that evaluate to 1 +or 0. The file's own comment explains why this is a structured expression +tree and not a raw SQL string: a SQL string generated column would push +backend-specific syntax into a schema declaration, and the in-memory +backend has no SQL engine to evaluate it against. A structured expression +has exactly one meaning, so SQLite and PostgreSQL render it to native +`GENERATED ALWAYS AS (...) STORED` DDL and the in-memory backend evaluates +it directly — three faithful realizations of one description, with no +escape hatch for one backend to interpret it differently from another. + +`renderSQL()` turns an expression into a SQL fragment shared by SQLite and +PostgreSQL, because both use the same bitwise-operator syntax and +double-quoted identifiers; the one exception is bitwise XOR, which SQLite +lacks as an operator, so both backends instead render it as the +algebraically equivalent `(a | b) - (a & b)`. `evaluate(_:)` computes the +same result directly against an in-memory row dictionary, for the +in-memory backend. `integerValue(_:)` extracts an `Int64` from any +integer-family `TypedValue` (an `.int`, `.bitmap`, `.bool`, or `.hlc`), +returning zero for anything else or for an absent column, which is the +in-memory evaluator's sentinel for "no meaningful value here." + +### EstateConfiguration.swift + +This file provides `EstateConfiguration`, the one value that fully +describes how to open one estate's storage, and `BackendConfiguration`, +the closed choice of physical engine (`sqlite`, `postgresql`, `inMemory`) +with its connection parameters. `EstateConfiguration` also carries the +estate's encryption mode, cache configuration, and novel-token tagger +choice, each defaulted so that existing code that does not mention these +newer fields keeps compiling and keeps behaving exactly as it did before +they existed — a plaintext, uncached, HMM-tagged estate is what a caller +gets by simply not asking for anything else. + +`queueSibling(filename:)` is a more specialized function: it derives a +second `EstateConfiguration` pointing at a companion database file that +sits beside the estate's own file — used, for example, so a work-queue +kit gets its own small database without needing separate key +distribution or a separate configuration path. For a SQLite estate at +`/.sqlite`, asking for the sibling `"queue.sqlite"` produces +`/.queue.sqlite` — the estate's own file stem is folded into +the sibling name so two different estates in the same directory can never +collide. For an in-memory estate, the sibling is also in-memory (both +ephemeral, which is the correct pairing for tests). For a PostgreSQL +estate, the function throws `StorageError.featureGated`: the design +deliberately defers a PostgreSQL queue sibling until later, and it fails +loudly rather than silently returning a half-working configuration. + +The private helper `deriveQueueSiblingID(parentID:filename:)` computes the +sibling's estate ID deterministically — by XOR-folding the filename's +UTF-8 bytes into a 16-byte tag and then XOR-ing that tag with the parent +UUID's raw bytes — so the same parent estate and the same filename always +produce the same sibling ID, with no call to `UUID()` anywhere on this +path. Determinism here matters because every process that opens the same +estate must agree on the same sibling ID without any coordination. + +### EstateCacheConfig.swift + +This file provides `EstateCacheConfig`, the small value type that turns +row caching on or off for one estate and bounds how much a cache is +allowed to hold and how sensitive the content it holds may be. +`ceilingBytes` is clamped to be non-negative at construction, and +`sensitivityThreshold` is clamped to at most 2 — because sensitivity level +3, "Secret" in the ARIA adjective scale this kit shares with the rest of +the system, must never be cached under any configuration, and clamping at +construction enforces that invariant without relying on every caller to +remember the numeric boundary. `EstateCacheConfig.disabled` is the +zero-change default: an estate that never mentions caching gets exactly +the pre-caching behavior. + +### EncryptionMode.swift + +This file provides the three at-rest encryption modes and the +configuration value that carries one of them per estate. +`EncryptionMode.plaintext` stores content verbatim — encryption, if any, +happens only at a sharing boundary elsewhere in the system. +`.rowEncryption` (Mode 2) encrypts one column's content per row under a +key the row's own `keyID` column names. `.fullDatabase` (Mode 3) encrypts +the entire SQLite file, schema included, via SQLCipher's `PRAGMA key` at +the connection layer, which makes the per-row seam a no-op for that mode +because the whole file is already ciphertext on disk. A fourth mode, +database-plus-threshold encryption for a FedRAMP-tier deployment, is +deliberately absent from this enum — the file's comment stresses that the +capability does not exist in this build at all, so adding it later is a +conscious, reviewed act rather than an accidental unlock. + +`EstateEncryptionConfig` bundles the mode with a key identifier and the +actual `SymmetricKey`. Its stored `key` property is `package`-scoped +rather than `public`: the SQLite backend, in a sibling module of the same +Swift package, needs it, but nothing outside the package should ever see +it. The convenience initializer `init(_ mode:)` mints a fresh, full-entropy +256-bit key and a UUID key identifier for either encrypting mode, and +mints neither for plaintext. `usesRowCrypto` is `true` only for Mode 2 — +Mode 3 protects everything at the file level, so the per-row seam has +nothing to do there. `fullDatabaseKeyHex` renders the whole-file key as +lowercase hex for the SQLCipher `PRAGMA key` statement and is documented +as something that must never be logged. + +### RowCrypto.swift + +This file provides the per-row AES-GCM-256 encryption used by +`EncryptionMode.rowEncryption`, plus the three write/read seam functions +that every backend calls at the same two moments: just before a write and +just after a read. Living in `PersistenceKit` core rather than in one +backend module is the point: both `PersistenceKitSQLite` and +`PersistenceKitPostgreSQL` call the exact same code, so the two backends +produce byte-compatible ciphertext envelopes without either one knowing +about the other. + +`AeadProvider` is a small protocol seam — AEAD stands for Authenticated +Encryption with Associated Data, a class of algorithm that both encrypts +data and proves it was not tampered with. Any type conforming to +`AeadProvider` can replace the default algorithm without touching any +call site; the file names this as the extension point a future +FIPS-validated provider would use. `CryptoKitAeadProvider` is the default +implementation, backed by Apple's CryptoKit. It generates a fresh random +96-bit nonce on every single encrypt call — reusing a nonce under the same +key is the one mistake that breaks AES-GCM's security guarantee — and +returns the three parts concatenated as `[12-byte nonce][16-byte +tag][ciphertext]`, so a later decrypt is self-contained from the stored +bytes alone with nothing else to look up. + +`encryptedForWrite(_:config:provider:)` is the write-side seam: for an +estate using row encryption, it encrypts the row's `content` column and +stamps a `keyID` column recording which key sealed it; for any other mode, +or for a row with no `content` column, it returns the values unchanged. +`decryptedForRead(_:config:provider:)` reverses this on read, but only +when the row's stored `keyID` matches the estate's own key identifier — a +mismatched `keyID` means the row was sealed under a key this estate does +not hold, so the function passes the row through as still-ciphertext +rather than attempting a decrypt that would only fail as an authentication +error. `assertContentKeyIDInvariant(_:table:config:)` is a final structural +guard: on an encrypting estate, any content-bearing row reaching a backend +write path with `.text` content and no `keyID` means the encryption seam +did not run somewhere upstream, and the function throws rather than let +that plaintext be persisted where a later read could never recover it. + +### CachingRowStore.swift + +This file provides `CachingRowStore`, a decorator that wraps any +`RowStore` and serves an in-memory hot tier of recently-read rows, +transparently — the file's own comment states the guarantee plainly: +every operation returns results identical to what the unwrapped backing +store would have returned, and the cache changes only latency, never +correctness. + +The cache key is not just `(table, row key)`; it is that pair combined +with an `AsOfCoordinate`. A `.present` (current-state) read and an +`.asOf(hlc)` (past-state) read of the same row are cached as two entirely +separate entries, because a snapshot read against a pinned, immutable +past state can be cached forever — the GC pin mechanism guarantees that +data will not be vacuumed out from under it — while a present-state read +must be evicted the instant the row changes. `insert`, `upsert`, +`update`, and `delete` all evict the affected present-state entry (or, for +a batch predicate that does not identify one specific row, evict every +present-state entry for that table) after the underlying write succeeds. +An optional `parentChainProvider` closure, supplied by the calling kit, +lets a write also evict cached Merkle-aggregate entries for that row's +ancestors, so a cached rollup value never goes stale just because one leaf +changed underneath it. + +Admission to the cache is sensitivity-gated. `isAdmissible(_:)` reads a +row's `provenance` bitmap column (when present) and decodes a six-bit +sensitivity field at bits 30–35, mapping the scale-gapped raw values 0, +16, 32, and 48 to the ordinals Normal, Elevated, Restricted, and Secret. +Secret content is rejected unconditionally, and anything above the +configured `sensitivityThreshold` is rejected too. An unrecognized bit +pattern, or a `provenance` value of the wrong `TypedValue` case, is also +rejected — the comments call this failing closed, meaning that any doubt +about a row's sensitivity keeps it out of the cache rather than risking +that sensitive content sits unencrypted in process memory longer than it +should. Eviction, when the estimated byte total exceeds the configured +ceiling, removes the least-recently-used entry first, tracked by a +monotonically increasing access counter rather than by wall-clock time. + +All of this mutable state — the entry dictionary, the access counter, the +running byte total — lives inside a private `actor`, `CacheActor`. Because +every stored property of the actor is itself `Sendable`, wrapping the +state in an actor is what lets the outer `CachingRowStore` class, a +`final class` with no locks of its own, safely conform to `Sendable` under +Swift 6's strict concurrency checking. + +### CacheInvalidator.swift + +This file provides `CacheInvalidator`, the bridge between a +`StorageObserver`'s live change stream and a `CachingRowStore`'s +invalidation method — needed for the case where some other writer +bypasses the specific `CachingRowStore` instance entirely (for example, +a second process, or a raw connection opened for a migration) and +mutates the backing store directly. Without this bridge, the cache would +serve stale data forever after such a write, because it would never learn +the write happened. + +One `CacheInvalidator` manages every table it is told to watch through a +single detached background task that fans out one child task per table, +using Swift's structured-concurrency `withTaskGroup`. `cancel()` cancels +the whole tree at once, and `deinit` calls `cancel()` automatically so a +caller who simply lets the invalidator go out of scope does not leak a +background task. The initializer's documentation is careful to flag one +narrow race: subscriptions start inside the background task, so a write +issued in the same instant as `init()` could in principle race ahead of +subscription registration; a caller with strict-ordering requirements is +advised to yield briefly (for example with `Task.sleep`) before relying on +the very first write being observed. + +### HashingRowStore.swift + +This file provides `HashingRowStore`, a decorator that intercepts writes +to any table marked hashable in the schema, computes a content hash for +the row, and emits a `DirtyChainEvent` so an integrity tree elsewhere in +the system can be kept current incrementally rather than rebuilt from +scratch on every write. PersistenceKit deliberately does not import a +hashing library itself; the actual hash function arrives as an injected +`ContentHashProvider` closure supplied by the calling kit (which does +import the hashing library), so this package stays free of that +dependency while still owning the mechanism that wires hashing into every +write path. + +The trickiest part of this file is keeping the hash correct on partial +writes. An `insert` always carries the row's full column set, so hashing +the incoming `values` directly is correct. An `update`, however, typically +carries only the columns being changed — hashing just that partial +dictionary would produce a hash for a row that never actually existed in +storage. `augmentWithHashForKnownKey(table:rowKey:mergedValues:)` solves +this by pre-reading the row's current, full state, merging the incoming +changes on top of it, and hashing that merged result — the same approach +`upsert` uses when it resolves to an update against an existing row rather +than a fresh insert. Before hashing, the function strips any existing +`content_hash` column from the input, so the insert path and the +update-via-merge path always hash the same set of data columns and never +diverge just because one of them happened to already carry a stale hash +value. + +`emitDirtyChain(table:rowKey:hashResult:)` delivers the resulting +`DirtyChainEvent` to an injected `ObserverRegistryRef` closure, which is +`nil` for backends without observer support — the hash is still computed +and stored, but the notification step is simply skipped. + +### ErasureLedger.swift + +This file provides the append-only record of what has been erased, +without ever storing the erased content itself. A "drawer," in this +context, is one unit of stored content (the term comes from the estate's +own filing metaphor, not from this package). `ErasureLedgerEntry` pairs a +`drawerId` with the `HLC` at which it was erased. The `erasure_ledger` +table this module declares is marked append-only in its +`TableDeclaration` — every backend enforces this at the storage layer, so +even a caller that somehow bypasses the ledger's own API cannot mutate or +remove an entry once written, which is exactly the tamper-evidence +property an erasure record needs. + +`recordErasure(rowStore:drawerId:erasedHlc:)` inserts one entry and +throws `StorageError.duplicateKey` if that drawer was already erased — +each drawer is erased exactly once. `isErased(rowStore:drawerId:)` is the +fast point-lookup used on every read that might need to hide erased +content; `lookupErasure(rowStore:drawerId:)` returns the full entry when a +caller needs the erasure timestamp itself, not just a yes/no answer. + +### ErasureOverlay.swift + +This file provides the two-phase, fail-closed filter that actually hides +erased content from a query result, built on top of the ledger this +module does not itself define. `ErasureOverlayConfig` is supplied by a +higher kit and carries two pieces of entity-specific knowledge +PersistenceKit does not have on its own: `extractErasureId`, a closure +that pulls the erasure-ledger key out of a result row (or returns `nil` +for a row type that is not subject to erasure at all), and +`contentColumns`, the list of column names to null out when a row turns +out to be erased. + +`ErasureOverlay.apply(rows:config:rowStore:)` runs the two phases the file +describes at the top: phase one is an ordinary query that already +happened before this function is called; phase two walks the returned +rows one at a time, checks each one's erasure ID against the ledger, and +nulls its content columns if erased. The important design decision is +what happens when the ledger check itself throws an error instead of +returning a clean yes-or-no answer: the row is dropped from the result +entirely. This is the fail-closed half of "two-phase fail-closed" — an +uncertain answer about whether a row was erased is treated the same as +"it was," because showing content that should have been erased is a worse +failure than temporarily hiding content that was not. + +### GCPin.swift + +This file provides the query that tells a garbage-collection pass which +rows it must not touch yet. GC, short for garbage collection, is any +maintenance pass that reclaims space by deleting rows that are no longer +needed — for example, old versions of a row that a new version has +superseded. The problem this file solves: a snapshot taken earlier (via +`SnapshotRegistry`) might still need to read an "old" row that the GC pass +would otherwise consider safe to delete. + +`GCPin.minimumRetainableHlc(rowStore:)` answers this by finding the +smallest HLC across every currently-registered snapshot — the oldest live +snapshot's timestamp is the pin. Any row whose HLC is at or after that +pin must survive a GC pass; anything strictly older is safe to reclaim. +When no snapshots exist at all, the function returns `nil`, meaning +nothing is pinned and every row is fair game for GC. `isPinned(rowStore: +rowHlc:)` is the convenience per-row check built on top of the same query. + +### SnapshotRegistry.swift + +This file provides the durable record of named point-in-time snapshots +and the cryptographic attestations that accompany them. A snapshot, in +this design, is not a copy of any data — it is a registry row recording +one HLC (the moment the snapshot was taken) plus one or more attestation +rows recording what a Merkle root looked like at that moment for a given +subject. PersistenceKit itself knows nothing about what a "subject" is — +the `subjectKind` and `subjectId` strings are supplied by whichever +higher-level kit is taking the snapshot. + +`SnapshotId` is a UUID-backed opaque identifier, minted fresh by +`SnapshotId.mint()`. `SnapshotRegistryOps.createSnapshot(rowStore:hlc: +label:createdAt:attestations:)` mints an ID, inserts one registry row, and +inserts one attestation row per subject passed in, returning the +completed `SnapshotRecord` to the caller. `listSnapshots(rowStore:)` walks +every registered snapshot in HLC order; `deleteSnapshot(rowStore: +snapshotId:)` removes a snapshot's attestation rows before removing its +registry row (child rows first, so no orphaned attestation can ever +reference a deleted registry entry) and reports whether anything existed +to delete. `attestations(rowStore:snapshotId:)` reads back every +attestation for one snapshot, ordered by subject for deterministic +output. The remaining private helpers handle the mechanical work of +turning a `StorageRow` into a typed `SnapshotRecord` or +`SnapshotAttestation` and back. + +### NoOpObserver.swift + +This file provides `NoOpObserver`, a trivial `StorageObserver` +conformance whose three subscription methods each return a stream that +finishes immediately, delivering nothing. It exists as a placeholder for +any backend or test path that has no real change-notification mechanism +yet but still needs to satisfy the `Storage` protocol's requirement for +an `observer` property — the PostgreSQL backend, for instance, uses this +type directly rather than implementing a live PostgreSQL notification +channel. + +### NovelTokenTaggerChoice.swift + +This file provides the estate-creation-time choice of which tagger +classifies a "novel token" — a word not found in a language model's +existing vocabulary — when an estate needs to do that classification work +somewhere in its processing pipeline. The two cases are `.hmm`, a +deterministic Hidden Markov Model tagger that produces the exact same +output on every platform and is therefore safe to use across a federation +of devices, and `.nlTagger`, Apple's `NaturalLanguage` framework tagger, +which can be more accurate on Apple hardware but is not available outside +Apple platforms and is not guaranteed to produce the same output across +different OS versions. + +This choice is fixed permanently at estate creation in this version of +the design; changing it later, with re-tagging of existing content, is +explicitly deferred to a future version. The file's extensive comments +spell out the practical consequence: an estate created with `.nlTagger` +cannot safely federate (share and compare memories) with an +`.hmm`-tagged estate without re-tagging all of its content first, because +the two taggers can disagree on the same word. Enforcing that constraint +automatically is also deferred; for now, a caller who selects `.nlTagger` +is responsible for not mixing that estate into a federation with +incompatible estates. `NovelTokenTaggerChoice.default` is `.hmm` — the +safe, federation-compatible baseline every estate gets unless a caller +explicitly opts into the Apple-only alternative. An identically-named, +independently-defined enum exists in `LatticeLib`, in a different package; +the two packages deliberately do not share this type by importing one +from the other, so that neither package depends on the other. + +### PersistenceKitTelemetry.swift + +This file provides `reportStorageStats(_:estateID:now:)`, the function +that turns a `StorageIntrospection.stats(now:)` snapshot into a stream of +named metrics through `IntellectusLib`, a lightweight, zero-dependency +telemetry library. The function's entire behavior is gated on +`Intellectus.isEnabled`, which defaults to `false`. When telemetry is +disabled — the default for essentially every deployment — the function +returns after a single atomic boolean load, without ever calling +`stats(now:)` and without building a single metric object; the file's +comment quantifies this cost at roughly one nanosecond, making the +disabled path effectively free. + +When telemetry is enabled, the function emits one metric per non-`nil` +field of the captured `StorageStats`, each under the `persistence.db.*` +namespace and each tagged with the emitting kit's name and the estate's +ID, so a monitoring backend can filter and group by estate. Because +`StorageStats` fields are `nil` exactly where a backend cannot supply +them, the emitted metric set naturally differs by backend without this +function needing to know anything about which backend produced the +snapshot — an in-memory estate simply never emits a `wal_frames` metric, +because that field was never populated in the first place. Every +timestamp passed to `Intellectus.report(_:)` comes from the caller-supplied +`now` parameter, never from a fresh `Date()` call inside this function, +which is a deliberate determinism rule followed throughout the whole +package: any code that might run identically twice must never read the +wall clock itself. + +## Target: PersistenceKitInMemory + +### InMemoryStorage.swift + +This file provides `InMemoryStorage`, the backend used for tests and for +any storage that should live only as long as the current process, plus +the `InMemoryStateActor` that owns every piece of mutable state behind +Swift's actor isolation. There is no file on disk and no network +connection; every table is a plain Swift dictionary held in memory. + +The most carefully documented piece of this file is how `transaction( +isolation:_:)` implements rollback. Rather than running the caller's +block against a private, detached copy of the state and only replacing +the live state on success, it runs the block directly against the live +actor state and takes a snapshot copy only for the error path. The file's +own comment explains why the seemingly simpler "copy, mutate, replace on +success" approach is actually wrong: if any other write reaches the live +actor state between the moment the snapshot was taken and the moment a +successful transaction would replace the whole state with its +snapshot-derived copy, that concurrent write would be silently erased by +the replacement. The comment traces this exact bug to a real incident — a +burst of concurrent inserts racing a transaction lost five to ten percent +of queued work. Running against live state avoids the problem entirely, +at the cost of needing an explicit rollback-to-snapshot path when the +block throws. + +Change notifications during a transaction are buffered rather than +delivered immediately, for a related reason: if a subscriber were notified +of a write that the transaction later rolled back, it would have observed +something that, from the estate's point of view, never actually happened. +`beginNotificationBuffering()` starts collecting notifications instead of +delivering them; `commitNotifications()` flushes the buffer to observers +only after the block has returned successfully; the rollback path +discards the buffer instead. `insertRow`, `upsertRow`, `updateRows`, and +`deleteRows` are the four mutating operations, each of which computes any +declared `GeneratedColumn` values via the static +`materializeGenerated(_:_:)` helper before storing the row, matching what +SQLite and PostgreSQL compute natively so a query against any backend +returns the same generated values. `queryRows(...)` applies the predicate +and ordering against full rows first and only narrows to the requested +column projection at the very end, so a projected query can still filter +or sort on a column it does not actually return — matching SQLite's +`SELECT`-list-versus-`WHERE`/`ORDER BY` semantics exactly. + +`InMemoryStorage` also conforms to `StorageIntrospection`, reporting an +approximate byte size (a flat per-row estimate plus exact blob byte +counts — described in the file as a rough signal, not a precise +allocator measurement), a live row and blob count, and a monotonically +increasing rollback counter. + +### InMemoryRowStore.swift + +This file provides the thin `RowStore` conformance that simply forwards +every call to the shared `InMemoryStateActor`. Its one piece of design +worth noting is the column-projection overload: because the in-memory +backend already holds the full row in memory, projecting away unrequested +columns saves no actual transfer cost the way it does for SQLite — the +comment is explicit that the point of implementing it anyway is +consistency, so a `StorageRow` returned from a projected in-memory query +is byte-identical in shape to the same projected query against SQLite, +and code under test cannot accidentally depend on a column being present +just because the in-memory backend happened to still have it in memory. + +### InMemoryBlobStore.swift + +This file provides the `BlobStore` conformance for the in-memory backend +— five one-line forwarding methods to the shared state actor's blob +dictionary. There is nothing backend-specific to explain here beyond what +`BlobStore.swift` already documents; this file exists purely to satisfy +the protocol against the actor's storage. + +### InMemoryAuditLog.swift + +This file provides the `AuditLog` conformance for the in-memory backend, +again a thin forwarding layer to the shared state actor, which itself +de-duplicates on `(eventID, hlc)` exactly as the protocol's idempotence +contract requires. + +### InMemoryObserver.swift + +This file provides `ObserverRegistry`, the subscription bookkeeping +shared by every in-memory row store, blob store, and audit log instance +backed by the same state actor, and `InMemoryObserver`, the thin +`StorageObserver` conformance built on top of it. The registry +deliberately does not use actor isolation for its subscriber lists; +instead it uses a plain `NSLock`. The file's comment explains the +trade-off precisely: registering a subscription must be synchronous, with +the stream already recording it before `observe()` returns to the caller, +so that a change notified the very next line of code can never race ahead +of the subscription being recorded. An `actor`-based registry would force +`register` to be an `async` function, reopening exactly that race. This +mirrors the equivalent Rust observer hub, which registers synchronously +for the same reason. + +`notify(_:)` snapshots the list of matching subscribers under the lock +and then yields to each of them entirely outside the lock, so that a +subscription's own termination handler — which also needs the lock to +remove itself from the list — can never deadlock against an in-flight +notification. Row, blob, and dirty-chain subscriptions are tracked in +three separate dictionaries but follow the same synchronous-register, +lock-scoped-snapshot, unlocked-delivery pattern throughout. + +### PredicateEvaluator.swift + +This file provides the in-memory interpreter for `StoragePredicate` — +the only backend that evaluates the predicate tree directly against +Swift values in memory rather than compiling it to a query string first. +`PredicateEvaluator.evaluate(_:against:)` walks the tree recursively, +handling every case from `Predicate.swift` by reading the named column +out of a `[String: TypedValue]` row (treating a missing column the same +as an explicit `.null`) and applying the matching comparison, bitmask +test, or `LIKE`-style text match. `likeMatch(_:pattern:)` translates a +SQL `LIKE` pattern (`%` for any run of characters, `_` for exactly one) +into an `NSRegularExpression`, so the in-memory backend matches the same +patterns SQL's `LIKE` operator would. + +`TypedValueComparator.compare(_:_:)`, in the same file, is the shared +ordering function used both by predicate comparisons and by `ORDER BY` +sorting: `nil` (that is, `.null`) sorts before every other value, and two +values of the same case compare by their underlying Swift value, with +`.hlc` compared by its packed integer form (`HLC.packed`, defined at the +bottom of `InMemoryStorage.swift`) to get a single total order over an +HLC's three component fields. + +## Target: PersistenceKitSQLite + +### SQLiteStorage.swift + +This file provides `SQLiteStorage`, the `Storage` conformance for the +on-device backend, and the much larger `SQLiteBackend` actor that does +essentially all of the real work — schema migration, every row and blob +operation, audit logging, and health introspection — serialized through +one actor per estate, matching the file's opening comment that a SQLite +estate gets exactly one connection, with SQLite's own WAL (write-ahead +log) mode handling concurrent readers safely underneath. + +Schema migration walks a schema's declared `Migration` list, running each +pending migration inside its own `BEGIN IMMEDIATE`/`COMMIT` pair and +rolling back and throwing `StorageError.migrationFailed` if any operation +in that migration fails. `applyMigrations(_:)` is written to be safe to +call even on a completely fresh SQLite file that was never opened through +`openSchema(_:)` first — a documented real scenario, since at least one +caller invokes `migrate(to:)` directly — by re-running the idempotent +table- and migrations-table-creation steps before checking what actually +needs to migrate. + +Every row-mutating method — `insertRow`, `upsertRow`, `updateRows`, +`deleteRows`, `queryRows` — begins by calling `validateSQLIdentifier` on +every table and column name it is about to interpolate into a SQL string. +The file's comments tie this directly to a named security fix +(SECFIX-WS2-PK F9): double-quoting a SQL identifier is not sufficient +protection if the identifier itself can contain a double-quote character, +because that character can escape the delimiter and alter the query, so +every name is checked against a strict allow-list before it is ever +placed into a SQL string. `insertRow` and `queryRows` are also where the +at-rest row-encryption seam actually runs: `insertRow` calls +`encryptedForWrite` before binding values, and `queryRows` calls +`decryptedForRead` after reading them back, both of which are no-ops on a +plaintext or whole-database-encrypted estate. + +`readColumn(stmt:index:schema:columnName:table:)` is the file's most +carefully reasoned function: it decides, column by column, whether a raw +SQLite value should be trusted as-is or treated as corrupt. The +distinction the comment draws is between a type-tolerant decode (a valid +value stored under SQLite's flexible column affinity, which is passed +through as-is) and a genuine parse failure on a column whose declared +type says it should parse (an unparseable UUID or timestamp string), +which throws `StorageError.corruptStoredValue` rather than ever +fabricating a random UUID or an epoch-zero date in its place. +`queryRowsSkipCorrupt(...)` reuses the same column-reading logic but +catches exactly that error per row, logs it, counts it, and continues to +the next row — the cursor-level implementation `RowStore`'s protocol +default can only approximate by discarding an entire query's results. + +`storageStats(now:)` implements `StorageIntrospection` for SQLite by +reading a handful of read-only `PRAGMA` statements (`page_size`, +`page_count`, `freelist_count`) and deriving the WAL frame count directly +from the `-wal` sidecar file's size on disk rather than calling `PRAGMA +wal_checkpoint`, because that particular PRAGMA can fail with a locked +error if a concurrent operation is in flight, even from inside the same +serializing actor. + +### SQLiteConnection.swift + +This file provides `SQLiteConnection`, a thin Swift wrapper around the C +`sqlite3` API, `SQLiteStatement`, the prepared-statement wrapper used for +every parameterized query, and `ISO8601`, the shared timestamp +formatting and parsing utility used everywhere a `.timestamp` `TypedValue` +crosses the SQLite boundary. + +Opening a connection does several things in a specific, safety-relevant +order. First, it checks whether the target path is a symbolic link and +refuses to open it if so — the comment names this CAND-052, a defense +against a pre-planted symlink that could otherwise redirect SQLite's +writes to an arbitrary file elsewhere on disk, and notes that the check +uses `lstat` semantics (via `resourceValues(forKeys:)`) so it identifies +the symlink itself rather than following it to whatever it points at. +Second, if the estate uses whole-database encryption, it issues `PRAGMA +key` with the estate's hex-encoded key before any other statement runs — +this must be the very first statement, because SQLCipher cannot even read +the schema on page one of the file without the correct key already +applied. Third, it applies Apple's file-level Data Protection, a +best-effort step that layers OS-level at-rest protection on top of +SQLCipher's own encryption. Finally, it sets the durability pragmas: WAL +journal mode, `NORMAL` synchronous durability, a WAL auto-checkpoint +threshold, a busy timeout, and foreign-key enforcement. + +`ISO8601` deserves its own explanation because two very different +performance and correctness stories live in it. `string(from:)` first +clamps an out-of-range `Date` (for example one accidentally derived from +milliseconds mistaken for seconds) into the range `ISO8601DateFormatter` +can actually parse back — years 0001 through 9999 — because writing an +unparseable string like `+59009-...` would silently corrupt any future +read of that row; a clamp with a logged warning is judged better than a +value nothing can ever read back correctly. `date(from:)` tries a hand- +written, allocation-free parser, `fastParseCanonicalUTC(_:)`, before +falling back to the much slower ICU-backed `ISO8601DateFormatter`. The +comment explains why the fast path exists at all: a stack-sampling +profile during a large import showed the formatter's `date(from:)` +consuming roughly eighty percent of total CPU time, because a Merkle +rollup re-decodes every row's timestamp on every insert, making the parse +cost scale quadratically with import size. The fast parser recognizes +only the exact canonical shape this kit itself writes and returns `nil` +for anything even slightly different (a numeric time-zone offset, +lowercase letters, trailing garbage), which sends control back to the +slow, fully general formatters — so correctness never regresses, and the +fast path is verified byte-for-byte against the formatters in a dedicated +test suite. + +### SQLiteStores.swift + +This file provides `SQLiteRowStore`, `SQLiteBlobStore`, `SQLiteAuditLog`, +and `SQLiteTransaction` — four small wrapper types that each forward to +the shared `SQLiteBackend` actor, giving the actor's internal methods +their public `RowStore`/`BlobStore`/`AuditLog`/`StorageTransaction` +faces. `SQLiteRowStore.querySkipCorrupt(...)` is the one method here with +real logic of its own: rather than inheriting `RowStore`'s protocol- +extension default (which can only discard an entire failed query), it +calls `SQLiteBackend.queryRowsSkipCorrupt(...)` directly, so a corrupt +row found partway through a large corpus scan is skipped and logged +individually rather than aborting everything already read. + +### SQLiteSchema.swift + +This file provides `SQLiteSchema`, the enum of pure functions that +translate a `SchemaDeclaration` into SQLite's native DDL (data definition +language — the SQL used to create and alter tables, as opposed to the SQL +used to read and write rows). `nativeType(_:)` maps every +`ColumnType` case to its SQLite storage class — most map directly, but +`.uuid` and `.timestamp` both map to `TEXT` because SQLite has no native +types for either, `.hlc` maps to `INTEGER` because it is stored as a +packed 64-bit value, and `.fingerprint` maps to `BLOB` because it is +stored as raw bytes. `createTable(_:)` assembles one `CREATE TABLE IF NOT +EXISTS` statement covering ordinary columns, generated columns (always +rendered `STORED`, matching PostgreSQL's lack of a `VIRTUAL` form), +primary key, and unique constraints. `appendOnlyTriggers(_:)` emits a +`BEFORE UPDATE`/`BEFORE DELETE` trigger pair for any table declared +append-only, each one aborting the statement outright with `RAISE(ABORT, +...)` — the actual enforcement mechanism behind, for example, the erasure +ledger's append-only guarantee. + +This file also owns the three internal bookkeeping tables every SQLite +estate carries regardless of what schema a caller declares: +`_storagekit_migrations` (tracks each kit's applied schema version), +`_storagekit_audit` (the single source of truth for every audit event, +with full-precision HLC columns alongside the packed integer form — the +packed form loses a bit of the physical-time field for dates far enough +in the future, so the full-precision columns exist specifically so a +cold rebuild of an estate's last-known HLC never silently disagrees with +what a snapshot originally recorded), and `_storagekit_blobs` (the flat +key/bytes table backing `SQLiteBlobStore`). + +### SQLitePredicateCompiler.swift + +This file provides the translation from a `StoragePredicate` tree into a +parameterized SQLite `WHERE` clause plus its ordered list of bound +values. `compile(_:)` is the public entry point; the private recursive +`render(_:bindings:)` walks the tree case by case, validating every +column name it touches with `validateSQLIdentifier` before it is +interpolated, and appending every comparison value to the bindings array +rather than ever interpolating a value directly — values are always safe +because SQLite's own parameter-binding mechanism handles them, but column +and table identifiers cannot be bound as parameters in SQL, which is +exactly why they need the separate identifier-validation guard instead. + +### SQLiteIdentifierValidator.swift + +This file provides `validateSQLIdentifier(_:)`, the single free function +that every write path and the predicate compiler in this module call +before interpolating any caller-supplied name into a SQL string. The rule +is simple and strict: the first character must be a letter or an +underscore, and every following character must be a letter, a digit, or +an underscore — the safe subset of SQLite identifier syntax. The file's +own comment states the reason this exists as one shared function rather +than several private copies: a name containing a double-quote character +can escape the double-quote delimiter SQL uses around identifiers and +change what the query actually does, so double-quoting alone is not a +sufficient defense, and having one seam means the defense cannot silently +drift out of sync between the several call sites that need it. + +### SQLiteObserver.swift + +This file provides `SQLiteObserverRegistry`, an `actor` holding the +subscriber lists for both row-change and blob-change notifications on one +SQLite estate, and `SQLiteObserver`, the public `StorageObserver` +conformance built on top of it. The file's own comment explains a real +limitation of SQLite's native `sqlite3_update_hook`: it fires for every +table, including the internal blob table, but it only reports which +operation happened on which table and row ID — it never carries the +actual column values. Because a blob-change notification needs to carry +the bytes that were written, blob notifications cannot be reconstructed +from the hook at all; instead, `SQLiteBackend`'s `putBlob`/`deleteBlob` +methods call this registry's `notifyBlob(_:)` directly, at the exact +point where the bytes are already in hand. + +### KeychainKeyStore.swift + +This file provides `KeychainKeyStore`, the Apple-platform source of the +whole-database (Mode 3) encryption key — the Apple counterpart to the +Rust port's per-estate key file on disk. One 256-bit key is stored per +estate in the system Keychain, under an account name derived +deterministically from the estate's own file path (`estateAccount(for:)` +standardizes the path and hashes it with SHA-256), so any process that is +told the same estate's file path — the main app and a managed +background server it spawns, in particular — computes the same account +name and therefore loads the same key without any separate key- +distribution step. `loadOrCreateKey()` reads an existing key if one is +present, or generates a fresh cryptographically random key and stores it +if not, handling the race where two callers try to create the key +simultaneously by having the loser simply re-read what the winner wrote. +`deleteKey()` disposes of an estate's key permanently when the estate +itself is deleted, and is documented as idempotent — calling it on an +estate whose key is already gone is a success, not an error. The stored +key's Keychain accessibility level, `afterFirstUnlockThisDeviceOnly`, is +chosen specifically so a background process can still read the key after +the very first unlock following a device restart, without the key ever +leaving the device or syncing to iCloud. + +## Target: PersistenceKitPostgreSQL + +### PostgreSQLStorage.swift + +This file provides `PostgreSQLStorage`, the `Storage` conformance for the +server backend, and the `PostgreSQLBackend` actor that holds the schema +declaration and drives migrations, transactions, and health +introspection against a connection pool. The most distinctive design +decision here is estate isolation: rather than one PostgreSQL database +per estate, every estate gets its own PostgreSQL schema — a namespace +within one shared database — named `pk_`. Every pooled connection for that estate pins its +`search_path` to that schema (keeping `public` on the path too, so any +shared extension still resolves), which is the PostgreSQL analogue of +SQLite's one-file-per-estate model: many estates can share one database +server without their tables ever colliding. + +Schema-version bookkeeping uses a simple key-value table, +`_storagekit_meta`, rather than a dedicated migrations table like +SQLite's: a per-kit version is stored under the composite key +`"schema_version:"`, and a running-maximum global version is kept +under the plain key `"schema_version"` so the no-argument +`currentSchemaVersion()` still returns something meaningful even when +several kits share one estate. `storageStats(now:)` implements +`StorageIntrospection` for PostgreSQL by querying `pg_database_size`, +`pg_stat_database` (for buffer-cache hit ratio and transaction counters), +and `pg_locks` joined against `pg_database` (for lock contention) — three +different system views, each documented in the file with the exact +formula used to turn its raw counters into the `StorageStats` fields. + +### PostgreSQLPool.swift + +This file provides `PostgreSQLPool`, a fixed-size connection pool built as +an `actor` so its bookkeeping — available connections, in-use count, +waiters — never needs a separate lock. `acquire()` returns an idle +connection immediately if one exists, opens a brand-new one if the pool +has not yet reached its configured size, or otherwise suspends the caller +on a `CheckedContinuation` until either a connection frees up or a timeout +elapses, at which point the waiter is resumed with +`StorageError.poolExhausted`. Every freshly-opened connection is +immediately pinned to the estate's dedicated PostgreSQL schema by issuing +`CREATE SCHEMA IF NOT EXISTS` followed by `SET search_path`, and the +connection is closed rather than kept if that two-statement setup fails +partway through, so a half-configured connection is never handed back to +a caller. + +`parseTLSMode(host:)` resolves the pool's TLS behavior from the +`ARIA_MCP_POSTGRES_TLS` environment variable, defaulting to `prefer` (try +TLS, fall back to plaintext if the server does not offer it) whenever the +variable is absent or holds an unrecognized value — the comment is +explicit that even a loopback connection defaults to `prefer` rather than +`disable`, so a caller who genuinely wants plaintext on loopback has to +say so explicitly rather than getting it as an accidental default. + +### PostgreSQLConnection.swift + +This file provides free functions that bridge between PersistenceKit's +`TypedValue` and the `postgres-nio` client library's own binding and +decoding types. `executeSimple(_:logger:)` and +`executeParameterized(_:bindings:logger:)` are small `PostgresConnection` +extension methods that wrap every underlying `postgres-nio` error in +`StorageError.backendError`, so a caller never has to catch a +`postgres-nio`-specific error type directly. `makeBindings(_:)` converts +an array of `TypedValue` into the positional `$1, $2, ...` bindings +PostgreSQL's parameterized-query protocol expects, handling each +`TypedValue` case's specific wire representation — a `.fingerprint`, for +example, is serialized as 32 raw bytes in a fixed block order so it +round-trips exactly. + +`decodeRow(_:columns:)` and the private `decodeCell(_:type:)` do the +reverse: given a returned `PostgresRow` and the caller's expected column +types, they decode each named cell into the matching `TypedValue` case, +falling back to `.null` if a cell fails to decode as its declared type +rather than throwing — a deliberately more permissive stance than the +SQLite backend's `corruptStoredValue` throw, reflecting that PostgreSQL's +own wire protocol already enforces column types at a lower level than +SQLite's flexible column affinity does. + +### PostgreSQLPredicateCompiler.swift + +This file provides the PostgreSQL analogue of `SQLitePredicateCompiler`: +the same `StoragePredicate` tree compiled to PostgreSQL's own SQL dialect, +using `$1, $2, ...` positional parameters instead of SQLite's `?` +placeholders. The case-by-case translation logic is otherwise the same +shape as the SQLite compiler — every column name is validated with +`validatePSQLIdentifier` before being interpolated, and every comparison +value becomes a bound parameter rather than inline text. One PostgreSQL- +specific detail: the bitmask predicate cases (`bitmaskAll`, `bitwiseEq`) +have to reference two already-appended bindings by their final numeric +position (`bindings.count - 1` and `bindings.count`), because PostgreSQL's +positional parameters are numbered by the order they were bound, unlike +SQLite's simple `?` placeholders which do not carry a number at all. + +### PostgreSQLSchema.swift + +This file provides `PostgreSQLSchemaEmitter`, the PostgreSQL analogue of +`SQLiteSchema`: pure functions translating a `SchemaDeclaration` into +PostgreSQL DDL. `typeSQL(_:)` maps each `ColumnType` to its PostgreSQL +native type — notably `.uuid` maps to a real `UUID` column (unlike +SQLite's `TEXT` fallback), `.timestamp` maps to `TIMESTAMPTZ`, and `.json` +maps to `JSONB`, PostgreSQL's indexable binary JSON type. Because +PostgreSQL has no `CREATE TRIGGER IF NOT EXISTS`, `appendOnlyTriggerStatements(_:)` +achieves the same idempotence SQLite gets for free by first dropping any +existing trigger of the same name and then creating it fresh, backed by +one shared `appendOnlyFunctionSQL` trigger function (created with `CREATE +OR REPLACE`, so re-running schema setup never fails) that raises an +exception naming the offending table via PostgreSQL's `TG_TABLE_NAME` +variable — one function serving every append-only table in the schema, +rather than one function generated per table. + +### PostgreSQLStores.swift + +This file provides `PostgreSQLRowStore`, `PostgreSQLBlobStore`, and +`PostgreSQLAuditLog` — the concrete `RowStore`, `BlobStore`, and +`AuditLog` conformances for the PostgreSQL backend, each able to run +either against a pooled connection acquired for a single call or against +a `PostgreSQLTransactionContext`'s already-open connection when called +from inside a transaction block. Every write method here validates SQL +identifiers with `validatePSQLIdentifier`, exactly as the SQLite backend +does, before assembling its SQL string. + +`PostgreSQLRowStore.query(...)` reads the schema's declared column list +for the target table (rather than issuing a bare `SELECT *`) specifically +so a generated column's PostgreSQL-computed value is decoded with the +correct `ColumnType`, and it runs every returned row's values through +`decryptedForRead` before wrapping them in a `StorageRow`. `update(...)` +and `delete(...)` both compile their predicate starting at parameter +index one past the last `SET`-clause binding, using the private +`renderPredicate(_:startIndex:bindings:)` helper, which compiles the +predicate normally and then renumbers every `$N` placeholder in reverse +order — high numbers first — specifically to avoid `$10` being +accidentally rewritten by a naive forward replacement of `$1`. +`PostgreSQLBlobStore` and `PostgreSQLAuditLog` each lazily create their +own backing table (`_storagekit_blobs`, `_storagekit_audit`) on first use +via `CREATE TABLE IF NOT EXISTS`, rather than requiring the caller's +schema declaration to know about them — the same design PersistenceKit +uses for SQLite's internal tables. `decodeAuditEvent(_:)` decodes the +audit row's required fields strictly (a decode failure there propagates +as a thrown error) while treating optional before-state fields +permissively (`try?`, because `NULL` is the expected, valid value for "no +prior state"). + +### PostgreSQLIdentifierValidator.swift + +This file provides `validatePSQLIdentifier(_:)`, the PostgreSQL-module +analogue of `SQLiteIdentifierValidator.swift`'s `validateSQLIdentifier`, +enforcing the exact same rule — first character a letter or underscore, +every later character a letter, digit, or underscore. The file's own +comment names this as one of three independent seams implementing the +identical rule: this one, the SQLite module's, and the Rust port's +`validate_sql_identifier` — three copies by necessity, since Swift +extensions cannot share a free function across two separate targets +without a shared dependency neither module otherwise needs, but each +copy is the single seam within its own module, so within a module the +rule cannot silently drift. + +## Target: PersistenceKitReplication + +### ReplicationTypes.swift + +This file provides the two public types every other file in this module +builds on. `ReplicationCursor` is the opaque watermark a caller stores +and passes back into a later incremental sync: it carries the highest HLC +seen across every copied row and audit event, plus counts of rows, +audit events, and blobs written during that run. `ReplicationError` is +the closed error type specific to replication — `schemaMismatch` (source +and destination disagree on schema version or kit ID; replication +deliberately refuses to auto-migrate either side) and `storageFailure` +(an underlying `StorageError` surfaced during the copy, re-wrapped as a +plain string so `ReplicationError` itself can stay `Equatable`, which a +raw `Error` cannot generally guarantee). + +### StorageReplicator.swift + +This file provides `StorageReplicator`, the full-snapshot replication +primitive: `replicate(from:to:schema:)` and its two direction-named +conveniences, `flush(from:into:schema:)` (in-memory into durable storage) +and `hydrate(into:from:schema:)` (durable storage into a fresh in-memory +instance). "Full snapshot" means exactly what it says — every run copies +every row in every schema-declared table, every audit event, and every +blob, regardless of what changed since the last run. The function is +still idempotent, though: every row upsert uses the table's own primary +key as its conflict column, not the random UUID a fresh `RowHandle` would +carry, so running the same flush twice against an unchanged source +updates existing destination rows in place rather than ever inserting a +second copy. + +The implementation runs in three ordered steps. Step one is a schema +gate: source and destination must report the exact same per-kit schema +version, or the function throws `ReplicationError.schemaMismatch` +immediately rather than attempting any copy. Step two reads the entire +source — every table's rows (with any generated columns filtered out of +each row, because writing a value into a column the destination computes +itself would be rejected by both SQL backends), every audit event, and +every blob — into one `Sendable` in-memory payload, before the +destination transaction ever opens; the file's comment explains this +ordering exists so the destination is never left holding a long-lived +serializable transaction open while potentially slow source I/O runs +underneath it. Step three writes that entire payload into the destination +inside one serializable transaction, so a crash or a thrown error partway +through leaves the destination exactly as it was before the flush began. +A blob-deletion pass at the end of step three removes any destination +blob key absent from the source snapshot, closing a gap the file +documents by its fix identifier (SECFIX-WS2-PK F5): without it, a blob +deleted at the source would linger forever in the destination after every +future full-snapshot flush, because an additive-only copy never notices +an absence. + +### IncrementalReplicationSession.swift + +This file provides `IncrementalReplicationSession`, the observer-driven +alternative to a full snapshot: instead of copying everything on every +run, it watches a source's live change stream and, on demand, syncs only +the rows and blobs that actually changed since the last sync. The file's +opening comment lays out the design choice explicitly: rather than a +durable dirty-row table written on every observed change (which would tie +this backend-agnostic module to one particular storage schema), the +session accumulates dirty identifiers purely in memory, inside two actors +— `DirtySet` for rows and `BlobDirtySet` for blobs — and re-reads each +dirty row's current state from the source at sync time rather than +trusting whatever the original change notification said, because the row +may have changed again since it was first marked dirty. If the process +restarts, the in-memory dirty-set is lost; the documented recovery is +simply to fall back to a full snapshot, which is always a safe substitute +for a lost incremental history. + +`DirtyKey` identifies one dirty row by its table name and its primary- +key values, encoded into a sortable string so that draining the dirty set +always produces the same processing order across repeated runs — this +determinism is what makes two different processes independently syncing +the same dirty set produce an identical, and therefore safely repeatable, +sequence of destination writes. `DirtySet.accumulate(_:)` extracts the +primary-key columns from an observed `TableChange`'s values dictionary, +logging and skipping (rather than crashing) if a conforming-but-buggy +backend ever emits a change missing one of those columns. `BlobDirtySet` +tracks the same idea for blobs, but because a blob change notification +already carries its own payload bytes, `BlobDirtySet` has no need to +re-read anything from the source at sync time — a `put` supersedes an +earlier `put` for the same key, and a `delete` supersedes a `put`, +last-write-wins. + +`sync(from:to:fromCursor:)` is the operation a caller actually invokes. +It repeats the same schema gate `StorageReplicator` uses, drains both +dirty sets, re-scans each dirty row from the source (turning an +now-missing row into a destination delete rather than an upsert, since +the row was deleted at the source between the original change +notification and this re-scan), fetches only the audit events newer than +the previous cursor's watermark, and commits everything inside one +serializable destination transaction — the same fail-loud, all-or-nothing +shape `StorageReplicator` uses for a full snapshot. The retry-preservation +contract is the piece unique to the incremental path: if anything fails +after the dirty sets have already been drained, the drained keys and blob +operations are restored into the sets — awaited synchronously in the +`catch` block, never handed off to a detached background task, because a +fire-and-forget restore could race an immediate caller retry and silently +lose the very keys it was trying to preserve — so a subsequent retry +re-attempts exactly the rows and blobs the failed run did not finish. + +## Rust Port and Conformance + +The `rust/` directory contains a second implementation of PersistenceKit's +design for use outside the Swift/Apple ecosystem: the same closed +`StoragePredicate` algebra, the same `TypedValue` case set, the same +`Storage`/`RowStore`/`BlobStore`/`AuditLog`/`StorageObserver` trait +surface, and all three backends (`InMemoryStorage`, `SqliteStorage`, +`PostgresStorage`), plus Rust equivalents of the caching decorator, the +hashing decorator, the encryption seam, and both replication modes. The +Rust traits are synchronous (`Result`) rather than +`async`, because the Rust backends do no real asynchronous I/O of their +own; the port's `README.md` notes that Swift's `async` requirement comes +from Swift actors specifically, and a future backend that does need +asynchronous I/O could wrap its own runtime without changing the trait +shape. + +Unlike LatticeLib's two legs, the Swift and Rust sides of PersistenceKit +are not gated by a shared, byte-identical conformance-fixture corpus. +PersistenceKit's cross-platform contract is a shared protocol shape and a +shared wire format — the same `TypedValue` cases, the same predicate +tree, the same identifier-validation rule — rather than a promise that +identical input always produces bit-identical output on both legs. +Each side carries its own test suite: the Rust SQLite and PostgreSQL +backends each run a ten-test conformance battery covering schema, rows, +predicates, blobs, audit, generated columns, transactions, append-only +enforcement, and introspection, exercised directly against the Rust +trait implementations rather than cross-checked against the Swift +package's own test output. diff --git a/packages/kits/PersistenceKit/docs/INTERFACE_DOCTRINE.md b/packages/kits/PersistenceKit/docs/INTERFACE_DOCTRINE.md deleted file mode 100644 index 1f5c284..0000000 --- a/packages/kits/PersistenceKit/docs/INTERFACE_DOCTRINE.md +++ /dev/null @@ -1,273 +0,0 @@ -# PersistenceKit Interface Doctrine - -For coding agents implementing kits that consume PersistenceKit (LocusKit, VectorKit, CorpusKit, GeniusLocusKit, anything else). This document is the contract. - -If you violate the doctrine, the abstraction breaks and the kit graph rots. Read it before writing code. - -## 1. Always go through protocols - -Consume `any RowStore`, `any BlobStore`, `any VectorIndex`, `any AuditLog`, `any Storage`. Never reference `InMemoryStorage`, `SQLiteStorage`, or `PostgreSQLStorage` from a downstream kit. Backend selection is the application's job, not the kit's. - -```swift -// CORRECT -final class LocusKit { - let storage: any Storage - init(storage: any Storage) { self.storage = storage } -} - -// WRONG -import PersistenceKitSQLite // never in a downstream kit -final class LocusKit { - let storage: SQLiteStorage // never -} -``` - -## 2. Declare your schema once, in code - -Every kit owns one `SchemaDeclaration`. Build it as a `let` constant inside the kit. Pass it to `Storage.open(schema:)` at estate-open time. PersistenceKit emits backend-specific DDL. - -```swift -public enum LocusKitSchema { - public static let declaration = SchemaDeclaration( - kitID: "LocusKit", - version: 1, - tables: [ - TableDeclaration( - name: "drawers", - columns: [ - .uuid("row_id"), - .bitmap("adjective"), - .bitmap("operational"), - .bitmap("provenance"), - .text("verbatim"), - .timestamp("captured_at"), - .int("udc_code"), - .int("qid_pointer", nullable: true) - ], - primaryKey: ["row_id"] - ) - ], - indices: [ - IndexDeclaration(name: "idx_drawers_adjective", table: "drawers", columns: ["adjective"]), - IndexDeclaration(name: "idx_drawers_captured_at", table: "drawers", columns: ["captured_at"]) - ] - ) -} -``` - -Schemas are append-only across versions. Bump `version` and add a `Migration` when you change anything. Never edit a `TableDeclaration` already in production. - -## 3. Never write raw SQL - -If you find yourself building a SQL string, stop. The work belongs in `StoragePredicate` or `OrderClause`. The backend compiles. You declare intent. - -```swift -// CORRECT -let active = try await storage.rowStore.query( - table: "drawers", - where: .and([ - .bitmaskAll(Column(table: "drawers", name: "operational"), mask: 0x01), - .bitmaskNone(Column(table: "drawers", name: "operational"), mask: 0x80) - ]), - orderBy: [OrderClause(column: Column(table: "drawers", name: "captured_at"), direction: .descending)], - limit: 50, - offset: nil -) - -// WRONG -let active = try connection.prepare("SELECT * FROM drawers WHERE ...") -``` - -The exception: if you need an operation that StoragePredicate cannot express, propose a new case in the closed enum via a decision record. Do not work around it with raw SQL. - -## 4. Use TypedValue exclusively - -Cross every kit boundary with TypedValue, not native Swift types. The encoding is the backend's problem. - -```swift -// CORRECT -values: [ - "captured_at": .timestamp(Date()), - "row_id": .uuid(rowID), - "adjective": .bitmap(0x01) -] - -// WRONG -values: [ - "captured_at": Date(), // not a TypedValue - "row_id": rowID.uuidString, // backend type leakage - "adjective": 1 // ambiguous between .int and .bitmap -] -``` - -`.bitmap(_)` and `.int(_)` are semantically distinct even though both are Int64. Use `.bitmap` for the three bitmap columns; `.int` for everything else. - -## 5. Atomic work uses transactions - -Operations that must commit together go inside `storage.transaction { ... }`. The block receives a `StorageTransaction` whose sub-stores share the same connection. Use them; do not call `storage.rowStore` from inside the block (that would acquire a separate connection). - -```swift -try await storage.transaction { txn in - let handle = try await txn.rowStore.insert(table: "drawers", values: ...) - try await txn.auditLog.append(captureEvent(for: handle)) -} -``` - -If the block throws, the transaction rolls back atomically. If it returns, the transaction commits. - -The capture verb crosses rowStore and auditLog; it always uses a transaction. Likewise for mutate, withdraw, expunge. - -## 6. Audit events get a fresh eventID per emit - -Every `AuditEvent` you construct gets a fresh `UUID()` for `eventID`. Never reuse an eventID; never derive it from row state. The compound key `(eventID, hlc)` makes append idempotent at the storage layer; ConvergenceKit relies on this property. - -```swift -// CORRECT -let event = AuditEvent( - eventID: UUID(), // fresh - estateUuid: estateID, - rowId: handle.key, - hlc: hlcGenerator.advance(), - verb: "capture", - // ... -) -try await txn.auditLog.append(event) -``` - -HLC comes from SubstrateLib's HLCGenerator. One generator per estate. Generators are monotonic; do not use raw timestamps. - -## 7. Never reach inside another kit's tables - -Each kit owns its tables. LocusKit owns `drawers`, `tunnels`, `kg_facts`, etc. VectorKit owns `rag_vectors`. CorpusKit owns `chunks`. Do not write SQL or queries that read another kit's tables from your kit. If you need cross-kit data, ask via that kit's API. - -The two tables PersistenceKit owns are internal and start with `_storagekit_`: `_storagekit_meta`, `_storagekit_blobs`, `_storagekit_audit`, `_storagekit_vectors`, `_storagekit_vector_meta`. Never touch them from downstream code. - -## 8. Test against InMemory in CI; SQLite for parity - -Per-kit test suites use `PersistenceKitInMemory` for speed (in-memory, no disk, no extension loading). Add `PersistenceKitSQLite` parity tests for any code path that depends on backend behavior (predicate semantics, ordering, vector distance). PostgreSQL tests are gated on `POSTGRES_TEST_URL`. - -```swift -func makeStorage() -> any Storage { - InMemoryStorage(configuration: EstateConfiguration( - estateID: UUID(), - backend: .inMemory - )) -} -``` - -When the conformance fixture suite lands (mission 2 final piece), your kit will run a small set of conformance fixtures against every backend. - -## 9. Migrations are forward-only - -When you change your schema, bump `version` and append a `Migration` with the operations to reach the new version. Never edit an existing migration. Never remove a migration. - -```swift -public static let declaration = SchemaDeclaration( - kitID: "LocusKit", - version: 2, // bumped from 1 - tables: [...], // current target shape - migrations: [ - Migration(fromVersion: 1, toVersion: 2, operations: [ - .addColumn(table: "drawers", column: .text("dialect", nullable: true)) - ]) - ] -) -``` - -A failed migration leaves the schema at the last successfully-applied version. Operators check `currentSchemaVersion()` (global max) or `currentSchemaVersion(for: kitID)` (per-kit) after failure. In multi-kit deployments, use the kitID-scoped variant to avoid misdiagnosing failure in one kit as failure in another. - -## 10. Bitmap predicate semantics - -Three operators cover spec §7.9: - -- `.bitmaskAll(col, mask: M)` → `(col & M) == M`. "all bits in M are set" -- `.bitmaskAny(col, mask: M)` → `(col & M) != 0`. "at least one bit in M is set" -- `.bitmaskNone(col, mask: M)` → `(col & M) == 0`. "no bits in M are set" - -Plus `.bitwiseEq(col, expected: E, mask: M)` → `(col & M) == E` for stateful bit-pattern matches. - -Compose with `.and([...])` and `.or([...])`. The mandatory filter ordering from spec §7.9.5 (tombstone exclusion, default state filters, then user predicates) is enforced by the kit constructing the predicate, not by PersistenceKit. Wrap user-supplied predicates: - -```swift -let final: StoragePredicate = .all([ - // 1. tombstone exclusion (always first) - .bitmaskNone(opCol, mask: TOMBSTONE_BIT), - // 2. default state filter (active rows) - .bitmaskAll(opCol, mask: ACTIVE_BIT), - // 3. user predicate - userPredicate -]) -``` - -## 11. Vector index conventions - -Vector dimensionality is fixed at first `add`. All subsequent vectors must match. To change dimensionality, drop and recreate. - -Distance metrics: `.cosine`, `.l2`, `.dot`. SQLite vec0 returns L2 by default regardless of metric requested; the abstraction tolerates this for v1.0 since callers normalize on the way in. PostgreSQL pgvector honors the metric per query. - -Vector metadata is per-vector key/value, encoded as JSON internally. Filter predicates on metadata work but currently run in-memory after the k-NN result returns (JSONB column-level predicate compilation is a v1.x improvement). Use metadata filters sparingly when k is large. - -## 12. EstateConfiguration is opaque - -The kit consuming Storage never inspects `configuration.backend`. The estate handle is opaque; the protocols are the contract. If your kit needs to know "am I on SQLite or PostgreSQL?", you have a layering bug. - -If your kit needs a behavior that differs by backend (e.g. concurrency tuning), that's a missing protocol method. Propose it via decision record. - -## 13. Sendable everywhere - -Every type that crosses an actor boundary or escapes into an `async` context must be Sendable. PersistenceKit's public types all are. Your downstream types must be too. Use `@unchecked Sendable` with a documented rationale (locking, immutable-after-init, etc.) only when needed. - -## 14. StorageObserver - -`Storage.observer` exposes change notifications. Subscribe to a table by event set; you receive an `AsyncStream` that fires on commits. Multiple subscribers on the same table coexist. - -```swift -let stream = storage.observer.observe(table: "jobs", events: [.insert]) -for await change in stream { - // wake up; do the work - guard let key = change.rowKey else { continue } - handleNewJob(key) -} -``` - -Use cases at v1.0: - -- QueueKit's `watch()` for filesystem-backed and PersistenceKit-backed queues -- GeniusLocusKit Brain layer standing signals waking on audit log appends -- ConvergenceKit replicating row-level changes outbound (when sync is enabled on the PersistenceKit instance) - -Backend semantics: - -- **InMemory**: notifications are reliably delivered to all matching subscribers -- **SQLite**: per-row notifications for `insert` and per-row notifications for `upsert`; `update` and `delete` fire coarse "something changed" notifications (rowKey may be nil) because bulk operations don't compose into per-row events without query rewriting -- **PostgreSQL**: NoOpObserver in v1.0; LISTEN/NOTIFY integration is v1.x - -Delivery is at-least-once. Subscribers must tolerate seeing a change after the row has already been deleted (race). - -Writes do not block on subscribers. AsyncStream uses `bufferingOldest(1024)`; slow subscribers may miss events under load. - -## 15. Error handling - -PersistenceKit throws `StorageError` for backend-attributable failures. Your kit can map these to kit-specific errors but should preserve the underlying cause: - -```swift -do { - try await storage.rowStore.insert(table: "drawers", values: ...) -} catch let error as StorageError { - throw LocusKitError.captureFailed(underlying: error) -} -``` - -Do not swallow StorageError. The error type carries diagnostic information operators need. - -## 16. When in doubt, file a decision record - -If you find yourself wanting to: - -- Add a case to TypedValue, ColumnType, StoragePredicate, or any closed enum in PersistenceKit -- Add a new protocol method to RowStore, BlobStore, VectorIndex, AuditLog, or Storage -- Change the audit log compound key -- Add a backend-specific escape hatch -- Treat one backend differently from another in downstream code - -Stop. Write a decision record in `docs/decisions/` proposing the change. The closed-enum design depends on every change being deliberate. diff --git a/packages/kits/PersistenceKit/docs/OVERVIEW.md b/packages/kits/PersistenceKit/docs/OVERVIEW.md new file mode 100644 index 0000000..da120d7 --- /dev/null +++ b/packages/kits/PersistenceKit/docs/OVERVIEW.md @@ -0,0 +1,279 @@ +--- +doc: OVERVIEW +package: PersistenceKit +repo: moot-system +authored_commit: 909513d0a8ecb1e9e903af9f4d25b3e4f2528242 +authored_date: 2026-07-04 +sources: + - path: Sources/PersistenceKit/AuditLog.swift + blob: ca4c0c9623056a49ad888a7a77e720a7519556bb + - path: Sources/PersistenceKit/BlobStore.swift + blob: f052c3ecc693d5414d57ad2c6da5fb4c5fe28f79 + - path: Sources/PersistenceKit/CacheInvalidator.swift + blob: 0a844b8037404dd72d08df6887ceee1e8c6014f2 + - path: Sources/PersistenceKit/CachingRowStore.swift + blob: 7edf9a86eb31dd9a17abd97b7baa9e8d5425e266 + - path: Sources/PersistenceKit/Column.swift + blob: 3cfc3ba856ebe15e8756ae65dae736cbd2c288a6 + - path: Sources/PersistenceKit/EncryptionMode.swift + blob: fdb7aa7ec63088ae09739f5e2a8e1468c386b987 + - path: Sources/PersistenceKit/ErasureLedger.swift + blob: eb2ca35f69783eead6049949bbaad68decfc9915 + - path: Sources/PersistenceKit/ErasureOverlay.swift + blob: d5a86f2baa10647134f285d6996a8e62cbe23ace + - path: Sources/PersistenceKit/EstateCacheConfig.swift + blob: 7003951cb5b2b075b6bb0c49b5fe372a97bfeb3a + - path: Sources/PersistenceKit/EstateConfiguration.swift + blob: be91569405e5c48a99859b4595540a59bd1a1994 + - path: Sources/PersistenceKit/GCPin.swift + blob: 73336dc4c693fcfa285a7eff8ed6520f27951ff8 + - path: Sources/PersistenceKit/GeneratedColumn.swift + blob: 34e466c80a0e9aac5d24303eaad132901333470b + - path: Sources/PersistenceKit/HashingRowStore.swift + blob: fe233ed2f4fda177373ad105d14fa321d325d0df + - path: Sources/PersistenceKit/NoOpObserver.swift + blob: 99f62b6889df50c5f09fce749f363637891f116c + - path: Sources/PersistenceKit/NovelTokenTaggerChoice.swift + blob: 9588d6303431883a2c8e0ee727f7ebe3e3f6d139 + - path: Sources/PersistenceKit/PersistenceKitTelemetry.swift + blob: a24e54dfb97beb86dac9422bbfc7cc3c2d959d73 + - path: Sources/PersistenceKit/Predicate.swift + blob: 522802c842dfc7573137efd7a3f36300d5201468 + - path: Sources/PersistenceKit/RowCrypto.swift + blob: 6678e426c0902cdae6d164904001526756e40916 + - path: Sources/PersistenceKit/RowStore.swift + blob: 9a96ff89528ca786d51224324cacb78cf810037d + - path: Sources/PersistenceKit/Schema.swift + blob: 5a6894c68d6f29eecfe9acc34d7c06ee2a655ac3 + - path: Sources/PersistenceKit/SnapshotRegistry.swift + blob: 88087bda544165c4d24c514a4f3e0641a620512e + - path: Sources/PersistenceKit/Storage.swift + blob: 7484a40913b28cb11a6bd2e3ea822dc8fe8eb63e + - path: Sources/PersistenceKit/StorageError.swift + blob: 743c2d1a24c7bafedb217e4c6bbf30ca20d03be8 + - path: Sources/PersistenceKit/StorageIntrospection.swift + blob: 35522b6601037246907bb88ebb2fb2eb5ea9b0e2 + - path: Sources/PersistenceKit/StorageObserver.swift + blob: f0a8b61a15344e02193137a3393be345dd51cf25 + - path: Sources/PersistenceKit/Transaction.swift + blob: d58d478618b31cc3985275be6ceee84a6fc222de + - path: Sources/PersistenceKit/TypedValue.swift + blob: ec124b7aebd86f64f67fea6656396d374666b741 + - path: Sources/PersistenceKitInMemory/InMemoryAuditLog.swift + blob: 1ba83d2408d3384d4d8e4f286fa2db743a433cc9 + - path: Sources/PersistenceKitInMemory/InMemoryBlobStore.swift + blob: e02822bde83c899b6d18ab8d82b90afba6abb5ac + - path: Sources/PersistenceKitInMemory/InMemoryObserver.swift + blob: 71b98c9da2d869a81a78265b7ef1f64a57a907af + - path: Sources/PersistenceKitInMemory/InMemoryRowStore.swift + blob: 2f7c92612c46b8a097e4211147c924f8356e47d1 + - path: Sources/PersistenceKitInMemory/InMemoryStorage.swift + blob: 9820c771bfe39ab738eea1f3b1623491fcfb1326 + - path: Sources/PersistenceKitInMemory/PredicateEvaluator.swift + blob: e8735422ff87addff6a2a3a89da85c495d07f07c + - path: Sources/PersistenceKitPostgreSQL/PostgreSQLConnection.swift + blob: 5a282a6064a69a543a5d650ccc7eeeae2c5a3e4f + - path: Sources/PersistenceKitPostgreSQL/PostgreSQLIdentifierValidator.swift + blob: 120cf0c1576a7db45d79bf6865e2f15cff09a5ac + - path: Sources/PersistenceKitPostgreSQL/PostgreSQLPool.swift + blob: eddc0c3af4135565e15d2d763b0d24240973dd6f + - path: Sources/PersistenceKitPostgreSQL/PostgreSQLPredicateCompiler.swift + blob: 6f0791e0372b09cf2605bae4cf149e48bbba2834 + - path: Sources/PersistenceKitPostgreSQL/PostgreSQLSchema.swift + blob: f165f0877b96c87e2f7de46012f8f8acb3c03cc8 + - path: Sources/PersistenceKitPostgreSQL/PostgreSQLStorage.swift + blob: 849926f9e6def788fae2f38757ad44bcd52a18a0 + - path: Sources/PersistenceKitPostgreSQL/PostgreSQLStores.swift + blob: bff90838d1324d82ae76a0895f15399537e9e343 + - path: Sources/PersistenceKitReplication/IncrementalReplicationSession.swift + blob: 2f90378146e043d75c739d400d98c7770e07af78 + - path: Sources/PersistenceKitReplication/ReplicationTypes.swift + blob: bb13c63d1febd50ad46c1814610d7f6c31a33112 + - path: Sources/PersistenceKitReplication/StorageReplicator.swift + blob: f5acc5993c8647e53c127c6871a1282c24b1c427 + - path: Sources/PersistenceKitSQLite/KeychainKeyStore.swift + blob: 0071732291a7cb6ce0777bd230a6188276fb4f32 + - path: Sources/PersistenceKitSQLite/SQLiteConnection.swift + blob: ece56dc7e25e67656bb37f5222c18c1166c750cc + - path: Sources/PersistenceKitSQLite/SQLiteIdentifierValidator.swift + blob: 713339c137d6af1cfbba5a3584e05bdda70b42c5 + - path: Sources/PersistenceKitSQLite/SQLiteObserver.swift + blob: 2cb61ed75dae5ab81a3cdfb5c9581731d25fd960 + - path: Sources/PersistenceKitSQLite/SQLitePredicateCompiler.swift + blob: 5770adb2024c54ea6921651632ca65ee84d416af + - path: Sources/PersistenceKitSQLite/SQLiteSchema.swift + blob: 4ba6fc175fe9d486b17af1088a23da64041b12ae + - path: Sources/PersistenceKitSQLite/SQLiteStorage.swift + blob: 417e63cc07b60295ac874187ebb796bf9f3b3c86 + - path: Sources/PersistenceKitSQLite/SQLiteStores.swift + blob: 76499ffb70d979f0d13e8cf9e32bc38ff28ffdb5 +--- + +# PersistenceKit Overview + +## What This Library Does + +PersistenceKit is the storage layer for MOOTx01. MOOTx01 is an on-device AI +memory system. It stores what an AI observes over time and helps the AI +recall it later. Every fact, drawer, and audit record that a MOOTx01 estate +keeps passes through PersistenceKit before it reaches a disk or a database. +An estate is one user's complete memory store in MOOTx01. PersistenceKit +does not decide what a memory means. It decides how a memory is written, +read, and kept safe once another kit hands it over as typed rows and bytes. + +PersistenceKit gives every higher kit the same four operations — insert a +row, fetch a blob, append an audit event, watch a table for changes — no +matter which physical engine sits underneath. The engine can be SQLite on a +phone, PostgreSQL on a shared server, or a plain in-memory table for a unit +test. The kit that calls PersistenceKit writes the same code either way. + +## The Problem It Solves + +MOOTx01 runs in more than one place. A single estate might live only on one +iPhone, or it might run against a shared PostgreSQL server for a managed +deployment, or it might exist only for the length of a test. Without a +storage abstraction, every kit that reads or writes memory would have to +know three different database APIs and three different sets of quirks. +Worse, a kit built against SQLite could not be pointed at PostgreSQL +without a rewrite. + +PersistenceKit solves this with one typed contract — the `Storage` +protocol — and three backends that all satisfy it: `PersistenceKitSQLite`, +`PersistenceKitPostgreSQL`, and `PersistenceKitInMemory`. A caller writes +against `Storage`, `RowStore`, `BlobStore`, and `AuditLog` once. Swapping +the backend means changing one line of configuration, not the calling code. + +A second problem is that "one interface, three engines" is not enough by +itself. Real deployments also need at-rest encryption, tamper-evident +audit trails, safe deletion of sensitive content, a cache that never lies +about what is really stored, and a way to copy an entire estate from one +backend to another. PersistenceKit builds all of these as layers on top of +the same four-operation core, so a kit that only needs plain rows never +pays for the features it does not use, and a kit that needs encryption or +replication gets it without leaving the same protocol surface. + +PersistenceKit deliberately does not own vector similarity search. A +separate library, VectorKit, owns dense-embedding nearest-neighbor search. +PersistenceKit's job toward that workload is narrower: every backend must +support the storage needs a vector index has — storing a vector payload in +a row, reading many rows back in bulk, counting rows, and deleting rows. +This is the accommodation contract. PersistenceKit accommodates the +workload; it does not implement the search. + +## How It Works + +At the center of the library is one protocol, `Storage`. A type conforms to +`Storage` by providing four sub-stores — `RowStore` for typed rows, +`BlobStore` for raw bytes keyed by string, `AuditLog` for an append-only +history of changes, and `StorageObserver` for live change notifications — +plus schema and transaction management. `EstateConfiguration` tells a +`Storage` implementation which backend to open, which encryption mode to +use, and which cache settings apply. + +A schema in PersistenceKit is not raw SQL. It is a typed Swift value — +`SchemaDeclaration`, built from `TableDeclaration` and `ColumnDeclaration` +values. A kit that wants to store data describes its tables once, as Swift +structs, and each backend translates that description into its own native +form: SQLite gets `CREATE TABLE` statements, PostgreSQL gets `CREATE TABLE` +statements in its own dialect, and the in-memory backend just allocates a +dictionary. Queries follow the same pattern: `StoragePredicate` is a closed +tree of comparison and bitmap operators that each backend compiles to its +own query language, so no caller ever writes a raw SQL string. + +Three backends satisfy `Storage`. `PersistenceKitSQLite` is the on-device +engine — one file per estate, encrypted at rest through a vendored +SQLCipher build when the estate asks for whole-database encryption. +`PersistenceKitPostgreSQL` is the server engine, built on the `postgres-nio` +client, with one PostgreSQL schema per estate so many estates can share one +database server without their tables colliding. `PersistenceKitInMemory` is +the test and prototyping engine — no file, no network, gone when the +process exits. + +On top of the core protocol, PersistenceKit layers decorators and +free-function toolkits that any backend can use without changing its own +code: + +- `CachingRowStore` wraps a `RowStore` with an in-memory hot tier. Rows + above a configured sensitivity level are never cached, and the cache + guarantees that every read returns exactly what the backing store would + have returned — caching only changes speed, never correctness. +- `HashingRowStore` wraps a `RowStore` and computes a content hash on every + write to a table marked hashable, then reports the write up a parent + chain so a Merkle-style integrity tree can stay current without a full + rescan. +- `RowCrypto` and its write/read seam functions apply per-row AES-GCM + encryption to a table's `content` column when an estate is configured for + row-level encryption, independent of whichever backend stores the row. +- `ErasureLedger` and `ErasureOverlay` implement right-to-be-forgotten + deletion: the ledger records that a piece of content was erased, and the + overlay nulls that content out of every read afterward, failing closed + (dropping the row) if the erasure check itself cannot be completed. +- `SnapshotRegistry` and `GCPin` let a kit take a named, attested snapshot + of the estate's state and pin the oldest live snapshot's timestamp so a + later garbage-collection pass never deletes data a snapshot still needs. +- `PersistenceKitReplication` copies an entire estate's rows, audit events, + and blobs from one `Storage` to another, either as a one-shot full + snapshot or as an incremental sync driven by a live change-observation + session. + +## How the Pieces Fit + +Figure 1 shows the library's topology — its major parts and how a write or +a read moves between them. + +![Figure 1. Topology of PersistenceKit](topology.svg) + +*Figure 1. Topology of PersistenceKit. A calling kit reaches storage only +through the four core protocols. Decorators wrap a backend's `RowStore` +transparently. Cross-cutting concerns (encryption, erasure, snapshots) sit +beside the core surface and are invoked by name, not by inheritance. +Replication reads one `Storage` and writes another. Dashed regions mark +backend-owned engines and the cross-process encryption key store.* + +A typical write enters through a kit-held reference to `any RowStore`. If +the estate has caching enabled, that reference is really a +`CachingRowStore` wrapping the real backend's row store; if hash-on-write +is configured, a `HashingRowStore` may sit in front of that. The write +descends through zero or more decorators until it reaches the concrete +backend — `SQLiteRowStore`, `PostgreSQLRowStore`, or `InMemoryRowStore` — +which applies the row-encryption seam (a no-op unless the estate uses row +encryption), checks the content/keyID invariant, validates every SQL +identifier it is about to interpolate, and commits the row. Every backend +notifies its `StorageObserver` afterward, so decorators, cache +invalidators, and replication sessions downstream can react. + +A typical read is the same path in reverse, with one addition: any content +subject to the right-to-be-forgotten contract passes through +`ErasureOverlay.apply(...)` after the backend query returns, so an erased +row's content columns come back `nil` even though its skeleton (its ID, +timestamps, and lattice anchors) survives for referential integrity. + +Encryption, erasure, and snapshots are not separate storage engines. They +are pure functions and small actors that any backend calls at the right +moment — `encryptedForWrite`/`decryptedForRead` around a write or a read, +`ErasureLedgerOps.isErased` inside the overlay, `GCPin.minimumRetainableHlc` +before a vacuum. This design keeps the cross-cutting logic in one place +(`PersistenceKit` core) instead of duplicating it inside every backend, and +it is why the SQLite and PostgreSQL backends produce byte-compatible +encrypted envelopes: they call the same `RowCrypto` code. + +## What Ships in the Package + +The package ships five Swift targets. `PersistenceKit` is the core: the +`Storage`/`RowStore`/`BlobStore`/`AuditLog`/`StorageObserver` protocols, the +typed value and predicate algebra, schema declarations, the caching and +hashing decorators, and the encryption, erasure, snapshot, and telemetry +toolkits. `PersistenceKitInMemory` is the test backend. `PersistenceKitSQLite` +is the on-device backend, built against a vendored `SQLCipher` C target so +whole-database encryption never depends on the host OS's system SQLite. +`PersistenceKitPostgreSQL` is the server backend, built on `postgres-nio` and +`NIOSSL`. `PersistenceKitReplication` is the estate-copying primitive, +depending only on the core protocol surface so it works against any pair of +conforming backends. + +The package also ships a Rust port in `rust/`, mirroring the same trait +surface, the same closed predicate algebra, and the same three backends, +for use outside the Swift/Apple ecosystem. The two ports share design intent +but are not conformance-gated against each other the way LatticeLib's two +legs are: PersistenceKit's cross-platform contract is the shared protocol +shape and wire format, not byte-identical output from a shared corpus. diff --git a/packages/kits/PersistenceKit/docs/topology.svg b/packages/kits/PersistenceKit/docs/topology.svg new file mode 100644 index 0000000..74f1ccb --- /dev/null +++ b/packages/kits/PersistenceKit/docs/topology.svg @@ -0,0 +1,164 @@ + + + + + + + + + + + + + PersistenceKit: one protocol, three backends, cross-cutting toolkits + + + + Calling kit + any RowStore + + + + HashingRowStore + hash-on-write, dirty chain + + + CachingRowStore + LRU hot tier, sensitivity-gated + + + + Storage protocol + RowStore+BlobStore+AuditLog+Observer + + + + + + + + + SQLiteStorage + one file, SQLCipher + + + PostgreSQLStorage + one schema per estate + + + InMemoryStorage + ephemeral, tests + + + + + + + + Apple-only key custody + + KeychainKeyStore — per-estate 256-bit key + + + + + Cross-cutting toolkits — called by name, not by inheritance + + + RowCrypto + AES-GCM row seam + encrypt/decrypt write+read + + + ErasureLedger + Overlay + right-to-be-forgotten + fail-closed on read + + + SnapshotRegistry + + GCPin + attested snapshots, vacuum floor + + + + + + + + + + PersistenceKitReplication — operates on any two Storage instances + + + Source estate + any Storage + + + StorageReplicator + full snapshot: rows, audit, + blobs, one serializable txn + + + IncrementalReplicationSession + observer-driven dirty set, + watermark + re-scan + + + Destination estate + any Storage + + + + + + + Both paths read the source, then write the + destination inside one serializable transaction — + a crash mid-flush leaves the destination unchanged. + + + + + diff --git a/packages/kits/QueueKit/docs/AGENT_MAP.md b/packages/kits/QueueKit/docs/AGENT_MAP.md new file mode 100644 index 0000000..659f0f8 --- /dev/null +++ b/packages/kits/QueueKit/docs/AGENT_MAP.md @@ -0,0 +1,179 @@ +--- +doc: AGENT_MAP +package: QueueKit +repo: moot-system +authored_commit: 909513d0a8ecb1e9e903af9f4d25b3e4f2528242 +authored_date: 2026-07-04 +sources: + - path: Sources/QueueKit/DrainLease.swift + blob: 3f1df6aa1fab44c993bd49d96de179ff03450303 + - path: Sources/QueueKit/FilesystemBackend.swift + blob: c0e5f536e20b1def11875e57b587e8392b6f936d + - path: Sources/QueueKit/Job.swift + blob: 5f39e97a8113e47de2f2a18563446072de8ec6cf + - path: Sources/QueueKit/ObservationStatus.swift + blob: a32b0e8c93311ce498d12ffb4df864a2652c707e + - path: Sources/QueueKit/PersistenceKitBackend.swift + blob: 96175a8b27c8b6e929f417d95b141f727c5179d7 + - path: Sources/QueueKit/QueueBackend.swift + blob: eb67e0b821bc9bfdec8cfde621c5e70d4295d331 + - path: Sources/QueueKit/QueueError.swift + blob: 2697c4b7404b9e04259267cd8f4008030ebb6754 + - path: Sources/QueueKit/QueueKit.swift + blob: 60dfaa1e8f92ec051810b50d8b9cadc47388c02f + - path: Sources/QueueKit/QueueKitTelemetry.swift + blob: 29024f112bf133012283205175aa336b8d80d7c9 + - path: Sources/QueueKit/Watcher.swift + blob: cf2b270b9c60da34f7a25c016f8c18b6ba6149e4 +--- + +# AGENT_MAP — QueueKit + +PURPOSE: general-purpose durable work queue. Producer→`send`/`writeBatch`→backend; consumer→`drain`/`watch`→claims jobs; consumer→`reply`→terminal completion. Two swappable backends (`FilesystemBackend` POSIX maildir, `PersistenceKitBackend` shared SQLite table) behind one `QueueBackend` protocol and one public `QueueKit` facade. Stream-scoped ops (ADR-021 Decision 7 / T1) let multiple consumers share one queue without stealing each other's jobs. + +DEPS: imports SubstrateTypes (HLC/HLCGenerator — hybrid logical clock), PersistenceKit (Storage, rowStore, transaction, observer — PersistenceKitBackend only), IntellectusLib (self-report telemetry, DECISION_LIFT_PACKAGE_SWIFT_RULE_2026-05-28). ConvergenceKit is application-layer composition and deliberately NOT a dependency (spec §11). Imported by: none within this repo checkout; source comments name CorpusKit and GeniusLocusKit as external mount-time consumers (not present in moot-system). Rust port in rust/ (8 files, ~3k lines) mirrors facade + both backends, PersistenceKitBackend behind `persistencekit` cargo feature. Python port in python/queuekit/ mirrors FilesystemBackend only (spec §2: Python has no PersistenceKit backend). All three gated by shared conformance fixtures in Tests/QueueKitTests/Fixtures/*.json. + +ENTRY POINTS (most callers need only these): +- QueueKit.swift:52 `QueueKit.init(root:hlcGenerator:) throws` — mount FilesystemBackend at root, create maildir, clean stale tmp/ +- QueueKit.swift:65 `QueueKit.init(backend:root:)` — mount explicit backend (PersistenceKitBackend, or a test double) +- QueueKit.swift:72 `send(_ job: Job) async throws` — enqueue one job +- QueueKit.swift:85 `drain() async throws -> [(job: Job, sessionID: SessionID)]` — claim all available jobs +- QueueKit.swift:129 `reply(to:status:artifacts:) async throws` — mark one claimed job terminal +- QueueKit.swift:123 `watch(handler:) async throws` — subscribe a per-job handler + +## Symbol Table + +### Facade — QueueKit.swift +- :28 `staleTmpThreshold = 5 * 60` — tmp/ files older than this on init are deleted (crash between write and rename; never visible in new/, safe to remove) +- :30 `final class QueueKit: Sendable` +- :39 `latencyWindow` — nonisolated(unsafe); caller (GLK scheduler actor / serial test) owns exclusivity +- :47 `estateTag: String = "unknown"` — nonisolated(unsafe); set ONCE at mount before any drain(), never again +- :52 `init(root:hlcGenerator:) throws` — see ENTRY POINTS +- :65 `init(backend:root:)` — see ENTRY POINTS +- :72 `send(_:)` / :81 `send(batch:) -> Int` — writeBatch twin; FS backend fsyncs new/ ONCE for the batch +- :85 `drain()` / :108 `drain(stream:)` — claim; wraps backend call with reportQueueStats timing +- :123 `watch(handler:)` — forwards to backend.watch +- :129 `reply(to:status:artifacts:)` — guards status.isTerminal, throws .invalidTerminalStatus otherwise +- :148 `reply(session:status:) -> Int` — fast path ONLY on PersistenceKitBackend (`as?` cast); returns 0 on FilesystemBackend (caller falls back to per-job reply) +- :171 `reply(batch:) -> Int` — routes to backend.completeBatch; FS backend's one-scan/one-fsync path +- :193 `reclaimInFlight(stream:) -> Int` — GATE: call ONLY immediately after DrainLease.tryAcquire SUCCEEDED for stream; fast path ONLY on PersistenceKitBackend, returns 0 otherwise +- :204 `inFlight()` / :214 `pendingCount()` / :224 `pendingCount(stream:)` — read-only depth probes; pendingCount()+inFlight().count = total outstanding work +- :261 `awaitDrain(pollInterval: .milliseconds(20), timeout: .seconds(30)) throws` — polls both frontiers; returns promptly if already empty; throws .drainTimeout, never hangs +- :292 `awaitDrain(stream:pollInterval:timeout:)` — stream-scoped twin; global awaitDrain would block forever on OTHER streams' jobs a scoped drainer never claims +- :314 `completed(streamID:)` +- :322 `maildirSubdirs = ["tmp","new","cur","done"]` +- :324 `ensureMaildir(root:) throws` — creates 4 subdirs if absent +- :341 `cleanStaleTmpFiles(root:) throws` — deletes tmp/ entries older than staleTmpThreshold + +### Backend contract — QueueBackend.swift +- :9 `protocol QueueBackend: Sendable` +- :10 `write(_:)`, :23 `drainAvailable()`, :30 `watch(handler:)`, :34 `complete(_:status:artifacts:)`, :52 `inFlight()`, :54 `completed(streamID:)` — must-implement core +- :21 `writeBatch(_:) -> Int` — default (extension :92-102) loops write; override for shared-cost batching +- :28 `pendingCount()` — depth probe used by telemetry +- :48 `completeBatch(_:) -> Int` — default (extension :104-116) loops complete; override for shared-cost batching +- :66 `drainAvailable(stream:)` — default (extension :76-84) claims ALL streams then filters — steals other streams' jobs, MUST override for real stream isolation +- :73 `pendingCount(stream:)` — default (extension :86-89) delegates to all-streams pendingCount() (wrong count unless overridden) + +### Wire types — Job.swift +- :22 `struct JobID: RawRepresentable` — :31 `generate()` = UUID → 32 lowercase hex, no hyphens (avoids colliding with maildir filename hyphen separator) +- :51 `struct StreamID: RawRepresentable` — workload/lane name +- :68 `struct SessionID: RawRepresentable` — :75 `mint()` = one per drain() call; groups a claimed batch +- :80 `struct ToolName: RawRepresentable` +- :85 `enum ArtifactRef` — file_path|commit_hash|signal_file|trajectory_step_id; explicit `type` field on wire (:91 CodingKeys) +- :131 `indirect enum CodableValue` — open caller extensions blob: null/bool/int/double/string/array/object; round-trips verbatim +- :167 `struct Job: Identifiable, Hashable` — id/streamID/submittedAt(HLC)/priority/payload/extensions +- :193 `Job` CodingKeys — PINNED snake_case wire keys: id, stream_id, submitted_at{physical_time,logical_count,node_id}, priority, payload(base64url), extensions +- :254 `struct MissionContext` — ONLY type using default camelCase Codable synthesis (not part of pinned job envelope) +- :236 `WireFormat.base64urlEncode(_:)` / :245 `base64urlDecode(_:)` — RFC 4648 §5, no padding +- :289 `WireFormat.filename(for:) -> String` — `{sortableHLC}-{streamID}-{jobID}` +- :296 `WireFormat.sortableHLC(_:)` — `{phys:016d}-{logical:08d}-{node_unsigned:010d}`; zero-padding makes lexicographic filename sort == HLC sort +- :307 `WireFormat.encoder` — sortedKeys + withoutEscapingSlashes — byte-stable across Swift/Rust/Python (conformance requirement) +- :313 `WireFormat.decoder` +- :321 `struct SignalFile` — job_id/status/artifacts/completed_at; same pinned-key discipline as Job + +### Status/errors +- ObservationStatus.swift:9 `enum ObservationStatus: String` — running|done|done_with_concerns|needs_context|blocked +- ObservationStatus.swift:16 `isTerminal` — only .running is false; gates every reply/complete call +- QueueError.swift:7 `enum QueueError: Error` — directoryCreationFailed, writeFailed, renameFailed, decodingFailed, unknownTool, jobNotFound, watcherFailed, staleTmpFile, backendUnavailable, invalidTerminalStatus, drainTimeout(pending:inFlight:), invalidIdentifier(id:reason:) + +### FilesystemBackend.swift (POSIX maildir; semantics derived from Postfix deliver_maildir()) +- :43 `final class FilesystemBackend: QueueBackend, @unchecked Sendable` +- :32 `private final class HLCBox` — NSLock-guarded HLCGenerator (value type; concurrent send() must serialize) +- :74 `validateIdentifier(_:) throws` (private static) — rejects empty/`.`/`..`/path-separator/control-char; MUST run before ANY path is built from a caller identifier (path-traversal guard, CAND-023, mirrored in Rust+Python) +- :96 `write(_:)` — validate → O_CREAT|O_EXCL write to tmp/, mode 0600, fsync, rename to new/, fsync new/ dir +- :129 `writeBatch(_:) -> Int` — per-job write+rename WITHOUT fsync, ONE fsync of new/ dir at end; AT-LEAST-ONCE (reindex re-derives from estate on crash) +- :160 `atomicWriteAndRename(data:tmpPath:newPath:newDir:) throws` (static) — the full durable write; ENOENT rename retried once, EXDEV (tmp/new on different filesystems) fails hard +- :273 `writeAndRenameNoFsync(data:tmpPath:newPath:) throws` (static) — batch primitive, no per-file/dir fsync +- :340 `pendingCount()` — file count in new/ +- :363 `drainAvailable(stream:)` — reads+decodes each new/ file BEFORE claiming (read ≠ claim); only rename-claims if job.streamID == stream; non-matching files left untouched in new/ (no steal-then-unclaim) +- :428 `pendingCount(stream:)` — decodes each new/ file, non-claiming +- :445 `drainAvailable()` — rename ALL new/→cur/ first, then decode; undecodable file → moved to done/ (poison, never retried) +- :516 `reclaimInFlight() -> Int` — GATE: call ONLY at mount before any drain session is live (fresh process owns no in-flight work by construction); unscoped (cur/ has no per-stream separation) +- :549 `watch(handler:)` — Watcher.watchNewDirectory → drainAvailable() → handler per job +- :567 `complete(_:status:artifacts:)` — find file in cur/ by `-{jobID}` suffix; writes SignalFile to done/ BEFORE renaming job file (crash between = job still reclaimable, never orphaned-signal-no-job) +- :631 `completeBatch(_:) -> Int` — ONE cur/ scan → jobID→filename index (keys on suffix after LAST `-`; JobIDs are 32 dashless hex so this is unambiguous); per-completion write+rename without fsync; ONE done/ dir fsync at end +- :701 `listJobs(in:filter:)` (private) — skips `.signal` sidecars, decodes, optional stream filter + +### PersistenceKitBackend.swift (spec §10 v1.1 — 5 invariants enforced) +- :34 `queueKitTableName = "queuekit_jobs"` +- :36 `enum QueueKitSchema` — :40 `declaration()` — columns incl. status/session_id/signal_status/artifacts; :62 `appendOnly: false` MUST stay false (invariant 5); indices: status, (status,phys,logical,node) claim-order, (stream_id,status) +- :84 `final class PersistenceKitBackend: QueueBackend, @unchecked Sendable` +- :91 `openSchema(on:) throws` (static) — opens schema on a Storage +- :97 `write(_:)` — BARE rowStore.insert, status="new", NO transaction (invariant 1) +- :143 `writeBatch(_:) -> Int` — ONE `.readCommitted` transaction wrapping N inserts; readCommitted safe because new-row inserts cannot conflict with the claim's serializable new→cur UPDATE (claim sees pre-batch snapshot or post-batch rows, never a torn view) +- :172 `pendingCount()` — COUNT WHERE status='new' +- :189 `drainAvailable(stream:)` / :248 `drainAvailable()` — SINGLE-PASS CLAIM (invariant 3): one `.serializable` txn = guarded bulk UPDATE new→cur under fresh session_id, THEN SELECT rows WHERE session_id=session (same txn, sees own writes); O(N) not O(N²); session-keyed readback prevents cross-drainer double-count +- :237 `pendingCount(stream:)` — COUNT WHERE status='new' AND stream_id=? +- :324 `completeSession(_:status:) -> Int` — NOT on QueueBackend protocol (concrete-only); ONE guarded UPDATE WHERE session_id=? AND status='cur'; backs QueueKit.reply(session:status:) +- :350 `watch(handler:)` — storage.observer.observe(insert) per invariant 2 (event = wake ONLY, never read job fields from event); drains once BEFORE first await (catches pre-subscription inserts), then per wake +- :378 `drainUntilEmpty` (private static) — loops drainAvailable() until a pass claims 0; absorbs coalesced/dropped observer wakes; a claim ERROR must propagate (not `try? ?? []`) or a fault masquerades as "queue empty" +- :400 `complete(_:status:artifacts:)` — guarded UPDATE WHERE id=? AND status='cur'; affected==0 → throws .jobNotFound +- :451 `reclaimInFlight(stream:) -> Int` — GATE identical to FilesystemBackend.reclaimInFlight: ONLY after DrainLease.tryAcquire success for stream; resets cur→new, clears session_id, stream-scoped (shared table holds multiple streams) +- :468 `inFlight()` / :472 `completed(streamID:)` → :480 `listJobs(status:streamID:)` (private) +- :500 `decodeRow(_:)` (private static) — malformed extensions JSON → falls back to empty dict, not a decode failure + +### DrainLease.swift — stream-keyed heartbeat-TTL lock (NOT PID-liveness; portable, no FFI) +- :49 `struct DrainLease: Sendable` — pure value type; lets Corpus's non-isolated deinit call release() directly +- :55 `owner` — `"pid-{pid}-{instanceToken}"`; nonce defeats PID-reuse impersonation after crash +- :59 `ttl` — default 15s = Rust DRAIN_LEASE_TTL_SECS +- :64 `heartbeatInterval = 5` (static) — heartbeat cadence, well inside ttl +- :73 `init(directory:stream:instanceToken:ttl:)` — lease file = `/.drain.lease`; stream name sanitized to alnum/_/- (else `_`) +- :93 `tryAcquire(now:) -> Bool` — false only if fresh lease held by ANOTHER owner; else (re)claims + re-reads to resolve write races (atomic replace = last-writer-wins) +- :110 `heartbeat(now:)` — refresh; does NOT check ownership, caller's responsibility +- :119 `isHeldByOther(now:) -> Bool` — freshness check without attempting acquire +- :127 `release()` — deletes file ONLY if still owned by self (never undoes a takeover) +- File format: `\n`, atomic replace write + +### Watcher.swift — directory wake source (spurious wakes OK; drainAvailable is authority) +- :27 `enum Watcher` +- :37 `watchNewDirectory(at:onChange:) throws` — calls onChange() once immediately (drain pre-attach work) before platform loop +- :63 `watchKQueue` (Darwin only) — DispatchSource kqueue VNODE (write/extend/delete/rename), O_EVTONLY fd, µs latency +- :134 `openInotify` (Linux, private) — IN_CREATE|IN_MOVED_TO; returns -1 on watch-limit/unsupported-fs (caller falls back) +- :162 `runInotifyLoop` (Linux, private) — poll() 500ms timeout for responsive cancellation; drains fd, doesn't parse events +- :216 `watchPoll` — 200ms snapshot-diff poll; Linux fallback AND sole watcher on other platforms; ~100ms average latency +- :240 `final class ContinuationBox` — bridges DispatchSource cancel handler (DispatchQueue callback) to async wait() + +### Telemetry — QueueKitTelemetry.swift (off-path cost ~1ns; metric namespace `queue.*`) +- :27 `struct QueueLatencyWindow` — capacity-100 rolling sample list +- :44 `percentile(_:) -> Double` — P7-secfix: guards non-finite/out-of-range p BEFORE index computation (NaN/inf could crash) +- :68 `reportQueueStats(backend:drained:drainStart:now:estateTag:window:) async` — gate: `Intellectus.isEnabled`; emits queue.depth (or queue.depth_unavailable on read failure — NEVER fabricates 0), queue.drain_count, queue.idle_nonempty (skipped if depth unknown), queue.latency_p50_ms, queue.latency_p95_ms, queue.head_of_line_age_s + +## INVARIANTS / GOTCHAS + +- WIRE FORMAT IS THE CONTRACT. Job/SignalFile Codable keys (snake_case, nested HLC, base64url payload) are pinned across Swift/Rust/Python. Any change must be mirrored in rust/src/job.rs and python/queuekit/job.py and pass the shared fixtures in Tests/QueueKitTests/Fixtures/*.json (rust/tests/conformance.rs reads the SAME files). +- JobID.generate() deliberately omits hyphens (32 dashless hex) — required so WireFormat.filename's own hyphen separators are unambiguous. Do not "prettify" back to UUID-with-hyphens form. +- validateIdentifier (FilesystemBackend) MUST run before any path is constructed from a caller-supplied stream_id or job id, at EVERY write/complete/completeBatch entry point. Mirrored in Rust and Python — edit all three or none. +- reclaimInFlight (both backends) is safe ONLY immediately after a successful DrainLease.tryAcquire for that stream. Calling it without a fresh lease can yank a job out from under a live drainer. FilesystemBackend's unscoped reclaimInFlight() is mount-time-only, by construction (fresh process = no in-flight work of its own). +- reply(session:status:) and reclaimInFlight(stream:) have real fast paths ONLY on PersistenceKitBackend (checked via `as?`); every other backend returns 0, and callers MUST have a per-job fallback. +- QueueBackend's default drainAvailable(stream:) and pendingCount(stream:) extensions are CORRECTNESS TRAPS if relied upon: the default drain claims and then re-releases other streams' jobs (transient full-queue claim), and the default pendingCount(stream:) reports the ALL-STREAMS count. Both concrete backends override correctly; a THIRD backend implementation MUST also override both. +- PersistenceKitBackend.write() is a bare insert, NEVER wrap it in a transaction (invariant 1). writeBatch's .readCommitted (not .serializable) is deliberate — see drainAvailable's serializable claim for why this is still race-free. +- PersistenceKitBackend.watch()'s observer event is a WAKE SIGNAL ONLY (invariant 2) — never read job data from the TableChange event; always re-enter through drainAvailable(). drainUntilEmpty must propagate claim errors, never `try? ?? []`, or a live fault silently reads as "queue empty." +- appendOnly MUST stay false on the queuekit_jobs table declaration (invariant 5) — rows are mutated in place through the new→cur→done lifecycle. +- FilesystemBackend.completeBatch's jobID lookup keys on the filename suffix after the LAST `-`. This is only unambiguous because JobIDs are 32 dashless hex characters (JobID.generate()); a job id containing a `-` would break this index. +- estateTag on QueueKit is `nonisolated(unsafe)` — set it ONCE at mount, before any drain() call, never again during concurrent use. Same discipline for the internal latencyWindow. +- awaitDrain / awaitDrain(stream:) are polling latches (20ms default interval, 30s default timeout), not push notifications — there is no native "queue just emptied" event on either backend. Always returns promptly on an already-empty queue; always throws QueueError.drainTimeout rather than hanging past the deadline. +- Telemetry never fabricates queue.depth=0 on a read failure — it emits queue.depth_unavailable instead, and skips every metric that depends on a known depth. Do not "simplify" this to `(try? pendingCount()) ?? 0`. +- Watcher wakes are ALWAYS advisory/spurious-tolerant. drainAvailable() (via atomic rename / serializable UPDATE) is the sole claim authority on every platform. +- DrainLease is heartbeat-TTL, not PID-liveness — worst-case takeover latency is one full TTL (15s). Do not add OS-specific process-liveness checks; portability across macOS/Linux/Windows is a design constraint, not an oversight. +- Pinned constants — changing requires a conformance regen across all 3 ports: staleTmpThreshold 300s, DrainLease ttl 15s / heartbeatInterval 5s, Watcher poll interval 200ms / Linux poll() timeout 500ms, QueueLatencyWindow capacity 100, awaitDrain pollInterval 20ms / timeout 30s, NovelToken-style file mode 0600 on queue files. +- ConvergenceKit is intentionally NOT a QueueKit dependency (spec §11) — it is application-layer composition sitting above this kit. Do not add it as a Package.swift dependency. diff --git a/packages/kits/QueueKit/docs/DETAILS.md b/packages/kits/QueueKit/docs/DETAILS.md new file mode 100644 index 0000000..78a8517 --- /dev/null +++ b/packages/kits/QueueKit/docs/DETAILS.md @@ -0,0 +1,567 @@ +--- +doc: DETAILS +package: QueueKit +repo: moot-system +authored_commit: 909513d0a8ecb1e9e903af9f4d25b3e4f2528242 +authored_date: 2026-07-04 +sources: + - path: Sources/QueueKit/DrainLease.swift + blob: 3f1df6aa1fab44c993bd49d96de179ff03450303 + - path: Sources/QueueKit/FilesystemBackend.swift + blob: c0e5f536e20b1def11875e57b587e8392b6f936d + - path: Sources/QueueKit/Job.swift + blob: 5f39e97a8113e47de2f2a18563446072de8ec6cf + - path: Sources/QueueKit/ObservationStatus.swift + blob: a32b0e8c93311ce498d12ffb4df864a2652c707e + - path: Sources/QueueKit/PersistenceKitBackend.swift + blob: 96175a8b27c8b6e929f417d95b141f727c5179d7 + - path: Sources/QueueKit/QueueBackend.swift + blob: eb67e0b821bc9bfdec8cfde621c5e70d4295d331 + - path: Sources/QueueKit/QueueError.swift + blob: 2697c4b7404b9e04259267cd8f4008030ebb6754 + - path: Sources/QueueKit/QueueKit.swift + blob: 60dfaa1e8f92ec051810b50d8b9cadc47388c02f + - path: Sources/QueueKit/QueueKitTelemetry.swift + blob: 29024f112bf133012283205175aa336b8d80d7c9 + - path: Sources/QueueKit/Watcher.swift + blob: cf2b270b9c60da34f7a25c016f8c18b6ba6149e4 +--- + +# QueueKit Details + +This document walks through every source file in the package. Read +`OVERVIEW.md` first for the big picture. Files appear here in pipeline +order: the wire-format types, the error vocabulary, the backend contract, +the public facade, the two concrete backends, and finally the supporting +infrastructure — the drain lease, the directory watcher, and telemetry. + +## Job.swift + +This file provides the wire-format types every job and completion signal is +built from: `JobID`, `StreamID`, `SessionID`, `ToolName`, `ArtifactRef`, +`CodableValue`, `Job` itself, `MissionContext`, `WireFormat`, and +`SignalFile`. + +A job needs an identifier that is both unique and safe to use as a filename, +because the filesystem backend uses it directly in a path. `JobID.generate()` +builds one from a UUID (a 128-bit random identifier), rendered as thirty-two +lowercase hexadecimal characters with the hyphens removed — this avoids +the hyphen colliding with the hyphen QueueKit uses as a separator inside +maildir filenames. `StreamID` names the workload a job belongs to (for +example, "encode" or "dreaming"); jobs on different streams can share one +queue without a consumer of one stream ever seeing the other's jobs. +`SessionID.mint()` stamps one drain call: every job a single `drain()` +claims shares one session identifier, which is what lets a caller retire +a whole claimed batch in one step later. + +`ArtifactRef` names something a completed job produced — a file, a commit, +a signal file, or a step in a recorded trajectory — tagged with an explicit +`type` field on the wire so a decoder can tell the four cases apart. +`CodableValue` is a caller-defined extensions blob: an open-ended value that +can be a string, a number, a boolean, null, an array, or a nested object. +It exists so a job's `extensions` dictionary can carry arbitrary structured +data supplied by whatever produced the job, and have every field of it +survive a `send()`/`drain()` round trip unchanged, without QueueKit having +to know the shape of that data in advance. + +`Job` is the unit of work: an identifier, a stream, an HLC submission +timestamp, a priority, a payload of raw bytes, and the extensions +dictionary. Its custom `Codable` conformance is not decorative — it +implements the exact wire format the package specification fixes: keys are +`snake_case` (`stream_id`, not `streamID`), the HLC nests as +`physical_time`/`logical_count`/`node_id`, and the payload is base64url +text, not raw JSON bytes. This wire format is a cross-language contract: the +Rust and Python ports must produce and consume the identical bytes, so every +field name, nesting shape, and encoding choice here is pinned, not a matter +of Swift convention. + +`MissionContext` is the one type in this file that uses ordinary Swift +`Codable` key synthesis (`camelCase` property names, no custom coding keys) +rather than the pinned wire format, because it is not part of the job +envelope; it is a caller-supplied payload shape that has no cross-language +byte-identity requirement. + +`WireFormat.filename(for:)` builds the canonical maildir filename for a job: +a sortable HLC string, the stream identifier, and the job identifier, +joined by hyphens. `WireFormat.sortableHLC(_:)` is why the filename sorts +correctly by claim order: it zero-pads each HLC component to a fixed width +(sixteen digits for physical time, eight for the logical count, ten for the +node identifier) so that ordinary lexicographic string sorting — the only +sort a plain directory listing gives you — matches HLC's own ordering. +`WireFormat.encoder` fixes sorted keys and no escaped slashes, so the exact +same job encodes to the exact same bytes on every run and every platform; +this is what makes the shared conformance fixtures meaningful. + +`SignalFile` is the durable record a completed job leaves behind: which job, +what terminal status it ended in, what artifacts it produced, and when. Its +`Codable` conformance mirrors `Job`'s: pinned `snake_case` keys and a nested +HLC, because a signal file is read by whichever consumer or tool eventually +inspects the `done/` directory, in whichever language that consumer happens +to be written in. + +## ObservationStatus.swift + +This file provides `ObservationStatus`, the small enumeration a job's +outcome is reported in: `.running`, `.done`, `.doneWithConcerns`, +`.needsContext`, or `.blocked`. + +The type is owned here, not duplicated, because more than one package needs +to agree on what a finished job's outcome can be; AgentHarness, mentioned in +the file's header comment, imports it directly from QueueKit rather than +declaring its own copy. `isTerminal` is the one piece of logic on the type: +only `.running` is non-terminal. `reply(to:status:)` on the `QueueKit` +facade checks this before accepting a completion, so a caller cannot +accidentally "finish" a job with the status that means it has not finished. + +## QueueError.swift + +This file provides `QueueError`, the complete error vocabulary a caller of +QueueKit can receive. + +Each case names one specific failure: a directory could not be created, a +write failed, a rename failed, a job's stored bytes could not be decoded, a +job identifier could not be found, the watcher failed, a stale temporary +file was found, the backend is unavailable, a caller tried to complete a job +with a non-terminal status, `awaitDrain` timed out, or a caller-supplied +identifier was unsafe to use in a path. Keeping every distinct failure as +its own case, rather than a generic wrapped error, is what lets a caller +handle a `jobNotFound` differently from a `renameFailed` without inspecting +error text. `drainTimeout(pending:inFlight:)` and +`invalidIdentifier(id:reason:)` each carry the diagnostic detail a caller +needs to explain, respectively, how far the queue was from empty when +`awaitDrain` gave up, and exactly why a submitted identifier was rejected. + +## QueueBackend.swift + +This file provides `QueueBackend`, the protocol every concrete backend +conforms to, plus two protocol extensions supplying default +implementations for the operations a backend does not have to implement +specially. + +The protocol is the seam that lets the public `QueueKit` class stay ignorant +of which storage mechanism is underneath it: every method the facade +exposes has a matching method here, and `QueueKit` does nothing but forward +to whichever `QueueBackend` it was mounted with. `write(_:)`, +`drainAvailable()`, `watch(handler:)`, `complete(_:status:artifacts:)`, +`inFlight()`, and `completed(streamID:)` are the core operations every +backend must implement from scratch. + +`writeBatch(_:)` and `completeBatch(_:)` are bulk twins of `write` and +`complete`. The first default extension implements them by simply looping +the single-item version — correct for any backend, but not necessarily +fast. A backend earns the right to skip that loop by overriding these two +methods with a mechanism that pays a shared cost once for the whole batch +instead of once per job; both concrete backends do exactly that, for +reasons explained under their own headings below. + +`drainAvailable(stream:)` and `pendingCount(stream:)` are the stream-scoped +forms of `drainAvailable()` and `pendingCount()`. The second protocol +extension's default implementation of `drainAvailable(stream:)` claims +every stream's jobs through the all-streams `drainAvailable()` and then +filters the result down to the requested stream — which is correct, but +claims jobs belonging to other streams that then have to be re-enqueued, so +the file's own comment warns that a concrete backend should override it +rather than rely on the default. Both `FilesystemBackend` and +`PersistenceKitBackend` do override it, each in the way suited to its own +storage. + +## QueueKit.swift + +This file provides the public facade: the `QueueKit` class itself, its +maildir directory-management helpers, and the four permanent methods — +`send`, `drain`, `watch`, `reply` — that every caller uses regardless of +which backend is mounted underneath. + +`QueueKit` holds a `backend` (anything conforming to `QueueBackend`), an +optional `root` URL (present only when the mounted backend is a +filesystem backend), a rolling latency window for telemetry, and an +`estateTag` string used to tag telemetry metrics by estate. Two +initializers exist: `init(root:hlcGenerator:)` is the common path — it +builds a `FilesystemBackend` at the given root and cleans any stale +temporary files left behind by a prior crash. `init(backend:root:)` mounts +an already-constructed backend directly; this is how `PersistenceKitBackend` +gets attached, and how tests substitute an in-memory backend. + +`send(_:)` and its batch twin `send(batch:)` hand jobs to the backend. +The batch form exists because the filesystem backend's per-job `write` +pays a filesystem synchronization cost — a call that forces the operating +system to guarantee the write has actually reached durable storage — on +every single job; a bulk producer enqueuing tens of thousands of jobs at +once (a full reindex, for instance) can instead route through +`writeBatch`, which the filesystem backend implements as one durability +barrier for the whole batch rather than one per job. + +`drain()` and the stream-scoped `drain(stream:)` claim available jobs and, +around the backend call, measure how long the claim took and report it +through `reportQueueStats`. Both return the same shape: a list of pairs, +each a claimed `Job` and the `SessionID` that drain call minted. + +`watch(handler:)` subscribes a handler that is invoked for every job as it +becomes available; the facade does nothing but forward to the backend's own +`watch`, because the two backends drive this very differently (a directory +watcher for the filesystem backend, a database change observer for the +PersistenceKit backend). + +`reply(to:status:artifacts:)` is the single-job completion path; it checks +`status.isTerminal` before forwarding to the backend, so a caller cannot +mark a job "complete" with the one status (`.running`) that means it is +not. `reply(session:status:)` completes every job claimed under one drain +call's session in a single pass; it only has a fast path on +`PersistenceKitBackend` (checked with an `as?` cast), because only that +backend's shared table can look up an entire session's claimed rows in one +guarded update. On any other backend it returns `0`, signaling the caller +to fall back to per-job `reply`. `reply(batch:)` completes an explicit list +of job/status pairs in one pass, routing to the backend's `completeBatch`; +this is the path a corpus-drain caller uses on the filesystem backend, which +has no session fast path but does have a batched completion. + +`reclaimInFlight(stream:)` resets every job stuck in-flight for one stream +back to pending, so the next `drain(stream:)` re-claims and re-drives it. +The doc comment is explicit about the precondition: call this only +immediately after successfully acquiring that stream's `DrainLease`, because +a freshly acquired lease is the guarantee that the job's prior claimant is +actually gone, not merely slow — without that guarantee, this method could +yank a job out from under a drain worker that is still processing it. Like +`reply(session:status:)`, it only has a fast path on `PersistenceKitBackend`; +`FilesystemBackend` instead exposes an unscoped `reclaimInFlight()` that +runs once at mount, because its `cur/` directory has no stream separation +in the first place. + +`inFlight()`, `pendingCount()`, and `pendingCount(stream:)` are read-only +depth probes: how many jobs are claimed but not yet replied to, and how +many are waiting to be claimed. `pendingCount() + inFlight().count` is the +total outstanding work a drain still has left to do — a status reader can +compute that without claiming or draining anything. + +`awaitDrain(pollInterval:timeout:)` and its stream-scoped twin +`awaitDrain(stream:pollInterval:timeout:)` block until a queue (or one +stream of it) is empty on both frontiers: nothing pending, nothing +in-flight. Both are polling loops, not push-based latches, because the +maildir backend has no native "the queue just emptied" event to wait on; +each iteration re-reads the live counts, so progress made by a concurrently +running drain worker between polls is observed on the very next tick. Both +return promptly when the queue is already empty — the first check can +succeed without ever sleeping — and both throw `QueueError.drainTimeout` +rather than blocking forever if the deadline passes with work still +outstanding, so a stuck or crashed drain worker surfaces as an error +instead of a hang. + +`ensureMaildir(root:)` and `cleanStaleTmpFiles(root:)` are the maildir +directory-management helpers `FilesystemBackend`'s initializer calls: the +first creates the four subdirectories (`tmp`, `new`, `cur`, `done`) if +absent, and the second deletes any file in `tmp/` older than +`staleTmpThreshold` (five minutes). A file stranded in `tmp/` means a +process crashed between writing it and renaming it into `new/`; because it +was never visible in `new/`, no consumer could have claimed it, so removing +it on the next mount is safe. + +## FilesystemBackend.swift + +This file provides `FilesystemBackend`, the POSIX maildir-style queue +implementation: no database, just four directories and atomic filesystem +operations. Its semantics are deliberately derived from `deliver_maildir()`, +the mail-delivery routine at the heart of the Postfix mail server — a +decades-proven pattern for handing off many small files between +uncoordinated processes without a lock file. + +`validateIdentifier(_:)` runs before any filesystem path is built from a +caller-supplied identifier (a stream identifier or a job identifier). It +rejects an empty string, a `.` or `..` path component, a `/` or `\` path +separator, and any ASCII control character. Without this check, a +maliciously or accidentally crafted stream identifier containing a path +separator could make a job's path resolve outside the queue's root +directory — a path-traversal risk. The Rust and Python ports enforce the +identical rule. + +`write(_:)` implements the maildir handoff in the classic three steps: +create the job's encoded bytes in `tmp/` with a mode that only the owning +process can read (defense in depth, since a queued payload may carry +sensitive estate content), force those bytes to durable storage, then +atomically rename the file into `new/`. `atomicWriteAndRename(...)` is +where those steps actually happen, using `open` with `O_CREAT | O_EXCL` (an +exclusive create only one caller can win), then a raw `write`/`fsync` +sequence, then `rename`, then a final `fsync` of the destination +directory itself — an extra step needed because on POSIX filesystems, a +directory's own metadata (which files it currently contains) is a separate +piece of durable state from the file contents, and both must be forced to +disk for the write to survive a crash. A `rename` failure because the +destination temporarily vanished (`ENOENT`) is retried once, since another +process cleaning `new/` mid-flight is a recoverable race, not a fatal one. + +`writeBatch(_:)` is the bulk twin: it writes and renames every job in the +batch with `writeAndRenameNoFsync`, which skips the per-file durability +step, and forces only `new/`'s directory metadata to disk once at the end. +A crash before that single barrier can lose some just-written jobs, but the +comment records why this is an acceptable trade for a bulk producer +specifically: the reindex that calls `writeBatch` derives its jobs from the +estate itself, so a lost job is simply re-enqueued on the next resume — an +at-least-once guarantee, not a promise that every job survives every +possible crash instant. Ordinary streaming `write` keeps its per-job +durability unchanged; only a caller that opts into the batch API accepts +this weaker bound. + +`pendingCount()` counts the files in `new/` — one file, one pending job, +no decoding required, which is why it stays cheap even as a queue grows. + +`drainAvailable(stream:)` is the stream-scoped claim. Because the maildir +filename already encodes the stream (`{hlc}-{stream}-{jobid}`), a naive +implementation could filter by filename alone, but the code instead reads +and decodes each file's actual `streamID` field before claiming it, leaving +files whose stream does not match entirely untouched in `new/` — a +concurrent drainer for a different stream is never affected by this one's +scan. `pendingCount(stream:)` performs the equivalent non-claiming count. + +`drainAvailable()`, the all-streams claim, renames every file in `new/` +into `cur/` first and only then reads and decodes each one; a file that +fails to decode is moved straight to `done/` rather than left to jam every +future drain pass, since a job whose stored bytes are unparseable can never +become processable by simply trying again. + +`reclaimInFlight()` moves everything left in `cur/` back to `new/`. Its doc +comment states its precondition plainly: call it only at mount, before any +drain session is live, because a freshly started process cannot have +claimed anything itself, so anything already sitting in `cur/` must be a +crash orphan from whichever process ran before it. + +`watch(handler:)` wires `Watcher.watchNewDirectory` to `drainAvailable()`: +every time the watcher signals that `new/` may have changed, the backend +drains whatever is currently claimable and hands each job to the caller's +handler. + +`complete(_:status:artifacts:)` finds the job's file in `cur/` by matching +the filename suffix (every maildir filename ends in `-{jobID}`), writes the +`SignalFile` recording the outcome, and only then moves the job's own file +into `done/`. Writing the signal first matters: if the process crashes +between the two steps, the job is still visible in `cur/` on the next +mount, ready to be reclaimed, rather than silently vanished with an +orphaned signal and no job record. + +`completeBatch(_:)` is the batched form: one scan of `cur/` builds a +job-identifier-to-filename index, every completion in the batch is written +without a per-file durability step, and one call forces `done/`'s directory +metadata to disk at the very end. The safety argument mirrors +`writeBatch`'s: a crash before that final barrier leaves the not-yet-moved +jobs in `cur/`, where they are reclaimed and reprocessed — safe because +ingest is idempotent (processing the same job twice causes no harm). + +`inFlight()` and `completed(streamID:)` both delegate to the private +`listJobs(in:filter:)`, which lists a directory, skips `.signal` sidecar +files, decodes each remaining file as a `Job`, and optionally filters by +stream. + +## PersistenceKitBackend.swift + +This file provides `PersistenceKitBackend`, the second concrete backend: +jobs live as rows in a table inside PersistenceKit, MOOTx01's storage kit, +rather than as files on disk. The file's own header comment states five +invariants the implementation is built to preserve; the sections below +explain each one as it is enforced in code. + +`QueueKitSchema.declaration()` defines the table's shape: one row per job, +with columns for the job's own fields plus queue bookkeeping — `status` +(`"new"`, `"cur"`, or `"done"`), `signal_status` and `artifacts` (filled in +only once a job completes), and `session_id` (which drain call, if any, +currently holds this row). Three indices back the operations that matter: +one for a plain status count, one on `(status, physical_time, +logical_count, node_id)` for claiming jobs in HLC order, and one on +`(stream_id, status)` for the stream-scoped operations. The table's +`appendOnly` flag is explicitly `false` — invariant 5 — because a queue row +must be mutable: the same row moves from `new` to `cur` to `done` in place. + +`write(_:)` is a bare insert into the table with `status: "new"` — no +enclosing transaction, per invariant 1. A transaction would offer no benefit +here (there is nothing else to make atomic with a single insert) and would +only add overhead to the hottest, most frequent operation in the backend. + +`writeBatch(_:)` wraps every insert in the batch in one `.readCommitted` +transaction — the transactional-isolation equivalent of the filesystem +backend's single durability barrier. The comment explains why the weaker +`.readCommitted` isolation (rather than the `.serializable` isolation the +claim uses) is still correct here: these are all inserts of brand-new rows, +which cannot conflict with the claim's job of flipping existing `new` rows +to `cur` — a claim that begins before the batch commits simply will not see +the batch's rows yet, and will see them on its next pass instead. + +`pendingCount()` and `pendingCount(stream:)` are single-row-count reads +with a `status = 'new'` predicate (and, for the stream form, an additional +`stream_id` predicate) — no claim, no lock beyond an ordinary read. + +`drainAvailable(stream:)` and the all-streams `drainAvailable()` both +implement the claim as one `.serializable` transaction containing exactly +two steps: an `UPDATE` that flips every matching `new` row to `cur` under +one freshly minted session identifier, and a `SELECT` that reads back +precisely the rows carrying that session identifier, ordered by HLC. This +single-pass shape — invariant 3, the status-guarded atomic transition — is +what the file's comment calls out as a deliberate improvement over an +earlier design that issued one guarded update per row: claiming a batch of +jobs that way cost one predicate scan per job, so a bulk claim over a large +queue became quadratic in the number of jobs. Reading back by this call's +own unique session, rather than by `status = 'cur'` generally, is what keeps +two concurrent drainers from ever double-counting the same claimed row — +each call's session tags only the rows it itself just claimed. + +`completeSession(_:status:)` completes every row claimed under one session +in a single guarded update — the batch twin of the per-job `complete`, and +the reason `QueueKit.reply(session:status:)` has a fast path only on this +backend. It is declared directly on the class rather than as part of the +`QueueBackend` protocol, because only the concrete `PersistenceKitBackend` +handle (not the abstract protocol) has anything meaningful to complete a +whole session against. + +`watch(handler:)` subscribes to `storage.observer`, per invariant 2, and +treats every event strictly as a wake signal — it never reads a job's +fields out of the event payload itself, only out of a subsequent +`drainAvailable()` call. `drainUntilEmpty` is the loop this drives: rather +than draining once per event, it drains repeatedly until a pass claims +nothing, because a burst of inserts can coalesce into fewer observer events +than there are actual rows, and draining only once per event would strand +whichever rows' wake got folded away. The initial drain before the +subscription's first `await` matters for the same reason from the other +direction: a job inserted between mount and subscription would otherwise be +invisible to any wake, since its insert event predates the subscription. + +`complete(_:status:artifacts:)` is a single guarded update keyed on both +the job identifier and `status = 'cur'`; if nothing matched, the job was +never claimed (or was already completed), so the method throws +`QueueError.jobNotFound` rather than silently doing nothing. + +`reclaimInFlight(stream:)` resets a stream's `cur` rows back to `new`, +clearing `session_id`. Its doc comment repeats the same safety argument as +the facade method it backs: this is safe to call only immediately after +successfully acquiring the stream's `DrainLease`, because the lease's +staleness gate is what rules out a live drainer still holding those rows. + +`inFlight()` and `completed(streamID:)` both route through the private +`listJobs(status:streamID:)`, and `decodeRow(_:)` turns one database row +back into a `Job`, falling back to an empty extensions dictionary if the +stored JSON for it fails to parse rather than failing the whole read. + +## DrainLease.swift + +This file provides `DrainLease`, a heartbeat-based, stream-keyed lock that +lets many processes share one estate's queue while guaranteeing that only +one of them actively drains a given stream at a time. Two different streams +get entirely independent leases and can both be held simultaneously — this +is what lets, for example, an encode drainer and a dreaming drainer run at +once without either blocking on the other. + +The design is deliberately not a check on whether the prior holder's +process is still alive. A PID-liveness check would need OS-specific code — +querying the process table through platform-specific system calls — which +the file's header comment rules out in favor of something that works +identically on macOS, Linux, and Windows: a lease file holding an owner +string and a wall-clock timestamp. `tryAcquire(now:)` succeeds when the +lease file is absent, already owned by the caller, or its timestamp is +older than `ttl` (fifteen seconds) — in every other case, another drainer +holds a fresh lease, and the caller must stand down. The owner string +itself combines the process identifier with a caller-supplied nonce, so +that if the operating system reuses a process identifier after a crash, the +reused identifier cannot be mistaken for the original holder. + +`heartbeat(now:)` refreshes the timestamp while a drainer actively holds +the lease; the doc comment directs a caller to invoke it roughly every five +seconds — well inside the fifteen-second TTL — so an ordinary slow drain +pass never has its own lease expire out from under it. `isHeldByOther(now:)` +answers the same freshness question as `tryAcquire` without attempting to +take the lease, for a caller that only wants to check before deciding +whether to try. `release()` deletes the lease file on clean shutdown, but +only if the caller still owns it, so that a takeover that already happened +is never undone by a late release from the prior holder. + +The worst-case cost of a crashed holder is one full TTL: a takeover cannot +happen sooner than fifteen seconds after the last heartbeat, because a +fresher lease has to be honestly waited out. This is the trade the design +accepts for staying free of any OS-specific liveness check. + +## Watcher.swift + +This file provides `Watcher`, the directory-change wake source +`FilesystemBackend.watch()` is built on. Its job is narrow and its contract +is explicit: call `onChange` whenever the watched directory's contents may +have changed, allowing spurious wakes, because `drainAvailable()` is always +the actual authority on what can be claimed — a false-positive wake costs +one wasted scan, never a correctness problem. + +Three platform strategies exist, tried in order of preference. +`watchKQueue` (Darwin only, meaning macOS and iOS) opens the directory with +`O_EVTONLY` and attaches a `DispatchSource` watching kqueue VNODE events — +write, extend, delete, rename — giving wake latency in the microseconds +with no external dependency. `watchLinux` first attempts `openInotify`, +which sets up an inotify watch for `IN_CREATE` and `IN_MOVED_TO` (the +second of which is the event the atomic tmp-to-new rename actually fires); +`runInotifyLoop` then polls that file descriptor with a 500-millisecond +timeout so that task cancellation is still checked promptly between +events, and reads and discards whatever event bytes arrive, since the +handler only needs to know that something happened, not what. If +`openInotify` cannot establish a watch — the kernel's inotify watch limit +is exhausted, or the filesystem does not support inotify — `watchLinux` +falls back automatically to `watchPoll`. Every other platform uses +`watchPoll` directly: a plain 200-millisecond snapshot comparison of the +directory's file listing, correct everywhere but capable of missing a +change for up to one poll interval (100 milliseconds on average). + +Every path calls `onChange` once immediately, before entering its +platform-specific wait loop, so that any work already present when the +watcher attaches is not stranded waiting for the next actual filesystem +event. `ContinuationBox` is the small bridge the Darwin path uses to let an +`async` caller wait on a `DispatchSource`'s cancel handler, which fires on +an ordinary `DispatchQueue` callback rather than inside Swift's structured +concurrency. + +## QueueKitTelemetry.swift + +This file provides `QueueLatencyWindow` and `reportQueueStats`, the +self-report telemetry QueueKit emits through IntellectusLib, MOOTx01's +telemetry library, after every `drain()` call. + +The reporting path is additive and deliberately cheap when unused: the +very first check inside `reportQueueStats` is a single atomic flag read, +`Intellectus.isEnabled`, and the function returns immediately when it is +false, at a cost the file's header comment puts at about one nanosecond. +When telemetry is enabled, the function makes one `pendingCount()` call and +emits up to six named metrics, all tagged with the estate identifier and +the kit name so a fleet of estates can be compared metric by metric. + +`QueueLatencyWindow` keeps a fixed-capacity rolling list of recent drain +latencies (one hundred samples by default) and computes an arbitrary +percentile from it on demand. `percentile(_:)` guards its input explicitly: +a percentile argument that is not a finite number between 0 and 100 (for +example `NaN` or a value produced by a caller's arithmetic error) returns +`0` rather than computing an array index from it, because converting a +non-finite floating-point value to an integer index can crash the program +outright on some inputs. + +`reportQueueStats` treats a failed `pendingCount()` read as its own signal, +not as a zero. Reporting `queue.depth = 0` when the read actually failed +would be indistinguishable from an honestly empty queue, and would tell +whoever is watching the telemetry "everything is drained" when the truth is +"the read itself did not work." Instead, a failed read skips the +`queue.depth` metric entirely, in favor of a dedicated +`queue.depth_unavailable` counter, and every metric that can only be +computed honestly from a known depth (`queue.idle_nonempty`, part of +`queue.head_of_line_age_s`) is skipped along with it. `queue.drain_count` +and the two latency percentiles have no such dependency and are always +reported. + +## Rust and Python Ports and Conformance + +The `rust/` directory contains a second, independent implementation of the +package: eight source files (`error.rs`, `job.rs`, `backend.rs`, +`filesystem.rs`, `facade.rs`, `drain_lease.rs`, and, behind a +`persistencekit` feature flag, `persistencekit.rs`), reimplementing the +facade, both backends, and the drain lease for non-Apple hosts. The +`python/queuekit/` directory contains a third implementation covering only +`FilesystemBackend` — the package specification restricts Python to that +one backend, since Python tooling has no need of the PersistenceKit +database path. + +All three implementations are gated by the same conformance fixtures, kept +in `Tests/QueueKitTests/Fixtures/` as recorded job and completion-signal +input/output pairs. Each fixture pins one exact set of bytes; Rust's own +`rust/tests/conformance.rs` loads the identical fixture files the Swift +test suite reads and asserts that Rust's encoder produces the same bytes +Swift already produced. This is what turns "the wire format is a contract" +from an assertion in this document into something a test suite actually +checks: change a field name, a key order, or an encoding rule in any one +implementation, and the shared fixtures catch the drift immediately in +whichever implementation was not updated to match. diff --git a/packages/kits/QueueKit/docs/OVERVIEW.md b/packages/kits/QueueKit/docs/OVERVIEW.md new file mode 100644 index 0000000..c328d9c --- /dev/null +++ b/packages/kits/QueueKit/docs/OVERVIEW.md @@ -0,0 +1,172 @@ +--- +doc: OVERVIEW +package: QueueKit +repo: moot-system +authored_commit: 909513d0a8ecb1e9e903af9f4d25b3e4f2528242 +authored_date: 2026-07-04 +sources: + - path: Sources/QueueKit/DrainLease.swift + blob: 3f1df6aa1fab44c993bd49d96de179ff03450303 + - path: Sources/QueueKit/FilesystemBackend.swift + blob: c0e5f536e20b1def11875e57b587e8392b6f936d + - path: Sources/QueueKit/Job.swift + blob: 5f39e97a8113e47de2f2a18563446072de8ec6cf + - path: Sources/QueueKit/ObservationStatus.swift + blob: a32b0e8c93311ce498d12ffb4df864a2652c707e + - path: Sources/QueueKit/PersistenceKitBackend.swift + blob: 96175a8b27c8b6e929f417d95b141f727c5179d7 + - path: Sources/QueueKit/QueueBackend.swift + blob: eb67e0b821bc9bfdec8cfde621c5e70d4295d331 + - path: Sources/QueueKit/QueueError.swift + blob: 2697c4b7404b9e04259267cd8f4008030ebb6754 + - path: Sources/QueueKit/QueueKit.swift + blob: 60dfaa1e8f92ec051810b50d8b9cadc47388c02f + - path: Sources/QueueKit/QueueKitTelemetry.swift + blob: 29024f112bf133012283205175aa336b8d80d7c9 + - path: Sources/QueueKit/Watcher.swift + blob: cf2b270b9c60da34f7a25c016f8c18b6ba6149e4 +--- + +# QueueKit Overview + +## What This Library Does + +QueueKit is a general-purpose work queue. It lets one part of MOOTx01 hand a +unit of work to another part without the two parts running in the same +process, or even at the same moment. MOOTx01 is an on-device AI memory +system. It stores what an AI observes over time and helps the AI recall it +later. Many of its jobs — encoding a memory, running a dreaming pass over the +estate, replaying a signal — are best done off to the side, by a separate +worker, so the part of the system that noticed the work does not have to wait +for it to finish. + +QueueKit is a Kit. A Kit is a larger package that composes libraries into a +subsystem; kits may depend on libraries, never the reverse. QueueKit lives in +moot-system, the repository that houses the kits that hold MOOTx01 together +at the system level. + +The unit of work is called a `Job`. A job carries an identifier, the name of +the stream it belongs to, a submission timestamp, a priority, and an +arbitrary payload of bytes. A producer calls `send` to hand a job to the +queue. A consumer calls `drain` to claim available jobs, does the work, and +calls `reply` to mark each job finished. QueueKit exposes exactly four +permanent methods for this cycle — `send`, `drain`, `watch`, `reply` — no +matter which storage mechanism backs the queue underneath. + +## The Problem It Solves + +Handing work across a process boundary is harder than it looks. Three +requirements make it hard, and QueueKit exists to meet all three at once. + +First, the handoff must survive a crash. If the producing process, the +queue itself, or the consuming process dies partway through, no job may be +silently lost or silently claimed twice. A job is claimed exactly once at a +time, and a completed job leaves a durable record of how it finished. + +Second, independent workloads must not interfere with each other while +sharing one queue. An encode worker and a dreaming worker might both draw +from the same estate's queue, but neither should be able to claim, or block +on, the other's jobs. QueueKit calls a workload a stream, and every +operation has a stream-scoped form so that consumers can share one queue +without stepping on each other. + +Third, the format on disk or in the database must mean the same thing no +matter which programming language wrote or reads it. MOOTx01 ships Swift +code for Apple devices, Rust code for cross-platform daemons and services, +and Python code for command-line tooling. A job enqueued by one language's +producer must be decodable, byte for byte, by another language's consumer. +QueueKit therefore treats its wire format — the exact bytes a job or a +completion signal is encoded as — as a contract, gated by conformance +fixtures shared across all three implementations. + +## How It Works + +QueueKit separates the fixed four-method interface from the storage +mechanism underneath it. A backend is anything that conforms to the +`QueueBackend` protocol: it can write a job, claim the available jobs, watch +for new ones, and mark a job complete. The public `QueueKit` class does not +know or care which backend it holds; it forwards every call and adds only +the telemetry and the await-drain convenience described below. + +Two backends ship today. `FilesystemBackend` needs no database. It lays out +four subdirectories under a root — `tmp`, `new`, `cur`, `done` — following +the maildir layout that mail servers such as Postfix have used for decades +to hand off messages between processes without a lock file: a job is written +into `tmp`, then atomically renamed into `new`; claiming it is another atomic +rename into `cur`; finishing it renames it into `done`. Every step relies on +the filesystem's own atomicity guarantee, so two processes racing to claim +the same job can never both win. + +`PersistenceKitBackend` stores jobs as rows in a table inside PersistenceKit, +MOOTx01's storage kit. It suits an estate that already keeps a shared, +encrypted SQLite database open for other purposes. Claiming a batch of jobs +here is one guarded database transaction that flips every available row from +`new` to `cur` under a single session identifier, then reads back exactly the +rows that transaction claimed — an approach that scales as one pass over the +table rather than one query per job. + +Every job carries a submission timestamp built from a Hybrid Logical Clock, +or HLC — a clock design that combines wall-clock time with a small counter +so that events from different machines or processes still sort into one +consistent, gap-free order, even when the machines' clocks disagree slightly. +QueueKit uses the HLC ordering to decide which job is oldest, both when a +consumer wants first-in-first-out delivery and when telemetry reports how +long the oldest waiting job has been sitting in the queue. + +Two supporting mechanisms keep multi-process operation honest. `DrainLease` +is a heartbeat lock, keyed by stream, that lets many processes share one +estate while guaranteeing that only one of them actively drains a given +stream at a time; a stale lease (its holder has not renewed it recently) is +automatically taken over. `Watcher` gives the filesystem backend an +efficient way to notice new work without spinning a loop: it uses the +kqueue mechanism on Apple platforms, inotify on Linux, and a plain +200-millisecond poll everywhere else, falling back automatically if the +faster mechanism cannot be set up. + +## How the Pieces Fit + +Figure 1 shows the library's topology — its major parts and how a job moves +through them. + +![Figure 1. Topology of QueueKit](topology.svg) + +*Figure 1. Topology of QueueKit. A producer calls the `QueueKit` facade, +which delegates to whichever backend is mounted. Dashed regions mark +external kit boundaries — the encrypted database PersistenceKit owns, and +the telemetry sink IntellectusLib owns — plus the maildir directories that +are the filesystem backend's own on-disk state.* + +A caller mounts `QueueKit` once, over either backend, and afterward talks +only to the facade. `send` and the batch form `send(batch:)` hand jobs to +the backend. `drain` and the stream-scoped `drain(stream:)` claim available +jobs and return them together with a session identifier that groups +everything claimed in that one call. `watch` subscribes a handler that fires +for every job as it becomes available, driven by `Watcher` on the filesystem +backend and by a database change observer on the PersistenceKit backend. +`reply` — in its single-job, per-session, and per-batch forms — marks +claimed jobs finished with a terminal status and, on the filesystem backend, +writes a durable signal file recording how the job ended. + +`awaitDrain` is a convenience built on top of the four permanent methods +rather than a new backend capability: it polls the pending count and the +in-flight count until both reach zero, so a bulk caller — an importer +finishing a batch, a test verifying a pipeline emptied — can wait for every +enqueued job to finish without wiring its own polling loop. + +Telemetry is additive and off by default. When enabled, `QueueKitTelemetry` +reports queue depth, drain throughput, claim latency percentiles, and the +age of the oldest waiting job to IntellectusLib, MOOTx01's self-report +telemetry library, tagged by estate so a fleet of estates can be watched +side by side. + +## What Ships in the Package + +The package ships ten Swift source files implementing the facade, the two +backends, and their supporting infrastructure; a Rust port in `rust/` (eight +source files, roughly three thousand lines) that reimplements both backends +for non-Apple hosts; and a Python port in `python/queuekit/` that +reimplements the filesystem backend only, for command-line tooling that has +no need of a database backend. A shared set of conformance fixtures under +`Tests/QueueKitTests/Fixtures/` — recorded job and completion-signal +input/output pairs — gates all three implementations to the same wire +format, byte for byte. diff --git a/packages/kits/QueueKit/docs/topology.svg b/packages/kits/QueueKit/docs/topology.svg new file mode 100644 index 0000000..e688be5 --- /dev/null +++ b/packages/kits/QueueKit/docs/topology.svg @@ -0,0 +1,128 @@ + + + + + + + + + + + + + QueueKit: a job crosses a process boundary + + + + Producer + send / send(batch:) + + + QueueKit facade + drain · watch · reply + awaitDrain + + + FilesystemBackend + POSIX maildir, no DB + + + PersistenceKitBackend + shared SQLite table + + + Consumer + drain worker / watch handler + + + + + + + + + + + reply(to:status:) / reply(session:) / reply(batch:) + + + + Watcher + kqueue · inotify · poll + + + DrainLease + per-stream heartbeat TTL + + + QueueKitTelemetry + queue.* metrics + + + + wake (spurious OK) + + + + + gates reclaimInFlight + + + + reportQueueStats after drain() + + + + FilesystemBackend's own on-disk state + + Maildir + tmp/ → new/ → cur/ → done/ + + + External kit boundary — PersistenceKit + + queue.sqlite + queuekit_jobs table + + + External kit boundary — IntellectusLib + + Telemetry sink + off by default, ~1 ns gate + + + + + + diff --git a/packages/libs/LoopbackHTTP/docs/AGENT_MAP.md b/packages/libs/LoopbackHTTP/docs/AGENT_MAP.md new file mode 100644 index 0000000..2cf0aa5 --- /dev/null +++ b/packages/libs/LoopbackHTTP/docs/AGENT_MAP.md @@ -0,0 +1,79 @@ +--- +doc: AGENT_MAP +package: LoopbackHTTP +repo: moot-system +authored_commit: 909513d0a8ecb1e9e903af9f4d25b3e4f2528242 +authored_date: 2026-07-04 +sources: + - path: Sources/LoopbackHTTP/HTTPWire.swift + blob: c07525c2583c06aa8406287eef5cb17279b9b86a + - path: Sources/LoopbackHTTP/POSIXSocket.swift + blob: 6dfba4dee7ad3121d47c8bfc937854c6f13ad1b1 +--- + +# AGENT_MAP — LoopbackHTTP + +PURPOSE: zero-dependency, loopback-pinned HTTP/1.1 server primitive (transport + wire parsing/framing only; no auth/policy). Bind loopback socket → accept → recv bytes → HTTPRequest.read parses → caller builds HTTPResponse or drives SSEStream → send/sendAll writes bytes back. + +DEPS: imports Foundation (HTTPWire.swift); Glibc (Linux) / Darwin (Apple) via `#if canImport(Glibc)` guard (POSIXSocket.swift). No external SwiftPM packages (zero-dependency kit rule). No rust/ port — extraction target is OS transport glue, exempt from the Swift/Rust parity discipline (ADR-LOOPBACKHTTP-001); parity for the MCP transport is enforced at the JSON-RPC wire, not here. Imported by: moot-mgr (monitor daemon dashboard read-API + control listener), resident mootx01 MCP daemon (JSON-RPC transport, SSE notifications). + +ENTRY POINTS (most callers need only these): +- POSIXSocket.swift:43 `POSIXSocket.listenLoopbackTCP(port:) throws -> (fd, port)` — bind+listen, loopback-only +- HTTPWire.swift:95 `HTTPRequest.read(fd:maxHeaderBytes:maxBodyBytes:) -> HTTPRequest?` — parse one request off a connected fd +- HTTPWire.swift:201 `HTTPResponse.send(fd:)` — serialize + write one response +- HTTPWire.swift:252/:259 `SSEStream.writeHead()` / `.send(_:)` — SSE head + data frames + +## Symbol Table + +### Transport — POSIXSocket.swift +- :32 `enum POSIXSocket` — namespace; all calls synchronous/blocking, run off cooperative pool +- :43 `listenLoopbackTCP(port:) throws -> (fd: Int32, port: UInt16)` — binds hard-pinned to 127.0.0.1 (never INADDR_ANY); port 0 = OS-assigned, read back via getsockname; listen backlog 16 +- :91 `listenUnix(path:) throws -> Int32` — UDS listener; unlinks stale path first; chmod 0600 BEFORE listen (closes bindable-but-world-readable window); listen backlog 16 +- :134 `acceptOne(_ listenFD:) -> Int32?` — accept one conn; nil on failure, caller decides retry +- :141 `recv(_:max:) -> Data?` — up to `max` bytes; empty non-nil Data = clean EOF; nil = read error +- :150 `sendAll(_:_:) -> Bool` — loops write() until all bytes sent or a write fails; single source of truth for "fully flushed to socket" used by both HTTPResponse.send and SSEStream +- :166 `enum SocketError: Error, Sendable, Equatable` — structured errors (project rule: enums not bare optionals) +- :168 `case syscall(String, Int32)` — named syscall + errno +- :170 `case pathTooLong` — UDS path exceeds fixed sun_path buffer + +### Request — HTTPWire.swift +- :27 `struct HTTPRequest: Sendable` — method/path(no query)/query(raw, no '?')/headers(lowercased keys)/body +- :36 `init(method:path:query:headers:body:)` — public memberwise; used directly by tests to synthesize requests +- :46 `bearerToken: String?` — reads Authorization, case-insensitive "bearer " prefix strip + trim; CONVENIENCE ONLY, no validation +- :54 `origin: String?` — raw Origin header passthrough; CONVENIENCE ONLY, no allow-list check +- :69 `wantsEventStream: Bool` — Accept:text/event-stream OR exact query param `stream=1` (split on '&', exact match — NOT substring; see GOTCHAS) +- :95 `static func read(fd:maxHeaderBytes:maxBodyBytes:) -> HTTPRequest?` — buffers until "\r\n\r\n"; nil if header block exceeds maxHeaderBytes before terminator found, or on recv failure +- :114 `private static func parse(buffer:headerEnd:fd:maxBodyBytes:) -> HTTPRequest?` — request-line split, header lowercase-key parse, Content-Length-bounded body read (truncates silently at maxBodyBytes, does not error) + +### Response — HTTPWire.swift +- :164 `struct HTTPResponse: Sendable` — status/headers/body, caller-composed (not a closed consumer-specific case set — see GOTCHAS) +- :169 `init(status:headers:body:)` — headers/body default to empty +- :176 `static func json(status:body:) -> HTTPResponse` — Content-Type: application/json +- :182 `static func asset(contentType:body:) -> HTTPResponse` — 200 + Cache-Control: no-store (redeployed UI never served stale) +- :191 `static var notFound: HTTPResponse` — 404 + `{"error":"not_found"}` +- :201 `func send(fd:)` — computes Content-Length from actual body (overrides any caller value); header order: Content-Type, Content-Length, then rest sorted alphabetically, then Connection: close ALWAYS last (caller-supplied Connection does not suppress it) +- :221 `static func reason(_ status:) -> String` — reason phrase for 200/400/401/403/404/413/500; default "OK" + +### SSE — HTTPWire.swift +- :242 `struct SSEStream: Sendable` — long-lived push connection wrapper; caller owns source/cadence/lifetime/fd close +- :245 `init(fd:)` +- :252 `func writeHead() -> Bool` — call exactly once before any frame; false = peer already gone +- :259 `func send(_ payload:) -> Bool` — writes `"data: \(payload)\n\n"`; false = write failed, caller should stop +- :264 `static let responseHead: Data` — fixed head: 200 OK, text/event-stream, no-cache, keep-alive + +## INVARIANTS / GOTCHAS + +- AUTH-FREE INVARIANT (ADR-LOOPBACKHTTP-001 condition 3): no auth/policy logic anywhere in this package. `bearerToken`/`origin` are read-only conveniences; accept/reject is composed ABOVE this library by each consumer (none in Community Edition, bearer+Origin in moot-mgr, OAuth 2.1 in EE-only remote layer). Do not add auth enforcement here — it would break the "one binary, all editions" property. +- LOOPBACK-ONLY BY CONSTRUCTION: `listenLoopbackTCP` binds the literal `127.0.0.1` (0x7F000001 big-endian), never `INADDR_ANY`/0.0.0.0. Do not parameterize the bind address to accept a caller-supplied host. +- UDS PERMISSION WINDOW: `listenUnix` must chmod 0600 BEFORE `listen()`, not after — the current ordering closes the world-readable window; do not reorder. +- SSE CSRF NOTE: `wantsEventStream` via `?stream=1` is a plain GET, so browser CORS preflight does NOT run. A cross-origin page can open an SSE connection if the loopback port is reachable. Any consumer serving sensitive data over SSE MUST check `request.origin` itself before upgrading — this library does not and will not enforce it (see AUTH-FREE INVARIANT). +- `wantsEventStream` query check is EXACT-MATCH on a `&`-split parameter, not `query.contains("stream=1")`. The substring form was a real bug (matched "stream=10", "mystream=1", "x=stream=1"); regression-locked by Tests/LoopbackHTTPTests/WantsEventStreamTests.swift. Do not revert to substring matching. +- Header names are lowercased on parse (`parse(buffer:...)`) so all internal/consumer lookups (`headers["content-length"]`, `headers["authorization"]`) use lowercase keys only — HTTP header names are case-insensitive on the wire but this dictionary is not. +- Body truncation, not rejection: `HTTPRequest.read`/`parse` silently truncate a body at `maxBodyBytes` when Content-Length exceeds it; they do not return nil or an error. A listener that must reject oversize bodies (e.g. the MCP `tools/call` listener) needs to check Content-Length itself before/after calling, or size its cap so truncation cannot corrupt a valid request. +- `maxHeaderBytes`/`maxBodyBytes` are per-call parameters, not fixed constants (default 64 KiB each) — deliberately, so a small dashboard control listener and a larger MCP listener each pick their own cap (ADR-LOOPBACKHTTP-001 condition 2). +- `HTTPResponse.send` always appends `Connection: close` last, even if the caller supplied their own `Connection` header (that value is emitted earlier, in the sorted section, and does not suppress the trailing one). Every response this library writes closes the connection; there is no keep-alive path for ordinary responses (SSE is the sole long-lived exception). +- `HTTPResponse`/`SSEStream` are open, caller-composed value types, not a closed enum of consumer-specific response shapes — this genericity (added in P1a) is what let a second consumer (resident MCP daemon) reuse the library without modifying it. Do not narrow them back to a fixed case set for one consumer's convenience. +- All `POSIXSocket` calls are synchronous/blocking. Callers must run them off a dedicated Thread or `Task.detached`, never inline on a thread that must stay responsive. +- Network.framework (`NWListener`) was tried and rejected: it returns POSIXErrorCode 22/EINVAL binding a listening socket in this project's command-line (non-app-bundle) build environment, on every configuration tested. Do not reintroduce it as "simpler" without re-verifying that constraint is gone. +- Swift-only, no Rust port: this library is OS-transport glue, not deterministic substrate compute, so it is exempt from the Swift/Rust byte-parity discipline that governs libraries like LatticeLib. ARIA_MCP-rust hand-rolls its own std::net transport under a no-FFI rule; the two sides agree only at the JSON-RPC message level. +- Zero external SwiftPM dependencies (Package.swift target has none) — adding one requires re-justifying against the kit dependency rules that motivated this extraction in the first place. diff --git a/packages/libs/LoopbackHTTP/docs/DETAILS.md b/packages/libs/LoopbackHTTP/docs/DETAILS.md new file mode 100644 index 0000000..36b409d --- /dev/null +++ b/packages/libs/LoopbackHTTP/docs/DETAILS.md @@ -0,0 +1,250 @@ +--- +doc: DETAILS +package: LoopbackHTTP +repo: moot-system +authored_commit: 909513d0a8ecb1e9e903af9f4d25b3e4f2528242 +authored_date: 2026-07-04 +sources: + - path: Sources/LoopbackHTTP/HTTPWire.swift + blob: c07525c2583c06aa8406287eef5cb17279b9b86a + - path: Sources/LoopbackHTTP/POSIXSocket.swift + blob: 6dfba4dee7ad3121d47c8bfc937854c6f13ad1b1 +--- + +# LoopbackHTTP Details + +This document walks through every source file in the package. Read +`OVERVIEW.md` first for the big picture. Files appear here in pipeline order: +the socket transport first, then the HTTP protocol layer built on top of it. + +## POSIXSocket.swift + +This file provides `POSIXSocket`, a set of small blocking-socket helpers, and +`SocketError`, the error type they throw. Everything in this file is +synchronous: a call blocks the calling thread until the operating system +answers. Callers are expected to run these calls off a dedicated thread or a +detached task, never on a thread that must stay responsive. + +Two design choices explain the whole file. First, the code targets plain +POSIX sockets rather than Apple's `Network.framework`, because that +higher-level framework refused to bind a listening server socket in this +project's command-line build environment; it returned a low-level POSIX error +code on every configuration tried, a restriction tied to running outside an +app bundle. Plain sockets bind correctly in that same environment, so the +library uses them directly, and the same security guarantees apply either +way. Second, the file imports `Glibc` on Linux and `Darwin` on Apple +platforms through a compile-time check, so the same source file compiles on +both without any conditional logic beyond the import line. + +`listenLoopbackTCP(port:)` opens a TCP listening socket and binds it to +`127.0.0.1`, never to `0.0.0.0` (which would accept connections from any +network interface, not just the local machine). The bind address is a +literal numeric value, not a name the operating system could resolve +differently, so this guarantee cannot be weakened by a misconfigured host +file or DNS entry. Passing port `0` lets the operating system pick an unused +port; the function reads the actual bound port back with `getsockname` and +returns it, which matters because the caller often needs to tell another +process which port to connect to. Why this function matters: it is the one +place in the library that decides which network interface can reach the +server at all, and it decides in favor of the strictest possible answer, +every time, regardless of what the caller asks for. + +`listenUnix(path:)` opens a Unix-domain socket at a filesystem path instead of +a network port. A Unix-domain socket is a connection endpoint that lives in +the filesystem and can only be reached by other processes on the same +machine; a web browser has no way to speak to one at all. This makes it a +natural channel for the most sensitive operations, since browser-based +attacks are structurally unable to reach it. The function removes any stale +socket file left over from a previous crashed run before binding, so a dead +file cannot block a fresh start. After binding, it changes the file's +permissions to owner-read-write-only (mode 0600) before calling `listen`, so +there is no window during which the socket file exists but is readable by +other users on the machine. Why this matters: the chmod step closing that +window is what turns "only this user can connect" from a hope into a +guarantee enforced by the filesystem itself. + +`acceptOne(_:)` accepts one pending connection on a listening socket and +returns the new connection's descriptor, or `nil` if the accept call failed. +It is a thin wrapper; the decision of whether and how to retry after a +failure belongs to the caller, not to this function. + +`recv(_:max:)` reads up to `max` bytes from a connected socket and returns +them as `Data`, or `nil` on a read error. An empty (but non-nil) result means +the peer closed the connection cleanly; a caller reading in a loop uses that +to know when to stop. + +`sendAll(_:_:)` writes every byte of `data` to a socket, looping until the +whole buffer is sent or a write fails. A single call to the underlying system +write is not guaranteed to send the whole buffer at once — the kernel is free +to accept less — so this function is the one place other files in the +package go to guarantee that a full response, or a full SSE frame, actually +reaches the wire. It returns `false` on any failed write, which a caller uses +as its signal that the peer is gone and the connection should be torn down. + +`SocketError` is the file's structured error type. The package's error +handling rule prefers enumerated error cases over returning an optional +value with no explanation, so that a caller (or a log line) can say exactly +which system call failed and with which error number, rather than only that +something failed. `.syscall(String, Int32)` names the failing call and +carries its `errno` value; `.pathTooLong` covers the one Unix-domain-socket +case where the caller's path does not fit the fixed-size buffer the +operating system's socket address structure provides. + +## HTTPWire.swift + +This file provides three public types: `HTTPRequest`, a parsed incoming +request; `HTTPResponse`, an outgoing response the caller builds; and +`SSEStream`, the narrower Server-Sent-Events framing used for a long-lived +push connection. All three depend on `POSIXSocket` for the actual byte +transfer; nothing in this file opens a socket itself. + +The file's scope is deliberately narrow. It is not a general-purpose HTTP +server: it parses a request line, a set of headers, and an optional body +bounded by a `Content-Length` header, and it writes exactly one response per +connection before that connection closes. It does not support chunked +transfer encoding, HTTP/1.1 keep-alive across many requests on the same +response path, or HTTP/2. Every response this library writes closes the +connection afterward (except an SSE stream, which the caller keeps open on +purpose). + +### HTTPRequest + +`HTTPRequest` is a plain value holding the request `method` (for example, +`"GET"`), the `path` with any query string already removed, the raw `query` +string that followed the `?` (or an empty string if there was none), a +lowercase-keyed table of `headers`, and the request `body` as `Data`. Storing +the path and query separately, rather than leaving the caller to split a +combined URL string, means every consumer performs that split identically +instead of each one writing its own (and possibly buggy) parsing. + +`bearerToken` reads the `Authorization` header, checks that it begins with +the case-insensitive prefix `"bearer "`, and returns the remaining token text +with surrounding whitespace trimmed, or `nil` if the header is absent or does +not match. Why this matters: this is a convenience reader only. The library +extracts the token text so a caller does not have to; it never decides +whether that token is valid. The decision to accept or reject a request +based on this value belongs entirely to the code above this library, in +keeping with the package's edition-neutral, authentication-free design. + +`origin` reads the `Origin` header directly, with no further processing. A +browser sets this header to say which web page's script initiated the +request. Like `bearerToken`, this is exposed purely so a caller can inspect +it; the library does not compare it against an allow-list itself. + +`wantsEventStream` decides whether the caller likely wants an SSE response +instead of an ordinary one. It returns `true` if the request's `Accept` +header mentions `text/event-stream`, or if the query string carries an exact +`stream=1` parameter. The exact-parameter check matters because an earlier +version of this logic used a substring check, `query.contains("stream=1")`, +which incorrectly matched `stream=10`, `mystream=1`, and `x=stream=1` — none +of which mean what `stream=1` means. The current implementation splits the +query on `&` and compares each resulting parameter for exact equality to +`"stream=1"`, which cannot be fooled by a longer number or an unrelated +parameter name that happens to contain the same substring. A dedicated +regression test file, `Tests/LoopbackHTTPTests/WantsEventStreamTests.swift`, +exists specifically to keep this fix from silently regressing. This same +property carries a documented security note: because an SSE request made +this way is an ordinary browser `GET` request, the browser's cross-origin +preflight check never runs for it. A page on an unrelated origin can +therefore open an SSE connection to a loopback server if that server is +reachable at all. Any consumer that serves sensitive data over SSE must check +the `origin` value itself and refuse an unexpected origin before upgrading to +a stream; this library does not perform that check, again by the +authentication-free design. + +`HTTPRequest.read(fd:maxHeaderBytes:maxBodyBytes:)` is the entry point that +turns raw socket bytes into a parsed request. It reads repeatedly from the +socket, accumulating bytes into a buffer, until it finds the header +terminator (`"\r\n\r\n"`, the blank line that ends an HTTP header block). If +the accumulated header block ever exceeds `maxHeaderBytes` before that +terminator appears, the function gives up and returns `nil` rather than +continuing to buffer an unbounded amount of untrusted network data — a cap +that protects the server's memory against a hostile or broken client sending +an endless header. The caps are parameters rather than fixed constants +because different listeners have different legitimate needs: a small +dashboard control listener wants a small cap, while a listener serving +`tools/call` requests for the Model Context Protocol wants a larger one, so +each caller can choose the cap that matches its own traffic without every +listener sharing one compromise value. + +The private `parse(buffer:headerEnd:fd:maxBodyBytes:)` helper does the actual +parsing once the header terminator has been found. It reads the request +line, splits it into method and target, and splits the target on the first +`?` into path and query. It then walks the remaining header lines, splitting +each on its first colon, trimming whitespace, and lowercasing the header +name so that later lookups (`headers["content-length"]`, for example) do not +have to guess whether a client sent `Content-Length` or `content-length` or +some other capitalization; HTTP header names are case-insensitive, and this +normalization is what makes the rest of the library able to treat them as +plain lowercase string keys. If a `Content-Length` header is present, the +function reads exactly that many more bytes from the socket, up to +`maxBodyBytes`. If the declared length exceeds the cap, the body is silently +truncated to the cap rather than the request being rejected; a caller for +whom truncation would be dangerous is expected to set its cap high enough +that truncation cannot happen, or to check the `Content-Length` header itself +before calling `read` at all. + +### HTTPResponse + +`HTTPResponse` is a plain value: a `status` code, a table of `headers`, and a +`body`. Unlike `HTTPRequest`, which the library builds for the caller, this +type is built by the caller and only serialized by the library. This +inversion is deliberate: the earlier version of this code hard-coded a +closed set of response shapes for one specific consumer, and generalizing it +to a plain value that any caller can construct is what let a second consumer +(the resident MCP transport) reuse the same library without waiting for new +cases to be added here every time its needs changed. + +Three static convenience constructors cover the shapes most callers need +without limiting them to only those shapes. `json(status:body:)` builds a +response with an `application/json` content type. `asset(contentType:body:)` +builds a `200 OK` response carrying `Cache-Control: no-store`, a header that +tells the browser never to reuse a cached copy; this matters for a locally +served dashboard, where a stale cached page after a new build would show +outdated controls or, worse, controls that no longer match the running +server's API. `notFound` is a ready-made `404` response with a small +consistent JSON error body, so every listener reports a missing route the +same way. + +`send(fd:)` serializes the response and writes it to the socket through +`POSIXSocket.sendAll`. It computes `Content-Length` itself from the actual +body byte count and overrides anything the caller supplied for that header, +because a caller-supplied length that does not match the real body would +produce a response the receiving HTTP client cannot parse correctly. Headers +are written in a fixed, deterministic order: `Content-Type` and +`Content-Length` first if present, then every other header sorted +alphabetically, and finally a `Connection: close` line that is always +appended last regardless of anything the caller set for that header. Fixing +the order this way is not required by the HTTP specification, which allows +headers in any order, but it makes every response byte-for-byte reproducible +for testing and logging, which matters more here than flexibility that no +caller needs. `reason(_:)` maps a status code to its human-readable reason +phrase (for example, `404` to `"Not Found"`) for the small fixed set of +statuses this library emits, falling back to `"OK"` for anything else since +the reason phrase is advisory text that HTTP clients are not required to +interpret. + +### SSEStream + +`SSEStream` frames the Server-Sent-Events wire format for one open +connection, identified by its socket descriptor `fd`. Unlike `HTTPRequest` +and `HTTPResponse`, which each cover one request-response exchange, an +`SSEStream` value is meant to be held by the caller for the life of a +long-running push connection; the library plays no role in deciding how long +that is, what triggers each message, or when to close it. + +`writeHead()` writes the fixed SSE response head — the status line and the +three headers (`Content-Type: text/event-stream`, `Cache-Control: no-cache`, +`Connection: keep-alive`) a client needs to recognize an event stream — and +must be called exactly once, before any data frame. It returns `false` if the +write failed, which the documentation notes means the peer is already gone; +a caller seeing `false` should close the connection rather than attempt to +send frames into a stream nobody is reading. `send(_:)` writes one SSE +`data:` frame carrying the given payload string, in the wire format +`"data: \n\n"` that the SSE specification requires (the payload +text followed by exactly one blank line). It also returns `false` on a failed +write, for the same reason. Both callers of this type in MOOTx01 — moot-mgr's +dashboard, which drives the stream from polling its own store, and the +resident MCP transport, which drives it from JSON-RPC notifications — decide +their own cadence and lifetime entirely outside this file; `SSEStream` only +guarantees that whatever they send is framed correctly on the wire. diff --git a/packages/libs/LoopbackHTTP/docs/OVERVIEW.md b/packages/libs/LoopbackHTTP/docs/OVERVIEW.md new file mode 100644 index 0000000..16ed926 --- /dev/null +++ b/packages/libs/LoopbackHTTP/docs/OVERVIEW.md @@ -0,0 +1,110 @@ +--- +doc: OVERVIEW +package: LoopbackHTTP +repo: moot-system +authored_commit: 909513d0a8ecb1e9e903af9f4d25b3e4f2528242 +authored_date: 2026-07-04 +sources: + - path: Sources/LoopbackHTTP/HTTPWire.swift + blob: c07525c2583c06aa8406287eef5cb17279b9b86a + - path: Sources/LoopbackHTTP/POSIXSocket.swift + blob: 6dfba4dee7ad3121d47c8bfc937854c6f13ad1b1 +--- + +# LoopbackHTTP Overview + +## What This Library Does + +LoopbackHTTP is a small HTTP/1.1 server primitive. It listens on a socket, reads +one request, and lets the caller write one response. It also knows how to +frame a Server-Sent-Events (SSE) stream, a way for a server to push a +continuing series of small text messages to a browser over one open +connection. The library does not decide what any of that traffic means; it +only moves the bytes correctly. + +The library binds strictly to the loopback address, `127.0.0.1`. A loopback +address is one that only the same machine can reach; no other computer on the +network can connect to it. Every listener this library creates is reachable +only from processes running on the same device. + +## The Problem It Solves + +Two programs inside MOOTx01 need a local HTTP server: the moot-mgr monitor +daemon, which serves a small dashboard, and the resident mootx01 MCP daemon, +which serves the Model Context Protocol used by AI tool integrations. Before +this library existed, each program hand-rolled its own socket and parsing +code. Two hand-rolled copies drift apart over time: one gets a security fix, +the other does not; one handles a header correctly, the other does not. +LoopbackHTTP was extracted from moot-mgr, under decision record +ADR-LOOPBACKHTTP-001, so both programs share one audited implementation +instead of maintaining two. + +The library adds no external package dependency. It wraps only the system C +socket API supplied by the operating system (`libc` on Linux, `Darwin` on +Apple platforms). A larger, general-purpose HTTP package such as SwiftNIO was +available but was rejected: it would pull in a dependency graph broader than +this narrow need justifies, and the moot-system kit rules ask each library to +justify every dependency it takes. Even Apple's own `Network.framework` was +tried and rejected for a concrete reason: its listener type refused to bind a +listening server socket in this project's command-line build environment, +failing with a low-level error code on every configuration tested. Plain +POSIX sockets bind correctly in that same environment, so the library builds +on them directly. + +The library is deliberately authentication-free. It exposes two convenience +readers on a parsed request, `bearerToken` and `origin`, so a caller can read +those header values easily. But the library itself never decides whether a +request is authorized. That decision is composed above the transport, one +layer up, by each consumer: the Community Edition of moot-mgr enforces no +policy at all, the full moot-mgr enforces a bearer token plus an Origin +header check, and an Enterprise Edition-only remote layer enforces OAuth +2.1. Keeping the transport policy-free lets the identical compiled library +ship inside every edition unchanged; only the layer above it differs. + +The library is Swift-only. MOOTx01 keeps a byte-for-byte parity discipline +between a Swift implementation and a Rust implementation for deterministic +computation, such as classification or fingerprinting, where both legs must +produce the exact same answer from the exact same input. Binding a socket and +parsing HTTP headers is operating-system transport glue, not that kind of +deterministic computation, so it falls outside the parity rule. The Rust side +of MOOTx01's MCP transport (ARIA_MCP-rust) hand-rolls its own separate +networking code under a rule that forbids it from calling into Swift or C +libraries across a foreign-function boundary; the two sides only have to +agree at the JSON-RPC message level, not at the socket level. + +## How It Works + +The library is organized in two layers, one file each. + +`POSIXSocket.swift` is the transport layer. It opens a listening socket bound +to `127.0.0.1` on a chosen TCP port, or opens a Unix-domain socket at a +filesystem path locked to owner-only access. It accepts one incoming +connection at a time and reads or writes raw bytes on the resulting +connection. + +`HTTPWire.swift` is the protocol layer, built on top. `HTTPRequest.read(fd:)` +reads raw bytes from a connected socket and parses them into a structured +request: a method, a path, a query string, a table of headers, and a body. +`HTTPResponse` is a value the caller builds — a status code, headers, and a +body — which the library then serializes into correct HTTP/1.1 wire bytes and +writes back to the socket. `SSEStream` frames the narrower SSE wire format: +one head line written once, then any number of `data:` frames written over +time on a connection the caller keeps open. + +A typical request passes through the library in one direction only: bytes +arrive from `POSIXSocket`, `HTTPRequest.read` turns them into a request value, +the caller inspects that value and decides what to do, and the caller's +resulting `HTTPResponse` or `SSEStream` frames go back out through +`POSIXSocket`. Nothing loops back, and no component here holds state between +one request and the next connection. That simple, one-way shape is why this +library needs no larger topology diagram: two files, a handful of value +types, and a straight line from incoming bytes to outgoing bytes. + +## What Ships in the Package + +The package ships two Swift source files and no bundled resources, artifacts, +or Rust port. `Package.swift` declares one library product, `LoopbackHTTP`, +with no external dependencies, targeting macOS 26 and iOS 26 while also +building on Linux Swift through a compile-time import guard in +`POSIXSocket.swift` that selects `Glibc` there and `Darwin` on Apple +platforms. diff --git a/packages/libs/ObserverSink/docs/AGENT_MAP.md b/packages/libs/ObserverSink/docs/AGENT_MAP.md new file mode 100644 index 0000000..36daea4 --- /dev/null +++ b/packages/libs/ObserverSink/docs/AGENT_MAP.md @@ -0,0 +1,104 @@ +--- +doc: AGENT_MAP +package: ObserverSink +repo: moot-system +authored_commit: 909513d0a8ecb1e9e903af9f4d25b3e4f2528242 +authored_date: 2026-07-04 +sources: + - path: Sources/ObserverSink/PersistenceStatsSink.swift + blob: e76c55599795f4bc85b860a4901dc9ab04dd201f + - path: Sources/ObserverSink/StatsStore.swift + blob: 483c736feffd82821a8d77030afde0e182d320fc +--- + +# AGENT_MAP — ObserverSink + +PURPOSE: reusable PersistenceKit-backed telemetry sink for the MOOTx01 manager pipeline (Manager 1.0, Phase 0.5). IntellectusLib.report(_:) → PersistenceStatsSink.receive(_:) → flag-gated SQLite write into StatsStore (metric_samples / event_samples). Also owns topology_snapshots (governor write, dashboard read) and store-level retention. + +DEPS: imports IntellectusLib (StatsSink protocol, StatSample/EventKind datum), PersistenceKit + PersistenceKitSQLite (Storage/SchemaDeclaration/RowStore, SQLiteStorage backend), Foundation, OSLog. Dependency hierarchy (no inversion, per MANAGER_1.0_PLAN.md §4): IntellectusLib (floor) → PersistenceKit (kit) → ObserverSink (this lib). Imported by: none within the moot-system repo at this commit — Package.swift comments name the intended consumers (Intellectus install site in AriaResident/aria-mcp, moot-mgr manager) as living outside this repo. Rust port in rust/ (crate `observer-sink`) mirrors schema + flag semantics exactly; conformance tests in Tests/ObserverSinkTests/ObserverSinkConformanceTests.swift and rust/tests/conformance.rs gate parity (16 scenarios). + +ENTRY POINTS (most callers need only these): +- PersistenceStatsSink.swift:102 `PersistenceStatsSink.init(store:dropboxID:)` — construct the sink; store must already be `.open()`ed +- PersistenceStatsSink.swift:123 `receive(_ sample: StatSample)` — StatsSink conformance; called by Intellectus.report(_:), never call directly +- StatsStore.swift:362 `StatsStore.init(url:) throws` — construct the store +- StatsStore.swift:384 `open() async throws` — apply schema/migrations + seed control rows (idempotent) + +## Symbol Table + +### Sink — PersistenceStatsSink.swift +- :77 `struct PersistenceStatsSink: StatsSink` — Sendable; all mutable state lives in StatsStore, not here +- :82 `store: StatsStore` (private) +- :89 `dropboxID: String` (private) — tags every written row; identifies source consumer/process +- :91 `logger` (private) — `Logger(subsystem: "com.mootx01.kit", category: "ObserverSink")` +- :102 `init(store:dropboxID:)` — see ENTRY POINTS +- :123 `receive(_:)` — synchronous, non-blocking; dispatches unstructured Task; inside: isMonitoringEnabled() gate → insertMetric/insertEvent by StatSample case → catch-and-log, never throws/propagates + +### Schema constants — StatsStore.swift `enum StatsStoreSchema` +- :49 `enum StatsStoreSchema` — table/column name constants; typo → compile error, not silent empty query +- :54 `metricSamplesTable = "metric_samples"` / :57 `eventSamplesTable = "event_samples"` / :60 `controlTable = "control"` / :117 `topologySnapshotsTable = "topology_snapshots"` +- :65 `tsColumn = "ts"` (TEXT ISO-8601 on every table) / :68 `dropboxIDColumn = "dropbox_id"` +- :73 `nameColumn`, :76 `valueColumn` (REAL), :79 `tagsColumn` (TEXT JSON) — metric_samples +- :84 `kindColumn` (EventKind raw string), :87 `nounTypeColumn` (INTEGER ordinal), :91 `rowIDColumn = "estate_row_id"` (NOT the synthetic PK), :94 `estateColumn` — event_samples +- :99 `keyColumn` (PK), :102 `controlValueColumn` — control +- :107 `monitoringKey = "monitoring"` (value "1"/"0") / :112 `retentionCutoffKey = "retention_cutoff"` (ISO-8601, default epoch-zero) +- :120 `generatedAtColumn`, :123 `payloadColumn`, :132 `topologyFingerprintColumn` (nullable, v3) — topology_snapshots + +### Store — StatsStore.swift `final class StatsStore: Sendable` +- :160 `final class StatsStore: Sendable` — wraps SQLiteStorage +- :164 `storage: SQLiteStorage` (private) / :165 `logger` (private) +- :175 `static let schemaVersion = 3` — v1: metric/event/control; v2: +topology_snapshots; v3: +topology_fingerprint (nullable) +- :193 `static let schema: SchemaDeclaration` — 4 tables, 2 indices (ts on metric_samples/event_samples), 2 migrations (v1→v2 createTable, v2→v3 addColumn) +- :317 migrations block — v1→v2 :321, v2→v3 :340; SQLite backend creates fresh DBs at latest version directly (migrations replay only matters for InMemory/upgrade path) +- :362 `init(url:) throws` — fresh random estateID per StatsStore instance; no I/O +- :384 `open() async throws` — applies schema, then :396/:398 `seedControlIfAbsent` for monitoring="0" and retention_cutoff=epoch-zero; SEED-IF-ABSENT not upsert +- :412 `seedControlIfAbsent(key:value:)` (private) — existence check then insert; preserves operator-set values across reopen +- :433 `close() async` — idempotent +- :448 `isMonitoringEnabled() async throws -> Bool` — control["monitoring"]=="1"; missing row → false (safe default) +- :470 `setMonitoringEnabled(_:) async throws` — upsert on keyColumn; manager-only caller by convention (sink never writes this) +- :496 `insertMetric(name:value:tags:ts:dropboxID:) async throws` — assigns row_id UUID; tags→JSON via encodeTagsJSON; ts(Double epoch)→.timestamp(Date) at this boundary +- :532 `insertEvent(kind:nounType:rowID:estate:ts:dropboxID:) async throws` — rowID stored in estate_row_id column, NOT row_id (that's the synthetic PK) +- :576 `writeTopologySnapshot(estate:generatedAt:payload:fingerprint:) async throws` — upsert keyed on estate (latest-wins, no history); throws StorageError.invalidQuery if payload isn't valid UTF-8; fingerprint nil → .null column +- :615 `latestTopologySnapshot(estate:) async throws -> Data?` — estate=nil → newest generated_at across ALL estates (dashboard "all" view); uses :679 generatedAtInstant to tolerate both .timestamp and .text read-back reps (SQLite vs InMemory) — DO NOT match only .timestamp, that was a real bug (all rows tie at .distantPast under SQLite) +- :655 `loadTopologyFingerprint(estate:) async throws -> String?` — nil if no snapshot / pre-v3 row / written without fingerprint +- :679 `generatedAtInstant(_:) -> Date` (private) — see above +- :696 `queryMetrics(dropboxID:) async throws -> [MetricRow]` — full-table (optionally dropbox-filtered) read, ts ascending +- :725 `queryMetricsByNames(_:dropboxID:) async throws -> [MetricRow]` — `WHERE name IN (...)`; empty names → [] immediately; USE THIS not queryMetrics+filter on hot read paths +- :769 `countMetrics() async throws -> Int` — COUNT(*), no row decode +- :784 `queryEvents(dropboxID:) async throws -> [EventRow]` — full-table (optionally dropbox-filtered) read, ts ascending +- :818 `deleteMetricsBefore(cutoff:now:) async throws -> Int` — ts < cutoff; updates retention_cutoff control row to `cutoff` (not `now`) +- :842 `deleteEventsBefore(cutoff:now:) async throws -> Int` — same semantics, event_samples +- :873 `storageStats(now:) async throws -> StorageStats?` — store's OWN SQLite backend health (WAL frames, file size, page/freelist counts), not an observed estate's storage; via StorageIntrospection, SQLite backend always returns non-nil +- :883 `recordRetentionCutoff(_:now:) async throws` (private) — upserts retention_cutoff as ISO-8601 +- :907 `static let iso8601Formatter: DateFormatter` — "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'", UTC, en_US_POSIX; matches SQLiteStorage's timestamp codec +- :924 `encodeTagsJSON(_:) -> String` (private) — `.sortedKeys` for determinism; empty/unencodable → "{}" +- :939 `static func decodeTagsJSON(_:) -> [String: String]` — parse failure → [:] (forward-compatible) + +### Row types +- :951 `struct MetricRow: Sendable` — rowID/name/value/tags/ts/dropboxID; :966 `init?(storageRow:)` returns nil on any column type mismatch (compactMap-friendly) +- :986 `struct EventRow: Sendable` — rowID/kind/nounType/rowIDStr/estate/ts/dropboxID; :1003 `init?(storageRow:)` same nil-on-mismatch contract + +### Rust port (rust/, crate observer_sink) — mirrors above 1:1 except where Rust semantics differ +- rust/src/sink.rs `PersistenceStatsSink::receive` — INLINE synchronous I/O (no Task/thread spawn); Rust StatsSink trait is sync and SqliteStorage is Mutex-backed +- rust/src/store.rs `StatsStore::SCHEMA_VERSION = 3`, `make_schema()` — line-by-line mirror of Swift `StatsStore.schema` +- rust/src/store.rs `epoch_to_iso8601` / `iso8601_to_epoch` (crate-internal) — explicit ISO-8601 codec (no DateFormatter equivalent); clamps NaN/±inf/out-of-range to year 0001–9999 window, never panics (unit-tested at bottom of store.rs) +- rust/src/store.rs `write_topology_snapshot_bytes` — Rust-only convenience: lossy UTF-8 conversion of raw bytes; Swift has no equivalent because Foundation's `String(data:encoding:)` guard is a runtime check callers already handle inline + +## INVARIANTS / GOTCHAS + +- Seed-if-absent, NEVER upsert, for control-row defaults in `open()`. Upserting monitoring on every open would reset an operator's "on" back to "off" on every process restart. This exact bug was fixed in Swift commit 852821cc (see rust/src/store.rs comment); do not reintroduce it in either port. +- `isMonitoringEnabled()` fails safe: missing row → `false` (off), never assume on. +- `PersistenceStatsSink` only READS the monitoring flag; only the manager process calls `setMonitoringEnabled`. Do not add a write path to the sink. +- `receive(_:)` MUST stay synchronous and non-blocking (StatsSink protocol contract). All I/O happens off the calling thread/task (Swift: unstructured Task; Rust: inline sync, which is fine because Rust's I/O is not on an async hot path here). +- No in-process buffering at v1.0 by design — one Task per sample in Swift. Do not add batching without also updating OVERVIEW/DETAILS and the design comment in PersistenceStatsSink.swift. +- `event_samples.estate_row_id` is NOT the table's primary key — `row_id` (synthetic UUID) is the PK; `estate_row_id` carries the estate's own UUID for that row. Never conflate the two when reading/writing. +- Retention (`deleteMetricsBefore`/`deleteEventsBefore`) takes `cutoff` AND `now` as caller-supplied parameters. NO `Date()` / `SystemTime::now()` call inside either engine — determinism contract, both ports. +- `recordRetentionCutoff` stores `cutoff` (the boundary applied), not `now` (when the pass ran) — intentional, dashboard-facing choice. +- `latestTopologySnapshot`/`latest_topology_snapshot` newest-across-estates comparison MUST tolerate both `.timestamp` and `.text` read-back representations for `generated_at` (SQLite returns text, InMemory returns timestamp). Matching only one representation reintroduces a real, previously-shipped bug where SQLite rows all tie at `.distantPast` / `i64::MIN` and an arbitrary row wins. +- `topology_snapshots` is latest-wins per estate (PRIMARY KEY = estate) — no history retained. This is intentional; do not add versioning without a schema bump and a doc update. +- `topology_fingerprint` column is nullable (v3, additive). Absent/null means "no fingerprint computed yet," not an error — callers (autonomic governor) treat null as "recompute once." +- Timestamps are TEXT (ISO-8601 UTC, millisecond precision) on every table. No REAL/numeric timestamp column exists anywhere in this schema. Swift boundary: `Date(timeIntervalSince1970: ts)` at insert, `iso8601Formatter` for retention-cutoff string formatting. Rust boundary: `epoch_to_iso8601`/`iso8601_to_epoch`. +- Rust's `epoch_to_iso8601` clamps to the year 0001–9999 window and never panics on NaN/±infinity/out-of-range f64 — required because a malformed 5-digit or negative year would break every lexicographic TEXT comparison retention and topology-snapshot ordering rely on. Six unit tests in rust/src/store.rs lock this behavior; do not remove them when refactoring. +- Tag maps: encoded with `.sortedKeys` (Swift) / serialized from a `BTreeMap` (Rust) for deterministic JSON — needed for reproducible test fixtures, not for correctness of the round trip itself. Decode failure → empty map in both ports (forward-compatible, never throws). +- `queryMetricsByNames` is the required hot-path read API — issues `WHERE name IN (...)`; do not reintroduce `queryMetrics` + Swift-side filtering on a read path that matters for latency. +- No pinned data artifacts in this package (unlike LatticeLib-style libs) — every table starts empty; there is nothing to version-check against a build-time asset. +- Both-ports parity is enforced ONLY by the two conformance test suites (Tests/ObserverSinkTests + rust/tests/conformance.rs), not by any shared fixture file. Changing schema, flag semantics, or timestamp encoding in one leg requires updating the other leg AND both suites by hand. diff --git a/packages/libs/ObserverSink/docs/DETAILS.md b/packages/libs/ObserverSink/docs/DETAILS.md new file mode 100644 index 0000000..47c7409 --- /dev/null +++ b/packages/libs/ObserverSink/docs/DETAILS.md @@ -0,0 +1,341 @@ +--- +doc: DETAILS +package: ObserverSink +repo: moot-system +authored_commit: 909513d0a8ecb1e9e903af9f4d25b3e4f2528242 +authored_date: 2026-07-04 +sources: + - path: Sources/ObserverSink/PersistenceStatsSink.swift + blob: e76c55599795f4bc85b860a4901dc9ab04dd201f + - path: Sources/ObserverSink/StatsStore.swift + blob: 483c736feffd82821a8d77030afde0e182d320fc +--- + +# ObserverSink Details + +This document walks through both source files in the package. Read +`OVERVIEW.md` first for the big picture. `PersistenceStatsSink.swift` +comes first because it is the entry point every caller uses; +`StatsStore.swift` follows because it is what the sink calls into. + +## PersistenceStatsSink.swift + +This file provides `PersistenceStatsSink`, the concrete implementation of +IntellectusLib's `StatsSink` protocol. It is the only type in the package +that a typical caller touches directly. + +A `StatsSink` is any type that can receive one `StatSample` at a time. +IntellectusLib defines the protocol and a global installation point; +`PersistenceStatsSink` is the implementation that actually saves samples +by writing them into a `StatsStore`. A host program constructs one sink, +installs it with `Intellectus.install(sink:)`, and turns monitoring on +with `Intellectus.setEnabled(true)`. From that point on, every call to +`Intellectus.report(_:)` anywhere in the process reaches this sink's +`receive(_:)` method. + +The struct holds three pieces of state: the `store` it writes to, a +`dropboxID` string identifying this consumer, and a `Logger` for +diagnostics. All three are `let` constants, and the struct conforms to +`Sendable` — required because `StatsSink` values are held in a +process-wide global and called from any thread. No mutable state lives in +the sink itself; anything that changes over time (the monitoring flag, the +stored rows) lives inside `StatsStore`, which manages its own +concurrency. + +`init(store:dropboxID:)` is a plain constructor. It does not open the +store — the caller must call `store.open()` first, because opening +involves asynchronous I/O that a synchronous initializer cannot perform. +This split between construction and opening matters when a host wants to +retry a failed open without recreating the sink. + +`receive(_:)` is the method that matters. It is declared as a synchronous, +non-blocking function because the `StatsSink` protocol documents that +requirement: whatever calls `Intellectus.report(_:)` must not stall +waiting for a database write. To honor that, `receive(_:)` captures the +store, the dropbox ID, and the logger — each `Sendable` — and immediately +starts an unstructured `Task`. The method then returns; the caller never +waits. + +Inside the `Task`, the real work happens in three steps. First, it calls +`store.isMonitoringEnabled()`. This is the flag-row check described in +`OVERVIEW.md`: a cheap read of one row in the `control` table. If +monitoring is off, the sample is discarded and a debug-level log line is +written — debug level specifically, because this check runs on every +single sample when monitoring is off, and a louder log level would flood +the console. Second, if monitoring is on, the code switches on the +`StatSample` case: a `.metric` sample calls `store.insertMetric(...)`, an +`.event` sample calls `store.insertEvent(...)`. Third, the whole block is +wrapped in a `do`/`catch`; any thrown error — from the flag read or from +either insert — is logged at `.error` level and never rethrown. A +telemetry sink that could crash the process it is monitoring would defeat +its own purpose, so every failure mode here ends in a log line, never a +propagated error. + +The file's header comments record a deliberate scope limit for this +version: there is no in-process buffering. Each sample launches its own +`Task`, and SQLite's write-ahead log mode is relied on to make concurrent +writes from many such tasks safe and reasonably efficient. The comments +name a possible future improvement — batching many samples behind a timer +— and explain why it was not built yet: stats recording sits off the +hot substrate path, so an occasional dropped sample under heavy load is an +acceptable cost, and the simpler design is easier to reason about. + +## StatsStore.swift + +This file provides `StatsStore`, the SQLite-backed component that owns the +telemetry schema, and `StatsStoreSchema`, a namespace of string constants +that name its tables and columns. + +### Schema constants — `StatsStoreSchema` + +Every table name and column name that any part of this package touches is +declared once, as a `public static let`, inside `StatsStoreSchema`. The +file's own comment explains the reason: gathering the strings in one place +turns a typo into a compile error instead of a query that silently matches +zero rows. `monitoringKey` and `retentionCutoffKey` additionally name the +two well-known rows inside the `control` table, since that table is a +generic key-value store rather than one column per concept. + +### Schema declaration — `StatsStore.schema` + +`StatsStore.schema` is a `SchemaDeclaration` value — a data structure from +PersistenceKit that describes tables, columns, indices, and migrations +without executing any SQL directly. `StatsStore.schemaVersion` (currently +`3`) is the version number PersistenceKit checks against this +declaration's migrations to decide whether an existing database needs to +be upgraded. + +The schema declares four tables. `metric_samples` and `event_samples` hold +the telemetry rows this library exists to record; both use a +`.uuid("row_id")` synthetic primary key that the store assigns, so callers +never have to generate or track one. `control` is the key-value table for +the monitoring flag and the retention-cutoff timestamp; its primary key is +the `key` column itself, which is what lets `setMonitoringEnabled` use an +upsert rather than a delete-then-insert. `topology_snapshots`, added in +schema version 2, holds one row per estate — its primary key is the +`estate` column, so writing a new snapshot for an estate automatically +replaces the old one instead of accumulating history. + +Two indices — `idx_metric_samples_ts` and `idx_event_samples_ts` — exist +purely to make retention fast. Retention deletes rows with `ts` below a +cutoff; without an index on `ts`, each retention pass would scan the +entire table. + +Two migrations are declared, one per schema version bump. +`fromVersion: 1, toVersion: 2` adds the `topology_snapshots` table; the +comment notes this is purely additive; no existing row is touched, and the +new table starts empty until the autonomic governor's next duty cycle +populates it. `fromVersion: 2, toVersion: 3` adds the nullable +`topology_fingerprint` column to that same table, again additively — any +row written before this version simply reads back with that column `nil`. + +### Lifecycle — `init`, `open()`, `close()` + +`init(url:)` builds the underlying `SQLiteStorage` for a database file at +`url`, generating a fresh random estate identifier for the storage +configuration. It performs no I/O beyond what opening a SQLite connection +requires. + +`open()` does two things: it asks PersistenceKit's storage layer to apply +`StatsStore.schema` (creating tables and running any pending migrations), +then it seeds two default rows in the `control` table — `"monitoring"` set +to `"0"` and `"retention_cutoff"` set to the epoch-zero ISO-8601 string. +Both seeds go through `seedControlIfAbsent(key:value:)`, a private helper +that checks whether a row with that key already exists before inserting. +This distinction — seed-if-absent rather than upsert — matters because the +monitoring flag is meant to be a persistent switch: if `open()` blindly +overwrote it every time, an operator's choice to turn monitoring on would +be silently reset to off on every process restart. Seeding only when +absent makes the first `open()` call install the defaults and every later +call on the same database a no-op for those two rows. + +`close()` closes the underlying storage. It is documented as idempotent — +safe to call more than once — which matters for callers that close a store +in both a normal shutdown path and a `defer` block. + +### Monitoring flag — `isMonitoringEnabled()`, `setMonitoringEnabled(_:)` + +`isMonitoringEnabled()` reads the single `control` row whose key is +`"monitoring"` and returns `true` only if its value is the string `"1"`. +If the row is missing entirely — for instance, if a caller queries a store +that was never opened — the function returns `false`. This fail-safe +default matters because a missing flag should never be interpreted as +"recording is on." + +`setMonitoringEnabled(_:)` writes `"1"` or `"0"` to that same row using an +upsert keyed on the `key` column, so it works whether or not the row +already exists. The manager process is the only intended caller of this +method; `PersistenceStatsSink` only ever reads the flag, never sets it. +This asymmetry is what lets one manager control every consumer's recording +behavior without touching the consumers themselves. + +### Writing samples — `insertMetric(...)`, `insertEvent(...)` + +`insertMetric(name:value:tags:ts:dropboxID:)` inserts one row into +`metric_samples`. It generates the row's UUID primary key itself, encodes +the `tags` dictionary to a JSON string with `encodeTagsJSON(_:)`, and +converts the caller-supplied `ts` — epoch seconds, a `Double` — to a +`Date` for storage. The schema stores timestamps as ISO-8601 text rather +than a native numeric column; the conversion from epoch seconds happens +here, at the boundary between the caller's representation and the store's, +so every other part of the codebase can treat `ts` as a plain number. + +`insertEvent(kind:nounType:rowID:estate:ts:dropboxID:)` does the same for +`event_samples`. Its `rowID` parameter is stored in the `estate_row_id` +column rather than a column literally named `row_id`, because that name is +already taken by the table's own synthetic primary key; the file's +comments call this out explicitly so a future reader does not confuse the +estate's row identifier with the row's own storage identifier. + +### Topology snapshots — `writeTopologySnapshot(...)`, `latestTopologySnapshot(estate:)`, `loadTopologyFingerprint(estate:)` + +`writeTopologySnapshot(estate:generatedAt:payload:fingerprint:)` upserts +one row per estate into `topology_snapshots`, keyed on the `estate` +column, so a new snapshot always replaces the previous one for that +estate rather than accumulating history. The `payload` parameter is raw +`Data` that must decode as UTF-8 text; the function throws if it does not, +because the autonomic governor that calls this method always produces +valid UTF-8 JSON, and a failure here signals a bug worth surfacing rather +than silently storing garbage. The optional `fingerprint` parameter is a +stable, process-independent hash of the topology inputs; when supplied, it +lets a restarting governor compare against a freshly computed fingerprint +and skip re-reading the estate's full set of drawers, tunnels, and facts +when nothing has changed. When omitted, the column is written as `nil`. + +`latestTopologySnapshot(estate:)` reads the payload back. Passing a +specific estate identifier is a primary-key lookup that returns at most +one row. Passing `nil` instead asks for the single newest snapshot across +every estate in the table — the behavior the dashboard's "all estates" +view relies on, since it has no single estate identifier to filter by. +Comparing "newest" requires care: the column is declared as a timestamp, +but PersistenceKit's SQLite backend reads timestamp columns back as plain +text, while its in-memory backend used in tests reads them back as native +timestamp values. `generatedAtInstant(_:)` is a small private helper that +normalizes either representation to a `Date` before comparing, because an +earlier version of this code matched only the timestamp representation, +which meant every row tied under SQLite and an arbitrary one won — a bug +the in-memory-backed tests could not catch because they never exercised +this mismatch. + +`loadTopologyFingerprint(estate:)` is the read side of the fingerprint +parameter described above: a primary-key lookup that returns `nil` when no +snapshot exists yet, when the row predates schema version 3, or when a +snapshot was written without a fingerprint. + +### Reading samples — `queryMetrics`, `queryMetricsByNames`, `queryEvents`, `countMetrics` + +`queryMetrics(dropboxID:)` and `queryEvents(dropboxID:)` return every row +in their respective tables, optionally filtered to one consumer's +dropbox, ordered by timestamp ascending. Both decode raw storage rows into +the public `MetricRow` and `EventRow` structs, skipping any row that fails +to decode rather than throwing — a defensive choice that tolerates a +malformed row without losing the rest of the query's results. + +`queryMetricsByNames(_:dropboxID:)` exists for a specific reason: reading +every metric row and filtering client-side does not scale once the table +holds many distinct metric names. This method issues a `WHERE name IN +(...)` predicate so the database itself narrows the result set, and it +short-circuits to an empty array without querying at all when the caller +passes an empty name set. The method's documentation says explicitly that +hot read paths should use this instead of `queryMetrics` plus a Swift-side +filter. + +`countMetrics()` answers "how many metric rows exist" with a `COUNT(*)` +query that never decodes a single row. It backs a dashboard total-count +display without paying the cost of reading and parsing every row just to +throw away everything except the count. + +### Retention — `deleteMetricsBefore(cutoff:now:)`, `deleteEventsBefore(cutoff:now:)` + +Both methods delete rows whose `ts` is strictly before a caller-supplied +`cutoff`, then update the `retention_cutoff` control row through the +private `recordRetentionCutoff(_:now:)` helper. Neither method reads the +system clock; `cutoff` and `now` both come from the caller. This +determinism rule — no `Date()` calls inside the engine — is what lets a +test construct an exact, reproducible retention scenario instead of racing +against wall-clock time. The stored cutoff records the boundary that was +applied, not the moment retention ran, because a dashboard showing "data +older than this point was removed" is more useful to an operator than a +timestamp of when the deletion happened to execute. + +### DB-layer health — `storageStats(now:)` + +`storageStats(now:)` reports statistics about the stats store's own SQLite +file — page counts, WAL frame counts, file size — as distinct from any +statistics about an observed estate's own storage. It delegates to +PersistenceKit's `StorageIntrospection` capability, which the SQLite +backend implements directly, so the call always succeeds in practice; the +optional return type exists only to keep the API honest for a +hypothetical backend that does not support introspection. The `now` +parameter follows the same determinism rule as retention: the caller +supplies the timestamp to stamp on the snapshot. + +### JSON tag encoding — `encodeTagsJSON(_:)`, `decodeTagsJSON(_:)` + +Metric tags are a flat `[String: String]` map, stored as a JSON text +column rather than a separate table, because the tag set is small and +simple enough that a second table would add complexity without adding +value. `encodeTagsJSON(_:)` encodes with `.sortedKeys` so that the same +tag map always produces the same JSON string — useful for tests and for +any future deduplication logic that might compare encoded rows. +`decodeTagsJSON(_:)` is forward-compatible by design: a string that fails +to parse decodes to an empty dictionary rather than throwing, so a future +change to the tag encoding cannot break old rows written before the +change. + +### Row result types — `MetricRow`, `EventRow` + +Both are plain `Sendable` structs that mirror a decoded database row. +Their failable initializers, `init?(storageRow:)`, pattern-match every +expected column and its expected `TypedValue` case; if any column is +missing or has an unexpected type, the initializer returns `nil` rather +than crashing. This is what lets `queryMetrics` and `queryEvents` use +`compactMap` to silently drop malformed rows instead of failing the whole +query. + +## Rust Port and Conformance + +The `rust/` directory contains a second implementation of this library: +`src/lib.rs` re-exports the public surface, `src/sink.rs` implements +`PersistenceStatsSink`, and `src/store.rs` implements `StatsStore` and +`StatsStoreSchema` against the Rust build of PersistenceKit. The two ports +share table names, column names, and the monitoring-flag semantics +exactly; the schema-declaration code in `store.rs`'s `make_schema()` +function is a line-by-line mirror of the Swift `StatsStore.schema` value. + +The two legs differ where Rust and Swift semantics genuinely differ, +rather than translating one language's idioms into the other +mechanically. `PersistenceStatsSink::receive` in Rust performs its +database write inline, synchronously, because the Rust `StatsSink` trait +is synchronous and Rust's `SqliteStorage` is safely accessible without an +async runtime; the Swift version's `Task` dispatch exists only because +Swift's `Storage` protocol is actor-isolated and asynchronous. The +monitoring-flag check and the discard-on-off behavior are identical in +both. + +The Rust `store.rs` file also owns timestamp conversion explicitly through +two free functions, `epoch_to_iso8601` and `iso8601_to_epoch`, because Rust +has no equivalent to Foundation's `DateFormatter` built in. These functions +clamp out-of-range or non-finite inputs (`NaN`, infinities, values outside +the year 0001–9999 window) to the nearest valid boundary instead of +panicking or producing malformed text; a block of unit tests at the bottom +of `store.rs` checks every one of these edge cases directly, since a +malformed ISO-8601 string would break every string comparison retention +and topology-snapshot ordering depend on. The Rust store also exposes one +convenience method the Swift store does not need, +`write_topology_snapshot_bytes`, which accepts raw bytes and performs a +lossy UTF-8 conversion — useful because Rust's type system otherwise +forces a caller to prove UTF-8 validity before calling the primary method, +a proof Swift's runtime guard does not require its callers to construct in +advance. + +Conformance between the two ports is enforced by two independent but +matching test suites: `Tests/ObserverSinkTests/ObserverSinkConformanceTests.swift` +and `rust/tests/conformance.rs`. Both exercise the same scenarios — schema +version, control-row seeding, the monitoring flag and its persistence +across a close-and-reopen cycle, metric and event round-trips, retention +roll-off for both tables, tag JSON round-tripping, the topology-snapshot +read/write/latest-wins/per-estate-isolation paths, and the DB-layer health +check — over the same table and column names. When you change either leg's +schema or flag semantics, update both files and both test suites; nothing +in either build enforces this automatically. diff --git a/packages/libs/ObserverSink/docs/OVERVIEW.md b/packages/libs/ObserverSink/docs/OVERVIEW.md new file mode 100644 index 0000000..c78914f --- /dev/null +++ b/packages/libs/ObserverSink/docs/OVERVIEW.md @@ -0,0 +1,129 @@ +--- +doc: OVERVIEW +package: ObserverSink +repo: moot-system +authored_commit: 909513d0a8ecb1e9e903af9f4d25b3e4f2528242 +authored_date: 2026-07-04 +sources: + - path: Sources/ObserverSink/PersistenceStatsSink.swift + blob: e76c55599795f4bc85b860a4901dc9ab04dd201f + - path: Sources/ObserverSink/StatsStore.swift + blob: 483c736feffd82821a8d77030afde0e182d320fc +--- + +# ObserverSink Overview + +## What This Library Does + +ObserverSink records telemetry for the MOOTx01 manager pipeline. Telemetry +is data that describes what a running program is doing, separate from the +memories the program stores. A companion library, IntellectusLib, defines +two shapes of telemetry: a metric (a named number, such as a latency +measurement) and an event (a record that one memory row was captured or +otherwise acted on). IntellectusLib defines only the shapes and a plug-in +point; it does not know how to save anything to disk. + +ObserverSink is the plug-in that saves them. It ships two pieces. The +first, `PersistenceStatsSink`, is the receiver that IntellectusLib calls +each time new telemetry arrives. The second, `StatsStore`, is a small +SQLite database and the code that reads and writes it. Together they turn +a stream of in-memory telemetry values into durable rows that a separate +manager process can later query. + +## The Problem It Solves + +A running MOOTx01 process is a black box unless something records what it +does. An operator who wants to know how fast captures are running, how +many think-cycles fired, or whether a given estate is behaving normally +needs that information written somewhere durable, because the process +itself may restart or be inspected only after the fact. An estate is one +user's complete memory store in MOOTx01. + +Writing telemetry to disk on every observation is expensive if done +carelessly, and it is dangerous if it cannot be turned off. ObserverSink +solves both problems with one mechanism: a flag row in the database +itself. Before writing anything, the sink checks whether monitoring is +switched on. The manager — not the process being observed — owns that +switch, so it can enable or disable recording for every consumer at once, +without restarting anything. This is the flag-row signal mechanism +described in the source comments as Bob's confirmed choice for Manager 1.0. + +The library also has to work without slowing down the code it observes. +`receive(_:)`, the method IntellectusLib calls, must return immediately +because it may be called from time-sensitive code paths. ObserverSink +meets this requirement by never performing file I/O on the calling +thread; the actual database write always happens elsewhere. + +## How It Works + +Each telemetry value follows the same path. IntellectusLib calls +`PersistenceStatsSink.receive(_:)` with one `StatSample`, which is either +a `.metric` case or an `.event` case. The sink captures what it needs and +starts an unstructured background task, then returns immediately. The +calling code never waits for the database write to finish. + +Inside that background task, the sink first asks the store whether +monitoring is enabled. This is a database read, but a cheap one: the +`control` table holds only a handful of rows. If the flag is off, the +sample is dropped and nothing else happens. If it is on, the sink asks the +store to serialize the sample into the correct table: `.metric` samples go +into `metric_samples`, `.event` samples go into `event_samples`. If the +write fails for any reason, the error is logged and swallowed — a +telemetry failure must never crash the program it is describing. + +The `StatsStore` also owns two other kinds of data beyond metrics and +events. A `control` table holds the monitoring flag itself, plus a +timestamp recording when data was last rolled off through retention. A +`topology_snapshots` table holds one row per estate: the latest picture of +that estate's structure, written by a separate background process (the +autonomic governor) and served back verbatim to dashboards that ask for +it. This table is not telemetry in the metric/event sense, but it shares +the same store, the same open/migrate machinery, and the same principle: +one writer, many readers, no history kept beyond the latest value. + +Retention keeps the two sample tables from growing without bound. The +manager periodically calls `deleteMetricsBefore(cutoff:now:)` and +`deleteEventsBefore(cutoff:now:)` with a cutoff timestamp it computes +itself; the store never reads the system clock on its own. This keeps the +store's behavior fully determined by its caller's inputs, which matters +for testing and for reasoning about what a given retention pass will do. + +## How the Pieces Fit + +Figure 1 shows the library's topology — its major parts and how data moves +between them. + +![Figure 1. Topology of ObserverSink](topology.svg) + +*Figure 1. Topology of ObserverSink. A telemetry sample flows from the +caller through the sink's background task, past the monitoring-flag gate, +into the matching SQLite table. A separate flow lets the autonomic +governor publish topology snapshots that a dashboard reads back later.* + +`Intellectus.report(_:)`, part of IntellectusLib, is the only caller of +`PersistenceStatsSink.receive(_:)`. The sink holds a reference to a +`StatsStore` and a `dropboxID` string that tags every row it writes with +the identity of the process that produced it — this lets one shared store +serve many concurrent consumers (for example, several `aria-mcp` process +instances) without their rows becoming mixed up. SQLite's write-ahead log +mode makes concurrent writers from separate consumers safe without extra +locking in this library. + +`StatsStore` wraps a `SQLiteStorage` value supplied by PersistenceKit, a +kit that provides the schema-declaration and row-storage machinery this +library builds on. `StatsStore` never talks to SQLite directly; it +describes its four tables as a `SchemaDeclaration` and lets PersistenceKit +create, migrate, and query them. + +## What Ships in the Package + +The package ships two Swift source files — `PersistenceStatsSink.swift` +and `StatsStore.swift` — plus a Rust port in `rust/` that reimplements the +same schema and the same sink logic for non-Swift consumers. Both legs are +exercised by parallel conformance test suites (`Tests/ObserverSinkTests` +in Swift, `rust/tests/conformance.rs` in Rust) that check the same sixteen +scenarios: schema version, control-row seeding, the monitoring flag, +metric and event round-trips, retention roll-off, tag encoding, and the +topology-snapshot read/write paths. There are no pinned data artifacts in +this package; every table starts empty and is populated only by the +processes that use it at run time. diff --git a/packages/libs/ObserverSink/docs/topology.svg b/packages/libs/ObserverSink/docs/topology.svg new file mode 100644 index 0000000..975a8e8 --- /dev/null +++ b/packages/libs/ObserverSink/docs/topology.svg @@ -0,0 +1,104 @@ + + + + + + + + + + + + + ObserverSink: telemetry sample in, durable row out + + + + Intellectus.report + caller (IntellectusLib) + + + PersistenceStatsSink + receive() → Task, flag gate + + + StatsStore + insertMetric / insertEvent + + + + + + + + + SQLite stats store (StatsStore, schema v3) + + + control + monitoring flag + + + metric_samples + name, value, tags, ts + + + event_samples + kind, noun_type, estate + + + topology_snapshots + 1 row/estate, latest-wins + + + + + + + + + Autonomic Governor + writeTopologySnapshot + + + + + + Manager / moot-mgr + setMonitoringEnabled + deleteMetricsBefore / deleteEventsBefore + + + + + + + + Dashboard reads + queryMetrics(ByNames) · queryEvents · + countMetrics · latestTopologySnapshot + + + + + From cf1d0676c989cee39c5a570ba43207da3721bc44 Mon Sep 17 00:00:00 2001 From: Fable Date: Sat, 4 Jul 2026 22:15:03 -0500 Subject: [PATCH 2/2] =?UTF-8?q?docs(sdk):=20rewrite=20prose=20to=20the=20f?= =?UTF-8?q?our=20immutable=20laws=20=E2=80=94=20FK=20<=3D=208.0,=20no=20em?= =?UTF-8?q?-dashes?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Full rewrite of OVERVIEW.md and DETAILS.md per package: Turabian/Chicago, reading level 8.0 or less, no sentence over 2 commas outside simple lists, no 'not X, but Y' or 'it is not X, it is Y' forms. AGENT_MAP and topology.svg scrubbed of em-dashes only. Front matter unchanged. Gated by scripts/docs-style-lint.py (mootx01-ee): 65/65 pass; docs-scrub clean. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01Pzkqg3bpbbhfuWRsYomJ3o --- .../kits/ConvergenceKit/docs/AGENT_MAP.md | 170 +- packages/kits/ConvergenceKit/docs/DETAILS.md | 665 +++--- packages/kits/ConvergenceKit/docs/OVERVIEW.md | 209 +- .../kits/PersistenceKit/docs/AGENT_MAP.md | 392 +-- packages/kits/PersistenceKit/docs/DETAILS.md | 2102 +++++++++-------- packages/kits/PersistenceKit/docs/OVERVIEW.md | 287 +-- .../kits/PersistenceKit/docs/topology.svg | 8 +- packages/kits/QueueKit/docs/AGENT_MAP.md | 230 +- packages/kits/QueueKit/docs/DETAILS.md | 964 ++++---- packages/kits/QueueKit/docs/OVERVIEW.md | 208 +- packages/kits/QueueKit/docs/topology.svg | 4 +- packages/libs/LoopbackHTTP/docs/AGENT_MAP.md | 88 +- packages/libs/LoopbackHTTP/docs/DETAILS.md | 411 ++-- packages/libs/LoopbackHTTP/docs/OVERVIEW.md | 147 +- packages/libs/ObserverSink/docs/AGENT_MAP.md | 126 +- packages/libs/ObserverSink/docs/DETAILS.md | 528 +++-- packages/libs/ObserverSink/docs/OVERVIEW.md | 150 +- 17 files changed, 3472 insertions(+), 3217 deletions(-) diff --git a/packages/kits/ConvergenceKit/docs/AGENT_MAP.md b/packages/kits/ConvergenceKit/docs/AGENT_MAP.md index a65ee40..3051db8 100644 --- a/packages/kits/ConvergenceKit/docs/AGENT_MAP.md +++ b/packages/kits/ConvergenceKit/docs/AGENT_MAP.md @@ -25,116 +25,116 @@ sources: blob: 5d3556a08c8bc80ec6b824ceb4e0838db9c95ecc --- -# AGENT_MAP — ConvergenceKit +# AGENT_MAP: ConvergenceKit PURPOSE: replicates PersistenceKit rows across device/estate boundaries. One protocol (SyncEngine), three interchangeable backends (None/CloudKit/Federation). App writes only to PersistenceKit; backend observes via StorageObserver, ships via SyncRecord wire format, applies inbound through PersistenceKit's own write path (RowStore.upsert/insert/delete), which fires StorageObserver again on the receiving side. -DEPS: core target imports SubstrateTypes (HLC, Fingerprint256), PersistenceKit (Storage, StorageObserver, TableChange, TypedValue). ConvergenceKitCloudKit adds CloudKit, os. ConvergenceKitFederation adds Crypto (swift-crypto, Ed25519), os. ConvergenceKitNone adds nothing beyond core. Imported by: NONE within the SDK at this commit — application-layer composition only; QueueKit's Package.swift explicitly excludes it as a dependency (spec §11, DECISION_KIT_GRAPH_REFACTOR_2026-05-19.md). Rust port in rust/ mirrors core types + wire format + None/Federation backends; CloudKit has no Rust port (Apple-only, by design). +DEPS: core target imports SubstrateTypes (HLC, Fingerprint256), PersistenceKit (Storage, StorageObserver, TableChange, TypedValue). ConvergenceKitCloudKit adds CloudKit, os. ConvergenceKitFederation adds Crypto (swift-crypto, Ed25519), os. ConvergenceKitNone adds nothing beyond core. Imported by: NONE within the SDK at this commit: application-layer composition only; QueueKit's Package.swift explicitly excludes it as a dependency (spec §11, DECISION_KIT_GRAPH_REFACTOR_2026-05-19.md). Rust port in rust/ mirrors core types + wire format + None/Federation backends; CloudKit has no Rust port (Apple-only, by design). ENTRY POINTS (most callers need only these): -- SyncEngine.swift:26 `SyncEngine.enable(manifest:storage:) async throws` — call once before push/pull/subscribe -- SyncEngine.swift:34/:38 `push() async throws -> SyncReceipt` / `pull() async throws -> SyncReceipt` — one-shot cycles -- SyncEngine.swift:41 `subscribe() -> AsyncStream` — live feed for UI -- FederationSyncEngine.swift:80 `FederationSyncEngine.pair(with:via:family:)` — Federation-only, establishes peer relationship before push/pull do anything +- SyncEngine.swift:26 `SyncEngine.enable(manifest:storage:) async throws`: call once before push/pull/subscribe +- SyncEngine.swift:34/:38 `push() async throws -> SyncReceipt` / `pull() async throws -> SyncReceipt`: one-shot cycles +- SyncEngine.swift:41 `subscribe() -> AsyncStream`: live feed for UI +- FederationSyncEngine.swift:80 `FederationSyncEngine.pair(with:via:family:)`: Federation-only, establishes peer relationship before push/pull do anything ## Symbol Table -### Protocol — SyncEngine.swift -- :21 `protocol SyncEngine: Sendable` — 4 methods + 1 async property, every backend conforms -- :26 `enable(manifest:storage:) async throws` — starts local-write observation; establishes remote subscriptions if any -- :30 `disable() async throws` — idempotent teardown -- :34 `push()` / :38 `pull()` — one-shot, return SyncReceipt +### Protocol: SyncEngine.swift +- :21 `protocol SyncEngine: Sendable`: 4 methods + 1 async property, every backend conforms +- :26 `enable(manifest:storage:) async throws`: starts local-write observation; establishes remote subscriptions if any +- :30 `disable() async throws`: idempotent teardown +- :34 `push()` / :38 `pull()`: one-shot, return SyncReceipt - :41 `subscribe() -> AsyncStream` - :44 `state: SyncState { get async }` -### Core types — SyncTypes.swift -- :22 `enum SyncDirection: String` — bidirectional | pushOnly | pullOnly -- :29 `enum ConflictPolicy: String` — lastWriterWinsByHLC (default) | appendOnly | localWins | remoteWins -- :42 `struct SyncedTable` — name/direction/primaryKeyColumn/conflictPolicy; :54 init defaults direction=.bidirectional, conflictPolicy=.lastWriterWinsByHLC -- :70 `struct SyncManifest` — kitID/schemaVersion/zoneIdentifier/tables; :94 `table(named:) -> SyncedTable?` -- :100 `struct SyncReceipt` — pushed/pulled/conflicts/timestamp; :113 `.empty` shared zero-value -- :117 `enum SyncEvent` — .remoteChangesApplied(count:) / .pushCompleted(receipt:) / .peerConnected(identity:) / .peerDisconnected(identity:reason:) / .error(SyncError) -- :126 `enum SyncState` — .disabled | .enabled(zone:lastPushAt:lastPullAt:) | .syncing(direction:) | .error(_:retryAt:) -- :134 `enum SyncError: Error, Equatable` — notEnabled/alreadyEnabled/schemaMismatch/kitMismatch/transportFailure/decodingFailure/encodingFailure/peerUnreachable/authenticationFailed/unsupportedTable/corruptRemoteIdentity(recordName:) — last case: NEVER fabricate a UUID for an unparseable remote recordName, see CKRecordMapping.decode - -### Wire format — SyncRecord.swift -- :28 `struct SyncRecord: Codable` — table/event/rowKey/values/hlc/schemaVersion/kitID; explicit CodingKeys — Rust serde renames match verbatim, do not let these drift -- :63 `enum SyncEventKind: String, Codable` — insert|update|delete; :68 `init(from: StorageEvent)`, :76 `.asStorageEvent` — Codable mirror, kept separate from PersistenceKit.StorageEvent deliberately -- :87 `struct PackedHLC: Codable, Hashable` — physicalTime/logicalCount/nodeID; Codable wrapper of SubstrateTypes.HLC -- :112 `struct SyncValueMap: Codable` — wraps [String:TypedValue] as [String:SyncValueBox] -- :144 `struct SyncValueBox` — kind + payload (13 TypedValue cases); adjacently-tagged JSON matching Rust `#[serde(tag="kind",content="payload")]` -- :206 `SyncValueBox: Codable` extension — :211 encode omits payload key for .null (matches Rust unit-variant omission); :227 timestamp encode guards `interval.isFinite` + Int64 range BEFORE `Int64(interval)` cast — Int64(_:) traps on NaN/±inf/overflow, corrupt inbound data can produce these -- :296 `struct FingerprintWire: Codable, Hashable` — block0..3 UInt64, Codable wrapper of SubstrateTypes.Fingerprint256 - -### None backend — ConvergenceKitNone.swift -- :28 `final class NoSyncEngine: SyncEngine` — passthrough; push/pull return .empty once enabled, throw .notEnabled before -- :53 `subscribe()` — stream that NEVER emits; closes only on caller task cancellation -- :64 `actor StateActor` — enable throws .alreadyEnabled on repeat; tracks manifest only for state reporting - -### CloudKit mapping — CKRecordMapping.swift -- :26 `enum CKRecordMapping` — generic row↔CKRecord translator, driven by table name + manifest, no per-table hardcoding -- :31 `recordType(kitID:table:) -> String` — "\(kitID)_\(table)" +### Core types: SyncTypes.swift +- :22 `enum SyncDirection: String`: bidirectional | pushOnly | pullOnly +- :29 `enum ConflictPolicy: String`: lastWriterWinsByHLC (default) | appendOnly | localWins | remoteWins +- :42 `struct SyncedTable`: name/direction/primaryKeyColumn/conflictPolicy; :54 init defaults direction=.bidirectional, conflictPolicy=.lastWriterWinsByHLC +- :70 `struct SyncManifest`: kitID/schemaVersion/zoneIdentifier/tables; :94 `table(named:) -> SyncedTable?` +- :100 `struct SyncReceipt`: pushed/pulled/conflicts/timestamp; :113 `.empty` shared zero-value +- :117 `enum SyncEvent`: .remoteChangesApplied(count:) / .pushCompleted(receipt:) / .peerConnected(identity:) / .peerDisconnected(identity:reason:) / .error(SyncError) +- :126 `enum SyncState`: .disabled | .enabled(zone:lastPushAt:lastPullAt:) | .syncing(direction:) | .error(_:retryAt:) +- :134 `enum SyncError: Error, Equatable`: notEnabled/alreadyEnabled/schemaMismatch/kitMismatch/transportFailure/decodingFailure/encodingFailure/peerUnreachable/authenticationFailed/unsupportedTable/corruptRemoteIdentity(recordName:): last case: NEVER fabricate a UUID for an unparseable remote recordName, see CKRecordMapping.decode + +### Wire format: SyncRecord.swift +- :28 `struct SyncRecord: Codable`: table/event/rowKey/values/hlc/schemaVersion/kitID; explicit CodingKeys: Rust serde renames match verbatim, do not let these drift +- :63 `enum SyncEventKind: String, Codable`: insert|update|delete; :68 `init(from: StorageEvent)`, :76 `.asStorageEvent`: Codable mirror, kept separate from PersistenceKit.StorageEvent deliberately +- :87 `struct PackedHLC: Codable, Hashable`: physicalTime/logicalCount/nodeID; Codable wrapper of SubstrateTypes.HLC +- :112 `struct SyncValueMap: Codable`: wraps [String:TypedValue] as [String:SyncValueBox] +- :144 `struct SyncValueBox`: kind + payload (13 TypedValue cases); adjacently-tagged JSON matching Rust `#[serde(tag="kind",content="payload")]` +- :206 `SyncValueBox: Codable` extension: :211 encode omits payload key for .null (matches Rust unit-variant omission); :227 timestamp encode guards `interval.isFinite` + Int64 range BEFORE `Int64(interval)` cast: Int64(_:) traps on NaN/±inf/overflow, corrupt inbound data can produce these +- :296 `struct FingerprintWire: Codable, Hashable`: block0..3 UInt64, Codable wrapper of SubstrateTypes.Fingerprint256 + +### None backend: ConvergenceKitNone.swift +- :28 `final class NoSyncEngine: SyncEngine`: passthrough; push/pull return .empty once enabled, throw .notEnabled before +- :53 `subscribe()`: stream that NEVER emits; closes only on caller task cancellation +- :64 `actor StateActor`: enable throws .alreadyEnabled on repeat; tracks manifest only for state reporting + +### CloudKit mapping: CKRecordMapping.swift +- :26 `enum CKRecordMapping`: generic row↔CKRecord translator, driven by table name + manifest, no per-table hardcoding +- :31 `recordType(kitID:table:) -> String`: "\(kitID)_\(table)" - :36 `recordID(rowKey:zone:) -> CKRecord.ID` -- :43 `record(from:table:rowKey:hlc:schemaVersion:kitID:zone:) throws -> CKRecord` — adds reserved _syncHLC/_syncSchemaVersion/_syncKitID fields -- :65 `decode(_:) throws -> DecodedRecord` — reverse; throws .decodingFailure if _syncHLC/_syncSchemaVersion missing; recovers table from "kitID_table" split; **throws .corruptRemoteIdentity if recordName is not a valid UUID — NEVER fabricate a fresh UUID here** -- :103 `assign(value:to:forKey:)` (private) — TypedValue→CKRecord field; .array throws .encodingFailure (unsupported); .fingerprint packs 4×UInt64 LE into 32 bytes -- :147 `typedValue(from:)` (private) — CKRecord field→TypedValue; NSNumber objCType sniffed for bool/float/int disambiguation -- :172 `packed(_:) -> Int64` / :179 `unpacked(_:) -> HLC` — HLC↔Int64, PINNED bit layout: 48 bits physical | 12 bits logical | 4 bits node — changing widths breaks previously-stored _syncHLC values -- :192 `struct SyncMeta` — hlc/schemaVersion/kitID extracted from _sync* fields -- :198 `struct DecodedRecord` — table/rowKey/values (clean, no _sync* keys)/syncMeta + convenience accessors - -### CloudKit engine — CloudKitSyncEngine.swift -- :34 `final class CloudKitSyncEngine: SyncEngine` — :43 init(containerIdentifier:) defers CKContainer resolution to enable() so tests can construct without iCloud entitlement -- :78 `actor CloudKitStateActor` — owns container/manifest/storage/pendingOutbound/serverChangeToken/hlcGenerator -- :112 `enable(manifest:storage:)` — creates CK zone (tolerates already-exists); subscribes storage.observer.observe per push-eligible table → recordOutbound -- :106 `hlcGenerator = HLCGenerator(nodeID: Int32.random(in: 1...0x0F))` — mints HLC only when an observed change carries none -- :175 `push()` — drains pendingOutbound; builds CKRecords via CKRecordMapping.record; one modifyRecords(saving:deleting:) call; HLC: prefer change.hlc, else hlcGenerator.send(now:) (NOT currentTime() — send advances the logical counter, avoiding same-millisecond collisions) -- :254 `pull()` — recordZoneChanges(inZoneWith:since:); per-record gate: kitID match → schemaVersion match → table declared & not pushOnly; any failure caught+logged+counted as conflict, does NOT abort the batch; deletions are bare CKRecord.ID (no table info) — attempted against every non-pushOnly manifest table -- :339 `applyInbound(_:syncedTable:storage:)` (internal, not private — LWW tests call via @testable) — dispatches 4 ConflictPolicy arms; .lastWriterWinsByHLC reads existing row's _syncHLC as EITHER .hlc (InMemory) OR .int (SQLite/Postgres, raw packed value) — must handle both - -### Federation identity — FederationIdentity.swift -- :24 `struct PeerIdentity: Hashable` — 32-byte Ed25519 public key -- :32 `struct LocalIdentity` — :36 init() generates fresh Curve25519.Signing.PrivateKey per engine instance; :42 init(privateKeyBytes:) restores from caller bytes — module does NOT persist identity itself +- :43 `record(from:table:rowKey:hlc:schemaVersion:kitID:zone:) throws -> CKRecord`: adds reserved _syncHLC/_syncSchemaVersion/_syncKitID fields +- :65 `decode(_:) throws -> DecodedRecord`: reverse; throws .decodingFailure if _syncHLC/_syncSchemaVersion missing; recovers table from "kitID_table" split; **throws .corruptRemoteIdentity if recordName is not a valid UUID: NEVER fabricate a fresh UUID here** +- :103 `assign(value:to:forKey:)` (private): TypedValue→CKRecord field; .array throws .encodingFailure (unsupported); .fingerprint packs 4×UInt64 LE into 32 bytes +- :147 `typedValue(from:)` (private): CKRecord field→TypedValue; NSNumber objCType sniffed for bool/float/int disambiguation +- :172 `packed(_:) -> Int64` / :179 `unpacked(_:) -> HLC`: HLC↔Int64, PINNED bit layout: 48 bits physical | 12 bits logical | 4 bits node: changing widths breaks previously-stored _syncHLC values +- :192 `struct SyncMeta`: hlc/schemaVersion/kitID extracted from _sync* fields +- :198 `struct DecodedRecord`: table/rowKey/values (clean, no _sync* keys)/syncMeta + convenience accessors + +### CloudKit engine: CloudKitSyncEngine.swift +- :34 `final class CloudKitSyncEngine: SyncEngine`: :43 init(containerIdentifier:) defers CKContainer resolution to enable() so tests can construct without iCloud entitlement +- :78 `actor CloudKitStateActor`: owns container/manifest/storage/pendingOutbound/serverChangeToken/hlcGenerator +- :112 `enable(manifest:storage:)`: creates CK zone (tolerates already-exists); subscribes storage.observer.observe per push-eligible table → recordOutbound +- :106 `hlcGenerator = HLCGenerator(nodeID: Int32.random(in: 1...0x0F))`: mints HLC only when an observed change carries none +- :175 `push()`: drains pendingOutbound; builds CKRecords via CKRecordMapping.record; one modifyRecords(saving:deleting:) call; HLC: prefer change.hlc, else hlcGenerator.send(now:) (NOT currentTime(): send advances the logical counter, avoiding same-millisecond collisions) +- :254 `pull()`: recordZoneChanges(inZoneWith:since:); per-record gate: kitID match → schemaVersion match → table declared & not pushOnly; any failure caught+logged+counted as conflict, does NOT abort the batch; deletions are bare CKRecord.ID (no table info): attempted against every non-pushOnly manifest table +- :339 `applyInbound(_:syncedTable:storage:)` (internal, not private: LWW tests call via @testable): dispatches 4 ConflictPolicy arms; .lastWriterWinsByHLC reads existing row's _syncHLC as EITHER .hlc (InMemory) OR .int (SQLite/Postgres, raw packed value): must handle both + +### Federation identity: FederationIdentity.swift +- :24 `struct PeerIdentity: Hashable`: 32-byte Ed25519 public key +- :32 `struct LocalIdentity`: :36 init() generates fresh Curve25519.Signing.PrivateKey per engine instance; :42 init(privateKeyBytes:) restores from caller bytes: module does NOT persist identity itself - :48 `sign(_:) throws -> Data` -- :53 `enum FederationSignature` — :54 `verify(_:of:by:) -> Bool` — returns false (not throw) for both malformed pubkey bytes and bad signature +- :53 `enum FederationSignature`: :54 `verify(_:of:by:) -> Bool`: returns false (not throw) for both malformed pubkey bytes and bad signature -### Federation pairing shapes — HyperplaneFamilyExchange.swift -- :27 `struct HyperplaneFamilySpec: Codable, Hashable` — seed + dimension (default 256); seed alone reproduces the family deterministically on both sides -- :42 `struct PairingProposal: Codable` — proposerPublicKey/proposedFamily/nonce -- :54 `struct PairingAcceptance: Codable` — accepterPublicKey/acceptedFamily/signatureOfProposal -- NOTE: these types are defined but NOT YET wired into FederationSyncEngine.pair — v1.0 pair() takes an already-agreed HyperplaneFamilySpec directly, does not negotiate/sign Proposal/Acceptance +### Federation pairing shapes: HyperplaneFamilyExchange.swift +- :27 `struct HyperplaneFamilySpec: Codable, Hashable`: seed + dimension (default 256); seed alone reproduces the family deterministically on both sides +- :42 `struct PairingProposal: Codable`: proposerPublicKey/proposedFamily/nonce +- :54 `struct PairingAcceptance: Codable`: accepterPublicKey/acceptedFamily/signatureOfProposal +- NOTE: these types are defined but NOT YET wired into FederationSyncEngine.pair: v1.0 pair() takes an already-agreed HyperplaneFamilySpec directly, does not negotiate/sign Proposal/Acceptance -### Federation engine — FederationSyncEngine.swift +### Federation engine: FederationSyncEngine.swift - :39 `final class FederationSyncEngine: SyncEngine` -- :80 `pair(with:via:family:) async throws` — exchanges pubkeys + family spec; in-process only at v1.0 +- :80 `pair(with:via:family:) async throws`: exchanges pubkeys + family spec; in-process only at v1.0 - :84 `identity: LocalIdentity { get async }` -- :98 `enum PayloadKind: UInt8, Codable` — .syncRecordBatch = 0x01 (only v1.0 variant); 0x02 RESERVED for fieldWriteEventBatch — never reassign -- :128 `envelopeSigningBytes(senderPublicKey:payloadKind:payload:hlc:) -> Data` — PINNED byte layout (all LE): pubkey(32) | kind(1) | payload_len(4) | payload(N) | hlc.physicalTime(8) | hlc.logicalCount(4) | hlc.nodeID(4); byte-identical to Rust `envelope_signing_bytes` in federation.rs — signature covers THIS, not raw payload (closes relabel/replay seam) -- :174 `struct SignedEnvelope: Codable` — senderPublicKey/payloadKind/payload/signature/hlc; batch-level hlc minted AFTER all record HLCs in the batch (strictly later) -- :211 `protocol Relay: Sendable` — send(to:message:) / drain(for:) — transport extension point; hosted relay is a drop-in conformer -- :220 `final class FederationRelay: Relay` — in-process only at v1.0, NSLock-protected [pubkey: [SignedEnvelope]] inbox dict -- :243 `actor FederationStateActor` — :244 `localIdentity = LocalIdentity()` generated once per actor -- :263 `enable(manifest:storage:)` — same observer-wiring pattern as CloudKitStateActor -- :279 `disable()` — cancels observer tasks THEN AWAITS each task's completion before clearing state (deliberately different from a cancel-only teardown — closes a race where a buffered change lands after disable() returns) -- :318 `pair(with:via:family:)` — registers peer locally, calls peerActor.acceptPeering symmetrically — one call registers both sides -- :331 `push()` — builds SyncRecord batch from pendingOutbound (HLC: change.hlc ?? hlcGenerator.send(now:), same rule as CloudKit), JSON-encodes, mints batch HLC, signs via envelopeSigningBytes, sends SignedEnvelope to every peer's relay -- :422 `pull()` — per envelope, in order: (1) senderPublicKey MUST equal the specific paired peer's key (ADR-013 — valid self-signature alone does not prove pairing authorization), (2) payloadKind MUST be .syncRecordBatch, (3) signature MUST verify over envelopeSigningBytes, (4) only then JSON-decode; each surviving SyncRecord re-checked for kitID/schemaVersion/table-declared before applyInbound -- :526 `applyInbound(_:syncedTable:storage:)` (internal, not private — LWW tests) — 4 ConflictPolicy arms × insert/update vs delete (delete gets full policy dispatch here, unlike CloudKit which receives bare record IDs with no policy routing): .appendOnly rejects remote deletes outright; .lastWriterWinsByHLC gates delete on stale-HLC check same as upsert path; .remoteWins deletes unconditionally; .localWins rejects all remote deletes +- :98 `enum PayloadKind: UInt8, Codable`: .syncRecordBatch = 0x01 (only v1.0 variant); 0x02 RESERVED for fieldWriteEventBatch: never reassign +- :128 `envelopeSigningBytes(senderPublicKey:payloadKind:payload:hlc:) -> Data`: PINNED byte layout (all LE): pubkey(32) | kind(1) | payload_len(4) | payload(N) | hlc.physicalTime(8) | hlc.logicalCount(4) | hlc.nodeID(4); byte-identical to Rust `envelope_signing_bytes` in federation.rs: signature covers THIS, not raw payload (closes relabel/replay seam) +- :174 `struct SignedEnvelope: Codable`: senderPublicKey/payloadKind/payload/signature/hlc; batch-level hlc minted AFTER all record HLCs in the batch (strictly later) +- :211 `protocol Relay: Sendable`: send(to:message:) / drain(for:): transport extension point; hosted relay is a drop-in conformer +- :220 `final class FederationRelay: Relay`: in-process only at v1.0, NSLock-protected [pubkey: [SignedEnvelope]] inbox dict +- :243 `actor FederationStateActor`: :244 `localIdentity = LocalIdentity()` generated once per actor +- :263 `enable(manifest:storage:)`: same observer-wiring pattern as CloudKitStateActor +- :279 `disable()`: cancels observer tasks THEN AWAITS each task's completion before clearing state (deliberately different from a cancel-only teardown: closes a race where a buffered change lands after disable() returns) +- :318 `pair(with:via:family:)`: registers peer locally, calls peerActor.acceptPeering symmetrically: one call registers both sides +- :331 `push()`: builds SyncRecord batch from pendingOutbound (HLC: change.hlc ?? hlcGenerator.send(now:), same rule as CloudKit), JSON-encodes, mints batch HLC, signs via envelopeSigningBytes, sends SignedEnvelope to every peer's relay +- :422 `pull()`: per envelope, in order: (1) senderPublicKey MUST equal the specific paired peer's key (ADR-013: valid self-signature alone does not prove pairing authorization), (2) payloadKind MUST be .syncRecordBatch, (3) signature MUST verify over envelopeSigningBytes, (4) only then JSON-decode; each surviving SyncRecord re-checked for kitID/schemaVersion/table-declared before applyInbound +- :526 `applyInbound(_:syncedTable:storage:)` (internal, not private: LWW tests): 4 ConflictPolicy arms × insert/update vs delete (delete gets full policy dispatch here, unlike CloudKit which receives bare record IDs with no policy routing): .appendOnly rejects remote deletes outright; .lastWriterWinsByHLC gates delete on stale-HLC check same as upsert path; .remoteWins deletes unconditionally; .localWins rejects all remote deletes ## INVARIANTS / GOTCHAS -- Application code NEVER calls a backend's push/pull for ordinary use — it writes to PersistenceKit; the enabled backend observes via StorageObserver and does the rest. Inbound changes apply through PersistenceKit's normal RowStore path (upsert/insert/delete), which is what wakes downstream StorageObserver watchers on the receiving side — do not special-case "just-synced" rows. +- Application code NEVER calls a backend's push/pull for ordinary use: it writes to PersistenceKit; the enabled backend observes via StorageObserver and does the rest. Inbound changes apply through PersistenceKit's normal RowStore path (upsert/insert/delete), which is what wakes downstream StorageObserver watchers on the receiving side: do not special-case "just-synced" rows. - CKRecordMapping.decode: an unparseable `recordID.recordName` MUST throw `.corruptRemoteIdentity`, never fabricate a fresh UUID. A fabricated identity creates a phantom row that desyncs forever (never matches the real remote row on later rounds). - HLC bit layout in CKRecordMapping.packed/unpacked is PINNED: 48/12/4 bits (physical/logical/node). Changing widths invalidates every previously stored `_syncHLC` value. - envelopeSigningBytes layout is PINNED and cross-port: pubkey(32)|kind(1)|payload_len(4 LE)|payload|hlc.physicalTime(8 LE)|hlc.logicalCount(4 LE)|hlc.nodeID(4 LE). Must stay byte-identical to Rust's `envelope_signing_bytes` in `rust/src/federation.rs`, or cross-port signatures fail verification. - SyncRecord / SyncValueBox / PackedHLC / FingerprintWire CodingKeys are an explicit cross-port JSON contract (Rust serde field renames match these strings verbatim). Do not let automatic Codable key derivation replace these without checking the Rust side. -- PayloadKind 0x02 is RESERVED for `fieldWriteEventBatch` — never assign it to anything else; a receiver on an older build would silently misinterpret a reused byte value. +- PayloadKind 0x02 is RESERVED for `fieldWriteEventBatch`: never assign it to anything else; a receiver on an older build would silently misinterpret a reused byte value. - Federation pull() enforces sender==paired-peer-key BEFORE trusting a valid signature (ADR-013): a syntactically valid self-signed envelope from a non-paired sender must be rejected, not merely a badly-signed one. -- `_syncHLC` on a stored row can arrive as TypedValue `.hlc` (InMemory backend, preserves the type) OR `.int` (SQLite/Postgres, raw packed integer, because those schemas don't declare the column as `.hlc`). Both `CloudKitStateActor.applyInbound` and `FederationStateActor.applyInbound` must handle both cases — do not assume one representation. +- `_syncHLC` on a stored row can arrive as TypedValue `.hlc` (InMemory backend, preserves the type) OR `.int` (SQLite/Postgres, raw packed integer, because those schemas don't declare the column as `.hlc`). Both `CloudKitStateActor.applyInbound` and `FederationStateActor.applyInbound` must handle both cases: do not assume one representation. - lastWriterWinsByHLC gates BOTH upserts and deletes on HLC comparison in Federation's applyInbound; CloudKit's applyInbound only gates upserts this way because CloudKit deletes arrive as bare record IDs with no policy dispatch at all (deletes just happen, scoped only by "table is in the manifest and not pushOnly"). -- FederationStateActor.disable() cancels observer tasks THEN awaits their completion before clearing state — do not reduce this to cancel-only; a buffered change could otherwise land in the outbox after disable() returns. -- HLC minting for locally-observed changes with no HLC of their own always uses `HLCGenerator.send(now:)`, never a read-only clock snapshot (`currentTime()` equivalent) — `send` advances the logical counter so two changes minted in the same millisecond do not collide. Same rule in both CloudKitStateActor.push and FederationStateActor.push. -- Timestamp encoding in SyncValueBox guards `interval.isFinite` and Int64-range BEFORE casting to Int64 — `Int64(_:)` traps (crashes) on NaN/±infinity/out-of-range Double. Corrupt inbound sync data can produce such a value; the guard converts a potential crash into a catchable EncodingError. +- FederationStateActor.disable() cancels observer tasks THEN awaits their completion before clearing state: do not reduce this to cancel-only; a buffered change could otherwise land in the outbox after disable() returns. +- HLC minting for locally-observed changes with no HLC of their own always uses `HLCGenerator.send(now:)`, never a read-only clock snapshot (`currentTime()` equivalent): `send` advances the logical counter so two changes minted in the same millisecond do not collide. Same rule in both CloudKitStateActor.push and FederationStateActor.push. +- Timestamp encoding in SyncValueBox guards `interval.isFinite` and Int64-range BEFORE casting to Int64: `Int64(_:)` traps (crashes) on NaN/±infinity/out-of-range Double. Corrupt inbound sync data can produce such a value; the guard converts a potential crash into a catchable EncodingError. - Public API stability per README: adding a case to SyncDirection, ConflictPolicy, SyncEvent, SyncState, or SyncError is a breaking change requiring a major version bump and a decision record. Backend additions (a fourth SyncEngine conformer) are additive, not breaking. - CloudKit backend has NO Rust port by design (Apple-only; Swift side always handles iCloud transport). Do not expect rust/ to mirror ConvergenceKitCloudKit. -- ConvergenceKit is intentionally NOT a dependency of other in-SDK kits at this commit (e.g., QueueKit's Package.swift excludes it per spec §11) — it is wired in at the application-composition layer, not the kit-graph layer. +- ConvergenceKit is intentionally NOT a dependency of other in-SDK kits at this commit (e.g., QueueKit's Package.swift excludes it per spec §11): it is wired in at the application-composition layer, not the kit-graph layer. diff --git a/packages/kits/ConvergenceKit/docs/DETAILS.md b/packages/kits/ConvergenceKit/docs/DETAILS.md index 6050a4a..27cb06b 100644 --- a/packages/kits/ConvergenceKit/docs/DETAILS.md +++ b/packages/kits/ConvergenceKit/docs/DETAILS.md @@ -29,8 +29,9 @@ sources: This document walks through every source file in the package. Read `OVERVIEW.md` first for the big picture. Files appear here in pipeline -order: the shared types and protocol that every backend depends on, the -wire format, then the three backends from simplest to most complex. +order. First come the shared types and the protocol every backend depends +on. Next comes the wire format. Then come the three backends, from +simplest to most complex. ## SyncTypes.swift @@ -38,151 +39,158 @@ This file provides the core enumerations and value types shared by every backend: `SyncDirection`, `ConflictPolicy`, `SyncedTable`, `SyncManifest`, `SyncReceipt`, `SyncEvent`, `SyncState`, and `SyncError`. -`SyncDirection` states which way one table replicates: `bidirectional`, -`pushOnly`, or `pullOnly`. Declaring direction per table matters because not -every table should travel both ways; a table that only a phone should ever -write, for example, can be marked `pushOnly` from the phone and `pullOnly` -everywhere else, and a backend that receives such a table simply skips it. +`SyncDirection` states which way one table replicates. The three choices +are `bidirectional`, `pushOnly`, and `pullOnly`. Direction matters per +table because not every table should travel both ways. A phone-only table +can be marked `pushOnly` from the phone. Every other device marks that same +table `pullOnly`. A backend that receives such a table simply skips it. `ConflictPolicy` states how a receiver resolves a row that changed on both -sides. `lastWriterWinsByHLC` is the default: it compares the Hybrid Logical -Clock, described in `OVERVIEW.md`, on the incoming record against the clock -on the stored row, and the more recent write survives. `appendOnly` treats -the table as a write-once audit log, where the row's own key already makes -duplicate delivery harmless, so incoming rows are always upserted rather -than compared. `localWins` and `remoteWins` are the two absolute policies, -useful when one side of a sync relationship is known in advance to be more -authoritative than the other. - -`SyncedTable.init(name:direction:primaryKeyColumn:conflictPolicy:)` bundles -one table's replication rules; `direction` and `conflictPolicy` default to -`bidirectional` and `lastWriterWinsByHLC` so a simple table declaration only -needs a name and a primary key column. `SyncManifest` is the full -declaration for one sync session: a `kitID` and `schemaVersion` that -identify the application and its data version, a `zoneIdentifier` naming -where remote data lives (a CloudKit zone name or a federation-facing label), -and the list of `SyncedTable` entries. `SyncManifest.table(named:)` looks up -one table's rules by name; every backend calls this before deciding whether +sides. `lastWriterWinsByHLC` is the default. It compares the Hybrid +Logical Clock, described in `OVERVIEW.md`, on the incoming record against +the clock on the stored row. The more recent write survives. `appendOnly` +treats the table as a write-once audit log. The row's own key already +makes duplicate delivery harmless, so incoming rows are always upserted +rather than compared. `localWins` and `remoteWins` are the two absolute +policies. They suit a case where one side of a sync relationship is known +in advance to be more authoritative than the other. + +`SyncedTable.init(name:direction:primaryKeyColumn:conflictPolicy:)` +bundles one table's replication rules. `direction` and `conflictPolicy` +default to `bidirectional` and `lastWriterWinsByHLC`. A simple table +declaration therefore needs only a name and a primary key column. +`SyncManifest` is the full declaration for one sync session. It carries a +`kitID` and a `schemaVersion` that identify the application and its data +version. It carries a `zoneIdentifier` naming where remote data lives, +either a CloudKit zone name or a federation-facing label. It carries the +list of `SyncedTable` entries. `SyncManifest.table(named:)` looks up one +table's rules by name. Every backend calls this before it decides whether and how to apply an incoming or outgoing change. -`SyncReceipt` summarizes one push or pull cycle: how many rows moved in each -direction, how many conflicts were detected, and when the cycle finished. -`SyncReceipt.empty` is the shared zero-value every backend returns when -there is nothing to do, so callers can treat "nothing happened" and "an -empty batch happened" identically. +`SyncReceipt` summarizes one push or pull cycle. It reports how many rows +moved in each direction, how many conflicts turned up, and when the cycle +finished. `SyncReceipt.empty` is the shared zero-value every backend +returns when there is nothing to do. Callers can then treat "nothing +happened" and "an empty batch happened" the same way. `SyncEvent` is what a live subscription emits: remote changes applied, a -push completed, a peer connected or disconnected, or an error. `SyncState` -is the coarser summary a settings screen would bind to: disabled, enabled -with a zone and last-activity timestamps, actively syncing in a direction, -or stuck in an error with an optional retry time. +push completed, a peer connected, a peer disconnected, or an error. +`SyncState` is the coarser summary a settings screen would bind to. One +case is disabled. One case is enabled, carrying a zone and last-activity +timestamps. One case is actively syncing in a direction. One case is +stuck in an error, carrying an optional retry time. `SyncError` covers every failure a backend can report: not enabled yet, -already enabled, a schema or kit mismatch on an incoming record, a transport -failure, an encoding or decoding failure, an unreachable peer, a failed -authentication, an unsupported table, and `corruptRemoteIdentity`. That last -case exists for a specific reason explained where it is thrown, in -`CKRecordMapping.swift`: a remote record whose identifier cannot be read as -a valid row key must never be given a fabricated one, because a fabricated -identity would create a phantom row that never matches the real one on -later sync rounds. `SyncError` conforms to `Equatable` so tests can assert -exactly which error a call produced. +already enabled, a transport failure, an encoding failure, a decoding +failure, an unreachable peer, a failed authentication, an unsupported +table, and `corruptRemoteIdentity`. It also covers a schema mismatch and +a kit mismatch on an incoming record. `corruptRemoteIdentity` exists for a +specific reason. `CKRecordMapping.swift` explains that reason where the +error is thrown. A remote record +whose identifier cannot be read as a valid row key must never be given a +fabricated one. A fabricated identity would create a phantom row. That +phantom row would never match the real one on later sync rounds. +`SyncError` conforms to `Equatable` so tests can assert exactly which error +a call produced. ## SyncEngine.swift -This file provides `SyncEngine`, the protocol every backend conforms to. It -is the seam that lets application code, and the other two documents in this -package, describe "a backend" once instead of three times. - -The protocol has four methods and one property. `enable(manifest:storage:)` -must be called once before any other method; it is where a backend -establishes remote subscriptions, if any, and starts observing local -writes. `disable()` tears the same things down and must be safe to call more -than once. `push()` and `pull()` are the one-shot operations that move data -in each direction and return a `SyncReceipt`. `subscribe()` returns an -`AsyncStream` for callers that want a live feed rather than -polling. `state` is a computed, asynchronously readable property for -user-interface bindings that only need the coarse `SyncState` snapshot. - -Declaring `enable`/`push`/`pull` as `async throws` rather than synchronous -matters because every real backend performs network or disk work; declaring -the protocol this way once means every conforming backend, and every caller, -handles that uniformly. +This file provides `SyncEngine`, the protocol every backend conforms to. +It is the seam that lets application code, and the other two documents in +this package, describe "a backend" once instead of three times. + +The protocol has four methods and one property. +`enable(manifest:storage:)` must be called once before any other method. +It is where a backend establishes remote subscriptions, if any, and starts +observing local writes. `disable()` tears the same things down. It must be +safe to call more than once. `push()` and `pull()` are the one-shot +operations that move data in each direction. Each returns a `SyncReceipt`. +`subscribe()` returns an `AsyncStream` for callers that want a +live feed rather than polling. `state` is a computed, asynchronously +readable property. It suits a user interface that only needs the coarse +`SyncState` snapshot. + +`enable`, `push`, and `pull` are declared `async throws` rather than +synchronous. Every real backend performs network or disk work, so this +choice matters. Declaring the protocol this way once means every +conforming backend, and every caller, handles that work the same way. ## SyncRecord.swift This file provides the wire format for one replicated row change: `SyncRecord`, plus the smaller types it is built from. -`SyncRecord` carries a table name, an event kind, a row key, the changed -column values (or `nil` for a delete), an HLC, a schema version, and a kit -identifier. It declares its `CodingKeys` explicitly rather than relying on -Swift's automatic key derivation, because the SDK's Rust port decodes the -exact same JSON, using serde field renames that must match these strings -character for character. An automatic default would still work today, but +`SyncRecord` carries a table name, an event kind, a row key, an HLC, a +schema version, and a kit identifier. It also carries the changed column +values, or `nil` for a delete. It declares its `CodingKeys` explicitly. It does not rely on +Swift's automatic key derivation. The SDK's Rust port decodes the exact +same JSON, using serde field renames that must match these strings +character for character. An automatic default would still work today. It would silently break the two ports' agreement the moment either side's naming convention changed without the other noticing. `SyncEventKind` is a Codable, string-backed mirror of PersistenceKit's -`StorageEvent` (`insert`, `update`, `delete`). `SyncEventKind.init(from:)` +`StorageEvent`. It has three cases: `insert`, `update`, and `delete`. +`SyncEventKind.init(from:)` and `.asStorageEvent` convert both ways. A record's event kind is -duplicated here, rather than reusing `StorageEvent` directly, because -`StorageEvent` belongs to PersistenceKit and is not guaranteed to be a -stable wire format; `SyncEventKind` is ConvergenceKit's own promise about -what a byte on the wire means. +duplicated here rather than reusing `StorageEvent` directly. `StorageEvent` +belongs to PersistenceKit. It is not guaranteed to be a stable wire +format. `SyncEventKind` is ConvergenceKit's own promise about what a byte +on the wire means. -`PackedHLC` is a Codable wrapper around `SubstrateTypes.HLC`, exposing the -same three integer fields (`physicalTime`, `logicalCount`, `nodeID`) -directly, because the raw `HLC` type is not itself built for a stable wire -encoding. +`PackedHLC` is a Codable wrapper around `SubstrateTypes.HLC`. It exposes +the same three integer fields directly: `physicalTime`, `logicalCount`, +and `nodeID`. It exists because the raw `HLC` type is not itself built for +a stable wire encoding. `SyncValueMap` wraps `[String: TypedValue]`, PersistenceKit's row-value type, as `[String: SyncValueBox]`. `TypedValue` covers thirteen kinds of -column value, several of which (an HLC, a fingerprint, a nested array) do -not map onto a single native JSON type, so `SyncValueBox` gives each value -an explicit `kind` string tag alongside its `payload`. This is an -adjacently-tagged encoding, matching Rust serde's -`#[serde(tag = "kind", content = "payload")]` attribute, chosen specifically -so a Rust decoder and a Swift decoder agree on the same JSON shape without -either language's default enum encoding leaking through. +column value. Several of these do not map onto a single native JSON type. +They include an HLC, a fingerprint, and a nested array. `SyncValueBox` +gives each value an explicit `kind` string tag alongside its `payload`. +This is an adjacently-tagged encoding. It matches Rust serde's +`#[serde(tag = "kind", content = "payload")]` attribute. The SDK chose it +specifically so a Rust decoder and a Swift decoder agree on the same JSON +shape, without either language's default enum encoding leaking through. `SyncValueBox.encode(to:)` and `.init(from:)` implement that tagging by -hand. Two cases deserve attention. The `.null` case omits the `payload` key -entirely on encode, matching how Rust serde omits content for a unit -variant; a decoder that expected a `payload` key here would fail against a -Rust-produced null. The `.timestamp` case converts a `Date` to whole epoch -seconds as an `Int64`, and guards the conversion first: `Int64(_:)` traps -the program on a `Double` that is not finite or is outside `Int64`'s range, -and a corrupt or maliciously crafted inbound record could carry such a -value. The guard turns a potential crash into a normal, catchable -`EncodingError` instead. - -`FingerprintWire` is a Codable wrapper around `SubstrateTypes.Fingerprint256` -exposing its four 64-bit blocks directly, for the same reason `PackedHLC` -exists: the underlying substrate type is not itself declared Codable in a -form guaranteed stable across releases, so the wire format owns its own -copy of the shape. +hand. Two cases deserve attention. The `.null` case omits the `payload` +key entirely on encode. This matches how Rust serde omits content for a +unit variant. A decoder that expected a `payload` key here would fail +against a Rust-produced null. The `.timestamp` case converts a `Date` to +whole epoch seconds as an `Int64`. It guards the conversion first. +`Int64(_:)` traps the program on a `Double` that is not finite or is +outside `Int64`'s range. A corrupt or maliciously crafted inbound record +could carry such a value. The guard turns a potential crash into a +normal, catchable `EncodingError` instead. + +`FingerprintWire` is a Codable wrapper around +`SubstrateTypes.Fingerprint256`. It exposes that type's four 64-bit blocks +directly, for the same reason `PackedHLC` exists. The underlying substrate +type is not itself declared Codable in a form guaranteed stable across +releases, so the wire format owns its own copy of the shape. ## ConvergenceKitNone.swift -This file provides `NoSyncEngine`, the passthrough backend, and its private -`StateActor`. +This file provides `NoSyncEngine`, the passthrough backend, and its +private `StateActor`. -`NoSyncEngine` conforms to `SyncEngine` with the smallest possible bodies: +`NoSyncEngine` conforms to `SyncEngine` with the smallest possible bodies. `enable(manifest:storage:)` records that sync is on and remembers the -manifest, only for the sake of reporting a `SyncState`; `push()` and -`pull()` each check that `enable` was called and then return -`SyncReceipt.empty`; `subscribe()` returns a stream that never emits -anything, closed only when the caller cancels its own task. There is +manifest, only for the sake of reporting a `SyncState`. `push()` and +`pull()` each check that `enable` was called, then return +`SyncReceipt.empty`. `subscribe()` returns a stream that never emits +anything. It closes only when the caller cancels its own task. There is nothing here to synchronize against, so the state lives in `StateActor`, a -Swift actor, purely so that `enable`/`disable`/`isEnabled` reads and writes -are safe to call from concurrent tasks without ConvergenceKit's own locking -code. +Swift actor. It exists so that `enable`, `disable`, and `isEnabled` reads +and writes stay safe under concurrent calls. ConvergenceKit needs no +locking code of its own to guarantee that safety. This backend exists so that development builds, unit tests, and genuinely single-device deployments can enable the same `SyncManifest`-driven code -path as a production backend, without paying for a real remote connection -or writing separate no-sync branches throughout the application. +path as a production backend. They do this without paying for a real +remote connection, and without writing separate no-sync branches +throughout the application. ## CKRecordMapping.swift @@ -190,269 +198,288 @@ This file provides `CKRecordMapping`, the generic translator between a PersistenceKit row and Apple's `CKRecord`, and the two result types it produces. -The mapping is generic rather than hand-written per table because a -CloudKit record type and its fields are derived entirely from data already -known at runtime: the table's name and the manifest's `kitID`. +The mapping is generic rather than hand-written per table. A CloudKit +record type and its fields derive entirely from data already known at +runtime: the table's name and the manifest's `kitID`. `recordType(kitID:table:)` builds the CloudKit record type string as `"\(kitID)_\(table)"`. `recordID(rowKey:zone:)` builds a `CKRecord.ID` from -the row's UUID and the zone. Because both functions are pure string and -identifier arithmetic, adding a new synced table to a manifest never -requires new mapping code in this file. +the row's UUID and the zone. Both functions are pure string and identifier +arithmetic. Adding a new synced table to a manifest never requires new +mapping code in this file. `record(from:table:rowKey:hlc:schemaVersion:kitID:zone:)` converts a row's -values into a `CKRecord`, then adds three reserved fields — -`_syncHLC`, `_syncSchemaVersion`, `_syncKitID` — that carry ConvergenceKit's +values into a `CKRecord`. It then adds three reserved fields: `_syncHLC`, +`_syncSchemaVersion`, and `_syncKitID`. These fields carry ConvergenceKit's own metadata inside an otherwise ordinary CloudKit record. The private -`assign(value:to:forKey:)` helper switches over every `TypedValue` case; the -`.fingerprint` case packs the four 64-bit blocks into 32 bytes of -little-endian `Data`, and the `.array` case throws -`SyncError.encodingFailure`, because CKRecord has no native representation -for a nested list of typed values at this version. +`assign(value:to:forKey:)` helper switches over every `TypedValue` case. +The `.fingerprint` case packs the four 64-bit blocks into thirty-two bytes +of little-endian `Data`. The `.array` case throws +`SyncError.encodingFailure`, because `CKRecord` has no native +representation for a nested list of typed values at this version. `decode(_:)` reverses the process. It reads the three reserved fields back -out, and rejects a record missing `_syncHLC` or `_syncSchemaVersion` as a -decoding failure, because a record without sync metadata cannot be -attributed to any schema version or conflict-resolved correctly. It then -recovers the original table name from the `kitID_table` record type by -splitting on the first underscore. The `rowKey` recovery is the one place -this file enforces a hard rule explained in its own comment: if -`record.recordID.recordName` does not parse as a `UUID`, the function throws -`SyncError.corruptRemoteIdentity` rather than inventing a fresh UUID. A -fabricated identity would silently create a new local row every sync round, -because the fabricated UUID could never match the real one; refusing to -guess turns a slow, invisible data leak into a single visible error the -caller can log and skip. +out. It rejects a record missing `_syncHLC` or `_syncSchemaVersion` as a +decoding failure. A record without sync metadata cannot be attributed to +any schema version. It also cannot be conflict-resolved correctly. The +function then recovers the original table name from the `kitID_table` +record type, by splitting on the first underscore. The `rowKey` recovery +is the one place this file enforces a hard rule, explained in its own +comment. If `record.recordID.recordName` does not parse as a `UUID`, the +function throws `SyncError.corruptRemoteIdentity` rather than inventing a +fresh UUID. A fabricated identity would silently create a new local row +every sync round, because the fabricated UUID could never match the real +one. Refusing to guess turns a slow, invisible data leak into a single +visible error the caller can log and skip. `packed(_:)` and `unpacked(_:)` convert an `HLC` to and from a single -`Int64` for storage in a CloudKit `NSNumber` field, using a fixed bit -layout: 48 bits of physical time, 12 bits of logical count, 4 bits of node -identifier. The layout is a contract, not an implementation detail — any -change to the bit widths would make previously stored `_syncHLC` values -unreadable. +`Int64` for storage in a CloudKit `NSNumber` field. They use a fixed bit +layout: forty-eight bits of physical time, twelve bits of logical count, +and four bits of node identifier. This layout is a contract, not an +implementation detail. Any change to the bit widths would make previously +stored `_syncHLC` values unreadable. `SyncMeta` and `DecodedRecord` are the two small result types `decode(_:)` -returns: `SyncMeta` isolates the three reserved fields, and `DecodedRecord` -pairs them with the table name, row key, and clean application values (no -`_sync*` keys). Keeping the reserved fields out of `values` matters because -`values` round-trips directly into PersistenceKit's own row-value type, -which should never see ConvergenceKit's internal bookkeeping fields mixed -in with application data. +returns. `SyncMeta` isolates the three reserved fields. `DecodedRecord` +pairs them with the table name, row key, and clean application values, +with no `_sync*` keys. Keeping the reserved fields out of `values` matters, +because `values` round-trips directly into PersistenceKit's own row-value +type. That type should never see ConvergenceKit's internal bookkeeping +fields mixed in with application data. ## CloudKitSyncEngine.swift This file provides `CloudKitSyncEngine`, the CloudKit-backed `SyncEngine` conformance, and its private `CloudKitStateActor`. -`CloudKitSyncEngine.init(containerIdentifier:)` accepts an optional CloudKit -container identifier, or `nil` to resolve `CKContainer.default()` later. -Resolution is deliberately deferred to `enable()` rather than done in -`init`, specifically so a test can construct the engine without an iCloud -entitlement configured; only calling `enable()`, `push()`, or `pull()` +`CloudKitSyncEngine.init(containerIdentifier:)` accepts an optional +CloudKit container identifier, or `nil` to resolve `CKContainer.default()` +later. Resolution is deliberately deferred to `enable()` rather than done +in `init`. This lets a test construct the engine without an iCloud +entitlement configured. Only calling `enable()`, `push()`, or `pull()` actually touches CloudKit. -`CloudKitStateActor.enable(manifest:storage:)` creates the estate's CloudKit -zone (tolerating a "zone already exists" failure as expected, not an error) -and then, for every manifest table that is not `pullOnly`, subscribes to -`storage.observer.observe(table:events:)` and forwards every observed -`TableChange` into `pendingOutbound`. This is the mechanism, described in -`OVERVIEW.md`, that turns an ordinary PersistenceKit write into a queued -outbound sync record without the application calling anything sync-specific. - -`push()` drains `pendingOutbound`, and for each change looks up the table's -`SyncedTable` in the manifest, skipping tables the manifest does not -declare or has marked `pullOnly`. For an insert or update it builds a -`CKRecord` via `CKRecordMapping.record`; for a delete it builds only the +`CloudKitStateActor.enable(manifest:storage:)` creates the estate's +CloudKit zone. It tolerates a "zone already exists" failure as expected, +not as an error. Then, for every manifest table that is not `pullOnly`, it +subscribes to `storage.observer.observe(table:events:)` and forwards every +observed `TableChange` into `pendingOutbound`. `OVERVIEW.md` describes this +mechanism. It turns an ordinary PersistenceKit write into a queued outbound +sync record. The application need not call anything sync-specific for +that to happen. + +`push()` drains `pendingOutbound`. For each change, it looks up the +table's `SyncedTable` in the manifest. It skips tables the manifest does +not declare, or has marked `pullOnly`. For an insert or update it builds a +`CKRecord` via `CKRecordMapping.record`. For a delete it builds only the `CKRecord.ID` to delete. The HLC used per record prefers one the change -already carries; if the observation carried none — true of the InMemory and -SQLite storage observers at this version — `push()` mints one from an -`HLCGenerator` seeded with a random node ID in `1...0x0F`, using +already carries. If the observation carried none, true of the InMemory +and SQLite storage observers at this version, `push()` mints one from an +`HLCGenerator` seeded with a random node ID in `1...0x0F`. It uses `send(now:)` rather than reading the clock without advancing it, so two changes minted in the same push never collide on an identical timestamp. -The accumulated saves and deletes are sent together in one +The accumulated saves and deletes go out together in one `modifyRecords(saving:deleting:savePolicy:atomically:)` call. -`pull()` fetches everything new since the last stored `serverChangeToken` -via `recordZoneChanges(inZoneWith:since:)`, then applies each pulled record -through `applyInbound`. Three checks gate every record before it is -applied: the decoded `kitID` must match the manifest's, the decoded -`schemaVersion` must match, and the table must be declared in the manifest -and not marked `pushOnly`. Any of the three failing, or a decode error, is -caught, logged, and counted as a conflict rather than aborting the whole -pull — one bad record must not stop the rest of the batch from applying. -Deletions arrive as bare `CKRecord.ID` values with no record type attached, -so `pull()` cannot know which table a deleted record belonged to; it -attempts the delete against every manifest table that is not `pushOnly`, -using the manifest itself as the only scope guard. - -`applyInbound(_:syncedTable:storage:)` is the method that actually enforces -`ConflictPolicy`. `.appendOnly` always upserts, relying on the row's own key -for idempotency. `.lastWriterWinsByHLC` reads back any existing row's stored -`_syncHLC` — which arrives as `.hlc` under the InMemory backend or as a raw -packed `.int` under SQLite and Postgres, because those schemas do not -declare the column's `TypedValue` kind — and silently discards the incoming -change if its HLC is older. A change that survives the comparison is -upserted with its own `_syncHLC`, `_syncSchemaVersion`, and `_syncKitID` -merged back into the row, which is what lets the *next* inbound write read -a comparison point back out. `.remoteWins` upserts unconditionally. -`.localWins` inserts only if no row with that primary key exists yet. The -method is declared with internal, not `private`, access specifically so LWW -(last-writer-wins) conflict tests can call it directly through -`@testable import`, bypassing the real CloudKit network stack. +`pull()` fetches everything new since the last stored `serverChangeToken`, +via `recordZoneChanges(inZoneWith:since:)`. It then applies each pulled +record through `applyInbound`. Three checks gate every record before it is +applied. The decoded `kitID` must match the manifest's own `kitID`. The +decoded `schemaVersion` must match. The table must be declared in the +manifest and not marked `pushOnly`. A failure on any of the three checks, +or a decode error, gets caught and logged. The record then counts as a +conflict, rather than aborting the whole pull. One bad record must not stop the rest of the +batch from applying. Deletions arrive as bare `CKRecord.ID` values with no +record type attached. `pull()` therefore cannot know which table a +deleted record belonged to. It attempts the delete against every manifest +table that is not `pushOnly`, using the manifest itself as the only scope +guard. + +`applyInbound(_:syncedTable:storage:)` is the method that actually +enforces `ConflictPolicy`. `.appendOnly` always upserts. It relies on the +row's own key for idempotency. `.lastWriterWinsByHLC` reads back any +existing row's stored `_syncHLC`. Under the InMemory backend this arrives +as `.hlc`. Under SQLite and Postgres it arrives as a raw packed `.int`, +because those schemas do not declare the column's `TypedValue` kind. The +method discards the incoming change silently if its HLC is older. A +change that survives the comparison gets upserted. Its own `_syncHLC`, +`_syncSchemaVersion`, and `_syncKitID` merge back into the row. That +merge is what lets the next inbound write read a comparison point back +out. `.remoteWins` upserts unconditionally. `.localWins` inserts only if +no row with that primary key exists yet. The method is declared with +internal access, not `private` access. It is declared this way specifically so +last-writer-wins conflict tests can call it directly through `@testable +import`, bypassing the real CloudKit network stack. ## FederationIdentity.swift This file provides the identity types federation pairing and signing are built from: `PeerIdentity`, `LocalIdentity`, and `FederationSignature`. -`PeerIdentity` is a thin wrapper around a 32-byte Ed25519 public key, -carried for a paired peer. `LocalIdentity.init()` generates a fresh Ed25519 -keypair in memory using `Curve25519.Signing.PrivateKey()`; every +`PeerIdentity` is a thin wrapper around a thirty-two-byte Ed25519 public +key, carried for a paired peer. `LocalIdentity.init()` generates a fresh +Ed25519 keypair in memory, using `Curve25519.Signing.PrivateKey()`. Every `FederationSyncEngine` instance calls this once, at construction, so each -estate has its own persistent-for-the-session signing identity. +estate has its own signing identity for the session. `LocalIdentity.init(privateKeyBytes:)` restores an identity from bytes a -caller already has, for a host application that wants to persist and reload -the estate's key across launches; this module deliberately does not persist -identity itself, leaving that decision to the host. `sign(_:)` produces an -Ed25519 signature over arbitrary data using the stored private key. +caller already has. This suits a host application that wants to persist +and reload the estate's key across launches. This module deliberately +does not persist identity itself. It leaves that decision to the host. +`sign(_:)` produces an Ed25519 signature over arbitrary data, using the +stored private key. `FederationSignature.verify(_:of:by:)` checks a signature against a raw -32-byte public key. It returns `false`, rather than throwing, both when the -public key bytes do not form a valid Curve25519 key and when the signature -itself does not match — a receiver treats a malformed peer key exactly like -a bad signature, because both mean the message cannot be trusted. +thirty-two-byte public key. It returns `false`, rather than throwing, in +two cases. It returns `false` when the public key bytes do not form a +valid Curve25519 key. It returns `false` when the signature itself does +not match. A receiver treats a malformed peer key exactly like a bad +signature, because both mean the message cannot be trusted. ## HyperplaneFamilyExchange.swift This file provides three Codable value types for the pairing handshake: `HyperplaneFamilySpec`, `PairingProposal`, and `PairingAcceptance`. -`HyperplaneFamilySpec` carries a `seed` and a `dimension` (default 256). -Two estates that share the same seed can each independently reproduce the -same hyperplane family — the substrate machinery, defined elsewhere, that -makes their 256-bit fingerprints directly comparable. Sharing only a seed, -rather than the derived family itself, keeps the pairing message small and -keeps the two sides' math guaranteed identical, because both derive it with +`HyperplaneFamilySpec` carries a `seed` and a `dimension`, which defaults +to two hundred fifty-six. Two estates that share the same seed can each +independently reproduce the same hyperplane family. That family is the +substrate machinery, defined elsewhere, that makes their 256-bit +fingerprints directly comparable. Sharing only a seed, rather than the +derived family itself, keeps the pairing message small. It also keeps the +two sides' math guaranteed identical, because both sides derive it with the same deterministic procedure from the same seed. -`PairingProposal` and `PairingAcceptance` model a proposer offering a family -spec plus a nonce (a value used once, to prevent a replay of an old -proposal), and an accepter countersigning that proposal. This file defines -only the shapes; as `FederationSyncEngine.swift` documents at its own -`pair(with:via:family:)` method, the shipped v1.0 pairing path does not yet -negotiate or sign these two structs — the caller supplies an -already-agreed `HyperplaneFamilySpec` directly. These types exist ahead of -that wiring so the wire shape is fixed and ready. +`PairingProposal` and `PairingAcceptance` model a handshake between two +roles. A proposer offers a family spec plus a nonce, a value used once to +prevent a replay of an old proposal. An accepter countersigns that +proposal. This file defines only the shapes. As `FederationSyncEngine.swift` +documents at its own `pair(with:via:family:)` method, the shipped v1.0 +pairing path does not yet negotiate or sign these two structs. The caller +supplies an already-agreed `HyperplaneFamilySpec` directly instead. These +types exist ahead of that wiring, so the wire shape is fixed and ready. ## FederationSyncEngine.swift This file provides `FederationSyncEngine`, the Ed25519-authenticated -peer-to-peer `SyncEngine` backend; the envelope format its messages travel -in; the `Relay` transport abstraction and its in-process implementation; -and `FederationStateActor`, where the actual protocol logic lives. +peer-to-peer `SyncEngine` backend. It also provides the envelope format +its messages travel in. It provides the `Relay` transport abstraction and +its in-process implementation. It provides `FederationStateActor`, where +the actual protocol logic lives. `PayloadKind` is a one-byte discriminator identifying what an envelope -carries. `.syncRecordBatch` (`0x01`) is the only kind that exists at v1.0; -`.fieldWriteEventBatch` (`0x02`) is reserved in a code comment for a future -payload and must never be reassigned to anything else, because a receiver -on an older version would silently misinterpret a reused byte value. - -`envelopeSigningBytes(senderPublicKey:payloadKind:payload:hlc:)` builds the -exact byte sequence a `SignedEnvelope`'s signature covers: the sender's -32-byte public key, the one-byte payload kind, a four-byte little-endian -payload length, the payload itself, and the HLC's three fields, each in a -fixed little-endian layout. Signing this constructed sequence, rather than -the raw JSON payload, closes what the code calls the relabel/replay seam: if -the signature only covered the payload bytes, an attacker who obtained a -valid signed payload could re-wrap it with a different sender key or -payload kind and have it re-verified as something it never was signed to -mean. The exact byte layout matters because the Rust port's -`envelope_signing_bytes` function must reproduce these bytes exactly, or a -signature made on one port fails verification on the other. - -`SignedEnvelope` is the wire message this signature protects: the sender's -public key, the payload kind, the opaque payload, the signature, and a -batch-level HLC. The batch HLC is minted after every record's own HLC, in -`push()`, so it is strictly later — a reader can trust the envelope's HLC as -a batch watermark distinct from any single record's HLC inside it. - -`Relay` is a two-method protocol — `send(to:message:)` and -`drain(for:)` — that abstracts the transport a `SignedEnvelope` travels -over. `FederationRelay` is the only conformer shipped at v1.0: an in-process, -lock-protected dictionary from a recipient's public key to a list of pending -envelopes. It exists so the full pairing, signing, and conflict-resolution -protocol can be exercised in tests without a real network, and so a future -hosted relay (a `SyncServer`) can conform to the same protocol as a drop-in -replacement without touching `FederationSyncEngine` itself. - -`FederationStateActor` is where enable, pairing, push, and pull are -implemented, as a Swift actor so its mutable state — peers, pending -outbound changes, subscribers — is safe under concurrent access without -manual locking. `enable(manifest:storage:)` mirrors -`CloudKitStateActor.enable`: it subscribes to `storage.observer.observe` for -every push-eligible table and appends observed changes to +carries. `.syncRecordBatch` (`0x01`) is the only kind that exists at +v1.0. `.fieldWriteEventBatch` (`0x02`) is reserved in a code comment for a +future payload. It must never be reassigned to anything else, because a +receiver on an older version would silently misinterpret a reused byte +value. + +`envelopeSigningBytes(senderPublicKey:payloadKind:payload:hlc:)` builds +the exact byte sequence a `SignedEnvelope`'s signature covers. The +sequence starts with the sender's public key, thirty-two bytes long. +Next comes the one-byte payload kind, then a four-byte little-endian +payload length, then the payload itself. Last comes the HLC's three +fields, each in a fixed little-endian layout. Signing this constructed +sequence, rather than the raw JSON payload, closes what the code calls +the relabel and replay seam. If the signature only covered the +payload bytes, an attacker who obtained a valid signed payload could +re-wrap it with a different sender key or payload kind. That re-wrapped +message would then re-verify as something it never was signed to mean. +The exact byte layout matters, because the Rust port's +`envelope_signing_bytes` function must reproduce these bytes exactly. A +signature made on one port would otherwise fail verification on the +other. + +`SignedEnvelope` is the wire message this signature protects: the +sender's public key, the payload kind, the opaque payload, the signature, +and a batch-level HLC. The batch HLC is minted after every record's own +HLC, in `push()`, so it is strictly later. A reader can trust the +envelope's HLC as a batch watermark distinct from any single record's HLC +inside it. + +`Relay` is a two-method protocol, `send(to:message:)` and `drain(for:)`, +that abstracts the transport a `SignedEnvelope` travels over. +`FederationRelay` is the only conformer shipped at v1.0. It is an +in-process, lock-protected dictionary from a recipient's public key to a +list of pending envelopes. It exists so the full pairing, signing, and +conflict-resolution protocol can be exercised in tests without a real +network. It also lets a future hosted relay, a `SyncServer`, conform to +the same protocol as a drop-in replacement. That replacement need not +touch `FederationSyncEngine` itself. + +This file provides `FederationStateActor`. It implements four operations: +enable, pairing, push, and pull. It is a Swift actor. This keeps its +mutable state safe under concurrent access, without manual locking code. +That state includes peers, pending outbound changes, and subscribers. +`enable(manifest:storage:)` mirrors +`CloudKitStateActor.enable`. It subscribes to `storage.observer.observe` +for every push-eligible table and appends observed changes to `pendingOutbound`. `disable()` differs from the CloudKit actor in one -deliberate way: it cancels every observer task and then *awaits* each one's -completion before clearing state, rather than only cancelling. The code -comment explains why — without awaiting, a change already in flight inside -a cancelled task could still land in the outbox after `disable()` returned, -which would violate the guarantee that disabling sync stops capturing -writes immediately. - -`pair(with:via:family:)` registers the peer's public key, relay, and family -spec locally, then calls `acceptPeering` on the peer's own actor so the -relationship is recorded symmetrically on both sides from one call — a test -or a caller need not call `pair` twice. Pairing here is a local bookkeeping -step, not yet the signed `PairingProposal`/`PairingAcceptance` handshake -`HyperplaneFamilyExchange.swift` defines the shapes for. - -`push()` converts every pending `TableChange` into a `SyncRecord` (minting +deliberate way. It cancels every observer task, and then awaits each +one's completion, before it clears state. It does not merely cancel. The +code comment explains why. Without awaiting, a change already in flight +inside a cancelled task could still land in the outbox after `disable()` +returned. That would violate the guarantee that disabling sync stops +capturing writes immediately. + +`pair(with:via:family:)` registers the peer's public key, relay, and +family spec locally. It then calls `acceptPeering` on the peer's own +actor, so the relationship gets recorded symmetrically on both sides from +one call. A test or a caller need not call `pair` twice. Pairing here is a +local bookkeeping step. It is not yet the signed +`PairingProposal`/`PairingAcceptance` handshake `HyperplaneFamilyExchange.swift` +defines the shapes for. + +`push()` converts every pending `TableChange` into a `SyncRecord`, minting an HLC the same way `CloudKitStateActor.push()` does, and for the same -reason), JSON-encodes the batch, builds and signs an envelope, and hands it -to every paired peer's relay. `pull()` drains each peer's relay inbox and, -for every envelope, runs four checks in order before trusting its contents: -the sender's public key must match the specific peer it arrived from (not -merely any known peer — the code comment cites ADR-013, since a valid -self-signature alone does not prove the pairing handshake was completed); -the payload kind must be a known one; the signature must verify over the -reconstructed canonical bytes; and only then is the JSON payload decoded. -A record that passes all four still individually re-checks its `kitID` and -`schemaVersion` against the manifest and its table against the manifest's -declared tables, exactly like `CloudKitSyncEngine.pull()`, before reaching +reason. It JSON-encodes the batch, builds and signs an envelope, and +hands it to every paired peer's relay. `pull()` drains each peer's relay +inbox. For every envelope, it runs four checks in order before trusting +its contents. First, the sender's public key must match the specific peer +it arrived from, not merely any known peer. The code comment cites +ADR-013 here, since a valid self-signature alone does not prove the +pairing handshake was completed. Second, the payload kind must be a known +one. Third, the signature must verify over the reconstructed canonical +bytes. Only then, fourth, is the JSON payload decoded. A record that +passes all four checks still faces individual checks after that. It must +match the manifest's `kitID` and `schemaVersion`. Its table must appear +among the manifest's declared tables, exactly as in +`CloudKitSyncEngine.pull()`. Only a record that clears every check reaches `applyInbound`. `applyInbound(_:syncedTable:storage:)` implements the same four -`ConflictPolicy` arms as `CloudKitStateActor.applyInbound`, extended to -cover delete events explicitly (`CloudKitSyncEngine` receives deletes as -bare CloudKit record IDs with no policy dispatch; federation's `SyncRecord` -always carries an explicit event kind, so its delete path can apply a -policy the same way inserts and updates do). Under -`.lastWriterWinsByHLC`, a stale delete — one whose HLC is older than the -row's stored `_syncHLC` — is silently rejected rather than removing a row a -peer has since updated; `.appendOnly` rejects every remote delete -outright, because an append-only table is write-once by definition; -`.remoteWins` deletes unconditionally; `.localWins` rejects every remote -delete, leaving local state authoritative. The method is internal, not -`private`, for the same testing reason as its CloudKit counterpart. +`ConflictPolicy` arms as `CloudKitStateActor.applyInbound`. It extends +them to cover delete events explicitly. `CloudKitSyncEngine` receives +deletes as bare CloudKit record IDs, with no policy dispatch. Federation's +`SyncRecord` always carries an explicit event kind, so its delete path can +apply a policy the same way inserts and updates do. Under +`.lastWriterWinsByHLC`, a stale delete is silently rejected. A stale +delete is one whose HLC is older than the row's stored `_syncHLC`. This +avoids removing a row a peer has since updated. `.appendOnly` rejects every +remote delete outright, because an append-only table is write-once by +definition. `.remoteWins` deletes unconditionally. `.localWins` rejects +every remote delete, leaving local state authoritative. The method is +internal, not `private`, for the same testing reason as its CloudKit +counterpart. ## Rust Port and Conformance The `rust/` directory contains a second implementation of most of this -package: the core types, the `SyncRecord` wire format, the `SyncEngine` -trait, and both the None and Federation backends, in `rust/src/types.rs`, -`record.rs`, `engine.rs`, `none.rs`, `federation.rs`, and `pairing.rs`. -CloudKit is Apple-only and has no Rust equivalent by design; the Swift -package alone handles that transport, as `rust/README.md` and -`rust/src/lib.rs` both state explicitly. - -The Rust and Swift Federation backends are gated by shared behavior rather -than a byte-fixture file the way some other SDK packages are: both ports -implement observer-driven outbox population, the same four `ConflictPolicy` -arms in `apply_record`/`applyInbound`, and identical Ed25519 envelope -signing bytes in `envelope_signing_bytes`/`envelopeSigningBytes`. The Rust -test suite (`rust/tests/`) exercises wire-format round trips, last-writer- -wins ordering, and inbound event routing per conflict policy, each -described in `rust/README.md` as mirroring a named Swift test file. When you -change conflict resolution, the envelope signing byte layout, or a -`SyncRecord` field name in either port, update both and re-run both test -suites — the JSON field names and the signing byte layout are the contract -between them. +package. It covers the core types, the wire format, and the protocol +trait. It also covers the None and Federation backends, ported to Rust. +These live in `rust/src/types.rs`, `record.rs`, `engine.rs`, `none.rs`, +`federation.rs`, and `pairing.rs`. CloudKit is Apple-only and has no Rust +equivalent by design. The Swift package alone handles that transport, as +`rust/README.md` and `rust/src/lib.rs` both state explicitly. + +The Rust and Swift Federation backends are gated by shared behavior. They +are not gated by a byte-fixture file, the way some other SDK packages +are. Both ports implement observer-driven outbox population. Both +implement the same four `ConflictPolicy` arms, in `apply_record` and +`applyInbound`. Both implement identical Ed25519 envelope signing bytes, +in `envelope_signing_bytes` and `envelopeSigningBytes`. The Rust test +suite lives in `rust/tests/`. It exercises wire-format round trips, +last-writer-wins ordering, and inbound event routing per conflict policy. +Each test is described in `rust/README.md` as mirroring a named Swift test +file. Update both ports together whenever you change conflict resolution, +the envelope signing byte layout, or a `SyncRecord` field name in either +one. Then re-run both test suites. The JSON field names and the signing +byte layout are the contract between them. diff --git a/packages/kits/ConvergenceKit/docs/OVERVIEW.md b/packages/kits/ConvergenceKit/docs/OVERVIEW.md index d16e2da..5968ded 100644 --- a/packages/kits/ConvergenceKit/docs/OVERVIEW.md +++ b/packages/kits/ConvergenceKit/docs/OVERVIEW.md @@ -29,20 +29,22 @@ sources: ## What This Library Does -ConvergenceKit copies the rows that PersistenceKit stores from one device or -one MOOTx01 estate to another. An estate is one user's complete memory store -in MOOTx01; estates can federate, which means separate estates share and -compare memories. PersistenceKit is the package that reads and writes rows on -one device. ConvergenceKit sits beside it and watches for changes, then ships -those changes somewhere else: another of the user's own devices through -Apple's CloudKit, or a different user's estate through a peer-to-peer -exchange the SDK calls federation. - -Application code never calls ConvergenceKit's sync methods directly for -ordinary reads and writes. It writes to PersistenceKit as usual. -ConvergenceKit is configured once, alongside PersistenceKit, to observe that -same storage. From that point on, replication is a side effect of normal -writes rather than a task the application must remember to perform. +ConvergenceKit copies rows between devices and between estates. PersistenceKit +is the package that stores those rows on one device. An estate is one user's +complete memory store in MOOTx01. Estates can federate. Federation means two +separate estates share and compare memories. + +ConvergenceKit sits beside PersistenceKit. It watches for changes there. It +ships each change somewhere else. The destination is one of two places. +The first is another of the user's own devices, through Apple's CloudKit. +The second is a different user's estate, through a peer-to-peer exchange. +The SDK calls this second path federation. + +Application code never calls ConvergenceKit's sync methods for ordinary reads +and writes. It writes to PersistenceKit as usual. A host configures +ConvergenceKit once, alongside PersistenceKit, to observe that same storage. +After that, replication is a side effect of normal writes. It is not a task +the application must remember to perform. ## The Problem It Solves @@ -50,114 +52,117 @@ A memory captured on one device is useless on another device until it travels there. Two hard problems stand between "captured here" and "visible there." -The first problem is conflicting writes. If the same row changes on two -devices before they synchronize, something must decide which change wins. -ConvergenceKit answers this with a per-table conflict policy declared once, -in advance, by the application. The default policy, `lastWriterWinsByHLC`, -settles conflicts using a Hybrid Logical Clock (HLC): a timestamp that -combines a device's wall-clock reading with a small counter, so that two -devices can order their changes consistently even when their clocks are not -perfectly synchronized. Three other policies exist for cases where -last-writer-wins is wrong: `appendOnly` for audit logs that must never -overwrite an entry, `localWins` for data a device should never let a peer -overwrite, and `remoteWins` for the opposite case. +The first problem is conflicting writes. A row can change on two devices +before they synchronize. Something must then decide which change wins. +ConvergenceKit answers this with a per-table conflict policy. The application +declares this policy once, in advance. The default policy is +`lastWriterWinsByHLC`. It settles conflicts with a Hybrid Logical Clock +(HLC). An HLC is a timestamp. It combines a device's wall-clock reading with +a small counter. This lets two devices order their changes consistently, +even when their clocks are not perfectly synchronized. Three other policies +cover cases where last-writer-wins is wrong. `appendOnly` suits audit logs +that must never overwrite an entry. `localWins` suits data a device should +never let a peer overwrite. `remoteWins` covers the opposite case. The second problem is trust. Federated sync crosses a perimeter between two -separate people's estates, so a receiving device must be able to prove that -an incoming batch of changes really came from the peer it paired with, and -was not altered in transit. ConvergenceKit answers this with Ed25519 -signatures. Ed25519 is a public-key signature scheme: each estate generates -a private key it never shares and a public key it hands to peers during -pairing. Every batch of changes is signed with the sender's private key -before it leaves the device, and the receiver checks the signature with the -sender's public key before applying anything. - -CloudKit sync does not need this second protection, because it stays inside -one person's Apple account, authenticated by Apple's own private database. -Federated sync crosses to a stranger's device, so it needs its own -authentication independent of any platform account system. +separate people's estates. A receiving device must be able to prove two +things about an incoming batch of changes. It must prove the batch came from +the peer it paired with. It must prove no one altered the batch in transit. +ConvergenceKit answers this with Ed25519 signatures. Ed25519 is a public-key +signature scheme. Each estate generates a private key it never shares. Each +estate also hands out a public key to peers during pairing. Every batch of +changes is signed with the sender's private key before it leaves the device. +The receiver checks that signature with the sender's public key before it +applies anything. + +CloudKit sync does not need this second protection. It stays inside one +person's Apple account. Apple's own private database already authenticates +it. Federated sync crosses to a stranger's device instead. It needs its own +authentication, independent of any platform account system. ## How It Works -ConvergenceKit defines one protocol, `SyncEngine`, and three backends that -conform to it: `NoSyncEngine`, `CloudKitSyncEngine`, and -`FederationSyncEngine`. An application picks exactly one backend per -PersistenceKit instance. All three backends expose the same four operations, -so application code that calls them does not need to know which backend is -active. - -`enable(manifest:storage:)` turns sync on. A `SyncManifest` is a declaration, -written once by the application, of which PersistenceKit tables to sync, in -which direction (`bidirectional`, `pushOnly`, or `pullOnly`), and under which -conflict policy. Enabling a backend also starts it watching PersistenceKit's -`StorageObserver` for local inserts, updates, and deletes on every table the -manifest lists as push-eligible. - -`push()` sends pending local changes outward and returns a `SyncReceipt` -summarizing what moved. `pull()` fetches pending remote changes and applies -them through PersistenceKit's own write path, which is what makes the -receiving side work correctly: an applied row is not a special sync -artifact, it is an ordinary row, so anything already watching PersistenceKit +ConvergenceKit defines one protocol, `SyncEngine`. Three backends conform to +it: `NoSyncEngine`, `CloudKitSyncEngine`, and `FederationSyncEngine`. An +application picks exactly one backend per PersistenceKit instance. All three +backends expose the same four operations. Application code that calls them +does not need to know which backend is active. + +`enable(manifest:storage:)` turns sync on. A `SyncManifest` is a +declaration. The application writes it once. It states which +PersistenceKit tables to sync. It states the direction for each table: +`bidirectional`, `pushOnly`, or `pullOnly`. It states the conflict policy +for each table. Enabling a backend also starts it watching +PersistenceKit's `StorageObserver`. It watches for local inserts, updates, +and deletes on every table the manifest lists as push-eligible. + +`push()` sends pending local changes outward. It returns a `SyncReceipt` +summarizing what moved. `pull()` fetches pending remote changes. It applies +them through PersistenceKit's own write path. This is what makes the +receiving side work correctly. An applied row is not a special sync +artifact. It is an ordinary row. Anything already watching PersistenceKit notices the change the normal way. `subscribe()` returns a live stream of -`SyncEvent` values (changes applied, pushes completed, peers connecting or -disconnecting, errors) for a user interface to display. - -Every backend that crosses a real network boundary needs a wire format for a -changed row. `SyncRecord` is that format: a table name, an event kind, a row -key, the changed column values, an HLC, a schema version, and a kit -identifier. The schema version and kit identifier exist so a receiver can -reject a record it does not understand, rather than misapply it. `SyncRecord` -is Codable with explicit coding keys, because the SDK maintains a Rust port -of large parts of the system, and both ports must produce the same JSON -field names for the same data. +`SyncEvent` values for a user interface to display. These events cover +changes applied, pushes completed, peers connecting, peers disconnecting, +and errors. + +Every backend that crosses a real network boundary needs a wire format for +a changed row. `SyncRecord` is that format. It carries a table name, an +event kind, a row key, the changed column values, an HLC, a schema +version, and a kit identifier. The schema version and kit identifier let a +receiver reject a record it does not understand. This is safer than +misapplying that record. `SyncRecord` is Codable with explicit coding +keys. The SDK maintains a Rust port of large parts of the system. Both +ports must produce the same JSON field names for the same data. ## How the Pieces Fit -Figure 1 shows the library's topology: the shared protocol and wire format -at the center, PersistenceKit on either end, and the three backends as -interchangeable implementations in between. +Figure 1 shows the library's topology. The shared protocol and wire format +sit at the center. PersistenceKit sits on either end. The three backends +appear as interchangeable implementations in between. ![Figure 1. Topology of ConvergenceKit](topology.svg) *Figure 1. Topology of ConvergenceKit. Application code writes only to PersistenceKit. Whichever backend is enabled observes those writes and ships -them outward; on the receiving side, applying an inbound change routes back +them outward. On the receiving side, applying an inbound change routes back through PersistenceKit's normal write path. Dashed regions mark the three -interchangeable backends and the external systems (CloudKit's private -database, the federation relay) each one talks to.* +interchangeable backends and the external systems each one talks to: CloudKit's +private database and the federation relay.* -`NoSyncEngine` is the simplest backend: it accepts `enable()`, then returns -empty receipts from `push()` and `pull()` forever. It exists for -single-device deployments, development, and tests, where wiring in a real -backend would add cost without adding value. +`NoSyncEngine` is the simplest backend. It accepts `enable()`. It then +returns empty receipts from `push()` and `pull()` forever. It exists for +single-device deployments, development, and tests. Wiring in a real backend +in those cases would add cost without adding value. `CloudKitSyncEngine` replicates rows through Apple's CloudKit private -database, one CloudKit zone per estate. `CKRecordMapping` is the piece that -makes this backend generic rather than hand-written per table: it converts -any PersistenceKit row into a `CKRecord` and back, driven entirely by the -table name and the manifest, so adding a new synced table never requires new -mapping code. +database, using one CloudKit zone per estate. `CKRecordMapping` is the piece +that makes this backend generic instead of hand-written per table. It +converts any PersistenceKit row into a `CKRecord`, and back again, driven +entirely by the table name and the manifest. Adding a new synced table +never requires new mapping code. `FederationSyncEngine` replicates rows to a different user's estate. Two -engines "pair" by exchanging Ed25519 public keys and a shared -`HyperplaneFamilySpec`, a parameter that lets both sides compare certain -256-bit fingerprints (short fixed-size codes computed from content) on equal -terms after pairing. Once paired, every pushed batch of `SyncRecord` values -is wrapped in a `SignedEnvelope`, signed, and handed to a `Relay`. At v1.0, -the shipped `Relay` implementation, `FederationRelay`, is in-process only, -holding pending messages in memory for peers running in the same process. -This is enough to exercise the full protocol, including signature -verification and conflict resolution, in tests; a wire transport that -reaches a peer on a different machine is future work. +engines "pair" by exchanging Ed25519 public keys. They also exchange a +shared `HyperplaneFamilySpec`. This is a parameter that lets both sides +compare certain 256-bit fingerprints on equal terms after pairing. A +fingerprint here is a short, fixed-size code computed from content. Once +paired, every pushed batch of `SyncRecord` values gets wrapped in a +`SignedEnvelope`. The engine signs that envelope and hands it to a `Relay`. +At v1.0, the shipped `Relay` implementation is `FederationRelay`. It is +in-process only. It holds pending messages in memory for peers running in +the same process. This is enough to exercise the full protocol in tests, +including signature verification and conflict resolution. A wire transport +that reaches a peer on a different machine is future work. ## What Ships in the Package -The package ships four Swift targets: `ConvergenceKit` (the protocol, the -wire format, and the shared types), `ConvergenceKitNone`, -`ConvergenceKitCloudKit`, and `ConvergenceKitFederation`. A `rust/` port -mirrors the core types, the wire format, and the None and Federation -backends; CloudKit is Apple-only and has no Rust equivalent by design, so -the Swift side always handles that transport. Every field name shared -between the two ports is pinned in code comments and exercised by tests on -both sides, because a receiver on one port must decode a record written by -the other exactly. +The package ships four Swift targets. `ConvergenceKit` holds the protocol, +the wire format, and the shared types. `ConvergenceKitNone`, +`ConvergenceKitCloudKit`, and `ConvergenceKitFederation` hold the three +backends. A `rust/` port mirrors the core types, the wire format, and the +None and Federation backends. CloudKit is Apple-only by design, so it has no +Rust equivalent. The Swift side always handles that transport. Every field +name shared between the two ports is pinned in code comments. Tests on both +sides exercise those names, because a receiver on one port must decode a +record written by the other exactly. diff --git a/packages/kits/PersistenceKit/docs/AGENT_MAP.md b/packages/kits/PersistenceKit/docs/AGENT_MAP.md index f27e7b1..28dc85b 100644 --- a/packages/kits/PersistenceKit/docs/AGENT_MAP.md +++ b/packages/kits/PersistenceKit/docs/AGENT_MAP.md @@ -109,7 +109,7 @@ sources: blob: 76499ffb70d979f0d13e8cf9e32bc38ff28ffdb5 --- -# AGENT_MAP — PersistenceKit +# AGENT_MAP: PersistenceKit PURPOSE: storage abstraction layer for MOOTx01 estates. One protocol surface (Storage = RowStore+BlobStore+AuditLog+StorageObserver), three @@ -118,10 +118,10 @@ interchangeable backends (SQLite/PostgreSQL/InMemory), decorators erasure ledger/overlay, snapshot registry+GC pin, telemetry) built on the core protocol, plus a full-estate replication module. Owns NO vector search (VectorKit owns k-NN; PersistenceKit only guarantees the -ACCOMMODATION contract — storage round-trip for vector payloads). +ACCOMMODATION contract: storage round-trip for vector payloads). DEPS: imports SubstrateTypes (HLC, Fingerprint256, AuditEvent, -AsOfCoordinate, ContentHash — defined upstream, NOT in this package), +AsOfCoordinate, ContentHash: defined upstream, NOT in this package), IntellectusLib (telemetry, zero-cost when disabled), CryptoKit (AES-GCM), Security/Keychain (Apple key store, SQLite target only), postgres-nio + NIOSSL (PostgreSQL target only), vendored SQLCipher C @@ -130,234 +130,234 @@ data (LocusKit and others up the graph); kits never import a backend target directly except at estate-construction time. Rust port in `rust/` mirrors the trait surface and backend set but is NOT conformance-fixture-gated against the Swift side (no shared byte- -identical corpus) — the cross-language contract is the protocol shape +identical corpus): the cross-language contract is the protocol shape and wire format only. ENTRY POINTS (most callers need only these): -- Storage.swift:16 `protocol Storage` — conform once per backend; exposes `.rowStore`/`.blobStore`/`.auditLog`/`.observer` -- Storage.swift:29 `Storage.open(schema:)` — bring backend up to declared schema -- Storage.swift:36 `Storage.transaction(isolation:_:)` — atomic multi-op block; ext. default isolation = .readCommitted -- RowStore.swift:33 `RowStore.insert/upsert/update/delete/query/count` — core row I/O -- EstateConfiguration.swift:8 `struct EstateConfiguration` — one value opens one Storage instance -- StorageReplicator.swift:89 `StorageReplicator.replicate(from:to:schema:)` — full-snapshot estate copy +- Storage.swift:16 `protocol Storage`: conform once per backend; exposes `.rowStore`/`.blobStore`/`.auditLog`/`.observer` +- Storage.swift:29 `Storage.open(schema:)`: bring backend up to declared schema +- Storage.swift:36 `Storage.transaction(isolation:_:)`: atomic multi-op block; ext. default isolation = .readCommitted +- RowStore.swift:33 `RowStore.insert/upsert/update/delete/query/count`: core row I/O +- EstateConfiguration.swift:8 `struct EstateConfiguration`: one value opens one Storage instance +- StorageReplicator.swift:89 `StorageReplicator.replicate(from:to:schema:)`: full-snapshot estate copy ## Symbol Table -### Core protocols — Storage.swift, RowStore.swift, BlobStore.swift, AuditLog.swift, StorageObserver.swift, StorageIntrospection.swift, Transaction.swift -- Storage.swift:16 `protocol Storage: Sendable` — rowStore/blobStore/auditLog/observer + open/close/transaction/migrate/currentSchemaVersion(for:) -- Storage.swift:58 `extension Storage.transaction(_:)` — default isolation .readCommitted +### Core protocols: Storage.swift, RowStore.swift, BlobStore.swift, AuditLog.swift, StorageObserver.swift, StorageIntrospection.swift, Transaction.swift +- Storage.swift:16 `protocol Storage: Sendable`: rowStore/blobStore/auditLog/observer + open/close/transaction/migrate/currentSchemaVersion(for:) +- Storage.swift:58 `extension Storage.transaction(_:)`: default isolation .readCommitted - RowStore.swift:8 `typealias RowKey = UUID` -- RowStore.swift:10 `struct StorageRow` — [String:TypedValue] wrapper, subscript by column name -- RowStore.swift:22 `struct RowHandle: Hashable` — (table, key) -- RowStore.swift:32 `protocol RowStore: Sendable` — insert/upsert/update/delete/query/count/querySkipCorrupt/query(...columns:)/begin·commit·rollbackTransaction (all true requirements, not ext defaults — needed for dynamic dispatch through `any RowStore`) -- RowStore.swift:72 `querySkipCorrupt(...)` — best-effort corpus scan, skips+counts StorageError.corruptStoredValue rows; NEVER use for point lookups -- RowStore.swift:96 `query(...columns:)` — column projection = no-blob read path; nil columns = full read (superset, always correct) -- RowStore.swift:194/214/236 `query/querySkipCorrupt(...asOf:)` — .present passthrough; .asOf(_) throws StorageError.featureGated("asOfQuery") until NT-L4+NT-P3 both merge -- RowStore.swift:260-266 ext defaults: beginTransaction/commitTransaction/rollbackTransaction — no-op (correct for backends w/o multi-stmt txn) +- RowStore.swift:10 `struct StorageRow`: [String:TypedValue] wrapper, subscript by column name +- RowStore.swift:22 `struct RowHandle: Hashable`: (table, key) +- RowStore.swift:32 `protocol RowStore: Sendable`: insert/upsert/update/delete/query/count/querySkipCorrupt/query(...columns:)/begin·commit·rollbackTransaction (all true requirements, not ext defaults: needed for dynamic dispatch through `any RowStore`) +- RowStore.swift:72 `querySkipCorrupt(...)`: best-effort corpus scan, skips+counts StorageError.corruptStoredValue rows; NEVER use for point lookups +- RowStore.swift:96 `query(...columns:)`: column projection = no-blob read path; nil columns = full read (superset, always correct) +- RowStore.swift:194/214/236 `query/querySkipCorrupt(...asOf:)`: .present passthrough; .asOf(_) throws StorageError.featureGated("asOfQuery") until NT-L4+NT-P3 both merge +- RowStore.swift:260-266 ext defaults: beginTransaction/commitTransaction/rollbackTransaction: no-op (correct for backends w/o multi-stmt txn) - BlobStore.swift:8 `typealias BlobKey = String` -- BlobStore.swift:10 `protocol BlobStore: Sendable` — put/get/delete/exists/size/listKeys; listKeys() order UNSPECIFIED (caller sorts) -- AuditLog.swift:23 `protocol AuditLog: Sendable` — append/appendBatch (idempotent on (eventID,hlc))/iterate(after:rowID:limit:)/eventsForRow/count -- StorageObserver.swift:26 `enum StorageEvent` — insert|update|delete -- StorageObserver.swift:32 `struct TableChange` — table/event/rowKey/values/hlc -- StorageObserver.swift:60 `enum BlobEvent` — put|delete -- StorageObserver.swift:73 `struct BlobChange` — key/event/bytes (bytes non-nil only on .put; last-write-wins on same key) -- StorageObserver.swift:111 `struct DirtyChainEvent` — changedRowId/parentNodeId/grandparentNodeId/contentHash/table — hash-on-write payload (NT-P2) -- StorageObserver.swift:140 `protocol StorageObserver: Sendable` — observe(table:events:)/observeBlobs()/observeDirtyChain(); delivery AT-LEAST-ONCE, order preserved PER-TABLE only -- StorageObserver.swift:168 ext default `observeDirtyChain()` — immediately-finished stream (back-compat for pre-hash-on-write observers) -- StorageIntrospection.swift:39 `struct StorageStats` — superset of fields across 3 backends; nil = "not measured" (see file's field/backend table); capturedAt is caller-injected (NEVER Date() inside engine) -- StorageIntrospection.swift:209 `protocol StorageIntrospection` — separate from Storage (additive capability); probe via `as? StorageIntrospection` -- Transaction.swift:9 `enum IsolationLevel` — readCommitted|repeatableRead|serializable -- Transaction.swift:15 `protocol StorageTransaction: Sendable` — rowStore/blobStore/auditLog (no nested txns, no savepoints in v1.0) +- BlobStore.swift:10 `protocol BlobStore: Sendable`: put/get/delete/exists/size/listKeys; listKeys() order UNSPECIFIED (caller sorts) +- AuditLog.swift:23 `protocol AuditLog: Sendable`: append/appendBatch (idempotent on (eventID,hlc))/iterate(after:rowID:limit:)/eventsForRow/count +- StorageObserver.swift:26 `enum StorageEvent`: insert|update|delete +- StorageObserver.swift:32 `struct TableChange`: table/event/rowKey/values/hlc +- StorageObserver.swift:60 `enum BlobEvent`: put|delete +- StorageObserver.swift:73 `struct BlobChange`: key/event/bytes (bytes non-nil only on .put; last-write-wins on same key) +- StorageObserver.swift:111 `struct DirtyChainEvent`: changedRowId/parentNodeId/grandparentNodeId/contentHash/table: hash-on-write payload (NT-P2) +- StorageObserver.swift:140 `protocol StorageObserver: Sendable`: observe(table:events:)/observeBlobs()/observeDirtyChain(); delivery AT-LEAST-ONCE, order preserved PER-TABLE only +- StorageObserver.swift:168 ext default `observeDirtyChain()`: immediately-finished stream (back-compat for pre-hash-on-write observers) +- StorageIntrospection.swift:39 `struct StorageStats`: superset of fields across 3 backends; nil = "not measured" (see file's field/backend table); capturedAt is caller-injected (NEVER Date() inside engine) +- StorageIntrospection.swift:209 `protocol StorageIntrospection`: separate from Storage (additive capability); probe via `as? StorageIntrospection` +- Transaction.swift:9 `enum IsolationLevel`: readCommitted|repeatableRead|serializable +- Transaction.swift:15 `protocol StorageTransaction: Sendable`: rowStore/blobStore/auditLog (no nested txns, no savepoints in v1.0) -### Value/type algebra — TypedValue.swift, Column.swift, Predicate.swift, Schema.swift, GeneratedColumn.swift, StorageError.swift -- TypedValue.swift:25 `enum TypedValue: Sendable, Hashable` — CLOSED 13-case wire format (null/bool/int/bitmap/float/text/blob/uuid/timestamp/json/hlc/fingerprint/array); new case = update every backend, deliberate cost -- Column.swift:9 `struct Column: Comparable` — (table,name); ordered by table then name for stable test fixtures -- Column.swift:24 `enum ColumnType` — uuid|bitmap|text|timestamp|float|int|bool|blob|json|hlc|fingerprint -- Predicate.swift:11 `indirect enum StoragePredicate` — CLOSED; logical(and/or/not/isTrue/isFalse) + comparison(eq/neq/lt/lte/gt/gte/isNull/isNotNull/in/like) + bitmap(bitmaskAll/Any/None/bitwiseEq, Int64 cols only) -- Predicate.swift:41 `StoragePredicate.all(_:)` — AND-combine w/ short-circuit: empty→isTrue, contains isFalse→isFalse -- Predicate.swift:55 `StoragePredicate.any(_:)` — OR-combine w/ short-circuit: empty→isFalse, contains isTrue→isTrue -- Predicate.swift:74 `struct OrderClause` — column + OrderDirection (default .ascending) -- Schema.swift:9 `struct SchemaDeclaration` — kitID/version/tables/indices/migrations -- Schema.swift:31 `struct TableDeclaration` — name/columns/primaryKey/uniqueConstraints/generatedColumns/appendOnly/hashable -- Schema.swift:82 `enum ColumnRole` — createdHlc|tombstonedHlc (as-of temporal filter tagging, ADR-017 §15) -- Schema.swift:90 `struct ColumnDeclaration` — name/type/nullable/defaultValue/role -- Schema.swift:128 `struct Migration` — fromVersion/toVersion/[SchemaOperation] -- Schema.swift:140 `enum SchemaOperation` — createTable/dropTable/addColumn/dropColumn/renameColumn/addIndex/dropIndex/custom(sqlite:postgresql:) +### Value/type algebra: TypedValue.swift, Column.swift, Predicate.swift, Schema.swift, GeneratedColumn.swift, StorageError.swift +- TypedValue.swift:25 `enum TypedValue: Sendable, Hashable`: CLOSED 13-case wire format (null/bool/int/bitmap/float/text/blob/uuid/timestamp/json/hlc/fingerprint/array); new case = update every backend, deliberate cost +- Column.swift:9 `struct Column: Comparable`: (table,name); ordered by table then name for stable test fixtures +- Column.swift:24 `enum ColumnType`: uuid|bitmap|text|timestamp|float|int|bool|blob|json|hlc|fingerprint +- Predicate.swift:11 `indirect enum StoragePredicate`: CLOSED; logical(and/or/not/isTrue/isFalse) + comparison(eq/neq/lt/lte/gt/gte/isNull/isNotNull/in/like) + bitmap(bitmaskAll/Any/None/bitwiseEq, Int64 cols only) +- Predicate.swift:41 `StoragePredicate.all(_:)`: AND-combine w/ short-circuit: empty→isTrue, contains isFalse→isFalse +- Predicate.swift:55 `StoragePredicate.any(_:)`: OR-combine w/ short-circuit: empty→isFalse, contains isTrue→isTrue +- Predicate.swift:74 `struct OrderClause`: column + OrderDirection (default .ascending) +- Schema.swift:9 `struct SchemaDeclaration`: kitID/version/tables/indices/migrations +- Schema.swift:31 `struct TableDeclaration`: name/columns/primaryKey/uniqueConstraints/generatedColumns/appendOnly/hashable +- Schema.swift:82 `enum ColumnRole`: createdHlc|tombstonedHlc (as-of temporal filter tagging, ADR-017 §15) +- Schema.swift:90 `struct ColumnDeclaration`: name/type/nullable/defaultValue/role +- Schema.swift:128 `struct Migration`: fromVersion/toVersion/[SchemaOperation] +- Schema.swift:140 `enum SchemaOperation`: createTable/dropTable/addColumn/dropColumn/renameColumn/addIndex/dropIndex/custom(sqlite:postgresql:) - Schema.swift:153-209 `ColumnDeclaration` static convenience ctors (.uuid/.bitmap/.text/.timestamp/.int/.float/.bool/.blob/.json/.hlc/.createdHlc/.tombstonedHlc/.fingerprint) -- Schema.swift:214-230 `TableDeclaration.createdHlcColumn/tombstonedHlcColumn/supportsAsOfFilter` — derive as-of eligibility from column roles -- GeneratedColumn.swift:43 `struct GeneratedColumn` — name/type/expression; ALWAYS STORED (PG has no VIRTUAL) -- GeneratedColumn.swift:63 `indirect enum GeneratedExpression` — column/literal/bitAnd/bitOr/bitXor/shiftRight/shiftLeft/equal/notEqual; evaluates to Int64 (bool=0/1) -- GeneratedColumn.swift:91 `renderSQL()` — shared SQLite+PG renderer; XOR emitted as (a|b)-(a&b) (SQLite lacks native XOR op) -- GeneratedColumn.swift:122 `evaluate(_:)` — InMemory direct evaluation against row dict -- GeneratedColumn.swift:150 `integerValue(_:)` — extract Int64 from int-family TypedValue; non-integer/absent → 0 (InMemory sentinel) -- StorageError.swift:5 `enum StorageError: Error, Equatable` — CLOSED; notable cases: :22 corruptStoredValue (NEVER fabricate a default — throw), :28 invalidConfiguration (fail-closed on platform-invalid config), :37 featureGated(feature:) (as-of query gate), :44 invalidIdentifier(name:) (SQL identifier injection guard, SECFIX-WS2-PK) +- Schema.swift:214-230 `TableDeclaration.createdHlcColumn/tombstonedHlcColumn/supportsAsOfFilter`: derive as-of eligibility from column roles +- GeneratedColumn.swift:43 `struct GeneratedColumn`: name/type/expression; ALWAYS STORED (PG has no VIRTUAL) +- GeneratedColumn.swift:63 `indirect enum GeneratedExpression`: column/literal/bitAnd/bitOr/bitXor/shiftRight/shiftLeft/equal/notEqual; evaluates to Int64 (bool=0/1) +- GeneratedColumn.swift:91 `renderSQL()`: shared SQLite+PG renderer; XOR emitted as (a|b)-(a&b) (SQLite lacks native XOR op) +- GeneratedColumn.swift:122 `evaluate(_:)`: InMemory direct evaluation against row dict +- GeneratedColumn.swift:150 `integerValue(_:)`: extract Int64 from int-family TypedValue; non-integer/absent → 0 (InMemory sentinel) +- StorageError.swift:5 `enum StorageError: Error, Equatable`: CLOSED; notable cases: :22 corruptStoredValue (NEVER fabricate a default: throw), :28 invalidConfiguration (fail-closed on platform-invalid config), :37 featureGated(feature:) (as-of query gate), :44 invalidIdentifier(name:) (SQL identifier injection guard, SECFIX-WS2-PK) -### Estate configuration — EstateConfiguration.swift, EstateCacheConfig.swift, EncryptionMode.swift, NovelTokenTaggerChoice.swift -- EstateConfiguration.swift:8 `struct EstateConfiguration` — estateID/backend/encryptionConfig(default .plaintext)/cacheConfig(default .disabled)/novelTokenTagger(default .hmm) -- EstateConfiguration.swift:45 `enum BackendConfiguration` — .sqlite(url:busyTimeout:) | .postgresql(connectionString:poolSize:connectionTimeout:idleTimeout:) | .inMemory -- EstateConfiguration.swift:108 `queueSibling(filename:)` — derives sibling EstateConfiguration for per-estate queue DB; .sqlite→sibling file `.`; .inMemory→sibling inMemory; .postgresql→ throws featureGated (deferred, ADR-021) -- EstateConfiguration.swift:166 `deriveQueueSiblingID(parentID:filename:)` (private) — deterministic XOR-fold, NO UUID() call, same parent+filename ⇒ same sibling ID always -- EstateCacheConfig.swift:23 `struct EstateCacheConfig` — enabled/ceilingBytes(clamped ≥0)/sensitivityThreshold(clamped ≤2 — Secret=3 NEVER cacheable) -- EstateCacheConfig.swift:53 `EstateCacheConfig.disabled` — zero-change default, cache off -- EncryptionMode.swift:20 `enum EncryptionMode` — .plaintext (Mode1) | .rowEncryption (Mode2, per-row AES-GCM) | .fullDatabase (Mode3, SQLCipher whole-file); Mode4 (DB+threshold/FedRAMP) DELIBERATELY ABSENT — not a build capability -- EncryptionMode.swift:44 `struct EstateEncryptionConfig` — mode/keyIdentifier/`package`-scoped key (SymmetricKey?, never public, never logged) -- EncryptionMode.swift:68 `init(_ mode:)` — mints fresh 256-bit key + UUID keyID for encrypting modes; nil/nil for .plaintext -- EncryptionMode.swift:88 `fullDatabase(key:)` — builds Mode3 config from caller-supplied 256-bit key (Keychain-sourced) -- EncryptionMode.swift:99 `usesRowCrypto` — true ONLY for .rowEncryption (Mode3 has no per-row seam — whole file is ciphertext) -- EncryptionMode.swift:106 `fullDatabaseKeyHex` — lowercase hex for `PRAGMA key`; NEVER logged -- NovelTokenTaggerChoice.swift:43 `enum NovelTokenTaggerChoice: Sendable, Hashable, Codable` — .hmm (default, cross-platform-deterministic, federation-safe) | .nlTagger (Apple-only, non-deterministic across OS versions, federation-INCOMPATIBLE w/o re-tag — enforcement deferred to v1.1); FIXED AT ESTATE CREATION, no change-after-creation in v1.0; mirrors identically-named PersistenceKit-independent enum in LatticeLib — bridge by switch, no shared import -- NovelTokenTaggerChoice.swift:74 `NovelTokenTaggerChoice.default` — .hmm +### Estate configuration: EstateConfiguration.swift, EstateCacheConfig.swift, EncryptionMode.swift, NovelTokenTaggerChoice.swift +- EstateConfiguration.swift:8 `struct EstateConfiguration`: estateID/backend/encryptionConfig(default .plaintext)/cacheConfig(default .disabled)/novelTokenTagger(default .hmm) +- EstateConfiguration.swift:45 `enum BackendConfiguration`: .sqlite(url:busyTimeout:) | .postgresql(connectionString:poolSize:connectionTimeout:idleTimeout:) | .inMemory +- EstateConfiguration.swift:108 `queueSibling(filename:)`: derives sibling EstateConfiguration for per-estate queue DB; .sqlite→sibling file `.`; .inMemory→sibling inMemory; .postgresql→ throws featureGated (deferred, ADR-021) +- EstateConfiguration.swift:166 `deriveQueueSiblingID(parentID:filename:)` (private): deterministic XOR-fold, NO UUID() call, same parent+filename ⇒ same sibling ID always +- EstateCacheConfig.swift:23 `struct EstateCacheConfig`: enabled/ceilingBytes(clamped ≥0)/sensitivityThreshold(clamped ≤2: Secret=3 NEVER cacheable) +- EstateCacheConfig.swift:53 `EstateCacheConfig.disabled`: zero-change default, cache off +- EncryptionMode.swift:20 `enum EncryptionMode`: .plaintext (Mode1) | .rowEncryption (Mode2, per-row AES-GCM) | .fullDatabase (Mode3, SQLCipher whole-file); Mode4 (DB+threshold/FedRAMP) DELIBERATELY ABSENT: not a build capability +- EncryptionMode.swift:44 `struct EstateEncryptionConfig`: mode/keyIdentifier/`package`-scoped key (SymmetricKey?, never public, never logged) +- EncryptionMode.swift:68 `init(_ mode:)`: mints fresh 256-bit key + UUID keyID for encrypting modes; nil/nil for .plaintext +- EncryptionMode.swift:88 `fullDatabase(key:)`: builds Mode3 config from caller-supplied 256-bit key (Keychain-sourced) +- EncryptionMode.swift:99 `usesRowCrypto`: true ONLY for .rowEncryption (Mode3 has no per-row seam: whole file is ciphertext) +- EncryptionMode.swift:106 `fullDatabaseKeyHex`: lowercase hex for `PRAGMA key`; NEVER logged +- NovelTokenTaggerChoice.swift:43 `enum NovelTokenTaggerChoice: Sendable, Hashable, Codable`: .hmm (default, cross-platform-deterministic, federation-safe) | .nlTagger (Apple-only, non-deterministic across OS versions, federation-INCOMPATIBLE w/o re-tag: enforcement deferred to v1.1); FIXED AT ESTATE CREATION, no change-after-creation in v1.0; mirrors identically-named PersistenceKit-independent enum in LatticeLib: bridge by switch, no shared import +- NovelTokenTaggerChoice.swift:74 `NovelTokenTaggerChoice.default`: .hmm -### Row-level crypto — RowCrypto.swift -- RowCrypto.swift:61 `protocol AeadProvider` (package) — swap point for AEAD algorithm; MUST generate fresh random nonce per encrypt call, MUST return [nonce][tag][ciphertext], MUST throw (never return garbage) on auth failure -- RowCrypto.swift:81 `struct CryptoKitAeadProvider: AeadProvider` — default; AES-GCM-256; :87 nonceByteCount=12, :88 tagByteCount=16 -- RowCrypto.swift:144 `enum RowCrypto` (package) — :151 encrypt(_:key:provider:), :168 decrypt(_:key:provider:); default provider = CryptoKitAeadProvider() -- RowCrypto.swift:191 `encryptedForWrite(_:config:provider:)` (package) — no-op unless usesRowCrypto + "content" column present as .text; encrypts + stamps keyID -- RowCrypto.swift:220 `decryptedForRead(_:config:provider:)` (package) — no-op unless keyID present AND matches config.keyIdentifier (mismatched keyID ⇒ pass through as ciphertext, NEVER attempt decrypt under wrong key) -- RowCrypto.swift:249 `assertContentKeyIDInvariant(_:table:config:)` (package) — structural guard (FUP-D/E-1): throws constraintViolation if encrypting estate has .text content with empty/absent keyID (seam didn't run) +### Row-level crypto: RowCrypto.swift +- RowCrypto.swift:61 `protocol AeadProvider` (package): swap point for AEAD algorithm; MUST generate fresh random nonce per encrypt call, MUST return [nonce][tag][ciphertext], MUST throw (never return garbage) on auth failure +- RowCrypto.swift:81 `struct CryptoKitAeadProvider: AeadProvider`: default; AES-GCM-256; :87 nonceByteCount=12, :88 tagByteCount=16 +- RowCrypto.swift:144 `enum RowCrypto` (package): :151 encrypt(_:key:provider:), :168 decrypt(_:key:provider:); default provider = CryptoKitAeadProvider() +- RowCrypto.swift:191 `encryptedForWrite(_:config:provider:)` (package): no-op unless usesRowCrypto + "content" column present as .text; encrypts + stamps keyID +- RowCrypto.swift:220 `decryptedForRead(_:config:provider:)` (package): no-op unless keyID present AND matches config.keyIdentifier (mismatched keyID ⇒ pass through as ciphertext, NEVER attempt decrypt under wrong key) +- RowCrypto.swift:249 `assertContentKeyIDInvariant(_:table:config:)` (package): structural guard (FUP-D/E-1): throws constraintViolation if encrypting estate has .text content with empty/absent keyID (seam didn't run) -### Caching + hash-on-write decorators — CachingRowStore.swift, CacheInvalidator.swift, HashingRowStore.swift -- CachingRowStore.swift:46 `typealias ParentChainProvider = @Sendable (String, RowKey) -> [RowHandle]` — kit-supplied Merkle ancestor lookup -- CachingRowStore.swift:52 `final class CachingRowStore: RowStore, Sendable` — LRU hot-tier decorator; TRANSPARENCY GUARANTEE: results always identical to unwrapped backing store -- CachingRowStore.swift:164 `query(...asOf:)` — cache key = (RowHandle, AsOfCoordinate); .present entries evicted on write, .asOf(hlc) entries NEVER evicted (immutable, GC-pinned) -- CachingRowStore.swift:218 `invalidate(table:key:)` — external-write invalidation hook, called by CacheInvalidator -- CachingRowStore.swift:302 `extractKey(from:)` — cache lookups ONLY feasible for single-key `.eq(_, .uuid)` predicates; all else bypasses cache -- CachingRowStore.swift:334 `actor CacheActor` — owns entries/accessCounter/totalBytes; makes outer class Sendable w/o manual locks -- CachingRowStore.swift:421 `isAdmissible(_:)` (private, in CacheActor) — sensitivity gate: reads `provenance` bitmap bits[30:35] scale-gapped (0/16/32/48→ordinal 0-3); ordinal==3(Secret) ALWAYS rejected; absent column→admit; unparseable/unrecognized→REJECT (fail-closed) -- CachingRowStore.swift:482 `evictLRU()` (private) — O(n) min-scan by accessOrder; acceptable at cache-sized N -- CacheInvalidator.swift:34 `final class CacheInvalidator: Sendable` — StorageObserver→CachingRowStore.invalidate bridge for external writers -- CacheInvalidator.swift:52 `init(cache:observer:tables:)` — subscriptions registered before first await inside Task.detached; race note: a write issued in the SAME instant as init() may precede subscription (caller should yield if strict ordering required) -- CacheInvalidator.swift:92 `cancel()` / deinit — cancels root task ⇒ propagates to all per-table child tasks -- HashingRowStore.swift:36 `typealias ContentHashProvider = @Sendable (table:rowKey:values:) -> ContentHash` — kit-injected hash fn (PersistenceKit never imports a hash impl) +### Caching + hash-on-write decorators: CachingRowStore.swift, CacheInvalidator.swift, HashingRowStore.swift +- CachingRowStore.swift:46 `typealias ParentChainProvider = @Sendable (String, RowKey) -> [RowHandle]`: kit-supplied Merkle ancestor lookup +- CachingRowStore.swift:52 `final class CachingRowStore: RowStore, Sendable`: LRU hot-tier decorator; TRANSPARENCY GUARANTEE: results always identical to unwrapped backing store +- CachingRowStore.swift:164 `query(...asOf:)`: cache key = (RowHandle, AsOfCoordinate); .present entries evicted on write, .asOf(hlc) entries NEVER evicted (immutable, GC-pinned) +- CachingRowStore.swift:218 `invalidate(table:key:)`: external-write invalidation hook, called by CacheInvalidator +- CachingRowStore.swift:302 `extractKey(from:)`: cache lookups ONLY feasible for single-key `.eq(_, .uuid)` predicates; all else bypasses cache +- CachingRowStore.swift:334 `actor CacheActor`: owns entries/accessCounter/totalBytes; makes outer class Sendable w/o manual locks +- CachingRowStore.swift:421 `isAdmissible(_:)` (private, in CacheActor): sensitivity gate: reads `provenance` bitmap bits[30:35] scale-gapped (0/16/32/48→ordinal 0-3); ordinal==3(Secret) ALWAYS rejected; absent column→admit; unparseable/unrecognized→REJECT (fail-closed) +- CachingRowStore.swift:482 `evictLRU()` (private): O(n) min-scan by accessOrder; acceptable at cache-sized N +- CacheInvalidator.swift:34 `final class CacheInvalidator: Sendable`: StorageObserver→CachingRowStore.invalidate bridge for external writers +- CacheInvalidator.swift:52 `init(cache:observer:tables:)`: subscriptions registered before first await inside Task.detached; race note: a write issued in the SAME instant as init() may precede subscription (caller should yield if strict ordering required) +- CacheInvalidator.swift:92 `cancel()` / deinit: cancels root task ⇒ propagates to all per-table child tasks +- HashingRowStore.swift:36 `typealias ContentHashProvider = @Sendable (table:rowKey:values:) -> ContentHash`: kit-injected hash fn (PersistenceKit never imports a hash impl) - HashingRowStore.swift:49 `typealias HashParentChainProvider = @Sendable (table:rowKey:) -> (parentNodeId:grandparentNodeId:)?` -- HashingRowStore.swift:55 `struct HashOnWriteConfig` — hashableTables/hashProvider/parentChainProvider -- HashingRowStore.swift:81 `final class HashingRowStore: RowStore, @unchecked Sendable` — decorator chain position: caller → HashingRowStore → CachingRowStore → backend -- HashingRowStore.swift:295 `augmentWithHashForKnownKey(table:rowKey:mergedValues:)` (private) — UPDATE/upsert-as-update path: hashes FULL merged row (not partial SET dict) per SECFIX-WS2-PK F6; strips stale content_hash before hashing so insert vs update paths agree -- HashingRowStore.swift:330 `augmentWithHash(table:values:)` (private) — INSERT path; extracts rowKey from "id" column by convention -- HashingRowStore.swift:369 `emitDirtyChain(...)` (private) — delivers DirtyChainEvent via injected ObserverRegistryRef (nil = computed but not delivered) +- HashingRowStore.swift:55 `struct HashOnWriteConfig`: hashableTables/hashProvider/parentChainProvider +- HashingRowStore.swift:81 `final class HashingRowStore: RowStore, @unchecked Sendable`: decorator chain position: caller → HashingRowStore → CachingRowStore → backend +- HashingRowStore.swift:295 `augmentWithHashForKnownKey(table:rowKey:mergedValues:)` (private): UPDATE/upsert-as-update path: hashes FULL merged row (not partial SET dict) per SECFIX-WS2-PK F6; strips stale content_hash before hashing so insert vs update paths agree +- HashingRowStore.swift:330 `augmentWithHash(table:values:)` (private): INSERT path; extracts rowKey from "id" column by convention +- HashingRowStore.swift:369 `emitDirtyChain(...)` (private): delivers DirtyChainEvent via injected ObserverRegistryRef (nil = computed but not delivered) -### Erasure — ErasureLedger.swift, ErasureOverlay.swift -- ErasureLedger.swift:15 `struct ErasureLedgerEntry` — drawerId + erasedHlc -- ErasureLedger.swift:28 `enum ErasureLedgerTables` — .ledger = "erasure_ledger" -- ErasureLedger.swift:35 `ErasureLedgerSchema.ledgerTable` — TableDeclaration, appendOnly:true (UPDATE/DELETE throw appendOnlyViolation at storage layer) -- ErasureLedger.swift:53 `ErasureLedgerOps.recordErasure(...)` — throws duplicateKey if drawerId already erased (erase-once contract) -- ErasureLedger.swift:68 `ErasureLedgerOps.isErased(...)` — fast point-lookup, hot path for every erasure-subject read -- ErasureLedger.swift:86 `ErasureLedgerOps.lookupErasure(...)` — full entry incl. erasedHlc -- ErasureOverlay.swift:22 `struct ErasureOverlayConfig` — extractErasureId(row)->String? (nil=not erasure-subject) + contentColumns([String]) supplied by calling kit -- ErasureOverlay.swift:64 `ErasureOverlay.apply(rows:config:rowStore:)` — TWO-PHASE FAIL-CLOSED: phase1=query already ran; phase2=per-row ledger check; ledger-check THROW ⇒ row DROPPED entirely (never shown-when-uncertain) -- ErasureOverlay.swift:96 `nullContentColumns(row:columns:)` — nulls listed columns, preserves skeleton (id/hlc/timestamps) +### Erasure: ErasureLedger.swift, ErasureOverlay.swift +- ErasureLedger.swift:15 `struct ErasureLedgerEntry`: drawerId + erasedHlc +- ErasureLedger.swift:28 `enum ErasureLedgerTables`: .ledger = "erasure_ledger" +- ErasureLedger.swift:35 `ErasureLedgerSchema.ledgerTable`: TableDeclaration, appendOnly:true (UPDATE/DELETE throw appendOnlyViolation at storage layer) +- ErasureLedger.swift:53 `ErasureLedgerOps.recordErasure(...)`: throws duplicateKey if drawerId already erased (erase-once contract) +- ErasureLedger.swift:68 `ErasureLedgerOps.isErased(...)`: fast point-lookup, hot path for every erasure-subject read +- ErasureLedger.swift:86 `ErasureLedgerOps.lookupErasure(...)`: full entry incl. erasedHlc +- ErasureOverlay.swift:22 `struct ErasureOverlayConfig`: extractErasureId(row)->String? (nil=not erasure-subject) + contentColumns([String]) supplied by calling kit +- ErasureOverlay.swift:64 `ErasureOverlay.apply(rows:config:rowStore:)`: TWO-PHASE FAIL-CLOSED: phase1=query already ran; phase2=per-row ledger check; ledger-check THROW ⇒ row DROPPED entirely (never shown-when-uncertain) +- ErasureOverlay.swift:96 `nullContentColumns(row:columns:)`: nulls listed columns, preserves skeleton (id/hlc/timestamps) -### Snapshots + GC — SnapshotRegistry.swift, GCPin.swift -- SnapshotRegistry.swift:18 `struct SnapshotId: Hashable` — :24 `.mint()` UUID-backed -- SnapshotRegistry.swift:32 `struct SnapshotRecord` — snapshotId/hlc/label/createdAt -- SnapshotRegistry.swift:47 `struct SnapshotAttestation` — snapshotId/subjectKind/subjectId/merkleRoot/keyVersion?; subject semantics owned by CALLING kit, not PersistenceKit -- SnapshotRegistry.swift:73 `enum SnapshotTables` — .registry="snapshot_registry", .attestations="snapshot_attestations" -- SnapshotRegistry.swift:83/98 `SnapshotSchema.registryTable` / `.attestationsTable` — TableDeclarations -- SnapshotRegistry.swift:130 `SnapshotRegistryOps.createSnapshot(...)` — mints SnapshotId, inserts registry row + N attestation rows -- SnapshotRegistry.swift:161 `.listSnapshots(...)` — HLC ascending -- SnapshotRegistry.swift:177 `.deleteSnapshot(...)` — attestations deleted BEFORE registry row (child-first) -- SnapshotRegistry.swift:201 `.attestations(rowStore:snapshotId:)` — ordered by (subjectKind, subjectId) -- GCPin.swift:22 `GCPin.minimumRetainableHlc(rowStore:)` — MIN(hlc) across all snapshots; nil = nothing pinned = all vacuumable -- GCPin.swift:44 `GCPin.isPinned(rowStore:rowHlc:)` — rowHlc.packed >= minRetainableHlc.packed +### Snapshots + GC: SnapshotRegistry.swift, GCPin.swift +- SnapshotRegistry.swift:18 `struct SnapshotId: Hashable`: :24 `.mint()` UUID-backed +- SnapshotRegistry.swift:32 `struct SnapshotRecord`: snapshotId/hlc/label/createdAt +- SnapshotRegistry.swift:47 `struct SnapshotAttestation`: snapshotId/subjectKind/subjectId/merkleRoot/keyVersion?; subject semantics owned by CALLING kit, not PersistenceKit +- SnapshotRegistry.swift:73 `enum SnapshotTables`: .registry="snapshot_registry", .attestations="snapshot_attestations" +- SnapshotRegistry.swift:83/98 `SnapshotSchema.registryTable` / `.attestationsTable`: TableDeclarations +- SnapshotRegistry.swift:130 `SnapshotRegistryOps.createSnapshot(...)`: mints SnapshotId, inserts registry row + N attestation rows +- SnapshotRegistry.swift:161 `.listSnapshots(...)`: HLC ascending +- SnapshotRegistry.swift:177 `.deleteSnapshot(...)`: attestations deleted BEFORE registry row (child-first) +- SnapshotRegistry.swift:201 `.attestations(rowStore:snapshotId:)`: ordered by (subjectKind, subjectId) +- GCPin.swift:22 `GCPin.minimumRetainableHlc(rowStore:)`: MIN(hlc) across all snapshots; nil = nothing pinned = all vacuumable +- GCPin.swift:44 `GCPin.isPinned(rowStore:rowHlc:)`: rowHlc.packed >= minRetainableHlc.packed -### Telemetry + observer stubs — PersistenceKitTelemetry.swift, NoOpObserver.swift -- PersistenceKitTelemetry.swift:81 `reportStorageStats(_:estateID:now:)` — OFF by default; single Atomic load+branch when Intellectus.isEnabled==false (~1ns, no stats() call); emits `persistence.db.*` namespace, one metric per non-nil StorageStats field; `now` ALWAYS caller-injected (never Date() inline) -- NoOpObserver.swift:10 `final class NoOpObserver: StorageObserver, Sendable` — immediately-finished streams; used directly by PostgreSQLStorage (no live PG notification channel) +### Telemetry + observer stubs: PersistenceKitTelemetry.swift, NoOpObserver.swift +- PersistenceKitTelemetry.swift:81 `reportStorageStats(_:estateID:now:)`: OFF by default; single Atomic load+branch when Intellectus.isEnabled==false (~1ns, no stats() call); emits `persistence.db.*` namespace, one metric per non-nil StorageStats field; `now` ALWAYS caller-injected (never Date() inline) +- NoOpObserver.swift:10 `final class NoOpObserver: StorageObserver, Sendable`: immediately-finished streams; used directly by PostgreSQLStorage (no live PG notification channel) ### Backend: PersistenceKitInMemory -- InMemoryStorage.swift:24 `final class InMemoryStorage: Storage, Sendable` — no persistence across process runs; always effectively-serializable isolation (full state snapshot per txn) -- InMemoryStorage.swift:78 `transaction(isolation:_:)` — mutates LIVE actor state directly (NOT a detached-copy-then-replace) — replace-on-commit would silently drop concurrent bare inserts landing between snapshot and replace (real incident: 5-10% lost QueueKit sends under burst); rollback restores pre-txn snapshot -- InMemoryStorage.swift:102/110 `beginNotificationBuffering()` / `commitNotifications()` — buffer during txn, flush only on successful commit (SECFIX-WS2-PK F2: rolled-back writes must never reach observers) -- InMemoryStorage.swift:144 `actor InMemoryStateActor` — sole owner of InMemoryState; :462 `materializeGenerated(_:_:)` static — computes GeneratedColumn values identically to SQL backends' STORED columns -- InMemoryStorage.swift:388 `queryRows(...)` — predicate/order/pagination applied to FULL row first, projection applied LAST (matches SQL ORDER-BY-on-unselected-column semantics) -- InMemoryStorage.swift:635 `HLC.packed` (extension) — canonical bit-pack: physical<<16 | logical<<4 | node -- InMemoryRowStore.swift:19 `final class InMemoryRowStore: RowStore, Sendable` — thin forward to state actor; column-projection kept for cross-backend StorageRow shape parity, not for a real transfer saving -- InMemoryBlobStore.swift:6 `final class InMemoryBlobStore: BlobStore, Sendable` — forwards to actor blob dict -- InMemoryAuditLog.swift:19 `final class InMemoryAuditLog: AuditLog, Sendable` — forwards to actor; dedup on (eventID,hlc) enforced in actor -- InMemoryObserver.swift:15 `final class ObserverRegistry: @unchecked Sendable` — NSLock-based (NOT actor) so `register` is SYNCHRONOUS — subscription recorded before observe() returns, no race window for immediately-following write; mirrors Rust ObserverHub::subscribe -- InMemoryObserver.swift:104 `notify(_:)` — matching subs snapshotted UNDER lock, yielded OUTSIDE lock (avoids deadlock vs. continuation onTermination which also locks) -- PredicateEvaluator.swift:6 `enum PredicateEvaluator` — in-memory StoragePredicate interpreter (only backend evaluating the tree directly rather than compiling to a query string) -- PredicateEvaluator.swift:61 `likeMatch(_:pattern:)` — SQL LIKE→NSRegularExpression (%→.*, _→.) -- PredicateEvaluator.swift:80 `enum TypedValueComparator` — :81 `compare(_:_:)` shared ordering; .null sorts first; .hlc compared via .packed (single total order) +- InMemoryStorage.swift:24 `final class InMemoryStorage: Storage, Sendable`: no persistence across process runs; always effectively-serializable isolation (full state snapshot per txn) +- InMemoryStorage.swift:78 `transaction(isolation:_:)`: mutates LIVE actor state directly (NOT a detached-copy-then-replace): replace-on-commit would silently drop concurrent bare inserts landing between snapshot and replace (real incident: 5-10% lost QueueKit sends under burst); rollback restores pre-txn snapshot +- InMemoryStorage.swift:102/110 `beginNotificationBuffering()` / `commitNotifications()`: buffer during txn, flush only on successful commit (SECFIX-WS2-PK F2: rolled-back writes must never reach observers) +- InMemoryStorage.swift:144 `actor InMemoryStateActor`: sole owner of InMemoryState; :462 `materializeGenerated(_:_:)` static: computes GeneratedColumn values identically to SQL backends' STORED columns +- InMemoryStorage.swift:388 `queryRows(...)`: predicate/order/pagination applied to FULL row first, projection applied LAST (matches SQL ORDER-BY-on-unselected-column semantics) +- InMemoryStorage.swift:635 `HLC.packed` (extension): canonical bit-pack: physical<<16 | logical<<4 | node +- InMemoryRowStore.swift:19 `final class InMemoryRowStore: RowStore, Sendable`: thin forward to state actor; column-projection kept for cross-backend StorageRow shape parity, not for a real transfer saving +- InMemoryBlobStore.swift:6 `final class InMemoryBlobStore: BlobStore, Sendable`: forwards to actor blob dict +- InMemoryAuditLog.swift:19 `final class InMemoryAuditLog: AuditLog, Sendable`: forwards to actor; dedup on (eventID,hlc) enforced in actor +- InMemoryObserver.swift:15 `final class ObserverRegistry: @unchecked Sendable`: NSLock-based (NOT actor) so `register` is SYNCHRONOUS: subscription recorded before observe() returns, no race window for immediately-following write; mirrors Rust ObserverHub::subscribe +- InMemoryObserver.swift:104 `notify(_:)`: matching subs snapshotted UNDER lock, yielded OUTSIDE lock (avoids deadlock vs. continuation onTermination which also locks) +- PredicateEvaluator.swift:6 `enum PredicateEvaluator`: in-memory StoragePredicate interpreter (only backend evaluating the tree directly rather than compiling to a query string) +- PredicateEvaluator.swift:61 `likeMatch(_:pattern:)`: SQL LIKE→NSRegularExpression (%→.*, _→.) +- PredicateEvaluator.swift:80 `enum TypedValueComparator`: :81 `compare(_:_:)` shared ordering; .null sorts first; .hlc compared via .packed (single total order) ### Backend: PersistenceKitSQLite -- SQLiteStorage.swift:22 `final class SQLiteStorage: Storage, Sendable` — one connection per estate -- SQLiteStorage.swift:110 `actor SQLiteBackend` — owns connection + inTransaction flag + pendingBlobNotifications (buffered during txn, SECFIX-WS2-PK F3) -- SQLiteStorage.swift:411 `insertRow` / :461 `upsertRow` / :495 `updateRows` / :535 `deleteRows` / :559 `queryRows` — ALL validate every table/column identifier via `validateSQLIdentifier` before interpolation (SECFIX-WS2-PK F9); insertRow+queryRows run RowCrypto encrypt/decrypt seam -- SQLiteStorage.swift:656 `queryRowsSkipCorrupt(...)` — cursor-level per-row skip+log on corruptStoredValue; other errors re-thrown (systemic) -- SQLiteStorage.swift:782 `storageStats(now:)` — PRAGMA page_size/page_count/freelist_count; WAL frame count derived from `-wal` FILE SIZE (NOT PRAGMA wal_checkpoint — can SQLITE_LOCKED even same-actor) -- SQLiteStorage.swift:917 `readColumn(...)` — type-tolerant decode (valid-but-coerced value passes through) vs. parse-failure (unparseable UUID/timestamp TEXT) → THROWS corruptStoredValue, never fabricates -- SQLiteConnection.swift:24 `final class SQLiteConnection: @unchecked Sendable` — thin sqlite3 C API wrapper -- SQLiteConnection.swift:29 `init(url:busyTimeout:keyHex:)` — ORDER MATTERS: symlink-refusal check (CAND-052, lstat semantics) → sqlite3_open_v2 → PRAGMA key (Mode3, MUST be first stmt) → Apple Data Protection → WAL/synchronous/busy_timeout/foreign_keys pragmas -- SQLiteConnection.swift:162 `final class SQLiteStatement` — prepared-stmt wrapper; :180 `bind(_:at:)` TypedValue→sqlite3_bind_*; :239 `step()` -- SQLiteConnection.swift:318 `enum ISO8601` — :356 `string(from:)` clamps Date to RFC-3339 range [0001,9999] before formatting (logs warning on clamp, never emits unparseable-back string); :392 `date(from:)` tries :412 `fastParseCanonicalUTC` FIRST (allocation-free, handles the exact canonical shape this kit writes — Merkle rollup re-decode was ~80% of CPU via ICU formatter before this fast path), falls back to ISO8601DateFormatter for anything else -- SQLiteStores.swift:22 `final class SQLiteRowStore: RowStore, Sendable` — forwards to SQLiteBackend; :75 `querySkipCorrupt` overrides protocol default with real cursor-level skip -- SQLiteStores.swift:90/102/119 `SQLiteBlobStore` / `SQLiteAuditLog` / `SQLiteTransaction` — thin forwards -- SQLiteSchema.swift:9 `enum SQLiteSchema` — :12 `nativeType(_:)` ColumnType→SQLite storage class (.uuid/.timestamp→TEXT, .hlc→INTEGER packed, .fingerprint→BLOB); :29 `createTable(_:)`; :68 `appendOnlyTriggers(_:)` emits BEFORE UPDATE/DELETE RAISE(ABORT) trigger pair +- SQLiteStorage.swift:22 `final class SQLiteStorage: Storage, Sendable`: one connection per estate +- SQLiteStorage.swift:110 `actor SQLiteBackend`: owns connection + inTransaction flag + pendingBlobNotifications (buffered during txn, SECFIX-WS2-PK F3) +- SQLiteStorage.swift:411 `insertRow` / :461 `upsertRow` / :495 `updateRows` / :535 `deleteRows` / :559 `queryRows`: ALL validate every table/column identifier via `validateSQLIdentifier` before interpolation (SECFIX-WS2-PK F9); insertRow+queryRows run RowCrypto encrypt/decrypt seam +- SQLiteStorage.swift:656 `queryRowsSkipCorrupt(...)`: cursor-level per-row skip+log on corruptStoredValue; other errors re-thrown (systemic) +- SQLiteStorage.swift:782 `storageStats(now:)`: PRAGMA page_size/page_count/freelist_count; WAL frame count derived from `-wal` FILE SIZE (NOT PRAGMA wal_checkpoint: can SQLITE_LOCKED even same-actor) +- SQLiteStorage.swift:917 `readColumn(...)`: type-tolerant decode (valid-but-coerced value passes through) vs. parse-failure (unparseable UUID/timestamp TEXT) → THROWS corruptStoredValue, never fabricates +- SQLiteConnection.swift:24 `final class SQLiteConnection: @unchecked Sendable`: thin sqlite3 C API wrapper +- SQLiteConnection.swift:29 `init(url:busyTimeout:keyHex:)`: ORDER MATTERS: symlink-refusal check (CAND-052, lstat semantics) → sqlite3_open_v2 → PRAGMA key (Mode3, MUST be first stmt) → Apple Data Protection → WAL/synchronous/busy_timeout/foreign_keys pragmas +- SQLiteConnection.swift:162 `final class SQLiteStatement`: prepared-stmt wrapper; :180 `bind(_:at:)` TypedValue→sqlite3_bind_*; :239 `step()` +- SQLiteConnection.swift:318 `enum ISO8601`: :356 `string(from:)` clamps Date to RFC-3339 range [0001,9999] before formatting (logs warning on clamp, never emits unparseable-back string); :392 `date(from:)` tries :412 `fastParseCanonicalUTC` FIRST (allocation-free, handles the exact canonical shape this kit writes: Merkle rollup re-decode was ~80% of CPU via ICU formatter before this fast path), falls back to ISO8601DateFormatter for anything else +- SQLiteStores.swift:22 `final class SQLiteRowStore: RowStore, Sendable`: forwards to SQLiteBackend; :75 `querySkipCorrupt` overrides protocol default with real cursor-level skip +- SQLiteStores.swift:90/102/119 `SQLiteBlobStore` / `SQLiteAuditLog` / `SQLiteTransaction`: thin forwards +- SQLiteSchema.swift:9 `enum SQLiteSchema`: :12 `nativeType(_:)` ColumnType→SQLite storage class (.uuid/.timestamp→TEXT, .hlc→INTEGER packed, .fingerprint→BLOB); :29 `createTable(_:)`; :68 `appendOnlyTriggers(_:)` emits BEFORE UPDATE/DELETE RAISE(ABORT) trigger pair - SQLiteSchema.swift:103/132/167 internal tables: `_storagekit_migrations` (kit_id PK) / `_storagekit_audit` (event_id+hlc PK, full-precision physical_time/logical_count/node_id columns alongside lossy packed hlc) / `_storagekit_blobs` (key PK) -- SQLitePredicateCompiler.swift:27 `SQLitePredicateCompiler.compile(_:)` — throws on invalidIdentifier; every column name validated before interpolation (SECFIX-WS2-PK F7) -- SQLiteIdentifierValidator.swift:28 `validateSQLIdentifier(_:)` — `[A-Za-z_][A-Za-z0-9_]*`; SINGLE seam for SQLite module (SQLiteStorage + SQLitePredicateCompiler both call this, no forked copy) -- SQLiteObserver.swift:26 `actor SQLiteObserverRegistry` — row subs via sqlite3_update_hook (op/table/rowid ONLY, no column values); blob subs fed directly by SQLiteBackend.putBlob/deleteBlob call sites (hook cannot carry bytes) -- KeychainKeyStore.swift:36 `struct KeychainKeyStore` (Apple-only, `#if canImport(Security)`) — Mode3 key source; :72 `estateAccount(for:)` SHA-256(standardized path)→per-estate Keychain account, so co-processes agree without coordination; :82 `loadOrCreateKey()` idempotent under concurrent first-callers; :111 `deleteKey()` idempotent (missing item = success); access class `kSecAttrAccessibleAfterFirstUnlockThisDeviceOnly` (background-readable post-first-unlock, device-only, no iCloud sync) +- SQLitePredicateCompiler.swift:27 `SQLitePredicateCompiler.compile(_:)`: throws on invalidIdentifier; every column name validated before interpolation (SECFIX-WS2-PK F7) +- SQLiteIdentifierValidator.swift:28 `validateSQLIdentifier(_:)`: `[A-Za-z_][A-Za-z0-9_]*`; SINGLE seam for SQLite module (SQLiteStorage + SQLitePredicateCompiler both call this, no forked copy) +- SQLiteObserver.swift:26 `actor SQLiteObserverRegistry`: row subs via sqlite3_update_hook (op/table/rowid ONLY, no column values); blob subs fed directly by SQLiteBackend.putBlob/deleteBlob call sites (hook cannot carry bytes) +- KeychainKeyStore.swift:36 `struct KeychainKeyStore` (Apple-only, `#if canImport(Security)`): Mode3 key source; :72 `estateAccount(for:)` SHA-256(standardized path)→per-estate Keychain account, so co-processes agree without coordination; :82 `loadOrCreateKey()` idempotent under concurrent first-callers; :111 `deleteKey()` idempotent (missing item = success); access class `kSecAttrAccessibleAfterFirstUnlockThisDeviceOnly` (background-readable post-first-unlock, device-only, no iCloud sync) ### Backend: PersistenceKitPostgreSQL -- PostgreSQLStorage.swift:23 `final class PostgreSQLStorage: Storage, Sendable` — observer = NoOpObserver() (no live PG change-notification impl) +- PostgreSQLStorage.swift:23 `final class PostgreSQLStorage: Storage, Sendable`: observer = NoOpObserver() (no live PG change-notification impl) - PostgreSQLStorage.swift:47 estate isolation: search_path pinned to `pk_` schema per connection (PG analogue of SQLite one-file-per-estate); `public` stays on path for shared extensions -- PostgreSQLStorage.swift:120 `actor PostgreSQLBackend` — :129 `encryptionConfig` is `nonisolated let` (immutable+Sendable, read sync by row stores for the crypto seam) +- PostgreSQLStorage.swift:120 `actor PostgreSQLBackend`: :129 `encryptionConfig` is `nonisolated let` (immutable+Sendable, read sync by row stores for the crypto seam) - PostgreSQLStorage.swift:206/333 per-kit schema version stored in `_storagekit_meta` under composite key `"schema_version:"`; global max under plain key `"schema_version"` -- PostgreSQLStorage.swift:246 `storageStats(now:)` — pg_database_size / pg_stat_database (blks_hit,blks_read,xact_commit,xact_rollback,deadlocks) / pg_locks⋈pg_database (granted=false ⇒ lockContention) -- PostgreSQLPool.swift:12 `actor PostgreSQLPool` — fixed-size; :38 `acquire()` reuse→open-new(if under size)→CheckedContinuation-wait w/ timeout→poolExhausted -- PostgreSQLPool.swift:101 `openConnection()` — every new conn: CREATE SCHEMA IF NOT EXISTS + SET search_path, closes conn on setup failure (never hands back half-configured conn) -- PostgreSQLPool.swift:168 `parseTLSMode(host:)` — `ARIA_MCP_POSTGRES_TLS` env var: disable|require|(absent/unrecognized→prefer, incl. loopback — explicit opt-out required for plaintext) -- PostgreSQLConnection.swift:23 `PostgresConnection` ext — :25 `executeSimple` / :33 `executeParameterized` wrap postgres-nio errors as StorageError.backendError -- PostgreSQLConnection.swift:45 `makeBindings(_:)` — TypedValue→PostgresBindings; .fingerprint serialized as 32 raw bytes fixed block order -- PostgreSQLConnection.swift:103/116 `decodeRow(_:columns:)` / `decodeCell(_:type:)` — decode failure → `.null` (NOT a throw — PG wire protocol already enforces column types at a lower level than SQLite affinity) -- PostgreSQLPredicateCompiler.swift:27 `.compile(_:)` — `$1,$2,...` positional params; :89/:102 bitmask cases reference prior bindings by FINAL numeric position (bindings.count-1 / bindings.count) -- PostgreSQLSchema.swift:8 `enum PostgreSQLSchemaEmitter` — :47 `typeSQL(_:)` (.uuid→UUID native, .timestamp→TIMESTAMPTZ, .json→JSONB); :83 `appendOnlyFunctionSQL` ONE shared plpgsql trigger fn (CREATE OR REPLACE, idempotent) for ALL append-only tables; :97 `appendOnlyTriggerStatements` DROP-then-CREATE per table (PG has no CREATE TRIGGER IF NOT EXISTS) -- PostgreSQLStores.swift:25 `final class PostgreSQLRowStore: RowStore, Sendable` — :34 `withConnection` routes to txn.connection if inside a PostgreSQLTransactionContext, else pool.acquire/release -- PostgreSQLStores.swift:240 `renderPredicate(_:startIndex:bindings:)` (private) — renumbers `$N` placeholders in REVERSE order (avoid $10 clobbered by naive forward $1 replace) -- PostgreSQLStores.swift:258/352 `PostgreSQLBlobStore` / `PostgreSQLAuditLog` — lazily CREATE TABLE IF NOT EXISTS own backing tables on first use (not in caller's SchemaDeclaration) -- PostgreSQLStores.swift:509 `decodeAuditEvent(_:)` — required fields `try` (throw on failure); optional before-state fields `try?` (NULL = valid "no prior state") -- PostgreSQLIdentifierValidator.swift:26 `validatePSQLIdentifier(_:)` — identical rule to SQLite's validator; independent copy (3 total incl. Rust `validate_sql_identifier`) — SECFIX-WS2-PK F7/F9/F10, one seam PER MODULE +- PostgreSQLStorage.swift:246 `storageStats(now:)`: pg_database_size / pg_stat_database (blks_hit,blks_read,xact_commit,xact_rollback,deadlocks) / pg_locks⋈pg_database (granted=false ⇒ lockContention) +- PostgreSQLPool.swift:12 `actor PostgreSQLPool`: fixed-size; :38 `acquire()` reuse→open-new(if under size)→CheckedContinuation-wait w/ timeout→poolExhausted +- PostgreSQLPool.swift:101 `openConnection()`: every new conn: CREATE SCHEMA IF NOT EXISTS + SET search_path, closes conn on setup failure (never hands back half-configured conn) +- PostgreSQLPool.swift:168 `parseTLSMode(host:)`: `ARIA_MCP_POSTGRES_TLS` env var: disable|require|(absent/unrecognized→prefer, incl. loopback: explicit opt-out required for plaintext) +- PostgreSQLConnection.swift:23 `PostgresConnection` ext: :25 `executeSimple` / :33 `executeParameterized` wrap postgres-nio errors as StorageError.backendError +- PostgreSQLConnection.swift:45 `makeBindings(_:)`: TypedValue→PostgresBindings; .fingerprint serialized as 32 raw bytes fixed block order +- PostgreSQLConnection.swift:103/116 `decodeRow(_:columns:)` / `decodeCell(_:type:)`: decode failure → `.null` (NOT a throw: PG wire protocol already enforces column types at a lower level than SQLite affinity) +- PostgreSQLPredicateCompiler.swift:27 `.compile(_:)`: `$1,$2,...` positional params; :89/:102 bitmask cases reference prior bindings by FINAL numeric position (bindings.count-1 / bindings.count) +- PostgreSQLSchema.swift:8 `enum PostgreSQLSchemaEmitter`: :47 `typeSQL(_:)` (.uuid→UUID native, .timestamp→TIMESTAMPTZ, .json→JSONB); :83 `appendOnlyFunctionSQL` ONE shared plpgsql trigger fn (CREATE OR REPLACE, idempotent) for ALL append-only tables; :97 `appendOnlyTriggerStatements` DROP-then-CREATE per table (PG has no CREATE TRIGGER IF NOT EXISTS) +- PostgreSQLStores.swift:25 `final class PostgreSQLRowStore: RowStore, Sendable`: :34 `withConnection` routes to txn.connection if inside a PostgreSQLTransactionContext, else pool.acquire/release +- PostgreSQLStores.swift:240 `renderPredicate(_:startIndex:bindings:)` (private): renumbers `$N` placeholders in REVERSE order (avoid $10 clobbered by naive forward $1 replace) +- PostgreSQLStores.swift:258/352 `PostgreSQLBlobStore` / `PostgreSQLAuditLog`: lazily CREATE TABLE IF NOT EXISTS own backing tables on first use (not in caller's SchemaDeclaration) +- PostgreSQLStores.swift:509 `decodeAuditEvent(_:)`: required fields `try` (throw on failure); optional before-state fields `try?` (NULL = valid "no prior state") +- PostgreSQLIdentifierValidator.swift:26 `validatePSQLIdentifier(_:)`: identical rule to SQLite's validator; independent copy (3 total incl. Rust `validate_sql_identifier`): SECFIX-WS2-PK F7/F9/F10, one seam PER MODULE ### PersistenceKitReplication -- ReplicationTypes.swift:24 `struct ReplicationCursor: Equatable` — hlcWatermark/rowsWritten/auditEventsWritten/blobsWritten -- ReplicationTypes.swift:51 `enum ReplicationError` — schemaMismatch(sourceVersion:destinationVersion:sourceKitID:destinationKitID:) | storageFailure(detail:) -- StorageReplicator.swift:69 `enum StorageReplicator` — :89 `replicate(from:to:schema:)` core; :103 `flush(from:into:schema:)` / :115 `hydrate(into:from:schema:)` direction-named wrappers -- StorageReplicator.swift:125 `replicateFull(...)` — 3 steps: (1) per-kit schema-version GATE (throws schemaMismatch, NO auto-migrate), (2) snapshotSource() reads ALL rows/audit/blobs into Sendable payload BEFORE opening dest txn, (3) one `.serializable` dest transaction does upsert(conflictColumns:=table.primaryKey — NOT RowHandle.key) -- StorageReplicator.swift:216 blob-delete propagation (SECFIX-WS2-PK F5) — destination keys ABSENT from source snapshot are deleted; additive-only copy would otherwise leak orphaned blobs forever -- StorageReplicator.swift:256 `snapshotSource(...)` — generated columns FILTERED OUT of each row before staging (destination recomputes); a blob present in listKeys() but absent on get() (TOCTOU) → THROWS storageFailure, never silently dropped -- IncrementalReplicationSession.swift:56 `actor BlobDirtySet` — :61 `accumulate` last-write-wins per key; :67 `drain()` sorted-by-key, atomic clear; :79 `restore(_:)` UNION semantics (does not overwrite newer dirt) -- IncrementalReplicationSession.swift:104 `struct DirtyKey: Comparable` — (table, pkEncoded) sorted string key; ORDER IS LOAD-BEARING — deterministic replay across independent sync runs -- IncrementalReplicationSession.swift:135 `actor DirtySet` — :163 `accumulate(_:)` extracts PK cols from TableChange.values; missing PK col or nil values dict ⇒ LOG+SKIP (never crash); :195 `drain()` sorted; :213 `restore(_:)` union (Set.insert no-op if present) -- IncrementalReplicationSession.swift:242 `final class IncrementalReplicationSession: Sendable` — :283 `start(source:schema:)` subscribes one Task per schema table + one blob Task; :266 deinit cancels all tasks -- IncrementalReplicationSession.swift:361 `sync(from:to:fromCursor:)` — schema-version gate (same as full snapshot) → drain both dirty sets → re-scan each dirty row from SOURCE (never trust the original notification's payload for rows — row may have changed again) → missing row ⇒ destination DELETE, present ⇒ UPSERT → audit events filtered to HLC > fromCursor.hlcWatermark → ONE serializable dest transaction -- IncrementalReplicationSession.swift:396-406 RETRY-PRESERVATION: on ANY error after drain, `dirtySet.restore` + `blobDirtySet.restore` are AWAITED SYNCHRONOUSLY inside the catch — NEVER fire-and-forget/detached — a detached restore could race an immediate caller retry and reproduce the lost-keys bug nondeterministically -- IncrementalReplicationSession.swift:521 `snapshotDirtyRows(...)` (private) — blob ops need NO source re-read (change event already carries payload); row ops DO re-scan (predicate = exact PK match, `pkPredicate(for:table:)` at :617) +- ReplicationTypes.swift:24 `struct ReplicationCursor: Equatable`: hlcWatermark/rowsWritten/auditEventsWritten/blobsWritten +- ReplicationTypes.swift:51 `enum ReplicationError`: schemaMismatch(sourceVersion:destinationVersion:sourceKitID:destinationKitID:) | storageFailure(detail:) +- StorageReplicator.swift:69 `enum StorageReplicator`: :89 `replicate(from:to:schema:)` core; :103 `flush(from:into:schema:)` / :115 `hydrate(into:from:schema:)` direction-named wrappers +- StorageReplicator.swift:125 `replicateFull(...)`: 3 steps: (1) per-kit schema-version GATE (throws schemaMismatch, NO auto-migrate), (2) snapshotSource() reads ALL rows/audit/blobs into Sendable payload BEFORE opening dest txn, (3) one `.serializable` dest transaction does upsert(conflictColumns:=table.primaryKey: NOT RowHandle.key) +- StorageReplicator.swift:216 blob-delete propagation (SECFIX-WS2-PK F5): destination keys ABSENT from source snapshot are deleted; additive-only copy would otherwise leak orphaned blobs forever +- StorageReplicator.swift:256 `snapshotSource(...)`: generated columns FILTERED OUT of each row before staging (destination recomputes); a blob present in listKeys() but absent on get() (TOCTOU) → THROWS storageFailure, never silently dropped +- IncrementalReplicationSession.swift:56 `actor BlobDirtySet`: :61 `accumulate` last-write-wins per key; :67 `drain()` sorted-by-key, atomic clear; :79 `restore(_:)` UNION semantics (does not overwrite newer dirt) +- IncrementalReplicationSession.swift:104 `struct DirtyKey: Comparable`: (table, pkEncoded) sorted string key; ORDER IS LOAD-BEARING: deterministic replay across independent sync runs +- IncrementalReplicationSession.swift:135 `actor DirtySet`: :163 `accumulate(_:)` extracts PK cols from TableChange.values; missing PK col or nil values dict ⇒ LOG+SKIP (never crash); :195 `drain()` sorted; :213 `restore(_:)` union (Set.insert no-op if present) +- IncrementalReplicationSession.swift:242 `final class IncrementalReplicationSession: Sendable`: :283 `start(source:schema:)` subscribes one Task per schema table + one blob Task; :266 deinit cancels all tasks +- IncrementalReplicationSession.swift:361 `sync(from:to:fromCursor:)`: schema-version gate (same as full snapshot) → drain both dirty sets → re-scan each dirty row from SOURCE (never trust the original notification's payload for rows: row may have changed again) → missing row ⇒ destination DELETE, present ⇒ UPSERT → audit events filtered to HLC > fromCursor.hlcWatermark → ONE serializable dest transaction +- IncrementalReplicationSession.swift:396-406 RETRY-PRESERVATION: on ANY error after drain, `dirtySet.restore` + `blobDirtySet.restore` are AWAITED SYNCHRONOUSLY inside the catch: NEVER fire-and-forget/detached: a detached restore could race an immediate caller retry and reproduce the lost-keys bug nondeterministically +- IncrementalReplicationSession.swift:521 `snapshotDirtyRows(...)` (private): blob ops need NO source re-read (change event already carries payload); row ops DO re-scan (predicate = exact PK match, `pkPredicate(for:table:)` at :617) ## INVARIANTS / GOTCHAS -- TypedValue is a CLOSED 13-case enum. Adding a case requires updating every backend (SQLite/PostgreSQL/InMemory/Rust) in the same change — this cost is deliberate, not an oversight. +- TypedValue is a CLOSED 13-case enum. Adding a case requires updating every backend (SQLite/PostgreSQL/InMemory/Rust) in the same change: this cost is deliberate, not an oversight. - StoragePredicate is CLOSED. Backends treat it as opaque data except when compiling; never special-case another backend's predicate shape. -- SQL IDENTIFIER INJECTION GUARD (SECFIX-WS2-PK F1/F7/F9/F10): every caller-supplied table/column name reaching a dynamically-built SQL string is validated against `[A-Za-z_][A-Za-z0-9_]*` FIRST — three independent seams (SQLiteIdentifierValidator.validateSQLIdentifier, PostgreSQLIdentifierValidator.validatePSQLIdentifier, Rust validate_sql_identifier), one per module, never forked within a module. Double-quoting alone is NOT sufficient — a name containing `"` escapes the delimiter. -- corruptStoredValue is THROWN, never papered over. A stored UUID/timestamp string that fails to parse must surface as an error — substituting a fabricated UUID or epoch-0 date is a silent data-identity lie. querySkipCorrupt / queryRowsSkipCorrupt exist specifically so a CORPUS scan (not a point lookup) can skip-and-log instead of aborting wholesale. -- as-of temporal query (.asOf(hlc)) is GATED OFF by default — every default implementation throws StorageError.featureGated("asOfQuery"). Do not implement/ungate a backend override until NT-L4 (lineage-wide expunge) and NT-P3 (erasure overlay) have both merged — ungating early risks resurfacing erased content through a stale snapshot read. -- CachingRowStore TRANSPARENCY GUARANTEE: every read must return exactly what the unwrapped backing store would. Cache changes latency only, NEVER correctness. Sensitivity gate fails CLOSED — unparseable/unrecognized provenance bit patterns are rejected from the cache, never admitted "just in case." +- SQL IDENTIFIER INJECTION GUARD (SECFIX-WS2-PK F1/F7/F9/F10): every caller-supplied table/column name reaching a dynamically-built SQL string is validated against `[A-Za-z_][A-Za-z0-9_]*` FIRST: three independent seams (SQLiteIdentifierValidator.validateSQLIdentifier, PostgreSQLIdentifierValidator.validatePSQLIdentifier, Rust validate_sql_identifier), one per module, never forked within a module. Double-quoting alone is NOT sufficient: a name containing `"` escapes the delimiter. +- corruptStoredValue is THROWN, never papered over. A stored UUID/timestamp string that fails to parse must surface as an error: substituting a fabricated UUID or epoch-0 date is a silent data-identity lie. querySkipCorrupt / queryRowsSkipCorrupt exist specifically so a CORPUS scan (not a point lookup) can skip-and-log instead of aborting wholesale. +- as-of temporal query (.asOf(hlc)) is GATED OFF by default: every default implementation throws StorageError.featureGated("asOfQuery"). Do not implement/ungate a backend override until NT-L4 (lineage-wide expunge) and NT-P3 (erasure overlay) have both merged: ungating early risks resurfacing erased content through a stale snapshot read. +- CachingRowStore TRANSPARENCY GUARANTEE: every read must return exactly what the unwrapped backing store would. Cache changes latency only, NEVER correctness. Sensitivity gate fails CLOSED: unparseable/unrecognized provenance bit patterns are rejected from the cache, never admitted "just in case." - Sensitivity ordinal 3 (Secret) is NEVER cacheable, enforced twice: EstateCacheConfig clamps sensitivityThreshold to ≤2 at construction, AND CacheActor.isAdmissible re-checks ordinal==3 unconditionally as defense-in-depth. - Decorator chain order matters: caller → HashingRowStore → CachingRowStore → backend. HashingRowStore must see the row BEFORE caching decides whether to admit it. -- HashingRowStore hashes the FULL committed row on UPDATE/upsert-as-update (pre-read + merge), never the partial SET-columns dict — a partial hash would diverge from what INSERT computes for identical final data (SECFIX-WS2-PK F6). -- RowCrypto: a keyID mismatch on read means "sealed under a key this estate does not hold" — pass the row through as still-ciphertext, do NOT attempt decrypt (would only surface as an AES-GCM auth failure). Mode 3 (fullDatabase) makes the per-row seam a permanent no-op (usesRowCrypto==false) — the whole file is already ciphertext. +- HashingRowStore hashes the FULL committed row on UPDATE/upsert-as-update (pre-read + merge), never the partial SET-columns dict: a partial hash would diverge from what INSERT computes for identical final data (SECFIX-WS2-PK F6). +- RowCrypto: a keyID mismatch on read means "sealed under a key this estate does not hold": pass the row through as still-ciphertext, do NOT attempt decrypt (would only surface as an AES-GCM auth failure). Mode 3 (fullDatabase) makes the per-row seam a permanent no-op (usesRowCrypto==false): the whole file is already ciphertext. - assertContentKeyIDInvariant is a STRUCTURAL guard, not just documentation: any encrypting-estate write path that reaches a backend with `.text` content and no/empty keyID throws constraintViolation. If you add a new content-bearing write path (a new upsert variant, a migration path), it must run the encryption seam BEFORE this guard sees it, or the guard will correctly reject the write. -- ErasureOverlay is FAIL-CLOSED: a ledger-check failure drops the row from the result entirely rather than showing possibly-erased content. This is NOT the same as failing open — silence (row absence) is the safe failure mode here. -- erasure_ledger table is appendOnly:true, enforced by the STORAGE LAYER (SQLite triggers / PostgreSQL trigger function / InMemory RowStore.update·delete throw) — not just by convention in ErasureLedgerOps. -- InMemoryStorage.transaction mutates LIVE actor state (not a detached copy replaced on commit) — this is a scar from a real lost-write incident (5-10% of QueueKit sends under burst). Do not "simplify" this back to copy-then-replace. +- ErasureOverlay is FAIL-CLOSED: a ledger-check failure drops the row from the result entirely rather than showing possibly-erased content. This is NOT the same as failing open: silence (row absence) is the safe failure mode here. +- erasure_ledger table is appendOnly:true, enforced by the STORAGE LAYER (SQLite triggers / PostgreSQL trigger function / InMemory RowStore.update·delete throw): not just by convention in ErasureLedgerOps. +- InMemoryStorage.transaction mutates LIVE actor state (not a detached copy replaced on commit): this is a scar from a real lost-write incident (5-10% of QueueKit sends under burst). Do not "simplify" this back to copy-then-replace. - Notification buffering (SECFIX-WS2-PK F2/F3) applies during EVERY transaction on EVERY backend (InMemory row+blob, SQLite blob): observers must never see a notification for a write that was later rolled back. Buffer before the block runs, flush only after commit, discard on rollback. -- IncrementalReplicationSession retry-preservation restore MUST be awaited synchronously inside the failure catch block, never dispatched to a detached Task — a detached restore races an immediate caller retry and can reproduce a lost-dirty-keys bug nondeterministically. +- IncrementalReplicationSession retry-preservation restore MUST be awaited synchronously inside the failure catch block, never dispatched to a detached Task: a detached restore races an immediate caller retry and can reproduce a lost-dirty-keys bug nondeterministically. - Full-snapshot replication (StorageReplicator) upserts on `table.primaryKey` as the conflict column, NEVER on RowHandle.key (which is a fresh random UUID per call and would insert duplicates on every re-run). -- Replication requires exact per-kit schema-version equality between source and destination (srcVersion == dstVersion == schema.version). No auto-migration, ever — a version mismatch is ReplicationError.schemaMismatch, not a best-effort attempt. -- Generated columns are ALWAYS STORED (never VIRTUAL) — PostgreSQL has no VIRTUAL form, and this keeps SQLite/PostgreSQL/InMemory semantically identical. Generated column names are filtered OUT of any row payload staged for upsert during replication (destination recomputes them; writing a value into a GENERATED column errors on both SQL backends). -- HLC.packed truncates physicalTime to 40 bits — lossy for far-future timestamps. SQLite's `_storagekit_audit` table therefore stores physical_time/logical_count/node_id as separate full-precision columns alongside the packed value; decode from those three columns, not from unpacking the packed column, or a cold rebuild's lastHLC can silently disagree with a snapshot's. -- ISO8601 fast-path parser (`fastParseCanonicalUTC`) exists purely for CPU cost: the general ISO8601DateFormatter was measured at ~80% of total CPU during large imports (Merkle rollup re-decodes every row's timestamp on every insert). The fast path recognizes ONLY the exact canonical shape this kit writes and returns nil (triggering formatter fallback) for anything else — never weaken its exactness to "handle more cases," that reintroduces the cost it exists to avoid. -- NovelTokenTaggerChoice is FIXED AT ESTATE CREATION in v1.0 — no change-after-creation path exists yet. `.nlTagger` estates cannot safely federate with `.hmm` estates without full re-tagging; this is not yet enforced automatically (v1.1 work) — caller discipline is the only guard today. +- Replication requires exact per-kit schema-version equality between source and destination (srcVersion == dstVersion == schema.version). No auto-migration, ever: a version mismatch is ReplicationError.schemaMismatch, not a best-effort attempt. +- Generated columns are ALWAYS STORED (never VIRTUAL): PostgreSQL has no VIRTUAL form, and this keeps SQLite/PostgreSQL/InMemory semantically identical. Generated column names are filtered OUT of any row payload staged for upsert during replication (destination recomputes them; writing a value into a GENERATED column errors on both SQL backends). +- HLC.packed truncates physicalTime to 40 bits: lossy for far-future timestamps. SQLite's `_storagekit_audit` table therefore stores physical_time/logical_count/node_id as separate full-precision columns alongside the packed value; decode from those three columns, not from unpacking the packed column, or a cold rebuild's lastHLC can silently disagree with a snapshot's. +- ISO8601 fast-path parser (`fastParseCanonicalUTC`) exists purely for CPU cost: the general ISO8601DateFormatter was measured at ~80% of total CPU during large imports (Merkle rollup re-decodes every row's timestamp on every insert). The fast path recognizes ONLY the exact canonical shape this kit writes and returns nil (triggering formatter fallback) for anything else: never weaken its exactness to "handle more cases," that reintroduces the cost it exists to avoid. +- NovelTokenTaggerChoice is FIXED AT ESTATE CREATION in v1.0: no change-after-creation path exists yet. `.nlTagger` estates cannot safely federate with `.hmm` estates without full re-tagging; this is not yet enforced automatically (v1.1 work): caller discipline is the only guard today. - Pinned/deterministic constants: sensitivityThreshold clamp ≤2, Secret ordinal ==3 always excluded, SQL identifier pattern `[A-Za-z_][A-Za-z0-9_]*`, ciphertext envelope layout [12-byte nonce][16-byte tag][ciphertext], KeychainKeyStore.keyByteCount=32, RFC-3339 round-trip range [0001-01-01, 9999-12-31]. -- PersistenceKit imports NO vector-search engine and must never grow one — the ACCOMMODATION contract (vector payload round-trip, bulk hydration, count, delete via the general RowStore/BlobStore surfaces) is the full extent of this package's obligation toward VectorKit's workload (ADR-008). +- PersistenceKit imports NO vector-search engine and must never grow one: the ACCOMMODATION contract (vector payload round-trip, bulk hydration, count, delete via the general RowStore/BlobStore surfaces) is the full extent of this package's obligation toward VectorKit's workload (ADR-008). diff --git a/packages/kits/PersistenceKit/docs/DETAILS.md b/packages/kits/PersistenceKit/docs/DETAILS.md index 15e2861..78e5ecd 100644 --- a/packages/kits/PersistenceKit/docs/DETAILS.md +++ b/packages/kits/PersistenceKit/docs/DETAILS.md @@ -109,1253 +109,1315 @@ sources: blob: 76499ffb70d979f0d13e8cf9e32bc38ff28ffdb5 --- -# PersistenceKit Details +# `PersistenceKit` Details This document walks through every source file in the package. Read -`OVERVIEW.md` first for the big picture. Files appear grouped by target, -in the order a reader should learn them: the core protocols and value -types first, then the decorators and cross-cutting toolkits built on top -of them, then the three backends that implement the core protocols, and -finally the replication module that operates across two backends. +`OVERVIEW.md` first for the big picture. Files appear grouped by +target, in the order a reader should learn them. The core protocols +and value types come first. The decorators and cross-cutting toolkits +built on top of them come next. The three backends that implement the +core protocols come after that. The replication module, which runs +across two backends, comes last. -## Target: PersistenceKit (core) +## Target: `PersistenceKit` (core) ### Storage.swift -This file provides the `Storage` protocol — the single entry point every -backend implements. A term of art first: a protocol in Swift is a contract -that a type promises to fulfill; any type that conforms to `Storage` can -be used anywhere `Storage` is expected, regardless of what physical -engine backs it. - -`Storage` bundles four sub-stores — `rowStore`, `blobStore`, `auditLog`, -`observer` — plus five life-cycle functions: `open(schema:)` creates files -or connections and brings the backend up to its declared schema version; -`close()` shuts it down cleanly and must be safe to call more than once; -`transaction(isolation:_:)` runs a block of work atomically, rolling back -every change if the block throws; and the two `currentSchemaVersion` -overloads report how far a backend's schema has migrated, either -globally or for one named kit. `migrate(to:)` advances the schema forward. -A protocol extension supplies a default isolation level of read-committed, -so most callers can write `storage.transaction { ... }` without naming a -level at all. +This file provides the `Storage` protocol, the single entry point +every backend implements. A term of art first. A protocol in Swift is +a contract that a type promises to fulfill. Any type that conforms to +`Storage` can be used anywhere `Storage` is expected. This holds +regardless of what physical engine backs it. + +`Storage` bundles four sub-stores: `rowStore`, `blobStore`, `auditLog`, +and `observer`. It also bundles five life-cycle functions. +`open(schema:)` creates files or connections. It also brings the +backend up to its declared schema version. `close()` shuts the backend +down cleanly. It must be safe to call more than once. +`transaction(isolation:_:)` runs a block of work as one step. It rolls +back every change if the block throws. The two `currentSchemaVersion` +overloads report how far a backend's schema has migrated. One reports +globally; the other reports for one named kit. `migrate(to:)` advances +the schema forward. A protocol extension supplies a default isolation +level of read-committed. Most callers can therefore write +`storage.transaction { ... }` without naming a level at all. ### RowStore.swift -This file provides the typed row input/output protocol, plus three small -supporting types: `RowKey` (a type alias for `UUID`), `StorageRow` (an -immutable, subscriptable wrapper around a `[String: TypedValue]` row), and -`RowHandle` (a `(table, key)` pair that identifies exactly one row). - -`RowStore` declares the operations every backend must support: `insert`, -`upsert`, `update`, `delete`, `query`, and `count`. Two further query -variants matter for performance and safety. `query(...columns:)` is the -column-projecting form: passing a specific list of column names means a -column such as a large text blob is never transferred out of storage at -all, which matters when a caller only needs a handful of small columns -from a wide table. `querySkipCorrupt(...)` is the resilience form: rather -than aborting an entire corpus scan the moment one row's stored value -fails to parse, it skips that row, counts it, and returns everything -else — the file's documentation is explicit that this method exists for -best-effort scans (like "every drawer in the estate"), never for a -single-row point lookup, where a corrupt value should always be a loud -error. +This file provides the typed row input and output protocol. It also +provides three small supporting types. `RowKey` is a type alias for +`UUID`. `StorageRow` is a fixed, subscriptable wrapper around a +`[String: TypedValue]` row. `RowHandle` is a `(table, key)` pair that +identifies exactly one row. + +`RowStore` declares the operations every backend must support: +`insert`, `upsert`, `update`, `delete`, `query`, and `count`. Two +further query variants matter for performance and safety. +`query(...columns:)` is the column-projecting form. A caller passes a +specific list of column names. A column such as a large text blob is +then never transferred out of storage at all. This matters when a +caller only needs a handful of small columns from a wide table. +`querySkipCorrupt(...)` is the resilience form. Rather than aborting an +entire corpus scan the moment one row's stored value fails to parse, +it skips that row. It counts the skipped row and returns everything +else. The file's own comment is explicit about this method's +purpose. It exists for best-effort scans, such as scanning every +drawer in the estate. It should never be used for a single-row point +lookup, where a corrupt value should always be a loud error. A protocol extension supplies default implementations for all of the -above (falling back to the plain, unprojected `query`), plus a family of -as-of temporal query overloads. As-of querying means asking what a row -looked like at a past point in time, identified by an `AsOfCoordinate` -(a type defined in the `SubstrateTypes` dependency). The default -implementation of the as-of overloads answers `.present` queries normally -and throws `StorageError.featureGated` for any `.asOf(hlc)` request — the -feature is deliberately turned off until two other in-flight pieces of -work (lineage-wide expunge and the erasure overlay) both ship, because -turning it on early could let an as-of read resurface content that was -supposed to have been erased. Finally, this file declares the write- -transaction boundary — `beginTransaction`, `commitTransaction`, -`rollbackTransaction` — as protocol requirements rather than -protocol-extension defaults. This distinction matters for a subtle -Swift reason: a protocol requirement dispatches to the concrete type even -when called through an `any RowStore` existential, while a -protocol-extension default does not always do so reliably. Because -`CachingRowStore` must forward these calls to whatever it wraps, they -have to be true requirements. +above. Each one falls back to the plain, full `query`. The +extension also supplies a family of as-of temporal query overloads. +As-of querying means asking what a row looked like at a past point in +time. That point is identified by an `AsOfCoordinate`, a type defined +in the `SubstrateTypes` dependency. The default implementation answers +`.present` queries normally. It throws `StorageError.featureGated` for +any `.asOf(hlc)` request. The feature stays off on purpose until two +other in-flight pieces of work both ship: lineage-wide expunge and the +erasure overlay. Turning it on early could let an as-of read resurface +content that was supposed to have been erased. + +Finally, this file declares the write-transaction boundary as protocol +requirements rather than protocol-extension defaults. These +requirements are `beginTransaction`, `commitTransaction`, and +`rollbackTransaction`. This distinction matters for a subtle Swift +reason. A protocol requirement dispatches to the concrete type even +when called through an `any RowStore` existential. A protocol-extension +default does not always do so reliably. `CachingRowStore` must forward +these calls to whatever it wraps. They have to be true requirements +for that forwarding to work. ### BlobStore.swift -This file provides the blob input/output protocol: `put`, `get`, +This file provides the blob input and output protocol: `put`, `get`, `delete`, `exists`, `size`, and `listKeys`. A blob is a chunk of raw -bytes identified by an arbitrary string key — typically a -content-addressed hash or a row's UUID combined with a column name. -`listKeys()` exists specifically so the replication primitive can -enumerate every blob in a backend for a full-snapshot copy; because -key order is unspecified and may differ between backends and even -between calls, any caller that needs a stable order (replication does) -sorts the returned keys itself. +bytes identified by any string key. That key is often a +content-addressed hash. It can also be a row's UUID combined with a +column name. `listKeys()` exists in particular so the replication +primitive can list every blob in a backend for a full-snapshot +copy. Key order is unspecified and may differ between backends and +even between calls. Any caller that needs a stable order, and +replication does, sorts the returned keys itself. ### AuditLog.swift This file provides the append-only history protocol. An audit log -records every change made to a row, in order, and never edits or removes -a past entry. `append` and `appendBatch` are both idempotent on the -compound key `(eventID, hlc)` — HLC stands for Hybrid Logical Clock, a -timestamp format that orders events consistently even across multiple -devices without perfectly synchronized clocks — so replaying the same -event twice (which can happen during sync) never creates a duplicate. +records every change made to a row, in order. It never edits or +removes a past entry. `append` and `appendBatch` are both idempotent on +the compound key `(eventID, hlc)`. HLC stands for Hybrid Logical Clock, +a timestamp format that orders events the same way every time across multiple +devices without perfectly matched up clocks. Replaying the same +event twice, which can happen during sync, never creates a duplicate. `iterate(after:rowID:limit:)` walks the log in HLC order and accepts a -resume cursor, and `eventsForRow(_:)` returns just one row's history for -building a point-in-time projection of that row. PersistenceKit's role -stops at durable, ordered storage of these events; enforcing the -conflict-resolution rules of a CRDT (a data structure designed so -concurrent edits always merge the same way) belongs to a higher kit. +resume cursor. `eventsForRow(_:)` returns just one row's history. A +caller uses this to build a point-in-time projection of that row. +`PersistenceKit`'s role stops at durable, ordered storage of these +events. Enforcing the conflict-resolution rules of a CRDT belongs to a +higher kit. A CRDT is a data structure designed so concurrent edits +always merge the same way. ### StorageObserver.swift -This file provides the live change-notification protocol, plus four -supporting event types. `StorageEvent` is the three kinds of row change -(`insert`, `update`, `delete`); `TableChange` bundles one such event with -its table, row key, values, and HLC. `BlobEvent` and `BlobChange` are the -blob-store analogues — a `put` event carries the written bytes so a -subscriber (specifically, the incremental replication session) can avoid -a second round-trip just to re-read what was written. `DirtyChainEvent` -is a third, more specialized notification: when a row in a table marked -hashable is written, the write also carries the row's new content hash -and its two nearest ancestors in a Merkle-style containment hierarchy — -the minimum information a downstream consumer needs to incrementally -recompute an integrity tree without rescanning it from scratch. - -`StorageObserver` itself declares three subscription methods — -`observe(table:events:)`, `observeBlobs()`, `observeDirtyChain()` — each -returning an `AsyncStream`, a Swift type for values that arrive over time. -Delivery is documented as at-least-once, and ordering is preserved within -one subscription but not promised across different tables. A protocol -extension supplies a default no-op stream for `observeDirtyChain()` so -older observer implementations, written before hash-on-write existed, -keep compiling without modification. +This file provides the live change-notification protocol. It also +provides four supporting event types. `StorageEvent` is the three +kinds of row change: `insert`, `update`, and `delete`. `TableChange` +bundles one such event with its table, row key, values, and HLC. +`BlobEvent` and `BlobChange` are the blob-store matches. A `put` +event carries the written bytes. A subscriber, in particular the +incremental replication session, can then avoid a second round-trip +just to re-read what was written. `DirtyChainEvent` is a third, more +special notification. When a row in a table marked hashable is +written, the write also carries the row's new content hash. It carries +the row's two nearest ancestors in a Merkle-style containment +hierarchy. A downstream consumer needs only this to recompute an +integrity tree bit by bit. It never has to rescan the tree from +scratch. + +`StorageObserver` itself declares three subscription methods: +`observe(table:events:)`, `observeBlobs()`, and `observeDirtyChain()`. +Each one returns an `AsyncStream`, a Swift type for values that arrive +over time. Delivery is documented as at-least-once. Ordering is +preserved within one subscription but not promised across different +tables. A protocol extension supplies a default no-op stream for +`observeDirtyChain()`. Older observer implementations, written before +hash-on-write existed, keep compiling without a change because of +this. ### StorageIntrospection.swift This file provides an optional, separate capability protocol for -reporting backend health, plus the `StorageStats` value type that carries -the numbers. `StorageIntrospection` is deliberately not merged into -`Storage` itself — it is a distinct protocol so that adding it never -breaks an existing conformer, and a caller checks for the capability with -`storage as? StorageIntrospection` rather than assuming every backend has it. +reporting backend health. It also provides the `StorageStats` value +type that carries the numbers. `StorageIntrospection` is on purpose +not merged into `Storage` itself. It is a distinct protocol so that +adding it never breaks an existing conformer. A caller checks for the +capability with `storage as? StorageIntrospection` rather than +assuming every backend has it. `StorageStats` holds a superset of fields that no single backend fully -populates: page-level statistics such as `pageSize` and `walFrameCount` -are SQLite-only; buffer cache and transaction counters such as -`cacheHitRatio` and `deadlockCount` are PostgreSQL-only; `rowCount` and -`blobCount` are InMemory-only. A backend that cannot supply a given field -sets it to `nil` rather than fabricating a zero, so a caller can tell "not -measured" apart from "measured as zero." The file's doc comment includes a -field-by-field table cross-referencing exactly which backend fills which -field — the authoritative reference for anyone adding a new statistic. +fills. Page-level statistics such as `pageSize` and +`walFrameCount` are SQLite-only. Buffer cache and transaction counters +such as `cacheHitRatio` and `deadlockCount` are PostgreSQL-only. +`rowCount` and `blobCount` are InMemory-only. A backend that cannot +supply a given field sets it to `nil` rather than fabricating a zero. +A caller can then tell "not measured" apart from "measured as zero." +The file's doc comment includes a field-by-field table. That table +cross-references exactly which backend fills which field. It is the +trusted reference for anyone adding a new statistic. ### Transaction.swift This file provides the transaction protocol and the three isolation levels a caller may request: `readCommitted`, `repeatableRead`, and `serializable`. `StorageTransaction` exposes the same three sub-stores -`Storage` does (`rowStore`, `blobStore`, `auditLog`) so that code written -inside a transaction block looks exactly like code written outside one — -the only difference is that every operation inside the block either all -commits together or all rolls back together. The file's own comment notes -that nested transactions and savepoints are out of scope for this version -of the design. +`Storage` does: `rowStore`, `blobStore`, and `auditLog`. Code written +inside a transaction block therefore looks exactly like code written +outside one. The only difference is that every operation inside the +block either all commits together or all rolls back together. The +file's own comment notes that nested transactions and savepoints are +out of scope for this version of the design. ### StorageError.swift -This file provides the single closed error type every PersistenceKit -operation can throw. Being a closed `enum` (as opposed to an open -protocol) means every possible failure is enumerated in one place, so a -caller's `switch` over a `StorageError` can be exhaustive. The cases cover -backend availability, schema and migration failures, constraint and -uniqueness violations, connection-pool exhaustion, transaction conflicts, -type mismatches, and three especially important safety cases: -`corruptStoredValue` (a stored value could not be parsed back to its -declared type — thrown instead of silently substituting a fabricated -default, because a fabricated UUID or an epoch-zero date would be a -quiet data-identity lie), `invalidConfiguration` (an `EstateConfiguration` -requests something impossible on the current platform, such as an -Apple-only tagger on a non-Apple host), and `invalidIdentifier` (a -caller-supplied SQL column or table name contains characters outside the -safe set, which is the guard that keeps a dynamically-built query from -becoming a SQL-injection vector even though the name is already -double-quoted). +This file provides the single closed error type every `PersistenceKit` +operation can throw. Being a closed `enum`, as opposed to an open +protocol, means every possible failure is listed in one place. A +caller's `switch` over a `StorageError` can therefore be exhaustive. +The cases cover backend availability, schema and migration failures, +and constraint and uniqueness violations. They also cover +connection-pool exhaustion, transaction conflicts, and type mismatches. +Three cases matter most for safety. `corruptStoredValue` fires when a +stored value could not be parsed back to its declared type. It is +thrown instead of silently using a fabricated default instead. This is because +a fabricated UUID or an epoch-zero date would be a quiet +data-identity lie. `invalidConfiguration` fires when an +`EstateConfiguration` requests something impossible on the current +platform, such as an Apple-only tagger on a non-Apple host. +`invalidIdentifier` fires when a caller-supplied SQL column or table +name contains characters outside the safe set. This is the guard that keeps a query built at run time from becoming +a SQL-injection vector. It matters even though the name is already +double-quoted. ### Column.swift -This file provides `Column`, a `(table, name)` pair used throughout the -predicate and schema types to reference one column, and `ColumnType`, the -closed set of value kinds a column can hold: `uuid`, `bitmap`, `text`, -`timestamp`, `float`, `int`, `bool`, `blob`, `json`, `hlc`, and -`fingerprint`. `Column` conforms to `Comparable` (ordered first by table, -then by name) purely so test fixtures and other code that sorts columns -gets a stable, repeatable order. +This file provides `Column`, a `(table, name)` pair used throughout +the predicate and schema types to reference one column. It also +provides `ColumnType`. This is the closed set of value kinds a column +can hold. The kinds are `uuid`, `bitmap`, `text`, `timestamp`, `float`, +`int`, `bool`, `blob`, `json`, `hlc`, and `fingerprint`. `Column` +conforms to `Comparable`, ordered first by table and then by name. +This exists purely so test fixtures and other code that sorts columns +gets a stable, steady order. ### TypedValue.swift This file provides `TypedValue`, the tagged union that carries every -value crossing the PersistenceKit boundary — the wire format of the whole -library. Every backend pattern-matches on the case and emits its own -native representation: SQLite might store a `.uuid` as a `TEXT` column, -PostgreSQL might store it as a native `UUID` column, but callers on both -sides always see the same Swift enum. The case set is deliberately -closed; the file's own comment says adding a new case is expected to -require updating every backend, and that this cost is the right price to -pay for true backend portability — a new value kind can never partially -work on only some backends. Two of the twelve cases, `.hlc` and -`.fingerprint`, exist specifically because PersistenceKit is forbidden -from reimplementing substrate math: the `HLC` and `Fingerprint256` types -themselves come from the `SubstrateTypes` dependency, and PersistenceKit -only needs a slot to carry them. +value crossing the `PersistenceKit` boundary. It is the wire format of +the whole library. Every backend pattern-matches on the case and emits +its own native representation. SQLite might store a `.uuid` as a +`TEXT` column. PostgreSQL might store it as a native `UUID` column. +Callers on both sides always see the same Swift enum. The case set is +on purpose closed. The file's own comment says adding a new case is +expected to require updating every backend. Paying this cost is worth it. It means a value works the same way +on every backend. A new value kind can never partially work on only +some backends. Two of the twelve cases, `.hlc` +and `.fingerprint`, exist for a specific reason. `PersistenceKit` is +forbidden from building again substrate math. The `HLC` and +`Fingerprint256` types themselves come from the `SubstrateTypes` +dependency. `PersistenceKit` only needs a slot to carry them. ### Predicate.swift This file provides `StoragePredicate`, the closed query-condition tree -every backend compiles to its own query language, plus `OrderClause` and -`OrderDirection` for sorting. The predicate cases fall into three -families: logical (`and`, `or`, `not`, `isTrue`, `isFalse`), comparison -(`eq`, `neq`, `lt`, `lte`, `gt`, `gte`, `isNull`, `isNotNull`, `in`, -`like`), and bitmap (`bitmaskAll`, `bitmaskAny`, `bitmaskNone`, -`bitwiseEq`, all restricted to integer-family columns). PersistenceKit -treats a predicate as opaque data except when compiling it — no backend -ever needs to special-case another backend's query shape, because the -tree is the same tree everywhere. - -The two static helpers `all(_:)` and `any(_:)` build an `and`/`or` tree -from a list of predicates with useful short-circuiting: an empty list of -conditions to AND together collapses to `isTrue` (an always-true filter, -because "no restrictions" should match everything), an empty list to OR -together collapses to `isFalse`, and a list already containing a -contradictory trivial case (`isFalse` inside an `all`, `isTrue` inside an -`any`) collapses the whole expression immediately rather than building a +every backend compiles to its own query language. It also provides +`OrderClause` and `OrderDirection` for sorting. The predicate cases +fall into three families. The logical family covers `and`, `or`, +`not`, `isTrue`, and `isFalse`. The comparison family covers `eq`, +`neq`, `lt`, `lte`, `gt`, `gte`, `isNull`, `isNotNull`, `in`, and +`like`. The bitmap family covers `bitmaskAll`, `bitmaskAny`, +`bitmaskNone`, and `bitwiseEq`. All four apply only to integer-family +columns. `PersistenceKit` treats a predicate as opaque data except when +compiling it. No backend ever needs to special-case another backend's +query shape. The tree is the same tree everywhere, so there is nothing +to special-case. + +The two static helpers `all(_:)` and `any(_:)` build an `and`/`or` +tree from a list of predicates. Both include useful short-circuiting. +An empty list of conditions to AND together collapses to `isTrue`, an +always-true filter, because "no restrictions" should match everything. +An empty list to OR together collapses to `isFalse`. A list that +already contains a conflicting trivial case collapses the whole +expression right away. This happens when `isFalse` sits inside an +`all`, or when `isTrue` sits inside an `any`, rather than building a needlessly large tree. ### GeneratedColumn.swift -This file provides first-class computed columns: a column whose value is -derived from other columns in the same row by a small, structured integer -expression, rather than stored directly by a caller. `GeneratedColumn` -names the column, its result type, and its `GeneratedExpression`. -`GeneratedExpression` is a closed, recursive enum covering exactly the -bit-field algebra a bitmap-heavy schema needs: reading another column -(`.column`), a constant (`.literal`), bitwise AND/OR/XOR, left and right -shift by a fixed amount, and equality/inequality tests that evaluate to 1 -or 0. The file's own comment explains why this is a structured expression -tree and not a raw SQL string: a SQL string generated column would push -backend-specific syntax into a schema declaration, and the in-memory -backend has no SQL engine to evaluate it against. A structured expression -has exactly one meaning, so SQLite and PostgreSQL render it to native -`GENERATED ALWAYS AS (...) STORED` DDL and the in-memory backend evaluates -it directly — three faithful realizations of one description, with no -escape hatch for one backend to interpret it differently from another. - -`renderSQL()` turns an expression into a SQL fragment shared by SQLite and -PostgreSQL, because both use the same bitwise-operator syntax and -double-quoted identifiers; the one exception is bitwise XOR, which SQLite -lacks as an operator, so both backends instead render it as the -algebraically equivalent `(a | b) - (a & b)`. `evaluate(_:)` computes the -same result directly against an in-memory row dictionary, for the +This file provides first-class computed columns. A computed column +holds a value derived from other columns in the same row. A small, +structured integer expression performs that derivation, rather than a +caller storing the value directly. `GeneratedColumn` names the column, +its result type, and its `GeneratedExpression`. `GeneratedExpression` +is a closed, recursive enum. It covers exactly the bit-field algebra a +bitmap-heavy schema needs: reading another column (`.column`), a +constant (`.literal`). Three cases cover bitwise AND, OR, and XOR. +Two more cases cover a left shift and a right shift by a fixed amount. +A final pair of cases test equality and inequality, each one giving +back one or zero. The file's own comment explains why this is a structured +expression tree rather than a raw SQL string. A SQL string generated +column would push backend-specific syntax into a schema declaration. +The in-memory backend also has no SQL engine to evaluate it against. A +structured expression has exactly one meaning. SQLite and PostgreSQL +render it to native `GENERATED ALWAYS AS (...) STORED` DDL. The +in-memory backend works out it directly. These are three faithful +versions of one description. No backend gets an escape hatch to +interpret it differently from another. + +`renderSQL()` turns an expression into a SQL fragment shared by SQLite +and PostgreSQL. Both use the same bitwise-operator syntax and +double-quoted identifiers. The one exception is bitwise XOR, which +SQLite lacks as an operator. Both backends instead render it as `(a | b) - (a & b)`, which gives +the same value using only AND and OR. `evaluate(_:)` computes +the same result directly against an in-memory row dictionary, for the in-memory backend. `integerValue(_:)` extracts an `Int64` from any -integer-family `TypedValue` (an `.int`, `.bitmap`, `.bool`, or `.hlc`), -returning zero for anything else or for an absent column, which is the -in-memory evaluator's sentinel for "no meaningful value here." +integer-family `TypedValue`: an `.int`, `.bitmap`, `.bool`, or `.hlc`. +It returns zero for anything else or for an absent column. That zero +is the in-memory evaluator's sentinel for "no meaningful value here." ### EstateConfiguration.swift This file provides `EstateConfiguration`, the one value that fully -describes how to open one estate's storage, and `BackendConfiguration`, -the closed choice of physical engine (`sqlite`, `postgresql`, `inMemory`) -with its connection parameters. `EstateConfiguration` also carries the -estate's encryption mode, cache configuration, and novel-token tagger -choice, each defaulted so that existing code that does not mention these -newer fields keeps compiling and keeps behaving exactly as it did before -they existed — a plaintext, uncached, HMM-tagged estate is what a caller -gets by simply not asking for anything else. - -`queueSibling(filename:)` is a more specialized function: it derives a -second `EstateConfiguration` pointing at a companion database file that -sits beside the estate's own file — used, for example, so a work-queue -kit gets its own small database without needing separate key -distribution or a separate configuration path. For a SQLite estate at -`/.sqlite`, asking for the sibling `"queue.sqlite"` produces -`/.queue.sqlite` — the estate's own file stem is folded into -the sibling name so two different estates in the same directory can never -collide. For an in-memory estate, the sibling is also in-memory (both -ephemeral, which is the correct pairing for tests). For a PostgreSQL -estate, the function throws `StorageError.featureGated`: the design -deliberately defers a PostgreSQL queue sibling until later, and it fails -loudly rather than silently returning a half-working configuration. - -The private helper `deriveQueueSiblingID(parentID:filename:)` computes the -sibling's estate ID deterministically — by XOR-folding the filename's -UTF-8 bytes into a 16-byte tag and then XOR-ing that tag with the parent -UUID's raw bytes — so the same parent estate and the same filename always -produce the same sibling ID, with no call to `UUID()` anywhere on this -path. Determinism here matters because every process that opens the same -estate must agree on the same sibling ID without any coordination. +describes how to open one estate's storage. It also provides +`BackendConfiguration`. This is the closed choice of physical engine: +`sqlite`, `postgresql`, or `inMemory`. It also carries that engine's +connection settings. `EstateConfiguration` itself carries the estate's +encryption mode, its cache setup, and its novel-token tagger choice. +Each of these has a default. Existing code that never mentions these +newer fields keeps compiling and keeps behaving exactly as it did +before they existed. A caller who asks for nothing else gets a +plaintext, uncached, HMM-tagged estate. + +`queueSibling(filename:)` is a more special function. It derives a +second `EstateConfiguration` that points at a companion database file +sitting beside the estate's own file. A work-queue kit, for example, +gets its own small database this way. It needs no separate key +distribution and no separate configuration path. Take a SQLite estate +at `/.sqlite`. Asking for the sibling `"queue.sqlite"` +produces `/.queue.sqlite`. The estate's own file stem is +folded into the sibling name. Two different estates in the same +directory can therefore never collide. For an in-memory estate, the +sibling is also in-memory. Both are ephemeral, which is the correct +pairing for tests. For a PostgreSQL estate, the function throws +`StorageError.featureGated`. The design puts off a PostgreSQL queue +sibling on purpose until later. It fails loudly rather than quietly +returning a half-working configuration. + +The private helper `deriveQueueSiblingID(parentID:filename:)` computes +the sibling's estate ID in a fixed way. It XOR-folds the filename's +UTF-8 bytes into a sixteen-byte tag. It then XORs that tag with the +parent UUID's raw bytes. The same parent estate and the same filename +therefore always produce the same sibling ID. No call to `UUID()` +happens anywhere on this path. This fixed behavior matters. Every +process that opens the same estate must agree on the same sibling ID +without talking to any other process. ### EstateCacheConfig.swift -This file provides `EstateCacheConfig`, the small value type that turns -row caching on or off for one estate and bounds how much a cache is -allowed to hold and how sensitive the content it holds may be. -`ceilingBytes` is clamped to be non-negative at construction, and -`sensitivityThreshold` is clamped to at most 2 — because sensitivity level -3, "Secret" in the ARIA adjective scale this kit shares with the rest of -the system, must never be cached under any configuration, and clamping at -construction enforces that invariant without relying on every caller to -remember the numeric boundary. `EstateCacheConfig.disabled` is the -zero-change default: an estate that never mentions caching gets exactly -the pre-caching behavior. +This file provides `EstateCacheConfig`, the small value type that +turns row caching on or off for one estate. It also bounds how much a +cache is allowed to hold. It bounds how sensitive the content it holds +may be. `ceilingBytes` is clamped to be non-negative at construction. +`sensitivityThreshold` is clamped to at most two. Sensitivity level +three is called "Secret" in the ARIA adjective scale this kit shares +with the rest of the system. Secret content must never be cached, no +matter how the estate is set up. Clamping at construction enforces +that rule. No caller needs to remember the numeric boundary on its +own. `EstateCacheConfig.disabled` is the zero-change default. An estate +that never mentions caching gets exactly the pre-caching behavior. ### EncryptionMode.swift -This file provides the three at-rest encryption modes and the -configuration value that carries one of them per estate. -`EncryptionMode.plaintext` stores content verbatim — encryption, if any, +This file provides the three at-rest encryption modes. It also +provides the configuration value that carries one of them per estate. +`EncryptionMode.plaintext` stores content as-is. Encryption, if any, happens only at a sharing boundary elsewhere in the system. -`.rowEncryption` (Mode 2) encrypts one column's content per row under a -key the row's own `keyID` column names. `.fullDatabase` (Mode 3) encrypts -the entire SQLite file, schema included, via SQLCipher's `PRAGMA key` at -the connection layer, which makes the per-row seam a no-op for that mode -because the whole file is already ciphertext on disk. A fourth mode, -database-plus-threshold encryption for a FedRAMP-tier deployment, is -deliberately absent from this enum — the file's comment stresses that the -capability does not exist in this build at all, so adding it later is a -conscious, reviewed act rather than an accidental unlock. +`.rowEncryption`, Mode 2, encrypts one column's content per row under a +key named by the row's own `keyID` column. `.fullDatabase` is Mode 3. +It encrypts the entire SQLite file, including its schema, through +SQLCipher's `PRAGMA key` at the connection layer. This makes the per-row seam a +no-op for that mode. The whole file is already ciphertext on disk, so +there is nothing left for the per-row seam to do. A fourth mode is on +purpose left out of this enum: database-plus-threshold encryption for +a FedRAMP-tier deployment. The file's comment stresses that this build +does not have that capability at all. Adding it later is a conscious, +reviewed act, not an accident. `EstateEncryptionConfig` bundles the mode with a key identifier and the actual `SymmetricKey`. Its stored `key` property is `package`-scoped -rather than `public`: the SQLite backend, in a sibling module of the same -Swift package, needs it, but nothing outside the package should ever see -it. The convenience initializer `init(_ mode:)` mints a fresh, full-entropy -256-bit key and a UUID key identifier for either encrypting mode, and -mints neither for plaintext. `usesRowCrypto` is `true` only for Mode 2 — -Mode 3 protects everything at the file level, so the per-row seam has -nothing to do there. `fullDatabaseKeyHex` renders the whole-file key as -lowercase hex for the SQLCipher `PRAGMA key` statement and is documented -as something that must never be logged. +rather than `public`. The SQLite backend, in a sibling module of the +same Swift package, needs it. Nothing outside the package should ever +see it. The convenience initializer `init(_ mode:)` mints a fresh, +full-entropy 256-bit key and a UUID key identifier for either +encrypting mode. It mints neither for plaintext. `usesRowCrypto` is +`true` only for Mode 2. Mode 3 protects everything at the file level, +so the per-row seam has nothing to do there. `fullDatabaseKeyHex` +renders the whole-file key as lowercase hex for the SQLCipher `PRAGMA +key` statement. It is documented as something that must never be +logged. ### RowCrypto.swift This file provides the per-row AES-GCM-256 encryption used by -`EncryptionMode.rowEncryption`, plus the three write/read seam functions -that every backend calls at the same two moments: just before a write and -just after a read. Living in `PersistenceKit` core rather than in one -backend module is the point: both `PersistenceKitSQLite` and -`PersistenceKitPostgreSQL` call the exact same code, so the two backends -produce byte-compatible ciphertext envelopes without either one knowing -about the other. - -`AeadProvider` is a small protocol seam — AEAD stands for Authenticated -Encryption with Associated Data, a class of algorithm that both encrypts -data and proves it was not tampered with. Any type conforming to -`AeadProvider` can replace the default algorithm without touching any -call site; the file names this as the extension point a future -FIPS-validated provider would use. `CryptoKitAeadProvider` is the default -implementation, backed by Apple's CryptoKit. It generates a fresh random -96-bit nonce on every single encrypt call — reusing a nonce under the same -key is the one mistake that breaks AES-GCM's security guarantee — and -returns the three parts concatenated as `[12-byte nonce][16-byte -tag][ciphertext]`, so a later decrypt is self-contained from the stored -bytes alone with nothing else to look up. - -`encryptedForWrite(_:config:provider:)` is the write-side seam: for an -estate using row encryption, it encrypts the row's `content` column and -stamps a `keyID` column recording which key sealed it; for any other mode, -or for a row with no `content` column, it returns the values unchanged. -`decryptedForRead(_:config:provider:)` reverses this on read, but only -when the row's stored `keyID` matches the estate's own key identifier — a -mismatched `keyID` means the row was sealed under a key this estate does -not hold, so the function passes the row through as still-ciphertext -rather than attempting a decrypt that would only fail as an authentication -error. `assertContentKeyIDInvariant(_:table:config:)` is a final structural -guard: on an encrypting estate, any content-bearing row reaching a backend -write path with `.text` content and no `keyID` means the encryption seam -did not run somewhere upstream, and the function throws rather than let -that plaintext be persisted where a later read could never recover it. +`EncryptionMode.rowEncryption`. It also provides the three write and +read seam functions every backend calls at the same two moments: just +before a write, and just after a read. Living in `PersistenceKit` core +rather than in one backend module is the point. Both +`PersistenceKitSQLite` and `PersistenceKitPostgreSQL` call the exact +same code. The two backends therefore produce byte-compatible +ciphertext envelopes without either one knowing about the other. + +`AeadProvider` is a small protocol seam. AEAD stands for Authenticated +Encryption with Associated Data. This is a class of algorithm that +both encrypts data and proves no one changed it. Any type that +conforms to `AeadProvider` can swap in for the default algorithm +without touching any call site. The file names this as the point +where a future FIPS-checked provider would plug in. +`CryptoKitAeadProvider` is the default version, backed by Apple's +CryptoKit. It makes a fresh random ninety-six-bit nonce on every +single encrypt call. Reusing a nonce under the same key is the one +mistake that breaks AES-GCM's whole guarantee. It returns the three +parts joined as `[12-byte nonce][16-byte tag][ciphertext]`. A later +decrypt is self-contained from the stored bytes alone. It has nothing +else to look up. + +`encryptedForWrite(_:config:provider:)` is the write-side seam. For an +estate using row encryption, it encrypts the row's `content` column. It +stamps a `keyID` column recording which key sealed it. For any other +mode, or for a row with no `content` column, it returns the values +unchanged. `decryptedForRead(_:config:provider:)` reverses this on +read, but only when the row's stored `keyID` matches the estate's own +key identifier. A mismatched `keyID` means the row was sealed under a +key this estate does not hold. The function passes the row through as +still-ciphertext rather than trying a decrypt that would only fail as +an authentication error. `assertContentKeyIDInvariant(_:table:config:)` +is a final structural guard. On an encrypting estate, any +content-bearing row that reaches a backend write path with `.text` +content and no `keyID` means the encryption seam did not run somewhere +upstream. The function throws rather than let that plaintext be stored +where a later read could never recover it. ### CachingRowStore.swift This file provides `CachingRowStore`, a decorator that wraps any -`RowStore` and serves an in-memory hot tier of recently-read rows, -transparently — the file's own comment states the guarantee plainly: -every operation returns results identical to what the unwrapped backing -store would have returned, and the cache changes only latency, never -correctness. +`RowStore` and serves an in-memory hot tier of recently-read rows. The +caller never has to know this layer is there. The file's own comment +states the guarantee plainly. Every operation returns results the same +as what the unwrapped backing store would have returned. The cache +changes only latency, never correctness. -The cache key is not just `(table, row key)`; it is that pair combined +The cache key is not just `(table, row key)`. It is that pair combined with an `AsOfCoordinate`. A `.present` (current-state) read and an -`.asOf(hlc)` (past-state) read of the same row are cached as two entirely -separate entries, because a snapshot read against a pinned, immutable -past state can be cached forever — the GC pin mechanism guarantees that -data will not be vacuumed out from under it — while a present-state read -must be evicted the instant the row changes. `insert`, `upsert`, -`update`, and `delete` all evict the affected present-state entry (or, for -a batch predicate that does not identify one specific row, evict every -present-state entry for that table) after the underlying write succeeds. -An optional `parentChainProvider` closure, supplied by the calling kit, -lets a write also evict cached Merkle-aggregate entries for that row's -ancestors, so a cached rollup value never goes stale just because one leaf -changed underneath it. - -Admission to the cache is sensitivity-gated. `isAdmissible(_:)` reads a -row's `provenance` bitmap column (when present) and decodes a six-bit -sensitivity field at bits 30–35, mapping the scale-gapped raw values 0, -16, 32, and 48 to the ordinals Normal, Elevated, Restricted, and Secret. -Secret content is rejected unconditionally, and anything above the -configured `sensitivityThreshold` is rejected too. An unrecognized bit -pattern, or a `provenance` value of the wrong `TypedValue` case, is also -rejected — the comments call this failing closed, meaning that any doubt -about a row's sensitivity keeps it out of the cache rather than risking -that sensitive content sits unencrypted in process memory longer than it -should. Eviction, when the estimated byte total exceeds the configured -ceiling, removes the least-recently-used entry first, tracked by a -monotonically increasing access counter rather than by wall-clock time. - -All of this mutable state — the entry dictionary, the access counter, the -running byte total — lives inside a private `actor`, `CacheActor`. Because -every stored property of the actor is itself `Sendable`, wrapping the -state in an actor is what lets the outer `CachingRowStore` class, a -`final class` with no locks of its own, safely conform to `Sendable` under -Swift 6's strict concurrency checking. +`.asOf(hlc)` (past-state) read of the same row are cached as two +entirely separate entries. A snapshot read against a pinned, fixed +past state can be cached forever. The GC pin mechanism guarantees that +data will not be swept away out from under it. A present-state read, +by contrast, must be dropped the instant the row changes. Four write +methods drop the affected present-state entry after the underlying +write succeeds: `insert`, `upsert`, `update`, and `delete`. For a +batch predicate that does not name one specific row, these same four +drop every present-state entry for that table instead. An optional `parentChainProvider` closure, +supplied by the calling kit, lets a write also drop cached Merkle- +aggregate entries for that row's ancestors. A cached rollup value never +goes stale just because one leaf changed underneath it. + +Admission to the cache is gated by sensitivity. `isAdmissible(_:)` reads +a row's `provenance` bitmap column, when present, and decodes a +six-bit sensitivity field at bits thirty through thirty-five. It maps the scale-gapped raw values zero, sixteen, thirty-two, and +forty-eight to four ordinals. The ordinals are Normal, Elevated, +Restricted, and Secret, in that order. Secret content is turned away +without exception. Anything above the +configured `sensitivityThreshold` is turned away too. An unrecognized +bit pattern, or a `provenance` value of the wrong `TypedValue` case, is +also turned away. The comments call this failing closed. Any doubt +about a row's sensitivity keeps it out of the cache, rather than +risking that sensitive content sits unencrypted in process memory +longer than it should. Eviction removes the least-recently-used entry first, whenever the +estimated byte total goes over the ceiling the estate sets. A counter +that only ever rises tracks recent use. The cache does not track +wall-clock time at all. + +All of this changeable state lives inside a private `actor`, +`CacheActor`. The state is the entry dictionary, the access counter, +and the running byte total. Every stored property of the actor is +itself `Sendable`. Wrapping the state in an actor is what lets the +outer `CachingRowStore` class safely conform to `Sendable` under Swift +6's strict concurrency checking. `CachingRowStore` itself is a `final +class` with no locks of its own. ### CacheInvalidator.swift This file provides `CacheInvalidator`, the bridge between a `StorageObserver`'s live change stream and a `CachingRowStore`'s -invalidation method — needed for the case where some other writer -bypasses the specific `CachingRowStore` instance entirely (for example, -a second process, or a raw connection opened for a migration) and -mutates the backing store directly. Without this bridge, the cache would -serve stale data forever after such a write, because it would never learn +invalidation method. This bridge is needed for the case where some +other writer bypasses the specific `CachingRowStore` instance +entirely. That writer might be a second process, or a raw connection +opened for a migration. Without this bridge, the cache would serve +stale data forever after such a write, because it would never learn the write happened. -One `CacheInvalidator` manages every table it is told to watch through a -single detached background task that fans out one child task per table, -using Swift's structured-concurrency `withTaskGroup`. `cancel()` cancels -the whole tree at once, and `deinit` calls `cancel()` automatically so a -caller who simply lets the invalidator go out of scope does not leak a -background task. The initializer's documentation is careful to flag one -narrow race: subscriptions start inside the background task, so a write -issued in the same instant as `init()` could in principle race ahead of -subscription registration; a caller with strict-ordering requirements is -advised to yield briefly (for example with `Task.sleep`) before relying on -the very first write being observed. +One `CacheInvalidator` watches every table it is told to watch through +a single detached background task. That task fans out one child task +per table, using Swift's structured-concurrency `withTaskGroup`. +`cancel()` cancels the whole tree at once. `deinit` calls `cancel()` on +its own. A caller who simply lets the invalidator go out of scope does +not leak a background task. The comment on `init(cache:observer:tables:)` +flags one narrow race. Subscriptions start inside the background task. +A write issued in the same instant as `init()` could in theory race +ahead of that subscription. A caller with strict-ordering needs should +yield briefly, for example with `Task.sleep`, before trusting that the +very first write was seen. ### HashingRowStore.swift -This file provides `HashingRowStore`, a decorator that intercepts writes -to any table marked hashable in the schema, computes a content hash for -the row, and emits a `DirtyChainEvent` so an integrity tree elsewhere in -the system can be kept current incrementally rather than rebuilt from -scratch on every write. PersistenceKit deliberately does not import a -hashing library itself; the actual hash function arrives as an injected -`ContentHashProvider` closure supplied by the calling kit (which does -import the hashing library), so this package stays free of that -dependency while still owning the mechanism that wires hashing into every -write path. +This file provides `HashingRowStore`, a decorator that intercepts +writes to any table marked hashable in the schema. It computes a +content hash for the row. It emits a `DirtyChainEvent`, so an +integrity tree elsewhere in the system can stay current bit by bit, +rather than being rebuilt from scratch on every write. +`PersistenceKit` on purpose does not import a hashing library itself. +The actual hash function arrives as an injected `ContentHashProvider` +closure supplied by the calling kit, which does import the hashing +library. This package stays free of that dependency while still owning +the mechanism that wires hashing into every write path. The trickiest part of this file is keeping the hash correct on partial -writes. An `insert` always carries the row's full column set, so hashing -the incoming `values` directly is correct. An `update`, however, typically -carries only the columns being changed — hashing just that partial -dictionary would produce a hash for a row that never actually existed in -storage. `augmentWithHashForKnownKey(table:rowKey:mergedValues:)` solves -this by pre-reading the row's current, full state, merging the incoming -changes on top of it, and hashing that merged result — the same approach -`upsert` uses when it resolves to an update against an existing row rather -than a fresh insert. Before hashing, the function strips any existing -`content_hash` column from the input, so the insert path and the -update-via-merge path always hash the same set of data columns and never -diverge just because one of them happened to already carry a stale hash -value. +writes. An `insert` always carries the row's full column set, so +hashing the incoming `values` directly is correct. An `update`, +however, often carries only the columns being changed. Hashing just +that partial dictionary would produce a hash for a row that never +actually existed in storage. +`augmentWithHashForKnownKey(table:rowKey:mergedValues:)` solves this by +pre-reading the row's current, full state. It merges the incoming +changes on top of it. It then hashes that merged result. `upsert` uses +this same approach when it resolves to an update against an existing +row rather than a fresh insert. Before hashing, the function strips +any existing `content_hash` column from the input. The insert path and +the update-via-merge path therefore always hash the same set of data +columns. Neither one ever drifts apart just because it happened to +already carry a stale hash value. `emitDirtyChain(table:rowKey:hashResult:)` delivers the resulting -`DirtyChainEvent` to an injected `ObserverRegistryRef` closure, which is -`nil` for backends without observer support — the hash is still computed -and stored, but the notification step is simply skipped. +`DirtyChainEvent` to an injected `ObserverRegistryRef` closure, which +is `nil` for backends without observer support. The hash is still +computed and stored either way. The notification step is simply +skipped when there is no closure to call. ### ErasureLedger.swift This file provides the append-only record of what has been erased, without ever storing the erased content itself. A "drawer," in this -context, is one unit of stored content (the term comes from the estate's -own filing metaphor, not from this package). `ErasureLedgerEntry` pairs a -`drawerId` with the `HLC` at which it was erased. The `erasure_ledger` -table this module declares is marked append-only in its -`TableDeclaration` — every backend enforces this at the storage layer, so -even a caller that somehow bypasses the ledger's own API cannot mutate or -remove an entry once written, which is exactly the tamper-evidence -property an erasure record needs. - -`recordErasure(rowStore:drawerId:erasedHlc:)` inserts one entry and -throws `StorageError.duplicateKey` if that drawer was already erased — -each drawer is erased exactly once. `isErased(rowStore:drawerId:)` is the -fast point-lookup used on every read that might need to hide erased -content; `lookupErasure(rowStore:drawerId:)` returns the full entry when a -caller needs the erasure timestamp itself, not just a yes/no answer. +context, is one unit of stored content. The term comes from the +estate's own filing metaphor, not from this package. +`ErasureLedgerEntry` pairs a `drawerId` with the `HLC` at which it was +erased. The `erasure_ledger` table this module declares is marked +append-only in its `TableDeclaration`. Every backend enforces this at +the storage layer. Even a caller that somehow bypasses the ledger's +own API cannot change or remove an entry once it is written. This is +exactly the tamper-evidence property an erasure record needs. + +`recordErasure(rowStore:drawerId:erasedHlc:)` inserts one entry. It +throws `StorageError.duplicateKey` if that drawer was already erased, +because each drawer is erased exactly once. +`isErased(rowStore:drawerId:)` is the fast point-lookup used on every +read that might need to hide erased content. +`lookupErasure(rowStore:drawerId:)` returns the full entry when a +caller needs the erasure time itself, not just a yes-or-no answer. ### ErasureOverlay.swift -This file provides the two-phase, fail-closed filter that actually hides -erased content from a query result, built on top of the ledger this -module does not itself define. `ErasureOverlayConfig` is supplied by a -higher kit and carries two pieces of entity-specific knowledge -PersistenceKit does not have on its own: `extractErasureId`, a closure -that pulls the erasure-ledger key out of a result row (or returns `nil` -for a row type that is not subject to erasure at all), and -`contentColumns`, the list of column names to null out when a row turns -out to be erased. - -`ErasureOverlay.apply(rows:config:rowStore:)` runs the two phases the file -describes at the top: phase one is an ordinary query that already -happened before this function is called; phase two walks the returned -rows one at a time, checks each one's erasure ID against the ledger, and -nulls its content columns if erased. The important design decision is -what happens when the ledger check itself throws an error instead of -returning a clean yes-or-no answer: the row is dropped from the result -entirely. This is the fail-closed half of "two-phase fail-closed" — an -uncertain answer about whether a row was erased is treated the same as -"it was," because showing content that should have been erased is a worse -failure than temporarily hiding content that was not. +This file provides the two-phase, fail-closed filter that actually +hides erased content from a query result. It builds on top of the +ledger, though this module does not itself define that ledger. +`ErasureOverlayConfig` is supplied by a higher kit. It carries two +pieces of entity-specific knowledge `PersistenceKit` does not have on +its own. `extractErasureId` is a closure that pulls the erasure-ledger +key out of a result row, or returns `nil` for a row type not subject +to erasure at all. `contentColumns` is the list of column names to +null out when a row turns out to be erased. + +`ErasureOverlay.apply(rows:config:rowStore:)` runs the two phases the +file describes at the top. Phase one is a plain query that +already ran before this function was called. Phase two walks the +returned rows one at a time. It checks each row's erasure ID against +the ledger. It nulls that row's content columns if the row was +erased. The important design choice is what happens when the ledger +check itself throws an error, instead of giving a clean yes-or-no +answer. The row is dropped from the result. This is the +fail-closed half of "two-phase fail-closed." An uncertain answer about +whether a row was erased is treated the same as "it was." Showing +content that should have been erased is a worse failure than +temporarily hiding content that was not. ### GCPin.swift -This file provides the query that tells a garbage-collection pass which -rows it must not touch yet. GC, short for garbage collection, is any -maintenance pass that reclaims space by deleting rows that are no longer -needed — for example, old versions of a row that a new version has -superseded. The problem this file solves: a snapshot taken earlier (via -`SnapshotRegistry`) might still need to read an "old" row that the GC pass -would otherwise consider safe to delete. +This file provides the query that tells a garbage-collection pass +which rows it must not touch yet. GC, short for garbage collection, is +any maintenance pass that reclaims space by deleting rows no longer +needed. An old version of a row that a newer version has replaced is +one example. The problem this file solves is simple to state. A +snapshot taken earlier, through `SnapshotRegistry`, might still need to +read an "old" row that a GC pass would otherwise think it safe to +delete. `GCPin.minimumRetainableHlc(rowStore:)` answers this by finding the -smallest HLC across every currently-registered snapshot — the oldest live -snapshot's timestamp is the pin. Any row whose HLC is at or after that -pin must survive a GC pass; anything strictly older is safe to reclaim. -When no snapshots exist at all, the function returns `nil`, meaning -nothing is pinned and every row is fair game for GC. `isPinned(rowStore: -rowHlc:)` is the convenience per-row check built on top of the same query. +smallest HLC across every currently-registered snapshot. The oldest +live snapshot's time stamp is the pin. Any row whose HLC sits at or +after that pin must survive a GC pass. Anything strictly older is safe +to reclaim. When no snapshots exist at all, the function returns +`nil`. Nothing is pinned in that case, and every row is fair game for +GC. `isPinned(rowStore:rowHlc:)` is the convenience per-row check built +on top of the same query. ### SnapshotRegistry.swift -This file provides the durable record of named point-in-time snapshots -and the cryptographic attestations that accompany them. A snapshot, in -this design, is not a copy of any data — it is a registry row recording -one HLC (the moment the snapshot was taken) plus one or more attestation -rows recording what a Merkle root looked like at that moment for a given -subject. PersistenceKit itself knows nothing about what a "subject" is — -the `subjectKind` and `subjectId` strings are supplied by whichever +This file provides the durable record of named point-in-time +snapshots. It also provides the signed proofs that go with them. A +snapshot, in this design, is not a copy of any data. It is a registry +row recording one HLC, the moment the snapshot was taken. It also adds +one or more proof rows. Each proof row records what a Merkle root +looked like at that moment for a given subject. `PersistenceKit` itself knows nothing about what a "subject" +is. The `subjectKind` and `subjectId` strings are supplied by whichever higher-level kit is taking the snapshot. -`SnapshotId` is a UUID-backed opaque identifier, minted fresh by -`SnapshotId.mint()`. `SnapshotRegistryOps.createSnapshot(rowStore:hlc: -label:createdAt:attestations:)` mints an ID, inserts one registry row, and -inserts one attestation row per subject passed in, returning the -completed `SnapshotRecord` to the caller. `listSnapshots(rowStore:)` walks -every registered snapshot in HLC order; `deleteSnapshot(rowStore: -snapshotId:)` removes a snapshot's attestation rows before removing its -registry row (child rows first, so no orphaned attestation can ever -reference a deleted registry entry) and reports whether anything existed -to delete. `attestations(rowStore:snapshotId:)` reads back every -attestation for one snapshot, ordered by subject for deterministic -output. The remaining private helpers handle the mechanical work of -turning a `StorageRow` into a typed `SnapshotRecord` or -`SnapshotAttestation` and back. +`SnapshotId` is a UUID-backed opaque ID, minted fresh by +`SnapshotId.mint()`. +`SnapshotRegistryOps.createSnapshot(rowStore:hlc:label:createdAt:attestations:)` +mints an ID, inserts one registry row, and inserts one proof row per +subject passed in. It returns the finished `SnapshotRecord` to the +caller. `listSnapshots(rowStore:)` walks every registered snapshot in +HLC order. `deleteSnapshot(rowStore:snapshotId:)` removes a snapshot's +proof rows before removing its registry row, child rows first, so no +orphaned proof can ever point at a deleted registry entry. It reports +whether anything existed to delete. `attestations(rowStore:snapshotId:)` +reads back every proof for one snapshot, ordered by subject for a +steady result. The remaining private helpers handle the mechanical +work. They turn a `StorageRow` into a typed `SnapshotRecord` or +`SnapshotAttestation`, and back again. ### NoOpObserver.swift -This file provides `NoOpObserver`, a trivial `StorageObserver` -conformance whose three subscription methods each return a stream that -finishes immediately, delivering nothing. It exists as a placeholder for -any backend or test path that has no real change-notification mechanism -yet but still needs to satisfy the `Storage` protocol's requirement for -an `observer` property — the PostgreSQL backend, for instance, uses this -type directly rather than implementing a live PostgreSQL notification +This file provides `NoOpObserver`, a plain `StorageObserver` that +satisfies the protocol without doing any real work. Its three +subscription methods each return a stream that finishes right away, +delivering nothing. It exists as a stand-in for any backend or test +path that has no real change-notification mechanism yet. That backend +still needs to satisfy the `Storage` protocol's requirement for an +`observer` property. The PostgreSQL backend, for instance, uses this +type directly. It does not implement a live PostgreSQL notification channel. ### NovelTokenTaggerChoice.swift This file provides the estate-creation-time choice of which tagger -classifies a "novel token" — a word not found in a language model's -existing vocabulary — when an estate needs to do that classification work -somewhere in its processing pipeline. The two cases are `.hmm`, a -deterministic Hidden Markov Model tagger that produces the exact same -output on every platform and is therefore safe to use across a federation -of devices, and `.nlTagger`, Apple's `NaturalLanguage` framework tagger, -which can be more accurate on Apple hardware but is not available outside -Apple platforms and is not guaranteed to produce the same output across -different OS versions. - -This choice is fixed permanently at estate creation in this version of -the design; changing it later, with re-tagging of existing content, is -explicitly deferred to a future version. The file's extensive comments -spell out the practical consequence: an estate created with `.nlTagger` -cannot safely federate (share and compare memories) with an -`.hmm`-tagged estate without re-tagging all of its content first, because -the two taggers can disagree on the same word. Enforcing that constraint -automatically is also deferred; for now, a caller who selects `.nlTagger` -is responsible for not mixing that estate into a federation with -incompatible estates. `NovelTokenTaggerChoice.default` is `.hmm` — the -safe, federation-compatible baseline every estate gets unless a caller -explicitly opts into the Apple-only alternative. An identically-named, -independently-defined enum exists in `LatticeLib`, in a different package; -the two packages deliberately do not share this type by importing one -from the other, so that neither package depends on the other. +classifies a "novel token." A novel token is a word not found in a +language model's own word list. An estate needs to sort words like +this somewhere in its processing pipeline. The two +cases are `.hmm` and `.nlTagger`. `.hmm` is a fixed Hidden Markov Model +tagger. It produces the exact same output on every platform, so it is +safe to use across a group of devices sharing memory. `.nlTagger` is +Apple's `NaturalLanguage` framework tagger. It can be more accurate on +Apple hardware. It does not work outside Apple platforms. It also does not always produce the same output across different +OS versions. + +This choice is fixed forever at estate creation in this version of the +design. Changing it later, with re-tagging of existing content, is on +purpose put off to a future version. The file's long comments spell +out the practical result. An estate created with `.nlTagger` cannot +safely federate with an `.hmm`-tagged estate without re-tagging all of +its content first. Federating means sharing and comparing memories +across estates. The two taggers can disagree on the same word. No +code enforces that rule on its own yet. A caller who picks +`.nlTagger` must keep that estate out of any federation with a +mismatched estate. `NovelTokenTaggerChoice.default` is `.hmm`. This is +the safe, federation-friendly baseline every estate gets, unless a +caller on purpose picks the Apple-only choice instead. An enum with +the same name exists in `LatticeLib`, a different package, defined on +its own. The two packages on purpose do not share this one type. Not +one imports it from the other, so neither package depends on the +other. ### PersistenceKitTelemetry.swift -This file provides `reportStorageStats(_:estateID:now:)`, the function -that turns a `StorageIntrospection.stats(now:)` snapshot into a stream of -named metrics through `IntellectusLib`, a lightweight, zero-dependency -telemetry library. The function's entire behavior is gated on -`Intellectus.isEnabled`, which defaults to `false`. When telemetry is -disabled — the default for essentially every deployment — the function -returns after a single atomic boolean load, without ever calling -`stats(now:)` and without building a single metric object; the file's -comment quantifies this cost at roughly one nanosecond, making the -disabled path effectively free. - -When telemetry is enabled, the function emits one metric per non-`nil` -field of the captured `StorageStats`, each under the `persistence.db.*` -namespace and each tagged with the emitting kit's name and the estate's -ID, so a monitoring backend can filter and group by estate. Because -`StorageStats` fields are `nil` exactly where a backend cannot supply -them, the emitted metric set naturally differs by backend without this -function needing to know anything about which backend produced the -snapshot — an in-memory estate simply never emits a `wal_frames` metric, -because that field was never populated in the first place. Every -timestamp passed to `Intellectus.report(_:)` comes from the caller-supplied -`now` parameter, never from a fresh `Date()` call inside this function, -which is a deliberate determinism rule followed throughout the whole -package: any code that might run identically twice must never read the -wall clock itself. +This file provides `reportStorageStats(_:estateID:now:)`. This function +turns a `StorageIntrospection.stats(now:)` snapshot into a stream of +named metrics through `IntellectusLib`, a light, zero-dependency +telemetry library. The function's whole behavior is gated on +`Intellectus.isEnabled`, which defaults to `false`. Telemetry is off +for nearly every deployment. When it is off, the function returns +after a single atomic boolean check. It never calls `stats(now:)`. It +never builds a single metric object. The file's own comment puts this +cost at roughly one nanosecond. The disabled path is close to free. + +When telemetry is on, the function emits one metric per non-`nil` +field of the captured `StorageStats`. Each metric sits under the +`persistence.db.*` namespace. Each one is tagged with the emitting +kit's name and the estate's ID, so a monitoring backend can filter and +group by estate. `StorageStats` fields are `nil` exactly where a +backend cannot supply them. The set of emitted metrics differs from +backend to backend. This function needs to know nothing about which +backend produced the snapshot. For example, an in-memory estate never +emits a `wal_frames` metric. That field was simply never filled in. +Every time stamp passed to `Intellectus.report(_:)` +comes from the caller-supplied `now` argument. It never comes from a +fresh `Date()` call inside this function. This is a rule followed +through the whole package. Any code that might run the same way twice +must never read the wall clock itself. ## Target: PersistenceKitInMemory ### InMemoryStorage.swift -This file provides `InMemoryStorage`, the backend used for tests and for -any storage that should live only as long as the current process, plus -the `InMemoryStateActor` that owns every piece of mutable state behind -Swift's actor isolation. There is no file on disk and no network -connection; every table is a plain Swift dictionary held in memory. - -The most carefully documented piece of this file is how `transaction( -isolation:_:)` implements rollback. Rather than running the caller's -block against a private, detached copy of the state and only replacing -the live state on success, it runs the block directly against the live -actor state and takes a snapshot copy only for the error path. The file's -own comment explains why the seemingly simpler "copy, mutate, replace on -success" approach is actually wrong: if any other write reaches the live -actor state between the moment the snapshot was taken and the moment a -successful transaction would replace the whole state with its -snapshot-derived copy, that concurrent write would be silently erased by -the replacement. The comment traces this exact bug to a real incident — a -burst of concurrent inserts racing a transaction lost five to ten percent -of queued work. Running against live state avoids the problem entirely, -at the cost of needing an explicit rollback-to-snapshot path when the -block throws. - -Change notifications during a transaction are buffered rather than -delivered immediately, for a related reason: if a subscriber were notified -of a write that the transaction later rolled back, it would have observed -something that, from the estate's point of view, never actually happened. -`beginNotificationBuffering()` starts collecting notifications instead of -delivering them; `commitNotifications()` flushes the buffer to observers -only after the block has returned successfully; the rollback path -discards the buffer instead. `insertRow`, `upsertRow`, `updateRows`, and -`deleteRows` are the four mutating operations, each of which computes any -declared `GeneratedColumn` values via the static -`materializeGenerated(_:_:)` helper before storing the row, matching what -SQLite and PostgreSQL compute natively so a query against any backend -returns the same generated values. `queryRows(...)` applies the predicate -and ordering against full rows first and only narrows to the requested -column projection at the very end, so a projected query can still filter -or sort on a column it does not actually return — matching SQLite's -`SELECT`-list-versus-`WHERE`/`ORDER BY` semantics exactly. - -`InMemoryStorage` also conforms to `StorageIntrospection`, reporting an -approximate byte size (a flat per-row estimate plus exact blob byte -counts — described in the file as a rough signal, not a precise -allocator measurement), a live row and blob count, and a monotonically -increasing rollback counter. +This file provides `InMemoryStorage`, the backend used for tests and +for any storage that should live only as long as the current process. +It also provides the `InMemoryStateActor` that owns every piece of +changeable state behind Swift's actor isolation. There is no file on +disk. There is no network connection. Every table is a plain Swift +dictionary held in memory. + +The trickiest part of this file is how +`transaction(isolation:_:)` implements rollback. It does not run the +caller's block against a private, separate copy of the state and then +replace the live state only on success. Instead, it runs the block +directly against the live actor state. It takes a snapshot copy only +for the error path. The file's own comment explains why the simpler +"copy, change, replace on success" approach is wrong. Say +some other write reaches the live actor state between the moment the +snapshot was taken and the moment a successful transaction would +replace the whole state with its snapshot-derived copy. That +concurrent write would be silently wiped out by that swap. The +comment traces this exact bug to a real incident. A burst of +concurrent inserts racing a transaction lost five to ten percent of +queued work. Running against live state avoids the problem. +The cost is needing a clear rollback-to-snapshot path when the block +throws. + +Reports of a change during a transaction are held back, rather than +sent right away, for a related reason. Say a subscriber were told +about a write that the transaction later rolled back. From the +estate's point of view, a rolled-back write never took place at all. +`beginNotificationBuffering()` starts collecting change reports +instead of sending them. `commitNotifications()` flushes the buffer to +observers only after the block has returned with success. The +rollback path throws the buffer away instead. Four operations change +stored rows: `insertRow`, `upsertRow`, `updateRows`, and `deleteRows`. +Each one works out any declared `GeneratedColumn` +values through the static `materializeGenerated(_:_:)` helper before +storing the row. This matches what SQLite and PostgreSQL compute on +their own, so a query against any backend returns the same generated +values. `queryRows(...)` applies the predicate and ordering against +full rows first. It only narrows to the requested column set at the +very end. A projected query can still filter or sort on a +column it does not actually return. This matches SQLite's own +`SELECT`-list-versus-`WHERE`/`ORDER BY` behavior, in full. + +`InMemoryStorage` also conforms to `StorageIntrospection`. It reports +a rough byte size, a live row and blob count, and a steadily rising +rollback counter. The byte size is a flat per-row guess plus exact +blob byte counts. The file's own comment calls this a rough signal, +not an exact count of memory used. ### InMemoryRowStore.swift This file provides the thin `RowStore` conformance that simply forwards every call to the shared `InMemoryStateActor`. Its one piece of design -worth noting is the column-projection overload: because the in-memory -backend already holds the full row in memory, projecting away unrequested -columns saves no actual transfer cost the way it does for SQLite — the -comment is explicit that the point of implementing it anyway is -consistency, so a `StorageRow` returned from a projected in-memory query -is byte-identical in shape to the same projected query against SQLite, -and code under test cannot accidentally depend on a column being present -just because the in-memory backend happened to still have it in memory. +worth noting is the column-projection overload. The in-memory backend +already holds the full row in memory, so leaving out columns nobody +asked for saves no real transfer cost, the way it does for SQLite. The +comment is clear that the point of building it anyway is one of +matching shape. A `StorageRow` returned from a projected in-memory +query is shaped just like the same projected query against SQLite. +Code under test cannot depend on a column being present by mistake, +just because the in-memory backend happened to still have it in +memory. ### InMemoryBlobStore.swift -This file provides the `BlobStore` conformance for the in-memory backend -— five one-line forwarding methods to the shared state actor's blob -dictionary. There is nothing backend-specific to explain here beyond what -`BlobStore.swift` already documents; this file exists purely to satisfy -the protocol against the actor's storage. +This file has the `BlobStore` conformance for the in-memory +backend. It is five one-line forwarding methods to the shared state +actor's blob dictionary. There is nothing backend-specific to explain +here beyond what `BlobStore.swift` already covers. This file exists +purely to satisfy the protocol against the actor's storage. ### InMemoryAuditLog.swift -This file provides the `AuditLog` conformance for the in-memory backend, -again a thin forwarding layer to the shared state actor, which itself -de-duplicates on `(eventID, hlc)` exactly as the protocol's idempotence -contract requires. +This file has the `AuditLog` conformance for the in-memory +backend. It is again a thin forwarding layer to the shared state +actor. That actor removes duplicates on `(eventID, hlc)`, just as the +protocol's idempotence contract requires. ### InMemoryObserver.swift -This file provides `ObserverRegistry`, the subscription bookkeeping -shared by every in-memory row store, blob store, and audit log instance -backed by the same state actor, and `InMemoryObserver`, the thin -`StorageObserver` conformance built on top of it. The registry -deliberately does not use actor isolation for its subscriber lists; -instead it uses a plain `NSLock`. The file's comment explains the -trade-off precisely: registering a subscription must be synchronous, with -the stream already recording it before `observe()` returns to the caller, -so that a change notified the very next line of code can never race ahead -of the subscription being recorded. An `actor`-based registry would force -`register` to be an `async` function, reopening exactly that race. This -mirrors the equivalent Rust observer hub, which registers synchronously -for the same reason. - -`notify(_:)` snapshots the list of matching subscribers under the lock -and then yields to each of them entirely outside the lock, so that a -subscription's own termination handler — which also needs the lock to -remove itself from the list — can never deadlock against an in-flight -notification. Row, blob, and dirty-chain subscriptions are tracked in -three separate dictionaries but follow the same synchronous-register, -lock-scoped-snapshot, unlocked-delivery pattern throughout. +This file provides `ObserverRegistry`. This is the subscription +bookkeeping shared by every in-memory row store, blob store, and +audit log instance. All three sit behind the same state actor. It also provides +`InMemoryObserver`, the thin `StorageObserver` conformance built on +top of it. The registry on purpose does not use actor isolation for +its subscriber lists. Instead, it uses a plain `NSLock`. The file's +comment lays out the trade-off in plain terms. Registering a +subscription must happen right away, with the stream already recording +it before `observe()` returns to the caller. A change reported the +very next line of code can then never race ahead of the subscription +being recorded. An `actor`-based registry would force `register` to be +an `async` function. That would reopen that same race all over again. This +mirrors the matching Rust observer hub, which registers right away for +the same reason. + +`notify(_:)` takes a snapshot of the list of matching subscribers +under the lock. It then sends to each of them outside the lock. A +subscription's own end-of-life handler, which also needs the lock to +remove itself from the list, can never deadlock against a report +already in flight. Row, blob, and dirty-chain +subscriptions are tracked in three separate dictionaries. All three +follow the same pattern: register right away, snapshot inside the +lock, and send outside the lock. ### PredicateEvaluator.swift -This file provides the in-memory interpreter for `StoragePredicate` — -the only backend that evaluates the predicate tree directly against -Swift values in memory rather than compiling it to a query string first. -`PredicateEvaluator.evaluate(_:against:)` walks the tree recursively, -handling every case from `Predicate.swift` by reading the named column -out of a `[String: TypedValue]` row (treating a missing column the same -as an explicit `.null`) and applying the matching comparison, bitmask -test, or `LIKE`-style text match. `likeMatch(_:pattern:)` translates a -SQL `LIKE` pattern (`%` for any run of characters, `_` for exactly one) -into an `NSRegularExpression`, so the in-memory backend matches the same -patterns SQL's `LIKE` operator would. +This file has the in-memory reader for `StoragePredicate`. It is +the only backend that works out the predicate tree directly against +Swift values in memory, rather than compiling it to a query string +first. `PredicateEvaluator.evaluate(_:against:)` walks the tree from +the top down. It handles every case from `Predicate.swift` by reading +the named column out of a `[String: TypedValue]` row, treating a +missing column the same as an explicit `.null`. It then applies the +matching comparison, bitmask test, or `LIKE`-style text match. +`likeMatch(_:pattern:)` turns a SQL `LIKE` pattern into an +`NSRegularExpression`. In that pattern, `%` stands for any run of +characters. `_` stands for just one character. The in-memory backend +matches the same patterns SQL's `LIKE` operator would. `TypedValueComparator.compare(_:_:)`, in the same file, is the shared -ordering function used both by predicate comparisons and by `ORDER BY` -sorting: `nil` (that is, `.null`) sorts before every other value, and two -values of the same case compare by their underlying Swift value, with -`.hlc` compared by its packed integer form (`HLC.packed`, defined at the -bottom of `InMemoryStorage.swift`) to get a single total order over an -HLC's three component fields. +ordering function used both by predicate comparisons and by `ORDER +BY` sorting. `nil`, that is `.null`, sorts before every other value. +Two values of the same case compare by their underlying Swift value. +`.hlc` is compared by its packed integer form instead. That form is +`HLC.packed`, defined at the bottom of `InMemoryStorage.swift`. Packing +gives a single total order over an HLC's three parts. ## Target: PersistenceKitSQLite ### SQLiteStorage.swift -This file provides `SQLiteStorage`, the `Storage` conformance for the -on-device backend, and the much larger `SQLiteBackend` actor that does -essentially all of the real work — schema migration, every row and blob -operation, audit logging, and health introspection — serialized through -one actor per estate, matching the file's opening comment that a SQLite -estate gets exactly one connection, with SQLite's own WAL (write-ahead -log) mode handling concurrent readers safely underneath. - -Schema migration walks a schema's declared `Migration` list, running each -pending migration inside its own `BEGIN IMMEDIATE`/`COMMIT` pair and -rolling back and throwing `StorageError.migrationFailed` if any operation -in that migration fails. `applyMigrations(_:)` is written to be safe to -call even on a completely fresh SQLite file that was never opened through -`openSchema(_:)` first — a documented real scenario, since at least one -caller invokes `migrate(to:)` directly — by re-running the idempotent -table- and migrations-table-creation steps before checking what actually -needs to migrate. - -Every row-mutating method — `insertRow`, `upsertRow`, `updateRows`, -`deleteRows`, `queryRows` — begins by calling `validateSQLIdentifier` on -every table and column name it is about to interpolate into a SQL string. -The file's comments tie this directly to a named security fix -(SECFIX-WS2-PK F9): double-quoting a SQL identifier is not sufficient -protection if the identifier itself can contain a double-quote character, -because that character can escape the delimiter and alter the query, so -every name is checked against a strict allow-list before it is ever -placed into a SQL string. `insertRow` and `queryRows` are also where the -at-rest row-encryption seam actually runs: `insertRow` calls -`encryptedForWrite` before binding values, and `queryRows` calls -`decryptedForRead` after reading them back, both of which are no-ops on a -plaintext or whole-database-encrypted estate. +This file has `SQLiteStorage`, the `Storage` conformance for the +on-device backend. It also has the much larger `SQLiteBackend` actor +that does almost all of the real work. One actor per estate handles it all: schema migration, row-and-blob +work, audit logging, and health checks. The file's opening comment +matches this design: a +SQLite estate gets exactly one connection. SQLite's own WAL, or +write-ahead log, mode handles concurrent readers safely underneath. + +Schema migration walks a schema's declared `Migration` list. It runs +each pending migration inside its own `BEGIN IMMEDIATE`/`COMMIT` pair. +It rolls back and throws `StorageError.migrationFailed` if any step in +that migration fails. `applyMigrations(_:)` is written to be safe to +call even on a brand-new SQLite file, one never opened through +`openSchema(_:)` first. This is a real case in practice, since at +least one caller calls `migrate(to:)` directly. It handles this by +re-running the safe-to-repeat table- and migrations-table-creation +steps before it checks what actually needs to migrate. + +Five row-changing methods start by calling `validateSQLIdentifier` +on every table and column name they are about to place into a SQL +string: `insertRow`, `upsertRow`, `updateRows`, `deleteRows`, and +`queryRows`. The file's comments tie this directly to a +named security fix, SECFIX-WS2-PK F9. Double-quoting a SQL name is not enough protection on its own. A name +might hold a double-quote character. That one character can escape +the mark and change the query. Every name is checked against a +strict allow-list before it is ever placed into a SQL string. +`insertRow` and `queryRows` are also where the at-rest row-encryption +seam actually runs. `insertRow` calls `encryptedForWrite` before +binding values. `queryRows` calls `decryptedForRead` after reading +them back. Both are no-ops on a plaintext or whole-database-encrypted +estate. `readColumn(stmt:index:schema:columnName:table:)` is the file's most -carefully reasoned function: it decides, column by column, whether a raw -SQLite value should be trusted as-is or treated as corrupt. The -distinction the comment draws is between a type-tolerant decode (a valid -value stored under SQLite's flexible column affinity, which is passed -through as-is) and a genuine parse failure on a column whose declared -type says it should parse (an unparseable UUID or timestamp string), -which throws `StorageError.corruptStoredValue` rather than ever -fabricating a random UUID or an epoch-zero date in its place. +carefully reasoned function. It decides, column by column, whether a +raw SQLite value should be trusted as-is or treated as corrupt. The +comment draws a line between two cases. One is a type-tolerant decode, +a valid value stored under SQLite's flexible column affinity, which is +passed through as-is. The other is a genuine parse failure on a +column whose declared type says it should parse, such as an +unparseable UUID or time-stamp string. That second case throws +`StorageError.corruptStoredValue` rather than ever making up a random +UUID or an epoch-zero date in its place. `queryRowsSkipCorrupt(...)` reuses the same column-reading logic but -catches exactly that error per row, logs it, counts it, and continues to -the next row — the cursor-level implementation `RowStore`'s protocol -default can only approximate by discarding an entire query's results. - -`storageStats(now:)` implements `StorageIntrospection` for SQLite by -reading a handful of read-only `PRAGMA` statements (`page_size`, -`page_count`, `freelist_count`) and deriving the WAL frame count directly -from the `-wal` sidecar file's size on disk rather than calling `PRAGMA -wal_checkpoint`, because that particular PRAGMA can fail with a locked -error if a concurrent operation is in flight, even from inside the same -serializing actor. +catches exactly that error per row. It logs the error, counts the row, +and moves on to the next one. This cursor-level version can do +something `RowStore`'s protocol default can only approximate, which is +to discard an entire query's results. + +`storageStats(now:)` gives `StorageIntrospection` for SQLite by +reading a handful of read-only `PRAGMA` statements: `page_size`, +`page_count`, and `freelist_count`. It works out the WAL frame count +straight from the `-wal` side file's size on disk. It does not call +`PRAGMA wal_checkpoint` for this. That pragma can fail with a locked +error if some other job is in flight. This can happen even from +inside the same serializing actor. ### SQLiteConnection.swift -This file provides `SQLiteConnection`, a thin Swift wrapper around the C -`sqlite3` API, `SQLiteStatement`, the prepared-statement wrapper used for -every parameterized query, and `ISO8601`, the shared timestamp -formatting and parsing utility used everywhere a `.timestamp` `TypedValue` -crosses the SQLite boundary. - -Opening a connection does several things in a specific, safety-relevant -order. First, it checks whether the target path is a symbolic link and -refuses to open it if so — the comment names this CAND-052, a defense -against a pre-planted symlink that could otherwise redirect SQLite's -writes to an arbitrary file elsewhere on disk, and notes that the check -uses `lstat` semantics (via `resourceValues(forKeys:)`) so it identifies +This file has `SQLiteConnection`, a thin Swift wrapper around the C +`sqlite3` API. It also has `SQLiteStatement`. This is the +prepared-statement wrapper used for every parameterized query. A +third piece, `ISO8601`, is the shared time-stamp formatting and +parsing tool. It runs everywhere a `.timestamp` `TypedValue` crosses +the SQLite boundary. + +Opening a connection does several things, in a specific order that +matters for safety. First, it checks whether the target path is a +symbolic link. It refuses to open the file if so. The comment names +this CAND-052, a defense against a pre-planted symlink that could +otherwise send SQLite's writes to some other file on disk. The check +uses `lstat` behavior, through `resourceValues(forKeys:)`, so it spots the symlink itself rather than following it to whatever it points at. -Second, if the estate uses whole-database encryption, it issues `PRAGMA -key` with the estate's hex-encoded key before any other statement runs — -this must be the very first statement, because SQLCipher cannot even read -the schema on page one of the file without the correct key already -applied. Third, it applies Apple's file-level Data Protection, a -best-effort step that layers OS-level at-rest protection on top of -SQLCipher's own encryption. Finally, it sets the durability pragmas: WAL -journal mode, `NORMAL` synchronous durability, a WAL auto-checkpoint -threshold, a busy timeout, and foreign-key enforcement. - -`ISO8601` deserves its own explanation because two very different -performance and correctness stories live in it. `string(from:)` first -clamps an out-of-range `Date` (for example one accidentally derived from -milliseconds mistaken for seconds) into the range `ISO8601DateFormatter` -can actually parse back — years 0001 through 9999 — because writing an -unparseable string like `+59009-...` would silently corrupt any future -read of that row; a clamp with a logged warning is judged better than a -value nothing can ever read back correctly. `date(from:)` tries a hand- -written, allocation-free parser, `fastParseCanonicalUTC(_:)`, before -falling back to the much slower ICU-backed `ISO8601DateFormatter`. The -comment explains why the fast path exists at all: a stack-sampling -profile during a large import showed the formatter's `date(from:)` -consuming roughly eighty percent of total CPU time, because a Merkle -rollup re-decodes every row's timestamp on every insert, making the parse -cost scale quadratically with import size. The fast parser recognizes -only the exact canonical shape this kit itself writes and returns `nil` -for anything even slightly different (a numeric time-zone offset, -lowercase letters, trailing garbage), which sends control back to the -slow, fully general formatters — so correctness never regresses, and the -fast path is verified byte-for-byte against the formatters in a dedicated -test suite. +Second, if the estate uses whole-database encryption, it issues +`PRAGMA key` with the estate's hex-coded key before any other +statement runs. This must be the very first statement. SQLCipher +cannot even read the schema on page one of the file without the +right key already applied. Third, it applies Apple's file-level Data +Protection, a best-effort step that layers OS-level at-rest +protection on top of SQLCipher's own encryption. It sets the durability pragmas last: WAL journal mode, `NORMAL` +synchronous durability, a WAL auto-checkpoint limit, a busy timeout, +and foreign-key rules. + +`ISO8601` deserves its own close look. Two very different stories +about speed and correctness live inside it. `string(from:)` first pins an out-of-range `Date` into the range +`ISO8601DateFormatter` can parse back: the years 0001 through 9999. +One example of an out-of-range date is one built by mistake from +milliseconds read as seconds. Writing an unparseable string like `+59009-...` would silently break +any future read of that row. A pinned value with a logged warning is +judged better than a value nothing can ever read back right. `date(from:)` first tries a hand-written, no-allocation parser: +`fastParseCanonicalUTC(_:)`. Only then does it fall back to the much +slower, ICU-backed `ISO8601DateFormatter`. The comment explains why the fast +path exists at all. A stack-sampling profile during a large import +showed the formatter's `date(from:)` using roughly eighty percent of +total CPU time. A Merkle rollup re-decodes every row's time stamp on +every insert, which makes the parse cost grow with the square of +import size. The fast parser knows only the exact canonical shape +this kit itself writes. It returns `nil` for anything even slightly +different: a numeric time-zone offset, lowercase letters, or trailing +junk. That sends control back to the slow, fully general formatters, +so correctness never slips. The fast path is checked byte-for-byte +against the formatters in its own test suite. ### SQLiteStores.swift -This file provides `SQLiteRowStore`, `SQLiteBlobStore`, `SQLiteAuditLog`, -and `SQLiteTransaction` — four small wrapper types that each forward to -the shared `SQLiteBackend` actor, giving the actor's internal methods -their public `RowStore`/`BlobStore`/`AuditLog`/`StorageTransaction` -faces. `SQLiteRowStore.querySkipCorrupt(...)` is the one method here with -real logic of its own: rather than inheriting `RowStore`'s protocol- -extension default (which can only discard an entire failed query), it -calls `SQLiteBackend.queryRowsSkipCorrupt(...)` directly, so a corrupt -row found partway through a large corpus scan is skipped and logged -individually rather than aborting everything already read. +This file has `SQLiteRowStore`, `SQLiteBlobStore`, `SQLiteAuditLog`, +and `SQLiteTransaction`. These are four small wrapper types that each +forward to the shared `SQLiteBackend` actor. They give the actor's own +methods their public `RowStore`, `BlobStore`, `AuditLog`, and +`StorageTransaction` faces. `SQLiteRowStore.querySkipCorrupt(...)` is +the one method here with real logic of its own. Rather than falling +back to `RowStore`'s protocol-extension default, which can only throw +away an entire failed query, it calls +`SQLiteBackend.queryRowsSkipCorrupt(...)` directly. A corrupt row found +partway through a large corpus scan is skipped and logged on its own, +rather than aborting everything already read. ### SQLiteSchema.swift -This file provides `SQLiteSchema`, the enum of pure functions that -translate a `SchemaDeclaration` into SQLite's native DDL (data definition -language — the SQL used to create and alter tables, as opposed to the SQL -used to read and write rows). `nativeType(_:)` maps every -`ColumnType` case to its SQLite storage class — most map directly, but -`.uuid` and `.timestamp` both map to `TEXT` because SQLite has no native -types for either, `.hlc` maps to `INTEGER` because it is stored as a -packed 64-bit value, and `.fingerprint` maps to `BLOB` because it is -stored as raw bytes. `createTable(_:)` assembles one `CREATE TABLE IF NOT -EXISTS` statement covering ordinary columns, generated columns (always -rendered `STORED`, matching PostgreSQL's lack of a `VIRTUAL` form), -primary key, and unique constraints. `appendOnlyTriggers(_:)` emits a -`BEFORE UPDATE`/`BEFORE DELETE` trigger pair for any table declared -append-only, each one aborting the statement outright with `RAISE(ABORT, -...)` — the actual enforcement mechanism behind, for example, the erasure -ledger's append-only guarantee. - -This file also owns the three internal bookkeeping tables every SQLite -estate carries regardless of what schema a caller declares: -`_storagekit_migrations` (tracks each kit's applied schema version), -`_storagekit_audit` (the single source of truth for every audit event, -with full-precision HLC columns alongside the packed integer form — the -packed form loses a bit of the physical-time field for dates far enough -in the future, so the full-precision columns exist specifically so a -cold rebuild of an estate's last-known HLC never silently disagrees with -what a snapshot originally recorded), and `_storagekit_blobs` (the flat -key/bytes table backing `SQLiteBlobStore`). +This file has `SQLiteSchema`, the enum of pure functions that turn a +`SchemaDeclaration` into SQLite's own DDL. DDL stands for data +definition language, the SQL used to build and change tables, as +opposed to the SQL used to read and write rows. `nativeType(_:)` maps +every `ColumnType` case to its SQLite storage class. Most map +directly, but `.uuid` and `.timestamp` both map to `TEXT`, because +SQLite has no native type for either. `.hlc` maps to `INTEGER`, +because it is stored as a packed sixty-four-bit value. `.fingerprint` +maps to `BLOB`, because it is stored as raw bytes. `createTable(_:)` +puts together one `CREATE TABLE IF NOT EXISTS` statement. It covers +plain columns, generated columns, a primary key, and unique rules. +Generated columns are always rendered `STORED`, to match PostgreSQL's +lack of a `VIRTUAL` form. `appendOnlyTriggers(_:)` emits a `BEFORE UPDATE`/`BEFORE +DELETE` trigger pair for any table marked append-only. Each trigger +stops the statement outright with `RAISE(ABORT, ...)`. This is the +real enforcement behind, for example, the erasure ledger's +append-only rule. + +This file also owns the three bookkeeping tables every SQLite estate +carries, no matter what schema a caller declares. `_storagekit_migrations` +tracks each kit's applied schema version. `_storagekit_audit` is the +one source of truth for every audit event. It stores full-precision HLC columns alongside the packed integer +form. The packed form loses a bit of the physical-time field for +dates far enough in the future. The full-precision columns exist so that a cold rebuild +of an estate's last-known HLC never quietly disagrees with what a +snapshot first recorded. `_storagekit_blobs` is the flat key-and-bytes +table backing `SQLiteBlobStore`. ### SQLitePredicateCompiler.swift -This file provides the translation from a `StoragePredicate` tree into a -parameterized SQLite `WHERE` clause plus its ordered list of bound -values. `compile(_:)` is the public entry point; the private recursive -`render(_:bindings:)` walks the tree case by case, validating every -column name it touches with `validateSQLIdentifier` before it is -interpolated, and appending every comparison value to the bindings array -rather than ever interpolating a value directly — values are always safe -because SQLite's own parameter-binding mechanism handles them, but column -and table identifiers cannot be bound as parameters in SQL, which is -exactly why they need the separate identifier-validation guard instead. +This file has the translation from a `StoragePredicate` tree into a +parameterized SQLite `WHERE` clause, plus its ordered list of bound +values. `compile(_:)` is the public entry point. The private recursive +`render(_:bindings:)` walks the tree case by case. It checks every +column name it touches with `validateSQLIdentifier` before that name +is placed into the string. It appends every comparison value to the +bindings array, rather than ever placing a value directly into the +text. Values are always safe this way, because SQLite's own +parameter-binding step handles them. Column and table names cannot be +bound as parameters in SQL, which is exactly why they need the +separate identifier check instead. ### SQLiteIdentifierValidator.swift -This file provides `validateSQLIdentifier(_:)`, the single free function -that every write path and the predicate compiler in this module call -before interpolating any caller-supplied name into a SQL string. The rule -is simple and strict: the first character must be a letter or an -underscore, and every following character must be a letter, a digit, or -an underscore — the safe subset of SQLite identifier syntax. The file's -own comment states the reason this exists as one shared function rather -than several private copies: a name containing a double-quote character -can escape the double-quote delimiter SQL uses around identifiers and -change what the query actually does, so double-quoting alone is not a -sufficient defense, and having one seam means the defense cannot silently -drift out of sync between the several call sites that need it. +This file has `validateSQLIdentifier(_:)`. Every write path and the +predicate compiler in this module call this one free function. They +call it before placing any caller-supplied name into a SQL string. The rule +is simple and strict. The first character must be a letter or an +underscore. Every character after that must be a letter, a digit, or +an underscore. This is the safe part of SQLite identifier syntax. The +file's own comment states why this exists as one shared function +rather than several private copies. A name holding a double-quote +character can escape the double-quote mark SQL uses around +identifiers. It can change what the query actually does. Double- +quoting alone is not a strong enough defense on its own. Having one +seam means the defense cannot quietly drift out of sync between the +several call sites that need it. ### SQLiteObserver.swift -This file provides `SQLiteObserverRegistry`, an `actor` holding the -subscriber lists for both row-change and blob-change notifications on one -SQLite estate, and `SQLiteObserver`, the public `StorageObserver` -conformance built on top of it. The file's own comment explains a real -limitation of SQLite's native `sqlite3_update_hook`: it fires for every -table, including the internal blob table, but it only reports which -operation happened on which table and row ID — it never carries the -actual column values. Because a blob-change notification needs to carry -the bytes that were written, blob notifications cannot be reconstructed -from the hook at all; instead, `SQLiteBackend`'s `putBlob`/`deleteBlob` -methods call this registry's `notifyBlob(_:)` directly, at the exact -point where the bytes are already in hand. +This file has `SQLiteObserverRegistry`, an `actor` holding the +subscriber lists for both row-change and blob-change notices on one +SQLite estate. It also has `SQLiteObserver`, the public +`StorageObserver` conformance built on top of it. The file's own +comment explains a real limit of SQLite's native +`sqlite3_update_hook`. It fires for every table, including the +internal blob table. It only reports which operation happened on +which table and row ID. It never carries the actual column values. +Because a blob-change notice needs to carry the bytes that were +written, blob notices cannot be rebuilt from the hook at all. Instead, +`SQLiteBackend`'s `putBlob`/`deleteBlob` methods call this registry's +`notifyBlob(_:)` directly, at the exact point where the bytes are +already in hand. ### KeychainKeyStore.swift -This file provides `KeychainKeyStore`, the Apple-platform source of the -whole-database (Mode 3) encryption key — the Apple counterpart to the -Rust port's per-estate key file on disk. One 256-bit key is stored per -estate in the system Keychain, under an account name derived -deterministically from the estate's own file path (`estateAccount(for:)` -standardizes the path and hashes it with SHA-256), so any process that is -told the same estate's file path — the main app and a managed -background server it spawns, in particular — computes the same account -name and therefore loads the same key without any separate key- -distribution step. `loadOrCreateKey()` reads an existing key if one is -present, or generates a fresh cryptographically random key and stores it -if not, handling the race where two callers try to create the key -simultaneously by having the loser simply re-read what the winner wrote. -`deleteKey()` disposes of an estate's key permanently when the estate -itself is deleted, and is documented as idempotent — calling it on an -estate whose key is already gone is a success, not an error. The stored -key's Keychain accessibility level, `afterFirstUnlockThisDeviceOnly`, is -chosen specifically so a background process can still read the key after -the very first unlock following a device restart, without the key ever -leaving the device or syncing to iCloud. +This file has `KeychainKeyStore`. This is the Apple-platform source +of the whole-database encryption key, Mode 3. It is the Apple match +to the Rust port's per-estate key file on disk. One 256-bit key is stored per estate in the system Keychain. Its +account name is worked out in a fixed way from the estate's own file +path. +`estateAccount(for:)` makes the path uniform and hashes it with +SHA-256. Any process told the same estate's file path works out the +same account name this way. The main app and a managed background +server it starts are one such pair of processes. Both load the same +key with no separate key-sharing step needed. `loadOrCreateKey()` reads an +existing key if one is present. If not, it makes a fresh, randomly +generated key and stores it. It handles the case where two callers try to make the key at the +same time. The loser simply re-reads what the winner wrote. `deleteKey()` gets rid of an estate's +key for good when the estate itself is deleted. It is documented as +safe to call twice: calling it on an estate whose key is already gone +counts as success, not an error. The stored key's Keychain access +level, `afterFirstUnlockThisDeviceOnly`, is picked for a specific +reason. A background process can still read the key after the very first +unlock following a device restart. The key never leaves the device. +It never syncs to iCloud. ## Target: PersistenceKitPostgreSQL ### PostgreSQLStorage.swift -This file provides `PostgreSQLStorage`, the `Storage` conformance for the -server backend, and the `PostgreSQLBackend` actor that holds the schema -declaration and drives migrations, transactions, and health -introspection against a connection pool. The most distinctive design -decision here is estate isolation: rather than one PostgreSQL database -per estate, every estate gets its own PostgreSQL schema — a namespace -within one shared database — named `pk_`. Every pooled connection for that estate pins its -`search_path` to that schema (keeping `public` on the path too, so any -shared extension still resolves), which is the PostgreSQL analogue of -SQLite's one-file-per-estate model: many estates can share one database -server without their tables ever colliding. +This file has `PostgreSQLStorage`, the `Storage` conformance for the +server backend. It also has the `PostgreSQLBackend` actor. That actor +holds the schema and drives migrations, transactions, and health +checks against a connection pool. The most distinct design choice +here is estate isolation. Rather than one PostgreSQL database per +estate, every estate gets its own PostgreSQL schema. A schema, here, +is a namespace within one shared database. Each one is named +`pk_`. Every pooled connection for +that estate pins its `search_path` to that schema. `public` stays on +the path too, so any shared extension still resolves. This is the +PostgreSQL match to SQLite's one-file-per-estate model. Many estates +can share one database server without their tables ever colliding. Schema-version bookkeeping uses a simple key-value table, `_storagekit_meta`, rather than a dedicated migrations table like -SQLite's: a per-kit version is stored under the composite key -`"schema_version:"`, and a running-maximum global version is kept -under the plain key `"schema_version"` so the no-argument -`currentSchemaVersion()` still returns something meaningful even when -several kits share one estate. `storageStats(now:)` implements -`StorageIntrospection` for PostgreSQL by querying `pg_database_size`, -`pg_stat_database` (for buffer-cache hit ratio and transaction counters), -and `pg_locks` joined against `pg_database` (for lock contention) — three -different system views, each documented in the file with the exact -formula used to turn its raw counters into the `StorageStats` fields. +SQLite's. A per-kit version is stored under the mixed key +`"schema_version:"`. A running highest version is kept under +the plain key `"schema_version"`. This way, the no-argument +`currentSchemaVersion()` still returns something useful even when +several kits share one estate. `storageStats(now:)` gives +`StorageIntrospection` for PostgreSQL by asking three different +system views. It queries `pg_database_size`. It queries +`pg_stat_database`, for buffer-cache hit ratio and transaction +counters. It queries `pg_locks` joined against `pg_database`, for lock +contention. The file documents the exact formula it uses to turn each +view's raw counters into `StorageStats` fields. ### PostgreSQLPool.swift -This file provides `PostgreSQLPool`, a fixed-size connection pool built as -an `actor` so its bookkeeping — available connections, in-use count, -waiters — never needs a separate lock. `acquire()` returns an idle -connection immediately if one exists, opens a brand-new one if the pool -has not yet reached its configured size, or otherwise suspends the caller -on a `CheckedContinuation` until either a connection frees up or a timeout -elapses, at which point the waiter is resumed with +This file has `PostgreSQLPool`, a fixed-size connection pool built as +an `actor`. Its bookkeeping never needs a separate lock this way. +That bookkeeping is open connections, in-use count, and waiters. +`acquire()` returns +an idle connection right away if one exists. It opens a brand-new one +if the pool has not yet reached its set size. Otherwise, it pauses the +caller on a `CheckedContinuation` until either a connection frees up +or a timeout passes. At that point, the waiter wakes back up with `StorageError.poolExhausted`. Every freshly-opened connection is -immediately pinned to the estate's dedicated PostgreSQL schema by issuing -`CREATE SCHEMA IF NOT EXISTS` followed by `SET search_path`, and the -connection is closed rather than kept if that two-statement setup fails -partway through, so a half-configured connection is never handed back to +pinned right away to the estate's own PostgreSQL schema. This happens +through `CREATE SCHEMA IF NOT EXISTS`, followed by `SET search_path`. +The connection is closed rather than kept if that two-step setup fails +partway through, so a half-set-up connection is never handed back to a caller. -`parseTLSMode(host:)` resolves the pool's TLS behavior from the -`ARIA_MCP_POSTGRES_TLS` environment variable, defaulting to `prefer` (try -TLS, fall back to plaintext if the server does not offer it) whenever the -variable is absent or holds an unrecognized value — the comment is -explicit that even a loopback connection defaults to `prefer` rather than -`disable`, so a caller who genuinely wants plaintext on loopback has to -say so explicitly rather than getting it as an accidental default. +`parseTLSMode(host:)` works out the pool's TLS behavior from the +`ARIA_MCP_POSTGRES_TLS` environment variable. It defaults to `prefer`, +meaning try TLS first and fall back to plaintext if the server does +not offer it, whenever the variable is missing or holds a value it +does not know. The comment is clear that even a loopback connection +defaults to `prefer` rather than `disable`. A caller who truly wants +plaintext on loopback has to say so on purpose, rather than getting it +by accident. ### PostgreSQLConnection.swift -This file provides free functions that bridge between PersistenceKit's +This file has free functions that bridge between `PersistenceKit`'s `TypedValue` and the `postgres-nio` client library's own binding and decoding types. `executeSimple(_:logger:)` and `executeParameterized(_:bindings:logger:)` are small `PostgresConnection` -extension methods that wrap every underlying `postgres-nio` error in +extension methods. They wrap every underlying `postgres-nio` error in `StorageError.backendError`, so a caller never has to catch a -`postgres-nio`-specific error type directly. `makeBindings(_:)` converts +`postgres-nio`-specific error type directly. `makeBindings(_:)` turns an array of `TypedValue` into the positional `$1, $2, ...` bindings -PostgreSQL's parameterized-query protocol expects, handling each -`TypedValue` case's specific wire representation — a `.fingerprint`, for -example, is serialized as 32 raw bytes in a fixed block order so it -round-trips exactly. +PostgreSQL's parameterized-query protocol expects. It handles each +`TypedValue` case's own wire shape. Take `.fingerprint`, for example. +It is written out as thirty-two raw bytes in a fixed block order. This +makes it round-trip exactly. `decodeRow(_:columns:)` and the private `decodeCell(_:type:)` do the -reverse: given a returned `PostgresRow` and the caller's expected column -types, they decode each named cell into the matching `TypedValue` case, -falling back to `.null` if a cell fails to decode as its declared type -rather than throwing — a deliberately more permissive stance than the -SQLite backend's `corruptStoredValue` throw, reflecting that PostgreSQL's -own wire protocol already enforces column types at a lower level than -SQLite's flexible column affinity does. +reverse. Given a returned `PostgresRow` and the caller's expected +column types, they decode each named cell into the matching +`TypedValue` case. If a cell fails to decode as its declared type, +they fall back to `.null` rather than throwing. This is a more open +stance than the SQLite backend's `corruptStoredValue` throw. It +reflects that PostgreSQL's own wire protocol already enforces column +types at a lower level than SQLite's flexible column affinity does. ### PostgreSQLPredicateCompiler.swift -This file provides the PostgreSQL analogue of `SQLitePredicateCompiler`: -the same `StoragePredicate` tree compiled to PostgreSQL's own SQL dialect, -using `$1, $2, ...` positional parameters instead of SQLite's `?` -placeholders. The case-by-case translation logic is otherwise the same -shape as the SQLite compiler — every column name is validated with -`validatePSQLIdentifier` before being interpolated, and every comparison -value becomes a bound parameter rather than inline text. One PostgreSQL- -specific detail: the bitmask predicate cases (`bitmaskAll`, `bitwiseEq`) -have to reference two already-appended bindings by their final numeric -position (`bindings.count - 1` and `bindings.count`), because PostgreSQL's -positional parameters are numbered by the order they were bound, unlike -SQLite's simple `?` placeholders which do not carry a number at all. +This file has the PostgreSQL match to `SQLitePredicateCompiler`: the +same `StoragePredicate` tree compiled to PostgreSQL's own SQL dialect. +It uses `$1, $2, ...` numbered spots instead of SQLite's `?` +placeholders. The case-by-case translation is otherwise the same +shape as the SQLite compiler. Every column name is checked with +`validatePSQLIdentifier` before it is placed into the string. Every +comparison value becomes a bound parameter rather than inline text. +One PostgreSQL-only detail stands out. Two bitmask predicate cases +work this way: `bitmaskAll` and `bitwiseEq`. Both must point at two +already-appended bindings by their final number: `bindings.count - 1` +and `bindings.count`. This is because PostgreSQL's numbered spots are +numbered by the order they were bound, unlike SQLite's plain `?` +placeholders, which carry no number at all. ### PostgreSQLSchema.swift -This file provides `PostgreSQLSchemaEmitter`, the PostgreSQL analogue of -`SQLiteSchema`: pure functions translating a `SchemaDeclaration` into -PostgreSQL DDL. `typeSQL(_:)` maps each `ColumnType` to its PostgreSQL -native type — notably `.uuid` maps to a real `UUID` column (unlike -SQLite's `TEXT` fallback), `.timestamp` maps to `TIMESTAMPTZ`, and `.json` -maps to `JSONB`, PostgreSQL's indexable binary JSON type. Because -PostgreSQL has no `CREATE TRIGGER IF NOT EXISTS`, `appendOnlyTriggerStatements(_:)` -achieves the same idempotence SQLite gets for free by first dropping any -existing trigger of the same name and then creating it fresh, backed by -one shared `appendOnlyFunctionSQL` trigger function (created with `CREATE -OR REPLACE`, so re-running schema setup never fails) that raises an -exception naming the offending table via PostgreSQL's `TG_TABLE_NAME` -variable — one function serving every append-only table in the schema, -rather than one function generated per table. +This file has `PostgreSQLSchemaEmitter`, the PostgreSQL match to +`SQLiteSchema`: pure functions that turn a `SchemaDeclaration` into +PostgreSQL DDL. `typeSQL(_:)` maps each `ColumnType` to its own +PostgreSQL native type. `.uuid` maps to a real `UUID` column, unlike +SQLite's `TEXT` fallback. `.timestamp` maps to `TIMESTAMPTZ`. +`.json` maps to `JSONB`, PostgreSQL's own binary, searchable JSON +type. PostgreSQL has no `CREATE TRIGGER IF NOT EXISTS`. +`appendOnlyTriggerStatements(_:)` gets the same repeat-safe behavior +SQLite gets for free. It first drops any existing trigger of the same +name, then creates it fresh. One shared `appendOnlyFunctionSQL` +trigger function backs every append-only table in the schema, rather +than one function generated per table. It is created with `CREATE OR +REPLACE`, so running schema setup again never fails. It raises an +error naming the table at fault, through PostgreSQL's `TG_TABLE_NAME` +variable. ### PostgreSQLStores.swift -This file provides `PostgreSQLRowStore`, `PostgreSQLBlobStore`, and -`PostgreSQLAuditLog` — the concrete `RowStore`, `BlobStore`, and -`AuditLog` conformances for the PostgreSQL backend, each able to run -either against a pooled connection acquired for a single call or against -a `PostgreSQLTransactionContext`'s already-open connection when called -from inside a transaction block. Every write method here validates SQL -identifiers with `validatePSQLIdentifier`, exactly as the SQLite backend -does, before assembling its SQL string. +This file has three concrete conformances for the PostgreSQL +backend: `PostgreSQLRowStore`, `PostgreSQLBlobStore`, and +`PostgreSQLAuditLog`. These match `RowStore`, `BlobStore`, and +`AuditLog`. Each one can run +either against a pooled connection grabbed for a single call, or +against a `PostgreSQLTransactionContext`'s already-open connection +when called from inside a transaction block. Every write method here +checks SQL names with `validatePSQLIdentifier`, exactly as the SQLite +backend does, before it puts together its SQL string. `PostgreSQLRowStore.query(...)` reads the schema's declared column list -for the target table (rather than issuing a bare `SELECT *`) specifically +for the target table, rather than issuing a bare `SELECT *`. This is so a generated column's PostgreSQL-computed value is decoded with the -correct `ColumnType`, and it runs every returned row's values through -`decryptedForRead` before wrapping them in a `StorageRow`. `update(...)` -and `delete(...)` both compile their predicate starting at parameter -index one past the last `SET`-clause binding, using the private -`renderPredicate(_:startIndex:bindings:)` helper, which compiles the -predicate normally and then renumbers every `$N` placeholder in reverse -order — high numbers first — specifically to avoid `$10` being -accidentally rewritten by a naive forward replacement of `$1`. -`PostgreSQLBlobStore` and `PostgreSQLAuditLog` each lazily create their -own backing table (`_storagekit_blobs`, `_storagekit_audit`) on first use -via `CREATE TABLE IF NOT EXISTS`, rather than requiring the caller's -schema declaration to know about them — the same design PersistenceKit -uses for SQLite's internal tables. `decodeAuditEvent(_:)` decodes the -audit row's required fields strictly (a decode failure there propagates -as a thrown error) while treating optional before-state fields -permissively (`try?`, because `NULL` is the expected, valid value for "no -prior state"). +right `ColumnType`. It also runs every returned row's values through +`decryptedForRead` before wrapping them in a `StorageRow`. +`update(...)` and `delete(...)` both compile their predicate starting +at the parameter spot right after the last `SET`-clause binding. They +use the private `renderPredicate(_:startIndex:bindings:)` helper. This +helper compiles the predicate as usual. It then renumbers every `$N` +placeholder in reverse order, high numbers first. It does this on +purpose, to avoid `$10` being wrongly rewritten by a simple forward +swap of `$1`. +`PostgreSQLBlobStore` and `PostgreSQLAuditLog` each lazily build their +own backing table on first use, through `CREATE TABLE IF NOT EXISTS`. +The tables are `_storagekit_blobs` and `_storagekit_audit`. Neither one needs the +caller's own schema to know about them. This is the same design +`PersistenceKit` uses for SQLite's own internal tables. +`decodeAuditEvent(_:)` decodes the audit row's required fields in a +strict way, so a decode failure there passes on as a thrown error. It +treats optional before-state fields in a looser way, with `try?`. +`NULL` is the expected, valid value for "no prior state," and that +value is always safe there. ### PostgreSQLIdentifierValidator.swift -This file provides `validatePSQLIdentifier(_:)`, the PostgreSQL-module -analogue of `SQLiteIdentifierValidator.swift`'s `validateSQLIdentifier`, -enforcing the exact same rule — first character a letter or underscore, -every later character a letter, digit, or underscore. The file's own -comment names this as one of three independent seams implementing the -identical rule: this one, the SQLite module's, and the Rust port's -`validate_sql_identifier` — three copies by necessity, since Swift -extensions cannot share a free function across two separate targets -without a shared dependency neither module otherwise needs, but each -copy is the single seam within its own module, so within a module the -rule cannot silently drift. +This file has `validatePSQLIdentifier(_:)`, the PostgreSQL-module +match to `SQLiteIdentifierValidator.swift`'s `validateSQLIdentifier`. +It enforces the exact same rule. The first character must be a +letter or underscore. Every later character must be a letter, a +digit, or an underscore. The file's own comment names this as one of three +copies that enforce this same rule on their own. The other two are +the SQLite module's copy and the Rust port's `validate_sql_identifier` +function. Three copies exist by need. Swift extensions cannot share a +free function across two separate targets without a shared dependency +neither module otherwise needs. Each copy is still the single seam +within its own module, so the rule cannot quietly drift within a +module. ## Target: PersistenceKitReplication ### ReplicationTypes.swift -This file provides the two public types every other file in this module +This file has the two public types every other file in this module builds on. `ReplicationCursor` is the opaque watermark a caller stores -and passes back into a later incremental sync: it carries the highest HLC -seen across every copied row and audit event, plus counts of rows, -audit events, and blobs written during that run. `ReplicationError` is -the closed error type specific to replication — `schemaMismatch` (source -and destination disagree on schema version or kit ID; replication -deliberately refuses to auto-migrate either side) and `storageFailure` -(an underlying `StorageError` surfaced during the copy, re-wrapped as a -plain string so `ReplicationError` itself can stay `Equatable`, which a -raw `Error` cannot generally guarantee). +and passes back into a later incremental sync. It carries the highest HLC seen across every copied row and audit +event. It also carries counts of rows, audit events, and blobs +written during that run. `ReplicationError` is +the closed error type for replication on its own. `schemaMismatch` +fires when source and destination disagree on schema version or kit +ID. Replication on purpose refuses to auto-migrate either side. +`storageFailure` wraps an underlying `StorageError` surfaced during +the copy, re-wrapped as a plain string. This lets `ReplicationError` +itself stay `Equatable`, which a raw `Error` cannot always guarantee. ### StorageReplicator.swift -This file provides `StorageReplicator`, the full-snapshot replication -primitive: `replicate(from:to:schema:)` and its two direction-named -conveniences, `flush(from:into:schema:)` (in-memory into durable storage) -and `hydrate(into:from:schema:)` (durable storage into a fresh in-memory -instance). "Full snapshot" means exactly what it says — every run copies -every row in every schema-declared table, every audit event, and every -blob, regardless of what changed since the last run. The function is -still idempotent, though: every row upsert uses the table's own primary -key as its conflict column, not the random UUID a fresh `RowHandle` would -carry, so running the same flush twice against an unchanged source -updates existing destination rows in place rather than ever inserting a -second copy. - -The implementation runs in three ordered steps. Step one is a schema -gate: source and destination must report the exact same per-kit schema +This file has `StorageReplicator`, the full-snapshot replication +tool. `replicate(from:to:schema:)` is the core function. Two +direction-named helpers sit on top of it: `flush(from:into:schema:)` +moves in-memory data into durable storage, and `hydrate(into:from:schema:)` +moves durable storage into a fresh in-memory copy. "Full snapshot" +means exactly what it says. Every run copies everything, no matter what changed since the last +run. This means every row in every schema-declared table, every audit +event, and every blob. The function still stays even on +repeat runs, though. Every row upsert uses the table's own primary +key as its conflict column, not the random UUID a fresh `RowHandle` +would carry. Running the same flush twice against an unchanged source +updates existing target rows in place, rather than ever +inserting a second copy. + +The work runs in three ordered steps. Step one is a schema gate. +Source and destination must report the exact same per-kit schema version, or the function throws `ReplicationError.schemaMismatch` -immediately rather than attempting any copy. Step two reads the entire -source — every table's rows (with any generated columns filtered out of -each row, because writing a value into a column the destination computes -itself would be rejected by both SQL backends), every audit event, and -every blob — into one `Sendable` in-memory payload, before the -destination transaction ever opens; the file's comment explains this -ordering exists so the destination is never left holding a long-lived -serializable transaction open while potentially slow source I/O runs -underneath it. Step three writes that entire payload into the destination -inside one serializable transaction, so a crash or a thrown error partway -through leaves the destination exactly as it was before the flush began. -A blob-deletion pass at the end of step three removes any destination -blob key absent from the source snapshot, closing a gap the file -documents by its fix identifier (SECFIX-WS2-PK F5): without it, a blob -deleted at the source would linger forever in the destination after every -future full-snapshot flush, because an additive-only copy never notices -an absence. +right away, rather than trying any copy at all. Step two reads the +entire source into memory: every table's rows, every audit event, +and every blob. Any generated columns are filtered out of each row +first, because writing a value into a column the target computes +on its own would be turned down by both SQL backends. All of this +goes into one `Sendable` in-memory payload, before the target +transaction ever opens. The +file's comment explains this order. It exists so the target is +never left holding a long-lived serializable transaction open while +slow source work runs underneath it. Step three writes that entire +payload into the target inside one plain transaction. A +crash or a thrown error partway through therefore leaves the +target exactly as it was before the flush began. A blob-deletion +pass at the end of step three removes any target blob key absent +from the source snapshot. This closes a gap the file names by its fix +ID, SECFIX-WS2-PK F5. Say this pass did not exist. A blob deleted at +the source would linger forever in the target, after every later +full-snapshot flush. A copy that only ever adds rows never notices +that something is missing. ### IncrementalReplicationSession.swift -This file provides `IncrementalReplicationSession`, the observer-driven -alternative to a full snapshot: instead of copying everything on every -run, it watches a source's live change stream and, on demand, syncs only -the rows and blobs that actually changed since the last sync. The file's -opening comment lays out the design choice explicitly: rather than a -durable dirty-row table written on every observed change (which would tie -this backend-agnostic module to one particular storage schema), the -session accumulates dirty identifiers purely in memory, inside two actors -— `DirtySet` for rows and `BlobDirtySet` for blobs — and re-reads each -dirty row's current state from the source at sync time rather than -trusting whatever the original change notification said, because the row -may have changed again since it was first marked dirty. If the process -restarts, the in-memory dirty-set is lost; the documented recovery is -simply to fall back to a full snapshot, which is always a safe substitute -for a lost incremental history. - -`DirtyKey` identifies one dirty row by its table name and its primary- -key values, encoded into a sortable string so that draining the dirty set -always produces the same processing order across repeated runs — this -determinism is what makes two different processes independently syncing -the same dirty set produce an identical, and therefore safely repeatable, -sequence of destination writes. `DirtySet.accumulate(_:)` extracts the -primary-key columns from an observed `TableChange`'s values dictionary, -logging and skipping (rather than crashing) if a conforming-but-buggy -backend ever emits a change missing one of those columns. `BlobDirtySet` -tracks the same idea for blobs, but because a blob change notification -already carries its own payload bytes, `BlobDirtySet` has no need to -re-read anything from the source at sync time — a `put` supersedes an -earlier `put` for the same key, and a `delete` supersedes a `put`, -last-write-wins. - -`sync(from:to:fromCursor:)` is the operation a caller actually invokes. -It repeats the same schema gate `StorageReplicator` uses, drains both -dirty sets, re-scans each dirty row from the source (turning an -now-missing row into a destination delete rather than an upsert, since -the row was deleted at the source between the original change -notification and this re-scan), fetches only the audit events newer than -the previous cursor's watermark, and commits everything inside one -serializable destination transaction — the same fail-loud, all-or-nothing -shape `StorageReplicator` uses for a full snapshot. The retry-preservation -contract is the piece unique to the incremental path: if anything fails -after the dirty sets have already been drained, the drained keys and blob -operations are restored into the sets — awaited synchronously in the -`catch` block, never handed off to a detached background task, because a -fire-and-forget restore could race an immediate caller retry and silently -lose the very keys it was trying to preserve — so a subsequent retry -re-attempts exactly the rows and blobs the failed run did not finish. +This file has `IncrementalReplicationSession`, the observer-driven +alternative to a full snapshot. Instead of copying everything on every +run, it watches a source's live change stream. On demand, it syncs +only the rows and blobs that truly changed since the last sync. The +file's opening comment lays out the design choice in plain terms. +A durable dirty-row table, written on every observed change, would +tie this backend-agnostic module to one particular storage schema. +The session avoids that. It gathers dirty IDs purely in memory +instead, inside two actors: `DirtySet` for rows and `BlobDirtySet` +for blobs. It re-reads each dirty row's current state from the source +at sync time. It does not trust whatever the original change notice +said. The row may have changed again since it was first marked dirty. +If the process restarts, the in-memory dirty set is lost. The fix for +that, as the file notes, is simple. Fall back to a full snapshot. That +always works as a safe stand-in for a lost incremental history. + +`DirtyKey` names one dirty row by its table name and its +primary-key values, packed into a sortable string. Draining the dirty +set therefore always produces the same order of work across repeated +runs. This fixed order is what makes two different processes +independently syncing the same dirty set produce an identical, and +so safely repeatable, run of target writes. +`DirtySet.accumulate(_:)` pulls the primary-key columns out of an +observed `TableChange`'s values. If a conforming-but-buggy backend +ever emits a change missing one of those columns, the function logs +it and skips it, rather than crashing. `BlobDirtySet` tracks the same idea for +blobs. A blob change notice already carries its own payload bytes, so +`BlobDirtySet` never needs to re-read anything from the source at +sync time. A `put` replaces an earlier `put` for the same key. A +`delete` replaces a `put`. The last write always wins. + +`sync(from:to:fromCursor:)` is the operation a caller actually calls. +It repeats the same schema gate `StorageReplicator` uses. It drains +both dirty sets. It re-scans each dirty row from the source. A +now-missing row turns into a target delete rather than an upsert. +This happens because the row was deleted at the source between the +original change notice and this re-scan. It fetches only the audit events newer than +the previous cursor's watermark. It commits everything inside one +plain target transaction, all in one go, the same fail-loud +shape `StorageReplicator` uses for a full snapshot. The +retry-keeping rule is the one piece unique to the incremental path. +Say anything fails after the dirty sets have already been drained. The +drained keys and blob operations go back into the sets. This restore +is awaited right there in the `catch` block. It is never handed off +to a detached background task. A fire-and-forget restore could race a +caller's immediate retry. It could quietly lose the very keys it was +trying to keep safe. A later retry then tries again on exactly the +rows and blobs the failed run did not finish. ## Rust Port and Conformance -The `rust/` directory contains a second implementation of PersistenceKit's -design for use outside the Swift/Apple ecosystem: the same closed -`StoragePredicate` algebra, the same `TypedValue` case set, the same -`Storage`/`RowStore`/`BlobStore`/`AuditLog`/`StorageObserver` trait -surface, and all three backends (`InMemoryStorage`, `SqliteStorage`, -`PostgresStorage`), plus Rust equivalents of the caching decorator, the -hashing decorator, the encryption seam, and both replication modes. The -Rust traits are synchronous (`Result`) rather than -`async`, because the Rust backends do no real asynchronous I/O of their -own; the port's `README.md` notes that Swift's `async` requirement comes -from Swift actors specifically, and a future backend that does need -asynchronous I/O could wrap its own runtime without changing the trait -shape. - -Unlike LatticeLib's two legs, the Swift and Rust sides of PersistenceKit -are not gated by a shared, byte-identical conformance-fixture corpus. -PersistenceKit's cross-platform contract is a shared protocol shape and a -shared wire format — the same `TypedValue` cases, the same predicate -tree, the same identifier-validation rule — rather than a promise that -identical input always produces bit-identical output on both legs. -Each side carries its own test suite: the Rust SQLite and PostgreSQL -backends each run a ten-test conformance battery covering schema, rows, -predicates, blobs, audit, generated columns, transactions, append-only -enforcement, and introspection, exercised directly against the Rust -trait implementations rather than cross-checked against the Swift -package's own test output. +The `rust/` folder holds a second build of `PersistenceKit`'s design, +for use outside the Swift and Apple world. It has the same closed `StoragePredicate` algebra and the same +`TypedValue` case set. It also has the same trait surface: `Storage`, +`RowStore`, `BlobStore`, `AuditLog`, and `StorageObserver`. It has all +three backends too: +`InMemoryStorage`, `SqliteStorage`, and `PostgresStorage`. It adds four things on top of these: the caching decorator, the +hashing decorator, the encryption seam, and both replication modes. +The Rust traits are plain. They use `Result` rather +than `async`. This is because the Rust backends do no real async +work of their own. The port's own `README.md` notes that Swift's `async` need comes +from Swift actors in particular. A future backend that does need +async work of its own could wrap its own runtime without changing the +trait shape. + +Unlike LatticeLib's two legs, the Swift and Rust sides of +`PersistenceKit` are not gated by a shared, byte-for-byte conformance- +fixture set. `PersistenceKit`'s cross-platform contract is a shared +protocol shape and a shared wire format: the same `TypedValue` cases, +the same predicate tree, the same identifier-checking rule. It is not +a promise that the same input always produces bit-identical output on +both legs. Each side runs its own test suite. The Rust SQLite and +PostgreSQL backends each run a ten-test conformance set covering +schema, rows, predicates, blobs, audit, generated columns, +transactions, append-only rules, and health checks. These tests run +straight against the Rust trait code, not cross-checked against the +Swift package's own test output. diff --git a/packages/kits/PersistenceKit/docs/OVERVIEW.md b/packages/kits/PersistenceKit/docs/OVERVIEW.md index da120d7..f687bd6 100644 --- a/packages/kits/PersistenceKit/docs/OVERVIEW.md +++ b/packages/kits/PersistenceKit/docs/OVERVIEW.md @@ -113,167 +113,188 @@ sources: ## What This Library Does -PersistenceKit is the storage layer for MOOTx01. MOOTx01 is an on-device AI -memory system. It stores what an AI observes over time and helps the AI -recall it later. Every fact, drawer, and audit record that a MOOTx01 estate -keeps passes through PersistenceKit before it reaches a disk or a database. -An estate is one user's complete memory store in MOOTx01. PersistenceKit -does not decide what a memory means. It decides how a memory is written, -read, and kept safe once another kit hands it over as typed rows and bytes. +PersistenceKit is the storage layer for MOOTx01. MOOTx01 is an on-device +AI memory system. It stores what an AI observes over time. It later +helps the AI recall what it observed. Every fact, drawer, and audit +record a MOOTx01 estate keeps passes through PersistenceKit first. Only +then does the record reach a disk or a database. An estate is one +user's complete memory store in MOOTx01. PersistenceKit does not decide +what a memory means. That job belongs to a higher kit. PersistenceKit +decides how a memory is written, read, and kept safe. It takes over +once another kit hands the memory over as typed rows and bytes. -PersistenceKit gives every higher kit the same four operations — insert a -row, fetch a blob, append an audit event, watch a table for changes — no -matter which physical engine sits underneath. The engine can be SQLite on a -phone, PostgreSQL on a shared server, or a plain in-memory table for a unit -test. The kit that calls PersistenceKit writes the same code either way. +PersistenceKit gives every higher kit the same four operations. A kit +can insert a row. A kit can fetch a blob. A kit can append an audit +event. A kit can watch a table for changes. This holds true no matter +which physical engine sits underneath. The engine can be SQLite on a +phone. It can be PostgreSQL on a shared server. It can also be a plain +in-memory table for a unit test. The calling kit writes the same code +in every case. ## The Problem It Solves -MOOTx01 runs in more than one place. A single estate might live only on one -iPhone, or it might run against a shared PostgreSQL server for a managed -deployment, or it might exist only for the length of a test. Without a -storage abstraction, every kit that reads or writes memory would have to -know three different database APIs and three different sets of quirks. -Worse, a kit built against SQLite could not be pointed at PostgreSQL -without a rewrite. +MOOTx01 runs in more than one place. A single estate might live only +on one iPhone. It might instead run against a shared PostgreSQL server +for a managed deployment. It might exist only for the length of a +test. Without a storage abstraction, every kit that touches memory +would need to know three different database APIs. Each API brings its +own quirks. Worse, a kit built against SQLite could not point at +PostgreSQL without a rewrite. -PersistenceKit solves this with one typed contract — the `Storage` -protocol — and three backends that all satisfy it: `PersistenceKitSQLite`, +PersistenceKit solves this with one typed contract: the `Storage` +protocol. Three backends satisfy that contract: `PersistenceKitSQLite`, `PersistenceKitPostgreSQL`, and `PersistenceKitInMemory`. A caller writes -against `Storage`, `RowStore`, `BlobStore`, and `AuditLog` once. Swapping -the backend means changing one line of configuration, not the calling code. +against `Storage`, `RowStore`, `BlobStore`, and `AuditLog` exactly once. +Swapping the backend then means changing one line of configuration. The +calling code itself never changes. -A second problem is that "one interface, three engines" is not enough by -itself. Real deployments also need at-rest encryption, tamper-evident -audit trails, safe deletion of sensitive content, a cache that never lies -about what is really stored, and a way to copy an entire estate from one -backend to another. PersistenceKit builds all of these as layers on top of -the same four-operation core, so a kit that only needs plain rows never -pays for the features it does not use, and a kit that needs encryption or -replication gets it without leaving the same protocol surface. +One interface across three engines is still not enough on its own. +Real deployments also need at-rest encryption. They need tamper-evident +audit trails. They need safe deletion of sensitive content. They need a +cache that never lies about what storage truly holds. They need a way +to copy an entire estate from one backend to another. PersistenceKit +builds all of these as layers on top of the same four-operation core. +A kit that only needs plain rows never pays for features it does not +use. A kit that needs encryption or replication gets it without +leaving the same protocol surface. PersistenceKit deliberately does not own vector similarity search. A -separate library, VectorKit, owns dense-embedding nearest-neighbor search. -PersistenceKit's job toward that workload is narrower: every backend must -support the storage needs a vector index has — storing a vector payload in -a row, reading many rows back in bulk, counting rows, and deleting rows. -This is the accommodation contract. PersistenceKit accommodates the -workload; it does not implement the search. +separate library, VectorKit, owns dense-embedding nearest-neighbor +search instead. PersistenceKit's job toward that workload stays narrow. +Every backend must support the storage needs a vector index has. It +must store a vector payload in a row. It must read many rows back in +bulk. It must count rows and delete rows. Call this the accommodation +contract. +PersistenceKit accommodates the workload. It does not implement the +search itself. ## How It Works -At the center of the library is one protocol, `Storage`. A type conforms to -`Storage` by providing four sub-stores — `RowStore` for typed rows, -`BlobStore` for raw bytes keyed by string, `AuditLog` for an append-only -history of changes, and `StorageObserver` for live change notifications — -plus schema and transaction management. `EstateConfiguration` tells a -`Storage` implementation which backend to open, which encryption mode to -use, and which cache settings apply. +One protocol sits at the center of the library: `Storage`. A type +conforms to `Storage` by providing four sub-stores. `RowStore` handles +typed rows. `BlobStore` handles raw bytes keyed by string. `AuditLog` +keeps an append-only history of changes. `StorageObserver` delivers live +change notifications. A conforming type also manages schema and +transactions. `EstateConfiguration` tells a `Storage` implementation +which backend to open. It also sets the encryption mode and the cache +settings to use. -A schema in PersistenceKit is not raw SQL. It is a typed Swift value — -`SchemaDeclaration`, built from `TableDeclaration` and `ColumnDeclaration` -values. A kit that wants to store data describes its tables once, as Swift -structs, and each backend translates that description into its own native -form: SQLite gets `CREATE TABLE` statements, PostgreSQL gets `CREATE TABLE` -statements in its own dialect, and the in-memory backend just allocates a -dictionary. Queries follow the same pattern: `StoragePredicate` is a closed -tree of comparison and bitmap operators that each backend compiles to its -own query language, so no caller ever writes a raw SQL string. +A schema in PersistenceKit is not raw SQL. It is a typed Swift value +called `SchemaDeclaration`, built from `TableDeclaration` and +`ColumnDeclaration` values. A kit that wants to store data describes +its tables once, as Swift structs. Each backend then translates that +description into its own native form. SQLite gets `CREATE TABLE` +statements. PostgreSQL gets `CREATE TABLE` statements in its own +dialect. The in-memory backend just allocates a dictionary. Queries +follow the same pattern. `StoragePredicate` is a closed tree of +comparison and bitmap operators. Each backend compiles that tree to its +own query language. No caller ever writes a raw SQL string. -Three backends satisfy `Storage`. `PersistenceKitSQLite` is the on-device -engine — one file per estate, encrypted at rest through a vendored -SQLCipher build when the estate asks for whole-database encryption. -`PersistenceKitPostgreSQL` is the server engine, built on the `postgres-nio` -client, with one PostgreSQL schema per estate so many estates can share one -database server without their tables colliding. `PersistenceKitInMemory` is -the test and prototyping engine — no file, no network, gone when the -process exits. +Three backends satisfy `Storage`. `PersistenceKitSQLite` is the +on-device engine. It uses one file per estate, encrypted at rest +through a vendored SQLCipher build when an estate asks for +whole-database encryption. `PersistenceKitPostgreSQL` is the server +engine, built on the `postgres-nio` client. It gives each estate its +own PostgreSQL schema, so many estates can share one database server +without their tables colliding. `PersistenceKitInMemory` is the test +and prototyping engine. It keeps no file and opens no network +connection, and it disappears when the process exits. -On top of the core protocol, PersistenceKit layers decorators and -free-function toolkits that any backend can use without changing its own -code: +The core protocol also carries decorators and free-function toolkits. +Any backend can use these without changing its own code. - `CachingRowStore` wraps a `RowStore` with an in-memory hot tier. Rows - above a configured sensitivity level are never cached, and the cache - guarantees that every read returns exactly what the backing store would - have returned — caching only changes speed, never correctness. -- `HashingRowStore` wraps a `RowStore` and computes a content hash on every - write to a table marked hashable, then reports the write up a parent - chain so a Merkle-style integrity tree can stay current without a full - rescan. -- `RowCrypto` and its write/read seam functions apply per-row AES-GCM - encryption to a table's `content` column when an estate is configured for - row-level encryption, independent of whichever backend stores the row. + above a configured sensitivity level are never cached. The cache + guarantees that every read returns exactly what the backing store + would return. Caching only changes speed, never correctness. +- `HashingRowStore` wraps a `RowStore` and computes a content hash on + every write to a table marked hashable. It then reports the write up + a parent chain, so a Merkle-style integrity tree can stay current + without a full rescan. +- `RowCrypto` and its write and read seam functions apply per-row + AES-GCM encryption to a table's `content` column. This runs whenever + an estate is configured for row-level encryption, independent of + whichever backend stores the row. - `ErasureLedger` and `ErasureOverlay` implement right-to-be-forgotten - deletion: the ledger records that a piece of content was erased, and the - overlay nulls that content out of every read afterward, failing closed - (dropping the row) if the erasure check itself cannot be completed. -- `SnapshotRegistry` and `GCPin` let a kit take a named, attested snapshot - of the estate's state and pin the oldest live snapshot's timestamp so a - later garbage-collection pass never deletes data a snapshot still needs. -- `PersistenceKitReplication` copies an entire estate's rows, audit events, - and blobs from one `Storage` to another, either as a one-shot full - snapshot or as an incremental sync driven by a live change-observation - session. + deletion. The ledger records that a piece of content was erased. The + overlay nulls that content out of every read afterward. It fails + closed, dropping the row, if the erasure check itself cannot + complete. +- `SnapshotRegistry` and `GCPin` let a kit take a named, attested + snapshot of the estate's state. They also pin the oldest live + snapshot's timestamp, so a later garbage-collection pass never + deletes data a snapshot still needs. +- `PersistenceKitReplication` copies an entire estate's rows, audit + events, and blobs from one `Storage` to another. It can run this + copy as a one-shot full snapshot or as an incremental sync driven by + a live change-observation session. ## How the Pieces Fit -Figure 1 shows the library's topology — its major parts and how a write or -a read moves between them. +Figure 1 shows the library's topology. It shows the major parts and +how a write or a read moves between them. ![Figure 1. Topology of PersistenceKit](topology.svg) -*Figure 1. Topology of PersistenceKit. A calling kit reaches storage only -through the four core protocols. Decorators wrap a backend's `RowStore` -transparently. Cross-cutting concerns (encryption, erasure, snapshots) sit -beside the core surface and are invoked by name, not by inheritance. -Replication reads one `Storage` and writes another. Dashed regions mark -backend-owned engines and the cross-process encryption key store.* +*Figure 1. Topology of PersistenceKit. A calling kit reaches storage +only through the four core protocols. Decorators wrap a backend's +`RowStore` transparently. Cross-cutting concerns sit beside the core +surface. Examples include encryption, erasure, and snapshots. Each is +invoked by name, not by inheritance. Replication reads one `Storage` and writes +another. Dashed regions mark backend-owned engines and the +cross-process encryption key store.* -A typical write enters through a kit-held reference to `any RowStore`. If -the estate has caching enabled, that reference is really a -`CachingRowStore` wrapping the real backend's row store; if hash-on-write -is configured, a `HashingRowStore` may sit in front of that. The write -descends through zero or more decorators until it reaches the concrete -backend — `SQLiteRowStore`, `PostgreSQLRowStore`, or `InMemoryRowStore` — -which applies the row-encryption seam (a no-op unless the estate uses row -encryption), checks the content/keyID invariant, validates every SQL -identifier it is about to interpolate, and commits the row. Every backend -notifies its `StorageObserver` afterward, so decorators, cache -invalidators, and replication sessions downstream can react. +A typical write enters through a kit-held reference to `any RowStore`. +If the estate has caching enabled, that reference is really a +`CachingRowStore` wrapping the real backend's row store. If hash-on-write +is also configured, a `HashingRowStore` may sit in front of that. The +write descends through zero or more decorators until it reaches the +concrete backend: `SQLiteRowStore`, `PostgreSQLRowStore`, or +`InMemoryRowStore`. That backend applies the row-encryption seam, a +no-op unless the estate uses row encryption. It checks the +content-and-keyID invariant. It validates every SQL identifier it is +about to interpolate. It then commits the row. Every backend notifies +its `StorageObserver` afterward. Decorators, cache invalidators, and +replication sessions downstream can then react. -A typical read is the same path in reverse, with one addition: any content -subject to the right-to-be-forgotten contract passes through -`ErasureOverlay.apply(...)` after the backend query returns, so an erased -row's content columns come back `nil` even though its skeleton (its ID, -timestamps, and lattice anchors) survives for referential integrity. +A typical read follows the same path in reverse, with one addition. +Any content subject to the right-to-be-forgotten contract passes +through `ErasureOverlay.apply(...)` after the backend query returns. An +erased row's content columns then come back `nil`. The row's skeleton +still survives for referential integrity. The skeleton is its ID, +timestamps, and lattice anchors. -Encryption, erasure, and snapshots are not separate storage engines. They -are pure functions and small actors that any backend calls at the right -moment — `encryptedForWrite`/`decryptedForRead` around a write or a read, -`ErasureLedgerOps.isErased` inside the overlay, `GCPin.minimumRetainableHlc` -before a vacuum. This design keeps the cross-cutting logic in one place -(`PersistenceKit` core) instead of duplicating it inside every backend, and -it is why the SQLite and PostgreSQL backends produce byte-compatible -encrypted envelopes: they call the same `RowCrypto` code. +Encryption, erasure, and snapshots are not separate storage engines. +They are pure functions and small actors that any backend calls at the +right moment. `encryptedForWrite` and `decryptedForRead` run around a +write or a read. `ErasureLedgerOps.isErased` runs inside the overlay. +`GCPin.minimumRetainableHlc` runs before a vacuum. This design keeps +the cross-cutting logic in one place, the `PersistenceKit` core, +instead of duplicating it inside every backend. It is also why the +SQLite and PostgreSQL backends produce byte-compatible encrypted +envelopes. Both call the same `RowCrypto` code. ## What Ships in the Package -The package ships five Swift targets. `PersistenceKit` is the core: the -`Storage`/`RowStore`/`BlobStore`/`AuditLog`/`StorageObserver` protocols, the -typed value and predicate algebra, schema declarations, the caching and -hashing decorators, and the encryption, erasure, snapshot, and telemetry -toolkits. `PersistenceKitInMemory` is the test backend. `PersistenceKitSQLite` -is the on-device backend, built against a vendored `SQLCipher` C target so -whole-database encryption never depends on the host OS's system SQLite. -`PersistenceKitPostgreSQL` is the server backend, built on `postgres-nio` and -`NIOSSL`. `PersistenceKitReplication` is the estate-copying primitive, -depending only on the core protocol surface so it works against any pair of -conforming backends. +The package ships five Swift targets. `PersistenceKit` is the core. It +holds the `Storage`, `RowStore`, `BlobStore`, `AuditLog`, and +`StorageObserver` protocols. It holds the typed value and predicate +algebra, the schema declarations, and the caching and hashing +decorators. It also holds the encryption, erasure, snapshot, and +telemetry toolkits. `PersistenceKitInMemory` is the test backend. +`PersistenceKitSQLite` is the on-device backend, built against a +vendored `SQLCipher` C target. Whole-database encryption never depends +on the host operating system's own SQLite because of this. +`PersistenceKitPostgreSQL` is the server backend, built on `postgres-nio` +and `NIOSSL`. `PersistenceKitReplication` is the estate-copying +primitive. It depends only on the core protocol surface, so it works +against any pair of conforming backends. -The package also ships a Rust port in `rust/`, mirroring the same trait -surface, the same closed predicate algebra, and the same three backends, -for use outside the Swift/Apple ecosystem. The two ports share design intent -but are not conformance-gated against each other the way LatticeLib's two -legs are: PersistenceKit's cross-platform contract is the shared protocol -shape and wire format, not byte-identical output from a shared corpus. +The package also ships a Rust port in `rust/`. That port mirrors the +same trait surface. It mirrors the same closed predicate algebra. It +mirrors the same three backends. It exists for use outside the Swift +and Apple ecosystem. The two ports share design intent. They are not +conformance-gated against each +other the way LatticeLib's two legs are. PersistenceKit's cross-platform +contract is the shared protocol shape and wire format. It is not a +promise of byte-identical output from a shared corpus. diff --git a/packages/kits/PersistenceKit/docs/topology.svg b/packages/kits/PersistenceKit/docs/topology.svg index 74f1ccb..b4c6f53 100644 --- a/packages/kits/PersistenceKit/docs/topology.svg +++ b/packages/kits/PersistenceKit/docs/topology.svg @@ -99,12 +99,12 @@ sources: Apple-only key custody - KeychainKeyStore — per-estate 256-bit key + KeychainKeyStore: per-estate 256-bit key - Cross-cutting toolkits — called by name, not by inheritance + Cross-cutting toolkits: called by name, not by inheritance RowCrypto @@ -129,7 +129,7 @@ sources: - PersistenceKitReplication — operates on any two Storage instances + PersistenceKitReplication: operates on any two Storage instances Source estate @@ -155,7 +155,7 @@ sources: Both paths read the source, then write the - destination inside one serializable transaction — + destination inside one serializable transaction: a crash mid-flush leaves the destination unchanged. diff --git a/packages/kits/QueueKit/docs/AGENT_MAP.md b/packages/kits/QueueKit/docs/AGENT_MAP.md index 659f0f8..e0597dd 100644 --- a/packages/kits/QueueKit/docs/AGENT_MAP.md +++ b/packages/kits/QueueKit/docs/AGENT_MAP.md @@ -27,153 +27,153 @@ sources: blob: cf2b270b9c60da34f7a25c016f8c18b6ba6149e4 --- -# AGENT_MAP — QueueKit +# AGENT_MAP: QueueKit PURPOSE: general-purpose durable work queue. Producer→`send`/`writeBatch`→backend; consumer→`drain`/`watch`→claims jobs; consumer→`reply`→terminal completion. Two swappable backends (`FilesystemBackend` POSIX maildir, `PersistenceKitBackend` shared SQLite table) behind one `QueueBackend` protocol and one public `QueueKit` facade. Stream-scoped ops (ADR-021 Decision 7 / T1) let multiple consumers share one queue without stealing each other's jobs. -DEPS: imports SubstrateTypes (HLC/HLCGenerator — hybrid logical clock), PersistenceKit (Storage, rowStore, transaction, observer — PersistenceKitBackend only), IntellectusLib (self-report telemetry, DECISION_LIFT_PACKAGE_SWIFT_RULE_2026-05-28). ConvergenceKit is application-layer composition and deliberately NOT a dependency (spec §11). Imported by: none within this repo checkout; source comments name CorpusKit and GeniusLocusKit as external mount-time consumers (not present in moot-system). Rust port in rust/ (8 files, ~3k lines) mirrors facade + both backends, PersistenceKitBackend behind `persistencekit` cargo feature. Python port in python/queuekit/ mirrors FilesystemBackend only (spec §2: Python has no PersistenceKit backend). All three gated by shared conformance fixtures in Tests/QueueKitTests/Fixtures/*.json. +DEPS: imports SubstrateTypes (HLC/HLCGenerator: hybrid logical clock), PersistenceKit (Storage, rowStore, transaction, observer: PersistenceKitBackend only), IntellectusLib (self-report telemetry, DECISION_LIFT_PACKAGE_SWIFT_RULE_2026-05-28). ConvergenceKit is application-layer composition and deliberately NOT a dependency (spec §11). Imported by: none within this repo checkout; source comments name CorpusKit and GeniusLocusKit as external mount-time consumers (not present in moot-system). Rust port in rust/ (8 files, ~3k lines) mirrors facade + both backends, PersistenceKitBackend behind `persistencekit` cargo feature. Python port in python/queuekit/ mirrors FilesystemBackend only (spec §2: Python has no PersistenceKit backend). All three gated by shared conformance fixtures in Tests/QueueKitTests/Fixtures/*.json. ENTRY POINTS (most callers need only these): -- QueueKit.swift:52 `QueueKit.init(root:hlcGenerator:) throws` — mount FilesystemBackend at root, create maildir, clean stale tmp/ -- QueueKit.swift:65 `QueueKit.init(backend:root:)` — mount explicit backend (PersistenceKitBackend, or a test double) -- QueueKit.swift:72 `send(_ job: Job) async throws` — enqueue one job -- QueueKit.swift:85 `drain() async throws -> [(job: Job, sessionID: SessionID)]` — claim all available jobs -- QueueKit.swift:129 `reply(to:status:artifacts:) async throws` — mark one claimed job terminal -- QueueKit.swift:123 `watch(handler:) async throws` — subscribe a per-job handler +- QueueKit.swift:52 `QueueKit.init(root:hlcGenerator:) throws`: mount FilesystemBackend at root, create maildir, clean stale tmp/ +- QueueKit.swift:65 `QueueKit.init(backend:root:)`: mount explicit backend (PersistenceKitBackend, or a test double) +- QueueKit.swift:72 `send(_ job: Job) async throws`: enqueue one job +- QueueKit.swift:85 `drain() async throws -> [(job: Job, sessionID: SessionID)]`: claim all available jobs +- QueueKit.swift:129 `reply(to:status:artifacts:) async throws`: mark one claimed job terminal +- QueueKit.swift:123 `watch(handler:) async throws`: subscribe a per-job handler ## Symbol Table -### Facade — QueueKit.swift -- :28 `staleTmpThreshold = 5 * 60` — tmp/ files older than this on init are deleted (crash between write and rename; never visible in new/, safe to remove) +### Facade: QueueKit.swift +- :28 `staleTmpThreshold = 5 * 60`: tmp/ files older than this on init are deleted (crash between write and rename; never visible in new/, safe to remove) - :30 `final class QueueKit: Sendable` -- :39 `latencyWindow` — nonisolated(unsafe); caller (GLK scheduler actor / serial test) owns exclusivity -- :47 `estateTag: String = "unknown"` — nonisolated(unsafe); set ONCE at mount before any drain(), never again -- :52 `init(root:hlcGenerator:) throws` — see ENTRY POINTS -- :65 `init(backend:root:)` — see ENTRY POINTS -- :72 `send(_:)` / :81 `send(batch:) -> Int` — writeBatch twin; FS backend fsyncs new/ ONCE for the batch -- :85 `drain()` / :108 `drain(stream:)` — claim; wraps backend call with reportQueueStats timing -- :123 `watch(handler:)` — forwards to backend.watch -- :129 `reply(to:status:artifacts:)` — guards status.isTerminal, throws .invalidTerminalStatus otherwise -- :148 `reply(session:status:) -> Int` — fast path ONLY on PersistenceKitBackend (`as?` cast); returns 0 on FilesystemBackend (caller falls back to per-job reply) -- :171 `reply(batch:) -> Int` — routes to backend.completeBatch; FS backend's one-scan/one-fsync path -- :193 `reclaimInFlight(stream:) -> Int` — GATE: call ONLY immediately after DrainLease.tryAcquire SUCCEEDED for stream; fast path ONLY on PersistenceKitBackend, returns 0 otherwise -- :204 `inFlight()` / :214 `pendingCount()` / :224 `pendingCount(stream:)` — read-only depth probes; pendingCount()+inFlight().count = total outstanding work -- :261 `awaitDrain(pollInterval: .milliseconds(20), timeout: .seconds(30)) throws` — polls both frontiers; returns promptly if already empty; throws .drainTimeout, never hangs -- :292 `awaitDrain(stream:pollInterval:timeout:)` — stream-scoped twin; global awaitDrain would block forever on OTHER streams' jobs a scoped drainer never claims +- :39 `latencyWindow`: nonisolated(unsafe); caller (GLK scheduler actor / serial test) owns exclusivity +- :47 `estateTag: String = "unknown"`: nonisolated(unsafe); set ONCE at mount before any drain(), never again +- :52 `init(root:hlcGenerator:) throws`: see ENTRY POINTS +- :65 `init(backend:root:)`: see ENTRY POINTS +- :72 `send(_:)` / :81 `send(batch:) -> Int`: writeBatch twin; FS backend fsyncs new/ ONCE for the batch +- :85 `drain()` / :108 `drain(stream:)`: claim; wraps backend call with reportQueueStats timing +- :123 `watch(handler:)`: forwards to backend.watch +- :129 `reply(to:status:artifacts:)`: guards status.isTerminal, throws .invalidTerminalStatus otherwise +- :148 `reply(session:status:) -> Int`: fast path ONLY on PersistenceKitBackend (`as?` cast); returns 0 on FilesystemBackend (caller falls back to per-job reply) +- :171 `reply(batch:) -> Int`: routes to backend.completeBatch; FS backend's one-scan/one-fsync path +- :193 `reclaimInFlight(stream:) -> Int`: GATE: call ONLY immediately after DrainLease.tryAcquire SUCCEEDED for stream; fast path ONLY on PersistenceKitBackend, returns 0 otherwise +- :204 `inFlight()` / :214 `pendingCount()` / :224 `pendingCount(stream:)`: read-only depth probes; pendingCount()+inFlight().count = total outstanding work +- :261 `awaitDrain(pollInterval: .milliseconds(20), timeout: .seconds(30)) throws`: polls both frontiers; returns promptly if already empty; throws .drainTimeout, never hangs +- :292 `awaitDrain(stream:pollInterval:timeout:)`: stream-scoped twin; global awaitDrain would block forever on OTHER streams' jobs a scoped drainer never claims - :314 `completed(streamID:)` - :322 `maildirSubdirs = ["tmp","new","cur","done"]` -- :324 `ensureMaildir(root:) throws` — creates 4 subdirs if absent -- :341 `cleanStaleTmpFiles(root:) throws` — deletes tmp/ entries older than staleTmpThreshold +- :324 `ensureMaildir(root:) throws`: creates 4 subdirs if absent +- :341 `cleanStaleTmpFiles(root:) throws`: deletes tmp/ entries older than staleTmpThreshold -### Backend contract — QueueBackend.swift +### Backend contract: QueueBackend.swift - :9 `protocol QueueBackend: Sendable` -- :10 `write(_:)`, :23 `drainAvailable()`, :30 `watch(handler:)`, :34 `complete(_:status:artifacts:)`, :52 `inFlight()`, :54 `completed(streamID:)` — must-implement core -- :21 `writeBatch(_:) -> Int` — default (extension :92-102) loops write; override for shared-cost batching -- :28 `pendingCount()` — depth probe used by telemetry -- :48 `completeBatch(_:) -> Int` — default (extension :104-116) loops complete; override for shared-cost batching -- :66 `drainAvailable(stream:)` — default (extension :76-84) claims ALL streams then filters — steals other streams' jobs, MUST override for real stream isolation -- :73 `pendingCount(stream:)` — default (extension :86-89) delegates to all-streams pendingCount() (wrong count unless overridden) - -### Wire types — Job.swift -- :22 `struct JobID: RawRepresentable` — :31 `generate()` = UUID → 32 lowercase hex, no hyphens (avoids colliding with maildir filename hyphen separator) -- :51 `struct StreamID: RawRepresentable` — workload/lane name -- :68 `struct SessionID: RawRepresentable` — :75 `mint()` = one per drain() call; groups a claimed batch +- :10 `write(_:)`, :23 `drainAvailable()`, :30 `watch(handler:)`, :34 `complete(_:status:artifacts:)`, :52 `inFlight()`, :54 `completed(streamID:)`: must-implement core +- :21 `writeBatch(_:) -> Int`: default (extension :92-102) loops write; override for shared-cost batching +- :28 `pendingCount()`: depth probe used by telemetry +- :48 `completeBatch(_:) -> Int`: default (extension :104-116) loops complete; override for shared-cost batching +- :66 `drainAvailable(stream:)`: default (extension :76-84) claims ALL streams then filters: steals other streams' jobs, MUST override for real stream isolation +- :73 `pendingCount(stream:)`: default (extension :86-89) delegates to all-streams pendingCount() (wrong count unless overridden) + +### Wire types: Job.swift +- :22 `struct JobID: RawRepresentable`: :31 `generate()` = UUID → 32 lowercase hex, no hyphens (avoids colliding with maildir filename hyphen separator) +- :51 `struct StreamID: RawRepresentable`: workload/lane name +- :68 `struct SessionID: RawRepresentable`: :75 `mint()` = one per drain() call; groups a claimed batch - :80 `struct ToolName: RawRepresentable` -- :85 `enum ArtifactRef` — file_path|commit_hash|signal_file|trajectory_step_id; explicit `type` field on wire (:91 CodingKeys) -- :131 `indirect enum CodableValue` — open caller extensions blob: null/bool/int/double/string/array/object; round-trips verbatim -- :167 `struct Job: Identifiable, Hashable` — id/streamID/submittedAt(HLC)/priority/payload/extensions -- :193 `Job` CodingKeys — PINNED snake_case wire keys: id, stream_id, submitted_at{physical_time,logical_count,node_id}, priority, payload(base64url), extensions -- :254 `struct MissionContext` — ONLY type using default camelCase Codable synthesis (not part of pinned job envelope) -- :236 `WireFormat.base64urlEncode(_:)` / :245 `base64urlDecode(_:)` — RFC 4648 §5, no padding -- :289 `WireFormat.filename(for:) -> String` — `{sortableHLC}-{streamID}-{jobID}` -- :296 `WireFormat.sortableHLC(_:)` — `{phys:016d}-{logical:08d}-{node_unsigned:010d}`; zero-padding makes lexicographic filename sort == HLC sort -- :307 `WireFormat.encoder` — sortedKeys + withoutEscapingSlashes — byte-stable across Swift/Rust/Python (conformance requirement) +- :85 `enum ArtifactRef`: file_path|commit_hash|signal_file|trajectory_step_id; explicit `type` field on wire (:91 CodingKeys) +- :131 `indirect enum CodableValue`: open caller extensions blob: null/bool/int/double/string/array/object; round-trips verbatim +- :167 `struct Job: Identifiable, Hashable`: id/streamID/submittedAt(HLC)/priority/payload/extensions +- :193 `Job` CodingKeys: PINNED snake_case wire keys: id, stream_id, submitted_at{physical_time,logical_count,node_id}, priority, payload(base64url), extensions +- :254 `struct MissionContext`: ONLY type using default camelCase Codable synthesis (not part of pinned job envelope) +- :236 `WireFormat.base64urlEncode(_:)` / :245 `base64urlDecode(_:)`: RFC 4648 §5, no padding +- :289 `WireFormat.filename(for:) -> String`: `{sortableHLC}-{streamID}-{jobID}` +- :296 `WireFormat.sortableHLC(_:)`: `{phys:016d}-{logical:08d}-{node_unsigned:010d}`; zero-padding makes lexicographic filename sort == HLC sort +- :307 `WireFormat.encoder`: sortedKeys + withoutEscapingSlashes: byte-stable across Swift/Rust/Python (conformance requirement) - :313 `WireFormat.decoder` -- :321 `struct SignalFile` — job_id/status/artifacts/completed_at; same pinned-key discipline as Job +- :321 `struct SignalFile`: job_id/status/artifacts/completed_at; same pinned-key discipline as Job ### Status/errors -- ObservationStatus.swift:9 `enum ObservationStatus: String` — running|done|done_with_concerns|needs_context|blocked -- ObservationStatus.swift:16 `isTerminal` — only .running is false; gates every reply/complete call -- QueueError.swift:7 `enum QueueError: Error` — directoryCreationFailed, writeFailed, renameFailed, decodingFailed, unknownTool, jobNotFound, watcherFailed, staleTmpFile, backendUnavailable, invalidTerminalStatus, drainTimeout(pending:inFlight:), invalidIdentifier(id:reason:) +- ObservationStatus.swift:9 `enum ObservationStatus: String`: running|done|done_with_concerns|needs_context|blocked +- ObservationStatus.swift:16 `isTerminal`: only .running is false; gates every reply/complete call +- QueueError.swift:7 `enum QueueError: Error`: directoryCreationFailed, writeFailed, renameFailed, decodingFailed, unknownTool, jobNotFound, watcherFailed, staleTmpFile, backendUnavailable, invalidTerminalStatus, drainTimeout(pending:inFlight:), invalidIdentifier(id:reason:) ### FilesystemBackend.swift (POSIX maildir; semantics derived from Postfix deliver_maildir()) - :43 `final class FilesystemBackend: QueueBackend, @unchecked Sendable` -- :32 `private final class HLCBox` — NSLock-guarded HLCGenerator (value type; concurrent send() must serialize) -- :74 `validateIdentifier(_:) throws` (private static) — rejects empty/`.`/`..`/path-separator/control-char; MUST run before ANY path is built from a caller identifier (path-traversal guard, CAND-023, mirrored in Rust+Python) -- :96 `write(_:)` — validate → O_CREAT|O_EXCL write to tmp/, mode 0600, fsync, rename to new/, fsync new/ dir -- :129 `writeBatch(_:) -> Int` — per-job write+rename WITHOUT fsync, ONE fsync of new/ dir at end; AT-LEAST-ONCE (reindex re-derives from estate on crash) -- :160 `atomicWriteAndRename(data:tmpPath:newPath:newDir:) throws` (static) — the full durable write; ENOENT rename retried once, EXDEV (tmp/new on different filesystems) fails hard -- :273 `writeAndRenameNoFsync(data:tmpPath:newPath:) throws` (static) — batch primitive, no per-file/dir fsync -- :340 `pendingCount()` — file count in new/ -- :363 `drainAvailable(stream:)` — reads+decodes each new/ file BEFORE claiming (read ≠ claim); only rename-claims if job.streamID == stream; non-matching files left untouched in new/ (no steal-then-unclaim) -- :428 `pendingCount(stream:)` — decodes each new/ file, non-claiming -- :445 `drainAvailable()` — rename ALL new/→cur/ first, then decode; undecodable file → moved to done/ (poison, never retried) -- :516 `reclaimInFlight() -> Int` — GATE: call ONLY at mount before any drain session is live (fresh process owns no in-flight work by construction); unscoped (cur/ has no per-stream separation) -- :549 `watch(handler:)` — Watcher.watchNewDirectory → drainAvailable() → handler per job -- :567 `complete(_:status:artifacts:)` — find file in cur/ by `-{jobID}` suffix; writes SignalFile to done/ BEFORE renaming job file (crash between = job still reclaimable, never orphaned-signal-no-job) -- :631 `completeBatch(_:) -> Int` — ONE cur/ scan → jobID→filename index (keys on suffix after LAST `-`; JobIDs are 32 dashless hex so this is unambiguous); per-completion write+rename without fsync; ONE done/ dir fsync at end -- :701 `listJobs(in:filter:)` (private) — skips `.signal` sidecars, decodes, optional stream filter - -### PersistenceKitBackend.swift (spec §10 v1.1 — 5 invariants enforced) +- :32 `private final class HLCBox`: NSLock-guarded HLCGenerator (value type; concurrent send() must serialize) +- :74 `validateIdentifier(_:) throws` (private static): rejects empty/`.`/`..`/path-separator/control-char; MUST run before ANY path is built from a caller identifier (path-traversal guard, CAND-023, mirrored in Rust+Python) +- :96 `write(_:)`: validate → O_CREAT|O_EXCL write to tmp/, mode 0600, fsync, rename to new/, fsync new/ dir +- :129 `writeBatch(_:) -> Int`: per-job write+rename WITHOUT fsync, ONE fsync of new/ dir at end; AT-LEAST-ONCE (reindex re-derives from estate on crash) +- :160 `atomicWriteAndRename(data:tmpPath:newPath:newDir:) throws` (static): the full durable write; ENOENT rename retried once, EXDEV (tmp/new on different filesystems) fails hard +- :273 `writeAndRenameNoFsync(data:tmpPath:newPath:) throws` (static): batch primitive, no per-file/dir fsync +- :340 `pendingCount()`: file count in new/ +- :363 `drainAvailable(stream:)`: reads+decodes each new/ file BEFORE claiming (read ≠ claim); only rename-claims if job.streamID == stream; non-matching files left untouched in new/ (no steal-then-unclaim) +- :428 `pendingCount(stream:)`: decodes each new/ file, non-claiming +- :445 `drainAvailable()`: rename ALL new/→cur/ first, then decode; undecodable file → moved to done/ (poison, never retried) +- :516 `reclaimInFlight() -> Int`: GATE: call ONLY at mount before any drain session is live (fresh process owns no in-flight work by construction); unscoped (cur/ has no per-stream separation) +- :549 `watch(handler:)`: Watcher.watchNewDirectory → drainAvailable() → handler per job +- :567 `complete(_:status:artifacts:)`: find file in cur/ by `-{jobID}` suffix; writes SignalFile to done/ BEFORE renaming job file (crash between = job still reclaimable, never orphaned-signal-no-job) +- :631 `completeBatch(_:) -> Int`: ONE cur/ scan → jobID→filename index (keys on suffix after LAST `-`; JobIDs are 32 dashless hex so this is unambiguous); per-completion write+rename without fsync; ONE done/ dir fsync at end +- :701 `listJobs(in:filter:)` (private): skips `.signal` sidecars, decodes, optional stream filter + +### PersistenceKitBackend.swift (spec §10 v1.1: 5 invariants enforced) - :34 `queueKitTableName = "queuekit_jobs"` -- :36 `enum QueueKitSchema` — :40 `declaration()` — columns incl. status/session_id/signal_status/artifacts; :62 `appendOnly: false` MUST stay false (invariant 5); indices: status, (status,phys,logical,node) claim-order, (stream_id,status) +- :36 `enum QueueKitSchema`: :40 `declaration()`: columns incl. status/session_id/signal_status/artifacts; :62 `appendOnly: false` MUST stay false (invariant 5); indices: status, (status,phys,logical,node) claim-order, (stream_id,status) - :84 `final class PersistenceKitBackend: QueueBackend, @unchecked Sendable` -- :91 `openSchema(on:) throws` (static) — opens schema on a Storage -- :97 `write(_:)` — BARE rowStore.insert, status="new", NO transaction (invariant 1) -- :143 `writeBatch(_:) -> Int` — ONE `.readCommitted` transaction wrapping N inserts; readCommitted safe because new-row inserts cannot conflict with the claim's serializable new→cur UPDATE (claim sees pre-batch snapshot or post-batch rows, never a torn view) -- :172 `pendingCount()` — COUNT WHERE status='new' -- :189 `drainAvailable(stream:)` / :248 `drainAvailable()` — SINGLE-PASS CLAIM (invariant 3): one `.serializable` txn = guarded bulk UPDATE new→cur under fresh session_id, THEN SELECT rows WHERE session_id=session (same txn, sees own writes); O(N) not O(N²); session-keyed readback prevents cross-drainer double-count -- :237 `pendingCount(stream:)` — COUNT WHERE status='new' AND stream_id=? -- :324 `completeSession(_:status:) -> Int` — NOT on QueueBackend protocol (concrete-only); ONE guarded UPDATE WHERE session_id=? AND status='cur'; backs QueueKit.reply(session:status:) -- :350 `watch(handler:)` — storage.observer.observe(insert) per invariant 2 (event = wake ONLY, never read job fields from event); drains once BEFORE first await (catches pre-subscription inserts), then per wake -- :378 `drainUntilEmpty` (private static) — loops drainAvailable() until a pass claims 0; absorbs coalesced/dropped observer wakes; a claim ERROR must propagate (not `try? ?? []`) or a fault masquerades as "queue empty" -- :400 `complete(_:status:artifacts:)` — guarded UPDATE WHERE id=? AND status='cur'; affected==0 → throws .jobNotFound -- :451 `reclaimInFlight(stream:) -> Int` — GATE identical to FilesystemBackend.reclaimInFlight: ONLY after DrainLease.tryAcquire success for stream; resets cur→new, clears session_id, stream-scoped (shared table holds multiple streams) +- :91 `openSchema(on:) throws` (static): opens schema on a Storage +- :97 `write(_:)`: BARE rowStore.insert, status="new", NO transaction (invariant 1) +- :143 `writeBatch(_:) -> Int`: ONE `.readCommitted` transaction wrapping N inserts; readCommitted safe because new-row inserts cannot conflict with the claim's serializable new→cur UPDATE (claim sees pre-batch snapshot or post-batch rows, never a torn view) +- :172 `pendingCount()`: COUNT WHERE status='new' +- :189 `drainAvailable(stream:)` / :248 `drainAvailable()`: SINGLE-PASS CLAIM (invariant 3): one `.serializable` txn = guarded bulk UPDATE new→cur under fresh session_id, THEN SELECT rows WHERE session_id=session (same txn, sees own writes); O(N) not O(N²); session-keyed readback prevents cross-drainer double-count +- :237 `pendingCount(stream:)`: COUNT WHERE status='new' AND stream_id=? +- :324 `completeSession(_:status:) -> Int`: NOT on QueueBackend protocol (concrete-only); ONE guarded UPDATE WHERE session_id=? AND status='cur'; backs QueueKit.reply(session:status:) +- :350 `watch(handler:)`: storage.observer.observe(insert) per invariant 2 (event = wake ONLY, never read job fields from event); drains once BEFORE first await (catches pre-subscription inserts), then per wake +- :378 `drainUntilEmpty` (private static): loops drainAvailable() until a pass claims 0; absorbs coalesced/dropped observer wakes; a claim ERROR must propagate (not `try? ?? []`) or a fault masquerades as "queue empty" +- :400 `complete(_:status:artifacts:)`: guarded UPDATE WHERE id=? AND status='cur'; affected==0 → throws .jobNotFound +- :451 `reclaimInFlight(stream:) -> Int`: GATE identical to FilesystemBackend.reclaimInFlight: ONLY after DrainLease.tryAcquire success for stream; resets cur→new, clears session_id, stream-scoped (shared table holds multiple streams) - :468 `inFlight()` / :472 `completed(streamID:)` → :480 `listJobs(status:streamID:)` (private) -- :500 `decodeRow(_:)` (private static) — malformed extensions JSON → falls back to empty dict, not a decode failure - -### DrainLease.swift — stream-keyed heartbeat-TTL lock (NOT PID-liveness; portable, no FFI) -- :49 `struct DrainLease: Sendable` — pure value type; lets Corpus's non-isolated deinit call release() directly -- :55 `owner` — `"pid-{pid}-{instanceToken}"`; nonce defeats PID-reuse impersonation after crash -- :59 `ttl` — default 15s = Rust DRAIN_LEASE_TTL_SECS -- :64 `heartbeatInterval = 5` (static) — heartbeat cadence, well inside ttl -- :73 `init(directory:stream:instanceToken:ttl:)` — lease file = `/.drain.lease`; stream name sanitized to alnum/_/- (else `_`) -- :93 `tryAcquire(now:) -> Bool` — false only if fresh lease held by ANOTHER owner; else (re)claims + re-reads to resolve write races (atomic replace = last-writer-wins) -- :110 `heartbeat(now:)` — refresh; does NOT check ownership, caller's responsibility -- :119 `isHeldByOther(now:) -> Bool` — freshness check without attempting acquire -- :127 `release()` — deletes file ONLY if still owned by self (never undoes a takeover) +- :500 `decodeRow(_:)` (private static): malformed extensions JSON → falls back to empty dict, not a decode failure + +### DrainLease.swift: stream-keyed heartbeat-TTL lock (NOT PID-liveness; portable, no FFI) +- :49 `struct DrainLease: Sendable`: pure value type; lets Corpus's non-isolated deinit call release() directly +- :55 `owner`: `"pid-{pid}-{instanceToken}"`; nonce defeats PID-reuse impersonation after crash +- :59 `ttl`: default 15s = Rust DRAIN_LEASE_TTL_SECS +- :64 `heartbeatInterval = 5` (static): heartbeat cadence, well inside ttl +- :73 `init(directory:stream:instanceToken:ttl:)`: lease file = `/.drain.lease`; stream name sanitized to alnum/_/- (else `_`) +- :93 `tryAcquire(now:) -> Bool`: false only if fresh lease held by ANOTHER owner; else (re)claims + re-reads to resolve write races (atomic replace = last-writer-wins) +- :110 `heartbeat(now:)`: refresh; does NOT check ownership, caller's responsibility +- :119 `isHeldByOther(now:) -> Bool`: freshness check without attempting acquire +- :127 `release()`: deletes file ONLY if still owned by self (never undoes a takeover) - File format: `\n`, atomic replace write -### Watcher.swift — directory wake source (spurious wakes OK; drainAvailable is authority) +### Watcher.swift: directory wake source (spurious wakes OK; drainAvailable is authority) - :27 `enum Watcher` -- :37 `watchNewDirectory(at:onChange:) throws` — calls onChange() once immediately (drain pre-attach work) before platform loop -- :63 `watchKQueue` (Darwin only) — DispatchSource kqueue VNODE (write/extend/delete/rename), O_EVTONLY fd, µs latency -- :134 `openInotify` (Linux, private) — IN_CREATE|IN_MOVED_TO; returns -1 on watch-limit/unsupported-fs (caller falls back) -- :162 `runInotifyLoop` (Linux, private) — poll() 500ms timeout for responsive cancellation; drains fd, doesn't parse events -- :216 `watchPoll` — 200ms snapshot-diff poll; Linux fallback AND sole watcher on other platforms; ~100ms average latency -- :240 `final class ContinuationBox` — bridges DispatchSource cancel handler (DispatchQueue callback) to async wait() - -### Telemetry — QueueKitTelemetry.swift (off-path cost ~1ns; metric namespace `queue.*`) -- :27 `struct QueueLatencyWindow` — capacity-100 rolling sample list -- :44 `percentile(_:) -> Double` — P7-secfix: guards non-finite/out-of-range p BEFORE index computation (NaN/inf could crash) -- :68 `reportQueueStats(backend:drained:drainStart:now:estateTag:window:) async` — gate: `Intellectus.isEnabled`; emits queue.depth (or queue.depth_unavailable on read failure — NEVER fabricates 0), queue.drain_count, queue.idle_nonempty (skipped if depth unknown), queue.latency_p50_ms, queue.latency_p95_ms, queue.head_of_line_age_s +- :37 `watchNewDirectory(at:onChange:) throws`: calls onChange() once immediately (drain pre-attach work) before platform loop +- :63 `watchKQueue` (Darwin only): DispatchSource kqueue VNODE (write/extend/delete/rename), O_EVTONLY fd, µs latency +- :134 `openInotify` (Linux, private): IN_CREATE|IN_MOVED_TO; returns -1 on watch-limit/unsupported-fs (caller falls back) +- :162 `runInotifyLoop` (Linux, private): poll() 500ms timeout for responsive cancellation; drains fd, doesn't parse events +- :216 `watchPoll`: 200ms snapshot-diff poll; Linux fallback AND sole watcher on other platforms; ~100ms average latency +- :240 `final class ContinuationBox`: bridges DispatchSource cancel handler (DispatchQueue callback) to async wait() + +### Telemetry: QueueKitTelemetry.swift (off-path cost ~1ns; metric namespace `queue.*`) +- :27 `struct QueueLatencyWindow`: capacity-100 rolling sample list +- :44 `percentile(_:) -> Double`: P7-secfix: guards non-finite/out-of-range p BEFORE index computation (NaN/inf could crash) +- :68 `reportQueueStats(backend:drained:drainStart:now:estateTag:window:) async`: gate: `Intellectus.isEnabled`; emits queue.depth (or queue.depth_unavailable on read failure: NEVER fabricates 0), queue.drain_count, queue.idle_nonempty (skipped if depth unknown), queue.latency_p50_ms, queue.latency_p95_ms, queue.head_of_line_age_s ## INVARIANTS / GOTCHAS - WIRE FORMAT IS THE CONTRACT. Job/SignalFile Codable keys (snake_case, nested HLC, base64url payload) are pinned across Swift/Rust/Python. Any change must be mirrored in rust/src/job.rs and python/queuekit/job.py and pass the shared fixtures in Tests/QueueKitTests/Fixtures/*.json (rust/tests/conformance.rs reads the SAME files). -- JobID.generate() deliberately omits hyphens (32 dashless hex) — required so WireFormat.filename's own hyphen separators are unambiguous. Do not "prettify" back to UUID-with-hyphens form. -- validateIdentifier (FilesystemBackend) MUST run before any path is constructed from a caller-supplied stream_id or job id, at EVERY write/complete/completeBatch entry point. Mirrored in Rust and Python — edit all three or none. +- JobID.generate() deliberately omits hyphens (32 dashless hex): required so WireFormat.filename's own hyphen separators are unambiguous. Do not "prettify" back to UUID-with-hyphens form. +- validateIdentifier (FilesystemBackend) MUST run before any path is constructed from a caller-supplied stream_id or job id, at EVERY write/complete/completeBatch entry point. Mirrored in Rust and Python: edit all three or none. - reclaimInFlight (both backends) is safe ONLY immediately after a successful DrainLease.tryAcquire for that stream. Calling it without a fresh lease can yank a job out from under a live drainer. FilesystemBackend's unscoped reclaimInFlight() is mount-time-only, by construction (fresh process = no in-flight work of its own). - reply(session:status:) and reclaimInFlight(stream:) have real fast paths ONLY on PersistenceKitBackend (checked via `as?`); every other backend returns 0, and callers MUST have a per-job fallback. - QueueBackend's default drainAvailable(stream:) and pendingCount(stream:) extensions are CORRECTNESS TRAPS if relied upon: the default drain claims and then re-releases other streams' jobs (transient full-queue claim), and the default pendingCount(stream:) reports the ALL-STREAMS count. Both concrete backends override correctly; a THIRD backend implementation MUST also override both. -- PersistenceKitBackend.write() is a bare insert, NEVER wrap it in a transaction (invariant 1). writeBatch's .readCommitted (not .serializable) is deliberate — see drainAvailable's serializable claim for why this is still race-free. -- PersistenceKitBackend.watch()'s observer event is a WAKE SIGNAL ONLY (invariant 2) — never read job data from the TableChange event; always re-enter through drainAvailable(). drainUntilEmpty must propagate claim errors, never `try? ?? []`, or a live fault silently reads as "queue empty." -- appendOnly MUST stay false on the queuekit_jobs table declaration (invariant 5) — rows are mutated in place through the new→cur→done lifecycle. +- PersistenceKitBackend.write() is a bare insert, NEVER wrap it in a transaction (invariant 1). writeBatch's .readCommitted (not .serializable) is deliberate: see drainAvailable's serializable claim for why this is still race-free. +- PersistenceKitBackend.watch()'s observer event is a WAKE SIGNAL ONLY (invariant 2): never read job data from the TableChange event; always re-enter through drainAvailable(). drainUntilEmpty must propagate claim errors, never `try? ?? []`, or a live fault silently reads as "queue empty." +- appendOnly MUST stay false on the queuekit_jobs table declaration (invariant 5): rows are mutated in place through the new→cur→done lifecycle. - FilesystemBackend.completeBatch's jobID lookup keys on the filename suffix after the LAST `-`. This is only unambiguous because JobIDs are 32 dashless hex characters (JobID.generate()); a job id containing a `-` would break this index. -- estateTag on QueueKit is `nonisolated(unsafe)` — set it ONCE at mount, before any drain() call, never again during concurrent use. Same discipline for the internal latencyWindow. -- awaitDrain / awaitDrain(stream:) are polling latches (20ms default interval, 30s default timeout), not push notifications — there is no native "queue just emptied" event on either backend. Always returns promptly on an already-empty queue; always throws QueueError.drainTimeout rather than hanging past the deadline. -- Telemetry never fabricates queue.depth=0 on a read failure — it emits queue.depth_unavailable instead, and skips every metric that depends on a known depth. Do not "simplify" this to `(try? pendingCount()) ?? 0`. +- estateTag on QueueKit is `nonisolated(unsafe)`: set it ONCE at mount, before any drain() call, never again during concurrent use. Same discipline for the internal latencyWindow. +- awaitDrain / awaitDrain(stream:) are polling latches (20ms default interval, 30s default timeout), not push notifications: there is no native "queue just emptied" event on either backend. Always returns promptly on an already-empty queue; always throws QueueError.drainTimeout rather than hanging past the deadline. +- Telemetry never fabricates queue.depth=0 on a read failure: it emits queue.depth_unavailable instead, and skips every metric that depends on a known depth. Do not "simplify" this to `(try? pendingCount()) ?? 0`. - Watcher wakes are ALWAYS advisory/spurious-tolerant. drainAvailable() (via atomic rename / serializable UPDATE) is the sole claim authority on every platform. -- DrainLease is heartbeat-TTL, not PID-liveness — worst-case takeover latency is one full TTL (15s). Do not add OS-specific process-liveness checks; portability across macOS/Linux/Windows is a design constraint, not an oversight. -- Pinned constants — changing requires a conformance regen across all 3 ports: staleTmpThreshold 300s, DrainLease ttl 15s / heartbeatInterval 5s, Watcher poll interval 200ms / Linux poll() timeout 500ms, QueueLatencyWindow capacity 100, awaitDrain pollInterval 20ms / timeout 30s, NovelToken-style file mode 0600 on queue files. -- ConvergenceKit is intentionally NOT a QueueKit dependency (spec §11) — it is application-layer composition sitting above this kit. Do not add it as a Package.swift dependency. +- DrainLease is heartbeat-TTL, not PID-liveness: worst-case takeover latency is one full TTL (15s). Do not add OS-specific process-liveness checks; portability across macOS/Linux/Windows is a design constraint, not an oversight. +- Pinned constants: changing requires a conformance regen across all 3 ports: staleTmpThreshold 300s, DrainLease ttl 15s / heartbeatInterval 5s, Watcher poll interval 200ms / Linux poll() timeout 500ms, QueueLatencyWindow capacity 100, awaitDrain pollInterval 20ms / timeout 30s, NovelToken-style file mode 0600 on queue files. +- ConvergenceKit is intentionally NOT a QueueKit dependency (spec §11): it is application-layer composition sitting above this kit. Do not add it as a Package.swift dependency. diff --git a/packages/kits/QueueKit/docs/DETAILS.md b/packages/kits/QueueKit/docs/DETAILS.md index 78a8517..4cc394b 100644 --- a/packages/kits/QueueKit/docs/DETAILS.md +++ b/packages/kits/QueueKit/docs/DETAILS.md @@ -31,537 +31,613 @@ sources: This document walks through every source file in the package. Read `OVERVIEW.md` first for the big picture. Files appear here in pipeline -order: the wire-format types, the error vocabulary, the backend contract, -the public facade, the two concrete backends, and finally the supporting -infrastructure — the drain lease, the directory watcher, and telemetry. +order. The wire-format types come first. The error vocabulary comes next, +followed by the backend contract and the public facade. Then come the two +concrete backends. Last comes the supporting infrastructure: the drain +lease, the directory watcher, and telemetry. ## Job.swift -This file provides the wire-format types every job and completion signal is -built from: `JobID`, `StreamID`, `SessionID`, `ToolName`, `ArtifactRef`, +This file provides the wire-format types every job and completion signal +is built from: `JobID`, `StreamID`, `SessionID`, `ToolName`, `ArtifactRef`, `CodableValue`, `Job` itself, `MissionContext`, `WireFormat`, and `SignalFile`. -A job needs an identifier that is both unique and safe to use as a filename, -because the filesystem backend uses it directly in a path. `JobID.generate()` -builds one from a UUID (a 128-bit random identifier), rendered as thirty-two -lowercase hexadecimal characters with the hyphens removed — this avoids -the hyphen colliding with the hyphen QueueKit uses as a separator inside -maildir filenames. `StreamID` names the workload a job belongs to (for -example, "encode" or "dreaming"); jobs on different streams can share one -queue without a consumer of one stream ever seeing the other's jobs. -`SessionID.mint()` stamps one drain call: every job a single `drain()` -claims shares one session identifier, which is what lets a caller retire -a whole claimed batch in one step later. - -`ArtifactRef` names something a completed job produced — a file, a commit, -a signal file, or a step in a recorded trajectory — tagged with an explicit -`type` field on the wire so a decoder can tell the four cases apart. -`CodableValue` is a caller-defined extensions blob: an open-ended value that -can be a string, a number, a boolean, null, an array, or a nested object. -It exists so a job's `extensions` dictionary can carry arbitrary structured -data supplied by whatever produced the job, and have every field of it -survive a `send()`/`drain()` round trip unchanged, without QueueKit having -to know the shape of that data in advance. - -`Job` is the unit of work: an identifier, a stream, an HLC submission -timestamp, a priority, a payload of raw bytes, and the extensions -dictionary. Its custom `Codable` conformance is not decorative — it -implements the exact wire format the package specification fixes: keys are -`snake_case` (`stream_id`, not `streamID`), the HLC nests as -`physical_time`/`logical_count`/`node_id`, and the payload is base64url -text, not raw JSON bytes. This wire format is a cross-language contract: the -Rust and Python ports must produce and consume the identical bytes, so every -field name, nesting shape, and encoding choice here is pinned, not a matter -of Swift convention. +A job needs an identifier that is both unique and safe to use as a +filename. The filesystem backend uses this identifier directly in a path. +`JobID.generate()` builds one from a UUID, a 128-bit random identifier. It +renders the UUID as thirty-two lowercase hexadecimal characters, with the +hyphens removed. This removal matters because QueueKit already uses a +hyphen as a separator inside maildir filenames, and the UUID's own hyphens +would otherwise collide with it. + +`StreamID` names the workload a job belongs to. Examples are "encode" or +"dreaming." Jobs on different streams can share one queue. A consumer of +one stream never sees another stream's jobs. + +`SessionID.mint()` stamps one drain call. Every job a single `drain()` +claims shares one session identifier. This is what lets a caller retire a +whole claimed batch in one step later. + +`ArtifactRef` names something a completed job produced. It can name a +file, a commit, or a signal file. It can also name a step in a recorded +trajectory. An explicit `type` field on the wire tags which of the four +cases applies, so a decoder can tell them apart. + +`CodableValue` is a caller-defined extensions blob. It is an open-ended +value. It can be a string, a number, a boolean, null, an array, or a +nested object. It exists so a job's `extensions` dictionary can carry +arbitrary structured data supplied by whatever produced the job. Every +field of that data survives a `send()`/`drain()` round trip unchanged. +QueueKit does not need to know the shape of that data in advance. + +`Job` is the unit of work. It carries an identifier, a stream, an HLC +submission timestamp, and a priority. It also carries a payload of raw +bytes and the extensions dictionary. Its custom `Codable` conformance is +not decorative. It implements the exact wire format the package +specification fixes. Keys use `snake_case`, so the wire uses `stream_id` +rather than `streamID`. The HLC nests as `physical_time`, `logical_count`, +and `node_id`. The payload is base64url text, rather than raw JSON bytes. + +This wire format is a cross-language contract. The Rust and Python ports +must produce and consume the identical bytes. Every field name, nesting +shape, and encoding choice here is pinned. None of it is a matter of Swift +convention. `MissionContext` is the one type in this file that uses ordinary Swift -`Codable` key synthesis (`camelCase` property names, no custom coding keys) -rather than the pinned wire format, because it is not part of the job -envelope; it is a caller-supplied payload shape that has no cross-language -byte-identity requirement. - -`WireFormat.filename(for:)` builds the canonical maildir filename for a job: -a sortable HLC string, the stream identifier, and the job identifier, -joined by hyphens. `WireFormat.sortableHLC(_:)` is why the filename sorts -correctly by claim order: it zero-pads each HLC component to a fixed width -(sixteen digits for physical time, eight for the logical count, ten for the -node identifier) so that ordinary lexicographic string sorting — the only -sort a plain directory listing gives you — matches HLC's own ordering. -`WireFormat.encoder` fixes sorted keys and no escaped slashes, so the exact -same job encodes to the exact same bytes on every run and every platform; -this is what makes the shared conformance fixtures meaningful. - -`SignalFile` is the durable record a completed job leaves behind: which job, -what terminal status it ended in, what artifacts it produced, and when. Its -`Codable` conformance mirrors `Job`'s: pinned `snake_case` keys and a nested -HLC, because a signal file is read by whichever consumer or tool eventually -inspects the `done/` directory, in whichever language that consumer happens -to be written in. +`Codable` key synthesis. It uses `camelCase` property names, and it needs +no custom coding keys. This type does not use the pinned wire format, +because it is not part of the job envelope. It is instead a +caller-supplied payload shape, with no cross-language byte-identity +requirement. + +`WireFormat.filename(for:)` builds the canonical maildir filename for a +job. The filename joins three parts by hyphens: a sortable HLC string, the +stream identifier, and the job identifier. + +`WireFormat.sortableHLC(_:)` is why the filename sorts correctly by claim +order. It zero-pads each HLC component to a fixed width: sixteen digits +for physical time, eight for the logical count, and ten for the node +identifier. This padding makes ordinary lexicographic string sorting match +HLC's own ordering. Lexicographic sorting is the only sort a plain +directory listing gives a caller. + +`WireFormat.encoder` fixes sorted keys and no escaped slashes. The exact +same job therefore encodes to the exact same bytes, on every run and every +platform. This is what makes the shared conformance fixtures meaningful. + +`SignalFile` is the durable record a completed job leaves behind. It +records four things. It records which job finished. It records what +terminal status resulted. It records what artifacts the job produced. It +records when the job finished. Its `Codable` conformance mirrors `Job`'s. +It uses the same pinned `snake_case` keys and the same nested HLC. This +consistency matters because whichever consumer or tool eventually inspects +the `done/` directory reads a signal file, regardless of which language +that consumer is written in. ## ObservationStatus.swift This file provides `ObservationStatus`, the small enumeration a job's -outcome is reported in: `.running`, `.done`, `.doneWithConcerns`, -`.needsContext`, or `.blocked`. +outcome is reported in. Its cases are `.running`, `.done`, +`.doneWithConcerns`, `.needsContext`, and `.blocked`. -The type is owned here, not duplicated, because more than one package needs -to agree on what a finished job's outcome can be; AgentHarness, mentioned in -the file's header comment, imports it directly from QueueKit rather than -declaring its own copy. `isTerminal` is the one piece of logic on the type: -only `.running` is non-terminal. `reply(to:status:)` on the `QueueKit` -facade checks this before accepting a completion, so a caller cannot -accidentally "finish" a job with the status that means it has not finished. +The type is owned here rather than duplicated. More than one package needs +to agree on what a finished job's outcome can be. The file's header +comment names AgentHarness as one such package. AgentHarness imports the +type directly from QueueKit, rather than declaring its own copy. + +`isTerminal` is the one piece of logic on the type. Only `.running` is +non-terminal. `reply(to:status:)` on the `QueueKit` facade checks +`isTerminal` before accepting a completion. This check stops a caller from +accidentally marking a job finished with the one status that means it has +not finished. ## QueueError.swift This file provides `QueueError`, the complete error vocabulary a caller of QueueKit can receive. -Each case names one specific failure: a directory could not be created, a -write failed, a rename failed, a job's stored bytes could not be decoded, a -job identifier could not be found, the watcher failed, a stale temporary -file was found, the backend is unavailable, a caller tried to complete a job -with a non-terminal status, `awaitDrain` timed out, or a caller-supplied -identifier was unsafe to use in a path. Keeping every distinct failure as -its own case, rather than a generic wrapped error, is what lets a caller -handle a `jobNotFound` differently from a `renameFailed` without inspecting -error text. `drainTimeout(pending:inFlight:)` and -`invalidIdentifier(id:reason:)` each carry the diagnostic detail a caller -needs to explain, respectively, how far the queue was from empty when -`awaitDrain` gave up, and exactly why a submitted identifier was rejected. +Each case names one specific failure. A directory could not be created. A +write failed. A rename failed. A job's stored bytes could not be decoded. +A job identifier could not be found. The watcher failed. A stale +temporary file was found. The backend was unavailable. A caller tried to +complete a job with a non-terminal status. `awaitDrain` timed out. A +caller-supplied identifier was unsafe to use in a path. + +Every distinct failure gets its own case, rather than a single generic +wrapped error. This design lets a caller handle a `jobNotFound` +differently from a `renameFailed`, without inspecting error text. + +`drainTimeout(pending:inFlight:)` and `invalidIdentifier(id:reason:)` each +carry extra diagnostic detail. `drainTimeout` explains how far the queue +was from empty when `awaitDrain` gave up. `invalidIdentifier` explains +exactly why a submitted identifier was rejected. ## QueueBackend.swift This file provides `QueueBackend`, the protocol every concrete backend -conforms to, plus two protocol extensions supplying default -implementations for the operations a backend does not have to implement -specially. - -The protocol is the seam that lets the public `QueueKit` class stay ignorant -of which storage mechanism is underneath it: every method the facade -exposes has a matching method here, and `QueueKit` does nothing but forward -to whichever `QueueBackend` it was mounted with. `write(_:)`, +conforms to. It also provides two protocol extensions. These extensions +supply default implementations for the operations a backend does not have +to implement specially. + +The protocol is the seam that lets the public `QueueKit` class stay +ignorant of which storage mechanism sits underneath it. Every method the +facade exposes has a matching method here. `QueueKit` does nothing but +forward to whichever `QueueBackend` it was mounted with. Six core +operations must be implemented from scratch by every backend: `write(_:)`, `drainAvailable()`, `watch(handler:)`, `complete(_:status:artifacts:)`, -`inFlight()`, and `completed(streamID:)` are the core operations every -backend must implement from scratch. +`inFlight()`, and `completed(streamID:)`. `writeBatch(_:)` and `completeBatch(_:)` are bulk twins of `write` and `complete`. The first default extension implements them by simply looping -the single-item version — correct for any backend, but not necessarily -fast. A backend earns the right to skip that loop by overriding these two -methods with a mechanism that pays a shared cost once for the whole batch -instead of once per job; both concrete backends do exactly that, for -reasons explained under their own headings below. - -`drainAvailable(stream:)` and `pendingCount(stream:)` are the stream-scoped -forms of `drainAvailable()` and `pendingCount()`. The second protocol -extension's default implementation of `drainAvailable(stream:)` claims -every stream's jobs through the all-streams `drainAvailable()` and then -filters the result down to the requested stream — which is correct, but -claims jobs belonging to other streams that then have to be re-enqueued, so -the file's own comment warns that a concrete backend should override it -rather than rely on the default. Both `FilesystemBackend` and -`PersistenceKitBackend` do override it, each in the way suited to its own -storage. +the single-item version. This loop is correct for any backend, but it is +not necessarily fast. A backend earns the right to skip the loop by +overriding these two methods. The override must use a mechanism that pays +a shared cost once for the whole batch, instead of once per job. Both +concrete backends do exactly that, for reasons explained under their own +headings below. + +`drainAvailable(stream:)` and `pendingCount(stream:)` are the +stream-scoped forms of `drainAvailable()` and `pendingCount()`. The second +protocol extension gives `drainAvailable(stream:)` a default +implementation. This default claims every stream's jobs through the +all-streams `drainAvailable()`, then filters the result down to the +requested stream. The result is correct, but it claims jobs belonging to +other streams, and those jobs then have to be re-enqueued. The file's own +comment warns that a concrete backend should override this default, +rather than rely on it. Both `FilesystemBackend` and `PersistenceKitBackend` +override it, each in the way suited to its own storage. ## QueueKit.swift -This file provides the public facade: the `QueueKit` class itself, its -maildir directory-management helpers, and the four permanent methods — -`send`, `drain`, `watch`, `reply` — that every caller uses regardless of +This file provides the public facade. The facade is the `QueueKit` class +itself, its maildir directory-management helpers, and the four permanent +methods every caller uses. Those four methods are `send`, `drain`, +`watch`, and `reply`. Every caller uses them the same way, regardless of which backend is mounted underneath. -`QueueKit` holds a `backend` (anything conforming to `QueueBackend`), an -optional `root` URL (present only when the mounted backend is a -filesystem backend), a rolling latency window for telemetry, and an -`estateTag` string used to tag telemetry metrics by estate. Two -initializers exist: `init(root:hlcGenerator:)` is the common path — it -builds a `FilesystemBackend` at the given root and cleans any stale +`QueueKit` holds four properties. It holds a `backend`, anything +conforming to `QueueBackend`. It holds an optional `root` URL, present +only when the mounted backend is a filesystem backend. It holds a rolling +latency window for telemetry. It holds an `estateTag` string, used to tag +telemetry metrics by estate. + +Two initializers exist. `init(root:hlcGenerator:)` is the common path. It +builds a `FilesystemBackend` at the given root, and it cleans any stale temporary files left behind by a prior crash. `init(backend:root:)` mounts -an already-constructed backend directly; this is how `PersistenceKitBackend` -gets attached, and how tests substitute an in-memory backend. - -`send(_:)` and its batch twin `send(batch:)` hand jobs to the backend. -The batch form exists because the filesystem backend's per-job `write` -pays a filesystem synchronization cost — a call that forces the operating -system to guarantee the write has actually reached durable storage — on -every single job; a bulk producer enqueuing tens of thousands of jobs at -once (a full reindex, for instance) can instead route through -`writeBatch`, which the filesystem backend implements as one durability -barrier for the whole batch rather than one per job. - -`drain()` and the stream-scoped `drain(stream:)` claim available jobs and, -around the backend call, measure how long the claim took and report it -through `reportQueueStats`. Both return the same shape: a list of pairs, -each a claimed `Job` and the `SessionID` that drain call minted. - -`watch(handler:)` subscribes a handler that is invoked for every job as it -becomes available; the facade does nothing but forward to the backend's own -`watch`, because the two backends drive this very differently (a directory -watcher for the filesystem backend, a database change observer for the -PersistenceKit backend). - -`reply(to:status:artifacts:)` is the single-job completion path; it checks -`status.isTerminal` before forwarding to the backend, so a caller cannot -mark a job "complete" with the one status (`.running`) that means it is -not. `reply(session:status:)` completes every job claimed under one drain -call's session in a single pass; it only has a fast path on -`PersistenceKitBackend` (checked with an `as?` cast), because only that -backend's shared table can look up an entire session's claimed rows in one -guarded update. On any other backend it returns `0`, signaling the caller -to fall back to per-job `reply`. `reply(batch:)` completes an explicit list -of job/status pairs in one pass, routing to the backend's `completeBatch`; -this is the path a corpus-drain caller uses on the filesystem backend, which -has no session fast path but does have a batched completion. +an already-constructed backend directly. This is how `PersistenceKitBackend` +gets attached. It is also how tests substitute an in-memory backend. + +`send(_:)` and its batch twin `send(batch:)` hand jobs to the backend. The +filesystem backend's per-job `write` pays a filesystem synchronization +cost on every single job. This cost is a call that forces the operating +system to guarantee the write has actually reached durable storage. The +batch form exists to avoid paying that cost per job. A bulk producer, +enqueuing tens of thousands of jobs at once during a full reindex, can +instead route through `writeBatch`. The filesystem backend implements +`writeBatch` as one durability barrier for the whole batch, rather than +one barrier per job. + +`drain()` and the stream-scoped `drain(stream:)` claim available jobs. +Around the backend call, each method measures how long the claim took, +and reports that duration through `reportQueueStats`. Both return the +same shape: a list of pairs. Each pair holds a claimed `Job` and the +`SessionID` that drain call minted. + +`watch(handler:)` subscribes a handler. The handler is invoked for every +job as it becomes available. The facade does nothing but forward to the +backend's own `watch`. The two backends drive this very differently. The +filesystem backend uses a directory watcher. The PersistenceKit backend +uses a database change observer. + +`reply(to:status:artifacts:)` is the single-job completion path. It +checks `status.isTerminal` before forwarding to the backend. This check +stops a caller from marking a job complete with the one status, +`.running`, that means the job has not finished. + +`reply(session:status:)` completes every job claimed under one drain +call's session, in a single pass. It has a real fast path only on +`PersistenceKitBackend`, checked with an `as?` cast. Only that backend's +shared table can look up an entire session's claimed rows in one guarded +update. On any other backend, `reply(session:status:)` returns `0`. That +return value signals the caller to fall back to per-job `reply`. + +`reply(batch:)` completes an explicit list of job and status pairs, in one +pass. It routes to the backend's `completeBatch`. This is the path a +corpus-drain caller uses on the filesystem backend, which has no session +fast path but does have a batched completion. `reclaimInFlight(stream:)` resets every job stuck in-flight for one stream -back to pending, so the next `drain(stream:)` re-claims and re-drives it. -The doc comment is explicit about the precondition: call this only -immediately after successfully acquiring that stream's `DrainLease`, because -a freshly acquired lease is the guarantee that the job's prior claimant is -actually gone, not merely slow — without that guarantee, this method could -yank a job out from under a drain worker that is still processing it. Like -`reply(session:status:)`, it only has a fast path on `PersistenceKitBackend`; -`FilesystemBackend` instead exposes an unscoped `reclaimInFlight()` that -runs once at mount, because its `cur/` directory has no stream separation -in the first place. +back to pending. The next `drain(stream:)` call then re-claims and +re-drives it. The doc comment states a strict precondition. A caller must +call this method only immediately after successfully acquiring that +stream's `DrainLease`. A freshly acquired lease guarantees the job's prior +claimant is actually gone, rather than merely slow. Without that +guarantee, this method could yank a job out from under a drain worker +still processing it. + +Like `reply(session:status:)`, `reclaimInFlight(stream:)` has a real fast +path only on `PersistenceKitBackend`. `FilesystemBackend` instead exposes +an unscoped `reclaimInFlight()` that runs once at mount. Its `cur/` +directory has no stream separation in the first place, so no scoping is +possible there. `inFlight()`, `pendingCount()`, and `pendingCount(stream:)` are read-only -depth probes: how many jobs are claimed but not yet replied to, and how -many are waiting to be claimed. `pendingCount() + inFlight().count` is the -total outstanding work a drain still has left to do — a status reader can -compute that without claiming or draining anything. +depth probes. `inFlight()` reports how many jobs are claimed but not yet +replied to. `pendingCount()` reports how many jobs are waiting to be +claimed. Adding `pendingCount()` to `inFlight().count` gives the total +outstanding work a drain still has left to do. A status reader can +compute this total without claiming or draining anything. `awaitDrain(pollInterval:timeout:)` and its stream-scoped twin -`awaitDrain(stream:pollInterval:timeout:)` block until a queue (or one -stream of it) is empty on both frontiers: nothing pending, nothing -in-flight. Both are polling loops, not push-based latches, because the -maildir backend has no native "the queue just emptied" event to wait on; -each iteration re-reads the live counts, so progress made by a concurrently -running drain worker between polls is observed on the very next tick. Both -return promptly when the queue is already empty — the first check can -succeed without ever sleeping — and both throw `QueueError.drainTimeout` -rather than blocking forever if the deadline passes with work still -outstanding, so a stuck or crashed drain worker surfaces as an error -instead of a hang. +`awaitDrain(stream:pollInterval:timeout:)` block until a queue, or one +stream of it, is empty on both frontiers. Nothing may be pending, and +nothing may be in-flight. Both methods are polling loops, rather than +push-based latches. The maildir backend has no native event for "the +queue just emptied," so there is nothing to wait on directly. Each +iteration re-reads the live counts. Progress made by a concurrently +running drain worker between polls is therefore observed on the very next +tick. + +Both methods return promptly when the queue is already empty. The first +check can succeed without the method ever sleeping. Both methods throw +`QueueError.drainTimeout` rather than blocking forever, if the deadline +passes with work still outstanding. A stuck or crashed drain worker +therefore surfaces as an error, instead of as a hang. `ensureMaildir(root:)` and `cleanStaleTmpFiles(root:)` are the maildir -directory-management helpers `FilesystemBackend`'s initializer calls: the -first creates the four subdirectories (`tmp`, `new`, `cur`, `done`) if -absent, and the second deletes any file in `tmp/` older than -`staleTmpThreshold` (five minutes). A file stranded in `tmp/` means a -process crashed between writing it and renaming it into `new/`; because it -was never visible in `new/`, no consumer could have claimed it, so removing -it on the next mount is safe. +directory-management helpers that `FilesystemBackend`'s initializer calls. +`ensureMaildir(root:)` creates four subdirectories if they are absent: +`tmp`, `new`, `cur`, and `done`. `cleanStaleTmpFiles(root:)` deletes +any file in `tmp/` older than `staleTmpThreshold`, five minutes. + +A file stranded in `tmp/` means a process crashed between writing it and +renaming it into `new/`. Because the file was never visible in `new/`, no +consumer could have claimed it. Removing it on the next mount is +therefore safe. ## FilesystemBackend.swift This file provides `FilesystemBackend`, the POSIX maildir-style queue -implementation: no database, just four directories and atomic filesystem -operations. Its semantics are deliberately derived from `deliver_maildir()`, -the mail-delivery routine at the heart of the Postfix mail server — a -decades-proven pattern for handing off many small files between -uncoordinated processes without a lock file. +implementation. It uses no database, just four directories and atomic +filesystem operations. Its semantics are deliberately derived from +`deliver_maildir()`, the mail-delivery routine at the heart of the Postfix +mail server. This routine is a decades-proven pattern. It hands off many +small files between uncoordinated processes, without a lock file. `validateIdentifier(_:)` runs before any filesystem path is built from a -caller-supplied identifier (a stream identifier or a job identifier). It -rejects an empty string, a `.` or `..` path component, a `/` or `\` path -separator, and any ASCII control character. Without this check, a -maliciously or accidentally crafted stream identifier containing a path -separator could make a job's path resolve outside the queue's root -directory — a path-traversal risk. The Rust and Python ports enforce the -identical rule. - -`write(_:)` implements the maildir handoff in the classic three steps: -create the job's encoded bytes in `tmp/` with a mode that only the owning -process can read (defense in depth, since a queued payload may carry -sensitive estate content), force those bytes to durable storage, then -atomically rename the file into `new/`. `atomicWriteAndRename(...)` is -where those steps actually happen, using `open` with `O_CREAT | O_EXCL` (an -exclusive create only one caller can win), then a raw `write`/`fsync` -sequence, then `rename`, then a final `fsync` of the destination -directory itself — an extra step needed because on POSIX filesystems, a -directory's own metadata (which files it currently contains) is a separate -piece of durable state from the file contents, and both must be forced to -disk for the write to survive a crash. A `rename` failure because the -destination temporarily vanished (`ENOENT`) is retried once, since another -process cleaning `new/` mid-flight is a recoverable race, not a fatal one. - -`writeBatch(_:)` is the bulk twin: it writes and renames every job in the -batch with `writeAndRenameNoFsync`, which skips the per-file durability -step, and forces only `new/`'s directory metadata to disk once at the end. -A crash before that single barrier can lose some just-written jobs, but the +caller-supplied identifier, whether a stream identifier or a job +identifier. It rejects an empty string. It rejects a `.` or `..` path +component. It rejects a `/` or `\` path separator. It rejects any ASCII +control character. Without this check, a maliciously or accidentally +crafted stream identifier containing a path separator could make a job's +path resolve outside the queue's root directory. That outcome is a +path-traversal risk. The Rust and Python ports enforce the identical rule. + +`write(_:)` implements the maildir handoff in three classic steps. First, +it creates the job's encoded bytes in `tmp/`, with a mode that only the +owning process can read. This restricted mode is defense in depth, since +a queued payload may carry sensitive estate content. Second, it forces +those bytes to durable storage. Third, it atomically renames the file +into `new/`. + +`atomicWriteAndRename(...)` is where those steps actually happen. It opens +the file with `O_CREAT | O_EXCL`, an exclusive create only one caller can +win. It then runs a raw `write` and `fsync` sequence, followed by +`rename`. It finishes with a final `fsync` of the destination directory +itself. This last step matters because a directory's own metadata is +separate durable state from the file contents, on POSIX filesystems. The +metadata records which files the directory currently contains. Both the +metadata and the file contents must be forced to disk, for the write to +survive a crash. A `rename` failure caused by the destination temporarily +vanishing, an `ENOENT` error, is retried once. Another process cleaning +`new/` mid-flight is a recoverable race, rather than a fatal one. + +`writeBatch(_:)` is the bulk twin. It writes and renames every job in the +batch using `writeAndRenameNoFsync`, which skips the per-file durability +step. It forces only `new/`'s directory metadata to disk once, at the +end. + +A crash before that single barrier can lose some just-written jobs. The comment records why this is an acceptable trade for a bulk producer -specifically: the reindex that calls `writeBatch` derives its jobs from the -estate itself, so a lost job is simply re-enqueued on the next resume — an -at-least-once guarantee, not a promise that every job survives every -possible crash instant. Ordinary streaming `write` keeps its per-job -durability unchanged; only a caller that opts into the batch API accepts -this weaker bound. - -`pendingCount()` counts the files in `new/` — one file, one pending job, -no decoding required, which is why it stays cheap even as a queue grows. - -`drainAvailable(stream:)` is the stream-scoped claim. Because the maildir -filename already encodes the stream (`{hlc}-{stream}-{jobid}`), a naive -implementation could filter by filename alone, but the code instead reads -and decodes each file's actual `streamID` field before claiming it, leaving -files whose stream does not match entirely untouched in `new/` — a +specifically. The reindex that calls `writeBatch` derives its jobs from +the estate itself, so a lost job is simply re-enqueued on the next resume. +This is an at-least-once guarantee. It is not a promise that every job +survives every possible crash instant. Ordinary streaming `write` keeps +its per-job durability unchanged. Only a caller that opts into the batch +API accepts this weaker bound. + +`pendingCount()` counts the files in `new/`. One file means one pending +job, so no decoding is required. This is why the count stays cheap even +as a queue grows. + +`drainAvailable(stream:)` is the stream-scoped claim. The maildir filename +already encodes the stream, in the form `{hlc}-{stream}-{jobid}`. A naive +implementation could filter by filename alone. The code instead reads and +decodes each file's actual `streamID` field before claiming it. Files +whose stream does not match are left entirely untouched in `new/`. A concurrent drainer for a different stream is never affected by this one's scan. `pendingCount(stream:)` performs the equivalent non-claiming count. `drainAvailable()`, the all-streams claim, renames every file in `new/` -into `cur/` first and only then reads and decodes each one; a file that -fails to decode is moved straight to `done/` rather than left to jam every -future drain pass, since a job whose stored bytes are unparseable can never -become processable by simply trying again. - -`reclaimInFlight()` moves everything left in `cur/` back to `new/`. Its doc -comment states its precondition plainly: call it only at mount, before any -drain session is live, because a freshly started process cannot have -claimed anything itself, so anything already sitting in `cur/` must be a -crash orphan from whichever process ran before it. - -`watch(handler:)` wires `Watcher.watchNewDirectory` to `drainAvailable()`: -every time the watcher signals that `new/` may have changed, the backend -drains whatever is currently claimable and hands each job to the caller's -handler. - -`complete(_:status:artifacts:)` finds the job's file in `cur/` by matching -the filename suffix (every maildir filename ends in `-{jobID}`), writes the -`SignalFile` recording the outcome, and only then moves the job's own file -into `done/`. Writing the signal first matters: if the process crashes -between the two steps, the job is still visible in `cur/` on the next -mount, ready to be reclaimed, rather than silently vanished with an -orphaned signal and no job record. - -`completeBatch(_:)` is the batched form: one scan of `cur/` builds a -job-identifier-to-filename index, every completion in the batch is written -without a per-file durability step, and one call forces `done/`'s directory -metadata to disk at the very end. The safety argument mirrors -`writeBatch`'s: a crash before that final barrier leaves the not-yet-moved -jobs in `cur/`, where they are reclaimed and reprocessed — safe because -ingest is idempotent (processing the same job twice causes no harm). +into `cur/` first. Only then does it read and decode each one. A file +that fails to decode is moved straight to `done/`, rather than left to +jam every future drain pass. A job whose stored bytes are unparseable can +never become processable by simply trying again. + +`reclaimInFlight()` moves everything left in `cur/` back to `new/`. Its +doc comment states its precondition plainly. A caller should call it only +at mount, before any drain session is live. A freshly started process +cannot have claimed anything itself. Anything already sitting in `cur/` +must therefore be a crash orphan from whichever process ran before it. + +`watch(handler:)` wires `Watcher.watchNewDirectory` to `drainAvailable()`. +Every time the watcher signals that `new/` may have changed, the backend +drains whatever is currently claimable, and hands each job to the +caller's handler. + +`complete(_:status:artifacts:)` finds the job's file in `cur/` by +matching the filename suffix. Every maildir filename ends in `-{jobID}`. +The method writes the `SignalFile` recording the outcome, and only then +moves the job's own file into `done/`. Writing the signal first matters. +If the process crashes between the two steps, the job is still visible +in `cur/` on the next mount, ready to be reclaimed. Without this order, +the job could vanish silently, leaving an orphaned signal and no job +record. + +`completeBatch(_:)` is the batched form. One scan of `cur/` builds a +job-identifier-to-filename index. Every completion in the batch is +written without a per-file durability step. One call forces `done/`'s +directory metadata to disk at the very end. The safety argument mirrors +`writeBatch`'s argument. A crash before that final barrier leaves the +not-yet-moved jobs in `cur/`, where they are reclaimed and reprocessed. +This is safe because ingest is idempotent, and processing the same job +twice causes no harm. `inFlight()` and `completed(streamID:)` both delegate to the private -`listJobs(in:filter:)`, which lists a directory, skips `.signal` sidecar -files, decodes each remaining file as a `Job`, and optionally filters by -stream. +`listJobs(in:filter:)`. This method lists a directory and skips +`.signal` sidecar files. It decodes each remaining file as a `Job`. It +can optionally filter by stream. ## PersistenceKitBackend.swift -This file provides `PersistenceKitBackend`, the second concrete backend: -jobs live as rows in a table inside PersistenceKit, MOOTx01's storage kit, -rather than as files on disk. The file's own header comment states five -invariants the implementation is built to preserve; the sections below -explain each one as it is enforced in code. - -`QueueKitSchema.declaration()` defines the table's shape: one row per job, -with columns for the job's own fields plus queue bookkeeping — `status` -(`"new"`, `"cur"`, or `"done"`), `signal_status` and `artifacts` (filled in -only once a job completes), and `session_id` (which drain call, if any, -currently holds this row). Three indices back the operations that matter: -one for a plain status count, one on `(status, physical_time, -logical_count, node_id)` for claiming jobs in HLC order, and one on -`(stream_id, status)` for the stream-scoped operations. The table's -`appendOnly` flag is explicitly `false` — invariant 5 — because a queue row -must be mutable: the same row moves from `new` to `cur` to `done` in place. - -`write(_:)` is a bare insert into the table with `status: "new"` — no -enclosing transaction, per invariant 1. A transaction would offer no benefit -here (there is nothing else to make atomic with a single insert) and would -only add overhead to the hottest, most frequent operation in the backend. +This file provides `PersistenceKitBackend`, the second concrete backend. +Jobs live as rows in a table inside PersistenceKit, MOOTx01's storage +kit, rather than as files on disk. The file's own header comment states +five invariants the implementation is built to preserve. The sections +below explain each one, as it is enforced in code. + +`QueueKitSchema.declaration()` defines the table's shape. There is one +row per job. Its columns hold the job's own fields, plus queue +bookkeeping. The bookkeeping columns are `status`, `signal_status`, +`artifacts`, and `session_id`. `status` is `"new"`, `"cur"`, or `"done"`. +`signal_status` and `artifacts` are filled in only once a job completes. +`session_id` names the drain call, if any, currently holding the row. + +Three indices back the operations that matter. One index backs a plain +status count. A second index, on `(status, physical_time, logical_count, +node_id)`, backs claiming jobs in HLC order. A third index, on +`(stream_id, status)`, backs the stream-scoped operations. The table's +`appendOnly` flag is explicitly `false`. This is invariant 5. A queue row +must be mutable, since the same row moves from `new` to `cur` to `done` +in place. + +`write(_:)` is a bare insert into the table, with `status: "new"`. It has +no enclosing transaction, per invariant 1. A transaction would offer no +benefit here, since there is nothing else to make atomic with a single +insert. A transaction would only add overhead to the hottest, most +frequent operation in the backend. `writeBatch(_:)` wraps every insert in the batch in one `.readCommitted` -transaction — the transactional-isolation equivalent of the filesystem -backend's single durability barrier. The comment explains why the weaker -`.readCommitted` isolation (rather than the `.serializable` isolation the -claim uses) is still correct here: these are all inserts of brand-new rows, -which cannot conflict with the claim's job of flipping existing `new` rows -to `cur` — a claim that begins before the batch commits simply will not see -the batch's rows yet, and will see them on its next pass instead. - -`pendingCount()` and `pendingCount(stream:)` are single-row-count reads -with a `status = 'new'` predicate (and, for the stream form, an additional -`stream_id` predicate) — no claim, no lock beyond an ordinary read. +transaction. This transaction is the isolation equivalent of the +filesystem backend's single durability barrier. The comment explains why +the weaker `.readCommitted` isolation is still correct here, rather than +the `.serializable` isolation the claim uses. These are all inserts of +brand-new rows. Brand-new rows cannot conflict with the claim's job of +flipping existing `new` rows to `cur`. A claim that begins before the +batch commits simply will not see the batch's rows yet. It will see them +on its next pass instead. + +`pendingCount()` and `pendingCount(stream:)` are single-row-count reads. +Both use a `status = 'new'` predicate. The stream form adds a +`stream_id` predicate. Neither read claims anything or locks beyond an +ordinary read. `drainAvailable(stream:)` and the all-streams `drainAvailable()` both -implement the claim as one `.serializable` transaction containing exactly -two steps: an `UPDATE` that flips every matching `new` row to `cur` under -one freshly minted session identifier, and a `SELECT` that reads back -precisely the rows carrying that session identifier, ordered by HLC. This -single-pass shape — invariant 3, the status-guarded atomic transition — is -what the file's comment calls out as a deliberate improvement over an -earlier design that issued one guarded update per row: claiming a batch of -jobs that way cost one predicate scan per job, so a bulk claim over a large -queue became quadratic in the number of jobs. Reading back by this call's -own unique session, rather than by `status = 'cur'` generally, is what keeps -two concurrent drainers from ever double-counting the same claimed row — -each call's session tags only the rows it itself just claimed. - -`completeSession(_:status:)` completes every row claimed under one session -in a single guarded update — the batch twin of the per-job `complete`, and -the reason `QueueKit.reply(session:status:)` has a fast path only on this -backend. It is declared directly on the class rather than as part of the -`QueueBackend` protocol, because only the concrete `PersistenceKitBackend` -handle (not the abstract protocol) has anything meaningful to complete a -whole session against. - -`watch(handler:)` subscribes to `storage.observer`, per invariant 2, and -treats every event strictly as a wake signal — it never reads a job's +implement the claim as one `.serializable` transaction. That transaction +has exactly two steps. First, an `UPDATE` flips every matching `new` row +to `cur`, under one freshly minted session identifier. Second, a `SELECT` +reads back precisely the rows carrying that session identifier, ordered +by HLC. This single-pass shape is invariant 3, the status-guarded atomic +transition. The file's comment calls it out as a deliberate improvement +over an earlier design, one that issued one guarded update per row. +Claiming a batch of jobs that older way cost one predicate scan per job, +so a bulk claim over a large queue became quadratic in the number of +jobs. Reading back by this call's own unique session, rather than by +`status = 'cur'` generally, keeps two concurrent drainers from ever +double-counting the same claimed row. Each call's session tags only the +rows it itself just claimed. + +`completeSession(_:status:)` completes every row claimed under one +session, in a single guarded update. It is the batch twin of the per-job +`complete`. It is the reason `QueueKit.reply(session:status:)` has a fast +path only on this backend. The method is declared directly on the class, +rather than as part of the `QueueBackend` protocol. Only the concrete +`PersistenceKitBackend` handle has anything meaningful to complete a +whole session against. The abstract protocol does not. + +`watch(handler:)` subscribes to `storage.observer`, per invariant 2. It +treats every event strictly as a wake signal. It never reads a job's fields out of the event payload itself, only out of a subsequent -`drainAvailable()` call. `drainUntilEmpty` is the loop this drives: rather -than draining once per event, it drains repeatedly until a pass claims -nothing, because a burst of inserts can coalesce into fewer observer events -than there are actual rows, and draining only once per event would strand -whichever rows' wake got folded away. The initial drain before the -subscription's first `await` matters for the same reason from the other -direction: a job inserted between mount and subscription would otherwise be -invisible to any wake, since its insert event predates the subscription. - -`complete(_:status:artifacts:)` is a single guarded update keyed on both -the job identifier and `status = 'cur'`; if nothing matched, the job was -never claimed (or was already completed), so the method throws -`QueueError.jobNotFound` rather than silently doing nothing. +`drainAvailable()` call. `drainUntilEmpty` is the loop this drives. +Rather than draining once per event, it drains repeatedly until a pass +claims nothing. A burst of inserts can coalesce into fewer observer +events than there are actual rows, so draining only once per event would +strand whichever rows' wake got folded away. The initial drain, run +before the subscription's first `await`, matters for the same reason in +reverse. A job inserted between mount and subscription would otherwise +be invisible to any wake, since its insert event predates the +subscription. + +`complete(_:status:artifacts:)` is a single guarded update, keyed on both +the job identifier and `status = 'cur'`. If nothing matched, the job was +never claimed, or it was already completed. The method throws +`QueueError.jobNotFound` in that case, rather than silently doing +nothing. `reclaimInFlight(stream:)` resets a stream's `cur` rows back to `new`, -clearing `session_id`. Its doc comment repeats the same safety argument as -the facade method it backs: this is safe to call only immediately after -successfully acquiring the stream's `DrainLease`, because the lease's -staleness gate is what rules out a live drainer still holding those rows. +and clears `session_id`. Its doc comment repeats the same safety argument +as the facade method it backs. Calling it is safe only immediately after +successfully acquiring the stream's `DrainLease`. The lease's staleness +gate is what rules out a live drainer still holding those rows. `inFlight()` and `completed(streamID:)` both route through the private -`listJobs(status:streamID:)`, and `decodeRow(_:)` turns one database row -back into a `Job`, falling back to an empty extensions dictionary if the -stored JSON for it fails to parse rather than failing the whole read. +`listJobs(status:streamID:)`. `decodeRow(_:)` turns one database row back +into a `Job`. If the stored JSON for a row's extensions fails to parse, +`decodeRow(_:)` falls back to an empty extensions dictionary, rather than +failing the whole read. ## DrainLease.swift -This file provides `DrainLease`, a heartbeat-based, stream-keyed lock that -lets many processes share one estate's queue while guaranteeing that only -one of them actively drains a given stream at a time. Two different streams -get entirely independent leases and can both be held simultaneously — this -is what lets, for example, an encode drainer and a dreaming drainer run at -once without either blocking on the other. - -The design is deliberately not a check on whether the prior holder's -process is still alive. A PID-liveness check would need OS-specific code — -querying the process table through platform-specific system calls — which -the file's header comment rules out in favor of something that works -identically on macOS, Linux, and Windows: a lease file holding an owner -string and a wall-clock timestamp. `tryAcquire(now:)` succeeds when the -lease file is absent, already owned by the caller, or its timestamp is -older than `ttl` (fifteen seconds) — in every other case, another drainer -holds a fresh lease, and the caller must stand down. The owner string -itself combines the process identifier with a caller-supplied nonce, so -that if the operating system reuses a process identifier after a crash, the -reused identifier cannot be mistaken for the original holder. +This file provides `DrainLease`, a heartbeat-based, stream-keyed lock. It +lets many processes share one estate's queue, while guaranteeing that +only one of them actively drains a given stream at a time. Two different +streams get entirely independent leases, and both can be held at once. +This is what lets an encode drainer and a dreaming drainer run at the +same time, for example, without either blocking on the other. + +The design deliberately avoids checking whether the prior holder's +process is still alive. A PID-liveness check would need OS-specific code, +querying the process table through platform-specific system calls. The +file's header comment rules this out. It favors something that works +identically on macOS, Linux, and Windows instead: a lease file holding an +owner string and a wall-clock timestamp. + +`tryAcquire(now:)` succeeds in three cases. It succeeds when the lease +file is absent. It succeeds when the lease is already owned by the +caller. It succeeds when the lease's timestamp is older than `ttl`, +fifteen seconds. In every other case, another drainer holds a fresh +lease, and the caller must stand down. The owner string itself combines +the process identifier with a caller-supplied nonce. This combination +matters because the operating system can reuse a process identifier +after a crash. Without the nonce, a reused identifier could be mistaken +for the original holder. `heartbeat(now:)` refreshes the timestamp while a drainer actively holds -the lease; the doc comment directs a caller to invoke it roughly every five -seconds — well inside the fifteen-second TTL — so an ordinary slow drain -pass never has its own lease expire out from under it. `isHeldByOther(now:)` -answers the same freshness question as `tryAcquire` without attempting to -take the lease, for a caller that only wants to check before deciding -whether to try. `release()` deletes the lease file on clean shutdown, but -only if the caller still owns it, so that a takeover that already happened +the lease. The doc comment directs a caller to invoke it roughly every +five seconds, well inside the fifteen-second TTL. This cadence keeps an +ordinary slow drain pass from ever having its own lease expire out from +under it. `isHeldByOther(now:)` answers the same freshness question as +`tryAcquire`, without attempting to take the lease. A caller that only +wants to check before deciding whether to try uses this method instead. +`release()` deletes the lease file on clean shutdown, but only if the +caller still owns it. This guard means a takeover that already happened is never undone by a late release from the prior holder. -The worst-case cost of a crashed holder is one full TTL: a takeover cannot -happen sooner than fifteen seconds after the last heartbeat, because a -fresher lease has to be honestly waited out. This is the trade the design -accepts for staying free of any OS-specific liveness check. +The worst-case cost of a crashed holder is one full TTL. A takeover +cannot happen sooner than fifteen seconds after the last heartbeat, +because a fresher lease has to be honestly waited out. This is the trade +the design accepts, for staying free of any OS-specific liveness check. ## Watcher.swift This file provides `Watcher`, the directory-change wake source -`FilesystemBackend.watch()` is built on. Its job is narrow and its contract -is explicit: call `onChange` whenever the watched directory's contents may -have changed, allowing spurious wakes, because `drainAvailable()` is always -the actual authority on what can be claimed — a false-positive wake costs -one wasted scan, never a correctness problem. +`FilesystemBackend.watch()` is built on. Its job is narrow, and its +contract is explicit. It calls `onChange` whenever the watched +directory's contents may have changed. Spurious wakes are allowed, +because `drainAvailable()` is always the actual authority on what can be +claimed. A false-positive wake costs one wasted scan. It is never a +correctness problem. Three platform strategies exist, tried in order of preference. -`watchKQueue` (Darwin only, meaning macOS and iOS) opens the directory with -`O_EVTONLY` and attaches a `DispatchSource` watching kqueue VNODE events — -write, extend, delete, rename — giving wake latency in the microseconds -with no external dependency. `watchLinux` first attempts `openInotify`, -which sets up an inotify watch for `IN_CREATE` and `IN_MOVED_TO` (the -second of which is the event the atomic tmp-to-new rename actually fires); -`runInotifyLoop` then polls that file descriptor with a 500-millisecond -timeout so that task cancellation is still checked promptly between -events, and reads and discards whatever event bytes arrive, since the -handler only needs to know that something happened, not what. If -`openInotify` cannot establish a watch — the kernel's inotify watch limit -is exhausted, or the filesystem does not support inotify — `watchLinux` -falls back automatically to `watchPoll`. Every other platform uses -`watchPoll` directly: a plain 200-millisecond snapshot comparison of the -directory's file listing, correct everywhere but capable of missing a -change for up to one poll interval (100 milliseconds on average). +`watchKQueue` runs only on Darwin platforms, meaning macOS and iOS. It +opens the directory with `O_EVTONLY`. It attaches a `DispatchSource` +watching kqueue VNODE +events: write, extend, delete, and rename. This strategy gives wake +latency in the microseconds, with no external dependency. + +`watchLinux` first attempts `openInotify`. `openInotify` sets up an +inotify watch for `IN_CREATE` and `IN_MOVED_TO`. The second of these is +the event the atomic tmp-to-new rename actually fires. `runInotifyLoop` +then polls that file descriptor with a five-hundred-millisecond timeout, +so that task cancellation is still checked promptly between events. It +reads and discards whatever event bytes arrive. The handler only needs to +know that something happened. It does not need to know what happened. If +`openInotify` cannot establish a watch, because the kernel's inotify +watch limit is exhausted or the filesystem does not support inotify, +`watchLinux` falls back automatically to `watchPoll`. + +Every other platform uses `watchPoll` directly. `watchPoll` is a plain +two-hundred-millisecond snapshot comparison of the directory's file +listing. It is correct everywhere, but it can miss a change for up to one +poll interval, about one hundred milliseconds on average. Every path calls `onChange` once immediately, before entering its -platform-specific wait loop, so that any work already present when the -watcher attaches is not stranded waiting for the next actual filesystem -event. `ContinuationBox` is the small bridge the Darwin path uses to let an -`async` caller wait on a `DispatchSource`'s cancel handler, which fires on -an ordinary `DispatchQueue` callback rather than inside Swift's structured -concurrency. +platform-specific wait loop. This first call means any work already +present when the watcher attaches is not stranded waiting for the next +actual filesystem event. `ContinuationBox` is the small bridge the Darwin +path uses. It lets an `async` caller wait on a `DispatchSource`'s cancel +handler. That handler fires on an ordinary `DispatchQueue` callback, +rather than inside Swift's structured concurrency. ## QueueKitTelemetry.swift -This file provides `QueueLatencyWindow` and `reportQueueStats`, the -self-report telemetry QueueKit emits through IntellectusLib, MOOTx01's -telemetry library, after every `drain()` call. +This file provides `QueueLatencyWindow` and `reportQueueStats`. These are +the self-report telemetry QueueKit emits through IntellectusLib, +MOOTx01's telemetry library, after every `drain()` call. -The reporting path is additive and deliberately cheap when unused: the -very first check inside `reportQueueStats` is a single atomic flag read, -`Intellectus.isEnabled`, and the function returns immediately when it is -false, at a cost the file's header comment puts at about one nanosecond. -When telemetry is enabled, the function makes one `pendingCount()` call and -emits up to six named metrics, all tagged with the estate identifier and -the kit name so a fleet of estates can be compared metric by metric. +The reporting path is additive, and it is deliberately cheap when +unused. The very first check inside `reportQueueStats` is a single +atomic flag read, `Intellectus.isEnabled`. The function returns +immediately when that flag is false. The file's header comment puts the +cost of this check at about one nanosecond. When telemetry is enabled, +the function makes one `pendingCount()` call. It then emits up to six +named metrics. Every metric is tagged with the estate identifier and the +kit name, so a fleet of estates can be compared metric by metric. `QueueLatencyWindow` keeps a fixed-capacity rolling list of recent drain -latencies (one hundred samples by default) and computes an arbitrary -percentile from it on demand. `percentile(_:)` guards its input explicitly: -a percentile argument that is not a finite number between 0 and 100 (for -example `NaN` or a value produced by a caller's arithmetic error) returns -`0` rather than computing an array index from it, because converting a -non-finite floating-point value to an integer index can crash the program -outright on some inputs. - -`reportQueueStats` treats a failed `pendingCount()` read as its own signal, -not as a zero. Reporting `queue.depth = 0` when the read actually failed -would be indistinguishable from an honestly empty queue, and would tell -whoever is watching the telemetry "everything is drained" when the truth is -"the read itself did not work." Instead, a failed read skips the -`queue.depth` metric entirely, in favor of a dedicated -`queue.depth_unavailable` counter, and every metric that can only be -computed honestly from a known depth (`queue.idle_nonempty`, part of -`queue.head_of_line_age_s`) is skipped along with it. `queue.drain_count` -and the two latency percentiles have no such dependency and are always -reported. +latencies, one hundred samples by default. It computes an arbitrary +percentile from that list on demand. `percentile(_:)` guards its input +explicitly. A percentile argument that is not a finite number between +zero and one hundred, for example `NaN` or a value produced by a +caller's arithmetic error, returns `0` rather than computing an array +index from it. Converting a non-finite floating-point value to an +integer index can crash the program outright on some inputs. + +`reportQueueStats` treats a failed `pendingCount()` read as its own +signal, rather than as a zero. Reporting `queue.depth = 0` when the read +actually failed would be indistinguishable from an honestly empty queue. +It would tell whoever is watching the telemetry that everything is +drained, when the truth is that the read itself did not work. Instead, a +failed read skips the `queue.depth` metric entirely, in favor of a +dedicated `queue.depth_unavailable` counter. Every metric that can only +be computed honestly from a known depth is skipped along with it. +`queue.idle_nonempty` is one such metric, as is part of +`queue.head_of_line_age_s`. `queue.drain_count` and the two latency +percentiles have no such dependency, and they are always reported. ## Rust and Python Ports and Conformance -The `rust/` directory contains a second, independent implementation of the -package: eight source files (`error.rs`, `job.rs`, `backend.rs`, -`filesystem.rs`, `facade.rs`, `drain_lease.rs`, and, behind a -`persistencekit` feature flag, `persistencekit.rs`), reimplementing the -facade, both backends, and the drain lease for non-Apple hosts. The -`python/queuekit/` directory contains a third implementation covering only -`FilesystemBackend` — the package specification restricts Python to that -one backend, since Python tooling has no need of the PersistenceKit -database path. - -All three implementations are gated by the same conformance fixtures, kept -in `Tests/QueueKitTests/Fixtures/` as recorded job and completion-signal -input/output pairs. Each fixture pins one exact set of bytes; Rust's own -`rust/tests/conformance.rs` loads the identical fixture files the Swift -test suite reads and asserts that Rust's encoder produces the same bytes -Swift already produced. This is what turns "the wire format is a contract" -from an assertion in this document into something a test suite actually -checks: change a field name, a key order, or an encoding rule in any one -implementation, and the shared fixtures catch the drift immediately in -whichever implementation was not updated to match. +The `rust/` directory contains a second, independent implementation of +the package. It has eight source files: `error.rs`, `job.rs`, +`backend.rs`, `filesystem.rs`, `facade.rs`, `drain_lease.rs`, +`persistencekit.rs`, and `lib.rs`. A `persistencekit` feature flag gates +`persistencekit.rs`. These files reimplement the facade, both backends, +and the drain lease, for non-Apple hosts. + +The `python/queuekit/` directory contains a third implementation. It +covers only `FilesystemBackend`. The package specification restricts +Python to that one backend, since Python tooling has no need of the +PersistenceKit database path. + +All three implementations are gated by the same conformance fixtures. +These fixtures live in `Tests/QueueKitTests/Fixtures/`, as recorded job +and completion-signal input and output pairs. Each fixture pins one +exact set of bytes. Rust's own `rust/tests/conformance.rs` loads the +identical fixture files the Swift test suite reads. It asserts that +Rust's encoder produces the same bytes Swift already produced. This +mechanism is what turns "the wire format is a contract" from an +assertion in this document into something a test suite actually checks. +Suppose a field name, a key order, or an encoding rule changes in any one +implementation. The shared fixtures catch the drift immediately. They +catch it in whichever implementation was not updated to match. diff --git a/packages/kits/QueueKit/docs/OVERVIEW.md b/packages/kits/QueueKit/docs/OVERVIEW.md index c328d9c..25fa763 100644 --- a/packages/kits/QueueKit/docs/OVERVIEW.md +++ b/packages/kits/QueueKit/docs/OVERVIEW.md @@ -32,141 +32,151 @@ sources: ## What This Library Does QueueKit is a general-purpose work queue. It lets one part of MOOTx01 hand a -unit of work to another part without the two parts running in the same -process, or even at the same moment. MOOTx01 is an on-device AI memory -system. It stores what an AI observes over time and helps the AI recall it -later. Many of its jobs — encoding a memory, running a dreaming pass over the -estate, replaying a signal — are best done off to the side, by a separate -worker, so the part of the system that noticed the work does not have to wait -for it to finish. +unit of work to another part. The two parts need not run in the same +process. They need not even run at the same moment. + +MOOTx01 is an on-device AI memory system. It stores what an AI observes over +time. It helps the AI recall that record later. Many of its jobs are best +done off to the side, by a separate worker. Encoding a memory is one +example. Running a dreaming pass over the estate is another. Replaying a +signal is a third. Handing off this work means the part of the system that +noticed it does not have to wait for it to finish. QueueKit is a Kit. A Kit is a larger package that composes libraries into a -subsystem; kits may depend on libraries, never the reverse. QueueKit lives in -moot-system, the repository that houses the kits that hold MOOTx01 together -at the system level. +subsystem. A kit may depend on a library. A library never depends back on a +kit. QueueKit lives in moot-system, the repository that houses the kits that +hold MOOTx01 together at the system level. The unit of work is called a `Job`. A job carries an identifier, the name of -the stream it belongs to, a submission timestamp, a priority, and an -arbitrary payload of bytes. A producer calls `send` to hand a job to the -queue. A consumer calls `drain` to claim available jobs, does the work, and -calls `reply` to mark each job finished. QueueKit exposes exactly four -permanent methods for this cycle — `send`, `drain`, `watch`, `reply` — no -matter which storage mechanism backs the queue underneath. +the stream it belongs to, and a submission timestamp. It also carries a +priority and an arbitrary payload of bytes. A producer calls `send` to hand +a job to the queue. A consumer calls `drain` to claim available jobs. The +consumer does the work, then calls `reply` to mark each job finished. +QueueKit exposes exactly four permanent methods for this cycle: `send`, +`drain`, `watch`, and `reply`. Every backend supports all four the same way, +no matter which storage mechanism sits underneath it. ## The Problem It Solves Handing work across a process boundary is harder than it looks. Three -requirements make it hard, and QueueKit exists to meet all three at once. +requirements make it hard. QueueKit exists to meet all three at once. -First, the handoff must survive a crash. If the producing process, the -queue itself, or the consuming process dies partway through, no job may be -silently lost or silently claimed twice. A job is claimed exactly once at a -time, and a completed job leaves a durable record of how it finished. +First, the handoff must survive a crash. The producing process might die +partway through. The queue itself might die. The consuming process might +die. In every case, no job may be lost without a trace, and no job may be +claimed twice at once. A job is claimed exactly once at a time. A completed +job leaves a durable record of how it finished. -Second, independent workloads must not interfere with each other while -sharing one queue. An encode worker and a dreaming worker might both draw -from the same estate's queue, but neither should be able to claim, or block -on, the other's jobs. QueueKit calls a workload a stream, and every -operation has a stream-scoped form so that consumers can share one queue -without stepping on each other. +Second, independent workloads must not interfere with each other while they +share one queue. An encode worker and a dreaming worker might both draw from +the same estate's queue. Neither should be able to claim, or block on, the +other's jobs. QueueKit calls a workload a stream. Every operation has a +stream-scoped form, so consumers can share one queue without stepping on +each other. Third, the format on disk or in the database must mean the same thing no -matter which programming language wrote or reads it. MOOTx01 ships Swift -code for Apple devices, Rust code for cross-platform daemons and services, -and Python code for command-line tooling. A job enqueued by one language's -producer must be decodable, byte for byte, by another language's consumer. -QueueKit therefore treats its wire format — the exact bytes a job or a -completion signal is encoded as — as a contract, gated by conformance -fixtures shared across all three implementations. +matter which programming language wrote it or read it. MOOTx01 ships Swift +code for Apple devices. It ships Rust code for cross-platform daemons and +services. It ships Python code for command-line tooling. A job enqueued by +one language's producer must be decodable, byte for byte, by another +language's consumer. QueueKit treats its wire format as a contract. The wire +format is the exact bytes a job or a completion signal is encoded as. +Conformance fixtures shared across all three implementations gate this +contract. ## How It Works QueueKit separates the fixed four-method interface from the storage mechanism underneath it. A backend is anything that conforms to the -`QueueBackend` protocol: it can write a job, claim the available jobs, watch -for new ones, and mark a job complete. The public `QueueKit` class does not -know or care which backend it holds; it forwards every call and adds only -the telemetry and the await-drain convenience described below. +`QueueBackend` protocol. A backend can write a job, claim the available +jobs, watch for new ones, and mark a job complete. The public `QueueKit` +class does not know or care which backend it holds. It forwards every call. +It adds only the telemetry and the await-drain convenience described below. Two backends ship today. `FilesystemBackend` needs no database. It lays out -four subdirectories under a root — `tmp`, `new`, `cur`, `done` — following -the maildir layout that mail servers such as Postfix have used for decades -to hand off messages between processes without a lock file: a job is written -into `tmp`, then atomically renamed into `new`; claiming it is another atomic -rename into `cur`; finishing it renames it into `done`. Every step relies on -the filesystem's own atomicity guarantee, so two processes racing to claim -the same job can never both win. +four subdirectories under a root: `tmp`, `new`, `cur`, and `done`. This +follows the maildir layout that mail servers such as Postfix have used for +decades, to hand off messages between processes without a lock file. A job +is written into `tmp`, then renamed atomically into `new`. Claiming it is +another atomic rename, into `cur`. Finishing it renames it into `done`. +Every step relies on the filesystem's own atomicity guarantee. Two +processes racing to claim the same job can never both win. `PersistenceKitBackend` stores jobs as rows in a table inside PersistenceKit, MOOTx01's storage kit. It suits an estate that already keeps a shared, encrypted SQLite database open for other purposes. Claiming a batch of jobs -here is one guarded database transaction that flips every available row from -`new` to `cur` under a single session identifier, then reads back exactly the -rows that transaction claimed — an approach that scales as one pass over the -table rather than one query per job. +here is one guarded database transaction. That transaction flips every +available row from `new` to `cur`, under a single session identifier. It +then reads back exactly the rows that transaction claimed. This scales as +one pass over the table, rather than as one query per job. Every job carries a submission timestamp built from a Hybrid Logical Clock, -or HLC — a clock design that combines wall-clock time with a small counter -so that events from different machines or processes still sort into one -consistent, gap-free order, even when the machines' clocks disagree slightly. -QueueKit uses the HLC ordering to decide which job is oldest, both when a -consumer wants first-in-first-out delivery and when telemetry reports how -long the oldest waiting job has been sitting in the queue. +or HLC. An HLC is a clock design that combines wall-clock time with a small +counter. Events from different machines or processes still sort into one +gap-free order this way, even when the machines' clocks disagree slightly. +QueueKit uses the HLC ordering to decide which job is oldest. This matters +when a consumer wants first-in-first-out delivery. It also matters when +telemetry reports how long the oldest waiting job has been sitting in the +queue. Two supporting mechanisms keep multi-process operation honest. `DrainLease` -is a heartbeat lock, keyed by stream, that lets many processes share one -estate while guaranteeing that only one of them actively drains a given -stream at a time; a stale lease (its holder has not renewed it recently) is -automatically taken over. `Watcher` gives the filesystem backend an -efficient way to notice new work without spinning a loop: it uses the -kqueue mechanism on Apple platforms, inotify on Linux, and a plain -200-millisecond poll everywhere else, falling back automatically if the -faster mechanism cannot be set up. +is a heartbeat lock, keyed by stream. It lets many processes share one +estate, while guaranteeing that only one of them actively drains a given +stream at a time. A stale lease, one whose holder has not renewed it +recently, is automatically taken over. `Watcher` gives the filesystem +backend an efficient way to notice new work without spinning a loop. It +uses the kqueue mechanism on Apple platforms. It uses inotify on Linux. +Everywhere else, it falls back to a plain poll every two hundred +milliseconds. It falls back automatically whenever the faster mechanism +cannot be set up. ## How the Pieces Fit -Figure 1 shows the library's topology — its major parts and how a job moves +Figure 1 shows the library's topology: its major parts, and how a job moves through them. ![Figure 1. Topology of QueueKit](topology.svg) -*Figure 1. Topology of QueueKit. A producer calls the `QueueKit` facade, -which delegates to whichever backend is mounted. Dashed regions mark -external kit boundaries — the encrypted database PersistenceKit owns, and -the telemetry sink IntellectusLib owns — plus the maildir directories that -are the filesystem backend's own on-disk state.* - -A caller mounts `QueueKit` once, over either backend, and afterward talks -only to the facade. `send` and the batch form `send(batch:)` hand jobs to -the backend. `drain` and the stream-scoped `drain(stream:)` claim available -jobs and return them together with a session identifier that groups -everything claimed in that one call. `watch` subscribes a handler that fires -for every job as it becomes available, driven by `Watcher` on the filesystem -backend and by a database change observer on the PersistenceKit backend. -`reply` — in its single-job, per-session, and per-batch forms — marks -claimed jobs finished with a terminal status and, on the filesystem backend, -writes a durable signal file recording how the job ended. - -`awaitDrain` is a convenience built on top of the four permanent methods -rather than a new backend capability: it polls the pending count and the -in-flight count until both reach zero, so a bulk caller — an importer -finishing a batch, a test verifying a pipeline emptied — can wait for every -enqueued job to finish without wiring its own polling loop. +*Figure 1. Topology of QueueKit. A producer calls the `QueueKit` facade. The +facade delegates to whichever backend is mounted. Dashed regions mark +external kit boundaries. One boundary is the encrypted database that +PersistenceKit owns. Another is the telemetry sink that IntellectusLib +owns. A third dashed region marks the maildir directories, the filesystem +backend's own on-disk state.* + +A caller mounts `QueueKit` once, over either backend. Afterward, the caller +talks only to the facade. `send` and the batch form `send(batch:)` hand +jobs to the backend. `drain` and the stream-scoped `drain(stream:)` claim +available jobs. Both return the claimed jobs together with a session +identifier. That identifier groups everything claimed in one call. `watch` +subscribes a handler that fires for every job as it becomes available. On +the filesystem backend, `Watcher` drives this. On the PersistenceKit +backend, a database change observer drives it instead. `reply` marks +claimed jobs finished with a terminal status. It has a single-job form, a +per-session form, and a per-batch form. On the filesystem backend, `reply` +also writes a durable signal file recording how the job ended. + +`awaitDrain` is a convenience. It is built on top of the four permanent +methods, rather than added as a new backend capability. It polls the +pending count and the in-flight count until both reach zero. A bulk caller +can use it to wait for every enqueued job to finish, without writing its own +polling loop. An importer finishing a batch is one example. A test verifying +that a pipeline emptied is another. Telemetry is additive and off by default. When enabled, `QueueKitTelemetry` -reports queue depth, drain throughput, claim latency percentiles, and the -age of the oldest waiting job to IntellectusLib, MOOTx01's self-report -telemetry library, tagged by estate so a fleet of estates can be watched -side by side. +reports queue depth, drain throughput, and claim latency percentiles. It +also reports the age of the oldest waiting job. It sends every report to +IntellectusLib, MOOTx01's self-report telemetry library. Each report is +tagged by estate, so a fleet of estates can be watched side by side. ## What Ships in the Package -The package ships ten Swift source files implementing the facade, the two -backends, and their supporting infrastructure; a Rust port in `rust/` (eight -source files, roughly three thousand lines) that reimplements both backends -for non-Apple hosts; and a Python port in `python/queuekit/` that -reimplements the filesystem backend only, for command-line tooling that has -no need of a database backend. A shared set of conformance fixtures under -`Tests/QueueKitTests/Fixtures/` — recorded job and completion-signal -input/output pairs — gates all three implementations to the same wire -format, byte for byte. +The package ships ten Swift source files. These implement the facade, the +two backends, and their supporting infrastructure. A Rust port lives in +`rust/`. It has eight source files and roughly three thousand lines. It +reimplements both backends, for non-Apple hosts. A Python port lives in +`python/queuekit/`. It reimplements the filesystem backend only, since +command-line tooling has no need of a database backend. A shared set of +conformance fixtures lives under `Tests/QueueKitTests/Fixtures/`. These are +recorded job and completion-signal input and output pairs. They gate all +three implementations to the same wire format, byte for byte. diff --git a/packages/kits/QueueKit/docs/topology.svg b/packages/kits/QueueKit/docs/topology.svg index e688be5..a69eacb 100644 --- a/packages/kits/QueueKit/docs/topology.svg +++ b/packages/kits/QueueKit/docs/topology.svg @@ -110,13 +110,13 @@ sources: tmp/ → new/ → cur/ → done/ - External kit boundary — PersistenceKit + External kit boundary: PersistenceKit queue.sqlite queuekit_jobs table - External kit boundary — IntellectusLib + External kit boundary: IntellectusLib Telemetry sink off by default, ~1 ns gate diff --git a/packages/libs/LoopbackHTTP/docs/AGENT_MAP.md b/packages/libs/LoopbackHTTP/docs/AGENT_MAP.md index 2cf0aa5..2bede22 100644 --- a/packages/libs/LoopbackHTTP/docs/AGENT_MAP.md +++ b/packages/libs/LoopbackHTTP/docs/AGENT_MAP.md @@ -11,69 +11,69 @@ sources: blob: 6dfba4dee7ad3121d47c8bfc937854c6f13ad1b1 --- -# AGENT_MAP — LoopbackHTTP +# AGENT_MAP: LoopbackHTTP PURPOSE: zero-dependency, loopback-pinned HTTP/1.1 server primitive (transport + wire parsing/framing only; no auth/policy). Bind loopback socket → accept → recv bytes → HTTPRequest.read parses → caller builds HTTPResponse or drives SSEStream → send/sendAll writes bytes back. -DEPS: imports Foundation (HTTPWire.swift); Glibc (Linux) / Darwin (Apple) via `#if canImport(Glibc)` guard (POSIXSocket.swift). No external SwiftPM packages (zero-dependency kit rule). No rust/ port — extraction target is OS transport glue, exempt from the Swift/Rust parity discipline (ADR-LOOPBACKHTTP-001); parity for the MCP transport is enforced at the JSON-RPC wire, not here. Imported by: moot-mgr (monitor daemon dashboard read-API + control listener), resident mootx01 MCP daemon (JSON-RPC transport, SSE notifications). +DEPS: imports Foundation (HTTPWire.swift); Glibc (Linux) / Darwin (Apple) via `#if canImport(Glibc)` guard (POSIXSocket.swift). No external SwiftPM packages (zero-dependency kit rule). No rust/ port; extraction target is OS transport glue, exempt from the Swift/Rust parity discipline (ADR-LOOPBACKHTTP-001); parity for the MCP transport is enforced at the JSON-RPC wire, not here. Imported by: moot-mgr (monitor daemon dashboard read-API + control listener), resident mootx01 MCP daemon (JSON-RPC transport, SSE notifications). ENTRY POINTS (most callers need only these): -- POSIXSocket.swift:43 `POSIXSocket.listenLoopbackTCP(port:) throws -> (fd, port)` — bind+listen, loopback-only -- HTTPWire.swift:95 `HTTPRequest.read(fd:maxHeaderBytes:maxBodyBytes:) -> HTTPRequest?` — parse one request off a connected fd -- HTTPWire.swift:201 `HTTPResponse.send(fd:)` — serialize + write one response -- HTTPWire.swift:252/:259 `SSEStream.writeHead()` / `.send(_:)` — SSE head + data frames +- POSIXSocket.swift:43 `POSIXSocket.listenLoopbackTCP(port:) throws -> (fd, port)`; bind+listen, loopback-only +- HTTPWire.swift:95 `HTTPRequest.read(fd:maxHeaderBytes:maxBodyBytes:) -> HTTPRequest?`; parse one request off a connected fd +- HTTPWire.swift:201 `HTTPResponse.send(fd:)`; serialize + write one response +- HTTPWire.swift:252/:259 `SSEStream.writeHead()` / `.send(_:)`; SSE head + data frames ## Symbol Table -### Transport — POSIXSocket.swift -- :32 `enum POSIXSocket` — namespace; all calls synchronous/blocking, run off cooperative pool -- :43 `listenLoopbackTCP(port:) throws -> (fd: Int32, port: UInt16)` — binds hard-pinned to 127.0.0.1 (never INADDR_ANY); port 0 = OS-assigned, read back via getsockname; listen backlog 16 -- :91 `listenUnix(path:) throws -> Int32` — UDS listener; unlinks stale path first; chmod 0600 BEFORE listen (closes bindable-but-world-readable window); listen backlog 16 -- :134 `acceptOne(_ listenFD:) -> Int32?` — accept one conn; nil on failure, caller decides retry -- :141 `recv(_:max:) -> Data?` — up to `max` bytes; empty non-nil Data = clean EOF; nil = read error -- :150 `sendAll(_:_:) -> Bool` — loops write() until all bytes sent or a write fails; single source of truth for "fully flushed to socket" used by both HTTPResponse.send and SSEStream -- :166 `enum SocketError: Error, Sendable, Equatable` — structured errors (project rule: enums not bare optionals) -- :168 `case syscall(String, Int32)` — named syscall + errno -- :170 `case pathTooLong` — UDS path exceeds fixed sun_path buffer +### Transport: POSIXSocket.swift +- :32 `enum POSIXSocket`: namespace; all calls synchronous/blocking, run off cooperative pool +- :43 `listenLoopbackTCP(port:) throws -> (fd: Int32, port: UInt16)`: binds hard-pinned to 127.0.0.1 (never INADDR_ANY); port 0 = OS-assigned, read back via getsockname; listen backlog 16 +- :91 `listenUnix(path:) throws -> Int32`: UDS listener; unlinks stale path first; chmod 0600 BEFORE listen (closes bindable-but-world-readable window); listen backlog 16 +- :134 `acceptOne(_ listenFD:) -> Int32?`: accept one conn; nil on failure, caller decides retry +- :141 `recv(_:max:) -> Data?`: up to `max` bytes; empty non-nil Data = clean EOF; nil = read error +- :150 `sendAll(_:_:) -> Bool`: loops write() until all bytes sent or a write fails; single source of truth for "fully flushed to socket" used by both HTTPResponse.send and SSEStream +- :166 `enum SocketError: Error, Sendable, Equatable`: structured errors (project rule: enums not bare optionals) +- :168 `case syscall(String, Int32)`: named syscall + errno +- :170 `case pathTooLong`: UDS path exceeds fixed sun_path buffer -### Request — HTTPWire.swift -- :27 `struct HTTPRequest: Sendable` — method/path(no query)/query(raw, no '?')/headers(lowercased keys)/body -- :36 `init(method:path:query:headers:body:)` — public memberwise; used directly by tests to synthesize requests -- :46 `bearerToken: String?` — reads Authorization, case-insensitive "bearer " prefix strip + trim; CONVENIENCE ONLY, no validation -- :54 `origin: String?` — raw Origin header passthrough; CONVENIENCE ONLY, no allow-list check -- :69 `wantsEventStream: Bool` — Accept:text/event-stream OR exact query param `stream=1` (split on '&', exact match — NOT substring; see GOTCHAS) -- :95 `static func read(fd:maxHeaderBytes:maxBodyBytes:) -> HTTPRequest?` — buffers until "\r\n\r\n"; nil if header block exceeds maxHeaderBytes before terminator found, or on recv failure -- :114 `private static func parse(buffer:headerEnd:fd:maxBodyBytes:) -> HTTPRequest?` — request-line split, header lowercase-key parse, Content-Length-bounded body read (truncates silently at maxBodyBytes, does not error) +### Request: HTTPWire.swift +- :27 `struct HTTPRequest: Sendable`: method/path(no query)/query(raw, no '?')/headers(lowercased keys)/body +- :36 `init(method:path:query:headers:body:)`: public memberwise; used directly by tests to synthesize requests +- :46 `bearerToken: String?`: reads Authorization, case-insensitive "bearer " prefix strip + trim; CONVENIENCE ONLY, no validation +- :54 `origin: String?`: raw Origin header passthrough; CONVENIENCE ONLY, no allow-list check +- :69 `wantsEventStream: Bool`: Accept:text/event-stream OR exact query param `stream=1` (split on '&', exact match; NOT substring; see GOTCHAS) +- :95 `static func read(fd:maxHeaderBytes:maxBodyBytes:) -> HTTPRequest?`: buffers until "\r\n\r\n"; nil if header block exceeds maxHeaderBytes before terminator found, or on recv failure +- :114 `private static func parse(buffer:headerEnd:fd:maxBodyBytes:) -> HTTPRequest?`: request-line split, header lowercase-key parse, Content-Length-bounded body read (truncates silently at maxBodyBytes, does not error) -### Response — HTTPWire.swift -- :164 `struct HTTPResponse: Sendable` — status/headers/body, caller-composed (not a closed consumer-specific case set — see GOTCHAS) -- :169 `init(status:headers:body:)` — headers/body default to empty -- :176 `static func json(status:body:) -> HTTPResponse` — Content-Type: application/json -- :182 `static func asset(contentType:body:) -> HTTPResponse` — 200 + Cache-Control: no-store (redeployed UI never served stale) -- :191 `static var notFound: HTTPResponse` — 404 + `{"error":"not_found"}` -- :201 `func send(fd:)` — computes Content-Length from actual body (overrides any caller value); header order: Content-Type, Content-Length, then rest sorted alphabetically, then Connection: close ALWAYS last (caller-supplied Connection does not suppress it) -- :221 `static func reason(_ status:) -> String` — reason phrase for 200/400/401/403/404/413/500; default "OK" +### Response: HTTPWire.swift +- :164 `struct HTTPResponse: Sendable`: status/headers/body, caller-composed (not a closed consumer-specific case set; see GOTCHAS) +- :169 `init(status:headers:body:)`: headers/body default to empty +- :176 `static func json(status:body:) -> HTTPResponse`: Content-Type: application/json +- :182 `static func asset(contentType:body:) -> HTTPResponse`: 200 + Cache-Control: no-store (redeployed UI never served stale) +- :191 `static var notFound: HTTPResponse`: 404 + `{"error":"not_found"}` +- :201 `func send(fd:)`: computes Content-Length from actual body (overrides any caller value); header order: Content-Type, Content-Length, then rest sorted alphabetically, then Connection: close ALWAYS last (caller-supplied Connection does not suppress it) +- :221 `static func reason(_ status:) -> String`: reason phrase for 200/400/401/403/404/413/500; default "OK" -### SSE — HTTPWire.swift -- :242 `struct SSEStream: Sendable` — long-lived push connection wrapper; caller owns source/cadence/lifetime/fd close +### SSE: HTTPWire.swift +- :242 `struct SSEStream: Sendable`: long-lived push connection wrapper; caller owns source/cadence/lifetime/fd close - :245 `init(fd:)` -- :252 `func writeHead() -> Bool` — call exactly once before any frame; false = peer already gone -- :259 `func send(_ payload:) -> Bool` — writes `"data: \(payload)\n\n"`; false = write failed, caller should stop -- :264 `static let responseHead: Data` — fixed head: 200 OK, text/event-stream, no-cache, keep-alive +- :252 `func writeHead() -> Bool`: call exactly once before any frame; false = peer already gone +- :259 `func send(_ payload:) -> Bool`: writes `"data: \(payload)\n\n"`; false = write failed, caller should stop +- :264 `static let responseHead: Data`: fixed head: 200 OK, text/event-stream, no-cache, keep-alive ## INVARIANTS / GOTCHAS -- AUTH-FREE INVARIANT (ADR-LOOPBACKHTTP-001 condition 3): no auth/policy logic anywhere in this package. `bearerToken`/`origin` are read-only conveniences; accept/reject is composed ABOVE this library by each consumer (none in Community Edition, bearer+Origin in moot-mgr, OAuth 2.1 in EE-only remote layer). Do not add auth enforcement here — it would break the "one binary, all editions" property. +- AUTH-FREE INVARIANT (ADR-LOOPBACKHTTP-001 condition 3): no auth/policy logic anywhere in this package. `bearerToken`/`origin` are read-only conveniences; accept/reject is composed ABOVE this library by each consumer (none in Community Edition, bearer+Origin in moot-mgr, OAuth 2.1 in EE-only remote layer). Do not add auth enforcement here; it would break the "one binary, all editions" property. - LOOPBACK-ONLY BY CONSTRUCTION: `listenLoopbackTCP` binds the literal `127.0.0.1` (0x7F000001 big-endian), never `INADDR_ANY`/0.0.0.0. Do not parameterize the bind address to accept a caller-supplied host. -- UDS PERMISSION WINDOW: `listenUnix` must chmod 0600 BEFORE `listen()`, not after — the current ordering closes the world-readable window; do not reorder. -- SSE CSRF NOTE: `wantsEventStream` via `?stream=1` is a plain GET, so browser CORS preflight does NOT run. A cross-origin page can open an SSE connection if the loopback port is reachable. Any consumer serving sensitive data over SSE MUST check `request.origin` itself before upgrading — this library does not and will not enforce it (see AUTH-FREE INVARIANT). +- UDS PERMISSION WINDOW: `listenUnix` must chmod 0600 BEFORE `listen()`, not after; the current ordering closes the world-readable window; do not reorder. +- SSE CSRF NOTE: `wantsEventStream` via `?stream=1` is a plain GET, so browser CORS preflight does NOT run. A cross-origin page can open an SSE connection if the loopback port is reachable. Any consumer serving sensitive data over SSE MUST check `request.origin` itself before upgrading; this library does not and will not enforce it (see AUTH-FREE INVARIANT). - `wantsEventStream` query check is EXACT-MATCH on a `&`-split parameter, not `query.contains("stream=1")`. The substring form was a real bug (matched "stream=10", "mystream=1", "x=stream=1"); regression-locked by Tests/LoopbackHTTPTests/WantsEventStreamTests.swift. Do not revert to substring matching. -- Header names are lowercased on parse (`parse(buffer:...)`) so all internal/consumer lookups (`headers["content-length"]`, `headers["authorization"]`) use lowercase keys only — HTTP header names are case-insensitive on the wire but this dictionary is not. +- Header names are lowercased on parse (`parse(buffer:...)`) so all internal/consumer lookups (`headers["content-length"]`, `headers["authorization"]`) use lowercase keys only; HTTP header names are case-insensitive on the wire but this dictionary is not. - Body truncation, not rejection: `HTTPRequest.read`/`parse` silently truncate a body at `maxBodyBytes` when Content-Length exceeds it; they do not return nil or an error. A listener that must reject oversize bodies (e.g. the MCP `tools/call` listener) needs to check Content-Length itself before/after calling, or size its cap so truncation cannot corrupt a valid request. -- `maxHeaderBytes`/`maxBodyBytes` are per-call parameters, not fixed constants (default 64 KiB each) — deliberately, so a small dashboard control listener and a larger MCP listener each pick their own cap (ADR-LOOPBACKHTTP-001 condition 2). +- `maxHeaderBytes`/`maxBodyBytes` are per-call parameters, not fixed constants (default 64 KiB each); deliberately, so a small dashboard control listener and a larger MCP listener each pick their own cap (ADR-LOOPBACKHTTP-001 condition 2). - `HTTPResponse.send` always appends `Connection: close` last, even if the caller supplied their own `Connection` header (that value is emitted earlier, in the sorted section, and does not suppress the trailing one). Every response this library writes closes the connection; there is no keep-alive path for ordinary responses (SSE is the sole long-lived exception). -- `HTTPResponse`/`SSEStream` are open, caller-composed value types, not a closed enum of consumer-specific response shapes — this genericity (added in P1a) is what let a second consumer (resident MCP daemon) reuse the library without modifying it. Do not narrow them back to a fixed case set for one consumer's convenience. +- `HTTPResponse`/`SSEStream` are open, caller-composed value types, not a closed enum of consumer-specific response shapes; this genericity (added in P1a) is what let a second consumer (resident MCP daemon) reuse the library without modifying it. Do not narrow them back to a fixed case set for one consumer's convenience. - All `POSIXSocket` calls are synchronous/blocking. Callers must run them off a dedicated Thread or `Task.detached`, never inline on a thread that must stay responsive. - Network.framework (`NWListener`) was tried and rejected: it returns POSIXErrorCode 22/EINVAL binding a listening socket in this project's command-line (non-app-bundle) build environment, on every configuration tested. Do not reintroduce it as "simpler" without re-verifying that constraint is gone. - Swift-only, no Rust port: this library is OS-transport glue, not deterministic substrate compute, so it is exempt from the Swift/Rust byte-parity discipline that governs libraries like LatticeLib. ARIA_MCP-rust hand-rolls its own std::net transport under a no-FFI rule; the two sides agree only at the JSON-RPC message level. -- Zero external SwiftPM dependencies (Package.swift target has none) — adding one requires re-justifying against the kit dependency rules that motivated this extraction in the first place. +- Zero external SwiftPM dependencies (Package.swift target has none); adding one requires re-justifying against the kit dependency rules that motivated this extraction in the first place. diff --git a/packages/libs/LoopbackHTTP/docs/DETAILS.md b/packages/libs/LoopbackHTTP/docs/DETAILS.md index 36b409d..4a14340 100644 --- a/packages/libs/LoopbackHTTP/docs/DETAILS.md +++ b/packages/libs/LoopbackHTTP/docs/DETAILS.md @@ -14,237 +14,262 @@ sources: # LoopbackHTTP Details This document walks through every source file in the package. Read -`OVERVIEW.md` first for the big picture. Files appear here in pipeline order: -the socket transport first, then the HTTP protocol layer built on top of it. +`OVERVIEW.md` first for the big picture. Files appear here in pipeline +order. The socket transport comes first, then the HTTP protocol layer +built on top of it. ## POSIXSocket.swift -This file provides `POSIXSocket`, a set of small blocking-socket helpers, and -`SocketError`, the error type they throw. Everything in this file is -synchronous: a call blocks the calling thread until the operating system -answers. Callers are expected to run these calls off a dedicated thread or a -detached task, never on a thread that must stay responsive. +This file provides `POSIXSocket`, a set of small blocking-socket helpers. +It also provides `SocketError`, the error type they throw. Everything in +this file is synchronous. A call blocks the calling thread until the +operating system answers. Callers should run these calls off a dedicated +thread or a detached task. They should never run on a thread that must stay +responsive. Two design choices explain the whole file. First, the code targets plain -POSIX sockets rather than Apple's `Network.framework`, because that -higher-level framework refused to bind a listening server socket in this -project's command-line build environment; it returned a low-level POSIX error -code on every configuration tried, a restriction tied to running outside an -app bundle. Plain sockets bind correctly in that same environment, so the +POSIX sockets rather than Apple's `Network.framework`. That higher-level +framework refused to bind a listening server socket in this project's +command-line build environment. It returned a low-level POSIX error code on +every configuration tried. The restriction is tied to running outside an +app bundle. Plain sockets bind correctly in that same environment. The library uses them directly, and the same security guarantees apply either -way. Second, the file imports `Glibc` on Linux and `Darwin` on Apple -platforms through a compile-time check, so the same source file compiles on -both without any conditional logic beyond the import line. - -`listenLoopbackTCP(port:)` opens a TCP listening socket and binds it to -`127.0.0.1`, never to `0.0.0.0` (which would accept connections from any -network interface, not just the local machine). The bind address is a -literal numeric value, not a name the operating system could resolve -differently, so this guarantee cannot be weakened by a misconfigured host -file or DNS entry. Passing port `0` lets the operating system pick an unused -port; the function reads the actual bound port back with `getsockname` and -returns it, which matters because the caller often needs to tell another -process which port to connect to. Why this function matters: it is the one -place in the library that decides which network interface can reach the -server at all, and it decides in favor of the strictest possible answer, -every time, regardless of what the caller asks for. - -`listenUnix(path:)` opens a Unix-domain socket at a filesystem path instead of -a network port. A Unix-domain socket is a connection endpoint that lives in -the filesystem and can only be reached by other processes on the same -machine; a web browser has no way to speak to one at all. This makes it a -natural channel for the most sensitive operations, since browser-based -attacks are structurally unable to reach it. The function removes any stale -socket file left over from a previous crashed run before binding, so a dead -file cannot block a fresh start. After binding, it changes the file's -permissions to owner-read-write-only (mode 0600) before calling `listen`, so -there is no window during which the socket file exists but is readable by -other users on the machine. Why this matters: the chmod step closing that -window is what turns "only this user can connect" from a hope into a -guarantee enforced by the filesystem itself. - -`acceptOne(_:)` accepts one pending connection on a listening socket and -returns the new connection's descriptor, or `nil` if the accept call failed. -It is a thin wrapper; the decision of whether and how to retry after a -failure belongs to the caller, not to this function. - -`recv(_:max:)` reads up to `max` bytes from a connected socket and returns -them as `Data`, or `nil` on a read error. An empty (but non-nil) result means -the peer closed the connection cleanly; a caller reading in a loop uses that -to know when to stop. - -`sendAll(_:_:)` writes every byte of `data` to a socket, looping until the -whole buffer is sent or a write fails. A single call to the underlying system -write is not guaranteed to send the whole buffer at once — the kernel is free -to accept less — so this function is the one place other files in the -package go to guarantee that a full response, or a full SSE frame, actually -reaches the wire. It returns `false` on any failed write, which a caller uses -as its signal that the peer is gone and the connection should be torn down. +way. + +Second, the file imports `Glibc` on Linux and `Darwin` on Apple platforms +through a compile-time check. The same source file compiles on both without +any conditional logic beyond the import line. + +`listenLoopbackTCP(port:)` opens a TCP listening socket. It binds the +socket to `127.0.0.1`, never to `0.0.0.0`. Binding to `0.0.0.0` would accept +connections from any network interface, not just the local machine. The +bind address is a literal numeric value. It is not a name the operating +system could resolve differently, so this guarantee cannot be weakened by a +misconfigured host file or DNS entry. + +Passing port zero lets the operating system pick an unused port. The +function reads the actual bound port back with `getsockname` and returns +it. This matters because the caller often needs to tell another process +which port to connect to. This function is the one place in the library +that decides which network interface can reach the server at all. It +decides in favor of the strictest possible answer every time, regardless of +what the caller asks for. + +`listenUnix(path:)` opens a Unix-domain socket at a filesystem path instead +of a network port. A Unix-domain socket is a connection endpoint that lives +in the filesystem. Only other processes on the same machine can reach it. A +web browser has no way to speak to one at all. This makes it a natural +channel for the most sensitive operations, since browser-based attacks +cannot structurally reach it. + +The function removes any stale socket file left over from a previous +crashed run before binding. A dead file cannot then block a fresh start. +After binding, it changes the file's permissions to owner-read-write-only +before calling `listen`. That mode is 0600. There is no window during which the +socket file exists but is readable by other users on the machine. This chmod +step is what turns "only this user can connect" from a hope into a +guarantee the filesystem itself enforces. + +`acceptOne(_:)` accepts one pending connection on a listening socket. It +returns the new connection's descriptor, or `nil` if the accept call +failed. It is a thin wrapper. The decision of whether and how to retry +after a failure belongs to the caller, not to this function. + +`recv(_:max:)` reads up to `max` bytes from a connected socket. It returns +them as `Data`, or `nil` on a read error. An empty but non-nil result means +the peer closed the connection cleanly. A caller reading in a loop uses that +signal to know when to stop. + +`sendAll(_:_:)` writes every byte of `data` to a socket. It loops until the +whole buffer is sent or a write fails. A single call to the underlying +system write is not guaranteed to send the whole buffer at once. The kernel +is free to accept less. This function is therefore the one place other +files in the package go to guarantee that a full response, or a full SSE +frame, actually reaches the wire. It returns `false` on any failed write. A +caller uses that as its signal that the peer is gone and the connection +should be torn down. `SocketError` is the file's structured error type. The package's error -handling rule prefers enumerated error cases over returning an optional -value with no explanation, so that a caller (or a log line) can say exactly -which system call failed and with which error number, rather than only that -something failed. `.syscall(String, Int32)` names the failing call and -carries its `errno` value; `.pathTooLong` covers the one Unix-domain-socket -case where the caller's path does not fit the fixed-size buffer the -operating system's socket address structure provides. +handling rule prefers enumerated error cases over an optional value with no +explanation. A caller, or a log line, can then say exactly which system +call failed. It can name the error number too, rather than only report that +something failed. `.syscall(String, Int32)` names the failing call and carries its +`errno` value. `.pathTooLong` covers the one Unix-domain-socket case where +the caller's path does not fit the fixed-size buffer the operating system's +socket address structure provides. ## HTTPWire.swift -This file provides three public types: `HTTPRequest`, a parsed incoming -request; `HTTPResponse`, an outgoing response the caller builds; and -`SSEStream`, the narrower Server-Sent-Events framing used for a long-lived -push connection. All three depend on `POSIXSocket` for the actual byte -transfer; nothing in this file opens a socket itself. +This file provides three public types. `HTTPRequest` is a parsed incoming +request. `HTTPResponse` is an outgoing response the caller builds. +`SSEStream` is the narrower Server-Sent-Events framing used for a +long-lived push connection. All three depend on `POSIXSocket` for the +actual byte transfer. Nothing in this file opens a socket itself. The file's scope is deliberately narrow. It is not a general-purpose HTTP -server: it parses a request line, a set of headers, and an optional body -bounded by a `Content-Length` header, and it writes exactly one response per +server. It parses a request line, a set of headers, and an optional body +bounded by a `Content-Length` header. It writes exactly one response per connection before that connection closes. It does not support chunked -transfer encoding, HTTP/1.1 keep-alive across many requests on the same -response path, or HTTP/2. Every response this library writes closes the -connection afterward (except an SSE stream, which the caller keeps open on -purpose). +transfer encoding, HTTP/1.1 keep-alive across many requests, or HTTP/2. +Every response this library writes closes the connection afterward. An SSE +stream is the one exception, since the caller keeps it open on purpose. ### HTTPRequest -`HTTPRequest` is a plain value holding the request `method` (for example, -`"GET"`), the `path` with any query string already removed, the raw `query` -string that followed the `?` (or an empty string if there was none), a -lowercase-keyed table of `headers`, and the request `body` as `Data`. Storing -the path and query separately, rather than leaving the caller to split a -combined URL string, means every consumer performs that split identically -instead of each one writing its own (and possibly buggy) parsing. - -`bearerToken` reads the `Authorization` header, checks that it begins with -the case-insensitive prefix `"bearer "`, and returns the remaining token text -with surrounding whitespace trimmed, or `nil` if the header is absent or does -not match. Why this matters: this is a convenience reader only. The library -extracts the token text so a caller does not have to; it never decides -whether that token is valid. The decision to accept or reject a request -based on this value belongs entirely to the code above this library, in -keeping with the package's edition-neutral, authentication-free design. +`HTTPRequest` is a plain value. It holds the request `method`, for example +`"GET"`. It holds the `path`, with any query string already removed. It +also holds the raw `query` string that followed the `?`. The `query` string +is empty if there was none. It also holds a lowercase-keyed table of `headers` and the +request `body` as `Data`. Storing the path and query separately means every +consumer performs that split identically. No consumer has to write its own, +possibly buggy, parsing. + +`bearerToken` reads the `Authorization` header. It checks that the header +begins with the case-insensitive prefix `"bearer "`. It returns the +remaining token text with surrounding whitespace trimmed, or `nil` if the +header is absent or does not match. This is a convenience reader only. The +library extracts the token text so a caller does not have to. It never +decides whether that token is valid. The decision to accept or reject a +request based on this value belongs entirely to the code above this +library. This keeps with the package's edition-neutral, authentication-free +design. `origin` reads the `Origin` header directly, with no further processing. A browser sets this header to say which web page's script initiated the -request. Like `bearerToken`, this is exposed purely so a caller can inspect -it; the library does not compare it against an allow-list itself. +request. Like `bearerToken`, this reader exists purely so a caller can +inspect the value. The library does not compare it against an allow-list +itself. `wantsEventStream` decides whether the caller likely wants an SSE response instead of an ordinary one. It returns `true` if the request's `Accept` -header mentions `text/event-stream`, or if the query string carries an exact -`stream=1` parameter. The exact-parameter check matters because an earlier -version of this logic used a substring check, `query.contains("stream=1")`, -which incorrectly matched `stream=10`, `mystream=1`, and `x=stream=1` — none -of which mean what `stream=1` means. The current implementation splits the -query on `&` and compares each resulting parameter for exact equality to -`"stream=1"`, which cannot be fooled by a longer number or an unrelated -parameter name that happens to contain the same substring. A dedicated -regression test file, `Tests/LoopbackHTTPTests/WantsEventStreamTests.swift`, -exists specifically to keep this fix from silently regressing. This same -property carries a documented security note: because an SSE request made -this way is an ordinary browser `GET` request, the browser's cross-origin +header mentions `text/event-stream`. It also returns `true` if the query +string carries an exact `stream=1` parameter. The exact-parameter check +matters for a specific reason. An earlier version of this logic used a +substring check, `query.contains("stream=1")`. That check incorrectly +matched `stream=10`, `mystream=1`, and `x=stream=1`. None of those mean what +`stream=1` means. + +The current implementation splits the query on `&`. It compares each +resulting parameter for exact equality to `"stream=1"`. A longer number or +an unrelated parameter name containing the same substring cannot fool it. +A dedicated regression test file, +`Tests/LoopbackHTTPTests/WantsEventStreamTests.swift`, exists specifically +to keep this fix from silently regressing. + +This same property carries a documented security note. An SSE request made +this way is an ordinary browser `GET` request. The browser's cross-origin preflight check never runs for it. A page on an unrelated origin can -therefore open an SSE connection to a loopback server if that server is -reachable at all. Any consumer that serves sensitive data over SSE must check -the `origin` value itself and refuse an unexpected origin before upgrading to -a stream; this library does not perform that check, again by the -authentication-free design. +therefore open an SSE connection to a loopback server, if that server is +reachable at all. Any consumer that serves sensitive data over SSE must +check the `origin` value itself. It must refuse an unexpected origin before +upgrading to a stream. This library does not perform that check, again by +the authentication-free design. `HTTPRequest.read(fd:maxHeaderBytes:maxBodyBytes:)` is the entry point that turns raw socket bytes into a parsed request. It reads repeatedly from the socket, accumulating bytes into a buffer, until it finds the header -terminator (`"\r\n\r\n"`, the blank line that ends an HTTP header block). If -the accumulated header block ever exceeds `maxHeaderBytes` before that -terminator appears, the function gives up and returns `nil` rather than -continuing to buffer an unbounded amount of untrusted network data — a cap -that protects the server's memory against a hostile or broken client sending -an endless header. The caps are parameters rather than fixed constants -because different listeners have different legitimate needs: a small -dashboard control listener wants a small cap, while a listener serving -`tools/call` requests for the Model Context Protocol wants a larger one, so -each caller can choose the cap that matches its own traffic without every -listener sharing one compromise value. - -The private `parse(buffer:headerEnd:fd:maxBodyBytes:)` helper does the actual -parsing once the header terminator has been found. It reads the request -line, splits it into method and target, and splits the target on the first -`?` into path and query. It then walks the remaining header lines, splitting -each on its first colon, trimming whitespace, and lowercasing the header -name so that later lookups (`headers["content-length"]`, for example) do not -have to guess whether a client sent `Content-Length` or `content-length` or -some other capitalization; HTTP header names are case-insensitive, and this -normalization is what makes the rest of the library able to treat them as -plain lowercase string keys. If a `Content-Length` header is present, the -function reads exactly that many more bytes from the socket, up to -`maxBodyBytes`. If the declared length exceeds the cap, the body is silently -truncated to the cap rather than the request being rejected; a caller for -whom truncation would be dangerous is expected to set its cap high enough -that truncation cannot happen, or to check the `Content-Length` header itself -before calling `read` at all. +terminator. The terminator is `"\r\n\r\n"`, the blank line that ends an HTTP +header block. If the accumulated header block ever exceeds +`maxHeaderBytes` before that terminator appears, the function gives up and +returns `nil`. It will not keep buffering an unbounded amount of untrusted +network data. This cap protects the server's memory against a hostile or +broken client sending an endless header. + +The caps are parameters rather than fixed constants, because different +listeners have different legitimate needs. A small dashboard control +listener wants a small cap. A listener serving `tools/call` requests for the +Model Context Protocol wants a larger one. Each caller can choose the cap +that matches its own traffic, without every listener sharing one compromise +value. + +The private `parse(buffer:headerEnd:fd:maxBodyBytes:)` helper does the +actual parsing once the header terminator has been found. It reads the +request line and splits it into method and target. It splits the target on +the first `?` into path and query. It then walks the remaining header +lines. Each line is split on its first colon. The function trims whitespace +from each side and lowercases the header name. + +This normalization matters. Later lookups, such as +`headers["content-length"]`, do not have to guess whether a client sent +`Content-Length` or `content-length` or some other capitalization. HTTP +header names are case-insensitive on the wire. This normalization is what +lets the rest of the library treat them as plain lowercase string keys. + +If a `Content-Length` header is present, the function reads exactly that +many more bytes from the socket, up to `maxBodyBytes`. If the declared +length exceeds the cap, the body is silently truncated to the cap rather +than the request being rejected. A caller for whom truncation would be +dangerous should set its cap high enough that truncation cannot happen. It +can also check the `Content-Length` header itself before calling `read` at +all. ### HTTPResponse -`HTTPResponse` is a plain value: a `status` code, a table of `headers`, and a -`body`. Unlike `HTTPRequest`, which the library builds for the caller, this -type is built by the caller and only serialized by the library. This -inversion is deliberate: the earlier version of this code hard-coded a -closed set of response shapes for one specific consumer, and generalizing it -to a plain value that any caller can construct is what let a second consumer -(the resident MCP transport) reuse the same library without waiting for new -cases to be added here every time its needs changed. +`HTTPResponse` is a plain value: a `status` code, a table of `headers`, and +a `body`. The library builds `HTTPRequest` for the caller. This type works +the other way around. The caller builds it, and the library only +serializes it. This inversion is deliberate. An earlier version of this +code hard-coded a closed set of response shapes for one specific consumer. +Generalizing it to a plain value that any caller can construct is what let +a second consumer, the resident MCP transport, reuse the same library +without waiting for new cases to be added here every time its needs +changed. -Three static convenience constructors cover the shapes most callers need +Three static convenience constructors cover the shapes most callers need, without limiting them to only those shapes. `json(status:body:)` builds a response with an `application/json` content type. `asset(contentType:body:)` -builds a `200 OK` response carrying `Cache-Control: no-store`, a header that -tells the browser never to reuse a cached copy; this matters for a locally -served dashboard, where a stale cached page after a new build would show -outdated controls or, worse, controls that no longer match the running -server's API. `notFound` is a ready-made `404` response with a small -consistent JSON error body, so every listener reports a missing route the -same way. +builds a `200 OK` response carrying `Cache-Control: no-store`. That header +tells the browser never to reuse a cached copy. This matters for a locally +served dashboard. A stale cached page after a new build would show outdated +controls, or worse, controls that no longer match the running server's API. +`notFound` is a ready-made `404` response with a small consistent JSON error +body. Every listener reports a missing route the same way. `send(fd:)` serializes the response and writes it to the socket through `POSIXSocket.sendAll`. It computes `Content-Length` itself from the actual -body byte count and overrides anything the caller supplied for that header, -because a caller-supplied length that does not match the real body would -produce a response the receiving HTTP client cannot parse correctly. Headers -are written in a fixed, deterministic order: `Content-Type` and -`Content-Length` first if present, then every other header sorted -alphabetically, and finally a `Connection: close` line that is always -appended last regardless of anything the caller set for that header. Fixing -the order this way is not required by the HTTP specification, which allows -headers in any order, but it makes every response byte-for-byte reproducible -for testing and logging, which matters more here than flexibility that no -caller needs. `reason(_:)` maps a status code to its human-readable reason -phrase (for example, `404` to `"Not Found"`) for the small fixed set of -statuses this library emits, falling back to `"OK"` for anything else since -the reason phrase is advisory text that HTTP clients are not required to -interpret. +body byte count. It overrides anything the caller supplied for that header. +A caller-supplied length that does not match the real body would produce a +response the receiving HTTP client cannot parse correctly. + +Headers are written in a fixed, deterministic order. `Content-Type` and +`Content-Length` come first, if present. Every other header follows, +sorted alphabetically. A `Connection: close` line is always appended last, +regardless of anything the caller set for that header. The HTTP +specification allows headers in any order, so fixing the order this way is +not required by the specification. It makes every response byte-for-byte +reproducible for testing and logging. That reproducibility matters more +here than flexibility no caller needs. + +`reason(_:)` maps a status code to its human-readable reason phrase, for +example `404` to `"Not Found"`, for the small fixed set of statuses this +library emits. It falls back to `"OK"` for anything else. The reason phrase +is advisory text. HTTP clients are not required to interpret it. ### SSEStream `SSEStream` frames the Server-Sent-Events wire format for one open -connection, identified by its socket descriptor `fd`. Unlike `HTTPRequest` -and `HTTPResponse`, which each cover one request-response exchange, an -`SSEStream` value is meant to be held by the caller for the life of a -long-running push connection; the library plays no role in deciding how long -that is, what triggers each message, or when to close it. - -`writeHead()` writes the fixed SSE response head — the status line and the -three headers (`Content-Type: text/event-stream`, `Cache-Control: no-cache`, -`Connection: keep-alive`) a client needs to recognize an event stream — and -must be called exactly once, before any data frame. It returns `false` if the -write failed, which the documentation notes means the peer is already gone; -a caller seeing `false` should close the connection rather than attempt to -send frames into a stream nobody is reading. `send(_:)` writes one SSE -`data:` frame carrying the given payload string, in the wire format -`"data: \n\n"` that the SSE specification requires (the payload -text followed by exactly one blank line). It also returns `false` on a failed -write, for the same reason. Both callers of this type in MOOTx01 — moot-mgr's -dashboard, which drives the stream from polling its own store, and the -resident MCP transport, which drives it from JSON-RPC notifications — decide -their own cadence and lifetime entirely outside this file; `SSEStream` only -guarantees that whatever they send is framed correctly on the wire. +connection, identified by its socket descriptor `fd`. `HTTPRequest` and +`HTTPResponse` each cover one request-response exchange. An `SSEStream` +value works differently. The caller holds it for the life of a +long-running push connection. The library plays no role in deciding how +long that is, what triggers each message, or when to close it. + +`writeHead()` writes the fixed SSE response head. That head is the status +line and three headers a client needs to recognize an event stream: +`Content-Type: text/event-stream`, `Cache-Control: no-cache`, and +`Connection: keep-alive`. It must be called exactly once, before any data +frame. It returns `false` if the write failed. The documentation notes that +a failed write means the peer is already gone. A caller seeing `false` +should close the connection rather than attempt to send frames into a +stream nobody is reading. + +`send(_:)` writes one SSE `data:` frame carrying the given payload string. +The wire format is `"data: \n\n"`, the payload text followed by +exactly one blank line, as the SSE specification requires. It also returns +`false` on a failed write, for the same reason. + +Both callers of this type in MOOTx01 decide their own cadence and lifetime +entirely outside this file. The moot-mgr dashboard drives the stream from +polling its own store. The resident MCP transport drives it from JSON-RPC +notifications. `SSEStream` only guarantees that whatever they send is +framed correctly on the wire. diff --git a/packages/libs/LoopbackHTTP/docs/OVERVIEW.md b/packages/libs/LoopbackHTTP/docs/OVERVIEW.md index 16ed926..c051eb2 100644 --- a/packages/libs/LoopbackHTTP/docs/OVERVIEW.md +++ b/packages/libs/LoopbackHTTP/docs/OVERVIEW.md @@ -15,96 +15,107 @@ sources: ## What This Library Does -LoopbackHTTP is a small HTTP/1.1 server primitive. It listens on a socket, reads -one request, and lets the caller write one response. It also knows how to -frame a Server-Sent-Events (SSE) stream, a way for a server to push a -continuing series of small text messages to a browser over one open -connection. The library does not decide what any of that traffic means; it -only moves the bytes correctly. +LoopbackHTTP is a small HTTP/1.1 server primitive. It listens on a socket. It +reads one request. It lets the caller write one response. + +The library also frames a Server-Sent-Events (SSE) stream. SSE is a way for a +server to push a series of small text messages to a browser. The messages +travel over one open connection. The library does not decide what the +traffic means. It only moves the bytes correctly. The library binds strictly to the loopback address, `127.0.0.1`. A loopback -address is one that only the same machine can reach; no other computer on the +address is one that only the same machine can reach. No other computer on the network can connect to it. Every listener this library creates is reachable -only from processes running on the same device. +only from processes on the same device. ## The Problem It Solves -Two programs inside MOOTx01 need a local HTTP server: the moot-mgr monitor -daemon, which serves a small dashboard, and the resident mootx01 MCP daemon, -which serves the Model Context Protocol used by AI tool integrations. Before -this library existed, each program hand-rolled its own socket and parsing -code. Two hand-rolled copies drift apart over time: one gets a security fix, -the other does not; one handles a header correctly, the other does not. -LoopbackHTTP was extracted from moot-mgr, under decision record -ADR-LOOPBACKHTTP-001, so both programs share one audited implementation +Two programs inside MOOTx01 need a local HTTP server. The moot-mgr monitor +daemon serves a small dashboard. The resident mootx01 MCP daemon serves the +Model Context Protocol used by AI tool integrations. + +Before this library existed, each program hand-rolled its own socket and +parsing code. Two hand-rolled copies drift apart over time. One gets a +security fix and the other does not. One handles a header correctly and the +other does not. + +LoopbackHTTP was extracted from moot-mgr under decision record +ADR-LOOPBACKHTTP-001. Both programs now share one audited implementation instead of maintaining two. The library adds no external package dependency. It wraps only the system C -socket API supplied by the operating system (`libc` on Linux, `Darwin` on -Apple platforms). A larger, general-purpose HTTP package such as SwiftNIO was -available but was rejected: it would pull in a dependency graph broader than -this narrow need justifies, and the moot-system kit rules ask each library to -justify every dependency it takes. Even Apple's own `Network.framework` was -tried and rejected for a concrete reason: its listener type refused to bind a -listening server socket in this project's command-line build environment, -failing with a low-level error code on every configuration tested. Plain -POSIX sockets bind correctly in that same environment, so the library builds -on them directly. +socket API supplied by the operating system: `libc` on Linux and `Darwin` on +Apple platforms. A larger, general-purpose HTTP package such as SwiftNIO was +available. It was rejected because it would pull in a dependency graph +broader than this narrow need justifies. The moot-system kit rules ask each +library to justify every dependency it takes. + +Even Apple's own `Network.framework` was tried. It was rejected for a +concrete reason. Its listener type refused to bind a listening server socket +in this project's command-line build environment. It failed with a +low-level error code on every configuration tested. Plain POSIX sockets bind +correctly in that same environment. The library builds on them directly. The library is deliberately authentication-free. It exposes two convenience readers on a parsed request, `bearerToken` and `origin`, so a caller can read -those header values easily. But the library itself never decides whether a -request is authorized. That decision is composed above the transport, one -layer up, by each consumer: the Community Edition of moot-mgr enforces no -policy at all, the full moot-mgr enforces a bearer token plus an Origin -header check, and an Enterprise Edition-only remote layer enforces OAuth -2.1. Keeping the transport policy-free lets the identical compiled library -ship inside every edition unchanged; only the layer above it differs. +those header values easily. The library itself never decides whether a +request is authorized. That decision sits one layer above the transport, in +each consumer. The Community Edition of moot-mgr enforces no policy at all. +The full moot-mgr enforces a bearer token plus an Origin header check. An +Enterprise Edition-only remote layer enforces OAuth 2.1. Keeping the +transport policy-free lets the identical compiled library ship inside every +edition unchanged. Only the layer above it differs. The library is Swift-only. MOOTx01 keeps a byte-for-byte parity discipline between a Swift implementation and a Rust implementation for deterministic -computation, such as classification or fingerprinting, where both legs must -produce the exact same answer from the exact same input. Binding a socket and -parsing HTTP headers is operating-system transport glue, not that kind of -deterministic computation, so it falls outside the parity rule. The Rust side -of MOOTx01's MCP transport (ARIA_MCP-rust) hand-rolls its own separate -networking code under a rule that forbids it from calling into Swift or C -libraries across a foreign-function boundary; the two sides only have to -agree at the JSON-RPC message level, not at the socket level. +computation. Classification and fingerprinting are two examples. Both legs +must produce the exact same answer from the exact same input. + +Binding a socket and parsing HTTP headers is operating-system transport +glue. It is not that kind of deterministic computation, so it falls outside +the parity rule. The Rust side of MOOTx01's MCP transport, ARIA_MCP-rust, +hand-rolls its own separate networking code. A project rule forbids that +code from calling into Swift or C libraries across a foreign-function +boundary. The two sides only have to agree at the JSON-RPC message level, +not at the socket level. ## How It Works The library is organized in two layers, one file each. -`POSIXSocket.swift` is the transport layer. It opens a listening socket bound -to `127.0.0.1` on a chosen TCP port, or opens a Unix-domain socket at a -filesystem path locked to owner-only access. It accepts one incoming -connection at a time and reads or writes raw bytes on the resulting -connection. - -`HTTPWire.swift` is the protocol layer, built on top. `HTTPRequest.read(fd:)` -reads raw bytes from a connected socket and parses them into a structured -request: a method, a path, a query string, a table of headers, and a body. -`HTTPResponse` is a value the caller builds — a status code, headers, and a -body — which the library then serializes into correct HTTP/1.1 wire bytes and -writes back to the socket. `SSEStream` frames the narrower SSE wire format: -one head line written once, then any number of `data:` frames written over -time on a connection the caller keeps open. - -A typical request passes through the library in one direction only: bytes -arrive from `POSIXSocket`, `HTTPRequest.read` turns them into a request value, -the caller inspects that value and decides what to do, and the caller's +`POSIXSocket.swift` is the transport layer. It opens a listening socket +bound to `127.0.0.1` on a chosen TCP port. It can also open a Unix-domain +socket at a filesystem path locked to owner-only access. It accepts one +incoming connection at a time. It reads or writes raw bytes on the +resulting connection. + +`HTTPWire.swift` is the protocol layer, built on top of the transport. +`HTTPRequest.read(fd:)` reads raw bytes from a connected socket. It parses +them into a structured request: a method, a path, a query string, a table +of headers, and a body. `HTTPResponse` is a value the caller builds: a +status code, headers, and a body. The library serializes that value into +correct HTTP/1.1 wire bytes and writes it back to the socket. `SSEStream` +frames the narrower SSE wire format. It writes one head line once, then any +number of `data:` frames over time, on a connection the caller keeps open. + +A typical request passes through the library in one direction only. Bytes +arrive from `POSIXSocket`. `HTTPRequest.read` turns them into a request +value. The caller inspects that value and decides what to do. The caller's resulting `HTTPResponse` or `SSEStream` frames go back out through -`POSIXSocket`. Nothing loops back, and no component here holds state between -one request and the next connection. That simple, one-way shape is why this -library needs no larger topology diagram: two files, a handful of value -types, and a straight line from incoming bytes to outgoing bytes. +`POSIXSocket`. Nothing loops back. No component here holds state between one +request and the next connection. + +That simple, one-way shape is why this library needs no larger topology +diagram. It is two files, a handful of value types, and a straight line from +incoming bytes to outgoing bytes. ## What Ships in the Package -The package ships two Swift source files and no bundled resources, artifacts, -or Rust port. `Package.swift` declares one library product, `LoopbackHTTP`, -with no external dependencies, targeting macOS 26 and iOS 26 while also -building on Linux Swift through a compile-time import guard in -`POSIXSocket.swift` that selects `Glibc` there and `Darwin` on Apple +The package ships two Swift source files and no bundled resources, +artifacts, or Rust port. `Package.swift` declares one library product, +`LoopbackHTTP`, with no external dependencies. It targets macOS 26 and iOS +26. + +The package also builds on Linux Swift. `POSIXSocket.swift` uses a +compile-time import guard that selects `Glibc` there and `Darwin` on Apple platforms. diff --git a/packages/libs/ObserverSink/docs/AGENT_MAP.md b/packages/libs/ObserverSink/docs/AGENT_MAP.md index 36daea4..dd6baa8 100644 --- a/packages/libs/ObserverSink/docs/AGENT_MAP.md +++ b/packages/libs/ObserverSink/docs/AGENT_MAP.md @@ -11,77 +11,77 @@ sources: blob: 483c736feffd82821a8d77030afde0e182d320fc --- -# AGENT_MAP — ObserverSink +# AGENT_MAP: ObserverSink PURPOSE: reusable PersistenceKit-backed telemetry sink for the MOOTx01 manager pipeline (Manager 1.0, Phase 0.5). IntellectusLib.report(_:) → PersistenceStatsSink.receive(_:) → flag-gated SQLite write into StatsStore (metric_samples / event_samples). Also owns topology_snapshots (governor write, dashboard read) and store-level retention. -DEPS: imports IntellectusLib (StatsSink protocol, StatSample/EventKind datum), PersistenceKit + PersistenceKitSQLite (Storage/SchemaDeclaration/RowStore, SQLiteStorage backend), Foundation, OSLog. Dependency hierarchy (no inversion, per MANAGER_1.0_PLAN.md §4): IntellectusLib (floor) → PersistenceKit (kit) → ObserverSink (this lib). Imported by: none within the moot-system repo at this commit — Package.swift comments name the intended consumers (Intellectus install site in AriaResident/aria-mcp, moot-mgr manager) as living outside this repo. Rust port in rust/ (crate `observer-sink`) mirrors schema + flag semantics exactly; conformance tests in Tests/ObserverSinkTests/ObserverSinkConformanceTests.swift and rust/tests/conformance.rs gate parity (16 scenarios). +DEPS: imports IntellectusLib (StatsSink protocol, StatSample/EventKind datum), PersistenceKit + PersistenceKitSQLite (Storage/SchemaDeclaration/RowStore, SQLiteStorage backend), Foundation, OSLog. Dependency hierarchy (no inversion, per MANAGER_1.0_PLAN.md §4): IntellectusLib (floor) → PersistenceKit (kit) → ObserverSink (this lib). Imported by: none within the moot-system repo at this commit: Package.swift comments name the intended consumers (Intellectus install site in AriaResident/aria-mcp, moot-mgr manager) as living outside this repo. Rust port in rust/ (crate `observer-sink`) mirrors schema + flag semantics exactly; conformance tests in Tests/ObserverSinkTests/ObserverSinkConformanceTests.swift and rust/tests/conformance.rs gate parity (16 scenarios). ENTRY POINTS (most callers need only these): -- PersistenceStatsSink.swift:102 `PersistenceStatsSink.init(store:dropboxID:)` — construct the sink; store must already be `.open()`ed -- PersistenceStatsSink.swift:123 `receive(_ sample: StatSample)` — StatsSink conformance; called by Intellectus.report(_:), never call directly -- StatsStore.swift:362 `StatsStore.init(url:) throws` — construct the store -- StatsStore.swift:384 `open() async throws` — apply schema/migrations + seed control rows (idempotent) +- PersistenceStatsSink.swift:102 `PersistenceStatsSink.init(store:dropboxID:)`: construct the sink; store must already be `.open()`ed +- PersistenceStatsSink.swift:123 `receive(_ sample: StatSample)`: StatsSink conformance; called by Intellectus.report(_:), never call directly +- StatsStore.swift:362 `StatsStore.init(url:) throws`: construct the store +- StatsStore.swift:384 `open() async throws`: apply schema/migrations + seed control rows (idempotent) ## Symbol Table -### Sink — PersistenceStatsSink.swift -- :77 `struct PersistenceStatsSink: StatsSink` — Sendable; all mutable state lives in StatsStore, not here +### Sink: PersistenceStatsSink.swift +- :77 `struct PersistenceStatsSink: StatsSink`: Sendable; all mutable state lives in StatsStore, not here - :82 `store: StatsStore` (private) -- :89 `dropboxID: String` (private) — tags every written row; identifies source consumer/process -- :91 `logger` (private) — `Logger(subsystem: "com.mootx01.kit", category: "ObserverSink")` -- :102 `init(store:dropboxID:)` — see ENTRY POINTS -- :123 `receive(_:)` — synchronous, non-blocking; dispatches unstructured Task; inside: isMonitoringEnabled() gate → insertMetric/insertEvent by StatSample case → catch-and-log, never throws/propagates +- :89 `dropboxID: String` (private): tags every written row; identifies source consumer/process +- :91 `logger` (private): `Logger(subsystem: "com.mootx01.kit", category: "ObserverSink")` +- :102 `init(store:dropboxID:)`: see ENTRY POINTS +- :123 `receive(_:)`: synchronous, non-blocking; dispatches unstructured Task; inside: isMonitoringEnabled() gate → insertMetric/insertEvent by StatSample case → catch-and-log, never throws/propagates -### Schema constants — StatsStore.swift `enum StatsStoreSchema` -- :49 `enum StatsStoreSchema` — table/column name constants; typo → compile error, not silent empty query +### Schema constants: StatsStore.swift `enum StatsStoreSchema` +- :49 `enum StatsStoreSchema`: table/column name constants; typo → compile error, not silent empty query - :54 `metricSamplesTable = "metric_samples"` / :57 `eventSamplesTable = "event_samples"` / :60 `controlTable = "control"` / :117 `topologySnapshotsTable = "topology_snapshots"` - :65 `tsColumn = "ts"` (TEXT ISO-8601 on every table) / :68 `dropboxIDColumn = "dropbox_id"` -- :73 `nameColumn`, :76 `valueColumn` (REAL), :79 `tagsColumn` (TEXT JSON) — metric_samples -- :84 `kindColumn` (EventKind raw string), :87 `nounTypeColumn` (INTEGER ordinal), :91 `rowIDColumn = "estate_row_id"` (NOT the synthetic PK), :94 `estateColumn` — event_samples -- :99 `keyColumn` (PK), :102 `controlValueColumn` — control +- :73 `nameColumn`, :76 `valueColumn` (REAL), :79 `tagsColumn` (TEXT JSON): metric_samples +- :84 `kindColumn` (EventKind raw string), :87 `nounTypeColumn` (INTEGER ordinal), :91 `rowIDColumn = "estate_row_id"` (NOT the synthetic PK), :94 `estateColumn`: event_samples +- :99 `keyColumn` (PK), :102 `controlValueColumn`: control - :107 `monitoringKey = "monitoring"` (value "1"/"0") / :112 `retentionCutoffKey = "retention_cutoff"` (ISO-8601, default epoch-zero) -- :120 `generatedAtColumn`, :123 `payloadColumn`, :132 `topologyFingerprintColumn` (nullable, v3) — topology_snapshots +- :120 `generatedAtColumn`, :123 `payloadColumn`, :132 `topologyFingerprintColumn` (nullable, v3): topology_snapshots -### Store — StatsStore.swift `final class StatsStore: Sendable` -- :160 `final class StatsStore: Sendable` — wraps SQLiteStorage +### Store: StatsStore.swift `final class StatsStore: Sendable` +- :160 `final class StatsStore: Sendable`: wraps SQLiteStorage - :164 `storage: SQLiteStorage` (private) / :165 `logger` (private) -- :175 `static let schemaVersion = 3` — v1: metric/event/control; v2: +topology_snapshots; v3: +topology_fingerprint (nullable) -- :193 `static let schema: SchemaDeclaration` — 4 tables, 2 indices (ts on metric_samples/event_samples), 2 migrations (v1→v2 createTable, v2→v3 addColumn) -- :317 migrations block — v1→v2 :321, v2→v3 :340; SQLite backend creates fresh DBs at latest version directly (migrations replay only matters for InMemory/upgrade path) -- :362 `init(url:) throws` — fresh random estateID per StatsStore instance; no I/O -- :384 `open() async throws` — applies schema, then :396/:398 `seedControlIfAbsent` for monitoring="0" and retention_cutoff=epoch-zero; SEED-IF-ABSENT not upsert -- :412 `seedControlIfAbsent(key:value:)` (private) — existence check then insert; preserves operator-set values across reopen -- :433 `close() async` — idempotent -- :448 `isMonitoringEnabled() async throws -> Bool` — control["monitoring"]=="1"; missing row → false (safe default) -- :470 `setMonitoringEnabled(_:) async throws` — upsert on keyColumn; manager-only caller by convention (sink never writes this) -- :496 `insertMetric(name:value:tags:ts:dropboxID:) async throws` — assigns row_id UUID; tags→JSON via encodeTagsJSON; ts(Double epoch)→.timestamp(Date) at this boundary -- :532 `insertEvent(kind:nounType:rowID:estate:ts:dropboxID:) async throws` — rowID stored in estate_row_id column, NOT row_id (that's the synthetic PK) -- :576 `writeTopologySnapshot(estate:generatedAt:payload:fingerprint:) async throws` — upsert keyed on estate (latest-wins, no history); throws StorageError.invalidQuery if payload isn't valid UTF-8; fingerprint nil → .null column -- :615 `latestTopologySnapshot(estate:) async throws -> Data?` — estate=nil → newest generated_at across ALL estates (dashboard "all" view); uses :679 generatedAtInstant to tolerate both .timestamp and .text read-back reps (SQLite vs InMemory) — DO NOT match only .timestamp, that was a real bug (all rows tie at .distantPast under SQLite) -- :655 `loadTopologyFingerprint(estate:) async throws -> String?` — nil if no snapshot / pre-v3 row / written without fingerprint -- :679 `generatedAtInstant(_:) -> Date` (private) — see above -- :696 `queryMetrics(dropboxID:) async throws -> [MetricRow]` — full-table (optionally dropbox-filtered) read, ts ascending -- :725 `queryMetricsByNames(_:dropboxID:) async throws -> [MetricRow]` — `WHERE name IN (...)`; empty names → [] immediately; USE THIS not queryMetrics+filter on hot read paths -- :769 `countMetrics() async throws -> Int` — COUNT(*), no row decode -- :784 `queryEvents(dropboxID:) async throws -> [EventRow]` — full-table (optionally dropbox-filtered) read, ts ascending -- :818 `deleteMetricsBefore(cutoff:now:) async throws -> Int` — ts < cutoff; updates retention_cutoff control row to `cutoff` (not `now`) -- :842 `deleteEventsBefore(cutoff:now:) async throws -> Int` — same semantics, event_samples -- :873 `storageStats(now:) async throws -> StorageStats?` — store's OWN SQLite backend health (WAL frames, file size, page/freelist counts), not an observed estate's storage; via StorageIntrospection, SQLite backend always returns non-nil -- :883 `recordRetentionCutoff(_:now:) async throws` (private) — upserts retention_cutoff as ISO-8601 -- :907 `static let iso8601Formatter: DateFormatter` — "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'", UTC, en_US_POSIX; matches SQLiteStorage's timestamp codec -- :924 `encodeTagsJSON(_:) -> String` (private) — `.sortedKeys` for determinism; empty/unencodable → "{}" -- :939 `static func decodeTagsJSON(_:) -> [String: String]` — parse failure → [:] (forward-compatible) +- :175 `static let schemaVersion = 3`: v1: metric/event/control; v2: +topology_snapshots; v3: +topology_fingerprint (nullable) +- :193 `static let schema: SchemaDeclaration`: 4 tables, 2 indices (ts on metric_samples/event_samples), 2 migrations (v1→v2 createTable, v2→v3 addColumn) +- :317 migrations block: v1→v2 :321, v2→v3 :340; SQLite backend creates fresh DBs at latest version directly (migrations replay only matters for InMemory/upgrade path) +- :362 `init(url:) throws`: fresh random estateID per StatsStore instance; no I/O +- :384 `open() async throws`: applies schema, then :396/:398 `seedControlIfAbsent` for monitoring="0" and retention_cutoff=epoch-zero; SEED-IF-ABSENT not upsert +- :412 `seedControlIfAbsent(key:value:)` (private): existence check then insert; preserves operator-set values across reopen +- :433 `close() async`: idempotent +- :448 `isMonitoringEnabled() async throws -> Bool`: control["monitoring"]=="1"; missing row → false (safe default) +- :470 `setMonitoringEnabled(_:) async throws`: upsert on keyColumn; manager-only caller by convention (sink never writes this) +- :496 `insertMetric(name:value:tags:ts:dropboxID:) async throws`: assigns row_id UUID; tags→JSON via encodeTagsJSON; ts(Double epoch)→.timestamp(Date) at this boundary +- :532 `insertEvent(kind:nounType:rowID:estate:ts:dropboxID:) async throws`: rowID stored in estate_row_id column, NOT row_id (that's the synthetic PK) +- :576 `writeTopologySnapshot(estate:generatedAt:payload:fingerprint:) async throws`: upsert keyed on estate (latest-wins, no history); throws StorageError.invalidQuery if payload isn't valid UTF-8; fingerprint nil → .null column +- :615 `latestTopologySnapshot(estate:) async throws -> Data?`: estate=nil → newest generated_at across ALL estates (dashboard "all" view); uses :679 generatedAtInstant to tolerate both .timestamp and .text read-back reps (SQLite vs InMemory): DO NOT match only .timestamp, that was a real bug (all rows tie at .distantPast under SQLite) +- :655 `loadTopologyFingerprint(estate:) async throws -> String?`: nil if no snapshot / pre-v3 row / written without fingerprint +- :679 `generatedAtInstant(_:) -> Date` (private): see above +- :696 `queryMetrics(dropboxID:) async throws -> [MetricRow]`: full-table (optionally dropbox-filtered) read, ts ascending +- :725 `queryMetricsByNames(_:dropboxID:) async throws -> [MetricRow]`: `WHERE name IN (...)`; empty names → [] immediately; USE THIS not queryMetrics+filter on hot read paths +- :769 `countMetrics() async throws -> Int`: COUNT(*), no row decode +- :784 `queryEvents(dropboxID:) async throws -> [EventRow]`: full-table (optionally dropbox-filtered) read, ts ascending +- :818 `deleteMetricsBefore(cutoff:now:) async throws -> Int`: ts < cutoff; updates retention_cutoff control row to `cutoff` (not `now`) +- :842 `deleteEventsBefore(cutoff:now:) async throws -> Int`: same semantics, event_samples +- :873 `storageStats(now:) async throws -> StorageStats?`: store's OWN SQLite backend health (WAL frames, file size, page/freelist counts), not an observed estate's storage; via StorageIntrospection, SQLite backend always returns non-nil +- :883 `recordRetentionCutoff(_:now:) async throws` (private): upserts retention_cutoff as ISO-8601 +- :907 `static let iso8601Formatter: DateFormatter`: "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'", UTC, en_US_POSIX; matches SQLiteStorage's timestamp codec +- :924 `encodeTagsJSON(_:) -> String` (private): `.sortedKeys` for determinism; empty/unencodable → "{}" +- :939 `static func decodeTagsJSON(_:) -> [String: String]`: parse failure → [:] (forward-compatible) ### Row types -- :951 `struct MetricRow: Sendable` — rowID/name/value/tags/ts/dropboxID; :966 `init?(storageRow:)` returns nil on any column type mismatch (compactMap-friendly) -- :986 `struct EventRow: Sendable` — rowID/kind/nounType/rowIDStr/estate/ts/dropboxID; :1003 `init?(storageRow:)` same nil-on-mismatch contract +- :951 `struct MetricRow: Sendable`: rowID/name/value/tags/ts/dropboxID; :966 `init?(storageRow:)` returns nil on any column type mismatch (compactMap-friendly) +- :986 `struct EventRow: Sendable`: rowID/kind/nounType/rowIDStr/estate/ts/dropboxID; :1003 `init?(storageRow:)` same nil-on-mismatch contract -### Rust port (rust/, crate observer_sink) — mirrors above 1:1 except where Rust semantics differ -- rust/src/sink.rs `PersistenceStatsSink::receive` — INLINE synchronous I/O (no Task/thread spawn); Rust StatsSink trait is sync and SqliteStorage is Mutex-backed -- rust/src/store.rs `StatsStore::SCHEMA_VERSION = 3`, `make_schema()` — line-by-line mirror of Swift `StatsStore.schema` -- rust/src/store.rs `epoch_to_iso8601` / `iso8601_to_epoch` (crate-internal) — explicit ISO-8601 codec (no DateFormatter equivalent); clamps NaN/±inf/out-of-range to year 0001–9999 window, never panics (unit-tested at bottom of store.rs) -- rust/src/store.rs `write_topology_snapshot_bytes` — Rust-only convenience: lossy UTF-8 conversion of raw bytes; Swift has no equivalent because Foundation's `String(data:encoding:)` guard is a runtime check callers already handle inline +### Rust port (rust/, crate observer_sink): mirrors above 1:1 except where Rust semantics differ +- rust/src/sink.rs `PersistenceStatsSink::receive`: INLINE synchronous I/O (no Task/thread spawn); Rust StatsSink trait is sync and SqliteStorage is Mutex-backed +- rust/src/store.rs `StatsStore::SCHEMA_VERSION = 3`, `make_schema()`: line-by-line mirror of Swift `StatsStore.schema` +- rust/src/store.rs `epoch_to_iso8601` / `iso8601_to_epoch` (crate-internal): explicit ISO-8601 codec (no DateFormatter equivalent); clamps NaN/±inf/out-of-range to year 0001–9999 window, never panics (unit-tested at bottom of store.rs) +- rust/src/store.rs `write_topology_snapshot_bytes`: Rust-only convenience: lossy UTF-8 conversion of raw bytes; Swift has no equivalent because Foundation's `String(data:encoding:)` guard is a runtime check callers already handle inline ## INVARIANTS / GOTCHAS @@ -89,16 +89,16 @@ ENTRY POINTS (most callers need only these): - `isMonitoringEnabled()` fails safe: missing row → `false` (off), never assume on. - `PersistenceStatsSink` only READS the monitoring flag; only the manager process calls `setMonitoringEnabled`. Do not add a write path to the sink. - `receive(_:)` MUST stay synchronous and non-blocking (StatsSink protocol contract). All I/O happens off the calling thread/task (Swift: unstructured Task; Rust: inline sync, which is fine because Rust's I/O is not on an async hot path here). -- No in-process buffering at v1.0 by design — one Task per sample in Swift. Do not add batching without also updating OVERVIEW/DETAILS and the design comment in PersistenceStatsSink.swift. -- `event_samples.estate_row_id` is NOT the table's primary key — `row_id` (synthetic UUID) is the PK; `estate_row_id` carries the estate's own UUID for that row. Never conflate the two when reading/writing. -- Retention (`deleteMetricsBefore`/`deleteEventsBefore`) takes `cutoff` AND `now` as caller-supplied parameters. NO `Date()` / `SystemTime::now()` call inside either engine — determinism contract, both ports. -- `recordRetentionCutoff` stores `cutoff` (the boundary applied), not `now` (when the pass ran) — intentional, dashboard-facing choice. +- No in-process buffering at v1.0 by design: one Task per sample in Swift. Do not add batching without also updating OVERVIEW/DETAILS and the design comment in PersistenceStatsSink.swift. +- `event_samples.estate_row_id` is NOT the table's primary key: `row_id` (synthetic UUID) is the PK; `estate_row_id` carries the estate's own UUID for that row. Never conflate the two when reading/writing. +- Retention (`deleteMetricsBefore`/`deleteEventsBefore`) takes `cutoff` AND `now` as caller-supplied parameters. NO `Date()` / `SystemTime::now()` call inside either engine: determinism contract, both ports. +- `recordRetentionCutoff` stores `cutoff` (the boundary applied), not `now` (when the pass ran): intentional, dashboard-facing choice. - `latestTopologySnapshot`/`latest_topology_snapshot` newest-across-estates comparison MUST tolerate both `.timestamp` and `.text` read-back representations for `generated_at` (SQLite returns text, InMemory returns timestamp). Matching only one representation reintroduces a real, previously-shipped bug where SQLite rows all tie at `.distantPast` / `i64::MIN` and an arbitrary row wins. -- `topology_snapshots` is latest-wins per estate (PRIMARY KEY = estate) — no history retained. This is intentional; do not add versioning without a schema bump and a doc update. -- `topology_fingerprint` column is nullable (v3, additive). Absent/null means "no fingerprint computed yet," not an error — callers (autonomic governor) treat null as "recompute once." +- `topology_snapshots` is latest-wins per estate (PRIMARY KEY = estate): no history retained. This is intentional; do not add versioning without a schema bump and a doc update. +- `topology_fingerprint` column is nullable (v3, additive). Absent/null means "no fingerprint computed yet," not an error: callers (autonomic governor) treat null as "recompute once." - Timestamps are TEXT (ISO-8601 UTC, millisecond precision) on every table. No REAL/numeric timestamp column exists anywhere in this schema. Swift boundary: `Date(timeIntervalSince1970: ts)` at insert, `iso8601Formatter` for retention-cutoff string formatting. Rust boundary: `epoch_to_iso8601`/`iso8601_to_epoch`. -- Rust's `epoch_to_iso8601` clamps to the year 0001–9999 window and never panics on NaN/±infinity/out-of-range f64 — required because a malformed 5-digit or negative year would break every lexicographic TEXT comparison retention and topology-snapshot ordering rely on. Six unit tests in rust/src/store.rs lock this behavior; do not remove them when refactoring. -- Tag maps: encoded with `.sortedKeys` (Swift) / serialized from a `BTreeMap` (Rust) for deterministic JSON — needed for reproducible test fixtures, not for correctness of the round trip itself. Decode failure → empty map in both ports (forward-compatible, never throws). -- `queryMetricsByNames` is the required hot-path read API — issues `WHERE name IN (...)`; do not reintroduce `queryMetrics` + Swift-side filtering on a read path that matters for latency. -- No pinned data artifacts in this package (unlike LatticeLib-style libs) — every table starts empty; there is nothing to version-check against a build-time asset. +- Rust's `epoch_to_iso8601` clamps to the year 0001–9999 window and never panics on NaN/±infinity/out-of-range f64: required because a malformed 5-digit or negative year would break every lexicographic TEXT comparison retention and topology-snapshot ordering rely on. Six unit tests in rust/src/store.rs lock this behavior; do not remove them when refactoring. +- Tag maps: encoded with `.sortedKeys` (Swift) / serialized from a `BTreeMap` (Rust) for deterministic JSON: needed for reproducible test fixtures, not for correctness of the round trip itself. Decode failure → empty map in both ports (forward-compatible, never throws). +- `queryMetricsByNames` is the required hot-path read API: issues `WHERE name IN (...)`; do not reintroduce `queryMetrics` + Swift-side filtering on a read path that matters for latency. +- No pinned data artifacts in this package (unlike LatticeLib-style libs): every table starts empty; there is nothing to version-check against a build-time asset. - Both-ports parity is enforced ONLY by the two conformance test suites (Tests/ObserverSinkTests + rust/tests/conformance.rs), not by any shared fixture file. Changing schema, flag semantics, or timestamp encoding in one leg requires updating the other leg AND both suites by hand. diff --git a/packages/libs/ObserverSink/docs/DETAILS.md b/packages/libs/ObserverSink/docs/DETAILS.md index 47c7409..93d13a9 100644 --- a/packages/libs/ObserverSink/docs/DETAILS.md +++ b/packages/libs/ObserverSink/docs/DETAILS.md @@ -15,327 +15,339 @@ sources: This document walks through both source files in the package. Read `OVERVIEW.md` first for the big picture. `PersistenceStatsSink.swift` -comes first because it is the entry point every caller uses; -`StatsStore.swift` follows because it is what the sink calls into. +comes first, because it is the entry point every caller uses. +`StatsStore.swift` follows, because it is what the sink calls into. ## PersistenceStatsSink.swift -This file provides `PersistenceStatsSink`, the concrete implementation of -IntellectusLib's `StatsSink` protocol. It is the only type in the package -that a typical caller touches directly. +This file provides `PersistenceStatsSink`. It is the concrete +implementation of IntellectusLib's `StatsSink` protocol, and the only +type in the package a typical caller touches directly. A `StatsSink` is any type that can receive one `StatSample` at a time. -IntellectusLib defines the protocol and a global installation point; -`PersistenceStatsSink` is the implementation that actually saves samples -by writing them into a `StatsStore`. A host program constructs one sink, -installs it with `Intellectus.install(sink:)`, and turns monitoring on -with `Intellectus.setEnabled(true)`. From that point on, every call to -`Intellectus.report(_:)` anywhere in the process reaches this sink's -`receive(_:)` method. - -The struct holds three pieces of state: the `store` it writes to, a -`dropboxID` string identifying this consumer, and a `Logger` for -diagnostics. All three are `let` constants, and the struct conforms to -`Sendable` — required because `StatsSink` values are held in a -process-wide global and called from any thread. No mutable state lives in -the sink itself; anything that changes over time (the monitoring flag, the -stored rows) lives inside `StatsStore`, which manages its own -concurrency. +IntellectusLib defines the protocol and a global installation point. +`PersistenceStatsSink` is the implementation that actually saves +samples, by writing them into a `StatsStore`. A host program constructs +one sink. It installs the sink with `Intellectus.install(sink:)`. It then +turns monitoring on with `Intellectus.setEnabled(true)`. From that point +on, every call to `Intellectus.report(_:)` anywhere in the process +reaches this sink's `receive(_:)` method. + +The struct holds three pieces of state. It holds the `store` it writes +to, a `dropboxID` string that identifies this consumer, and a `Logger` +for diagnostics. All three are `let` constants. The struct conforms to +`Sendable`, a requirement because `StatsSink` values live in a +process-wide global and get called from any thread. No mutable state +lives in the sink itself. Anything that changes over time, such as the +monitoring flag or the stored rows, lives inside `StatsStore`. `StatsStore` +manages its own concurrency. `init(store:dropboxID:)` is a plain constructor. It does not open the -store — the caller must call `store.open()` first, because opening -involves asynchronous I/O that a synchronous initializer cannot perform. -This split between construction and opening matters when a host wants to +store. The caller must call `store.open()` first, because opening +involves async I/O that a synchronous initializer cannot perform. This +split between construction and opening matters when a host wants to retry a failed open without recreating the sink. -`receive(_:)` is the method that matters. It is declared as a synchronous, -non-blocking function because the `StatsSink` protocol documents that -requirement: whatever calls `Intellectus.report(_:)` must not stall -waiting for a database write. To honor that, `receive(_:)` captures the -store, the dropbox ID, and the logger — each `Sendable` — and immediately -starts an unstructured `Task`. The method then returns; the caller never -waits. - -Inside the `Task`, the real work happens in three steps. First, it calls -`store.isMonitoringEnabled()`. This is the flag-row check described in -`OVERVIEW.md`: a cheap read of one row in the `control` table. If -monitoring is off, the sample is discarded and a debug-level log line is -written — debug level specifically, because this check runs on every -single sample when monitoring is off, and a louder log level would flood -the console. Second, if monitoring is on, the code switches on the -`StatSample` case: a `.metric` sample calls `store.insertMetric(...)`, an -`.event` sample calls `store.insertEvent(...)`. Third, the whole block is -wrapped in a `do`/`catch`; any thrown error — from the flag read or from -either insert — is logged at `.error` level and never rethrown. A -telemetry sink that could crash the process it is monitoring would defeat -its own purpose, so every failure mode here ends in a log line, never a +`receive(_:)` is the method that matters. The `StatsSink` protocol +documents a requirement: whatever calls `Intellectus.report(_:)` must +never stall waiting for a database write. So `receive(_:)` is declared as +a synchronous, non-blocking function. To honor that requirement, +`receive(_:)` captures three values first. It captures the store, the +dropbox ID, and the logger. Each value is `Sendable`. The method then +starts a background task and returns. The caller never waits. + +Inside the task, the real work happens in three steps. First, the code +calls `store.isMonitoringEnabled()`. This is the flag-row check described +in `OVERVIEW.md`: a cheap read of one row in the `control` table. If +monitoring is off, the sample is discarded, and a debug-level log line is +written. The level is debug specifically, because this check runs on +every single sample when monitoring is off. A louder log level would +flood the console. Second, if monitoring is on, the code switches on the +`StatSample` case. A `.metric` sample calls `store.insertMetric(...)`. An +`.event` sample calls `store.insertEvent(...)`. Third, the whole block +sits inside a `do`/`catch`. Any thrown error, whether from the flag read +or from either insert, gets logged at `.error` level and never rethrown. +A telemetry sink that could crash the process it monitors would defeat +its own purpose. So every failure mode here ends in a log line, never a propagated error. The file's header comments record a deliberate scope limit for this -version: there is no in-process buffering. Each sample launches its own -`Task`, and SQLite's write-ahead log mode is relied on to make concurrent -writes from many such tasks safe and reasonably efficient. The comments -name a possible future improvement — batching many samples behind a timer -— and explain why it was not built yet: stats recording sits off the -hot substrate path, so an occasional dropped sample under heavy load is an -acceptable cost, and the simpler design is easier to reason about. +version. There is no in-process buffering. Each sample launches its own +task. SQLite's write-ahead log mode keeps concurrent writes from many +such tasks safe and reasonably efficient. The comments name a possible +future improvement: batching many samples behind a timer. They also +explain why that improvement was not built yet. Stats recording sits off +the hot substrate path. An occasional dropped sample under heavy load is +an acceptable cost there. The simpler design is also easier to reason +about. ## StatsStore.swift -This file provides `StatsStore`, the SQLite-backed component that owns the -telemetry schema, and `StatsStoreSchema`, a namespace of string constants -that name its tables and columns. - -### Schema constants — `StatsStoreSchema` - -Every table name and column name that any part of this package touches is -declared once, as a `public static let`, inside `StatsStoreSchema`. The -file's own comment explains the reason: gathering the strings in one place -turns a typo into a compile error instead of a query that silently matches -zero rows. `monitoringKey` and `retentionCutoffKey` additionally name the -two well-known rows inside the `control` table, since that table is a -generic key-value store rather than one column per concept. - -### Schema declaration — `StatsStore.schema` - -`StatsStore.schema` is a `SchemaDeclaration` value — a data structure from -PersistenceKit that describes tables, columns, indices, and migrations -without executing any SQL directly. `StatsStore.schemaVersion` (currently -`3`) is the version number PersistenceKit checks against this -declaration's migrations to decide whether an existing database needs to -be upgraded. - -The schema declares four tables. `metric_samples` and `event_samples` hold -the telemetry rows this library exists to record; both use a -`.uuid("row_id")` synthetic primary key that the store assigns, so callers -never have to generate or track one. `control` is the key-value table for -the monitoring flag and the retention-cutoff timestamp; its primary key is -the `key` column itself, which is what lets `setMonitoringEnabled` use an -upsert rather than a delete-then-insert. `topology_snapshots`, added in -schema version 2, holds one row per estate — its primary key is the -`estate` column, so writing a new snapshot for an estate automatically -replaces the old one instead of accumulating history. - -Two indices — `idx_metric_samples_ts` and `idx_event_samples_ts` — exist -purely to make retention fast. Retention deletes rows with `ts` below a -cutoff; without an index on `ts`, each retention pass would scan the +This file provides `StatsStore`, the SQLite-backed component that owns +the telemetry schema. It also provides `StatsStoreSchema`, a namespace of +string constants that name the tables and columns. + +### Schema Constants: `StatsStoreSchema` + +Every table name and column name that this package touches gets declared +once, as a `public static let`, inside `StatsStoreSchema`. The file's own +comment explains the reason. Gathering the strings in one place turns a +typo into a compile error, instead of a query that silently matches zero +rows. `monitoringKey` and `retentionCutoffKey` additionally name the two +well-known rows inside the `control` table. That table is a generic +key-value store, rather than one column per concept. + +### Schema Declaration: `StatsStore.schema`. + +`StatsStore.schema` is a `SchemaDeclaration` value. This value comes from +PersistenceKit. It describes tables, columns, indices, and migrations. It +never runs any SQL directly. `StatsStore.schemaVersion` is currently +three. PersistenceKit checks this number against the declaration's +migrations. This check decides whether an existing database needs an +upgrade. + +The schema declares four tables. `metric_samples` and `event_samples` +hold the telemetry rows this library exists to record. Both use a +`.uuid("row_id")` synthetic primary key that the store assigns, so +callers never have to generate or track one. `control` is the key-value +table for the monitoring flag and the retention-cutoff timestamp. Its +primary key is the `key` column itself. That design is what lets +`setMonitoringEnabled` use an upsert, rather than a delete followed by an +insert. `topology_snapshots`, added in schema version two, holds one row +per estate. Its primary key is the `estate` column. Writing a new +snapshot for an estate automatically replaces the old one. History never +accumulates. + +Two indices exist purely to make retention fast: `idx_metric_samples_ts` +and `idx_event_samples_ts`. Retention deletes rows with `ts` below a +cutoff. Without an index on `ts`, each retention pass would scan the entire table. -Two migrations are declared, one per schema version bump. -`fromVersion: 1, toVersion: 2` adds the `topology_snapshots` table; the -comment notes this is purely additive; no existing row is touched, and the -new table starts empty until the autonomic governor's next duty cycle -populates it. `fromVersion: 2, toVersion: 3` adds the nullable -`topology_fingerprint` column to that same table, again additively — any -row written before this version simply reads back with that column `nil`. +Two migrations are declared, one per schema version bump. The migration +from version one to version two adds the `topology_snapshots` table. The +comment notes that this change is purely additive. No existing row is +touched, and the new table starts empty until the autonomic governor's +next duty cycle fills it. The migration from version two to version three +adds the nullable `topology_fingerprint` column to that same table, again +additively. Any row written before this version simply reads back with +that column set to `nil`. -### Lifecycle — `init`, `open()`, `close()` +### Lifecycle: `init`, `open()`, `close()` `init(url:)` builds the underlying `SQLiteStorage` for a database file at -`url`, generating a fresh random estate identifier for the storage +`url`. It generates a fresh random estate identifier for the storage configuration. It performs no I/O beyond what opening a SQLite connection requires. -`open()` does two things: it asks PersistenceKit's storage layer to apply -`StatsStore.schema` (creating tables and running any pending migrations), -then it seeds two default rows in the `control` table — `"monitoring"` set -to `"0"` and `"retention_cutoff"` set to the epoch-zero ISO-8601 string. -Both seeds go through `seedControlIfAbsent(key:value:)`, a private helper -that checks whether a row with that key already exists before inserting. -This distinction — seed-if-absent rather than upsert — matters because the -monitoring flag is meant to be a persistent switch: if `open()` blindly -overwrote it every time, an operator's choice to turn monitoring on would -be silently reset to off on every process restart. Seeding only when -absent makes the first `open()` call install the defaults and every later -call on the same database a no-op for those two rows. - -`close()` closes the underlying storage. It is documented as idempotent — -safe to call more than once — which matters for callers that close a store -in both a normal shutdown path and a `defer` block. - -### Monitoring flag — `isMonitoringEnabled()`, `setMonitoringEnabled(_:)` +`open()` does two things. First, it asks PersistenceKit's storage layer +to apply `StatsStore.schema`, creating tables and running any pending +migrations. Second, it seeds two default rows in the `control` table. +`"monitoring"` gets set to `"0"`, and `"retention_cutoff"` gets set to the +epoch-zero ISO-8601 string. Both seeds go through +`seedControlIfAbsent(key:value:)`, a private helper that checks whether a +row with that key already exists before inserting. This seed-if-absent +approach, rather than an upsert, matters for one reason. The monitoring +flag is meant to be a persistent switch. Suppose `open()` blindly +overwrote it every time. An operator's choice to turn monitoring on +would then reset to off silently, on every process restart. Seeding only +when the row is absent avoids this. The first `open()` call installs the +defaults. Every later call on the same database becomes a no-op for those +two rows. + +`close()` closes the underlying storage. It is documented as idempotent, +meaning it is safe to call more than once. That property matters for +callers that close a store in both a normal shutdown path and a `defer` +block. + +### Monitoring Flag: `isMonitoringEnabled()`, `setMonitoringEnabled(_:)` `isMonitoringEnabled()` reads the single `control` row whose key is -`"monitoring"` and returns `true` only if its value is the string `"1"`. -If the row is missing entirely — for instance, if a caller queries a store -that was never opened — the function returns `false`. This fail-safe -default matters because a missing flag should never be interpreted as +`"monitoring"`. It returns `true` only if that row's value is the string +`"1"`. If the row is missing entirely, for instance if a caller queries a +store that was never opened, the function returns `false`. This +fail-safe default matters. A missing flag should never read as "recording is on." -`setMonitoringEnabled(_:)` writes `"1"` or `"0"` to that same row using an -upsert keyed on the `key` column, so it works whether or not the row +`setMonitoringEnabled(_:)` writes `"1"` or `"0"` to that same row, using +an upsert keyed on the `key` column. So it works whether or not the row already exists. The manager process is the only intended caller of this -method; `PersistenceStatsSink` only ever reads the flag, never sets it. -This asymmetry is what lets one manager control every consumer's recording -behavior without touching the consumers themselves. +method. `PersistenceStatsSink` only ever reads the flag; it never sets +it. This asymmetry is what lets one manager control every consumer's +recording behavior, without touching the consumers themselves. -### Writing samples — `insertMetric(...)`, `insertEvent(...)` +### Writing Samples: `insertMetric(...)`, `insertEvent(...)` `insertMetric(name:value:tags:ts:dropboxID:)` inserts one row into -`metric_samples`. It generates the row's UUID primary key itself, encodes -the `tags` dictionary to a JSON string with `encodeTagsJSON(_:)`, and -converts the caller-supplied `ts` — epoch seconds, a `Double` — to a -`Date` for storage. The schema stores timestamps as ISO-8601 text rather -than a native numeric column; the conversion from epoch seconds happens -here, at the boundary between the caller's representation and the store's, -so every other part of the codebase can treat `ts` as a plain number. - -`insertEvent(kind:nounType:rowID:estate:ts:dropboxID:)` does the same for -`event_samples`. Its `rowID` parameter is stored in the `estate_row_id` -column rather than a column literally named `row_id`, because that name is -already taken by the table's own synthetic primary key; the file's -comments call this out explicitly so a future reader does not confuse the -estate's row identifier with the row's own storage identifier. - -### Topology snapshots — `writeTopologySnapshot(...)`, `latestTopologySnapshot(estate:)`, `loadTopologyFingerprint(estate:)` +`metric_samples`. It generates the row's UUID primary key itself. It +encodes the `tags` dictionary to a JSON string with `encodeTagsJSON(_:)`. +It converts the caller-supplied `ts`, epoch seconds as a `Double`, into a +`Date` for storage. The schema stores timestamps as ISO-8601 text, not as +a native numeric column. The conversion from epoch seconds happens here, +at the boundary between the caller's representation and the store's. So +every other part of the codebase can treat `ts` as a plain number. + +`insertEvent(kind:nounType:rowID:estate:ts:dropboxID:)` does the same job +for `event_samples`. Its `rowID` parameter is stored in the +`estate_row_id` column. It is not stored in a column literally named +`row_id`. That name is already taken by the table's own synthetic primary +key. The file's comments call this out explicitly. A future reader +should not confuse the estate's row identifier with the row's own +storage identifier. + +### Topology Snapshots: `writeTopologySnapshot(...)`, `latestTopologySnapshot(estate:)`, `loadTopologyFingerprint(estate:)`. `writeTopologySnapshot(estate:generatedAt:payload:fingerprint:)` upserts one row per estate into `topology_snapshots`, keyed on the `estate` -column, so a new snapshot always replaces the previous one for that -estate rather than accumulating history. The `payload` parameter is raw -`Data` that must decode as UTF-8 text; the function throws if it does not, -because the autonomic governor that calls this method always produces -valid UTF-8 JSON, and a failure here signals a bug worth surfacing rather -than silently storing garbage. The optional `fingerprint` parameter is a -stable, process-independent hash of the topology inputs; when supplied, it -lets a restarting governor compare against a freshly computed fingerprint -and skip re-reading the estate's full set of drawers, tunnels, and facts -when nothing has changed. When omitted, the column is written as `nil`. +column. So a new snapshot always replaces the previous one for that +estate, instead of accumulating history. The `payload` parameter is raw +`Data` that must decode as UTF-8 text. The function throws if it does +not, because the autonomic governor that calls this method always +produces valid UTF-8 JSON. A failure here signals a bug worth surfacing, +rather than silently storing garbage. The optional `fingerprint` +parameter is a stable, process-independent hash of the topology inputs. +When supplied, it lets a restarting governor compare against a freshly +computed fingerprint. The governor can then skip re-reading the estate's +full set of drawers, tunnels, and facts when nothing has changed. When +the parameter is omitted, the column is written as `nil`. `latestTopologySnapshot(estate:)` reads the payload back. Passing a specific estate identifier is a primary-key lookup that returns at most one row. Passing `nil` instead asks for the single newest snapshot across -every estate in the table — the behavior the dashboard's "all estates" -view relies on, since it has no single estate identifier to filter by. -Comparing "newest" requires care: the column is declared as a timestamp, -but PersistenceKit's SQLite backend reads timestamp columns back as plain -text, while its in-memory backend used in tests reads them back as native -timestamp values. `generatedAtInstant(_:)` is a small private helper that -normalizes either representation to a `Date` before comparing, because an -earlier version of this code matched only the timestamp representation, -which meant every row tied under SQLite and an arbitrary one won — a bug -the in-memory-backed tests could not catch because they never exercised -this mismatch. +every estate in the table. This is the behavior the dashboard's "all +estates" view relies on, since that view has no single estate identifier +to filter by. Comparing "newest" requires care. The column is declared as +a timestamp, but PersistenceKit's SQLite backend reads timestamp columns +back as plain text. Its in-memory backend, used in tests, reads them back +as native timestamp values instead. `generatedAtInstant(_:)` is a small +private helper that normalizes either representation to a `Date` before +comparing. An earlier version of this code matched only the timestamp +representation. Under that earlier version, every row tied under SQLite. +An arbitrary row won each time. This was a bug. The in-memory-backed +tests could not catch it, because they never exercised this mismatch. `loadTopologyFingerprint(estate:)` is the read side of the fingerprint -parameter described above: a primary-key lookup that returns `nil` when no -snapshot exists yet, when the row predates schema version 3, or when a -snapshot was written without a fingerprint. +parameter described above. It is a primary-key lookup. The lookup returns +`nil` in three cases: no snapshot exists yet, the row predates schema +version three, or a snapshot was written without a fingerprint. -### Reading samples — `queryMetrics`, `queryMetricsByNames`, `queryEvents`, `countMetrics` +### Reading Samples: `queryMetrics`, `queryMetricsByNames`, `queryEvents`, `countMetrics`. `queryMetrics(dropboxID:)` and `queryEvents(dropboxID:)` return every row -in their respective tables, optionally filtered to one consumer's -dropbox, ordered by timestamp ascending. Both decode raw storage rows into -the public `MetricRow` and `EventRow` structs, skipping any row that fails -to decode rather than throwing — a defensive choice that tolerates a -malformed row without losing the rest of the query's results. - -`queryMetricsByNames(_:dropboxID:)` exists for a specific reason: reading -every metric row and filtering client-side does not scale once the table -holds many distinct metric names. This method issues a `WHERE name IN -(...)` predicate so the database itself narrows the result set, and it -short-circuits to an empty array without querying at all when the caller -passes an empty name set. The method's documentation says explicitly that -hot read paths should use this instead of `queryMetrics` plus a Swift-side -filter. +in their respective tables. Each call can optionally filter to one +consumer's dropbox, and results come back ordered by timestamp ascending. +Both decode raw storage rows into the public `MetricRow` and `EventRow` +structs. Both skip any row that fails to decode. Neither throws. This is +a defensive choice. It tolerates a malformed row without losing the rest +of the query's results. + +`queryMetricsByNames(_:dropboxID:)` exists for a specific reason. Reading +every metric row and filtering it on the client side does not scale. It +breaks down once the table holds many distinct metric names. This method +issues a `WHERE name IN (...)` predicate. The database itself then +narrows the result set. The method also short-circuits to an empty +array, with no query at all, when the caller passes an empty name set. +The method's documentation states explicitly that hot read paths should +use this, instead of `queryMetrics` plus a Swift-side filter. `countMetrics()` answers "how many metric rows exist" with a `COUNT(*)` -query that never decodes a single row. It backs a dashboard total-count -display without paying the cost of reading and parsing every row just to -throw away everything except the count. - -### Retention — `deleteMetricsBefore(cutoff:now:)`, `deleteEventsBefore(cutoff:now:)` - -Both methods delete rows whose `ts` is strictly before a caller-supplied -`cutoff`, then update the `retention_cutoff` control row through the -private `recordRetentionCutoff(_:now:)` helper. Neither method reads the -system clock; `cutoff` and `now` both come from the caller. This -determinism rule — no `Date()` calls inside the engine — is what lets a -test construct an exact, reproducible retention scenario instead of racing -against wall-clock time. The stored cutoff records the boundary that was -applied, not the moment retention ran, because a dashboard showing "data -older than this point was removed" is more useful to an operator than a +query. This query never decodes a single row. It backs a dashboard +total-count display. It never pays the cost of reading and parsing every +row just to throw away everything except the count. + +### Retention: `deleteMetricsBefore(cutoff:now:)`, `deleteEventsBefore(cutoff:now:)` + +Both methods delete rows whose `ts` sits strictly before a +caller-supplied `cutoff`. Both then update the `retention_cutoff` control +row through the private `recordRetentionCutoff(_:now:)` helper. Neither +method reads the system clock. `cutoff` and `now` both come from the +caller. This determinism rule matters. The engine never calls `Date()` +internally. A test can then build an exact, reproducible retention +scenario. The test never has to race against wall-clock time. The stored +cutoff records the boundary that was applied. It does not record the +moment retention ran. A dashboard can then show "data older than this +point was removed." That message is more useful to an operator than a timestamp of when the deletion happened to execute. -### DB-layer health — `storageStats(now:)` +### DB-Layer Health: `storageStats(now:)` -`storageStats(now:)` reports statistics about the stats store's own SQLite -file — page counts, WAL frame counts, file size — as distinct from any -statistics about an observed estate's own storage. It delegates to -PersistenceKit's `StorageIntrospection` capability, which the SQLite -backend implements directly, so the call always succeeds in practice; the -optional return type exists only to keep the API honest for a -hypothetical backend that does not support introspection. The `now` -parameter follows the same determinism rule as retention: the caller +`storageStats(now:)` reports statistics about the stats store's own +SQLite file: page counts, WAL frame counts, file size. These are distinct +from any statistics about an observed estate's own storage. The method +delegates to PersistenceKit's `StorageIntrospection` capability, which +the SQLite backend implements directly. So the call always succeeds in +practice. The optional return type exists only to keep the API honest for +a hypothetical backend that does not support introspection. The `now` +parameter follows the same determinism rule as retention. The caller supplies the timestamp to stamp on the snapshot. -### JSON tag encoding — `encodeTagsJSON(_:)`, `decodeTagsJSON(_:)` +### JSON Tag Encoding: `encodeTagsJSON(_:)`, `decodeTagsJSON(_:)` -Metric tags are a flat `[String: String]` map, stored as a JSON text -column rather than a separate table, because the tag set is small and -simple enough that a second table would add complexity without adding -value. `encodeTagsJSON(_:)` encodes with `.sortedKeys` so that the same -tag map always produces the same JSON string — useful for tests and for -any future deduplication logic that might compare encoded rows. -`decodeTagsJSON(_:)` is forward-compatible by design: a string that fails -to parse decodes to an empty dictionary rather than throwing, so a future -change to the tag encoding cannot break old rows written before the -change. +Metric tags form a flat `[String: String]` map. The store keeps this map +as a JSON text column, not a separate table. The tag set is small and +simple. A second table would add complexity without adding value. `encodeTagsJSON(_:)` encodes with `.sortedKeys`, so +the same tag map always produces the same JSON string. This helps tests, +and it helps any future deduplication logic that might compare encoded +rows. `decodeTagsJSON(_:)` is forward-compatible by design. A string that +fails to parse decodes to an empty dictionary, rather than throwing. So a +future change to the tag encoding cannot break old rows written before +the change. -### Row result types — `MetricRow`, `EventRow` +### Row Result Types: `MetricRow`, `EventRow` Both are plain `Sendable` structs that mirror a decoded database row. Their failable initializers, `init?(storageRow:)`, pattern-match every -expected column and its expected `TypedValue` case; if any column is -missing or has an unexpected type, the initializer returns `nil` rather +expected column and its expected `TypedValue` case. If any column is +missing, or has an unexpected type, the initializer returns `nil` rather than crashing. This is what lets `queryMetrics` and `queryEvents` use -`compactMap` to silently drop malformed rows instead of failing the whole -query. +`compactMap` to silently drop malformed rows, instead of failing the +whole query. ## Rust Port and Conformance -The `rust/` directory contains a second implementation of this library: -`src/lib.rs` re-exports the public surface, `src/sink.rs` implements -`PersistenceStatsSink`, and `src/store.rs` implements `StatsStore` and -`StatsStoreSchema` against the Rust build of PersistenceKit. The two ports -share table names, column names, and the monitoring-flag semantics -exactly; the schema-declaration code in `store.rs`'s `make_schema()` +The `rust/` directory contains a second implementation of this library. +`src/lib.rs` re-exports the public surface. `src/sink.rs` implements +`PersistenceStatsSink`. `src/store.rs` implements `StatsStore` and +`StatsStoreSchema` against the Rust build of PersistenceKit. The two +ports share table names, column names, and the monitoring-flag semantics +exactly. The schema-declaration code in `store.rs`'s `make_schema()` function is a line-by-line mirror of the Swift `StatsStore.schema` value. -The two legs differ where Rust and Swift semantics genuinely differ, -rather than translating one language's idioms into the other +The two legs differ only where Rust and Swift semantics genuinely +differ, rather than translating one language's idioms into the other mechanically. `PersistenceStatsSink::receive` in Rust performs its -database write inline, synchronously, because the Rust `StatsSink` trait -is synchronous and Rust's `SqliteStorage` is safely accessible without an -async runtime; the Swift version's `Task` dispatch exists only because +database write inline and synchronously. The Rust `StatsSink` trait is +synchronous, and Rust's `SqliteStorage` is safely accessible without an +async runtime. The Swift version's task dispatch exists only because Swift's `Storage` protocol is actor-isolated and asynchronous. The -monitoring-flag check and the discard-on-off behavior are identical in -both. - -The Rust `store.rs` file also owns timestamp conversion explicitly through -two free functions, `epoch_to_iso8601` and `iso8601_to_epoch`, because Rust -has no equivalent to Foundation's `DateFormatter` built in. These functions -clamp out-of-range or non-finite inputs (`NaN`, infinities, values outside -the year 0001–9999 window) to the nearest valid boundary instead of -panicking or producing malformed text; a block of unit tests at the bottom -of `store.rs` checks every one of these edge cases directly, since a -malformed ISO-8601 string would break every string comparison retention -and topology-snapshot ordering depend on. The Rust store also exposes one -convenience method the Swift store does not need, -`write_topology_snapshot_bytes`, which accepts raw bytes and performs a -lossy UTF-8 conversion — useful because Rust's type system otherwise -forces a caller to prove UTF-8 validity before calling the primary method, -a proof Swift's runtime guard does not require its callers to construct in -advance. +monitoring-flag check and the discard-on-off behavior stay identical in +both languages. + +The Rust `store.rs` file also owns timestamp conversion explicitly, +through two free functions: `epoch_to_iso8601` and `iso8601_to_epoch`. +Rust has no equivalent to Foundation's `DateFormatter` built in. Some +inputs are unsafe on their own, such as `NaN`, infinities, and +out-of-range years. The valid year window runs from 0001 to 9999. These +functions clamp unsafe values to the nearest valid boundary in that +window. They never panic. They never produce malformed text. A block of +unit tests at the bottom of `store.rs` checks every one of +these edge cases directly. A malformed ISO-8601 string would break every +string comparison that retention and topology-snapshot ordering depend +on. The Rust store also exposes one convenience method the Swift store +does not need: `write_topology_snapshot_bytes`. This method accepts raw +bytes and performs a lossy UTF-8 conversion. It is useful because Rust's +type system otherwise forces a caller to prove UTF-8 validity before +calling the primary method. Swift's runtime guard does not require its +callers to construct that proof in advance. Conformance between the two ports is enforced by two independent but matching test suites: `Tests/ObserverSinkTests/ObserverSinkConformanceTests.swift` -and `rust/tests/conformance.rs`. Both exercise the same scenarios — schema -version, control-row seeding, the monitoring flag and its persistence -across a close-and-reopen cycle, metric and event round-trips, retention -roll-off for both tables, tag JSON round-tripping, the topology-snapshot -read/write/latest-wins/per-estate-isolation paths, and the DB-layer health -check — over the same table and column names. When you change either leg's -schema or flag semantics, update both files and both test suites; nothing -in either build enforces this automatically. +and `rust/tests/conformance.rs`. Both exercise the same scenarios over the +same table and column names. These scenarios include the schema version +and control-row seeding. They include the monitoring flag, plus its +persistence across a close-and-reopen cycle. They include metric and +event round-trips, retention roll-off for both tables, and tag JSON +round-tripping. They include the topology-snapshot read, write, and +latest-wins paths too. They include per-estate isolation and the +DB-layer health check as well. Changing either leg's schema or flag +semantics requires updating both files and both test suites by hand. +Nothing in either build enforces this automatically. diff --git a/packages/libs/ObserverSink/docs/OVERVIEW.md b/packages/libs/ObserverSink/docs/OVERVIEW.md index c78914f..c072be7 100644 --- a/packages/libs/ObserverSink/docs/OVERVIEW.md +++ b/packages/libs/ObserverSink/docs/OVERVIEW.md @@ -16,114 +16,120 @@ sources: ## What This Library Does ObserverSink records telemetry for the MOOTx01 manager pipeline. Telemetry -is data that describes what a running program is doing, separate from the -memories the program stores. A companion library, IntellectusLib, defines -two shapes of telemetry: a metric (a named number, such as a latency -measurement) and an event (a record that one memory row was captured or -otherwise acted on). IntellectusLib defines only the shapes and a plug-in -point; it does not know how to save anything to disk. +is data that describes what a running program does. It is separate from +the memories the program stores. A companion library, IntellectusLib, +defines two shapes of telemetry. A metric is a named number, such as a +latency measurement. An event is a record that one memory row was +captured or otherwise acted on. IntellectusLib defines only the shapes +and a plug-in point. It does not know how to save anything to disk. ObserverSink is the plug-in that saves them. It ships two pieces. The -first, `PersistenceStatsSink`, is the receiver that IntellectusLib calls -each time new telemetry arrives. The second, `StatsStore`, is a small -SQLite database and the code that reads and writes it. Together they turn -a stream of in-memory telemetry values into durable rows that a separate -manager process can later query. +first piece is `PersistenceStatsSink`. IntellectusLib calls it each time +new telemetry arrives. The second piece is `StatsStore`. It is a small +SQLite database, plus the code that reads and writes it. Together the two +pieces turn a stream of in-memory telemetry values into durable rows. A +separate manager process can later query those rows. ## The Problem It Solves -A running MOOTx01 process is a black box unless something records what it -does. An operator who wants to know how fast captures are running, how -many think-cycles fired, or whether a given estate is behaving normally -needs that information written somewhere durable, because the process -itself may restart or be inspected only after the fact. An estate is one -user's complete memory store in MOOTx01. +A running MOOTx01 process is a black box unless something records what +it does. An operator may want to know how fast captures run. An operator +may want to know how many think-cycles fired. An operator may want to +know whether a given estate behaves normally. An estate is one user's +complete memory store in MOOTx01. This kind of information needs a +durable home, because the process may restart or get inspected only +after the fact. Writing telemetry to disk on every observation is expensive if done -carelessly, and it is dangerous if it cannot be turned off. ObserverSink +carelessly. It is also dangerous if it cannot be turned off. ObserverSink solves both problems with one mechanism: a flag row in the database itself. Before writing anything, the sink checks whether monitoring is -switched on. The manager — not the process being observed — owns that -switch, so it can enable or disable recording for every consumer at once, -without restarting anything. This is the flag-row signal mechanism -described in the source comments as Bob's confirmed choice for Manager 1.0. +switched on. The manager owns that switch. The process being observed +does not. This lets the manager enable or disable recording for every +consumer at once, with no restart required. The source comments describe +this flag-row signal as Bob's confirmed choice for Manager 1.0. The library also has to work without slowing down the code it observes. -`receive(_:)`, the method IntellectusLib calls, must return immediately -because it may be called from time-sensitive code paths. ObserverSink -meets this requirement by never performing file I/O on the calling -thread; the actual database write always happens elsewhere. +`receive(_:)` is the method IntellectusLib calls. It must return +immediately, because it may run on time-sensitive code paths. ObserverSink +meets this requirement in a simple way. It never performs file I/O on the +calling thread. The actual database write always happens elsewhere. ## How It Works Each telemetry value follows the same path. IntellectusLib calls -`PersistenceStatsSink.receive(_:)` with one `StatSample`, which is either -a `.metric` case or an `.event` case. The sink captures what it needs and -starts an unstructured background task, then returns immediately. The +`PersistenceStatsSink.receive(_:)` with one `StatSample`. That sample is +either a `.metric` case or an `.event` case. The sink captures what it +needs and starts a background task. It then returns right away. The calling code never waits for the database write to finish. Inside that background task, the sink first asks the store whether -monitoring is enabled. This is a database read, but a cheap one: the +monitoring is enabled. This is a database read, but a cheap one. The `control` table holds only a handful of rows. If the flag is off, the -sample is dropped and nothing else happens. If it is on, the sink asks the -store to serialize the sample into the correct table: `.metric` samples go -into `metric_samples`, `.event` samples go into `event_samples`. If the -write fails for any reason, the error is logged and swallowed — a -telemetry failure must never crash the program it is describing. - -The `StatsStore` also owns two other kinds of data beyond metrics and -events. A `control` table holds the monitoring flag itself, plus a -timestamp recording when data was last rolled off through retention. A -`topology_snapshots` table holds one row per estate: the latest picture of -that estate's structure, written by a separate background process (the -autonomic governor) and served back verbatim to dashboards that ask for -it. This table is not telemetry in the metric/event sense, but it shares -the same store, the same open/migrate machinery, and the same principle: -one writer, many readers, no history kept beyond the latest value. +sample is dropped and nothing else happens. If the flag is on, the sink +asks the store to save the sample in the correct table. A `.metric` +sample goes into `metric_samples`. A `.event` sample goes into +`event_samples`. If the write fails for any reason, the sink logs the +error and swallows it. A telemetry failure must never crash the program +it describes. + +`StatsStore` also owns two other kinds of data, beyond metrics and +events. A `control` table holds the monitoring flag itself. It also holds +a timestamp that records when data was last rolled off through +retention. A `topology_snapshots` table holds one row per estate: the +latest picture of that estate's structure. A separate background process, +the autonomic governor, writes this picture. Dashboards ask for it and +get it back exactly as written. This table is not telemetry in the +metric or event sense. Even so, it shares the same store and the same +open-and-migrate machinery. It follows the same guiding principle too. +One writer produces the data. Many readers consume it. No history +persists beyond the latest value. Retention keeps the two sample tables from growing without bound. The manager periodically calls `deleteMetricsBefore(cutoff:now:)` and -`deleteEventsBefore(cutoff:now:)` with a cutoff timestamp it computes -itself; the store never reads the system clock on its own. This keeps the -store's behavior fully determined by its caller's inputs, which matters -for testing and for reasoning about what a given retention pass will do. +`deleteEventsBefore(cutoff:now:)`. It supplies the cutoff timestamp +itself; the store never reads the system clock on its own. This keeps +the store's behavior fully determined by its caller's inputs. That +property matters for testing, and for reasoning about what a given +retention pass will do. ## How the Pieces Fit -Figure 1 shows the library's topology — its major parts and how data moves +Figure 1 shows the library's topology: its major parts and how data moves between them. ![Figure 1. Topology of ObserverSink](topology.svg) *Figure 1. Topology of ObserverSink. A telemetry sample flows from the -caller through the sink's background task, past the monitoring-flag gate, -into the matching SQLite table. A separate flow lets the autonomic +caller through the sink's background task, past the monitoring-flag +gate, into the matching SQLite table. A separate flow lets the autonomic governor publish topology snapshots that a dashboard reads back later.* `Intellectus.report(_:)`, part of IntellectusLib, is the only caller of `PersistenceStatsSink.receive(_:)`. The sink holds a reference to a -`StatsStore` and a `dropboxID` string that tags every row it writes with -the identity of the process that produced it — this lets one shared store -serve many concurrent consumers (for example, several `aria-mcp` process -instances) without their rows becoming mixed up. SQLite's write-ahead log -mode makes concurrent writers from separate consumers safe without extra -locking in this library. - -`StatsStore` wraps a `SQLiteStorage` value supplied by PersistenceKit, a -kit that provides the schema-declaration and row-storage machinery this -library builds on. `StatsStore` never talks to SQLite directly; it +`StatsStore` and a `dropboxID` string. That string tags every row the +sink writes with the identity of the process that produced it. This lets +one shared store serve many concurrent consumers, for example several +`aria-mcp` process instances, without mixing up their rows. SQLite's +write-ahead log mode makes concurrent writers from separate consumers +safe, with no extra locking needed in this library. + +`StatsStore` wraps a `SQLiteStorage` value supplied by PersistenceKit. That +kit provides the schema-declaration and row-storage machinery this +library builds on. `StatsStore` never talks to SQLite directly. It describes its four tables as a `SchemaDeclaration` and lets PersistenceKit create, migrate, and query them. ## What Ships in the Package -The package ships two Swift source files — `PersistenceStatsSink.swift` -and `StatsStore.swift` — plus a Rust port in `rust/` that reimplements the -same schema and the same sink logic for non-Swift consumers. Both legs are -exercised by parallel conformance test suites (`Tests/ObserverSinkTests` -in Swift, `rust/tests/conformance.rs` in Rust) that check the same sixteen -scenarios: schema version, control-row seeding, the monitoring flag, -metric and event round-trips, retention roll-off, tag encoding, and the -topology-snapshot read/write paths. There are no pinned data artifacts in -this package; every table starts empty and is populated only by the -processes that use it at run time. +The package ships two Swift source files, `PersistenceStatsSink.swift` +and `StatsStore.swift`, plus a Rust port in `rust/`. The Rust port +reimplements the same schema and the same sink logic for consumers that +are not written in Swift. Parallel conformance test suites exercise both +legs: `Tests/ObserverSinkTests` for Swift, `rust/tests/conformance.rs` for +Rust. Both suites check the same sixteen scenarios. These cover the +schema version, control-row seeding, the monitoring flag, and metric and +event round-trips. They also cover retention roll-off, tag encoding, and +topology-snapshot reads and writes. This package pins no data artifacts. +Every table starts empty and fills only as the processes that use it +run.