diff --git a/packages/kits/ConvergenceKit/docs/AGENT_MAP.md b/packages/kits/ConvergenceKit/docs/AGENT_MAP.md new file mode 100644 index 0000000..3051db8 --- /dev/null +++ b/packages/kits/ConvergenceKit/docs/AGENT_MAP.md @@ -0,0 +1,140 @@ +--- +doc: AGENT_MAP +package: ConvergenceKit +repo: moot-system +authored_commit: 909513d0a8ecb1e9e903af9f4d25b3e4f2528242 +authored_date: 2026-07-04 +sources: + - path: Sources/ConvergenceKit/SyncEngine.swift + blob: 32e69e0ea5002d8b699866183371527f675023cd + - path: Sources/ConvergenceKit/SyncRecord.swift + blob: 8c29f63ea9f965f1ea75d3bbff71632f76699233 + - path: Sources/ConvergenceKit/SyncTypes.swift + blob: 52ed3cb07f0483d0e541b86a8da080791402d132 + - path: Sources/ConvergenceKitCloudKit/CKRecordMapping.swift + blob: 9b61903296533f8044bee5a4f93ca2dc1b76b653 + - path: Sources/ConvergenceKitCloudKit/CloudKitSyncEngine.swift + blob: 7dd43ccf5454739a3d1ae1cda0870cf507c479c6 + - path: Sources/ConvergenceKitFederation/FederationIdentity.swift + blob: b64358fe36fd241049e9f5bdbbb335f1f3d29464 + - path: Sources/ConvergenceKitFederation/FederationSyncEngine.swift + blob: 9bf722f8e7fc24ae12ac72d571c135da2b40af62 + - path: Sources/ConvergenceKitFederation/HyperplaneFamilyExchange.swift + blob: 4880e847918c253f3d99a02fdd65c15f172d32e1 + - path: Sources/ConvergenceKitNone/ConvergenceKitNone.swift + blob: 5d3556a08c8bc80ec6b824ceb4e0838db9c95ecc +--- + +# AGENT_MAP: ConvergenceKit + +PURPOSE: replicates PersistenceKit rows across device/estate boundaries. One protocol (SyncEngine), three interchangeable backends (None/CloudKit/Federation). App writes only to PersistenceKit; backend observes via StorageObserver, ships via SyncRecord wire format, applies inbound through PersistenceKit's own write path (RowStore.upsert/insert/delete), which fires StorageObserver again on the receiving side. + +DEPS: core target imports SubstrateTypes (HLC, Fingerprint256), PersistenceKit (Storage, StorageObserver, TableChange, TypedValue). ConvergenceKitCloudKit adds CloudKit, os. ConvergenceKitFederation adds Crypto (swift-crypto, Ed25519), os. ConvergenceKitNone adds nothing beyond core. Imported by: NONE within the SDK at this commit: application-layer composition only; QueueKit's Package.swift explicitly excludes it as a dependency (spec §11, DECISION_KIT_GRAPH_REFACTOR_2026-05-19.md). Rust port in rust/ mirrors core types + wire format + None/Federation backends; CloudKit has no Rust port (Apple-only, by design). + +ENTRY POINTS (most callers need only these): +- SyncEngine.swift:26 `SyncEngine.enable(manifest:storage:) async throws`: call once before push/pull/subscribe +- SyncEngine.swift:34/:38 `push() async throws -> SyncReceipt` / `pull() async throws -> SyncReceipt`: one-shot cycles +- SyncEngine.swift:41 `subscribe() -> AsyncStream`: live feed for UI +- FederationSyncEngine.swift:80 `FederationSyncEngine.pair(with:via:family:)`: Federation-only, establishes peer relationship before push/pull do anything + +## Symbol Table + +### Protocol: SyncEngine.swift +- :21 `protocol SyncEngine: Sendable`: 4 methods + 1 async property, every backend conforms +- :26 `enable(manifest:storage:) async throws`: starts local-write observation; establishes remote subscriptions if any +- :30 `disable() async throws`: idempotent teardown +- :34 `push()` / :38 `pull()`: one-shot, return SyncReceipt +- :41 `subscribe() -> AsyncStream` +- :44 `state: SyncState { get async }` + +### Core types: SyncTypes.swift +- :22 `enum SyncDirection: String`: bidirectional | pushOnly | pullOnly +- :29 `enum ConflictPolicy: String`: lastWriterWinsByHLC (default) | appendOnly | localWins | remoteWins +- :42 `struct SyncedTable`: name/direction/primaryKeyColumn/conflictPolicy; :54 init defaults direction=.bidirectional, conflictPolicy=.lastWriterWinsByHLC +- :70 `struct SyncManifest`: kitID/schemaVersion/zoneIdentifier/tables; :94 `table(named:) -> SyncedTable?` +- :100 `struct SyncReceipt`: pushed/pulled/conflicts/timestamp; :113 `.empty` shared zero-value +- :117 `enum SyncEvent`: .remoteChangesApplied(count:) / .pushCompleted(receipt:) / .peerConnected(identity:) / .peerDisconnected(identity:reason:) / .error(SyncError) +- :126 `enum SyncState`: .disabled | .enabled(zone:lastPushAt:lastPullAt:) | .syncing(direction:) | .error(_:retryAt:) +- :134 `enum SyncError: Error, Equatable`: notEnabled/alreadyEnabled/schemaMismatch/kitMismatch/transportFailure/decodingFailure/encodingFailure/peerUnreachable/authenticationFailed/unsupportedTable/corruptRemoteIdentity(recordName:): last case: NEVER fabricate a UUID for an unparseable remote recordName, see CKRecordMapping.decode + +### Wire format: SyncRecord.swift +- :28 `struct SyncRecord: Codable`: table/event/rowKey/values/hlc/schemaVersion/kitID; explicit CodingKeys: Rust serde renames match verbatim, do not let these drift +- :63 `enum SyncEventKind: String, Codable`: insert|update|delete; :68 `init(from: StorageEvent)`, :76 `.asStorageEvent`: Codable mirror, kept separate from PersistenceKit.StorageEvent deliberately +- :87 `struct PackedHLC: Codable, Hashable`: physicalTime/logicalCount/nodeID; Codable wrapper of SubstrateTypes.HLC +- :112 `struct SyncValueMap: Codable`: wraps [String:TypedValue] as [String:SyncValueBox] +- :144 `struct SyncValueBox`: kind + payload (13 TypedValue cases); adjacently-tagged JSON matching Rust `#[serde(tag="kind",content="payload")]` +- :206 `SyncValueBox: Codable` extension: :211 encode omits payload key for .null (matches Rust unit-variant omission); :227 timestamp encode guards `interval.isFinite` + Int64 range BEFORE `Int64(interval)` cast: Int64(_:) traps on NaN/±inf/overflow, corrupt inbound data can produce these +- :296 `struct FingerprintWire: Codable, Hashable`: block0..3 UInt64, Codable wrapper of SubstrateTypes.Fingerprint256 + +### None backend: ConvergenceKitNone.swift +- :28 `final class NoSyncEngine: SyncEngine`: passthrough; push/pull return .empty once enabled, throw .notEnabled before +- :53 `subscribe()`: stream that NEVER emits; closes only on caller task cancellation +- :64 `actor StateActor`: enable throws .alreadyEnabled on repeat; tracks manifest only for state reporting + +### CloudKit mapping: CKRecordMapping.swift +- :26 `enum CKRecordMapping`: generic row↔CKRecord translator, driven by table name + manifest, no per-table hardcoding +- :31 `recordType(kitID:table:) -> String`: "\(kitID)_\(table)" +- :36 `recordID(rowKey:zone:) -> CKRecord.ID` +- :43 `record(from:table:rowKey:hlc:schemaVersion:kitID:zone:) throws -> CKRecord`: adds reserved _syncHLC/_syncSchemaVersion/_syncKitID fields +- :65 `decode(_:) throws -> DecodedRecord`: reverse; throws .decodingFailure if _syncHLC/_syncSchemaVersion missing; recovers table from "kitID_table" split; **throws .corruptRemoteIdentity if recordName is not a valid UUID: NEVER fabricate a fresh UUID here** +- :103 `assign(value:to:forKey:)` (private): TypedValue→CKRecord field; .array throws .encodingFailure (unsupported); .fingerprint packs 4×UInt64 LE into 32 bytes +- :147 `typedValue(from:)` (private): CKRecord field→TypedValue; NSNumber objCType sniffed for bool/float/int disambiguation +- :172 `packed(_:) -> Int64` / :179 `unpacked(_:) -> HLC`: HLC↔Int64, PINNED bit layout: 48 bits physical | 12 bits logical | 4 bits node: changing widths breaks previously-stored _syncHLC values +- :192 `struct SyncMeta`: hlc/schemaVersion/kitID extracted from _sync* fields +- :198 `struct DecodedRecord`: table/rowKey/values (clean, no _sync* keys)/syncMeta + convenience accessors + +### CloudKit engine: CloudKitSyncEngine.swift +- :34 `final class CloudKitSyncEngine: SyncEngine`: :43 init(containerIdentifier:) defers CKContainer resolution to enable() so tests can construct without iCloud entitlement +- :78 `actor CloudKitStateActor`: owns container/manifest/storage/pendingOutbound/serverChangeToken/hlcGenerator +- :112 `enable(manifest:storage:)`: creates CK zone (tolerates already-exists); subscribes storage.observer.observe per push-eligible table → recordOutbound +- :106 `hlcGenerator = HLCGenerator(nodeID: Int32.random(in: 1...0x0F))`: mints HLC only when an observed change carries none +- :175 `push()`: drains pendingOutbound; builds CKRecords via CKRecordMapping.record; one modifyRecords(saving:deleting:) call; HLC: prefer change.hlc, else hlcGenerator.send(now:) (NOT currentTime(): send advances the logical counter, avoiding same-millisecond collisions) +- :254 `pull()`: recordZoneChanges(inZoneWith:since:); per-record gate: kitID match → schemaVersion match → table declared & not pushOnly; any failure caught+logged+counted as conflict, does NOT abort the batch; deletions are bare CKRecord.ID (no table info): attempted against every non-pushOnly manifest table +- :339 `applyInbound(_:syncedTable:storage:)` (internal, not private: LWW tests call via @testable): dispatches 4 ConflictPolicy arms; .lastWriterWinsByHLC reads existing row's _syncHLC as EITHER .hlc (InMemory) OR .int (SQLite/Postgres, raw packed value): must handle both + +### Federation identity: FederationIdentity.swift +- :24 `struct PeerIdentity: Hashable`: 32-byte Ed25519 public key +- :32 `struct LocalIdentity`: :36 init() generates fresh Curve25519.Signing.PrivateKey per engine instance; :42 init(privateKeyBytes:) restores from caller bytes: module does NOT persist identity itself +- :48 `sign(_:) throws -> Data` +- :53 `enum FederationSignature`: :54 `verify(_:of:by:) -> Bool`: returns false (not throw) for both malformed pubkey bytes and bad signature + +### Federation pairing shapes: HyperplaneFamilyExchange.swift +- :27 `struct HyperplaneFamilySpec: Codable, Hashable`: seed + dimension (default 256); seed alone reproduces the family deterministically on both sides +- :42 `struct PairingProposal: Codable`: proposerPublicKey/proposedFamily/nonce +- :54 `struct PairingAcceptance: Codable`: accepterPublicKey/acceptedFamily/signatureOfProposal +- NOTE: these types are defined but NOT YET wired into FederationSyncEngine.pair: v1.0 pair() takes an already-agreed HyperplaneFamilySpec directly, does not negotiate/sign Proposal/Acceptance + +### Federation engine: FederationSyncEngine.swift +- :39 `final class FederationSyncEngine: SyncEngine` +- :80 `pair(with:via:family:) async throws`: exchanges pubkeys + family spec; in-process only at v1.0 +- :84 `identity: LocalIdentity { get async }` +- :98 `enum PayloadKind: UInt8, Codable`: .syncRecordBatch = 0x01 (only v1.0 variant); 0x02 RESERVED for fieldWriteEventBatch: never reassign +- :128 `envelopeSigningBytes(senderPublicKey:payloadKind:payload:hlc:) -> Data`: PINNED byte layout (all LE): pubkey(32) | kind(1) | payload_len(4) | payload(N) | hlc.physicalTime(8) | hlc.logicalCount(4) | hlc.nodeID(4); byte-identical to Rust `envelope_signing_bytes` in federation.rs: signature covers THIS, not raw payload (closes relabel/replay seam) +- :174 `struct SignedEnvelope: Codable`: senderPublicKey/payloadKind/payload/signature/hlc; batch-level hlc minted AFTER all record HLCs in the batch (strictly later) +- :211 `protocol Relay: Sendable`: send(to:message:) / drain(for:): transport extension point; hosted relay is a drop-in conformer +- :220 `final class FederationRelay: Relay`: in-process only at v1.0, NSLock-protected [pubkey: [SignedEnvelope]] inbox dict +- :243 `actor FederationStateActor`: :244 `localIdentity = LocalIdentity()` generated once per actor +- :263 `enable(manifest:storage:)`: same observer-wiring pattern as CloudKitStateActor +- :279 `disable()`: cancels observer tasks THEN AWAITS each task's completion before clearing state (deliberately different from a cancel-only teardown: closes a race where a buffered change lands after disable() returns) +- :318 `pair(with:via:family:)`: registers peer locally, calls peerActor.acceptPeering symmetrically: one call registers both sides +- :331 `push()`: builds SyncRecord batch from pendingOutbound (HLC: change.hlc ?? hlcGenerator.send(now:), same rule as CloudKit), JSON-encodes, mints batch HLC, signs via envelopeSigningBytes, sends SignedEnvelope to every peer's relay +- :422 `pull()`: per envelope, in order: (1) senderPublicKey MUST equal the specific paired peer's key (ADR-013: valid self-signature alone does not prove pairing authorization), (2) payloadKind MUST be .syncRecordBatch, (3) signature MUST verify over envelopeSigningBytes, (4) only then JSON-decode; each surviving SyncRecord re-checked for kitID/schemaVersion/table-declared before applyInbound +- :526 `applyInbound(_:syncedTable:storage:)` (internal, not private: LWW tests): 4 ConflictPolicy arms × insert/update vs delete (delete gets full policy dispatch here, unlike CloudKit which receives bare record IDs with no policy routing): .appendOnly rejects remote deletes outright; .lastWriterWinsByHLC gates delete on stale-HLC check same as upsert path; .remoteWins deletes unconditionally; .localWins rejects all remote deletes + +## INVARIANTS / GOTCHAS + +- Application code NEVER calls a backend's push/pull for ordinary use: it writes to PersistenceKit; the enabled backend observes via StorageObserver and does the rest. Inbound changes apply through PersistenceKit's normal RowStore path (upsert/insert/delete), which is what wakes downstream StorageObserver watchers on the receiving side: do not special-case "just-synced" rows. +- CKRecordMapping.decode: an unparseable `recordID.recordName` MUST throw `.corruptRemoteIdentity`, never fabricate a fresh UUID. A fabricated identity creates a phantom row that desyncs forever (never matches the real remote row on later rounds). +- HLC bit layout in CKRecordMapping.packed/unpacked is PINNED: 48/12/4 bits (physical/logical/node). Changing widths invalidates every previously stored `_syncHLC` value. +- envelopeSigningBytes layout is PINNED and cross-port: pubkey(32)|kind(1)|payload_len(4 LE)|payload|hlc.physicalTime(8 LE)|hlc.logicalCount(4 LE)|hlc.nodeID(4 LE). Must stay byte-identical to Rust's `envelope_signing_bytes` in `rust/src/federation.rs`, or cross-port signatures fail verification. +- SyncRecord / SyncValueBox / PackedHLC / FingerprintWire CodingKeys are an explicit cross-port JSON contract (Rust serde field renames match these strings verbatim). Do not let automatic Codable key derivation replace these without checking the Rust side. +- PayloadKind 0x02 is RESERVED for `fieldWriteEventBatch`: never assign it to anything else; a receiver on an older build would silently misinterpret a reused byte value. +- Federation pull() enforces sender==paired-peer-key BEFORE trusting a valid signature (ADR-013): a syntactically valid self-signed envelope from a non-paired sender must be rejected, not merely a badly-signed one. +- `_syncHLC` on a stored row can arrive as TypedValue `.hlc` (InMemory backend, preserves the type) OR `.int` (SQLite/Postgres, raw packed integer, because those schemas don't declare the column as `.hlc`). Both `CloudKitStateActor.applyInbound` and `FederationStateActor.applyInbound` must handle both cases: do not assume one representation. +- lastWriterWinsByHLC gates BOTH upserts and deletes on HLC comparison in Federation's applyInbound; CloudKit's applyInbound only gates upserts this way because CloudKit deletes arrive as bare record IDs with no policy dispatch at all (deletes just happen, scoped only by "table is in the manifest and not pushOnly"). +- FederationStateActor.disable() cancels observer tasks THEN awaits their completion before clearing state: do not reduce this to cancel-only; a buffered change could otherwise land in the outbox after disable() returns. +- HLC minting for locally-observed changes with no HLC of their own always uses `HLCGenerator.send(now:)`, never a read-only clock snapshot (`currentTime()` equivalent): `send` advances the logical counter so two changes minted in the same millisecond do not collide. Same rule in both CloudKitStateActor.push and FederationStateActor.push. +- Timestamp encoding in SyncValueBox guards `interval.isFinite` and Int64-range BEFORE casting to Int64: `Int64(_:)` traps (crashes) on NaN/±infinity/out-of-range Double. Corrupt inbound sync data can produce such a value; the guard converts a potential crash into a catchable EncodingError. +- Public API stability per README: adding a case to SyncDirection, ConflictPolicy, SyncEvent, SyncState, or SyncError is a breaking change requiring a major version bump and a decision record. Backend additions (a fourth SyncEngine conformer) are additive, not breaking. +- CloudKit backend has NO Rust port by design (Apple-only; Swift side always handles iCloud transport). Do not expect rust/ to mirror ConvergenceKitCloudKit. +- ConvergenceKit is intentionally NOT a dependency of other in-SDK kits at this commit (e.g., QueueKit's Package.swift excludes it per spec §11): it is wired in at the application-composition layer, not the kit-graph layer. diff --git a/packages/kits/ConvergenceKit/docs/DETAILS.md b/packages/kits/ConvergenceKit/docs/DETAILS.md new file mode 100644 index 0000000..27cb06b --- /dev/null +++ b/packages/kits/ConvergenceKit/docs/DETAILS.md @@ -0,0 +1,485 @@ +--- +doc: DETAILS +package: ConvergenceKit +repo: moot-system +authored_commit: 909513d0a8ecb1e9e903af9f4d25b3e4f2528242 +authored_date: 2026-07-04 +sources: + - path: Sources/ConvergenceKit/SyncEngine.swift + blob: 32e69e0ea5002d8b699866183371527f675023cd + - path: Sources/ConvergenceKit/SyncRecord.swift + blob: 8c29f63ea9f965f1ea75d3bbff71632f76699233 + - path: Sources/ConvergenceKit/SyncTypes.swift + blob: 52ed3cb07f0483d0e541b86a8da080791402d132 + - path: Sources/ConvergenceKitCloudKit/CKRecordMapping.swift + blob: 9b61903296533f8044bee5a4f93ca2dc1b76b653 + - path: Sources/ConvergenceKitCloudKit/CloudKitSyncEngine.swift + blob: 7dd43ccf5454739a3d1ae1cda0870cf507c479c6 + - path: Sources/ConvergenceKitFederation/FederationIdentity.swift + blob: b64358fe36fd241049e9f5bdbbb335f1f3d29464 + - path: Sources/ConvergenceKitFederation/FederationSyncEngine.swift + blob: 9bf722f8e7fc24ae12ac72d571c135da2b40af62 + - path: Sources/ConvergenceKitFederation/HyperplaneFamilyExchange.swift + blob: 4880e847918c253f3d99a02fdd65c15f172d32e1 + - path: Sources/ConvergenceKitNone/ConvergenceKitNone.swift + blob: 5d3556a08c8bc80ec6b824ceb4e0838db9c95ecc +--- + +# ConvergenceKit Details + +This document walks through every source file in the package. Read +`OVERVIEW.md` first for the big picture. Files appear here in pipeline +order. First come the shared types and the protocol every backend depends +on. Next comes the wire format. Then come the three backends, from +simplest to most complex. + +## SyncTypes.swift + +This file provides the core enumerations and value types shared by every +backend: `SyncDirection`, `ConflictPolicy`, `SyncedTable`, `SyncManifest`, +`SyncReceipt`, `SyncEvent`, `SyncState`, and `SyncError`. + +`SyncDirection` states which way one table replicates. The three choices +are `bidirectional`, `pushOnly`, and `pullOnly`. Direction matters per +table because not every table should travel both ways. A phone-only table +can be marked `pushOnly` from the phone. Every other device marks that same +table `pullOnly`. A backend that receives such a table simply skips it. + +`ConflictPolicy` states how a receiver resolves a row that changed on both +sides. `lastWriterWinsByHLC` is the default. It compares the Hybrid +Logical Clock, described in `OVERVIEW.md`, on the incoming record against +the clock on the stored row. The more recent write survives. `appendOnly` +treats the table as a write-once audit log. The row's own key already +makes duplicate delivery harmless, so incoming rows are always upserted +rather than compared. `localWins` and `remoteWins` are the two absolute +policies. They suit a case where one side of a sync relationship is known +in advance to be more authoritative than the other. + +`SyncedTable.init(name:direction:primaryKeyColumn:conflictPolicy:)` +bundles one table's replication rules. `direction` and `conflictPolicy` +default to `bidirectional` and `lastWriterWinsByHLC`. A simple table +declaration therefore needs only a name and a primary key column. +`SyncManifest` is the full declaration for one sync session. It carries a +`kitID` and a `schemaVersion` that identify the application and its data +version. It carries a `zoneIdentifier` naming where remote data lives, +either a CloudKit zone name or a federation-facing label. It carries the +list of `SyncedTable` entries. `SyncManifest.table(named:)` looks up one +table's rules by name. Every backend calls this before it decides whether +and how to apply an incoming or outgoing change. + +`SyncReceipt` summarizes one push or pull cycle. It reports how many rows +moved in each direction, how many conflicts turned up, and when the cycle +finished. `SyncReceipt.empty` is the shared zero-value every backend +returns when there is nothing to do. Callers can then treat "nothing +happened" and "an empty batch happened" the same way. + +`SyncEvent` is what a live subscription emits: remote changes applied, a +push completed, a peer connected, a peer disconnected, or an error. +`SyncState` is the coarser summary a settings screen would bind to. One +case is disabled. One case is enabled, carrying a zone and last-activity +timestamps. One case is actively syncing in a direction. One case is +stuck in an error, carrying an optional retry time. + +`SyncError` covers every failure a backend can report: not enabled yet, +already enabled, a transport failure, an encoding failure, a decoding +failure, an unreachable peer, a failed authentication, an unsupported +table, and `corruptRemoteIdentity`. It also covers a schema mismatch and +a kit mismatch on an incoming record. `corruptRemoteIdentity` exists for a +specific reason. `CKRecordMapping.swift` explains that reason where the +error is thrown. A remote record +whose identifier cannot be read as a valid row key must never be given a +fabricated one. A fabricated identity would create a phantom row. That +phantom row would never match the real one on later sync rounds. +`SyncError` conforms to `Equatable` so tests can assert exactly which error +a call produced. + +## SyncEngine.swift + +This file provides `SyncEngine`, the protocol every backend conforms to. +It is the seam that lets application code, and the other two documents in +this package, describe "a backend" once instead of three times. + +The protocol has four methods and one property. +`enable(manifest:storage:)` must be called once before any other method. +It is where a backend establishes remote subscriptions, if any, and starts +observing local writes. `disable()` tears the same things down. It must be +safe to call more than once. `push()` and `pull()` are the one-shot +operations that move data in each direction. Each returns a `SyncReceipt`. +`subscribe()` returns an `AsyncStream` for callers that want a +live feed rather than polling. `state` is a computed, asynchronously +readable property. It suits a user interface that only needs the coarse +`SyncState` snapshot. + +`enable`, `push`, and `pull` are declared `async throws` rather than +synchronous. Every real backend performs network or disk work, so this +choice matters. Declaring the protocol this way once means every +conforming backend, and every caller, handles that work the same way. + +## SyncRecord.swift + +This file provides the wire format for one replicated row change: +`SyncRecord`, plus the smaller types it is built from. + +`SyncRecord` carries a table name, an event kind, a row key, an HLC, a +schema version, and a kit identifier. It also carries the changed column +values, or `nil` for a delete. It declares its `CodingKeys` explicitly. It does not rely on +Swift's automatic key derivation. The SDK's Rust port decodes the exact +same JSON, using serde field renames that must match these strings +character for character. An automatic default would still work today. It +would silently break the two ports' agreement the moment either side's +naming convention changed without the other noticing. + +`SyncEventKind` is a Codable, string-backed mirror of PersistenceKit's +`StorageEvent`. It has three cases: `insert`, `update`, and `delete`. +`SyncEventKind.init(from:)` +and `.asStorageEvent` convert both ways. A record's event kind is +duplicated here rather than reusing `StorageEvent` directly. `StorageEvent` +belongs to PersistenceKit. It is not guaranteed to be a stable wire +format. `SyncEventKind` is ConvergenceKit's own promise about what a byte +on the wire means. + +`PackedHLC` is a Codable wrapper around `SubstrateTypes.HLC`. It exposes +the same three integer fields directly: `physicalTime`, `logicalCount`, +and `nodeID`. It exists because the raw `HLC` type is not itself built for +a stable wire encoding. + +`SyncValueMap` wraps `[String: TypedValue]`, PersistenceKit's row-value +type, as `[String: SyncValueBox]`. `TypedValue` covers thirteen kinds of +column value. Several of these do not map onto a single native JSON type. +They include an HLC, a fingerprint, and a nested array. `SyncValueBox` +gives each value an explicit `kind` string tag alongside its `payload`. +This is an adjacently-tagged encoding. It matches Rust serde's +`#[serde(tag = "kind", content = "payload")]` attribute. The SDK chose it +specifically so a Rust decoder and a Swift decoder agree on the same JSON +shape, without either language's default enum encoding leaking through. + +`SyncValueBox.encode(to:)` and `.init(from:)` implement that tagging by +hand. Two cases deserve attention. The `.null` case omits the `payload` +key entirely on encode. This matches how Rust serde omits content for a +unit variant. A decoder that expected a `payload` key here would fail +against a Rust-produced null. The `.timestamp` case converts a `Date` to +whole epoch seconds as an `Int64`. It guards the conversion first. +`Int64(_:)` traps the program on a `Double` that is not finite or is +outside `Int64`'s range. A corrupt or maliciously crafted inbound record +could carry such a value. The guard turns a potential crash into a +normal, catchable `EncodingError` instead. + +`FingerprintWire` is a Codable wrapper around +`SubstrateTypes.Fingerprint256`. It exposes that type's four 64-bit blocks +directly, for the same reason `PackedHLC` exists. The underlying substrate +type is not itself declared Codable in a form guaranteed stable across +releases, so the wire format owns its own copy of the shape. + +## ConvergenceKitNone.swift + +This file provides `NoSyncEngine`, the passthrough backend, and its +private `StateActor`. + +`NoSyncEngine` conforms to `SyncEngine` with the smallest possible bodies. +`enable(manifest:storage:)` records that sync is on and remembers the +manifest, only for the sake of reporting a `SyncState`. `push()` and +`pull()` each check that `enable` was called, then return +`SyncReceipt.empty`. `subscribe()` returns a stream that never emits +anything. It closes only when the caller cancels its own task. There is +nothing here to synchronize against, so the state lives in `StateActor`, a +Swift actor. It exists so that `enable`, `disable`, and `isEnabled` reads +and writes stay safe under concurrent calls. ConvergenceKit needs no +locking code of its own to guarantee that safety. + +This backend exists so that development builds, unit tests, and genuinely +single-device deployments can enable the same `SyncManifest`-driven code +path as a production backend. They do this without paying for a real +remote connection, and without writing separate no-sync branches +throughout the application. + +## CKRecordMapping.swift + +This file provides `CKRecordMapping`, the generic translator between a +PersistenceKit row and Apple's `CKRecord`, and the two result types it +produces. + +The mapping is generic rather than hand-written per table. A CloudKit +record type and its fields derive entirely from data already known at +runtime: the table's name and the manifest's `kitID`. +`recordType(kitID:table:)` builds the CloudKit record type string as +`"\(kitID)_\(table)"`. `recordID(rowKey:zone:)` builds a `CKRecord.ID` from +the row's UUID and the zone. Both functions are pure string and identifier +arithmetic. Adding a new synced table to a manifest never requires new +mapping code in this file. + +`record(from:table:rowKey:hlc:schemaVersion:kitID:zone:)` converts a row's +values into a `CKRecord`. It then adds three reserved fields: `_syncHLC`, +`_syncSchemaVersion`, and `_syncKitID`. These fields carry ConvergenceKit's +own metadata inside an otherwise ordinary CloudKit record. The private +`assign(value:to:forKey:)` helper switches over every `TypedValue` case. +The `.fingerprint` case packs the four 64-bit blocks into thirty-two bytes +of little-endian `Data`. The `.array` case throws +`SyncError.encodingFailure`, because `CKRecord` has no native +representation for a nested list of typed values at this version. + +`decode(_:)` reverses the process. It reads the three reserved fields back +out. It rejects a record missing `_syncHLC` or `_syncSchemaVersion` as a +decoding failure. A record without sync metadata cannot be attributed to +any schema version. It also cannot be conflict-resolved correctly. The +function then recovers the original table name from the `kitID_table` +record type, by splitting on the first underscore. The `rowKey` recovery +is the one place this file enforces a hard rule, explained in its own +comment. If `record.recordID.recordName` does not parse as a `UUID`, the +function throws `SyncError.corruptRemoteIdentity` rather than inventing a +fresh UUID. A fabricated identity would silently create a new local row +every sync round, because the fabricated UUID could never match the real +one. Refusing to guess turns a slow, invisible data leak into a single +visible error the caller can log and skip. + +`packed(_:)` and `unpacked(_:)` convert an `HLC` to and from a single +`Int64` for storage in a CloudKit `NSNumber` field. They use a fixed bit +layout: forty-eight bits of physical time, twelve bits of logical count, +and four bits of node identifier. This layout is a contract, not an +implementation detail. Any change to the bit widths would make previously +stored `_syncHLC` values unreadable. + +`SyncMeta` and `DecodedRecord` are the two small result types `decode(_:)` +returns. `SyncMeta` isolates the three reserved fields. `DecodedRecord` +pairs them with the table name, row key, and clean application values, +with no `_sync*` keys. Keeping the reserved fields out of `values` matters, +because `values` round-trips directly into PersistenceKit's own row-value +type. That type should never see ConvergenceKit's internal bookkeeping +fields mixed in with application data. + +## CloudKitSyncEngine.swift + +This file provides `CloudKitSyncEngine`, the CloudKit-backed `SyncEngine` +conformance, and its private `CloudKitStateActor`. + +`CloudKitSyncEngine.init(containerIdentifier:)` accepts an optional +CloudKit container identifier, or `nil` to resolve `CKContainer.default()` +later. Resolution is deliberately deferred to `enable()` rather than done +in `init`. This lets a test construct the engine without an iCloud +entitlement configured. Only calling `enable()`, `push()`, or `pull()` +actually touches CloudKit. + +`CloudKitStateActor.enable(manifest:storage:)` creates the estate's +CloudKit zone. It tolerates a "zone already exists" failure as expected, +not as an error. Then, for every manifest table that is not `pullOnly`, it +subscribes to `storage.observer.observe(table:events:)` and forwards every +observed `TableChange` into `pendingOutbound`. `OVERVIEW.md` describes this +mechanism. It turns an ordinary PersistenceKit write into a queued outbound +sync record. The application need not call anything sync-specific for +that to happen. + +`push()` drains `pendingOutbound`. For each change, it looks up the +table's `SyncedTable` in the manifest. It skips tables the manifest does +not declare, or has marked `pullOnly`. For an insert or update it builds a +`CKRecord` via `CKRecordMapping.record`. For a delete it builds only the +`CKRecord.ID` to delete. The HLC used per record prefers one the change +already carries. If the observation carried none, true of the InMemory +and SQLite storage observers at this version, `push()` mints one from an +`HLCGenerator` seeded with a random node ID in `1...0x0F`. It uses +`send(now:)` rather than reading the clock without advancing it, so two +changes minted in the same push never collide on an identical timestamp. +The accumulated saves and deletes go out together in one +`modifyRecords(saving:deleting:savePolicy:atomically:)` call. + +`pull()` fetches everything new since the last stored `serverChangeToken`, +via `recordZoneChanges(inZoneWith:since:)`. It then applies each pulled +record through `applyInbound`. Three checks gate every record before it is +applied. The decoded `kitID` must match the manifest's own `kitID`. The +decoded `schemaVersion` must match. The table must be declared in the +manifest and not marked `pushOnly`. A failure on any of the three checks, +or a decode error, gets caught and logged. The record then counts as a +conflict, rather than aborting the whole pull. One bad record must not stop the rest of the +batch from applying. Deletions arrive as bare `CKRecord.ID` values with no +record type attached. `pull()` therefore cannot know which table a +deleted record belonged to. It attempts the delete against every manifest +table that is not `pushOnly`, using the manifest itself as the only scope +guard. + +`applyInbound(_:syncedTable:storage:)` is the method that actually +enforces `ConflictPolicy`. `.appendOnly` always upserts. It relies on the +row's own key for idempotency. `.lastWriterWinsByHLC` reads back any +existing row's stored `_syncHLC`. Under the InMemory backend this arrives +as `.hlc`. Under SQLite and Postgres it arrives as a raw packed `.int`, +because those schemas do not declare the column's `TypedValue` kind. The +method discards the incoming change silently if its HLC is older. A +change that survives the comparison gets upserted. Its own `_syncHLC`, +`_syncSchemaVersion`, and `_syncKitID` merge back into the row. That +merge is what lets the next inbound write read a comparison point back +out. `.remoteWins` upserts unconditionally. `.localWins` inserts only if +no row with that primary key exists yet. The method is declared with +internal access, not `private` access. It is declared this way specifically so +last-writer-wins conflict tests can call it directly through `@testable +import`, bypassing the real CloudKit network stack. + +## FederationIdentity.swift + +This file provides the identity types federation pairing and signing are +built from: `PeerIdentity`, `LocalIdentity`, and `FederationSignature`. + +`PeerIdentity` is a thin wrapper around a thirty-two-byte Ed25519 public +key, carried for a paired peer. `LocalIdentity.init()` generates a fresh +Ed25519 keypair in memory, using `Curve25519.Signing.PrivateKey()`. Every +`FederationSyncEngine` instance calls this once, at construction, so each +estate has its own signing identity for the session. +`LocalIdentity.init(privateKeyBytes:)` restores an identity from bytes a +caller already has. This suits a host application that wants to persist +and reload the estate's key across launches. This module deliberately +does not persist identity itself. It leaves that decision to the host. +`sign(_:)` produces an Ed25519 signature over arbitrary data, using the +stored private key. + +`FederationSignature.verify(_:of:by:)` checks a signature against a raw +thirty-two-byte public key. It returns `false`, rather than throwing, in +two cases. It returns `false` when the public key bytes do not form a +valid Curve25519 key. It returns `false` when the signature itself does +not match. A receiver treats a malformed peer key exactly like a bad +signature, because both mean the message cannot be trusted. + +## HyperplaneFamilyExchange.swift + +This file provides three Codable value types for the pairing handshake: +`HyperplaneFamilySpec`, `PairingProposal`, and `PairingAcceptance`. + +`HyperplaneFamilySpec` carries a `seed` and a `dimension`, which defaults +to two hundred fifty-six. Two estates that share the same seed can each +independently reproduce the same hyperplane family. That family is the +substrate machinery, defined elsewhere, that makes their 256-bit +fingerprints directly comparable. Sharing only a seed, rather than the +derived family itself, keeps the pairing message small. It also keeps the +two sides' math guaranteed identical, because both sides derive it with +the same deterministic procedure from the same seed. + +`PairingProposal` and `PairingAcceptance` model a handshake between two +roles. A proposer offers a family spec plus a nonce, a value used once to +prevent a replay of an old proposal. An accepter countersigns that +proposal. This file defines only the shapes. As `FederationSyncEngine.swift` +documents at its own `pair(with:via:family:)` method, the shipped v1.0 +pairing path does not yet negotiate or sign these two structs. The caller +supplies an already-agreed `HyperplaneFamilySpec` directly instead. These +types exist ahead of that wiring, so the wire shape is fixed and ready. + +## FederationSyncEngine.swift + +This file provides `FederationSyncEngine`, the Ed25519-authenticated +peer-to-peer `SyncEngine` backend. It also provides the envelope format +its messages travel in. It provides the `Relay` transport abstraction and +its in-process implementation. It provides `FederationStateActor`, where +the actual protocol logic lives. + +`PayloadKind` is a one-byte discriminator identifying what an envelope +carries. `.syncRecordBatch` (`0x01`) is the only kind that exists at +v1.0. `.fieldWriteEventBatch` (`0x02`) is reserved in a code comment for a +future payload. It must never be reassigned to anything else, because a +receiver on an older version would silently misinterpret a reused byte +value. + +`envelopeSigningBytes(senderPublicKey:payloadKind:payload:hlc:)` builds +the exact byte sequence a `SignedEnvelope`'s signature covers. The +sequence starts with the sender's public key, thirty-two bytes long. +Next comes the one-byte payload kind, then a four-byte little-endian +payload length, then the payload itself. Last comes the HLC's three +fields, each in a fixed little-endian layout. Signing this constructed +sequence, rather than the raw JSON payload, closes what the code calls +the relabel and replay seam. If the signature only covered the +payload bytes, an attacker who obtained a valid signed payload could +re-wrap it with a different sender key or payload kind. That re-wrapped +message would then re-verify as something it never was signed to mean. +The exact byte layout matters, because the Rust port's +`envelope_signing_bytes` function must reproduce these bytes exactly. A +signature made on one port would otherwise fail verification on the +other. + +`SignedEnvelope` is the wire message this signature protects: the +sender's public key, the payload kind, the opaque payload, the signature, +and a batch-level HLC. The batch HLC is minted after every record's own +HLC, in `push()`, so it is strictly later. A reader can trust the +envelope's HLC as a batch watermark distinct from any single record's HLC +inside it. + +`Relay` is a two-method protocol, `send(to:message:)` and `drain(for:)`, +that abstracts the transport a `SignedEnvelope` travels over. +`FederationRelay` is the only conformer shipped at v1.0. It is an +in-process, lock-protected dictionary from a recipient's public key to a +list of pending envelopes. It exists so the full pairing, signing, and +conflict-resolution protocol can be exercised in tests without a real +network. It also lets a future hosted relay, a `SyncServer`, conform to +the same protocol as a drop-in replacement. That replacement need not +touch `FederationSyncEngine` itself. + +This file provides `FederationStateActor`. It implements four operations: +enable, pairing, push, and pull. It is a Swift actor. This keeps its +mutable state safe under concurrent access, without manual locking code. +That state includes peers, pending outbound changes, and subscribers. +`enable(manifest:storage:)` mirrors +`CloudKitStateActor.enable`. It subscribes to `storage.observer.observe` +for every push-eligible table and appends observed changes to +`pendingOutbound`. `disable()` differs from the CloudKit actor in one +deliberate way. It cancels every observer task, and then awaits each +one's completion, before it clears state. It does not merely cancel. The +code comment explains why. Without awaiting, a change already in flight +inside a cancelled task could still land in the outbox after `disable()` +returned. That would violate the guarantee that disabling sync stops +capturing writes immediately. + +`pair(with:via:family:)` registers the peer's public key, relay, and +family spec locally. It then calls `acceptPeering` on the peer's own +actor, so the relationship gets recorded symmetrically on both sides from +one call. A test or a caller need not call `pair` twice. Pairing here is a +local bookkeeping step. It is not yet the signed +`PairingProposal`/`PairingAcceptance` handshake `HyperplaneFamilyExchange.swift` +defines the shapes for. + +`push()` converts every pending `TableChange` into a `SyncRecord`, minting +an HLC the same way `CloudKitStateActor.push()` does, and for the same +reason. It JSON-encodes the batch, builds and signs an envelope, and +hands it to every paired peer's relay. `pull()` drains each peer's relay +inbox. For every envelope, it runs four checks in order before trusting +its contents. First, the sender's public key must match the specific peer +it arrived from, not merely any known peer. The code comment cites +ADR-013 here, since a valid self-signature alone does not prove the +pairing handshake was completed. Second, the payload kind must be a known +one. Third, the signature must verify over the reconstructed canonical +bytes. Only then, fourth, is the JSON payload decoded. A record that +passes all four checks still faces individual checks after that. It must +match the manifest's `kitID` and `schemaVersion`. Its table must appear +among the manifest's declared tables, exactly as in +`CloudKitSyncEngine.pull()`. Only a record that clears every check reaches +`applyInbound`. + +`applyInbound(_:syncedTable:storage:)` implements the same four +`ConflictPolicy` arms as `CloudKitStateActor.applyInbound`. It extends +them to cover delete events explicitly. `CloudKitSyncEngine` receives +deletes as bare CloudKit record IDs, with no policy dispatch. Federation's +`SyncRecord` always carries an explicit event kind, so its delete path can +apply a policy the same way inserts and updates do. Under +`.lastWriterWinsByHLC`, a stale delete is silently rejected. A stale +delete is one whose HLC is older than the row's stored `_syncHLC`. This +avoids removing a row a peer has since updated. `.appendOnly` rejects every +remote delete outright, because an append-only table is write-once by +definition. `.remoteWins` deletes unconditionally. `.localWins` rejects +every remote delete, leaving local state authoritative. The method is +internal, not `private`, for the same testing reason as its CloudKit +counterpart. + +## Rust Port and Conformance + +The `rust/` directory contains a second implementation of most of this +package. It covers the core types, the wire format, and the protocol +trait. It also covers the None and Federation backends, ported to Rust. +These live in `rust/src/types.rs`, `record.rs`, `engine.rs`, `none.rs`, +`federation.rs`, and `pairing.rs`. CloudKit is Apple-only and has no Rust +equivalent by design. The Swift package alone handles that transport, as +`rust/README.md` and `rust/src/lib.rs` both state explicitly. + +The Rust and Swift Federation backends are gated by shared behavior. They +are not gated by a byte-fixture file, the way some other SDK packages +are. Both ports implement observer-driven outbox population. Both +implement the same four `ConflictPolicy` arms, in `apply_record` and +`applyInbound`. Both implement identical Ed25519 envelope signing bytes, +in `envelope_signing_bytes` and `envelopeSigningBytes`. The Rust test +suite lives in `rust/tests/`. It exercises wire-format round trips, +last-writer-wins ordering, and inbound event routing per conflict policy. +Each test is described in `rust/README.md` as mirroring a named Swift test +file. Update both ports together whenever you change conflict resolution, +the envelope signing byte layout, or a `SyncRecord` field name in either +one. Then re-run both test suites. The JSON field names and the signing +byte layout are the contract between them. diff --git a/packages/kits/ConvergenceKit/docs/INTERFACE_DOCTRINE.md b/packages/kits/ConvergenceKit/docs/INTERFACE_DOCTRINE.md deleted file mode 100644 index 0a3d9e0..0000000 --- a/packages/kits/ConvergenceKit/docs/INTERFACE_DOCTRINE.md +++ /dev/null @@ -1,115 +0,0 @@ -# ConvergenceKit Interface Doctrine - -For coding agents implementing kits that enable sync on their PersistenceKit instance. ConvergenceKit's contract is with PersistenceKit; downstream kits do not call ConvergenceKit directly except to declare and enable it. - -## 1. Sync is enabled at the application layer, not the kit - -The kit does not own its sync configuration. The application that composes the kit graph decides whether to sync, which backend, and which zone identifier. Kits declare the manifest of what *would* sync if sync were enabled; the application picks a SyncEngine and calls `enable`. - -```swift -// In the kit -public extension LocusKit { - public static func syncManifest(estateID: UUID) -> SyncManifest { - SyncManifest( - kitID: "LocusKit", - schemaVersion: 1, - zoneIdentifier: "LocusKit-\(estateID.uuidString)", - tables: [ - SyncedTable(name: "drawers", primaryKeyColumn: "row_id"), - SyncedTable(name: "tunnels", primaryKeyColumn: "tunnel_id"), - SyncedTable(name: "audit_events", primaryKeyColumn: "event_id", - conflictPolicy: .appendOnly) - ] - ) - } -} - -// In the application -let locusKit = LocusKit(storage: storage) -let sync = CloudKitSyncEngine() -try await sync.enable(manifest: LocusKit.syncManifest(estateID: estate.id), - storage: storage) -``` - -The kit never wires its own sync. The application does. - -## 2. Choose ConflictPolicy per table, not per kit - -Different tables have different conflict semantics. The audit log is append-only (`.appendOnly`); substrate noun tables project from the audit log and tolerate `.lastWriterWinsByHLC`. Queue jobs claim by HLC; settings might be `.localWins`. Per-table choice is the kit author's responsibility. - -```swift -SyncedTable(name: "audit_events", primaryKeyColumn: "event_id", - conflictPolicy: .appendOnly) -SyncedTable(name: "settings", primaryKeyColumn: "key", - conflictPolicy: .localWins) -SyncedTable(name: "jobs", primaryKeyColumn: "job_id", - conflictPolicy: .lastWriterWinsByHLC) -``` - -## 3. Sync direction is declarative - -Most tables are `.bidirectional`. Push-only (`.pushOnly`) is for tables a device emits but never receives (logs, telemetry). Pull-only (`.pullOnly`) is for tables a device consumes but never writes (read-only state from a central source). The kit declares; the application doesn't override per-call. - -## 4. Schema version must match across peers - -If two devices run different schema versions of the same kit, sync between them rejects with `SyncError.schemaMismatch`. The receiver queues the record (or drops it, depending on backend) until both sides upgrade. Document this for the application that orchestrates updates. - -Schema version is bumped whenever PersistenceKit's `SchemaDeclaration.version` is bumped. The two numbers must agree. - -## 5. The audit log uses appendOnly - -The audit log table that GeniusLocusKit owns has `(event_id, hlc)` as its compound primary key. ConvergenceKit's `.appendOnly` policy maps to an idempotent upsert on this key. Duplicate appends from sync replay are no-ops at the storage layer; the CRDT property holds. - -Other kits with append-only logs (queue audit, federation handshake history) use the same policy. - -## 6. The application catches SyncError - -ConvergenceKit operations throw `SyncError` for backend-attributable failures. The application decides retry policy. The kit does not see these errors unless the application surfaces them. - -```swift -do { - _ = try await sync.push() -} catch SyncError.transportFailure(let detail) { - // queue for retry; show offline indicator -} catch SyncError.schemaMismatch { - // prompt user to update the app -} -``` - -## 7. Subscribe for live updates, push/pull for one-shots - -`subscribe()` returns a long-running stream that fires events as sync activity happens. Use it for UI bindings ("syncing now," "last synced 30 seconds ago") and for waking up watchers that need notification when remote work arrives. - -`push()` and `pull()` are one-shots. Use them for explicit refresh ("pull on app foreground," "push before close") or in test paths where determinism matters. - -The two patterns coexist; subscribe stays open while push/pull run. - -## 8. CloudKit and Federation can run side by side - -A single PersistenceKit instance can have both a ConvergenceKit-CloudKit engine and a ConvergenceKit-Federation engine enabled simultaneously. The manifests pick different zones / peer sets; the engines observe the same StorageObserver independently. Multi-backend deployments (an app syncs its entities via CloudKit between a user's devices AND federates a GeniusLocus estate with a partner via ConvergenceKit-Federation) are natively supported. - -The cost is two observer subscriptions per table. Document the resource use in the application. - -## 9. Federation pairing is out of band - -Pairing two estates over Federation requires exchanging public keys plus the HyperplaneFamilySpec. At v1.0 this is in-process (`engineA.pair(with: engineB, via: relay, family:)`). Cross-machine pairing is a v1.x concern; expect QR code, NFC, or AirDrop as the eventual out-of-band channel. - -The kit does not initiate pairing. The application orchestrates it (pairing flow in the UI), and the kit's sync manifest just declares what flows once pairing is in place. - -## 10. Test against ConvergenceKitNone in CI - -CI test suites for kits with sync declarations use ConvergenceKitNone unless the test specifically exercises sync semantics. ConvergenceKitNone validates the manifest, succeeds on enable/disable, and returns empty receipts on push/pull. Fast and deterministic. - -For sync-specific tests use ConvergenceKitFederation in-process pairing (two engines, shared relay, real round-trip). For full CloudKit integration tests, gate on `CLOUDKIT_TEST_CONTAINER` and provision a test container per the project's CI setup. - -## 11. When in doubt, file a decision record - -If you find yourself wanting to: - -- Add a case to `SyncDirection`, `ConflictPolicy`, `SyncEvent`, `SyncState`, or `SyncError` -- Add a new method to `SyncEngine` -- Bypass schema checking -- Sync something that isn't a PersistenceKit row (large blob, file, stream) -- Add a backend-specific escape hatch - -Stop. Write a decision record in `docs/decisions/` proposing the change. The closed-enum design depends on every change being deliberate. diff --git a/packages/kits/ConvergenceKit/docs/OVERVIEW.md b/packages/kits/ConvergenceKit/docs/OVERVIEW.md new file mode 100644 index 0000000..5968ded --- /dev/null +++ b/packages/kits/ConvergenceKit/docs/OVERVIEW.md @@ -0,0 +1,168 @@ +--- +doc: OVERVIEW +package: ConvergenceKit +repo: moot-system +authored_commit: 909513d0a8ecb1e9e903af9f4d25b3e4f2528242 +authored_date: 2026-07-04 +sources: + - path: Sources/ConvergenceKit/SyncEngine.swift + blob: 32e69e0ea5002d8b699866183371527f675023cd + - path: Sources/ConvergenceKit/SyncRecord.swift + blob: 8c29f63ea9f965f1ea75d3bbff71632f76699233 + - path: Sources/ConvergenceKit/SyncTypes.swift + blob: 52ed3cb07f0483d0e541b86a8da080791402d132 + - path: Sources/ConvergenceKitCloudKit/CKRecordMapping.swift + blob: 9b61903296533f8044bee5a4f93ca2dc1b76b653 + - path: Sources/ConvergenceKitCloudKit/CloudKitSyncEngine.swift + blob: 7dd43ccf5454739a3d1ae1cda0870cf507c479c6 + - path: Sources/ConvergenceKitFederation/FederationIdentity.swift + blob: b64358fe36fd241049e9f5bdbbb335f1f3d29464 + - path: Sources/ConvergenceKitFederation/FederationSyncEngine.swift + blob: 9bf722f8e7fc24ae12ac72d571c135da2b40af62 + - path: Sources/ConvergenceKitFederation/HyperplaneFamilyExchange.swift + blob: 4880e847918c253f3d99a02fdd65c15f172d32e1 + - path: Sources/ConvergenceKitNone/ConvergenceKitNone.swift + blob: 5d3556a08c8bc80ec6b824ceb4e0838db9c95ecc +--- + +# ConvergenceKit Overview + +## What This Library Does + +ConvergenceKit copies rows between devices and between estates. PersistenceKit +is the package that stores those rows on one device. An estate is one user's +complete memory store in MOOTx01. Estates can federate. Federation means two +separate estates share and compare memories. + +ConvergenceKit sits beside PersistenceKit. It watches for changes there. It +ships each change somewhere else. The destination is one of two places. +The first is another of the user's own devices, through Apple's CloudKit. +The second is a different user's estate, through a peer-to-peer exchange. +The SDK calls this second path federation. + +Application code never calls ConvergenceKit's sync methods for ordinary reads +and writes. It writes to PersistenceKit as usual. A host configures +ConvergenceKit once, alongside PersistenceKit, to observe that same storage. +After that, replication is a side effect of normal writes. It is not a task +the application must remember to perform. + +## The Problem It Solves + +A memory captured on one device is useless on another device until it +travels there. Two hard problems stand between "captured here" and +"visible there." + +The first problem is conflicting writes. A row can change on two devices +before they synchronize. Something must then decide which change wins. +ConvergenceKit answers this with a per-table conflict policy. The application +declares this policy once, in advance. The default policy is +`lastWriterWinsByHLC`. It settles conflicts with a Hybrid Logical Clock +(HLC). An HLC is a timestamp. It combines a device's wall-clock reading with +a small counter. This lets two devices order their changes consistently, +even when their clocks are not perfectly synchronized. Three other policies +cover cases where last-writer-wins is wrong. `appendOnly` suits audit logs +that must never overwrite an entry. `localWins` suits data a device should +never let a peer overwrite. `remoteWins` covers the opposite case. + +The second problem is trust. Federated sync crosses a perimeter between two +separate people's estates. A receiving device must be able to prove two +things about an incoming batch of changes. It must prove the batch came from +the peer it paired with. It must prove no one altered the batch in transit. +ConvergenceKit answers this with Ed25519 signatures. Ed25519 is a public-key +signature scheme. Each estate generates a private key it never shares. Each +estate also hands out a public key to peers during pairing. Every batch of +changes is signed with the sender's private key before it leaves the device. +The receiver checks that signature with the sender's public key before it +applies anything. + +CloudKit sync does not need this second protection. It stays inside one +person's Apple account. Apple's own private database already authenticates +it. Federated sync crosses to a stranger's device instead. It needs its own +authentication, independent of any platform account system. + +## How It Works + +ConvergenceKit defines one protocol, `SyncEngine`. Three backends conform to +it: `NoSyncEngine`, `CloudKitSyncEngine`, and `FederationSyncEngine`. An +application picks exactly one backend per PersistenceKit instance. All three +backends expose the same four operations. Application code that calls them +does not need to know which backend is active. + +`enable(manifest:storage:)` turns sync on. A `SyncManifest` is a +declaration. The application writes it once. It states which +PersistenceKit tables to sync. It states the direction for each table: +`bidirectional`, `pushOnly`, or `pullOnly`. It states the conflict policy +for each table. Enabling a backend also starts it watching +PersistenceKit's `StorageObserver`. It watches for local inserts, updates, +and deletes on every table the manifest lists as push-eligible. + +`push()` sends pending local changes outward. It returns a `SyncReceipt` +summarizing what moved. `pull()` fetches pending remote changes. It applies +them through PersistenceKit's own write path. This is what makes the +receiving side work correctly. An applied row is not a special sync +artifact. It is an ordinary row. Anything already watching PersistenceKit +notices the change the normal way. `subscribe()` returns a live stream of +`SyncEvent` values for a user interface to display. These events cover +changes applied, pushes completed, peers connecting, peers disconnecting, +and errors. + +Every backend that crosses a real network boundary needs a wire format for +a changed row. `SyncRecord` is that format. It carries a table name, an +event kind, a row key, the changed column values, an HLC, a schema +version, and a kit identifier. The schema version and kit identifier let a +receiver reject a record it does not understand. This is safer than +misapplying that record. `SyncRecord` is Codable with explicit coding +keys. The SDK maintains a Rust port of large parts of the system. Both +ports must produce the same JSON field names for the same data. + +## How the Pieces Fit + +Figure 1 shows the library's topology. The shared protocol and wire format +sit at the center. PersistenceKit sits on either end. The three backends +appear as interchangeable implementations in between. + +![Figure 1. Topology of ConvergenceKit](topology.svg) + +*Figure 1. Topology of ConvergenceKit. Application code writes only to +PersistenceKit. Whichever backend is enabled observes those writes and ships +them outward. On the receiving side, applying an inbound change routes back +through PersistenceKit's normal write path. Dashed regions mark the three +interchangeable backends and the external systems each one talks to: CloudKit's +private database and the federation relay.* + +`NoSyncEngine` is the simplest backend. It accepts `enable()`. It then +returns empty receipts from `push()` and `pull()` forever. It exists for +single-device deployments, development, and tests. Wiring in a real backend +in those cases would add cost without adding value. + +`CloudKitSyncEngine` replicates rows through Apple's CloudKit private +database, using one CloudKit zone per estate. `CKRecordMapping` is the piece +that makes this backend generic instead of hand-written per table. It +converts any PersistenceKit row into a `CKRecord`, and back again, driven +entirely by the table name and the manifest. Adding a new synced table +never requires new mapping code. + +`FederationSyncEngine` replicates rows to a different user's estate. Two +engines "pair" by exchanging Ed25519 public keys. They also exchange a +shared `HyperplaneFamilySpec`. This is a parameter that lets both sides +compare certain 256-bit fingerprints on equal terms after pairing. A +fingerprint here is a short, fixed-size code computed from content. Once +paired, every pushed batch of `SyncRecord` values gets wrapped in a +`SignedEnvelope`. The engine signs that envelope and hands it to a `Relay`. +At v1.0, the shipped `Relay` implementation is `FederationRelay`. It is +in-process only. It holds pending messages in memory for peers running in +the same process. This is enough to exercise the full protocol in tests, +including signature verification and conflict resolution. A wire transport +that reaches a peer on a different machine is future work. + +## What Ships in the Package + +The package ships four Swift targets. `ConvergenceKit` holds the protocol, +the wire format, and the shared types. `ConvergenceKitNone`, +`ConvergenceKitCloudKit`, and `ConvergenceKitFederation` hold the three +backends. A `rust/` port mirrors the core types, the wire format, and the +None and Federation backends. CloudKit is Apple-only by design, so it has no +Rust equivalent. The Swift side always handles that transport. Every field +name shared between the two ports is pinned in code comments. Tests on both +sides exercise those names, because a receiver on one port must decode a +record written by the other exactly. diff --git a/packages/kits/ConvergenceKit/docs/topology.svg b/packages/kits/ConvergenceKit/docs/topology.svg new file mode 100644 index 0000000..1f65269 --- /dev/null +++ b/packages/kits/ConvergenceKit/docs/topology.svg @@ -0,0 +1,113 @@ + + + + + + + + + + + + + ConvergenceKit: PersistenceKit rows in, replicated rows out + + + + Application code + writes via PersistenceKit + + + PersistenceKit (local) + Storage · RowStore · StorageObserver + + + SyncEngine + enable · push · pull · subscribe + + + Remote + CloudKit database or peer's PersistenceKit + + + + + + push + + pull + + + + apply inbound via RowStore -> wakes local watchers + + + + Backends - one enabled per PersistenceKit instance; each conforms to SyncEngine + + + ConvergenceKitNone + passthrough; empty receipts + + + ConvergenceKitCloudKit + CKRecordMapping <-> CKRecord + + + ConvergenceKitFederation + Ed25519 SignedEnvelope + + + + + + + + + External systems (v1.0 transport) + + + CloudKit private database + Apple iCloud, per-estate zone + + + FederationRelay + in-process inbox (v1.0) + + + + + + + Cross-machine relay transport and the signed pairing handshake (HyperplaneFamilyExchange) are v1.x scope. + diff --git a/packages/kits/PersistenceKit/docs/AGENT_MAP.md b/packages/kits/PersistenceKit/docs/AGENT_MAP.md new file mode 100644 index 0000000..28dc85b --- /dev/null +++ b/packages/kits/PersistenceKit/docs/AGENT_MAP.md @@ -0,0 +1,363 @@ +--- +doc: AGENT_MAP +package: PersistenceKit +repo: moot-system +authored_commit: 909513d0a8ecb1e9e903af9f4d25b3e4f2528242 +authored_date: 2026-07-04 +sources: + - path: Sources/PersistenceKit/AuditLog.swift + blob: ca4c0c9623056a49ad888a7a77e720a7519556bb + - path: Sources/PersistenceKit/BlobStore.swift + blob: f052c3ecc693d5414d57ad2c6da5fb4c5fe28f79 + - path: Sources/PersistenceKit/CacheInvalidator.swift + blob: 0a844b8037404dd72d08df6887ceee1e8c6014f2 + - path: Sources/PersistenceKit/CachingRowStore.swift + blob: 7edf9a86eb31dd9a17abd97b7baa9e8d5425e266 + - path: Sources/PersistenceKit/Column.swift + blob: 3cfc3ba856ebe15e8756ae65dae736cbd2c288a6 + - path: Sources/PersistenceKit/EncryptionMode.swift + blob: fdb7aa7ec63088ae09739f5e2a8e1468c386b987 + - path: Sources/PersistenceKit/ErasureLedger.swift + blob: eb2ca35f69783eead6049949bbaad68decfc9915 + - path: Sources/PersistenceKit/ErasureOverlay.swift + blob: d5a86f2baa10647134f285d6996a8e62cbe23ace + - path: Sources/PersistenceKit/EstateCacheConfig.swift + blob: 7003951cb5b2b075b6bb0c49b5fe372a97bfeb3a + - path: Sources/PersistenceKit/EstateConfiguration.swift + blob: be91569405e5c48a99859b4595540a59bd1a1994 + - path: Sources/PersistenceKit/GCPin.swift + blob: 73336dc4c693fcfa285a7eff8ed6520f27951ff8 + - path: Sources/PersistenceKit/GeneratedColumn.swift + blob: 34e466c80a0e9aac5d24303eaad132901333470b + - path: Sources/PersistenceKit/HashingRowStore.swift + blob: fe233ed2f4fda177373ad105d14fa321d325d0df + - path: Sources/PersistenceKit/NoOpObserver.swift + blob: 99f62b6889df50c5f09fce749f363637891f116c + - path: Sources/PersistenceKit/NovelTokenTaggerChoice.swift + blob: 9588d6303431883a2c8e0ee727f7ebe3e3f6d139 + - path: Sources/PersistenceKit/PersistenceKitTelemetry.swift + blob: a24e54dfb97beb86dac9422bbfc7cc3c2d959d73 + - path: Sources/PersistenceKit/Predicate.swift + blob: 522802c842dfc7573137efd7a3f36300d5201468 + - path: Sources/PersistenceKit/RowCrypto.swift + blob: 6678e426c0902cdae6d164904001526756e40916 + - path: Sources/PersistenceKit/RowStore.swift + blob: 9a96ff89528ca786d51224324cacb78cf810037d + - path: Sources/PersistenceKit/Schema.swift + blob: 5a6894c68d6f29eecfe9acc34d7c06ee2a655ac3 + - path: Sources/PersistenceKit/SnapshotRegistry.swift + blob: 88087bda544165c4d24c514a4f3e0641a620512e + - path: Sources/PersistenceKit/Storage.swift + blob: 7484a40913b28cb11a6bd2e3ea822dc8fe8eb63e + - path: Sources/PersistenceKit/StorageError.swift + blob: 743c2d1a24c7bafedb217e4c6bbf30ca20d03be8 + - path: Sources/PersistenceKit/StorageIntrospection.swift + blob: 35522b6601037246907bb88ebb2fb2eb5ea9b0e2 + - path: Sources/PersistenceKit/StorageObserver.swift + blob: f0a8b61a15344e02193137a3393be345dd51cf25 + - path: Sources/PersistenceKit/Transaction.swift + blob: d58d478618b31cc3985275be6ceee84a6fc222de + - path: Sources/PersistenceKit/TypedValue.swift + blob: ec124b7aebd86f64f67fea6656396d374666b741 + - path: Sources/PersistenceKitInMemory/InMemoryAuditLog.swift + blob: 1ba83d2408d3384d4d8e4f286fa2db743a433cc9 + - path: Sources/PersistenceKitInMemory/InMemoryBlobStore.swift + blob: e02822bde83c899b6d18ab8d82b90afba6abb5ac + - path: Sources/PersistenceKitInMemory/InMemoryObserver.swift + blob: 71b98c9da2d869a81a78265b7ef1f64a57a907af + - path: Sources/PersistenceKitInMemory/InMemoryRowStore.swift + blob: 2f7c92612c46b8a097e4211147c924f8356e47d1 + - path: Sources/PersistenceKitInMemory/InMemoryStorage.swift + blob: 9820c771bfe39ab738eea1f3b1623491fcfb1326 + - path: Sources/PersistenceKitInMemory/PredicateEvaluator.swift + blob: e8735422ff87addff6a2a3a89da85c495d07f07c + - path: Sources/PersistenceKitPostgreSQL/PostgreSQLConnection.swift + blob: 5a282a6064a69a543a5d650ccc7eeeae2c5a3e4f + - path: Sources/PersistenceKitPostgreSQL/PostgreSQLIdentifierValidator.swift + blob: 120cf0c1576a7db45d79bf6865e2f15cff09a5ac + - path: Sources/PersistenceKitPostgreSQL/PostgreSQLPool.swift + blob: eddc0c3af4135565e15d2d763b0d24240973dd6f + - path: Sources/PersistenceKitPostgreSQL/PostgreSQLPredicateCompiler.swift + blob: 6f0791e0372b09cf2605bae4cf149e48bbba2834 + - path: Sources/PersistenceKitPostgreSQL/PostgreSQLSchema.swift + blob: f165f0877b96c87e2f7de46012f8f8acb3c03cc8 + - path: Sources/PersistenceKitPostgreSQL/PostgreSQLStorage.swift + blob: 849926f9e6def788fae2f38757ad44bcd52a18a0 + - path: Sources/PersistenceKitPostgreSQL/PostgreSQLStores.swift + blob: bff90838d1324d82ae76a0895f15399537e9e343 + - path: Sources/PersistenceKitReplication/IncrementalReplicationSession.swift + blob: 2f90378146e043d75c739d400d98c7770e07af78 + - path: Sources/PersistenceKitReplication/ReplicationTypes.swift + blob: bb13c63d1febd50ad46c1814610d7f6c31a33112 + - path: Sources/PersistenceKitReplication/StorageReplicator.swift + blob: f5acc5993c8647e53c127c6871a1282c24b1c427 + - path: Sources/PersistenceKitSQLite/KeychainKeyStore.swift + blob: 0071732291a7cb6ce0777bd230a6188276fb4f32 + - path: Sources/PersistenceKitSQLite/SQLiteConnection.swift + blob: ece56dc7e25e67656bb37f5222c18c1166c750cc + - path: Sources/PersistenceKitSQLite/SQLiteIdentifierValidator.swift + blob: 713339c137d6af1cfbba5a3584e05bdda70b42c5 + - path: Sources/PersistenceKitSQLite/SQLiteObserver.swift + blob: 2cb61ed75dae5ab81a3cdfb5c9581731d25fd960 + - path: Sources/PersistenceKitSQLite/SQLitePredicateCompiler.swift + blob: 5770adb2024c54ea6921651632ca65ee84d416af + - path: Sources/PersistenceKitSQLite/SQLiteSchema.swift + blob: 4ba6fc175fe9d486b17af1088a23da64041b12ae + - path: Sources/PersistenceKitSQLite/SQLiteStorage.swift + blob: 417e63cc07b60295ac874187ebb796bf9f3b3c86 + - path: Sources/PersistenceKitSQLite/SQLiteStores.swift + blob: 76499ffb70d979f0d13e8cf9e32bc38ff28ffdb5 +--- + +# AGENT_MAP: PersistenceKit + +PURPOSE: storage abstraction layer for MOOTx01 estates. One protocol +surface (Storage = RowStore+BlobStore+AuditLog+StorageObserver), three +interchangeable backends (SQLite/PostgreSQL/InMemory), decorators +(caching, hash-on-write) and cross-cutting toolkits (row encryption, +erasure ledger/overlay, snapshot registry+GC pin, telemetry) built on the +core protocol, plus a full-estate replication module. Owns NO vector +search (VectorKit owns k-NN; PersistenceKit only guarantees the +ACCOMMODATION contract: storage round-trip for vector payloads). + +DEPS: imports SubstrateTypes (HLC, Fingerprint256, AuditEvent, +AsOfCoordinate, ContentHash: defined upstream, NOT in this package), +IntellectusLib (telemetry, zero-cost when disabled), CryptoKit +(AES-GCM), Security/Keychain (Apple key store, SQLite target only), +postgres-nio + NIOSSL (PostgreSQL target only), vendored SQLCipher C +target (SQLite target only). Imported by: every kit that persists estate +data (LocusKit and others up the graph); kits never import a backend +target directly except at estate-construction time. Rust port in +`rust/` mirrors the trait surface and backend set but is NOT +conformance-fixture-gated against the Swift side (no shared byte- +identical corpus): the cross-language contract is the protocol shape +and wire format only. + +ENTRY POINTS (most callers need only these): +- Storage.swift:16 `protocol Storage`: conform once per backend; exposes `.rowStore`/`.blobStore`/`.auditLog`/`.observer` +- Storage.swift:29 `Storage.open(schema:)`: bring backend up to declared schema +- Storage.swift:36 `Storage.transaction(isolation:_:)`: atomic multi-op block; ext. default isolation = .readCommitted +- RowStore.swift:33 `RowStore.insert/upsert/update/delete/query/count`: core row I/O +- EstateConfiguration.swift:8 `struct EstateConfiguration`: one value opens one Storage instance +- StorageReplicator.swift:89 `StorageReplicator.replicate(from:to:schema:)`: full-snapshot estate copy + +## Symbol Table + +### Core protocols: Storage.swift, RowStore.swift, BlobStore.swift, AuditLog.swift, StorageObserver.swift, StorageIntrospection.swift, Transaction.swift +- Storage.swift:16 `protocol Storage: Sendable`: rowStore/blobStore/auditLog/observer + open/close/transaction/migrate/currentSchemaVersion(for:) +- Storage.swift:58 `extension Storage.transaction(_:)`: default isolation .readCommitted +- RowStore.swift:8 `typealias RowKey = UUID` +- RowStore.swift:10 `struct StorageRow`: [String:TypedValue] wrapper, subscript by column name +- RowStore.swift:22 `struct RowHandle: Hashable`: (table, key) +- RowStore.swift:32 `protocol RowStore: Sendable`: insert/upsert/update/delete/query/count/querySkipCorrupt/query(...columns:)/begin·commit·rollbackTransaction (all true requirements, not ext defaults: needed for dynamic dispatch through `any RowStore`) +- RowStore.swift:72 `querySkipCorrupt(...)`: best-effort corpus scan, skips+counts StorageError.corruptStoredValue rows; NEVER use for point lookups +- RowStore.swift:96 `query(...columns:)`: column projection = no-blob read path; nil columns = full read (superset, always correct) +- RowStore.swift:194/214/236 `query/querySkipCorrupt(...asOf:)`: .present passthrough; .asOf(_) throws StorageError.featureGated("asOfQuery") until NT-L4+NT-P3 both merge +- RowStore.swift:260-266 ext defaults: beginTransaction/commitTransaction/rollbackTransaction: no-op (correct for backends w/o multi-stmt txn) +- BlobStore.swift:8 `typealias BlobKey = String` +- BlobStore.swift:10 `protocol BlobStore: Sendable`: put/get/delete/exists/size/listKeys; listKeys() order UNSPECIFIED (caller sorts) +- AuditLog.swift:23 `protocol AuditLog: Sendable`: append/appendBatch (idempotent on (eventID,hlc))/iterate(after:rowID:limit:)/eventsForRow/count +- StorageObserver.swift:26 `enum StorageEvent`: insert|update|delete +- StorageObserver.swift:32 `struct TableChange`: table/event/rowKey/values/hlc +- StorageObserver.swift:60 `enum BlobEvent`: put|delete +- StorageObserver.swift:73 `struct BlobChange`: key/event/bytes (bytes non-nil only on .put; last-write-wins on same key) +- StorageObserver.swift:111 `struct DirtyChainEvent`: changedRowId/parentNodeId/grandparentNodeId/contentHash/table: hash-on-write payload (NT-P2) +- StorageObserver.swift:140 `protocol StorageObserver: Sendable`: observe(table:events:)/observeBlobs()/observeDirtyChain(); delivery AT-LEAST-ONCE, order preserved PER-TABLE only +- StorageObserver.swift:168 ext default `observeDirtyChain()`: immediately-finished stream (back-compat for pre-hash-on-write observers) +- StorageIntrospection.swift:39 `struct StorageStats`: superset of fields across 3 backends; nil = "not measured" (see file's field/backend table); capturedAt is caller-injected (NEVER Date() inside engine) +- StorageIntrospection.swift:209 `protocol StorageIntrospection`: separate from Storage (additive capability); probe via `as? StorageIntrospection` +- Transaction.swift:9 `enum IsolationLevel`: readCommitted|repeatableRead|serializable +- Transaction.swift:15 `protocol StorageTransaction: Sendable`: rowStore/blobStore/auditLog (no nested txns, no savepoints in v1.0) + +### Value/type algebra: TypedValue.swift, Column.swift, Predicate.swift, Schema.swift, GeneratedColumn.swift, StorageError.swift +- TypedValue.swift:25 `enum TypedValue: Sendable, Hashable`: CLOSED 13-case wire format (null/bool/int/bitmap/float/text/blob/uuid/timestamp/json/hlc/fingerprint/array); new case = update every backend, deliberate cost +- Column.swift:9 `struct Column: Comparable`: (table,name); ordered by table then name for stable test fixtures +- Column.swift:24 `enum ColumnType`: uuid|bitmap|text|timestamp|float|int|bool|blob|json|hlc|fingerprint +- Predicate.swift:11 `indirect enum StoragePredicate`: CLOSED; logical(and/or/not/isTrue/isFalse) + comparison(eq/neq/lt/lte/gt/gte/isNull/isNotNull/in/like) + bitmap(bitmaskAll/Any/None/bitwiseEq, Int64 cols only) +- Predicate.swift:41 `StoragePredicate.all(_:)`: AND-combine w/ short-circuit: empty→isTrue, contains isFalse→isFalse +- Predicate.swift:55 `StoragePredicate.any(_:)`: OR-combine w/ short-circuit: empty→isFalse, contains isTrue→isTrue +- Predicate.swift:74 `struct OrderClause`: column + OrderDirection (default .ascending) +- Schema.swift:9 `struct SchemaDeclaration`: kitID/version/tables/indices/migrations +- Schema.swift:31 `struct TableDeclaration`: name/columns/primaryKey/uniqueConstraints/generatedColumns/appendOnly/hashable +- Schema.swift:82 `enum ColumnRole`: createdHlc|tombstonedHlc (as-of temporal filter tagging, ADR-017 §15) +- Schema.swift:90 `struct ColumnDeclaration`: name/type/nullable/defaultValue/role +- Schema.swift:128 `struct Migration`: fromVersion/toVersion/[SchemaOperation] +- Schema.swift:140 `enum SchemaOperation`: createTable/dropTable/addColumn/dropColumn/renameColumn/addIndex/dropIndex/custom(sqlite:postgresql:) +- Schema.swift:153-209 `ColumnDeclaration` static convenience ctors (.uuid/.bitmap/.text/.timestamp/.int/.float/.bool/.blob/.json/.hlc/.createdHlc/.tombstonedHlc/.fingerprint) +- Schema.swift:214-230 `TableDeclaration.createdHlcColumn/tombstonedHlcColumn/supportsAsOfFilter`: derive as-of eligibility from column roles +- GeneratedColumn.swift:43 `struct GeneratedColumn`: name/type/expression; ALWAYS STORED (PG has no VIRTUAL) +- GeneratedColumn.swift:63 `indirect enum GeneratedExpression`: column/literal/bitAnd/bitOr/bitXor/shiftRight/shiftLeft/equal/notEqual; evaluates to Int64 (bool=0/1) +- GeneratedColumn.swift:91 `renderSQL()`: shared SQLite+PG renderer; XOR emitted as (a|b)-(a&b) (SQLite lacks native XOR op) +- GeneratedColumn.swift:122 `evaluate(_:)`: InMemory direct evaluation against row dict +- GeneratedColumn.swift:150 `integerValue(_:)`: extract Int64 from int-family TypedValue; non-integer/absent → 0 (InMemory sentinel) +- StorageError.swift:5 `enum StorageError: Error, Equatable`: CLOSED; notable cases: :22 corruptStoredValue (NEVER fabricate a default: throw), :28 invalidConfiguration (fail-closed on platform-invalid config), :37 featureGated(feature:) (as-of query gate), :44 invalidIdentifier(name:) (SQL identifier injection guard, SECFIX-WS2-PK) + +### Estate configuration: EstateConfiguration.swift, EstateCacheConfig.swift, EncryptionMode.swift, NovelTokenTaggerChoice.swift +- EstateConfiguration.swift:8 `struct EstateConfiguration`: estateID/backend/encryptionConfig(default .plaintext)/cacheConfig(default .disabled)/novelTokenTagger(default .hmm) +- EstateConfiguration.swift:45 `enum BackendConfiguration`: .sqlite(url:busyTimeout:) | .postgresql(connectionString:poolSize:connectionTimeout:idleTimeout:) | .inMemory +- EstateConfiguration.swift:108 `queueSibling(filename:)`: derives sibling EstateConfiguration for per-estate queue DB; .sqlite→sibling file `.`; .inMemory→sibling inMemory; .postgresql→ throws featureGated (deferred, ADR-021) +- EstateConfiguration.swift:166 `deriveQueueSiblingID(parentID:filename:)` (private): deterministic XOR-fold, NO UUID() call, same parent+filename ⇒ same sibling ID always +- EstateCacheConfig.swift:23 `struct EstateCacheConfig`: enabled/ceilingBytes(clamped ≥0)/sensitivityThreshold(clamped ≤2: Secret=3 NEVER cacheable) +- EstateCacheConfig.swift:53 `EstateCacheConfig.disabled`: zero-change default, cache off +- EncryptionMode.swift:20 `enum EncryptionMode`: .plaintext (Mode1) | .rowEncryption (Mode2, per-row AES-GCM) | .fullDatabase (Mode3, SQLCipher whole-file); Mode4 (DB+threshold/FedRAMP) DELIBERATELY ABSENT: not a build capability +- EncryptionMode.swift:44 `struct EstateEncryptionConfig`: mode/keyIdentifier/`package`-scoped key (SymmetricKey?, never public, never logged) +- EncryptionMode.swift:68 `init(_ mode:)`: mints fresh 256-bit key + UUID keyID for encrypting modes; nil/nil for .plaintext +- EncryptionMode.swift:88 `fullDatabase(key:)`: builds Mode3 config from caller-supplied 256-bit key (Keychain-sourced) +- EncryptionMode.swift:99 `usesRowCrypto`: true ONLY for .rowEncryption (Mode3 has no per-row seam: whole file is ciphertext) +- EncryptionMode.swift:106 `fullDatabaseKeyHex`: lowercase hex for `PRAGMA key`; NEVER logged +- NovelTokenTaggerChoice.swift:43 `enum NovelTokenTaggerChoice: Sendable, Hashable, Codable`: .hmm (default, cross-platform-deterministic, federation-safe) | .nlTagger (Apple-only, non-deterministic across OS versions, federation-INCOMPATIBLE w/o re-tag: enforcement deferred to v1.1); FIXED AT ESTATE CREATION, no change-after-creation in v1.0; mirrors identically-named PersistenceKit-independent enum in LatticeLib: bridge by switch, no shared import +- NovelTokenTaggerChoice.swift:74 `NovelTokenTaggerChoice.default`: .hmm + +### Row-level crypto: RowCrypto.swift +- RowCrypto.swift:61 `protocol AeadProvider` (package): swap point for AEAD algorithm; MUST generate fresh random nonce per encrypt call, MUST return [nonce][tag][ciphertext], MUST throw (never return garbage) on auth failure +- RowCrypto.swift:81 `struct CryptoKitAeadProvider: AeadProvider`: default; AES-GCM-256; :87 nonceByteCount=12, :88 tagByteCount=16 +- RowCrypto.swift:144 `enum RowCrypto` (package): :151 encrypt(_:key:provider:), :168 decrypt(_:key:provider:); default provider = CryptoKitAeadProvider() +- RowCrypto.swift:191 `encryptedForWrite(_:config:provider:)` (package): no-op unless usesRowCrypto + "content" column present as .text; encrypts + stamps keyID +- RowCrypto.swift:220 `decryptedForRead(_:config:provider:)` (package): no-op unless keyID present AND matches config.keyIdentifier (mismatched keyID ⇒ pass through as ciphertext, NEVER attempt decrypt under wrong key) +- RowCrypto.swift:249 `assertContentKeyIDInvariant(_:table:config:)` (package): structural guard (FUP-D/E-1): throws constraintViolation if encrypting estate has .text content with empty/absent keyID (seam didn't run) + +### Caching + hash-on-write decorators: CachingRowStore.swift, CacheInvalidator.swift, HashingRowStore.swift +- CachingRowStore.swift:46 `typealias ParentChainProvider = @Sendable (String, RowKey) -> [RowHandle]`: kit-supplied Merkle ancestor lookup +- CachingRowStore.swift:52 `final class CachingRowStore: RowStore, Sendable`: LRU hot-tier decorator; TRANSPARENCY GUARANTEE: results always identical to unwrapped backing store +- CachingRowStore.swift:164 `query(...asOf:)`: cache key = (RowHandle, AsOfCoordinate); .present entries evicted on write, .asOf(hlc) entries NEVER evicted (immutable, GC-pinned) +- CachingRowStore.swift:218 `invalidate(table:key:)`: external-write invalidation hook, called by CacheInvalidator +- CachingRowStore.swift:302 `extractKey(from:)`: cache lookups ONLY feasible for single-key `.eq(_, .uuid)` predicates; all else bypasses cache +- CachingRowStore.swift:334 `actor CacheActor`: owns entries/accessCounter/totalBytes; makes outer class Sendable w/o manual locks +- CachingRowStore.swift:421 `isAdmissible(_:)` (private, in CacheActor): sensitivity gate: reads `provenance` bitmap bits[30:35] scale-gapped (0/16/32/48→ordinal 0-3); ordinal==3(Secret) ALWAYS rejected; absent column→admit; unparseable/unrecognized→REJECT (fail-closed) +- CachingRowStore.swift:482 `evictLRU()` (private): O(n) min-scan by accessOrder; acceptable at cache-sized N +- CacheInvalidator.swift:34 `final class CacheInvalidator: Sendable`: StorageObserver→CachingRowStore.invalidate bridge for external writers +- CacheInvalidator.swift:52 `init(cache:observer:tables:)`: subscriptions registered before first await inside Task.detached; race note: a write issued in the SAME instant as init() may precede subscription (caller should yield if strict ordering required) +- CacheInvalidator.swift:92 `cancel()` / deinit: cancels root task ⇒ propagates to all per-table child tasks +- HashingRowStore.swift:36 `typealias ContentHashProvider = @Sendable (table:rowKey:values:) -> ContentHash`: kit-injected hash fn (PersistenceKit never imports a hash impl) +- HashingRowStore.swift:49 `typealias HashParentChainProvider = @Sendable (table:rowKey:) -> (parentNodeId:grandparentNodeId:)?` +- HashingRowStore.swift:55 `struct HashOnWriteConfig`: hashableTables/hashProvider/parentChainProvider +- HashingRowStore.swift:81 `final class HashingRowStore: RowStore, @unchecked Sendable`: decorator chain position: caller → HashingRowStore → CachingRowStore → backend +- HashingRowStore.swift:295 `augmentWithHashForKnownKey(table:rowKey:mergedValues:)` (private): UPDATE/upsert-as-update path: hashes FULL merged row (not partial SET dict) per SECFIX-WS2-PK F6; strips stale content_hash before hashing so insert vs update paths agree +- HashingRowStore.swift:330 `augmentWithHash(table:values:)` (private): INSERT path; extracts rowKey from "id" column by convention +- HashingRowStore.swift:369 `emitDirtyChain(...)` (private): delivers DirtyChainEvent via injected ObserverRegistryRef (nil = computed but not delivered) + +### Erasure: ErasureLedger.swift, ErasureOverlay.swift +- ErasureLedger.swift:15 `struct ErasureLedgerEntry`: drawerId + erasedHlc +- ErasureLedger.swift:28 `enum ErasureLedgerTables`: .ledger = "erasure_ledger" +- ErasureLedger.swift:35 `ErasureLedgerSchema.ledgerTable`: TableDeclaration, appendOnly:true (UPDATE/DELETE throw appendOnlyViolation at storage layer) +- ErasureLedger.swift:53 `ErasureLedgerOps.recordErasure(...)`: throws duplicateKey if drawerId already erased (erase-once contract) +- ErasureLedger.swift:68 `ErasureLedgerOps.isErased(...)`: fast point-lookup, hot path for every erasure-subject read +- ErasureLedger.swift:86 `ErasureLedgerOps.lookupErasure(...)`: full entry incl. erasedHlc +- ErasureOverlay.swift:22 `struct ErasureOverlayConfig`: extractErasureId(row)->String? (nil=not erasure-subject) + contentColumns([String]) supplied by calling kit +- ErasureOverlay.swift:64 `ErasureOverlay.apply(rows:config:rowStore:)`: TWO-PHASE FAIL-CLOSED: phase1=query already ran; phase2=per-row ledger check; ledger-check THROW ⇒ row DROPPED entirely (never shown-when-uncertain) +- ErasureOverlay.swift:96 `nullContentColumns(row:columns:)`: nulls listed columns, preserves skeleton (id/hlc/timestamps) + +### Snapshots + GC: SnapshotRegistry.swift, GCPin.swift +- SnapshotRegistry.swift:18 `struct SnapshotId: Hashable`: :24 `.mint()` UUID-backed +- SnapshotRegistry.swift:32 `struct SnapshotRecord`: snapshotId/hlc/label/createdAt +- SnapshotRegistry.swift:47 `struct SnapshotAttestation`: snapshotId/subjectKind/subjectId/merkleRoot/keyVersion?; subject semantics owned by CALLING kit, not PersistenceKit +- SnapshotRegistry.swift:73 `enum SnapshotTables`: .registry="snapshot_registry", .attestations="snapshot_attestations" +- SnapshotRegistry.swift:83/98 `SnapshotSchema.registryTable` / `.attestationsTable`: TableDeclarations +- SnapshotRegistry.swift:130 `SnapshotRegistryOps.createSnapshot(...)`: mints SnapshotId, inserts registry row + N attestation rows +- SnapshotRegistry.swift:161 `.listSnapshots(...)`: HLC ascending +- SnapshotRegistry.swift:177 `.deleteSnapshot(...)`: attestations deleted BEFORE registry row (child-first) +- SnapshotRegistry.swift:201 `.attestations(rowStore:snapshotId:)`: ordered by (subjectKind, subjectId) +- GCPin.swift:22 `GCPin.minimumRetainableHlc(rowStore:)`: MIN(hlc) across all snapshots; nil = nothing pinned = all vacuumable +- GCPin.swift:44 `GCPin.isPinned(rowStore:rowHlc:)`: rowHlc.packed >= minRetainableHlc.packed + +### Telemetry + observer stubs: PersistenceKitTelemetry.swift, NoOpObserver.swift +- PersistenceKitTelemetry.swift:81 `reportStorageStats(_:estateID:now:)`: OFF by default; single Atomic load+branch when Intellectus.isEnabled==false (~1ns, no stats() call); emits `persistence.db.*` namespace, one metric per non-nil StorageStats field; `now` ALWAYS caller-injected (never Date() inline) +- NoOpObserver.swift:10 `final class NoOpObserver: StorageObserver, Sendable`: immediately-finished streams; used directly by PostgreSQLStorage (no live PG notification channel) + +### Backend: PersistenceKitInMemory +- InMemoryStorage.swift:24 `final class InMemoryStorage: Storage, Sendable`: no persistence across process runs; always effectively-serializable isolation (full state snapshot per txn) +- InMemoryStorage.swift:78 `transaction(isolation:_:)`: mutates LIVE actor state directly (NOT a detached-copy-then-replace): replace-on-commit would silently drop concurrent bare inserts landing between snapshot and replace (real incident: 5-10% lost QueueKit sends under burst); rollback restores pre-txn snapshot +- InMemoryStorage.swift:102/110 `beginNotificationBuffering()` / `commitNotifications()`: buffer during txn, flush only on successful commit (SECFIX-WS2-PK F2: rolled-back writes must never reach observers) +- InMemoryStorage.swift:144 `actor InMemoryStateActor`: sole owner of InMemoryState; :462 `materializeGenerated(_:_:)` static: computes GeneratedColumn values identically to SQL backends' STORED columns +- InMemoryStorage.swift:388 `queryRows(...)`: predicate/order/pagination applied to FULL row first, projection applied LAST (matches SQL ORDER-BY-on-unselected-column semantics) +- InMemoryStorage.swift:635 `HLC.packed` (extension): canonical bit-pack: physical<<16 | logical<<4 | node +- InMemoryRowStore.swift:19 `final class InMemoryRowStore: RowStore, Sendable`: thin forward to state actor; column-projection kept for cross-backend StorageRow shape parity, not for a real transfer saving +- InMemoryBlobStore.swift:6 `final class InMemoryBlobStore: BlobStore, Sendable`: forwards to actor blob dict +- InMemoryAuditLog.swift:19 `final class InMemoryAuditLog: AuditLog, Sendable`: forwards to actor; dedup on (eventID,hlc) enforced in actor +- InMemoryObserver.swift:15 `final class ObserverRegistry: @unchecked Sendable`: NSLock-based (NOT actor) so `register` is SYNCHRONOUS: subscription recorded before observe() returns, no race window for immediately-following write; mirrors Rust ObserverHub::subscribe +- InMemoryObserver.swift:104 `notify(_:)`: matching subs snapshotted UNDER lock, yielded OUTSIDE lock (avoids deadlock vs. continuation onTermination which also locks) +- PredicateEvaluator.swift:6 `enum PredicateEvaluator`: in-memory StoragePredicate interpreter (only backend evaluating the tree directly rather than compiling to a query string) +- PredicateEvaluator.swift:61 `likeMatch(_:pattern:)`: SQL LIKE→NSRegularExpression (%→.*, _→.) +- PredicateEvaluator.swift:80 `enum TypedValueComparator`: :81 `compare(_:_:)` shared ordering; .null sorts first; .hlc compared via .packed (single total order) + +### Backend: PersistenceKitSQLite +- SQLiteStorage.swift:22 `final class SQLiteStorage: Storage, Sendable`: one connection per estate +- SQLiteStorage.swift:110 `actor SQLiteBackend`: owns connection + inTransaction flag + pendingBlobNotifications (buffered during txn, SECFIX-WS2-PK F3) +- SQLiteStorage.swift:411 `insertRow` / :461 `upsertRow` / :495 `updateRows` / :535 `deleteRows` / :559 `queryRows`: ALL validate every table/column identifier via `validateSQLIdentifier` before interpolation (SECFIX-WS2-PK F9); insertRow+queryRows run RowCrypto encrypt/decrypt seam +- SQLiteStorage.swift:656 `queryRowsSkipCorrupt(...)`: cursor-level per-row skip+log on corruptStoredValue; other errors re-thrown (systemic) +- SQLiteStorage.swift:782 `storageStats(now:)`: PRAGMA page_size/page_count/freelist_count; WAL frame count derived from `-wal` FILE SIZE (NOT PRAGMA wal_checkpoint: can SQLITE_LOCKED even same-actor) +- SQLiteStorage.swift:917 `readColumn(...)`: type-tolerant decode (valid-but-coerced value passes through) vs. parse-failure (unparseable UUID/timestamp TEXT) → THROWS corruptStoredValue, never fabricates +- SQLiteConnection.swift:24 `final class SQLiteConnection: @unchecked Sendable`: thin sqlite3 C API wrapper +- SQLiteConnection.swift:29 `init(url:busyTimeout:keyHex:)`: ORDER MATTERS: symlink-refusal check (CAND-052, lstat semantics) → sqlite3_open_v2 → PRAGMA key (Mode3, MUST be first stmt) → Apple Data Protection → WAL/synchronous/busy_timeout/foreign_keys pragmas +- SQLiteConnection.swift:162 `final class SQLiteStatement`: prepared-stmt wrapper; :180 `bind(_:at:)` TypedValue→sqlite3_bind_*; :239 `step()` +- SQLiteConnection.swift:318 `enum ISO8601`: :356 `string(from:)` clamps Date to RFC-3339 range [0001,9999] before formatting (logs warning on clamp, never emits unparseable-back string); :392 `date(from:)` tries :412 `fastParseCanonicalUTC` FIRST (allocation-free, handles the exact canonical shape this kit writes: Merkle rollup re-decode was ~80% of CPU via ICU formatter before this fast path), falls back to ISO8601DateFormatter for anything else +- SQLiteStores.swift:22 `final class SQLiteRowStore: RowStore, Sendable`: forwards to SQLiteBackend; :75 `querySkipCorrupt` overrides protocol default with real cursor-level skip +- SQLiteStores.swift:90/102/119 `SQLiteBlobStore` / `SQLiteAuditLog` / `SQLiteTransaction`: thin forwards +- SQLiteSchema.swift:9 `enum SQLiteSchema`: :12 `nativeType(_:)` ColumnType→SQLite storage class (.uuid/.timestamp→TEXT, .hlc→INTEGER packed, .fingerprint→BLOB); :29 `createTable(_:)`; :68 `appendOnlyTriggers(_:)` emits BEFORE UPDATE/DELETE RAISE(ABORT) trigger pair +- SQLiteSchema.swift:103/132/167 internal tables: `_storagekit_migrations` (kit_id PK) / `_storagekit_audit` (event_id+hlc PK, full-precision physical_time/logical_count/node_id columns alongside lossy packed hlc) / `_storagekit_blobs` (key PK) +- SQLitePredicateCompiler.swift:27 `SQLitePredicateCompiler.compile(_:)`: throws on invalidIdentifier; every column name validated before interpolation (SECFIX-WS2-PK F7) +- SQLiteIdentifierValidator.swift:28 `validateSQLIdentifier(_:)`: `[A-Za-z_][A-Za-z0-9_]*`; SINGLE seam for SQLite module (SQLiteStorage + SQLitePredicateCompiler both call this, no forked copy) +- SQLiteObserver.swift:26 `actor SQLiteObserverRegistry`: row subs via sqlite3_update_hook (op/table/rowid ONLY, no column values); blob subs fed directly by SQLiteBackend.putBlob/deleteBlob call sites (hook cannot carry bytes) +- KeychainKeyStore.swift:36 `struct KeychainKeyStore` (Apple-only, `#if canImport(Security)`): Mode3 key source; :72 `estateAccount(for:)` SHA-256(standardized path)→per-estate Keychain account, so co-processes agree without coordination; :82 `loadOrCreateKey()` idempotent under concurrent first-callers; :111 `deleteKey()` idempotent (missing item = success); access class `kSecAttrAccessibleAfterFirstUnlockThisDeviceOnly` (background-readable post-first-unlock, device-only, no iCloud sync) + +### Backend: PersistenceKitPostgreSQL +- PostgreSQLStorage.swift:23 `final class PostgreSQLStorage: Storage, Sendable`: observer = NoOpObserver() (no live PG change-notification impl) +- PostgreSQLStorage.swift:47 estate isolation: search_path pinned to `pk_` schema per connection (PG analogue of SQLite one-file-per-estate); `public` stays on path for shared extensions +- PostgreSQLStorage.swift:120 `actor PostgreSQLBackend`: :129 `encryptionConfig` is `nonisolated let` (immutable+Sendable, read sync by row stores for the crypto seam) +- PostgreSQLStorage.swift:206/333 per-kit schema version stored in `_storagekit_meta` under composite key `"schema_version:"`; global max under plain key `"schema_version"` +- PostgreSQLStorage.swift:246 `storageStats(now:)`: pg_database_size / pg_stat_database (blks_hit,blks_read,xact_commit,xact_rollback,deadlocks) / pg_locks⋈pg_database (granted=false ⇒ lockContention) +- PostgreSQLPool.swift:12 `actor PostgreSQLPool`: fixed-size; :38 `acquire()` reuse→open-new(if under size)→CheckedContinuation-wait w/ timeout→poolExhausted +- PostgreSQLPool.swift:101 `openConnection()`: every new conn: CREATE SCHEMA IF NOT EXISTS + SET search_path, closes conn on setup failure (never hands back half-configured conn) +- PostgreSQLPool.swift:168 `parseTLSMode(host:)`: `ARIA_MCP_POSTGRES_TLS` env var: disable|require|(absent/unrecognized→prefer, incl. loopback: explicit opt-out required for plaintext) +- PostgreSQLConnection.swift:23 `PostgresConnection` ext: :25 `executeSimple` / :33 `executeParameterized` wrap postgres-nio errors as StorageError.backendError +- PostgreSQLConnection.swift:45 `makeBindings(_:)`: TypedValue→PostgresBindings; .fingerprint serialized as 32 raw bytes fixed block order +- PostgreSQLConnection.swift:103/116 `decodeRow(_:columns:)` / `decodeCell(_:type:)`: decode failure → `.null` (NOT a throw: PG wire protocol already enforces column types at a lower level than SQLite affinity) +- PostgreSQLPredicateCompiler.swift:27 `.compile(_:)`: `$1,$2,...` positional params; :89/:102 bitmask cases reference prior bindings by FINAL numeric position (bindings.count-1 / bindings.count) +- PostgreSQLSchema.swift:8 `enum PostgreSQLSchemaEmitter`: :47 `typeSQL(_:)` (.uuid→UUID native, .timestamp→TIMESTAMPTZ, .json→JSONB); :83 `appendOnlyFunctionSQL` ONE shared plpgsql trigger fn (CREATE OR REPLACE, idempotent) for ALL append-only tables; :97 `appendOnlyTriggerStatements` DROP-then-CREATE per table (PG has no CREATE TRIGGER IF NOT EXISTS) +- PostgreSQLStores.swift:25 `final class PostgreSQLRowStore: RowStore, Sendable`: :34 `withConnection` routes to txn.connection if inside a PostgreSQLTransactionContext, else pool.acquire/release +- PostgreSQLStores.swift:240 `renderPredicate(_:startIndex:bindings:)` (private): renumbers `$N` placeholders in REVERSE order (avoid $10 clobbered by naive forward $1 replace) +- PostgreSQLStores.swift:258/352 `PostgreSQLBlobStore` / `PostgreSQLAuditLog`: lazily CREATE TABLE IF NOT EXISTS own backing tables on first use (not in caller's SchemaDeclaration) +- PostgreSQLStores.swift:509 `decodeAuditEvent(_:)`: required fields `try` (throw on failure); optional before-state fields `try?` (NULL = valid "no prior state") +- PostgreSQLIdentifierValidator.swift:26 `validatePSQLIdentifier(_:)`: identical rule to SQLite's validator; independent copy (3 total incl. Rust `validate_sql_identifier`): SECFIX-WS2-PK F7/F9/F10, one seam PER MODULE + +### PersistenceKitReplication +- ReplicationTypes.swift:24 `struct ReplicationCursor: Equatable`: hlcWatermark/rowsWritten/auditEventsWritten/blobsWritten +- ReplicationTypes.swift:51 `enum ReplicationError`: schemaMismatch(sourceVersion:destinationVersion:sourceKitID:destinationKitID:) | storageFailure(detail:) +- StorageReplicator.swift:69 `enum StorageReplicator`: :89 `replicate(from:to:schema:)` core; :103 `flush(from:into:schema:)` / :115 `hydrate(into:from:schema:)` direction-named wrappers +- StorageReplicator.swift:125 `replicateFull(...)`: 3 steps: (1) per-kit schema-version GATE (throws schemaMismatch, NO auto-migrate), (2) snapshotSource() reads ALL rows/audit/blobs into Sendable payload BEFORE opening dest txn, (3) one `.serializable` dest transaction does upsert(conflictColumns:=table.primaryKey: NOT RowHandle.key) +- StorageReplicator.swift:216 blob-delete propagation (SECFIX-WS2-PK F5): destination keys ABSENT from source snapshot are deleted; additive-only copy would otherwise leak orphaned blobs forever +- StorageReplicator.swift:256 `snapshotSource(...)`: generated columns FILTERED OUT of each row before staging (destination recomputes); a blob present in listKeys() but absent on get() (TOCTOU) → THROWS storageFailure, never silently dropped +- IncrementalReplicationSession.swift:56 `actor BlobDirtySet`: :61 `accumulate` last-write-wins per key; :67 `drain()` sorted-by-key, atomic clear; :79 `restore(_:)` UNION semantics (does not overwrite newer dirt) +- IncrementalReplicationSession.swift:104 `struct DirtyKey: Comparable`: (table, pkEncoded) sorted string key; ORDER IS LOAD-BEARING: deterministic replay across independent sync runs +- IncrementalReplicationSession.swift:135 `actor DirtySet`: :163 `accumulate(_:)` extracts PK cols from TableChange.values; missing PK col or nil values dict ⇒ LOG+SKIP (never crash); :195 `drain()` sorted; :213 `restore(_:)` union (Set.insert no-op if present) +- IncrementalReplicationSession.swift:242 `final class IncrementalReplicationSession: Sendable`: :283 `start(source:schema:)` subscribes one Task per schema table + one blob Task; :266 deinit cancels all tasks +- IncrementalReplicationSession.swift:361 `sync(from:to:fromCursor:)`: schema-version gate (same as full snapshot) → drain both dirty sets → re-scan each dirty row from SOURCE (never trust the original notification's payload for rows: row may have changed again) → missing row ⇒ destination DELETE, present ⇒ UPSERT → audit events filtered to HLC > fromCursor.hlcWatermark → ONE serializable dest transaction +- IncrementalReplicationSession.swift:396-406 RETRY-PRESERVATION: on ANY error after drain, `dirtySet.restore` + `blobDirtySet.restore` are AWAITED SYNCHRONOUSLY inside the catch: NEVER fire-and-forget/detached: a detached restore could race an immediate caller retry and reproduce the lost-keys bug nondeterministically +- IncrementalReplicationSession.swift:521 `snapshotDirtyRows(...)` (private): blob ops need NO source re-read (change event already carries payload); row ops DO re-scan (predicate = exact PK match, `pkPredicate(for:table:)` at :617) + +## INVARIANTS / GOTCHAS + +- TypedValue is a CLOSED 13-case enum. Adding a case requires updating every backend (SQLite/PostgreSQL/InMemory/Rust) in the same change: this cost is deliberate, not an oversight. +- StoragePredicate is CLOSED. Backends treat it as opaque data except when compiling; never special-case another backend's predicate shape. +- SQL IDENTIFIER INJECTION GUARD (SECFIX-WS2-PK F1/F7/F9/F10): every caller-supplied table/column name reaching a dynamically-built SQL string is validated against `[A-Za-z_][A-Za-z0-9_]*` FIRST: three independent seams (SQLiteIdentifierValidator.validateSQLIdentifier, PostgreSQLIdentifierValidator.validatePSQLIdentifier, Rust validate_sql_identifier), one per module, never forked within a module. Double-quoting alone is NOT sufficient: a name containing `"` escapes the delimiter. +- corruptStoredValue is THROWN, never papered over. A stored UUID/timestamp string that fails to parse must surface as an error: substituting a fabricated UUID or epoch-0 date is a silent data-identity lie. querySkipCorrupt / queryRowsSkipCorrupt exist specifically so a CORPUS scan (not a point lookup) can skip-and-log instead of aborting wholesale. +- as-of temporal query (.asOf(hlc)) is GATED OFF by default: every default implementation throws StorageError.featureGated("asOfQuery"). Do not implement/ungate a backend override until NT-L4 (lineage-wide expunge) and NT-P3 (erasure overlay) have both merged: ungating early risks resurfacing erased content through a stale snapshot read. +- CachingRowStore TRANSPARENCY GUARANTEE: every read must return exactly what the unwrapped backing store would. Cache changes latency only, NEVER correctness. Sensitivity gate fails CLOSED: unparseable/unrecognized provenance bit patterns are rejected from the cache, never admitted "just in case." +- Sensitivity ordinal 3 (Secret) is NEVER cacheable, enforced twice: EstateCacheConfig clamps sensitivityThreshold to ≤2 at construction, AND CacheActor.isAdmissible re-checks ordinal==3 unconditionally as defense-in-depth. +- Decorator chain order matters: caller → HashingRowStore → CachingRowStore → backend. HashingRowStore must see the row BEFORE caching decides whether to admit it. +- HashingRowStore hashes the FULL committed row on UPDATE/upsert-as-update (pre-read + merge), never the partial SET-columns dict: a partial hash would diverge from what INSERT computes for identical final data (SECFIX-WS2-PK F6). +- RowCrypto: a keyID mismatch on read means "sealed under a key this estate does not hold": pass the row through as still-ciphertext, do NOT attempt decrypt (would only surface as an AES-GCM auth failure). Mode 3 (fullDatabase) makes the per-row seam a permanent no-op (usesRowCrypto==false): the whole file is already ciphertext. +- assertContentKeyIDInvariant is a STRUCTURAL guard, not just documentation: any encrypting-estate write path that reaches a backend with `.text` content and no/empty keyID throws constraintViolation. If you add a new content-bearing write path (a new upsert variant, a migration path), it must run the encryption seam BEFORE this guard sees it, or the guard will correctly reject the write. +- ErasureOverlay is FAIL-CLOSED: a ledger-check failure drops the row from the result entirely rather than showing possibly-erased content. This is NOT the same as failing open: silence (row absence) is the safe failure mode here. +- erasure_ledger table is appendOnly:true, enforced by the STORAGE LAYER (SQLite triggers / PostgreSQL trigger function / InMemory RowStore.update·delete throw): not just by convention in ErasureLedgerOps. +- InMemoryStorage.transaction mutates LIVE actor state (not a detached copy replaced on commit): this is a scar from a real lost-write incident (5-10% of QueueKit sends under burst). Do not "simplify" this back to copy-then-replace. +- Notification buffering (SECFIX-WS2-PK F2/F3) applies during EVERY transaction on EVERY backend (InMemory row+blob, SQLite blob): observers must never see a notification for a write that was later rolled back. Buffer before the block runs, flush only after commit, discard on rollback. +- IncrementalReplicationSession retry-preservation restore MUST be awaited synchronously inside the failure catch block, never dispatched to a detached Task: a detached restore races an immediate caller retry and can reproduce a lost-dirty-keys bug nondeterministically. +- Full-snapshot replication (StorageReplicator) upserts on `table.primaryKey` as the conflict column, NEVER on RowHandle.key (which is a fresh random UUID per call and would insert duplicates on every re-run). +- Replication requires exact per-kit schema-version equality between source and destination (srcVersion == dstVersion == schema.version). No auto-migration, ever: a version mismatch is ReplicationError.schemaMismatch, not a best-effort attempt. +- Generated columns are ALWAYS STORED (never VIRTUAL): PostgreSQL has no VIRTUAL form, and this keeps SQLite/PostgreSQL/InMemory semantically identical. Generated column names are filtered OUT of any row payload staged for upsert during replication (destination recomputes them; writing a value into a GENERATED column errors on both SQL backends). +- HLC.packed truncates physicalTime to 40 bits: lossy for far-future timestamps. SQLite's `_storagekit_audit` table therefore stores physical_time/logical_count/node_id as separate full-precision columns alongside the packed value; decode from those three columns, not from unpacking the packed column, or a cold rebuild's lastHLC can silently disagree with a snapshot's. +- ISO8601 fast-path parser (`fastParseCanonicalUTC`) exists purely for CPU cost: the general ISO8601DateFormatter was measured at ~80% of total CPU during large imports (Merkle rollup re-decodes every row's timestamp on every insert). The fast path recognizes ONLY the exact canonical shape this kit writes and returns nil (triggering formatter fallback) for anything else: never weaken its exactness to "handle more cases," that reintroduces the cost it exists to avoid. +- NovelTokenTaggerChoice is FIXED AT ESTATE CREATION in v1.0: no change-after-creation path exists yet. `.nlTagger` estates cannot safely federate with `.hmm` estates without full re-tagging; this is not yet enforced automatically (v1.1 work): caller discipline is the only guard today. +- Pinned/deterministic constants: sensitivityThreshold clamp ≤2, Secret ordinal ==3 always excluded, SQL identifier pattern `[A-Za-z_][A-Za-z0-9_]*`, ciphertext envelope layout [12-byte nonce][16-byte tag][ciphertext], KeychainKeyStore.keyByteCount=32, RFC-3339 round-trip range [0001-01-01, 9999-12-31]. +- PersistenceKit imports NO vector-search engine and must never grow one: the ACCOMMODATION contract (vector payload round-trip, bulk hydration, count, delete via the general RowStore/BlobStore surfaces) is the full extent of this package's obligation toward VectorKit's workload (ADR-008). diff --git a/packages/kits/PersistenceKit/docs/DETAILS.md b/packages/kits/PersistenceKit/docs/DETAILS.md new file mode 100644 index 0000000..78e5ecd --- /dev/null +++ b/packages/kits/PersistenceKit/docs/DETAILS.md @@ -0,0 +1,1423 @@ +--- +doc: DETAILS +package: PersistenceKit +repo: moot-system +authored_commit: 909513d0a8ecb1e9e903af9f4d25b3e4f2528242 +authored_date: 2026-07-04 +sources: + - path: Sources/PersistenceKit/AuditLog.swift + blob: ca4c0c9623056a49ad888a7a77e720a7519556bb + - path: Sources/PersistenceKit/BlobStore.swift + blob: f052c3ecc693d5414d57ad2c6da5fb4c5fe28f79 + - path: Sources/PersistenceKit/CacheInvalidator.swift + blob: 0a844b8037404dd72d08df6887ceee1e8c6014f2 + - path: Sources/PersistenceKit/CachingRowStore.swift + blob: 7edf9a86eb31dd9a17abd97b7baa9e8d5425e266 + - path: Sources/PersistenceKit/Column.swift + blob: 3cfc3ba856ebe15e8756ae65dae736cbd2c288a6 + - path: Sources/PersistenceKit/EncryptionMode.swift + blob: fdb7aa7ec63088ae09739f5e2a8e1468c386b987 + - path: Sources/PersistenceKit/ErasureLedger.swift + blob: eb2ca35f69783eead6049949bbaad68decfc9915 + - path: Sources/PersistenceKit/ErasureOverlay.swift + blob: d5a86f2baa10647134f285d6996a8e62cbe23ace + - path: Sources/PersistenceKit/EstateCacheConfig.swift + blob: 7003951cb5b2b075b6bb0c49b5fe372a97bfeb3a + - path: Sources/PersistenceKit/EstateConfiguration.swift + blob: be91569405e5c48a99859b4595540a59bd1a1994 + - path: Sources/PersistenceKit/GCPin.swift + blob: 73336dc4c693fcfa285a7eff8ed6520f27951ff8 + - path: Sources/PersistenceKit/GeneratedColumn.swift + blob: 34e466c80a0e9aac5d24303eaad132901333470b + - path: Sources/PersistenceKit/HashingRowStore.swift + blob: fe233ed2f4fda177373ad105d14fa321d325d0df + - path: Sources/PersistenceKit/NoOpObserver.swift + blob: 99f62b6889df50c5f09fce749f363637891f116c + - path: Sources/PersistenceKit/NovelTokenTaggerChoice.swift + blob: 9588d6303431883a2c8e0ee727f7ebe3e3f6d139 + - path: Sources/PersistenceKit/PersistenceKitTelemetry.swift + blob: a24e54dfb97beb86dac9422bbfc7cc3c2d959d73 + - path: Sources/PersistenceKit/Predicate.swift + blob: 522802c842dfc7573137efd7a3f36300d5201468 + - path: Sources/PersistenceKit/RowCrypto.swift + blob: 6678e426c0902cdae6d164904001526756e40916 + - path: Sources/PersistenceKit/RowStore.swift + blob: 9a96ff89528ca786d51224324cacb78cf810037d + - path: Sources/PersistenceKit/Schema.swift + blob: 5a6894c68d6f29eecfe9acc34d7c06ee2a655ac3 + - path: Sources/PersistenceKit/SnapshotRegistry.swift + blob: 88087bda544165c4d24c514a4f3e0641a620512e + - path: Sources/PersistenceKit/Storage.swift + blob: 7484a40913b28cb11a6bd2e3ea822dc8fe8eb63e + - path: Sources/PersistenceKit/StorageError.swift + blob: 743c2d1a24c7bafedb217e4c6bbf30ca20d03be8 + - path: Sources/PersistenceKit/StorageIntrospection.swift + blob: 35522b6601037246907bb88ebb2fb2eb5ea9b0e2 + - path: Sources/PersistenceKit/StorageObserver.swift + blob: f0a8b61a15344e02193137a3393be345dd51cf25 + - path: Sources/PersistenceKit/Transaction.swift + blob: d58d478618b31cc3985275be6ceee84a6fc222de + - path: Sources/PersistenceKit/TypedValue.swift + blob: ec124b7aebd86f64f67fea6656396d374666b741 + - path: Sources/PersistenceKitInMemory/InMemoryAuditLog.swift + blob: 1ba83d2408d3384d4d8e4f286fa2db743a433cc9 + - path: Sources/PersistenceKitInMemory/InMemoryBlobStore.swift + blob: e02822bde83c899b6d18ab8d82b90afba6abb5ac + - path: Sources/PersistenceKitInMemory/InMemoryObserver.swift + blob: 71b98c9da2d869a81a78265b7ef1f64a57a907af + - path: Sources/PersistenceKitInMemory/InMemoryRowStore.swift + blob: 2f7c92612c46b8a097e4211147c924f8356e47d1 + - path: Sources/PersistenceKitInMemory/InMemoryStorage.swift + blob: 9820c771bfe39ab738eea1f3b1623491fcfb1326 + - path: Sources/PersistenceKitInMemory/PredicateEvaluator.swift + blob: e8735422ff87addff6a2a3a89da85c495d07f07c + - path: Sources/PersistenceKitPostgreSQL/PostgreSQLConnection.swift + blob: 5a282a6064a69a543a5d650ccc7eeeae2c5a3e4f + - path: Sources/PersistenceKitPostgreSQL/PostgreSQLIdentifierValidator.swift + blob: 120cf0c1576a7db45d79bf6865e2f15cff09a5ac + - path: Sources/PersistenceKitPostgreSQL/PostgreSQLPool.swift + blob: eddc0c3af4135565e15d2d763b0d24240973dd6f + - path: Sources/PersistenceKitPostgreSQL/PostgreSQLPredicateCompiler.swift + blob: 6f0791e0372b09cf2605bae4cf149e48bbba2834 + - path: Sources/PersistenceKitPostgreSQL/PostgreSQLSchema.swift + blob: f165f0877b96c87e2f7de46012f8f8acb3c03cc8 + - path: Sources/PersistenceKitPostgreSQL/PostgreSQLStorage.swift + blob: 849926f9e6def788fae2f38757ad44bcd52a18a0 + - path: Sources/PersistenceKitPostgreSQL/PostgreSQLStores.swift + blob: bff90838d1324d82ae76a0895f15399537e9e343 + - path: Sources/PersistenceKitReplication/IncrementalReplicationSession.swift + blob: 2f90378146e043d75c739d400d98c7770e07af78 + - path: Sources/PersistenceKitReplication/ReplicationTypes.swift + blob: bb13c63d1febd50ad46c1814610d7f6c31a33112 + - path: Sources/PersistenceKitReplication/StorageReplicator.swift + blob: f5acc5993c8647e53c127c6871a1282c24b1c427 + - path: Sources/PersistenceKitSQLite/KeychainKeyStore.swift + blob: 0071732291a7cb6ce0777bd230a6188276fb4f32 + - path: Sources/PersistenceKitSQLite/SQLiteConnection.swift + blob: ece56dc7e25e67656bb37f5222c18c1166c750cc + - path: Sources/PersistenceKitSQLite/SQLiteIdentifierValidator.swift + blob: 713339c137d6af1cfbba5a3584e05bdda70b42c5 + - path: Sources/PersistenceKitSQLite/SQLiteObserver.swift + blob: 2cb61ed75dae5ab81a3cdfb5c9581731d25fd960 + - path: Sources/PersistenceKitSQLite/SQLitePredicateCompiler.swift + blob: 5770adb2024c54ea6921651632ca65ee84d416af + - path: Sources/PersistenceKitSQLite/SQLiteSchema.swift + blob: 4ba6fc175fe9d486b17af1088a23da64041b12ae + - path: Sources/PersistenceKitSQLite/SQLiteStorage.swift + blob: 417e63cc07b60295ac874187ebb796bf9f3b3c86 + - path: Sources/PersistenceKitSQLite/SQLiteStores.swift + blob: 76499ffb70d979f0d13e8cf9e32bc38ff28ffdb5 +--- + +# `PersistenceKit` Details + +This document walks through every source file in the package. Read +`OVERVIEW.md` first for the big picture. Files appear grouped by +target, in the order a reader should learn them. The core protocols +and value types come first. The decorators and cross-cutting toolkits +built on top of them come next. The three backends that implement the +core protocols come after that. The replication module, which runs +across two backends, comes last. + +## Target: `PersistenceKit` (core) + +### Storage.swift + +This file provides the `Storage` protocol, the single entry point +every backend implements. A term of art first. A protocol in Swift is +a contract that a type promises to fulfill. Any type that conforms to +`Storage` can be used anywhere `Storage` is expected. This holds +regardless of what physical engine backs it. + +`Storage` bundles four sub-stores: `rowStore`, `blobStore`, `auditLog`, +and `observer`. It also bundles five life-cycle functions. +`open(schema:)` creates files or connections. It also brings the +backend up to its declared schema version. `close()` shuts the backend +down cleanly. It must be safe to call more than once. +`transaction(isolation:_:)` runs a block of work as one step. It rolls +back every change if the block throws. The two `currentSchemaVersion` +overloads report how far a backend's schema has migrated. One reports +globally; the other reports for one named kit. `migrate(to:)` advances +the schema forward. A protocol extension supplies a default isolation +level of read-committed. Most callers can therefore write +`storage.transaction { ... }` without naming a level at all. + +### RowStore.swift + +This file provides the typed row input and output protocol. It also +provides three small supporting types. `RowKey` is a type alias for +`UUID`. `StorageRow` is a fixed, subscriptable wrapper around a +`[String: TypedValue]` row. `RowHandle` is a `(table, key)` pair that +identifies exactly one row. + +`RowStore` declares the operations every backend must support: +`insert`, `upsert`, `update`, `delete`, `query`, and `count`. Two +further query variants matter for performance and safety. +`query(...columns:)` is the column-projecting form. A caller passes a +specific list of column names. A column such as a large text blob is +then never transferred out of storage at all. This matters when a +caller only needs a handful of small columns from a wide table. +`querySkipCorrupt(...)` is the resilience form. Rather than aborting an +entire corpus scan the moment one row's stored value fails to parse, +it skips that row. It counts the skipped row and returns everything +else. The file's own comment is explicit about this method's +purpose. It exists for best-effort scans, such as scanning every +drawer in the estate. It should never be used for a single-row point +lookup, where a corrupt value should always be a loud error. + +A protocol extension supplies default implementations for all of the +above. Each one falls back to the plain, full `query`. The +extension also supplies a family of as-of temporal query overloads. +As-of querying means asking what a row looked like at a past point in +time. That point is identified by an `AsOfCoordinate`, a type defined +in the `SubstrateTypes` dependency. The default implementation answers +`.present` queries normally. It throws `StorageError.featureGated` for +any `.asOf(hlc)` request. The feature stays off on purpose until two +other in-flight pieces of work both ship: lineage-wide expunge and the +erasure overlay. Turning it on early could let an as-of read resurface +content that was supposed to have been erased. + +Finally, this file declares the write-transaction boundary as protocol +requirements rather than protocol-extension defaults. These +requirements are `beginTransaction`, `commitTransaction`, and +`rollbackTransaction`. This distinction matters for a subtle Swift +reason. A protocol requirement dispatches to the concrete type even +when called through an `any RowStore` existential. A protocol-extension +default does not always do so reliably. `CachingRowStore` must forward +these calls to whatever it wraps. They have to be true requirements +for that forwarding to work. + +### BlobStore.swift + +This file provides the blob input and output protocol: `put`, `get`, +`delete`, `exists`, `size`, and `listKeys`. A blob is a chunk of raw +bytes identified by any string key. That key is often a +content-addressed hash. It can also be a row's UUID combined with a +column name. `listKeys()` exists in particular so the replication +primitive can list every blob in a backend for a full-snapshot +copy. Key order is unspecified and may differ between backends and +even between calls. Any caller that needs a stable order, and +replication does, sorts the returned keys itself. + +### AuditLog.swift + +This file provides the append-only history protocol. An audit log +records every change made to a row, in order. It never edits or +removes a past entry. `append` and `appendBatch` are both idempotent on +the compound key `(eventID, hlc)`. HLC stands for Hybrid Logical Clock, +a timestamp format that orders events the same way every time across multiple +devices without perfectly matched up clocks. Replaying the same +event twice, which can happen during sync, never creates a duplicate. +`iterate(after:rowID:limit:)` walks the log in HLC order and accepts a +resume cursor. `eventsForRow(_:)` returns just one row's history. A +caller uses this to build a point-in-time projection of that row. +`PersistenceKit`'s role stops at durable, ordered storage of these +events. Enforcing the conflict-resolution rules of a CRDT belongs to a +higher kit. A CRDT is a data structure designed so concurrent edits +always merge the same way. + +### StorageObserver.swift + +This file provides the live change-notification protocol. It also +provides four supporting event types. `StorageEvent` is the three +kinds of row change: `insert`, `update`, and `delete`. `TableChange` +bundles one such event with its table, row key, values, and HLC. +`BlobEvent` and `BlobChange` are the blob-store matches. A `put` +event carries the written bytes. A subscriber, in particular the +incremental replication session, can then avoid a second round-trip +just to re-read what was written. `DirtyChainEvent` is a third, more +special notification. When a row in a table marked hashable is +written, the write also carries the row's new content hash. It carries +the row's two nearest ancestors in a Merkle-style containment +hierarchy. A downstream consumer needs only this to recompute an +integrity tree bit by bit. It never has to rescan the tree from +scratch. + +`StorageObserver` itself declares three subscription methods: +`observe(table:events:)`, `observeBlobs()`, and `observeDirtyChain()`. +Each one returns an `AsyncStream`, a Swift type for values that arrive +over time. Delivery is documented as at-least-once. Ordering is +preserved within one subscription but not promised across different +tables. A protocol extension supplies a default no-op stream for +`observeDirtyChain()`. Older observer implementations, written before +hash-on-write existed, keep compiling without a change because of +this. + +### StorageIntrospection.swift + +This file provides an optional, separate capability protocol for +reporting backend health. It also provides the `StorageStats` value +type that carries the numbers. `StorageIntrospection` is on purpose +not merged into `Storage` itself. It is a distinct protocol so that +adding it never breaks an existing conformer. A caller checks for the +capability with `storage as? StorageIntrospection` rather than +assuming every backend has it. + +`StorageStats` holds a superset of fields that no single backend fully +fills. Page-level statistics such as `pageSize` and +`walFrameCount` are SQLite-only. Buffer cache and transaction counters +such as `cacheHitRatio` and `deadlockCount` are PostgreSQL-only. +`rowCount` and `blobCount` are InMemory-only. A backend that cannot +supply a given field sets it to `nil` rather than fabricating a zero. +A caller can then tell "not measured" apart from "measured as zero." +The file's doc comment includes a field-by-field table. That table +cross-references exactly which backend fills which field. It is the +trusted reference for anyone adding a new statistic. + +### Transaction.swift + +This file provides the transaction protocol and the three isolation +levels a caller may request: `readCommitted`, `repeatableRead`, and +`serializable`. `StorageTransaction` exposes the same three sub-stores +`Storage` does: `rowStore`, `blobStore`, and `auditLog`. Code written +inside a transaction block therefore looks exactly like code written +outside one. The only difference is that every operation inside the +block either all commits together or all rolls back together. The +file's own comment notes that nested transactions and savepoints are +out of scope for this version of the design. + +### StorageError.swift + +This file provides the single closed error type every `PersistenceKit` +operation can throw. Being a closed `enum`, as opposed to an open +protocol, means every possible failure is listed in one place. A +caller's `switch` over a `StorageError` can therefore be exhaustive. +The cases cover backend availability, schema and migration failures, +and constraint and uniqueness violations. They also cover +connection-pool exhaustion, transaction conflicts, and type mismatches. +Three cases matter most for safety. `corruptStoredValue` fires when a +stored value could not be parsed back to its declared type. It is +thrown instead of silently using a fabricated default instead. This is because +a fabricated UUID or an epoch-zero date would be a quiet +data-identity lie. `invalidConfiguration` fires when an +`EstateConfiguration` requests something impossible on the current +platform, such as an Apple-only tagger on a non-Apple host. +`invalidIdentifier` fires when a caller-supplied SQL column or table +name contains characters outside the safe set. This is the guard that keeps a query built at run time from becoming +a SQL-injection vector. It matters even though the name is already +double-quoted. + +### Column.swift + +This file provides `Column`, a `(table, name)` pair used throughout +the predicate and schema types to reference one column. It also +provides `ColumnType`. This is the closed set of value kinds a column +can hold. The kinds are `uuid`, `bitmap`, `text`, `timestamp`, `float`, +`int`, `bool`, `blob`, `json`, `hlc`, and `fingerprint`. `Column` +conforms to `Comparable`, ordered first by table and then by name. +This exists purely so test fixtures and other code that sorts columns +gets a stable, steady order. + +### TypedValue.swift + +This file provides `TypedValue`, the tagged union that carries every +value crossing the `PersistenceKit` boundary. It is the wire format of +the whole library. Every backend pattern-matches on the case and emits +its own native representation. SQLite might store a `.uuid` as a +`TEXT` column. PostgreSQL might store it as a native `UUID` column. +Callers on both sides always see the same Swift enum. The case set is +on purpose closed. The file's own comment says adding a new case is +expected to require updating every backend. Paying this cost is worth it. It means a value works the same way +on every backend. A new value kind can never partially work on only +some backends. Two of the twelve cases, `.hlc` +and `.fingerprint`, exist for a specific reason. `PersistenceKit` is +forbidden from building again substrate math. The `HLC` and +`Fingerprint256` types themselves come from the `SubstrateTypes` +dependency. `PersistenceKit` only needs a slot to carry them. + +### Predicate.swift + +This file provides `StoragePredicate`, the closed query-condition tree +every backend compiles to its own query language. It also provides +`OrderClause` and `OrderDirection` for sorting. The predicate cases +fall into three families. The logical family covers `and`, `or`, +`not`, `isTrue`, and `isFalse`. The comparison family covers `eq`, +`neq`, `lt`, `lte`, `gt`, `gte`, `isNull`, `isNotNull`, `in`, and +`like`. The bitmap family covers `bitmaskAll`, `bitmaskAny`, +`bitmaskNone`, and `bitwiseEq`. All four apply only to integer-family +columns. `PersistenceKit` treats a predicate as opaque data except when +compiling it. No backend ever needs to special-case another backend's +query shape. The tree is the same tree everywhere, so there is nothing +to special-case. + +The two static helpers `all(_:)` and `any(_:)` build an `and`/`or` +tree from a list of predicates. Both include useful short-circuiting. +An empty list of conditions to AND together collapses to `isTrue`, an +always-true filter, because "no restrictions" should match everything. +An empty list to OR together collapses to `isFalse`. A list that +already contains a conflicting trivial case collapses the whole +expression right away. This happens when `isFalse` sits inside an +`all`, or when `isTrue` sits inside an `any`, rather than building a +needlessly large tree. + +### GeneratedColumn.swift + +This file provides first-class computed columns. A computed column +holds a value derived from other columns in the same row. A small, +structured integer expression performs that derivation, rather than a +caller storing the value directly. `GeneratedColumn` names the column, +its result type, and its `GeneratedExpression`. `GeneratedExpression` +is a closed, recursive enum. It covers exactly the bit-field algebra a +bitmap-heavy schema needs: reading another column (`.column`), a +constant (`.literal`). Three cases cover bitwise AND, OR, and XOR. +Two more cases cover a left shift and a right shift by a fixed amount. +A final pair of cases test equality and inequality, each one giving +back one or zero. The file's own comment explains why this is a structured +expression tree rather than a raw SQL string. A SQL string generated +column would push backend-specific syntax into a schema declaration. +The in-memory backend also has no SQL engine to evaluate it against. A +structured expression has exactly one meaning. SQLite and PostgreSQL +render it to native `GENERATED ALWAYS AS (...) STORED` DDL. The +in-memory backend works out it directly. These are three faithful +versions of one description. No backend gets an escape hatch to +interpret it differently from another. + +`renderSQL()` turns an expression into a SQL fragment shared by SQLite +and PostgreSQL. Both use the same bitwise-operator syntax and +double-quoted identifiers. The one exception is bitwise XOR, which +SQLite lacks as an operator. Both backends instead render it as `(a | b) - (a & b)`, which gives +the same value using only AND and OR. `evaluate(_:)` computes +the same result directly against an in-memory row dictionary, for the +in-memory backend. `integerValue(_:)` extracts an `Int64` from any +integer-family `TypedValue`: an `.int`, `.bitmap`, `.bool`, or `.hlc`. +It returns zero for anything else or for an absent column. That zero +is the in-memory evaluator's sentinel for "no meaningful value here." + +### EstateConfiguration.swift + +This file provides `EstateConfiguration`, the one value that fully +describes how to open one estate's storage. It also provides +`BackendConfiguration`. This is the closed choice of physical engine: +`sqlite`, `postgresql`, or `inMemory`. It also carries that engine's +connection settings. `EstateConfiguration` itself carries the estate's +encryption mode, its cache setup, and its novel-token tagger choice. +Each of these has a default. Existing code that never mentions these +newer fields keeps compiling and keeps behaving exactly as it did +before they existed. A caller who asks for nothing else gets a +plaintext, uncached, HMM-tagged estate. + +`queueSibling(filename:)` is a more special function. It derives a +second `EstateConfiguration` that points at a companion database file +sitting beside the estate's own file. A work-queue kit, for example, +gets its own small database this way. It needs no separate key +distribution and no separate configuration path. Take a SQLite estate +at `/.sqlite`. Asking for the sibling `"queue.sqlite"` +produces `/.queue.sqlite`. The estate's own file stem is +folded into the sibling name. Two different estates in the same +directory can therefore never collide. For an in-memory estate, the +sibling is also in-memory. Both are ephemeral, which is the correct +pairing for tests. For a PostgreSQL estate, the function throws +`StorageError.featureGated`. The design puts off a PostgreSQL queue +sibling on purpose until later. It fails loudly rather than quietly +returning a half-working configuration. + +The private helper `deriveQueueSiblingID(parentID:filename:)` computes +the sibling's estate ID in a fixed way. It XOR-folds the filename's +UTF-8 bytes into a sixteen-byte tag. It then XORs that tag with the +parent UUID's raw bytes. The same parent estate and the same filename +therefore always produce the same sibling ID. No call to `UUID()` +happens anywhere on this path. This fixed behavior matters. Every +process that opens the same estate must agree on the same sibling ID +without talking to any other process. + +### EstateCacheConfig.swift + +This file provides `EstateCacheConfig`, the small value type that +turns row caching on or off for one estate. It also bounds how much a +cache is allowed to hold. It bounds how sensitive the content it holds +may be. `ceilingBytes` is clamped to be non-negative at construction. +`sensitivityThreshold` is clamped to at most two. Sensitivity level +three is called "Secret" in the ARIA adjective scale this kit shares +with the rest of the system. Secret content must never be cached, no +matter how the estate is set up. Clamping at construction enforces +that rule. No caller needs to remember the numeric boundary on its +own. `EstateCacheConfig.disabled` is the zero-change default. An estate +that never mentions caching gets exactly the pre-caching behavior. + +### EncryptionMode.swift + +This file provides the three at-rest encryption modes. It also +provides the configuration value that carries one of them per estate. +`EncryptionMode.plaintext` stores content as-is. Encryption, if any, +happens only at a sharing boundary elsewhere in the system. +`.rowEncryption`, Mode 2, encrypts one column's content per row under a +key named by the row's own `keyID` column. `.fullDatabase` is Mode 3. +It encrypts the entire SQLite file, including its schema, through +SQLCipher's `PRAGMA key` at the connection layer. This makes the per-row seam a +no-op for that mode. The whole file is already ciphertext on disk, so +there is nothing left for the per-row seam to do. A fourth mode is on +purpose left out of this enum: database-plus-threshold encryption for +a FedRAMP-tier deployment. The file's comment stresses that this build +does not have that capability at all. Adding it later is a conscious, +reviewed act, not an accident. + +`EstateEncryptionConfig` bundles the mode with a key identifier and the +actual `SymmetricKey`. Its stored `key` property is `package`-scoped +rather than `public`. The SQLite backend, in a sibling module of the +same Swift package, needs it. Nothing outside the package should ever +see it. The convenience initializer `init(_ mode:)` mints a fresh, +full-entropy 256-bit key and a UUID key identifier for either +encrypting mode. It mints neither for plaintext. `usesRowCrypto` is +`true` only for Mode 2. Mode 3 protects everything at the file level, +so the per-row seam has nothing to do there. `fullDatabaseKeyHex` +renders the whole-file key as lowercase hex for the SQLCipher `PRAGMA +key` statement. It is documented as something that must never be +logged. + +### RowCrypto.swift + +This file provides the per-row AES-GCM-256 encryption used by +`EncryptionMode.rowEncryption`. It also provides the three write and +read seam functions every backend calls at the same two moments: just +before a write, and just after a read. Living in `PersistenceKit` core +rather than in one backend module is the point. Both +`PersistenceKitSQLite` and `PersistenceKitPostgreSQL` call the exact +same code. The two backends therefore produce byte-compatible +ciphertext envelopes without either one knowing about the other. + +`AeadProvider` is a small protocol seam. AEAD stands for Authenticated +Encryption with Associated Data. This is a class of algorithm that +both encrypts data and proves no one changed it. Any type that +conforms to `AeadProvider` can swap in for the default algorithm +without touching any call site. The file names this as the point +where a future FIPS-checked provider would plug in. +`CryptoKitAeadProvider` is the default version, backed by Apple's +CryptoKit. It makes a fresh random ninety-six-bit nonce on every +single encrypt call. Reusing a nonce under the same key is the one +mistake that breaks AES-GCM's whole guarantee. It returns the three +parts joined as `[12-byte nonce][16-byte tag][ciphertext]`. A later +decrypt is self-contained from the stored bytes alone. It has nothing +else to look up. + +`encryptedForWrite(_:config:provider:)` is the write-side seam. For an +estate using row encryption, it encrypts the row's `content` column. It +stamps a `keyID` column recording which key sealed it. For any other +mode, or for a row with no `content` column, it returns the values +unchanged. `decryptedForRead(_:config:provider:)` reverses this on +read, but only when the row's stored `keyID` matches the estate's own +key identifier. A mismatched `keyID` means the row was sealed under a +key this estate does not hold. The function passes the row through as +still-ciphertext rather than trying a decrypt that would only fail as +an authentication error. `assertContentKeyIDInvariant(_:table:config:)` +is a final structural guard. On an encrypting estate, any +content-bearing row that reaches a backend write path with `.text` +content and no `keyID` means the encryption seam did not run somewhere +upstream. The function throws rather than let that plaintext be stored +where a later read could never recover it. + +### CachingRowStore.swift + +This file provides `CachingRowStore`, a decorator that wraps any +`RowStore` and serves an in-memory hot tier of recently-read rows. The +caller never has to know this layer is there. The file's own comment +states the guarantee plainly. Every operation returns results the same +as what the unwrapped backing store would have returned. The cache +changes only latency, never correctness. + +The cache key is not just `(table, row key)`. It is that pair combined +with an `AsOfCoordinate`. A `.present` (current-state) read and an +`.asOf(hlc)` (past-state) read of the same row are cached as two +entirely separate entries. A snapshot read against a pinned, fixed +past state can be cached forever. The GC pin mechanism guarantees that +data will not be swept away out from under it. A present-state read, +by contrast, must be dropped the instant the row changes. Four write +methods drop the affected present-state entry after the underlying +write succeeds: `insert`, `upsert`, `update`, and `delete`. For a +batch predicate that does not name one specific row, these same four +drop every present-state entry for that table instead. An optional `parentChainProvider` closure, +supplied by the calling kit, lets a write also drop cached Merkle- +aggregate entries for that row's ancestors. A cached rollup value never +goes stale just because one leaf changed underneath it. + +Admission to the cache is gated by sensitivity. `isAdmissible(_:)` reads +a row's `provenance` bitmap column, when present, and decodes a +six-bit sensitivity field at bits thirty through thirty-five. It maps the scale-gapped raw values zero, sixteen, thirty-two, and +forty-eight to four ordinals. The ordinals are Normal, Elevated, +Restricted, and Secret, in that order. Secret content is turned away +without exception. Anything above the +configured `sensitivityThreshold` is turned away too. An unrecognized +bit pattern, or a `provenance` value of the wrong `TypedValue` case, is +also turned away. The comments call this failing closed. Any doubt +about a row's sensitivity keeps it out of the cache, rather than +risking that sensitive content sits unencrypted in process memory +longer than it should. Eviction removes the least-recently-used entry first, whenever the +estimated byte total goes over the ceiling the estate sets. A counter +that only ever rises tracks recent use. The cache does not track +wall-clock time at all. + +All of this changeable state lives inside a private `actor`, +`CacheActor`. The state is the entry dictionary, the access counter, +and the running byte total. Every stored property of the actor is +itself `Sendable`. Wrapping the state in an actor is what lets the +outer `CachingRowStore` class safely conform to `Sendable` under Swift +6's strict concurrency checking. `CachingRowStore` itself is a `final +class` with no locks of its own. + +### CacheInvalidator.swift + +This file provides `CacheInvalidator`, the bridge between a +`StorageObserver`'s live change stream and a `CachingRowStore`'s +invalidation method. This bridge is needed for the case where some +other writer bypasses the specific `CachingRowStore` instance +entirely. That writer might be a second process, or a raw connection +opened for a migration. Without this bridge, the cache would serve +stale data forever after such a write, because it would never learn +the write happened. + +One `CacheInvalidator` watches every table it is told to watch through +a single detached background task. That task fans out one child task +per table, using Swift's structured-concurrency `withTaskGroup`. +`cancel()` cancels the whole tree at once. `deinit` calls `cancel()` on +its own. A caller who simply lets the invalidator go out of scope does +not leak a background task. The comment on `init(cache:observer:tables:)` +flags one narrow race. Subscriptions start inside the background task. +A write issued in the same instant as `init()` could in theory race +ahead of that subscription. A caller with strict-ordering needs should +yield briefly, for example with `Task.sleep`, before trusting that the +very first write was seen. + +### HashingRowStore.swift + +This file provides `HashingRowStore`, a decorator that intercepts +writes to any table marked hashable in the schema. It computes a +content hash for the row. It emits a `DirtyChainEvent`, so an +integrity tree elsewhere in the system can stay current bit by bit, +rather than being rebuilt from scratch on every write. +`PersistenceKit` on purpose does not import a hashing library itself. +The actual hash function arrives as an injected `ContentHashProvider` +closure supplied by the calling kit, which does import the hashing +library. This package stays free of that dependency while still owning +the mechanism that wires hashing into every write path. + +The trickiest part of this file is keeping the hash correct on partial +writes. An `insert` always carries the row's full column set, so +hashing the incoming `values` directly is correct. An `update`, +however, often carries only the columns being changed. Hashing just +that partial dictionary would produce a hash for a row that never +actually existed in storage. +`augmentWithHashForKnownKey(table:rowKey:mergedValues:)` solves this by +pre-reading the row's current, full state. It merges the incoming +changes on top of it. It then hashes that merged result. `upsert` uses +this same approach when it resolves to an update against an existing +row rather than a fresh insert. Before hashing, the function strips +any existing `content_hash` column from the input. The insert path and +the update-via-merge path therefore always hash the same set of data +columns. Neither one ever drifts apart just because it happened to +already carry a stale hash value. + +`emitDirtyChain(table:rowKey:hashResult:)` delivers the resulting +`DirtyChainEvent` to an injected `ObserverRegistryRef` closure, which +is `nil` for backends without observer support. The hash is still +computed and stored either way. The notification step is simply +skipped when there is no closure to call. + +### ErasureLedger.swift + +This file provides the append-only record of what has been erased, +without ever storing the erased content itself. A "drawer," in this +context, is one unit of stored content. The term comes from the +estate's own filing metaphor, not from this package. +`ErasureLedgerEntry` pairs a `drawerId` with the `HLC` at which it was +erased. The `erasure_ledger` table this module declares is marked +append-only in its `TableDeclaration`. Every backend enforces this at +the storage layer. Even a caller that somehow bypasses the ledger's +own API cannot change or remove an entry once it is written. This is +exactly the tamper-evidence property an erasure record needs. + +`recordErasure(rowStore:drawerId:erasedHlc:)` inserts one entry. It +throws `StorageError.duplicateKey` if that drawer was already erased, +because each drawer is erased exactly once. +`isErased(rowStore:drawerId:)` is the fast point-lookup used on every +read that might need to hide erased content. +`lookupErasure(rowStore:drawerId:)` returns the full entry when a +caller needs the erasure time itself, not just a yes-or-no answer. + +### ErasureOverlay.swift + +This file provides the two-phase, fail-closed filter that actually +hides erased content from a query result. It builds on top of the +ledger, though this module does not itself define that ledger. +`ErasureOverlayConfig` is supplied by a higher kit. It carries two +pieces of entity-specific knowledge `PersistenceKit` does not have on +its own. `extractErasureId` is a closure that pulls the erasure-ledger +key out of a result row, or returns `nil` for a row type not subject +to erasure at all. `contentColumns` is the list of column names to +null out when a row turns out to be erased. + +`ErasureOverlay.apply(rows:config:rowStore:)` runs the two phases the +file describes at the top. Phase one is a plain query that +already ran before this function was called. Phase two walks the +returned rows one at a time. It checks each row's erasure ID against +the ledger. It nulls that row's content columns if the row was +erased. The important design choice is what happens when the ledger +check itself throws an error, instead of giving a clean yes-or-no +answer. The row is dropped from the result. This is the +fail-closed half of "two-phase fail-closed." An uncertain answer about +whether a row was erased is treated the same as "it was." Showing +content that should have been erased is a worse failure than +temporarily hiding content that was not. + +### GCPin.swift + +This file provides the query that tells a garbage-collection pass +which rows it must not touch yet. GC, short for garbage collection, is +any maintenance pass that reclaims space by deleting rows no longer +needed. An old version of a row that a newer version has replaced is +one example. The problem this file solves is simple to state. A +snapshot taken earlier, through `SnapshotRegistry`, might still need to +read an "old" row that a GC pass would otherwise think it safe to +delete. + +`GCPin.minimumRetainableHlc(rowStore:)` answers this by finding the +smallest HLC across every currently-registered snapshot. The oldest +live snapshot's time stamp is the pin. Any row whose HLC sits at or +after that pin must survive a GC pass. Anything strictly older is safe +to reclaim. When no snapshots exist at all, the function returns +`nil`. Nothing is pinned in that case, and every row is fair game for +GC. `isPinned(rowStore:rowHlc:)` is the convenience per-row check built +on top of the same query. + +### SnapshotRegistry.swift + +This file provides the durable record of named point-in-time +snapshots. It also provides the signed proofs that go with them. A +snapshot, in this design, is not a copy of any data. It is a registry +row recording one HLC, the moment the snapshot was taken. It also adds +one or more proof rows. Each proof row records what a Merkle root +looked like at that moment for a given subject. `PersistenceKit` itself knows nothing about what a "subject" +is. The `subjectKind` and `subjectId` strings are supplied by whichever +higher-level kit is taking the snapshot. + +`SnapshotId` is a UUID-backed opaque ID, minted fresh by +`SnapshotId.mint()`. +`SnapshotRegistryOps.createSnapshot(rowStore:hlc:label:createdAt:attestations:)` +mints an ID, inserts one registry row, and inserts one proof row per +subject passed in. It returns the finished `SnapshotRecord` to the +caller. `listSnapshots(rowStore:)` walks every registered snapshot in +HLC order. `deleteSnapshot(rowStore:snapshotId:)` removes a snapshot's +proof rows before removing its registry row, child rows first, so no +orphaned proof can ever point at a deleted registry entry. It reports +whether anything existed to delete. `attestations(rowStore:snapshotId:)` +reads back every proof for one snapshot, ordered by subject for a +steady result. The remaining private helpers handle the mechanical +work. They turn a `StorageRow` into a typed `SnapshotRecord` or +`SnapshotAttestation`, and back again. + +### NoOpObserver.swift + +This file provides `NoOpObserver`, a plain `StorageObserver` that +satisfies the protocol without doing any real work. Its three +subscription methods each return a stream that finishes right away, +delivering nothing. It exists as a stand-in for any backend or test +path that has no real change-notification mechanism yet. That backend +still needs to satisfy the `Storage` protocol's requirement for an +`observer` property. The PostgreSQL backend, for instance, uses this +type directly. It does not implement a live PostgreSQL notification +channel. + +### NovelTokenTaggerChoice.swift + +This file provides the estate-creation-time choice of which tagger +classifies a "novel token." A novel token is a word not found in a +language model's own word list. An estate needs to sort words like +this somewhere in its processing pipeline. The two +cases are `.hmm` and `.nlTagger`. `.hmm` is a fixed Hidden Markov Model +tagger. It produces the exact same output on every platform, so it is +safe to use across a group of devices sharing memory. `.nlTagger` is +Apple's `NaturalLanguage` framework tagger. It can be more accurate on +Apple hardware. It does not work outside Apple platforms. It also does not always produce the same output across different +OS versions. + +This choice is fixed forever at estate creation in this version of the +design. Changing it later, with re-tagging of existing content, is on +purpose put off to a future version. The file's long comments spell +out the practical result. An estate created with `.nlTagger` cannot +safely federate with an `.hmm`-tagged estate without re-tagging all of +its content first. Federating means sharing and comparing memories +across estates. The two taggers can disagree on the same word. No +code enforces that rule on its own yet. A caller who picks +`.nlTagger` must keep that estate out of any federation with a +mismatched estate. `NovelTokenTaggerChoice.default` is `.hmm`. This is +the safe, federation-friendly baseline every estate gets, unless a +caller on purpose picks the Apple-only choice instead. An enum with +the same name exists in `LatticeLib`, a different package, defined on +its own. The two packages on purpose do not share this one type. Not +one imports it from the other, so neither package depends on the +other. + +### PersistenceKitTelemetry.swift + +This file provides `reportStorageStats(_:estateID:now:)`. This function +turns a `StorageIntrospection.stats(now:)` snapshot into a stream of +named metrics through `IntellectusLib`, a light, zero-dependency +telemetry library. The function's whole behavior is gated on +`Intellectus.isEnabled`, which defaults to `false`. Telemetry is off +for nearly every deployment. When it is off, the function returns +after a single atomic boolean check. It never calls `stats(now:)`. It +never builds a single metric object. The file's own comment puts this +cost at roughly one nanosecond. The disabled path is close to free. + +When telemetry is on, the function emits one metric per non-`nil` +field of the captured `StorageStats`. Each metric sits under the +`persistence.db.*` namespace. Each one is tagged with the emitting +kit's name and the estate's ID, so a monitoring backend can filter and +group by estate. `StorageStats` fields are `nil` exactly where a +backend cannot supply them. The set of emitted metrics differs from +backend to backend. This function needs to know nothing about which +backend produced the snapshot. For example, an in-memory estate never +emits a `wal_frames` metric. That field was simply never filled in. +Every time stamp passed to `Intellectus.report(_:)` +comes from the caller-supplied `now` argument. It never comes from a +fresh `Date()` call inside this function. This is a rule followed +through the whole package. Any code that might run the same way twice +must never read the wall clock itself. + +## Target: PersistenceKitInMemory + +### InMemoryStorage.swift + +This file provides `InMemoryStorage`, the backend used for tests and +for any storage that should live only as long as the current process. +It also provides the `InMemoryStateActor` that owns every piece of +changeable state behind Swift's actor isolation. There is no file on +disk. There is no network connection. Every table is a plain Swift +dictionary held in memory. + +The trickiest part of this file is how +`transaction(isolation:_:)` implements rollback. It does not run the +caller's block against a private, separate copy of the state and then +replace the live state only on success. Instead, it runs the block +directly against the live actor state. It takes a snapshot copy only +for the error path. The file's own comment explains why the simpler +"copy, change, replace on success" approach is wrong. Say +some other write reaches the live actor state between the moment the +snapshot was taken and the moment a successful transaction would +replace the whole state with its snapshot-derived copy. That +concurrent write would be silently wiped out by that swap. The +comment traces this exact bug to a real incident. A burst of +concurrent inserts racing a transaction lost five to ten percent of +queued work. Running against live state avoids the problem. +The cost is needing a clear rollback-to-snapshot path when the block +throws. + +Reports of a change during a transaction are held back, rather than +sent right away, for a related reason. Say a subscriber were told +about a write that the transaction later rolled back. From the +estate's point of view, a rolled-back write never took place at all. +`beginNotificationBuffering()` starts collecting change reports +instead of sending them. `commitNotifications()` flushes the buffer to +observers only after the block has returned with success. The +rollback path throws the buffer away instead. Four operations change +stored rows: `insertRow`, `upsertRow`, `updateRows`, and `deleteRows`. +Each one works out any declared `GeneratedColumn` +values through the static `materializeGenerated(_:_:)` helper before +storing the row. This matches what SQLite and PostgreSQL compute on +their own, so a query against any backend returns the same generated +values. `queryRows(...)` applies the predicate and ordering against +full rows first. It only narrows to the requested column set at the +very end. A projected query can still filter or sort on a +column it does not actually return. This matches SQLite's own +`SELECT`-list-versus-`WHERE`/`ORDER BY` behavior, in full. + +`InMemoryStorage` also conforms to `StorageIntrospection`. It reports +a rough byte size, a live row and blob count, and a steadily rising +rollback counter. The byte size is a flat per-row guess plus exact +blob byte counts. The file's own comment calls this a rough signal, +not an exact count of memory used. + +### InMemoryRowStore.swift + +This file provides the thin `RowStore` conformance that simply forwards +every call to the shared `InMemoryStateActor`. Its one piece of design +worth noting is the column-projection overload. The in-memory backend +already holds the full row in memory, so leaving out columns nobody +asked for saves no real transfer cost, the way it does for SQLite. The +comment is clear that the point of building it anyway is one of +matching shape. A `StorageRow` returned from a projected in-memory +query is shaped just like the same projected query against SQLite. +Code under test cannot depend on a column being present by mistake, +just because the in-memory backend happened to still have it in +memory. + +### InMemoryBlobStore.swift + +This file has the `BlobStore` conformance for the in-memory +backend. It is five one-line forwarding methods to the shared state +actor's blob dictionary. There is nothing backend-specific to explain +here beyond what `BlobStore.swift` already covers. This file exists +purely to satisfy the protocol against the actor's storage. + +### InMemoryAuditLog.swift + +This file has the `AuditLog` conformance for the in-memory +backend. It is again a thin forwarding layer to the shared state +actor. That actor removes duplicates on `(eventID, hlc)`, just as the +protocol's idempotence contract requires. + +### InMemoryObserver.swift + +This file provides `ObserverRegistry`. This is the subscription +bookkeeping shared by every in-memory row store, blob store, and +audit log instance. All three sit behind the same state actor. It also provides +`InMemoryObserver`, the thin `StorageObserver` conformance built on +top of it. The registry on purpose does not use actor isolation for +its subscriber lists. Instead, it uses a plain `NSLock`. The file's +comment lays out the trade-off in plain terms. Registering a +subscription must happen right away, with the stream already recording +it before `observe()` returns to the caller. A change reported the +very next line of code can then never race ahead of the subscription +being recorded. An `actor`-based registry would force `register` to be +an `async` function. That would reopen that same race all over again. This +mirrors the matching Rust observer hub, which registers right away for +the same reason. + +`notify(_:)` takes a snapshot of the list of matching subscribers +under the lock. It then sends to each of them outside the lock. A +subscription's own end-of-life handler, which also needs the lock to +remove itself from the list, can never deadlock against a report +already in flight. Row, blob, and dirty-chain +subscriptions are tracked in three separate dictionaries. All three +follow the same pattern: register right away, snapshot inside the +lock, and send outside the lock. + +### PredicateEvaluator.swift + +This file has the in-memory reader for `StoragePredicate`. It is +the only backend that works out the predicate tree directly against +Swift values in memory, rather than compiling it to a query string +first. `PredicateEvaluator.evaluate(_:against:)` walks the tree from +the top down. It handles every case from `Predicate.swift` by reading +the named column out of a `[String: TypedValue]` row, treating a +missing column the same as an explicit `.null`. It then applies the +matching comparison, bitmask test, or `LIKE`-style text match. +`likeMatch(_:pattern:)` turns a SQL `LIKE` pattern into an +`NSRegularExpression`. In that pattern, `%` stands for any run of +characters. `_` stands for just one character. The in-memory backend +matches the same patterns SQL's `LIKE` operator would. + +`TypedValueComparator.compare(_:_:)`, in the same file, is the shared +ordering function used both by predicate comparisons and by `ORDER +BY` sorting. `nil`, that is `.null`, sorts before every other value. +Two values of the same case compare by their underlying Swift value. +`.hlc` is compared by its packed integer form instead. That form is +`HLC.packed`, defined at the bottom of `InMemoryStorage.swift`. Packing +gives a single total order over an HLC's three parts. + +## Target: PersistenceKitSQLite + +### SQLiteStorage.swift + +This file has `SQLiteStorage`, the `Storage` conformance for the +on-device backend. It also has the much larger `SQLiteBackend` actor +that does almost all of the real work. One actor per estate handles it all: schema migration, row-and-blob +work, audit logging, and health checks. The file's opening comment +matches this design: a +SQLite estate gets exactly one connection. SQLite's own WAL, or +write-ahead log, mode handles concurrent readers safely underneath. + +Schema migration walks a schema's declared `Migration` list. It runs +each pending migration inside its own `BEGIN IMMEDIATE`/`COMMIT` pair. +It rolls back and throws `StorageError.migrationFailed` if any step in +that migration fails. `applyMigrations(_:)` is written to be safe to +call even on a brand-new SQLite file, one never opened through +`openSchema(_:)` first. This is a real case in practice, since at +least one caller calls `migrate(to:)` directly. It handles this by +re-running the safe-to-repeat table- and migrations-table-creation +steps before it checks what actually needs to migrate. + +Five row-changing methods start by calling `validateSQLIdentifier` +on every table and column name they are about to place into a SQL +string: `insertRow`, `upsertRow`, `updateRows`, `deleteRows`, and +`queryRows`. The file's comments tie this directly to a +named security fix, SECFIX-WS2-PK F9. Double-quoting a SQL name is not enough protection on its own. A name +might hold a double-quote character. That one character can escape +the mark and change the query. Every name is checked against a +strict allow-list before it is ever placed into a SQL string. +`insertRow` and `queryRows` are also where the at-rest row-encryption +seam actually runs. `insertRow` calls `encryptedForWrite` before +binding values. `queryRows` calls `decryptedForRead` after reading +them back. Both are no-ops on a plaintext or whole-database-encrypted +estate. + +`readColumn(stmt:index:schema:columnName:table:)` is the file's most +carefully reasoned function. It decides, column by column, whether a +raw SQLite value should be trusted as-is or treated as corrupt. The +comment draws a line between two cases. One is a type-tolerant decode, +a valid value stored under SQLite's flexible column affinity, which is +passed through as-is. The other is a genuine parse failure on a +column whose declared type says it should parse, such as an +unparseable UUID or time-stamp string. That second case throws +`StorageError.corruptStoredValue` rather than ever making up a random +UUID or an epoch-zero date in its place. +`queryRowsSkipCorrupt(...)` reuses the same column-reading logic but +catches exactly that error per row. It logs the error, counts the row, +and moves on to the next one. This cursor-level version can do +something `RowStore`'s protocol default can only approximate, which is +to discard an entire query's results. + +`storageStats(now:)` gives `StorageIntrospection` for SQLite by +reading a handful of read-only `PRAGMA` statements: `page_size`, +`page_count`, and `freelist_count`. It works out the WAL frame count +straight from the `-wal` side file's size on disk. It does not call +`PRAGMA wal_checkpoint` for this. That pragma can fail with a locked +error if some other job is in flight. This can happen even from +inside the same serializing actor. + +### SQLiteConnection.swift + +This file has `SQLiteConnection`, a thin Swift wrapper around the C +`sqlite3` API. It also has `SQLiteStatement`. This is the +prepared-statement wrapper used for every parameterized query. A +third piece, `ISO8601`, is the shared time-stamp formatting and +parsing tool. It runs everywhere a `.timestamp` `TypedValue` crosses +the SQLite boundary. + +Opening a connection does several things, in a specific order that +matters for safety. First, it checks whether the target path is a +symbolic link. It refuses to open the file if so. The comment names +this CAND-052, a defense against a pre-planted symlink that could +otherwise send SQLite's writes to some other file on disk. The check +uses `lstat` behavior, through `resourceValues(forKeys:)`, so it spots +the symlink itself rather than following it to whatever it points at. +Second, if the estate uses whole-database encryption, it issues +`PRAGMA key` with the estate's hex-coded key before any other +statement runs. This must be the very first statement. SQLCipher +cannot even read the schema on page one of the file without the +right key already applied. Third, it applies Apple's file-level Data +Protection, a best-effort step that layers OS-level at-rest +protection on top of SQLCipher's own encryption. It sets the durability pragmas last: WAL journal mode, `NORMAL` +synchronous durability, a WAL auto-checkpoint limit, a busy timeout, +and foreign-key rules. + +`ISO8601` deserves its own close look. Two very different stories +about speed and correctness live inside it. `string(from:)` first pins an out-of-range `Date` into the range +`ISO8601DateFormatter` can parse back: the years 0001 through 9999. +One example of an out-of-range date is one built by mistake from +milliseconds read as seconds. Writing an unparseable string like `+59009-...` would silently break +any future read of that row. A pinned value with a logged warning is +judged better than a value nothing can ever read back right. `date(from:)` first tries a hand-written, no-allocation parser: +`fastParseCanonicalUTC(_:)`. Only then does it fall back to the much +slower, ICU-backed `ISO8601DateFormatter`. The comment explains why the fast +path exists at all. A stack-sampling profile during a large import +showed the formatter's `date(from:)` using roughly eighty percent of +total CPU time. A Merkle rollup re-decodes every row's time stamp on +every insert, which makes the parse cost grow with the square of +import size. The fast parser knows only the exact canonical shape +this kit itself writes. It returns `nil` for anything even slightly +different: a numeric time-zone offset, lowercase letters, or trailing +junk. That sends control back to the slow, fully general formatters, +so correctness never slips. The fast path is checked byte-for-byte +against the formatters in its own test suite. + +### SQLiteStores.swift + +This file has `SQLiteRowStore`, `SQLiteBlobStore`, `SQLiteAuditLog`, +and `SQLiteTransaction`. These are four small wrapper types that each +forward to the shared `SQLiteBackend` actor. They give the actor's own +methods their public `RowStore`, `BlobStore`, `AuditLog`, and +`StorageTransaction` faces. `SQLiteRowStore.querySkipCorrupt(...)` is +the one method here with real logic of its own. Rather than falling +back to `RowStore`'s protocol-extension default, which can only throw +away an entire failed query, it calls +`SQLiteBackend.queryRowsSkipCorrupt(...)` directly. A corrupt row found +partway through a large corpus scan is skipped and logged on its own, +rather than aborting everything already read. + +### SQLiteSchema.swift + +This file has `SQLiteSchema`, the enum of pure functions that turn a +`SchemaDeclaration` into SQLite's own DDL. DDL stands for data +definition language, the SQL used to build and change tables, as +opposed to the SQL used to read and write rows. `nativeType(_:)` maps +every `ColumnType` case to its SQLite storage class. Most map +directly, but `.uuid` and `.timestamp` both map to `TEXT`, because +SQLite has no native type for either. `.hlc` maps to `INTEGER`, +because it is stored as a packed sixty-four-bit value. `.fingerprint` +maps to `BLOB`, because it is stored as raw bytes. `createTable(_:)` +puts together one `CREATE TABLE IF NOT EXISTS` statement. It covers +plain columns, generated columns, a primary key, and unique rules. +Generated columns are always rendered `STORED`, to match PostgreSQL's +lack of a `VIRTUAL` form. `appendOnlyTriggers(_:)` emits a `BEFORE UPDATE`/`BEFORE +DELETE` trigger pair for any table marked append-only. Each trigger +stops the statement outright with `RAISE(ABORT, ...)`. This is the +real enforcement behind, for example, the erasure ledger's +append-only rule. + +This file also owns the three bookkeeping tables every SQLite estate +carries, no matter what schema a caller declares. `_storagekit_migrations` +tracks each kit's applied schema version. `_storagekit_audit` is the +one source of truth for every audit event. It stores full-precision HLC columns alongside the packed integer +form. The packed form loses a bit of the physical-time field for +dates far enough in the future. The full-precision columns exist so that a cold rebuild +of an estate's last-known HLC never quietly disagrees with what a +snapshot first recorded. `_storagekit_blobs` is the flat key-and-bytes +table backing `SQLiteBlobStore`. + +### SQLitePredicateCompiler.swift + +This file has the translation from a `StoragePredicate` tree into a +parameterized SQLite `WHERE` clause, plus its ordered list of bound +values. `compile(_:)` is the public entry point. The private recursive +`render(_:bindings:)` walks the tree case by case. It checks every +column name it touches with `validateSQLIdentifier` before that name +is placed into the string. It appends every comparison value to the +bindings array, rather than ever placing a value directly into the +text. Values are always safe this way, because SQLite's own +parameter-binding step handles them. Column and table names cannot be +bound as parameters in SQL, which is exactly why they need the +separate identifier check instead. + +### SQLiteIdentifierValidator.swift + +This file has `validateSQLIdentifier(_:)`. Every write path and the +predicate compiler in this module call this one free function. They +call it before placing any caller-supplied name into a SQL string. The rule +is simple and strict. The first character must be a letter or an +underscore. Every character after that must be a letter, a digit, or +an underscore. This is the safe part of SQLite identifier syntax. The +file's own comment states why this exists as one shared function +rather than several private copies. A name holding a double-quote +character can escape the double-quote mark SQL uses around +identifiers. It can change what the query actually does. Double- +quoting alone is not a strong enough defense on its own. Having one +seam means the defense cannot quietly drift out of sync between the +several call sites that need it. + +### SQLiteObserver.swift + +This file has `SQLiteObserverRegistry`, an `actor` holding the +subscriber lists for both row-change and blob-change notices on one +SQLite estate. It also has `SQLiteObserver`, the public +`StorageObserver` conformance built on top of it. The file's own +comment explains a real limit of SQLite's native +`sqlite3_update_hook`. It fires for every table, including the +internal blob table. It only reports which operation happened on +which table and row ID. It never carries the actual column values. +Because a blob-change notice needs to carry the bytes that were +written, blob notices cannot be rebuilt from the hook at all. Instead, +`SQLiteBackend`'s `putBlob`/`deleteBlob` methods call this registry's +`notifyBlob(_:)` directly, at the exact point where the bytes are +already in hand. + +### KeychainKeyStore.swift + +This file has `KeychainKeyStore`. This is the Apple-platform source +of the whole-database encryption key, Mode 3. It is the Apple match +to the Rust port's per-estate key file on disk. One 256-bit key is stored per estate in the system Keychain. Its +account name is worked out in a fixed way from the estate's own file +path. +`estateAccount(for:)` makes the path uniform and hashes it with +SHA-256. Any process told the same estate's file path works out the +same account name this way. The main app and a managed background +server it starts are one such pair of processes. Both load the same +key with no separate key-sharing step needed. `loadOrCreateKey()` reads an +existing key if one is present. If not, it makes a fresh, randomly +generated key and stores it. It handles the case where two callers try to make the key at the +same time. The loser simply re-reads what the winner wrote. `deleteKey()` gets rid of an estate's +key for good when the estate itself is deleted. It is documented as +safe to call twice: calling it on an estate whose key is already gone +counts as success, not an error. The stored key's Keychain access +level, `afterFirstUnlockThisDeviceOnly`, is picked for a specific +reason. A background process can still read the key after the very first +unlock following a device restart. The key never leaves the device. +It never syncs to iCloud. + +## Target: PersistenceKitPostgreSQL + +### PostgreSQLStorage.swift + +This file has `PostgreSQLStorage`, the `Storage` conformance for the +server backend. It also has the `PostgreSQLBackend` actor. That actor +holds the schema and drives migrations, transactions, and health +checks against a connection pool. The most distinct design choice +here is estate isolation. Rather than one PostgreSQL database per +estate, every estate gets its own PostgreSQL schema. A schema, here, +is a namespace within one shared database. Each one is named +`pk_`. Every pooled connection for +that estate pins its `search_path` to that schema. `public` stays on +the path too, so any shared extension still resolves. This is the +PostgreSQL match to SQLite's one-file-per-estate model. Many estates +can share one database server without their tables ever colliding. + +Schema-version bookkeeping uses a simple key-value table, +`_storagekit_meta`, rather than a dedicated migrations table like +SQLite's. A per-kit version is stored under the mixed key +`"schema_version:"`. A running highest version is kept under +the plain key `"schema_version"`. This way, the no-argument +`currentSchemaVersion()` still returns something useful even when +several kits share one estate. `storageStats(now:)` gives +`StorageIntrospection` for PostgreSQL by asking three different +system views. It queries `pg_database_size`. It queries +`pg_stat_database`, for buffer-cache hit ratio and transaction +counters. It queries `pg_locks` joined against `pg_database`, for lock +contention. The file documents the exact formula it uses to turn each +view's raw counters into `StorageStats` fields. + +### PostgreSQLPool.swift + +This file has `PostgreSQLPool`, a fixed-size connection pool built as +an `actor`. Its bookkeeping never needs a separate lock this way. +That bookkeeping is open connections, in-use count, and waiters. +`acquire()` returns +an idle connection right away if one exists. It opens a brand-new one +if the pool has not yet reached its set size. Otherwise, it pauses the +caller on a `CheckedContinuation` until either a connection frees up +or a timeout passes. At that point, the waiter wakes back up with +`StorageError.poolExhausted`. Every freshly-opened connection is +pinned right away to the estate's own PostgreSQL schema. This happens +through `CREATE SCHEMA IF NOT EXISTS`, followed by `SET search_path`. +The connection is closed rather than kept if that two-step setup fails +partway through, so a half-set-up connection is never handed back to +a caller. + +`parseTLSMode(host:)` works out the pool's TLS behavior from the +`ARIA_MCP_POSTGRES_TLS` environment variable. It defaults to `prefer`, +meaning try TLS first and fall back to plaintext if the server does +not offer it, whenever the variable is missing or holds a value it +does not know. The comment is clear that even a loopback connection +defaults to `prefer` rather than `disable`. A caller who truly wants +plaintext on loopback has to say so on purpose, rather than getting it +by accident. + +### PostgreSQLConnection.swift + +This file has free functions that bridge between `PersistenceKit`'s +`TypedValue` and the `postgres-nio` client library's own binding and +decoding types. `executeSimple(_:logger:)` and +`executeParameterized(_:bindings:logger:)` are small `PostgresConnection` +extension methods. They wrap every underlying `postgres-nio` error in +`StorageError.backendError`, so a caller never has to catch a +`postgres-nio`-specific error type directly. `makeBindings(_:)` turns +an array of `TypedValue` into the positional `$1, $2, ...` bindings +PostgreSQL's parameterized-query protocol expects. It handles each +`TypedValue` case's own wire shape. Take `.fingerprint`, for example. +It is written out as thirty-two raw bytes in a fixed block order. This +makes it round-trip exactly. + +`decodeRow(_:columns:)` and the private `decodeCell(_:type:)` do the +reverse. Given a returned `PostgresRow` and the caller's expected +column types, they decode each named cell into the matching +`TypedValue` case. If a cell fails to decode as its declared type, +they fall back to `.null` rather than throwing. This is a more open +stance than the SQLite backend's `corruptStoredValue` throw. It +reflects that PostgreSQL's own wire protocol already enforces column +types at a lower level than SQLite's flexible column affinity does. + +### PostgreSQLPredicateCompiler.swift + +This file has the PostgreSQL match to `SQLitePredicateCompiler`: the +same `StoragePredicate` tree compiled to PostgreSQL's own SQL dialect. +It uses `$1, $2, ...` numbered spots instead of SQLite's `?` +placeholders. The case-by-case translation is otherwise the same +shape as the SQLite compiler. Every column name is checked with +`validatePSQLIdentifier` before it is placed into the string. Every +comparison value becomes a bound parameter rather than inline text. +One PostgreSQL-only detail stands out. Two bitmask predicate cases +work this way: `bitmaskAll` and `bitwiseEq`. Both must point at two +already-appended bindings by their final number: `bindings.count - 1` +and `bindings.count`. This is because PostgreSQL's numbered spots are +numbered by the order they were bound, unlike SQLite's plain `?` +placeholders, which carry no number at all. + +### PostgreSQLSchema.swift + +This file has `PostgreSQLSchemaEmitter`, the PostgreSQL match to +`SQLiteSchema`: pure functions that turn a `SchemaDeclaration` into +PostgreSQL DDL. `typeSQL(_:)` maps each `ColumnType` to its own +PostgreSQL native type. `.uuid` maps to a real `UUID` column, unlike +SQLite's `TEXT` fallback. `.timestamp` maps to `TIMESTAMPTZ`. +`.json` maps to `JSONB`, PostgreSQL's own binary, searchable JSON +type. PostgreSQL has no `CREATE TRIGGER IF NOT EXISTS`. +`appendOnlyTriggerStatements(_:)` gets the same repeat-safe behavior +SQLite gets for free. It first drops any existing trigger of the same +name, then creates it fresh. One shared `appendOnlyFunctionSQL` +trigger function backs every append-only table in the schema, rather +than one function generated per table. It is created with `CREATE OR +REPLACE`, so running schema setup again never fails. It raises an +error naming the table at fault, through PostgreSQL's `TG_TABLE_NAME` +variable. + +### PostgreSQLStores.swift + +This file has three concrete conformances for the PostgreSQL +backend: `PostgreSQLRowStore`, `PostgreSQLBlobStore`, and +`PostgreSQLAuditLog`. These match `RowStore`, `BlobStore`, and +`AuditLog`. Each one can run +either against a pooled connection grabbed for a single call, or +against a `PostgreSQLTransactionContext`'s already-open connection +when called from inside a transaction block. Every write method here +checks SQL names with `validatePSQLIdentifier`, exactly as the SQLite +backend does, before it puts together its SQL string. + +`PostgreSQLRowStore.query(...)` reads the schema's declared column list +for the target table, rather than issuing a bare `SELECT *`. This is +so a generated column's PostgreSQL-computed value is decoded with the +right `ColumnType`. It also runs every returned row's values through +`decryptedForRead` before wrapping them in a `StorageRow`. +`update(...)` and `delete(...)` both compile their predicate starting +at the parameter spot right after the last `SET`-clause binding. They +use the private `renderPredicate(_:startIndex:bindings:)` helper. This +helper compiles the predicate as usual. It then renumbers every `$N` +placeholder in reverse order, high numbers first. It does this on +purpose, to avoid `$10` being wrongly rewritten by a simple forward +swap of `$1`. +`PostgreSQLBlobStore` and `PostgreSQLAuditLog` each lazily build their +own backing table on first use, through `CREATE TABLE IF NOT EXISTS`. +The tables are `_storagekit_blobs` and `_storagekit_audit`. Neither one needs the +caller's own schema to know about them. This is the same design +`PersistenceKit` uses for SQLite's own internal tables. +`decodeAuditEvent(_:)` decodes the audit row's required fields in a +strict way, so a decode failure there passes on as a thrown error. It +treats optional before-state fields in a looser way, with `try?`. +`NULL` is the expected, valid value for "no prior state," and that +value is always safe there. + +### PostgreSQLIdentifierValidator.swift + +This file has `validatePSQLIdentifier(_:)`, the PostgreSQL-module +match to `SQLiteIdentifierValidator.swift`'s `validateSQLIdentifier`. +It enforces the exact same rule. The first character must be a +letter or underscore. Every later character must be a letter, a +digit, or an underscore. The file's own comment names this as one of three +copies that enforce this same rule on their own. The other two are +the SQLite module's copy and the Rust port's `validate_sql_identifier` +function. Three copies exist by need. Swift extensions cannot share a +free function across two separate targets without a shared dependency +neither module otherwise needs. Each copy is still the single seam +within its own module, so the rule cannot quietly drift within a +module. + +## Target: PersistenceKitReplication + +### ReplicationTypes.swift + +This file has the two public types every other file in this module +builds on. `ReplicationCursor` is the opaque watermark a caller stores +and passes back into a later incremental sync. It carries the highest HLC seen across every copied row and audit +event. It also carries counts of rows, audit events, and blobs +written during that run. `ReplicationError` is +the closed error type for replication on its own. `schemaMismatch` +fires when source and destination disagree on schema version or kit +ID. Replication on purpose refuses to auto-migrate either side. +`storageFailure` wraps an underlying `StorageError` surfaced during +the copy, re-wrapped as a plain string. This lets `ReplicationError` +itself stay `Equatable`, which a raw `Error` cannot always guarantee. + +### StorageReplicator.swift + +This file has `StorageReplicator`, the full-snapshot replication +tool. `replicate(from:to:schema:)` is the core function. Two +direction-named helpers sit on top of it: `flush(from:into:schema:)` +moves in-memory data into durable storage, and `hydrate(into:from:schema:)` +moves durable storage into a fresh in-memory copy. "Full snapshot" +means exactly what it says. Every run copies everything, no matter what changed since the last +run. This means every row in every schema-declared table, every audit +event, and every blob. The function still stays even on +repeat runs, though. Every row upsert uses the table's own primary +key as its conflict column, not the random UUID a fresh `RowHandle` +would carry. Running the same flush twice against an unchanged source +updates existing target rows in place, rather than ever +inserting a second copy. + +The work runs in three ordered steps. Step one is a schema gate. +Source and destination must report the exact same per-kit schema +version, or the function throws `ReplicationError.schemaMismatch` +right away, rather than trying any copy at all. Step two reads the +entire source into memory: every table's rows, every audit event, +and every blob. Any generated columns are filtered out of each row +first, because writing a value into a column the target computes +on its own would be turned down by both SQL backends. All of this +goes into one `Sendable` in-memory payload, before the target +transaction ever opens. The +file's comment explains this order. It exists so the target is +never left holding a long-lived serializable transaction open while +slow source work runs underneath it. Step three writes that entire +payload into the target inside one plain transaction. A +crash or a thrown error partway through therefore leaves the +target exactly as it was before the flush began. A blob-deletion +pass at the end of step three removes any target blob key absent +from the source snapshot. This closes a gap the file names by its fix +ID, SECFIX-WS2-PK F5. Say this pass did not exist. A blob deleted at +the source would linger forever in the target, after every later +full-snapshot flush. A copy that only ever adds rows never notices +that something is missing. + +### IncrementalReplicationSession.swift + +This file has `IncrementalReplicationSession`, the observer-driven +alternative to a full snapshot. Instead of copying everything on every +run, it watches a source's live change stream. On demand, it syncs +only the rows and blobs that truly changed since the last sync. The +file's opening comment lays out the design choice in plain terms. +A durable dirty-row table, written on every observed change, would +tie this backend-agnostic module to one particular storage schema. +The session avoids that. It gathers dirty IDs purely in memory +instead, inside two actors: `DirtySet` for rows and `BlobDirtySet` +for blobs. It re-reads each dirty row's current state from the source +at sync time. It does not trust whatever the original change notice +said. The row may have changed again since it was first marked dirty. +If the process restarts, the in-memory dirty set is lost. The fix for +that, as the file notes, is simple. Fall back to a full snapshot. That +always works as a safe stand-in for a lost incremental history. + +`DirtyKey` names one dirty row by its table name and its +primary-key values, packed into a sortable string. Draining the dirty +set therefore always produces the same order of work across repeated +runs. This fixed order is what makes two different processes +independently syncing the same dirty set produce an identical, and +so safely repeatable, run of target writes. +`DirtySet.accumulate(_:)` pulls the primary-key columns out of an +observed `TableChange`'s values. If a conforming-but-buggy backend +ever emits a change missing one of those columns, the function logs +it and skips it, rather than crashing. `BlobDirtySet` tracks the same idea for +blobs. A blob change notice already carries its own payload bytes, so +`BlobDirtySet` never needs to re-read anything from the source at +sync time. A `put` replaces an earlier `put` for the same key. A +`delete` replaces a `put`. The last write always wins. + +`sync(from:to:fromCursor:)` is the operation a caller actually calls. +It repeats the same schema gate `StorageReplicator` uses. It drains +both dirty sets. It re-scans each dirty row from the source. A +now-missing row turns into a target delete rather than an upsert. +This happens because the row was deleted at the source between the +original change notice and this re-scan. It fetches only the audit events newer than +the previous cursor's watermark. It commits everything inside one +plain target transaction, all in one go, the same fail-loud +shape `StorageReplicator` uses for a full snapshot. The +retry-keeping rule is the one piece unique to the incremental path. +Say anything fails after the dirty sets have already been drained. The +drained keys and blob operations go back into the sets. This restore +is awaited right there in the `catch` block. It is never handed off +to a detached background task. A fire-and-forget restore could race a +caller's immediate retry. It could quietly lose the very keys it was +trying to keep safe. A later retry then tries again on exactly the +rows and blobs the failed run did not finish. + +## Rust Port and Conformance + +The `rust/` folder holds a second build of `PersistenceKit`'s design, +for use outside the Swift and Apple world. It has the same closed `StoragePredicate` algebra and the same +`TypedValue` case set. It also has the same trait surface: `Storage`, +`RowStore`, `BlobStore`, `AuditLog`, and `StorageObserver`. It has all +three backends too: +`InMemoryStorage`, `SqliteStorage`, and `PostgresStorage`. It adds four things on top of these: the caching decorator, the +hashing decorator, the encryption seam, and both replication modes. +The Rust traits are plain. They use `Result` rather +than `async`. This is because the Rust backends do no real async +work of their own. The port's own `README.md` notes that Swift's `async` need comes +from Swift actors in particular. A future backend that does need +async work of its own could wrap its own runtime without changing the +trait shape. + +Unlike LatticeLib's two legs, the Swift and Rust sides of +`PersistenceKit` are not gated by a shared, byte-for-byte conformance- +fixture set. `PersistenceKit`'s cross-platform contract is a shared +protocol shape and a shared wire format: the same `TypedValue` cases, +the same predicate tree, the same identifier-checking rule. It is not +a promise that the same input always produces bit-identical output on +both legs. Each side runs its own test suite. The Rust SQLite and +PostgreSQL backends each run a ten-test conformance set covering +schema, rows, predicates, blobs, audit, generated columns, +transactions, append-only rules, and health checks. These tests run +straight against the Rust trait code, not cross-checked against the +Swift package's own test output. diff --git a/packages/kits/PersistenceKit/docs/INTERFACE_DOCTRINE.md b/packages/kits/PersistenceKit/docs/INTERFACE_DOCTRINE.md deleted file mode 100644 index 1f5c284..0000000 --- a/packages/kits/PersistenceKit/docs/INTERFACE_DOCTRINE.md +++ /dev/null @@ -1,273 +0,0 @@ -# PersistenceKit Interface Doctrine - -For coding agents implementing kits that consume PersistenceKit (LocusKit, VectorKit, CorpusKit, GeniusLocusKit, anything else). This document is the contract. - -If you violate the doctrine, the abstraction breaks and the kit graph rots. Read it before writing code. - -## 1. Always go through protocols - -Consume `any RowStore`, `any BlobStore`, `any VectorIndex`, `any AuditLog`, `any Storage`. Never reference `InMemoryStorage`, `SQLiteStorage`, or `PostgreSQLStorage` from a downstream kit. Backend selection is the application's job, not the kit's. - -```swift -// CORRECT -final class LocusKit { - let storage: any Storage - init(storage: any Storage) { self.storage = storage } -} - -// WRONG -import PersistenceKitSQLite // never in a downstream kit -final class LocusKit { - let storage: SQLiteStorage // never -} -``` - -## 2. Declare your schema once, in code - -Every kit owns one `SchemaDeclaration`. Build it as a `let` constant inside the kit. Pass it to `Storage.open(schema:)` at estate-open time. PersistenceKit emits backend-specific DDL. - -```swift -public enum LocusKitSchema { - public static let declaration = SchemaDeclaration( - kitID: "LocusKit", - version: 1, - tables: [ - TableDeclaration( - name: "drawers", - columns: [ - .uuid("row_id"), - .bitmap("adjective"), - .bitmap("operational"), - .bitmap("provenance"), - .text("verbatim"), - .timestamp("captured_at"), - .int("udc_code"), - .int("qid_pointer", nullable: true) - ], - primaryKey: ["row_id"] - ) - ], - indices: [ - IndexDeclaration(name: "idx_drawers_adjective", table: "drawers", columns: ["adjective"]), - IndexDeclaration(name: "idx_drawers_captured_at", table: "drawers", columns: ["captured_at"]) - ] - ) -} -``` - -Schemas are append-only across versions. Bump `version` and add a `Migration` when you change anything. Never edit a `TableDeclaration` already in production. - -## 3. Never write raw SQL - -If you find yourself building a SQL string, stop. The work belongs in `StoragePredicate` or `OrderClause`. The backend compiles. You declare intent. - -```swift -// CORRECT -let active = try await storage.rowStore.query( - table: "drawers", - where: .and([ - .bitmaskAll(Column(table: "drawers", name: "operational"), mask: 0x01), - .bitmaskNone(Column(table: "drawers", name: "operational"), mask: 0x80) - ]), - orderBy: [OrderClause(column: Column(table: "drawers", name: "captured_at"), direction: .descending)], - limit: 50, - offset: nil -) - -// WRONG -let active = try connection.prepare("SELECT * FROM drawers WHERE ...") -``` - -The exception: if you need an operation that StoragePredicate cannot express, propose a new case in the closed enum via a decision record. Do not work around it with raw SQL. - -## 4. Use TypedValue exclusively - -Cross every kit boundary with TypedValue, not native Swift types. The encoding is the backend's problem. - -```swift -// CORRECT -values: [ - "captured_at": .timestamp(Date()), - "row_id": .uuid(rowID), - "adjective": .bitmap(0x01) -] - -// WRONG -values: [ - "captured_at": Date(), // not a TypedValue - "row_id": rowID.uuidString, // backend type leakage - "adjective": 1 // ambiguous between .int and .bitmap -] -``` - -`.bitmap(_)` and `.int(_)` are semantically distinct even though both are Int64. Use `.bitmap` for the three bitmap columns; `.int` for everything else. - -## 5. Atomic work uses transactions - -Operations that must commit together go inside `storage.transaction { ... }`. The block receives a `StorageTransaction` whose sub-stores share the same connection. Use them; do not call `storage.rowStore` from inside the block (that would acquire a separate connection). - -```swift -try await storage.transaction { txn in - let handle = try await txn.rowStore.insert(table: "drawers", values: ...) - try await txn.auditLog.append(captureEvent(for: handle)) -} -``` - -If the block throws, the transaction rolls back atomically. If it returns, the transaction commits. - -The capture verb crosses rowStore and auditLog; it always uses a transaction. Likewise for mutate, withdraw, expunge. - -## 6. Audit events get a fresh eventID per emit - -Every `AuditEvent` you construct gets a fresh `UUID()` for `eventID`. Never reuse an eventID; never derive it from row state. The compound key `(eventID, hlc)` makes append idempotent at the storage layer; ConvergenceKit relies on this property. - -```swift -// CORRECT -let event = AuditEvent( - eventID: UUID(), // fresh - estateUuid: estateID, - rowId: handle.key, - hlc: hlcGenerator.advance(), - verb: "capture", - // ... -) -try await txn.auditLog.append(event) -``` - -HLC comes from SubstrateLib's HLCGenerator. One generator per estate. Generators are monotonic; do not use raw timestamps. - -## 7. Never reach inside another kit's tables - -Each kit owns its tables. LocusKit owns `drawers`, `tunnels`, `kg_facts`, etc. VectorKit owns `rag_vectors`. CorpusKit owns `chunks`. Do not write SQL or queries that read another kit's tables from your kit. If you need cross-kit data, ask via that kit's API. - -The two tables PersistenceKit owns are internal and start with `_storagekit_`: `_storagekit_meta`, `_storagekit_blobs`, `_storagekit_audit`, `_storagekit_vectors`, `_storagekit_vector_meta`. Never touch them from downstream code. - -## 8. Test against InMemory in CI; SQLite for parity - -Per-kit test suites use `PersistenceKitInMemory` for speed (in-memory, no disk, no extension loading). Add `PersistenceKitSQLite` parity tests for any code path that depends on backend behavior (predicate semantics, ordering, vector distance). PostgreSQL tests are gated on `POSTGRES_TEST_URL`. - -```swift -func makeStorage() -> any Storage { - InMemoryStorage(configuration: EstateConfiguration( - estateID: UUID(), - backend: .inMemory - )) -} -``` - -When the conformance fixture suite lands (mission 2 final piece), your kit will run a small set of conformance fixtures against every backend. - -## 9. Migrations are forward-only - -When you change your schema, bump `version` and append a `Migration` with the operations to reach the new version. Never edit an existing migration. Never remove a migration. - -```swift -public static let declaration = SchemaDeclaration( - kitID: "LocusKit", - version: 2, // bumped from 1 - tables: [...], // current target shape - migrations: [ - Migration(fromVersion: 1, toVersion: 2, operations: [ - .addColumn(table: "drawers", column: .text("dialect", nullable: true)) - ]) - ] -) -``` - -A failed migration leaves the schema at the last successfully-applied version. Operators check `currentSchemaVersion()` (global max) or `currentSchemaVersion(for: kitID)` (per-kit) after failure. In multi-kit deployments, use the kitID-scoped variant to avoid misdiagnosing failure in one kit as failure in another. - -## 10. Bitmap predicate semantics - -Three operators cover spec §7.9: - -- `.bitmaskAll(col, mask: M)` → `(col & M) == M`. "all bits in M are set" -- `.bitmaskAny(col, mask: M)` → `(col & M) != 0`. "at least one bit in M is set" -- `.bitmaskNone(col, mask: M)` → `(col & M) == 0`. "no bits in M are set" - -Plus `.bitwiseEq(col, expected: E, mask: M)` → `(col & M) == E` for stateful bit-pattern matches. - -Compose with `.and([...])` and `.or([...])`. The mandatory filter ordering from spec §7.9.5 (tombstone exclusion, default state filters, then user predicates) is enforced by the kit constructing the predicate, not by PersistenceKit. Wrap user-supplied predicates: - -```swift -let final: StoragePredicate = .all([ - // 1. tombstone exclusion (always first) - .bitmaskNone(opCol, mask: TOMBSTONE_BIT), - // 2. default state filter (active rows) - .bitmaskAll(opCol, mask: ACTIVE_BIT), - // 3. user predicate - userPredicate -]) -``` - -## 11. Vector index conventions - -Vector dimensionality is fixed at first `add`. All subsequent vectors must match. To change dimensionality, drop and recreate. - -Distance metrics: `.cosine`, `.l2`, `.dot`. SQLite vec0 returns L2 by default regardless of metric requested; the abstraction tolerates this for v1.0 since callers normalize on the way in. PostgreSQL pgvector honors the metric per query. - -Vector metadata is per-vector key/value, encoded as JSON internally. Filter predicates on metadata work but currently run in-memory after the k-NN result returns (JSONB column-level predicate compilation is a v1.x improvement). Use metadata filters sparingly when k is large. - -## 12. EstateConfiguration is opaque - -The kit consuming Storage never inspects `configuration.backend`. The estate handle is opaque; the protocols are the contract. If your kit needs to know "am I on SQLite or PostgreSQL?", you have a layering bug. - -If your kit needs a behavior that differs by backend (e.g. concurrency tuning), that's a missing protocol method. Propose it via decision record. - -## 13. Sendable everywhere - -Every type that crosses an actor boundary or escapes into an `async` context must be Sendable. PersistenceKit's public types all are. Your downstream types must be too. Use `@unchecked Sendable` with a documented rationale (locking, immutable-after-init, etc.) only when needed. - -## 14. StorageObserver - -`Storage.observer` exposes change notifications. Subscribe to a table by event set; you receive an `AsyncStream` that fires on commits. Multiple subscribers on the same table coexist. - -```swift -let stream = storage.observer.observe(table: "jobs", events: [.insert]) -for await change in stream { - // wake up; do the work - guard let key = change.rowKey else { continue } - handleNewJob(key) -} -``` - -Use cases at v1.0: - -- QueueKit's `watch()` for filesystem-backed and PersistenceKit-backed queues -- GeniusLocusKit Brain layer standing signals waking on audit log appends -- ConvergenceKit replicating row-level changes outbound (when sync is enabled on the PersistenceKit instance) - -Backend semantics: - -- **InMemory**: notifications are reliably delivered to all matching subscribers -- **SQLite**: per-row notifications for `insert` and per-row notifications for `upsert`; `update` and `delete` fire coarse "something changed" notifications (rowKey may be nil) because bulk operations don't compose into per-row events without query rewriting -- **PostgreSQL**: NoOpObserver in v1.0; LISTEN/NOTIFY integration is v1.x - -Delivery is at-least-once. Subscribers must tolerate seeing a change after the row has already been deleted (race). - -Writes do not block on subscribers. AsyncStream uses `bufferingOldest(1024)`; slow subscribers may miss events under load. - -## 15. Error handling - -PersistenceKit throws `StorageError` for backend-attributable failures. Your kit can map these to kit-specific errors but should preserve the underlying cause: - -```swift -do { - try await storage.rowStore.insert(table: "drawers", values: ...) -} catch let error as StorageError { - throw LocusKitError.captureFailed(underlying: error) -} -``` - -Do not swallow StorageError. The error type carries diagnostic information operators need. - -## 16. When in doubt, file a decision record - -If you find yourself wanting to: - -- Add a case to TypedValue, ColumnType, StoragePredicate, or any closed enum in PersistenceKit -- Add a new protocol method to RowStore, BlobStore, VectorIndex, AuditLog, or Storage -- Change the audit log compound key -- Add a backend-specific escape hatch -- Treat one backend differently from another in downstream code - -Stop. Write a decision record in `docs/decisions/` proposing the change. The closed-enum design depends on every change being deliberate. diff --git a/packages/kits/PersistenceKit/docs/OVERVIEW.md b/packages/kits/PersistenceKit/docs/OVERVIEW.md new file mode 100644 index 0000000..f687bd6 --- /dev/null +++ b/packages/kits/PersistenceKit/docs/OVERVIEW.md @@ -0,0 +1,300 @@ +--- +doc: OVERVIEW +package: PersistenceKit +repo: moot-system +authored_commit: 909513d0a8ecb1e9e903af9f4d25b3e4f2528242 +authored_date: 2026-07-04 +sources: + - path: Sources/PersistenceKit/AuditLog.swift + blob: ca4c0c9623056a49ad888a7a77e720a7519556bb + - path: Sources/PersistenceKit/BlobStore.swift + blob: f052c3ecc693d5414d57ad2c6da5fb4c5fe28f79 + - path: Sources/PersistenceKit/CacheInvalidator.swift + blob: 0a844b8037404dd72d08df6887ceee1e8c6014f2 + - path: Sources/PersistenceKit/CachingRowStore.swift + blob: 7edf9a86eb31dd9a17abd97b7baa9e8d5425e266 + - path: Sources/PersistenceKit/Column.swift + blob: 3cfc3ba856ebe15e8756ae65dae736cbd2c288a6 + - path: Sources/PersistenceKit/EncryptionMode.swift + blob: fdb7aa7ec63088ae09739f5e2a8e1468c386b987 + - path: Sources/PersistenceKit/ErasureLedger.swift + blob: eb2ca35f69783eead6049949bbaad68decfc9915 + - path: Sources/PersistenceKit/ErasureOverlay.swift + blob: d5a86f2baa10647134f285d6996a8e62cbe23ace + - path: Sources/PersistenceKit/EstateCacheConfig.swift + blob: 7003951cb5b2b075b6bb0c49b5fe372a97bfeb3a + - path: Sources/PersistenceKit/EstateConfiguration.swift + blob: be91569405e5c48a99859b4595540a59bd1a1994 + - path: Sources/PersistenceKit/GCPin.swift + blob: 73336dc4c693fcfa285a7eff8ed6520f27951ff8 + - path: Sources/PersistenceKit/GeneratedColumn.swift + blob: 34e466c80a0e9aac5d24303eaad132901333470b + - path: Sources/PersistenceKit/HashingRowStore.swift + blob: fe233ed2f4fda177373ad105d14fa321d325d0df + - path: Sources/PersistenceKit/NoOpObserver.swift + blob: 99f62b6889df50c5f09fce749f363637891f116c + - path: Sources/PersistenceKit/NovelTokenTaggerChoice.swift + blob: 9588d6303431883a2c8e0ee727f7ebe3e3f6d139 + - path: Sources/PersistenceKit/PersistenceKitTelemetry.swift + blob: a24e54dfb97beb86dac9422bbfc7cc3c2d959d73 + - path: Sources/PersistenceKit/Predicate.swift + blob: 522802c842dfc7573137efd7a3f36300d5201468 + - path: Sources/PersistenceKit/RowCrypto.swift + blob: 6678e426c0902cdae6d164904001526756e40916 + - path: Sources/PersistenceKit/RowStore.swift + blob: 9a96ff89528ca786d51224324cacb78cf810037d + - path: Sources/PersistenceKit/Schema.swift + blob: 5a6894c68d6f29eecfe9acc34d7c06ee2a655ac3 + - path: Sources/PersistenceKit/SnapshotRegistry.swift + blob: 88087bda544165c4d24c514a4f3e0641a620512e + - path: Sources/PersistenceKit/Storage.swift + blob: 7484a40913b28cb11a6bd2e3ea822dc8fe8eb63e + - path: Sources/PersistenceKit/StorageError.swift + blob: 743c2d1a24c7bafedb217e4c6bbf30ca20d03be8 + - path: Sources/PersistenceKit/StorageIntrospection.swift + blob: 35522b6601037246907bb88ebb2fb2eb5ea9b0e2 + - path: Sources/PersistenceKit/StorageObserver.swift + blob: f0a8b61a15344e02193137a3393be345dd51cf25 + - path: Sources/PersistenceKit/Transaction.swift + blob: d58d478618b31cc3985275be6ceee84a6fc222de + - path: Sources/PersistenceKit/TypedValue.swift + blob: ec124b7aebd86f64f67fea6656396d374666b741 + - path: Sources/PersistenceKitInMemory/InMemoryAuditLog.swift + blob: 1ba83d2408d3384d4d8e4f286fa2db743a433cc9 + - path: Sources/PersistenceKitInMemory/InMemoryBlobStore.swift + blob: e02822bde83c899b6d18ab8d82b90afba6abb5ac + - path: Sources/PersistenceKitInMemory/InMemoryObserver.swift + blob: 71b98c9da2d869a81a78265b7ef1f64a57a907af + - path: Sources/PersistenceKitInMemory/InMemoryRowStore.swift + blob: 2f7c92612c46b8a097e4211147c924f8356e47d1 + - path: Sources/PersistenceKitInMemory/InMemoryStorage.swift + blob: 9820c771bfe39ab738eea1f3b1623491fcfb1326 + - path: Sources/PersistenceKitInMemory/PredicateEvaluator.swift + blob: e8735422ff87addff6a2a3a89da85c495d07f07c + - path: Sources/PersistenceKitPostgreSQL/PostgreSQLConnection.swift + blob: 5a282a6064a69a543a5d650ccc7eeeae2c5a3e4f + - path: Sources/PersistenceKitPostgreSQL/PostgreSQLIdentifierValidator.swift + blob: 120cf0c1576a7db45d79bf6865e2f15cff09a5ac + - path: Sources/PersistenceKitPostgreSQL/PostgreSQLPool.swift + blob: eddc0c3af4135565e15d2d763b0d24240973dd6f + - path: Sources/PersistenceKitPostgreSQL/PostgreSQLPredicateCompiler.swift + blob: 6f0791e0372b09cf2605bae4cf149e48bbba2834 + - path: Sources/PersistenceKitPostgreSQL/PostgreSQLSchema.swift + blob: f165f0877b96c87e2f7de46012f8f8acb3c03cc8 + - path: Sources/PersistenceKitPostgreSQL/PostgreSQLStorage.swift + blob: 849926f9e6def788fae2f38757ad44bcd52a18a0 + - path: Sources/PersistenceKitPostgreSQL/PostgreSQLStores.swift + blob: bff90838d1324d82ae76a0895f15399537e9e343 + - path: Sources/PersistenceKitReplication/IncrementalReplicationSession.swift + blob: 2f90378146e043d75c739d400d98c7770e07af78 + - path: Sources/PersistenceKitReplication/ReplicationTypes.swift + blob: bb13c63d1febd50ad46c1814610d7f6c31a33112 + - path: Sources/PersistenceKitReplication/StorageReplicator.swift + blob: f5acc5993c8647e53c127c6871a1282c24b1c427 + - path: Sources/PersistenceKitSQLite/KeychainKeyStore.swift + blob: 0071732291a7cb6ce0777bd230a6188276fb4f32 + - path: Sources/PersistenceKitSQLite/SQLiteConnection.swift + blob: ece56dc7e25e67656bb37f5222c18c1166c750cc + - path: Sources/PersistenceKitSQLite/SQLiteIdentifierValidator.swift + blob: 713339c137d6af1cfbba5a3584e05bdda70b42c5 + - path: Sources/PersistenceKitSQLite/SQLiteObserver.swift + blob: 2cb61ed75dae5ab81a3cdfb5c9581731d25fd960 + - path: Sources/PersistenceKitSQLite/SQLitePredicateCompiler.swift + blob: 5770adb2024c54ea6921651632ca65ee84d416af + - path: Sources/PersistenceKitSQLite/SQLiteSchema.swift + blob: 4ba6fc175fe9d486b17af1088a23da64041b12ae + - path: Sources/PersistenceKitSQLite/SQLiteStorage.swift + blob: 417e63cc07b60295ac874187ebb796bf9f3b3c86 + - path: Sources/PersistenceKitSQLite/SQLiteStores.swift + blob: 76499ffb70d979f0d13e8cf9e32bc38ff28ffdb5 +--- + +# PersistenceKit Overview + +## What This Library Does + +PersistenceKit is the storage layer for MOOTx01. MOOTx01 is an on-device +AI memory system. It stores what an AI observes over time. It later +helps the AI recall what it observed. Every fact, drawer, and audit +record a MOOTx01 estate keeps passes through PersistenceKit first. Only +then does the record reach a disk or a database. An estate is one +user's complete memory store in MOOTx01. PersistenceKit does not decide +what a memory means. That job belongs to a higher kit. PersistenceKit +decides how a memory is written, read, and kept safe. It takes over +once another kit hands the memory over as typed rows and bytes. + +PersistenceKit gives every higher kit the same four operations. A kit +can insert a row. A kit can fetch a blob. A kit can append an audit +event. A kit can watch a table for changes. This holds true no matter +which physical engine sits underneath. The engine can be SQLite on a +phone. It can be PostgreSQL on a shared server. It can also be a plain +in-memory table for a unit test. The calling kit writes the same code +in every case. + +## The Problem It Solves + +MOOTx01 runs in more than one place. A single estate might live only +on one iPhone. It might instead run against a shared PostgreSQL server +for a managed deployment. It might exist only for the length of a +test. Without a storage abstraction, every kit that touches memory +would need to know three different database APIs. Each API brings its +own quirks. Worse, a kit built against SQLite could not point at +PostgreSQL without a rewrite. + +PersistenceKit solves this with one typed contract: the `Storage` +protocol. Three backends satisfy that contract: `PersistenceKitSQLite`, +`PersistenceKitPostgreSQL`, and `PersistenceKitInMemory`. A caller writes +against `Storage`, `RowStore`, `BlobStore`, and `AuditLog` exactly once. +Swapping the backend then means changing one line of configuration. The +calling code itself never changes. + +One interface across three engines is still not enough on its own. +Real deployments also need at-rest encryption. They need tamper-evident +audit trails. They need safe deletion of sensitive content. They need a +cache that never lies about what storage truly holds. They need a way +to copy an entire estate from one backend to another. PersistenceKit +builds all of these as layers on top of the same four-operation core. +A kit that only needs plain rows never pays for features it does not +use. A kit that needs encryption or replication gets it without +leaving the same protocol surface. + +PersistenceKit deliberately does not own vector similarity search. A +separate library, VectorKit, owns dense-embedding nearest-neighbor +search instead. PersistenceKit's job toward that workload stays narrow. +Every backend must support the storage needs a vector index has. It +must store a vector payload in a row. It must read many rows back in +bulk. It must count rows and delete rows. Call this the accommodation +contract. +PersistenceKit accommodates the workload. It does not implement the +search itself. + +## How It Works + +One protocol sits at the center of the library: `Storage`. A type +conforms to `Storage` by providing four sub-stores. `RowStore` handles +typed rows. `BlobStore` handles raw bytes keyed by string. `AuditLog` +keeps an append-only history of changes. `StorageObserver` delivers live +change notifications. A conforming type also manages schema and +transactions. `EstateConfiguration` tells a `Storage` implementation +which backend to open. It also sets the encryption mode and the cache +settings to use. + +A schema in PersistenceKit is not raw SQL. It is a typed Swift value +called `SchemaDeclaration`, built from `TableDeclaration` and +`ColumnDeclaration` values. A kit that wants to store data describes +its tables once, as Swift structs. Each backend then translates that +description into its own native form. SQLite gets `CREATE TABLE` +statements. PostgreSQL gets `CREATE TABLE` statements in its own +dialect. The in-memory backend just allocates a dictionary. Queries +follow the same pattern. `StoragePredicate` is a closed tree of +comparison and bitmap operators. Each backend compiles that tree to its +own query language. No caller ever writes a raw SQL string. + +Three backends satisfy `Storage`. `PersistenceKitSQLite` is the +on-device engine. It uses one file per estate, encrypted at rest +through a vendored SQLCipher build when an estate asks for +whole-database encryption. `PersistenceKitPostgreSQL` is the server +engine, built on the `postgres-nio` client. It gives each estate its +own PostgreSQL schema, so many estates can share one database server +without their tables colliding. `PersistenceKitInMemory` is the test +and prototyping engine. It keeps no file and opens no network +connection, and it disappears when the process exits. + +The core protocol also carries decorators and free-function toolkits. +Any backend can use these without changing its own code. + +- `CachingRowStore` wraps a `RowStore` with an in-memory hot tier. Rows + above a configured sensitivity level are never cached. The cache + guarantees that every read returns exactly what the backing store + would return. Caching only changes speed, never correctness. +- `HashingRowStore` wraps a `RowStore` and computes a content hash on + every write to a table marked hashable. It then reports the write up + a parent chain, so a Merkle-style integrity tree can stay current + without a full rescan. +- `RowCrypto` and its write and read seam functions apply per-row + AES-GCM encryption to a table's `content` column. This runs whenever + an estate is configured for row-level encryption, independent of + whichever backend stores the row. +- `ErasureLedger` and `ErasureOverlay` implement right-to-be-forgotten + deletion. The ledger records that a piece of content was erased. The + overlay nulls that content out of every read afterward. It fails + closed, dropping the row, if the erasure check itself cannot + complete. +- `SnapshotRegistry` and `GCPin` let a kit take a named, attested + snapshot of the estate's state. They also pin the oldest live + snapshot's timestamp, so a later garbage-collection pass never + deletes data a snapshot still needs. +- `PersistenceKitReplication` copies an entire estate's rows, audit + events, and blobs from one `Storage` to another. It can run this + copy as a one-shot full snapshot or as an incremental sync driven by + a live change-observation session. + +## How the Pieces Fit + +Figure 1 shows the library's topology. It shows the major parts and +how a write or a read moves between them. + +![Figure 1. Topology of PersistenceKit](topology.svg) + +*Figure 1. Topology of PersistenceKit. A calling kit reaches storage +only through the four core protocols. Decorators wrap a backend's +`RowStore` transparently. Cross-cutting concerns sit beside the core +surface. Examples include encryption, erasure, and snapshots. Each is +invoked by name, not by inheritance. Replication reads one `Storage` and writes +another. Dashed regions mark backend-owned engines and the +cross-process encryption key store.* + +A typical write enters through a kit-held reference to `any RowStore`. +If the estate has caching enabled, that reference is really a +`CachingRowStore` wrapping the real backend's row store. If hash-on-write +is also configured, a `HashingRowStore` may sit in front of that. The +write descends through zero or more decorators until it reaches the +concrete backend: `SQLiteRowStore`, `PostgreSQLRowStore`, or +`InMemoryRowStore`. That backend applies the row-encryption seam, a +no-op unless the estate uses row encryption. It checks the +content-and-keyID invariant. It validates every SQL identifier it is +about to interpolate. It then commits the row. Every backend notifies +its `StorageObserver` afterward. Decorators, cache invalidators, and +replication sessions downstream can then react. + +A typical read follows the same path in reverse, with one addition. +Any content subject to the right-to-be-forgotten contract passes +through `ErasureOverlay.apply(...)` after the backend query returns. An +erased row's content columns then come back `nil`. The row's skeleton +still survives for referential integrity. The skeleton is its ID, +timestamps, and lattice anchors. + +Encryption, erasure, and snapshots are not separate storage engines. +They are pure functions and small actors that any backend calls at the +right moment. `encryptedForWrite` and `decryptedForRead` run around a +write or a read. `ErasureLedgerOps.isErased` runs inside the overlay. +`GCPin.minimumRetainableHlc` runs before a vacuum. This design keeps +the cross-cutting logic in one place, the `PersistenceKit` core, +instead of duplicating it inside every backend. It is also why the +SQLite and PostgreSQL backends produce byte-compatible encrypted +envelopes. Both call the same `RowCrypto` code. + +## What Ships in the Package + +The package ships five Swift targets. `PersistenceKit` is the core. It +holds the `Storage`, `RowStore`, `BlobStore`, `AuditLog`, and +`StorageObserver` protocols. It holds the typed value and predicate +algebra, the schema declarations, and the caching and hashing +decorators. It also holds the encryption, erasure, snapshot, and +telemetry toolkits. `PersistenceKitInMemory` is the test backend. +`PersistenceKitSQLite` is the on-device backend, built against a +vendored `SQLCipher` C target. Whole-database encryption never depends +on the host operating system's own SQLite because of this. +`PersistenceKitPostgreSQL` is the server backend, built on `postgres-nio` +and `NIOSSL`. `PersistenceKitReplication` is the estate-copying +primitive. It depends only on the core protocol surface, so it works +against any pair of conforming backends. + +The package also ships a Rust port in `rust/`. That port mirrors the +same trait surface. It mirrors the same closed predicate algebra. It +mirrors the same three backends. It exists for use outside the Swift +and Apple ecosystem. The two ports share design intent. They are not +conformance-gated against each +other the way LatticeLib's two legs are. PersistenceKit's cross-platform +contract is the shared protocol shape and wire format. It is not a +promise of byte-identical output from a shared corpus. diff --git a/packages/kits/PersistenceKit/docs/topology.svg b/packages/kits/PersistenceKit/docs/topology.svg new file mode 100644 index 0000000..b4c6f53 --- /dev/null +++ b/packages/kits/PersistenceKit/docs/topology.svg @@ -0,0 +1,164 @@ + + + + + + + + + + + + + PersistenceKit: one protocol, three backends, cross-cutting toolkits + + + + Calling kit + any RowStore + + + + HashingRowStore + hash-on-write, dirty chain + + + CachingRowStore + LRU hot tier, sensitivity-gated + + + + Storage protocol + RowStore+BlobStore+AuditLog+Observer + + + + + + + + + SQLiteStorage + one file, SQLCipher + + + PostgreSQLStorage + one schema per estate + + + InMemoryStorage + ephemeral, tests + + + + + + + + Apple-only key custody + + KeychainKeyStore: per-estate 256-bit key + + + + + Cross-cutting toolkits: called by name, not by inheritance + + + RowCrypto + AES-GCM row seam + encrypt/decrypt write+read + + + ErasureLedger + Overlay + right-to-be-forgotten + fail-closed on read + + + SnapshotRegistry + + GCPin + attested snapshots, vacuum floor + + + + + + + + + + PersistenceKitReplication: operates on any two Storage instances + + + Source estate + any Storage + + + StorageReplicator + full snapshot: rows, audit, + blobs, one serializable txn + + + IncrementalReplicationSession + observer-driven dirty set, + watermark + re-scan + + + Destination estate + any Storage + + + + + + + Both paths read the source, then write the + destination inside one serializable transaction: + a crash mid-flush leaves the destination unchanged. + + + + + diff --git a/packages/kits/QueueKit/docs/AGENT_MAP.md b/packages/kits/QueueKit/docs/AGENT_MAP.md new file mode 100644 index 0000000..e0597dd --- /dev/null +++ b/packages/kits/QueueKit/docs/AGENT_MAP.md @@ -0,0 +1,179 @@ +--- +doc: AGENT_MAP +package: QueueKit +repo: moot-system +authored_commit: 909513d0a8ecb1e9e903af9f4d25b3e4f2528242 +authored_date: 2026-07-04 +sources: + - path: Sources/QueueKit/DrainLease.swift + blob: 3f1df6aa1fab44c993bd49d96de179ff03450303 + - path: Sources/QueueKit/FilesystemBackend.swift + blob: c0e5f536e20b1def11875e57b587e8392b6f936d + - path: Sources/QueueKit/Job.swift + blob: 5f39e97a8113e47de2f2a18563446072de8ec6cf + - path: Sources/QueueKit/ObservationStatus.swift + blob: a32b0e8c93311ce498d12ffb4df864a2652c707e + - path: Sources/QueueKit/PersistenceKitBackend.swift + blob: 96175a8b27c8b6e929f417d95b141f727c5179d7 + - path: Sources/QueueKit/QueueBackend.swift + blob: eb67e0b821bc9bfdec8cfde621c5e70d4295d331 + - path: Sources/QueueKit/QueueError.swift + blob: 2697c4b7404b9e04259267cd8f4008030ebb6754 + - path: Sources/QueueKit/QueueKit.swift + blob: 60dfaa1e8f92ec051810b50d8b9cadc47388c02f + - path: Sources/QueueKit/QueueKitTelemetry.swift + blob: 29024f112bf133012283205175aa336b8d80d7c9 + - path: Sources/QueueKit/Watcher.swift + blob: cf2b270b9c60da34f7a25c016f8c18b6ba6149e4 +--- + +# AGENT_MAP: QueueKit + +PURPOSE: general-purpose durable work queue. Producer→`send`/`writeBatch`→backend; consumer→`drain`/`watch`→claims jobs; consumer→`reply`→terminal completion. Two swappable backends (`FilesystemBackend` POSIX maildir, `PersistenceKitBackend` shared SQLite table) behind one `QueueBackend` protocol and one public `QueueKit` facade. Stream-scoped ops (ADR-021 Decision 7 / T1) let multiple consumers share one queue without stealing each other's jobs. + +DEPS: imports SubstrateTypes (HLC/HLCGenerator: hybrid logical clock), PersistenceKit (Storage, rowStore, transaction, observer: PersistenceKitBackend only), IntellectusLib (self-report telemetry, DECISION_LIFT_PACKAGE_SWIFT_RULE_2026-05-28). ConvergenceKit is application-layer composition and deliberately NOT a dependency (spec §11). Imported by: none within this repo checkout; source comments name CorpusKit and GeniusLocusKit as external mount-time consumers (not present in moot-system). Rust port in rust/ (8 files, ~3k lines) mirrors facade + both backends, PersistenceKitBackend behind `persistencekit` cargo feature. Python port in python/queuekit/ mirrors FilesystemBackend only (spec §2: Python has no PersistenceKit backend). All three gated by shared conformance fixtures in Tests/QueueKitTests/Fixtures/*.json. + +ENTRY POINTS (most callers need only these): +- QueueKit.swift:52 `QueueKit.init(root:hlcGenerator:) throws`: mount FilesystemBackend at root, create maildir, clean stale tmp/ +- QueueKit.swift:65 `QueueKit.init(backend:root:)`: mount explicit backend (PersistenceKitBackend, or a test double) +- QueueKit.swift:72 `send(_ job: Job) async throws`: enqueue one job +- QueueKit.swift:85 `drain() async throws -> [(job: Job, sessionID: SessionID)]`: claim all available jobs +- QueueKit.swift:129 `reply(to:status:artifacts:) async throws`: mark one claimed job terminal +- QueueKit.swift:123 `watch(handler:) async throws`: subscribe a per-job handler + +## Symbol Table + +### Facade: QueueKit.swift +- :28 `staleTmpThreshold = 5 * 60`: tmp/ files older than this on init are deleted (crash between write and rename; never visible in new/, safe to remove) +- :30 `final class QueueKit: Sendable` +- :39 `latencyWindow`: nonisolated(unsafe); caller (GLK scheduler actor / serial test) owns exclusivity +- :47 `estateTag: String = "unknown"`: nonisolated(unsafe); set ONCE at mount before any drain(), never again +- :52 `init(root:hlcGenerator:) throws`: see ENTRY POINTS +- :65 `init(backend:root:)`: see ENTRY POINTS +- :72 `send(_:)` / :81 `send(batch:) -> Int`: writeBatch twin; FS backend fsyncs new/ ONCE for the batch +- :85 `drain()` / :108 `drain(stream:)`: claim; wraps backend call with reportQueueStats timing +- :123 `watch(handler:)`: forwards to backend.watch +- :129 `reply(to:status:artifacts:)`: guards status.isTerminal, throws .invalidTerminalStatus otherwise +- :148 `reply(session:status:) -> Int`: fast path ONLY on PersistenceKitBackend (`as?` cast); returns 0 on FilesystemBackend (caller falls back to per-job reply) +- :171 `reply(batch:) -> Int`: routes to backend.completeBatch; FS backend's one-scan/one-fsync path +- :193 `reclaimInFlight(stream:) -> Int`: GATE: call ONLY immediately after DrainLease.tryAcquire SUCCEEDED for stream; fast path ONLY on PersistenceKitBackend, returns 0 otherwise +- :204 `inFlight()` / :214 `pendingCount()` / :224 `pendingCount(stream:)`: read-only depth probes; pendingCount()+inFlight().count = total outstanding work +- :261 `awaitDrain(pollInterval: .milliseconds(20), timeout: .seconds(30)) throws`: polls both frontiers; returns promptly if already empty; throws .drainTimeout, never hangs +- :292 `awaitDrain(stream:pollInterval:timeout:)`: stream-scoped twin; global awaitDrain would block forever on OTHER streams' jobs a scoped drainer never claims +- :314 `completed(streamID:)` +- :322 `maildirSubdirs = ["tmp","new","cur","done"]` +- :324 `ensureMaildir(root:) throws`: creates 4 subdirs if absent +- :341 `cleanStaleTmpFiles(root:) throws`: deletes tmp/ entries older than staleTmpThreshold + +### Backend contract: QueueBackend.swift +- :9 `protocol QueueBackend: Sendable` +- :10 `write(_:)`, :23 `drainAvailable()`, :30 `watch(handler:)`, :34 `complete(_:status:artifacts:)`, :52 `inFlight()`, :54 `completed(streamID:)`: must-implement core +- :21 `writeBatch(_:) -> Int`: default (extension :92-102) loops write; override for shared-cost batching +- :28 `pendingCount()`: depth probe used by telemetry +- :48 `completeBatch(_:) -> Int`: default (extension :104-116) loops complete; override for shared-cost batching +- :66 `drainAvailable(stream:)`: default (extension :76-84) claims ALL streams then filters: steals other streams' jobs, MUST override for real stream isolation +- :73 `pendingCount(stream:)`: default (extension :86-89) delegates to all-streams pendingCount() (wrong count unless overridden) + +### Wire types: Job.swift +- :22 `struct JobID: RawRepresentable`: :31 `generate()` = UUID → 32 lowercase hex, no hyphens (avoids colliding with maildir filename hyphen separator) +- :51 `struct StreamID: RawRepresentable`: workload/lane name +- :68 `struct SessionID: RawRepresentable`: :75 `mint()` = one per drain() call; groups a claimed batch +- :80 `struct ToolName: RawRepresentable` +- :85 `enum ArtifactRef`: file_path|commit_hash|signal_file|trajectory_step_id; explicit `type` field on wire (:91 CodingKeys) +- :131 `indirect enum CodableValue`: open caller extensions blob: null/bool/int/double/string/array/object; round-trips verbatim +- :167 `struct Job: Identifiable, Hashable`: id/streamID/submittedAt(HLC)/priority/payload/extensions +- :193 `Job` CodingKeys: PINNED snake_case wire keys: id, stream_id, submitted_at{physical_time,logical_count,node_id}, priority, payload(base64url), extensions +- :254 `struct MissionContext`: ONLY type using default camelCase Codable synthesis (not part of pinned job envelope) +- :236 `WireFormat.base64urlEncode(_:)` / :245 `base64urlDecode(_:)`: RFC 4648 §5, no padding +- :289 `WireFormat.filename(for:) -> String`: `{sortableHLC}-{streamID}-{jobID}` +- :296 `WireFormat.sortableHLC(_:)`: `{phys:016d}-{logical:08d}-{node_unsigned:010d}`; zero-padding makes lexicographic filename sort == HLC sort +- :307 `WireFormat.encoder`: sortedKeys + withoutEscapingSlashes: byte-stable across Swift/Rust/Python (conformance requirement) +- :313 `WireFormat.decoder` +- :321 `struct SignalFile`: job_id/status/artifacts/completed_at; same pinned-key discipline as Job + +### Status/errors +- ObservationStatus.swift:9 `enum ObservationStatus: String`: running|done|done_with_concerns|needs_context|blocked +- ObservationStatus.swift:16 `isTerminal`: only .running is false; gates every reply/complete call +- QueueError.swift:7 `enum QueueError: Error`: directoryCreationFailed, writeFailed, renameFailed, decodingFailed, unknownTool, jobNotFound, watcherFailed, staleTmpFile, backendUnavailable, invalidTerminalStatus, drainTimeout(pending:inFlight:), invalidIdentifier(id:reason:) + +### FilesystemBackend.swift (POSIX maildir; semantics derived from Postfix deliver_maildir()) +- :43 `final class FilesystemBackend: QueueBackend, @unchecked Sendable` +- :32 `private final class HLCBox`: NSLock-guarded HLCGenerator (value type; concurrent send() must serialize) +- :74 `validateIdentifier(_:) throws` (private static): rejects empty/`.`/`..`/path-separator/control-char; MUST run before ANY path is built from a caller identifier (path-traversal guard, CAND-023, mirrored in Rust+Python) +- :96 `write(_:)`: validate → O_CREAT|O_EXCL write to tmp/, mode 0600, fsync, rename to new/, fsync new/ dir +- :129 `writeBatch(_:) -> Int`: per-job write+rename WITHOUT fsync, ONE fsync of new/ dir at end; AT-LEAST-ONCE (reindex re-derives from estate on crash) +- :160 `atomicWriteAndRename(data:tmpPath:newPath:newDir:) throws` (static): the full durable write; ENOENT rename retried once, EXDEV (tmp/new on different filesystems) fails hard +- :273 `writeAndRenameNoFsync(data:tmpPath:newPath:) throws` (static): batch primitive, no per-file/dir fsync +- :340 `pendingCount()`: file count in new/ +- :363 `drainAvailable(stream:)`: reads+decodes each new/ file BEFORE claiming (read ≠ claim); only rename-claims if job.streamID == stream; non-matching files left untouched in new/ (no steal-then-unclaim) +- :428 `pendingCount(stream:)`: decodes each new/ file, non-claiming +- :445 `drainAvailable()`: rename ALL new/→cur/ first, then decode; undecodable file → moved to done/ (poison, never retried) +- :516 `reclaimInFlight() -> Int`: GATE: call ONLY at mount before any drain session is live (fresh process owns no in-flight work by construction); unscoped (cur/ has no per-stream separation) +- :549 `watch(handler:)`: Watcher.watchNewDirectory → drainAvailable() → handler per job +- :567 `complete(_:status:artifacts:)`: find file in cur/ by `-{jobID}` suffix; writes SignalFile to done/ BEFORE renaming job file (crash between = job still reclaimable, never orphaned-signal-no-job) +- :631 `completeBatch(_:) -> Int`: ONE cur/ scan → jobID→filename index (keys on suffix after LAST `-`; JobIDs are 32 dashless hex so this is unambiguous); per-completion write+rename without fsync; ONE done/ dir fsync at end +- :701 `listJobs(in:filter:)` (private): skips `.signal` sidecars, decodes, optional stream filter + +### PersistenceKitBackend.swift (spec §10 v1.1: 5 invariants enforced) +- :34 `queueKitTableName = "queuekit_jobs"` +- :36 `enum QueueKitSchema`: :40 `declaration()`: columns incl. status/session_id/signal_status/artifacts; :62 `appendOnly: false` MUST stay false (invariant 5); indices: status, (status,phys,logical,node) claim-order, (stream_id,status) +- :84 `final class PersistenceKitBackend: QueueBackend, @unchecked Sendable` +- :91 `openSchema(on:) throws` (static): opens schema on a Storage +- :97 `write(_:)`: BARE rowStore.insert, status="new", NO transaction (invariant 1) +- :143 `writeBatch(_:) -> Int`: ONE `.readCommitted` transaction wrapping N inserts; readCommitted safe because new-row inserts cannot conflict with the claim's serializable new→cur UPDATE (claim sees pre-batch snapshot or post-batch rows, never a torn view) +- :172 `pendingCount()`: COUNT WHERE status='new' +- :189 `drainAvailable(stream:)` / :248 `drainAvailable()`: SINGLE-PASS CLAIM (invariant 3): one `.serializable` txn = guarded bulk UPDATE new→cur under fresh session_id, THEN SELECT rows WHERE session_id=session (same txn, sees own writes); O(N) not O(N²); session-keyed readback prevents cross-drainer double-count +- :237 `pendingCount(stream:)`: COUNT WHERE status='new' AND stream_id=? +- :324 `completeSession(_:status:) -> Int`: NOT on QueueBackend protocol (concrete-only); ONE guarded UPDATE WHERE session_id=? AND status='cur'; backs QueueKit.reply(session:status:) +- :350 `watch(handler:)`: storage.observer.observe(insert) per invariant 2 (event = wake ONLY, never read job fields from event); drains once BEFORE first await (catches pre-subscription inserts), then per wake +- :378 `drainUntilEmpty` (private static): loops drainAvailable() until a pass claims 0; absorbs coalesced/dropped observer wakes; a claim ERROR must propagate (not `try? ?? []`) or a fault masquerades as "queue empty" +- :400 `complete(_:status:artifacts:)`: guarded UPDATE WHERE id=? AND status='cur'; affected==0 → throws .jobNotFound +- :451 `reclaimInFlight(stream:) -> Int`: GATE identical to FilesystemBackend.reclaimInFlight: ONLY after DrainLease.tryAcquire success for stream; resets cur→new, clears session_id, stream-scoped (shared table holds multiple streams) +- :468 `inFlight()` / :472 `completed(streamID:)` → :480 `listJobs(status:streamID:)` (private) +- :500 `decodeRow(_:)` (private static): malformed extensions JSON → falls back to empty dict, not a decode failure + +### DrainLease.swift: stream-keyed heartbeat-TTL lock (NOT PID-liveness; portable, no FFI) +- :49 `struct DrainLease: Sendable`: pure value type; lets Corpus's non-isolated deinit call release() directly +- :55 `owner`: `"pid-{pid}-{instanceToken}"`; nonce defeats PID-reuse impersonation after crash +- :59 `ttl`: default 15s = Rust DRAIN_LEASE_TTL_SECS +- :64 `heartbeatInterval = 5` (static): heartbeat cadence, well inside ttl +- :73 `init(directory:stream:instanceToken:ttl:)`: lease file = `/.drain.lease`; stream name sanitized to alnum/_/- (else `_`) +- :93 `tryAcquire(now:) -> Bool`: false only if fresh lease held by ANOTHER owner; else (re)claims + re-reads to resolve write races (atomic replace = last-writer-wins) +- :110 `heartbeat(now:)`: refresh; does NOT check ownership, caller's responsibility +- :119 `isHeldByOther(now:) -> Bool`: freshness check without attempting acquire +- :127 `release()`: deletes file ONLY if still owned by self (never undoes a takeover) +- File format: `\n`, atomic replace write + +### Watcher.swift: directory wake source (spurious wakes OK; drainAvailable is authority) +- :27 `enum Watcher` +- :37 `watchNewDirectory(at:onChange:) throws`: calls onChange() once immediately (drain pre-attach work) before platform loop +- :63 `watchKQueue` (Darwin only): DispatchSource kqueue VNODE (write/extend/delete/rename), O_EVTONLY fd, µs latency +- :134 `openInotify` (Linux, private): IN_CREATE|IN_MOVED_TO; returns -1 on watch-limit/unsupported-fs (caller falls back) +- :162 `runInotifyLoop` (Linux, private): poll() 500ms timeout for responsive cancellation; drains fd, doesn't parse events +- :216 `watchPoll`: 200ms snapshot-diff poll; Linux fallback AND sole watcher on other platforms; ~100ms average latency +- :240 `final class ContinuationBox`: bridges DispatchSource cancel handler (DispatchQueue callback) to async wait() + +### Telemetry: QueueKitTelemetry.swift (off-path cost ~1ns; metric namespace `queue.*`) +- :27 `struct QueueLatencyWindow`: capacity-100 rolling sample list +- :44 `percentile(_:) -> Double`: P7-secfix: guards non-finite/out-of-range p BEFORE index computation (NaN/inf could crash) +- :68 `reportQueueStats(backend:drained:drainStart:now:estateTag:window:) async`: gate: `Intellectus.isEnabled`; emits queue.depth (or queue.depth_unavailable on read failure: NEVER fabricates 0), queue.drain_count, queue.idle_nonempty (skipped if depth unknown), queue.latency_p50_ms, queue.latency_p95_ms, queue.head_of_line_age_s + +## INVARIANTS / GOTCHAS + +- WIRE FORMAT IS THE CONTRACT. Job/SignalFile Codable keys (snake_case, nested HLC, base64url payload) are pinned across Swift/Rust/Python. Any change must be mirrored in rust/src/job.rs and python/queuekit/job.py and pass the shared fixtures in Tests/QueueKitTests/Fixtures/*.json (rust/tests/conformance.rs reads the SAME files). +- JobID.generate() deliberately omits hyphens (32 dashless hex): required so WireFormat.filename's own hyphen separators are unambiguous. Do not "prettify" back to UUID-with-hyphens form. +- validateIdentifier (FilesystemBackend) MUST run before any path is constructed from a caller-supplied stream_id or job id, at EVERY write/complete/completeBatch entry point. Mirrored in Rust and Python: edit all three or none. +- reclaimInFlight (both backends) is safe ONLY immediately after a successful DrainLease.tryAcquire for that stream. Calling it without a fresh lease can yank a job out from under a live drainer. FilesystemBackend's unscoped reclaimInFlight() is mount-time-only, by construction (fresh process = no in-flight work of its own). +- reply(session:status:) and reclaimInFlight(stream:) have real fast paths ONLY on PersistenceKitBackend (checked via `as?`); every other backend returns 0, and callers MUST have a per-job fallback. +- QueueBackend's default drainAvailable(stream:) and pendingCount(stream:) extensions are CORRECTNESS TRAPS if relied upon: the default drain claims and then re-releases other streams' jobs (transient full-queue claim), and the default pendingCount(stream:) reports the ALL-STREAMS count. Both concrete backends override correctly; a THIRD backend implementation MUST also override both. +- PersistenceKitBackend.write() is a bare insert, NEVER wrap it in a transaction (invariant 1). writeBatch's .readCommitted (not .serializable) is deliberate: see drainAvailable's serializable claim for why this is still race-free. +- PersistenceKitBackend.watch()'s observer event is a WAKE SIGNAL ONLY (invariant 2): never read job data from the TableChange event; always re-enter through drainAvailable(). drainUntilEmpty must propagate claim errors, never `try? ?? []`, or a live fault silently reads as "queue empty." +- appendOnly MUST stay false on the queuekit_jobs table declaration (invariant 5): rows are mutated in place through the new→cur→done lifecycle. +- FilesystemBackend.completeBatch's jobID lookup keys on the filename suffix after the LAST `-`. This is only unambiguous because JobIDs are 32 dashless hex characters (JobID.generate()); a job id containing a `-` would break this index. +- estateTag on QueueKit is `nonisolated(unsafe)`: set it ONCE at mount, before any drain() call, never again during concurrent use. Same discipline for the internal latencyWindow. +- awaitDrain / awaitDrain(stream:) are polling latches (20ms default interval, 30s default timeout), not push notifications: there is no native "queue just emptied" event on either backend. Always returns promptly on an already-empty queue; always throws QueueError.drainTimeout rather than hanging past the deadline. +- Telemetry never fabricates queue.depth=0 on a read failure: it emits queue.depth_unavailable instead, and skips every metric that depends on a known depth. Do not "simplify" this to `(try? pendingCount()) ?? 0`. +- Watcher wakes are ALWAYS advisory/spurious-tolerant. drainAvailable() (via atomic rename / serializable UPDATE) is the sole claim authority on every platform. +- DrainLease is heartbeat-TTL, not PID-liveness: worst-case takeover latency is one full TTL (15s). Do not add OS-specific process-liveness checks; portability across macOS/Linux/Windows is a design constraint, not an oversight. +- Pinned constants: changing requires a conformance regen across all 3 ports: staleTmpThreshold 300s, DrainLease ttl 15s / heartbeatInterval 5s, Watcher poll interval 200ms / Linux poll() timeout 500ms, QueueLatencyWindow capacity 100, awaitDrain pollInterval 20ms / timeout 30s, NovelToken-style file mode 0600 on queue files. +- ConvergenceKit is intentionally NOT a QueueKit dependency (spec §11): it is application-layer composition sitting above this kit. Do not add it as a Package.swift dependency. diff --git a/packages/kits/QueueKit/docs/DETAILS.md b/packages/kits/QueueKit/docs/DETAILS.md new file mode 100644 index 0000000..4cc394b --- /dev/null +++ b/packages/kits/QueueKit/docs/DETAILS.md @@ -0,0 +1,643 @@ +--- +doc: DETAILS +package: QueueKit +repo: moot-system +authored_commit: 909513d0a8ecb1e9e903af9f4d25b3e4f2528242 +authored_date: 2026-07-04 +sources: + - path: Sources/QueueKit/DrainLease.swift + blob: 3f1df6aa1fab44c993bd49d96de179ff03450303 + - path: Sources/QueueKit/FilesystemBackend.swift + blob: c0e5f536e20b1def11875e57b587e8392b6f936d + - path: Sources/QueueKit/Job.swift + blob: 5f39e97a8113e47de2f2a18563446072de8ec6cf + - path: Sources/QueueKit/ObservationStatus.swift + blob: a32b0e8c93311ce498d12ffb4df864a2652c707e + - path: Sources/QueueKit/PersistenceKitBackend.swift + blob: 96175a8b27c8b6e929f417d95b141f727c5179d7 + - path: Sources/QueueKit/QueueBackend.swift + blob: eb67e0b821bc9bfdec8cfde621c5e70d4295d331 + - path: Sources/QueueKit/QueueError.swift + blob: 2697c4b7404b9e04259267cd8f4008030ebb6754 + - path: Sources/QueueKit/QueueKit.swift + blob: 60dfaa1e8f92ec051810b50d8b9cadc47388c02f + - path: Sources/QueueKit/QueueKitTelemetry.swift + blob: 29024f112bf133012283205175aa336b8d80d7c9 + - path: Sources/QueueKit/Watcher.swift + blob: cf2b270b9c60da34f7a25c016f8c18b6ba6149e4 +--- + +# QueueKit Details + +This document walks through every source file in the package. Read +`OVERVIEW.md` first for the big picture. Files appear here in pipeline +order. The wire-format types come first. The error vocabulary comes next, +followed by the backend contract and the public facade. Then come the two +concrete backends. Last comes the supporting infrastructure: the drain +lease, the directory watcher, and telemetry. + +## Job.swift + +This file provides the wire-format types every job and completion signal +is built from: `JobID`, `StreamID`, `SessionID`, `ToolName`, `ArtifactRef`, +`CodableValue`, `Job` itself, `MissionContext`, `WireFormat`, and +`SignalFile`. + +A job needs an identifier that is both unique and safe to use as a +filename. The filesystem backend uses this identifier directly in a path. +`JobID.generate()` builds one from a UUID, a 128-bit random identifier. It +renders the UUID as thirty-two lowercase hexadecimal characters, with the +hyphens removed. This removal matters because QueueKit already uses a +hyphen as a separator inside maildir filenames, and the UUID's own hyphens +would otherwise collide with it. + +`StreamID` names the workload a job belongs to. Examples are "encode" or +"dreaming." Jobs on different streams can share one queue. A consumer of +one stream never sees another stream's jobs. + +`SessionID.mint()` stamps one drain call. Every job a single `drain()` +claims shares one session identifier. This is what lets a caller retire a +whole claimed batch in one step later. + +`ArtifactRef` names something a completed job produced. It can name a +file, a commit, or a signal file. It can also name a step in a recorded +trajectory. An explicit `type` field on the wire tags which of the four +cases applies, so a decoder can tell them apart. + +`CodableValue` is a caller-defined extensions blob. It is an open-ended +value. It can be a string, a number, a boolean, null, an array, or a +nested object. It exists so a job's `extensions` dictionary can carry +arbitrary structured data supplied by whatever produced the job. Every +field of that data survives a `send()`/`drain()` round trip unchanged. +QueueKit does not need to know the shape of that data in advance. + +`Job` is the unit of work. It carries an identifier, a stream, an HLC +submission timestamp, and a priority. It also carries a payload of raw +bytes and the extensions dictionary. Its custom `Codable` conformance is +not decorative. It implements the exact wire format the package +specification fixes. Keys use `snake_case`, so the wire uses `stream_id` +rather than `streamID`. The HLC nests as `physical_time`, `logical_count`, +and `node_id`. The payload is base64url text, rather than raw JSON bytes. + +This wire format is a cross-language contract. The Rust and Python ports +must produce and consume the identical bytes. Every field name, nesting +shape, and encoding choice here is pinned. None of it is a matter of Swift +convention. + +`MissionContext` is the one type in this file that uses ordinary Swift +`Codable` key synthesis. It uses `camelCase` property names, and it needs +no custom coding keys. This type does not use the pinned wire format, +because it is not part of the job envelope. It is instead a +caller-supplied payload shape, with no cross-language byte-identity +requirement. + +`WireFormat.filename(for:)` builds the canonical maildir filename for a +job. The filename joins three parts by hyphens: a sortable HLC string, the +stream identifier, and the job identifier. + +`WireFormat.sortableHLC(_:)` is why the filename sorts correctly by claim +order. It zero-pads each HLC component to a fixed width: sixteen digits +for physical time, eight for the logical count, and ten for the node +identifier. This padding makes ordinary lexicographic string sorting match +HLC's own ordering. Lexicographic sorting is the only sort a plain +directory listing gives a caller. + +`WireFormat.encoder` fixes sorted keys and no escaped slashes. The exact +same job therefore encodes to the exact same bytes, on every run and every +platform. This is what makes the shared conformance fixtures meaningful. + +`SignalFile` is the durable record a completed job leaves behind. It +records four things. It records which job finished. It records what +terminal status resulted. It records what artifacts the job produced. It +records when the job finished. Its `Codable` conformance mirrors `Job`'s. +It uses the same pinned `snake_case` keys and the same nested HLC. This +consistency matters because whichever consumer or tool eventually inspects +the `done/` directory reads a signal file, regardless of which language +that consumer is written in. + +## ObservationStatus.swift + +This file provides `ObservationStatus`, the small enumeration a job's +outcome is reported in. Its cases are `.running`, `.done`, +`.doneWithConcerns`, `.needsContext`, and `.blocked`. + +The type is owned here rather than duplicated. More than one package needs +to agree on what a finished job's outcome can be. The file's header +comment names AgentHarness as one such package. AgentHarness imports the +type directly from QueueKit, rather than declaring its own copy. + +`isTerminal` is the one piece of logic on the type. Only `.running` is +non-terminal. `reply(to:status:)` on the `QueueKit` facade checks +`isTerminal` before accepting a completion. This check stops a caller from +accidentally marking a job finished with the one status that means it has +not finished. + +## QueueError.swift + +This file provides `QueueError`, the complete error vocabulary a caller of +QueueKit can receive. + +Each case names one specific failure. A directory could not be created. A +write failed. A rename failed. A job's stored bytes could not be decoded. +A job identifier could not be found. The watcher failed. A stale +temporary file was found. The backend was unavailable. A caller tried to +complete a job with a non-terminal status. `awaitDrain` timed out. A +caller-supplied identifier was unsafe to use in a path. + +Every distinct failure gets its own case, rather than a single generic +wrapped error. This design lets a caller handle a `jobNotFound` +differently from a `renameFailed`, without inspecting error text. + +`drainTimeout(pending:inFlight:)` and `invalidIdentifier(id:reason:)` each +carry extra diagnostic detail. `drainTimeout` explains how far the queue +was from empty when `awaitDrain` gave up. `invalidIdentifier` explains +exactly why a submitted identifier was rejected. + +## QueueBackend.swift + +This file provides `QueueBackend`, the protocol every concrete backend +conforms to. It also provides two protocol extensions. These extensions +supply default implementations for the operations a backend does not have +to implement specially. + +The protocol is the seam that lets the public `QueueKit` class stay +ignorant of which storage mechanism sits underneath it. Every method the +facade exposes has a matching method here. `QueueKit` does nothing but +forward to whichever `QueueBackend` it was mounted with. Six core +operations must be implemented from scratch by every backend: `write(_:)`, +`drainAvailable()`, `watch(handler:)`, `complete(_:status:artifacts:)`, +`inFlight()`, and `completed(streamID:)`. + +`writeBatch(_:)` and `completeBatch(_:)` are bulk twins of `write` and +`complete`. The first default extension implements them by simply looping +the single-item version. This loop is correct for any backend, but it is +not necessarily fast. A backend earns the right to skip the loop by +overriding these two methods. The override must use a mechanism that pays +a shared cost once for the whole batch, instead of once per job. Both +concrete backends do exactly that, for reasons explained under their own +headings below. + +`drainAvailable(stream:)` and `pendingCount(stream:)` are the +stream-scoped forms of `drainAvailable()` and `pendingCount()`. The second +protocol extension gives `drainAvailable(stream:)` a default +implementation. This default claims every stream's jobs through the +all-streams `drainAvailable()`, then filters the result down to the +requested stream. The result is correct, but it claims jobs belonging to +other streams, and those jobs then have to be re-enqueued. The file's own +comment warns that a concrete backend should override this default, +rather than rely on it. Both `FilesystemBackend` and `PersistenceKitBackend` +override it, each in the way suited to its own storage. + +## QueueKit.swift + +This file provides the public facade. The facade is the `QueueKit` class +itself, its maildir directory-management helpers, and the four permanent +methods every caller uses. Those four methods are `send`, `drain`, +`watch`, and `reply`. Every caller uses them the same way, regardless of +which backend is mounted underneath. + +`QueueKit` holds four properties. It holds a `backend`, anything +conforming to `QueueBackend`. It holds an optional `root` URL, present +only when the mounted backend is a filesystem backend. It holds a rolling +latency window for telemetry. It holds an `estateTag` string, used to tag +telemetry metrics by estate. + +Two initializers exist. `init(root:hlcGenerator:)` is the common path. It +builds a `FilesystemBackend` at the given root, and it cleans any stale +temporary files left behind by a prior crash. `init(backend:root:)` mounts +an already-constructed backend directly. This is how `PersistenceKitBackend` +gets attached. It is also how tests substitute an in-memory backend. + +`send(_:)` and its batch twin `send(batch:)` hand jobs to the backend. The +filesystem backend's per-job `write` pays a filesystem synchronization +cost on every single job. This cost is a call that forces the operating +system to guarantee the write has actually reached durable storage. The +batch form exists to avoid paying that cost per job. A bulk producer, +enqueuing tens of thousands of jobs at once during a full reindex, can +instead route through `writeBatch`. The filesystem backend implements +`writeBatch` as one durability barrier for the whole batch, rather than +one barrier per job. + +`drain()` and the stream-scoped `drain(stream:)` claim available jobs. +Around the backend call, each method measures how long the claim took, +and reports that duration through `reportQueueStats`. Both return the +same shape: a list of pairs. Each pair holds a claimed `Job` and the +`SessionID` that drain call minted. + +`watch(handler:)` subscribes a handler. The handler is invoked for every +job as it becomes available. The facade does nothing but forward to the +backend's own `watch`. The two backends drive this very differently. The +filesystem backend uses a directory watcher. The PersistenceKit backend +uses a database change observer. + +`reply(to:status:artifacts:)` is the single-job completion path. It +checks `status.isTerminal` before forwarding to the backend. This check +stops a caller from marking a job complete with the one status, +`.running`, that means the job has not finished. + +`reply(session:status:)` completes every job claimed under one drain +call's session, in a single pass. It has a real fast path only on +`PersistenceKitBackend`, checked with an `as?` cast. Only that backend's +shared table can look up an entire session's claimed rows in one guarded +update. On any other backend, `reply(session:status:)` returns `0`. That +return value signals the caller to fall back to per-job `reply`. + +`reply(batch:)` completes an explicit list of job and status pairs, in one +pass. It routes to the backend's `completeBatch`. This is the path a +corpus-drain caller uses on the filesystem backend, which has no session +fast path but does have a batched completion. + +`reclaimInFlight(stream:)` resets every job stuck in-flight for one stream +back to pending. The next `drain(stream:)` call then re-claims and +re-drives it. The doc comment states a strict precondition. A caller must +call this method only immediately after successfully acquiring that +stream's `DrainLease`. A freshly acquired lease guarantees the job's prior +claimant is actually gone, rather than merely slow. Without that +guarantee, this method could yank a job out from under a drain worker +still processing it. + +Like `reply(session:status:)`, `reclaimInFlight(stream:)` has a real fast +path only on `PersistenceKitBackend`. `FilesystemBackend` instead exposes +an unscoped `reclaimInFlight()` that runs once at mount. Its `cur/` +directory has no stream separation in the first place, so no scoping is +possible there. + +`inFlight()`, `pendingCount()`, and `pendingCount(stream:)` are read-only +depth probes. `inFlight()` reports how many jobs are claimed but not yet +replied to. `pendingCount()` reports how many jobs are waiting to be +claimed. Adding `pendingCount()` to `inFlight().count` gives the total +outstanding work a drain still has left to do. A status reader can +compute this total without claiming or draining anything. + +`awaitDrain(pollInterval:timeout:)` and its stream-scoped twin +`awaitDrain(stream:pollInterval:timeout:)` block until a queue, or one +stream of it, is empty on both frontiers. Nothing may be pending, and +nothing may be in-flight. Both methods are polling loops, rather than +push-based latches. The maildir backend has no native event for "the +queue just emptied," so there is nothing to wait on directly. Each +iteration re-reads the live counts. Progress made by a concurrently +running drain worker between polls is therefore observed on the very next +tick. + +Both methods return promptly when the queue is already empty. The first +check can succeed without the method ever sleeping. Both methods throw +`QueueError.drainTimeout` rather than blocking forever, if the deadline +passes with work still outstanding. A stuck or crashed drain worker +therefore surfaces as an error, instead of as a hang. + +`ensureMaildir(root:)` and `cleanStaleTmpFiles(root:)` are the maildir +directory-management helpers that `FilesystemBackend`'s initializer calls. +`ensureMaildir(root:)` creates four subdirectories if they are absent: +`tmp`, `new`, `cur`, and `done`. `cleanStaleTmpFiles(root:)` deletes +any file in `tmp/` older than `staleTmpThreshold`, five minutes. + +A file stranded in `tmp/` means a process crashed between writing it and +renaming it into `new/`. Because the file was never visible in `new/`, no +consumer could have claimed it. Removing it on the next mount is +therefore safe. + +## FilesystemBackend.swift + +This file provides `FilesystemBackend`, the POSIX maildir-style queue +implementation. It uses no database, just four directories and atomic +filesystem operations. Its semantics are deliberately derived from +`deliver_maildir()`, the mail-delivery routine at the heart of the Postfix +mail server. This routine is a decades-proven pattern. It hands off many +small files between uncoordinated processes, without a lock file. + +`validateIdentifier(_:)` runs before any filesystem path is built from a +caller-supplied identifier, whether a stream identifier or a job +identifier. It rejects an empty string. It rejects a `.` or `..` path +component. It rejects a `/` or `\` path separator. It rejects any ASCII +control character. Without this check, a maliciously or accidentally +crafted stream identifier containing a path separator could make a job's +path resolve outside the queue's root directory. That outcome is a +path-traversal risk. The Rust and Python ports enforce the identical rule. + +`write(_:)` implements the maildir handoff in three classic steps. First, +it creates the job's encoded bytes in `tmp/`, with a mode that only the +owning process can read. This restricted mode is defense in depth, since +a queued payload may carry sensitive estate content. Second, it forces +those bytes to durable storage. Third, it atomically renames the file +into `new/`. + +`atomicWriteAndRename(...)` is where those steps actually happen. It opens +the file with `O_CREAT | O_EXCL`, an exclusive create only one caller can +win. It then runs a raw `write` and `fsync` sequence, followed by +`rename`. It finishes with a final `fsync` of the destination directory +itself. This last step matters because a directory's own metadata is +separate durable state from the file contents, on POSIX filesystems. The +metadata records which files the directory currently contains. Both the +metadata and the file contents must be forced to disk, for the write to +survive a crash. A `rename` failure caused by the destination temporarily +vanishing, an `ENOENT` error, is retried once. Another process cleaning +`new/` mid-flight is a recoverable race, rather than a fatal one. + +`writeBatch(_:)` is the bulk twin. It writes and renames every job in the +batch using `writeAndRenameNoFsync`, which skips the per-file durability +step. It forces only `new/`'s directory metadata to disk once, at the +end. + +A crash before that single barrier can lose some just-written jobs. The +comment records why this is an acceptable trade for a bulk producer +specifically. The reindex that calls `writeBatch` derives its jobs from +the estate itself, so a lost job is simply re-enqueued on the next resume. +This is an at-least-once guarantee. It is not a promise that every job +survives every possible crash instant. Ordinary streaming `write` keeps +its per-job durability unchanged. Only a caller that opts into the batch +API accepts this weaker bound. + +`pendingCount()` counts the files in `new/`. One file means one pending +job, so no decoding is required. This is why the count stays cheap even +as a queue grows. + +`drainAvailable(stream:)` is the stream-scoped claim. The maildir filename +already encodes the stream, in the form `{hlc}-{stream}-{jobid}`. A naive +implementation could filter by filename alone. The code instead reads and +decodes each file's actual `streamID` field before claiming it. Files +whose stream does not match are left entirely untouched in `new/`. A +concurrent drainer for a different stream is never affected by this one's +scan. `pendingCount(stream:)` performs the equivalent non-claiming count. + +`drainAvailable()`, the all-streams claim, renames every file in `new/` +into `cur/` first. Only then does it read and decode each one. A file +that fails to decode is moved straight to `done/`, rather than left to +jam every future drain pass. A job whose stored bytes are unparseable can +never become processable by simply trying again. + +`reclaimInFlight()` moves everything left in `cur/` back to `new/`. Its +doc comment states its precondition plainly. A caller should call it only +at mount, before any drain session is live. A freshly started process +cannot have claimed anything itself. Anything already sitting in `cur/` +must therefore be a crash orphan from whichever process ran before it. + +`watch(handler:)` wires `Watcher.watchNewDirectory` to `drainAvailable()`. +Every time the watcher signals that `new/` may have changed, the backend +drains whatever is currently claimable, and hands each job to the +caller's handler. + +`complete(_:status:artifacts:)` finds the job's file in `cur/` by +matching the filename suffix. Every maildir filename ends in `-{jobID}`. +The method writes the `SignalFile` recording the outcome, and only then +moves the job's own file into `done/`. Writing the signal first matters. +If the process crashes between the two steps, the job is still visible +in `cur/` on the next mount, ready to be reclaimed. Without this order, +the job could vanish silently, leaving an orphaned signal and no job +record. + +`completeBatch(_:)` is the batched form. One scan of `cur/` builds a +job-identifier-to-filename index. Every completion in the batch is +written without a per-file durability step. One call forces `done/`'s +directory metadata to disk at the very end. The safety argument mirrors +`writeBatch`'s argument. A crash before that final barrier leaves the +not-yet-moved jobs in `cur/`, where they are reclaimed and reprocessed. +This is safe because ingest is idempotent, and processing the same job +twice causes no harm. + +`inFlight()` and `completed(streamID:)` both delegate to the private +`listJobs(in:filter:)`. This method lists a directory and skips +`.signal` sidecar files. It decodes each remaining file as a `Job`. It +can optionally filter by stream. + +## PersistenceKitBackend.swift + +This file provides `PersistenceKitBackend`, the second concrete backend. +Jobs live as rows in a table inside PersistenceKit, MOOTx01's storage +kit, rather than as files on disk. The file's own header comment states +five invariants the implementation is built to preserve. The sections +below explain each one, as it is enforced in code. + +`QueueKitSchema.declaration()` defines the table's shape. There is one +row per job. Its columns hold the job's own fields, plus queue +bookkeeping. The bookkeeping columns are `status`, `signal_status`, +`artifacts`, and `session_id`. `status` is `"new"`, `"cur"`, or `"done"`. +`signal_status` and `artifacts` are filled in only once a job completes. +`session_id` names the drain call, if any, currently holding the row. + +Three indices back the operations that matter. One index backs a plain +status count. A second index, on `(status, physical_time, logical_count, +node_id)`, backs claiming jobs in HLC order. A third index, on +`(stream_id, status)`, backs the stream-scoped operations. The table's +`appendOnly` flag is explicitly `false`. This is invariant 5. A queue row +must be mutable, since the same row moves from `new` to `cur` to `done` +in place. + +`write(_:)` is a bare insert into the table, with `status: "new"`. It has +no enclosing transaction, per invariant 1. A transaction would offer no +benefit here, since there is nothing else to make atomic with a single +insert. A transaction would only add overhead to the hottest, most +frequent operation in the backend. + +`writeBatch(_:)` wraps every insert in the batch in one `.readCommitted` +transaction. This transaction is the isolation equivalent of the +filesystem backend's single durability barrier. The comment explains why +the weaker `.readCommitted` isolation is still correct here, rather than +the `.serializable` isolation the claim uses. These are all inserts of +brand-new rows. Brand-new rows cannot conflict with the claim's job of +flipping existing `new` rows to `cur`. A claim that begins before the +batch commits simply will not see the batch's rows yet. It will see them +on its next pass instead. + +`pendingCount()` and `pendingCount(stream:)` are single-row-count reads. +Both use a `status = 'new'` predicate. The stream form adds a +`stream_id` predicate. Neither read claims anything or locks beyond an +ordinary read. + +`drainAvailable(stream:)` and the all-streams `drainAvailable()` both +implement the claim as one `.serializable` transaction. That transaction +has exactly two steps. First, an `UPDATE` flips every matching `new` row +to `cur`, under one freshly minted session identifier. Second, a `SELECT` +reads back precisely the rows carrying that session identifier, ordered +by HLC. This single-pass shape is invariant 3, the status-guarded atomic +transition. The file's comment calls it out as a deliberate improvement +over an earlier design, one that issued one guarded update per row. +Claiming a batch of jobs that older way cost one predicate scan per job, +so a bulk claim over a large queue became quadratic in the number of +jobs. Reading back by this call's own unique session, rather than by +`status = 'cur'` generally, keeps two concurrent drainers from ever +double-counting the same claimed row. Each call's session tags only the +rows it itself just claimed. + +`completeSession(_:status:)` completes every row claimed under one +session, in a single guarded update. It is the batch twin of the per-job +`complete`. It is the reason `QueueKit.reply(session:status:)` has a fast +path only on this backend. The method is declared directly on the class, +rather than as part of the `QueueBackend` protocol. Only the concrete +`PersistenceKitBackend` handle has anything meaningful to complete a +whole session against. The abstract protocol does not. + +`watch(handler:)` subscribes to `storage.observer`, per invariant 2. It +treats every event strictly as a wake signal. It never reads a job's +fields out of the event payload itself, only out of a subsequent +`drainAvailable()` call. `drainUntilEmpty` is the loop this drives. +Rather than draining once per event, it drains repeatedly until a pass +claims nothing. A burst of inserts can coalesce into fewer observer +events than there are actual rows, so draining only once per event would +strand whichever rows' wake got folded away. The initial drain, run +before the subscription's first `await`, matters for the same reason in +reverse. A job inserted between mount and subscription would otherwise +be invisible to any wake, since its insert event predates the +subscription. + +`complete(_:status:artifacts:)` is a single guarded update, keyed on both +the job identifier and `status = 'cur'`. If nothing matched, the job was +never claimed, or it was already completed. The method throws +`QueueError.jobNotFound` in that case, rather than silently doing +nothing. + +`reclaimInFlight(stream:)` resets a stream's `cur` rows back to `new`, +and clears `session_id`. Its doc comment repeats the same safety argument +as the facade method it backs. Calling it is safe only immediately after +successfully acquiring the stream's `DrainLease`. The lease's staleness +gate is what rules out a live drainer still holding those rows. + +`inFlight()` and `completed(streamID:)` both route through the private +`listJobs(status:streamID:)`. `decodeRow(_:)` turns one database row back +into a `Job`. If the stored JSON for a row's extensions fails to parse, +`decodeRow(_:)` falls back to an empty extensions dictionary, rather than +failing the whole read. + +## DrainLease.swift + +This file provides `DrainLease`, a heartbeat-based, stream-keyed lock. It +lets many processes share one estate's queue, while guaranteeing that +only one of them actively drains a given stream at a time. Two different +streams get entirely independent leases, and both can be held at once. +This is what lets an encode drainer and a dreaming drainer run at the +same time, for example, without either blocking on the other. + +The design deliberately avoids checking whether the prior holder's +process is still alive. A PID-liveness check would need OS-specific code, +querying the process table through platform-specific system calls. The +file's header comment rules this out. It favors something that works +identically on macOS, Linux, and Windows instead: a lease file holding an +owner string and a wall-clock timestamp. + +`tryAcquire(now:)` succeeds in three cases. It succeeds when the lease +file is absent. It succeeds when the lease is already owned by the +caller. It succeeds when the lease's timestamp is older than `ttl`, +fifteen seconds. In every other case, another drainer holds a fresh +lease, and the caller must stand down. The owner string itself combines +the process identifier with a caller-supplied nonce. This combination +matters because the operating system can reuse a process identifier +after a crash. Without the nonce, a reused identifier could be mistaken +for the original holder. + +`heartbeat(now:)` refreshes the timestamp while a drainer actively holds +the lease. The doc comment directs a caller to invoke it roughly every +five seconds, well inside the fifteen-second TTL. This cadence keeps an +ordinary slow drain pass from ever having its own lease expire out from +under it. `isHeldByOther(now:)` answers the same freshness question as +`tryAcquire`, without attempting to take the lease. A caller that only +wants to check before deciding whether to try uses this method instead. +`release()` deletes the lease file on clean shutdown, but only if the +caller still owns it. This guard means a takeover that already happened +is never undone by a late release from the prior holder. + +The worst-case cost of a crashed holder is one full TTL. A takeover +cannot happen sooner than fifteen seconds after the last heartbeat, +because a fresher lease has to be honestly waited out. This is the trade +the design accepts, for staying free of any OS-specific liveness check. + +## Watcher.swift + +This file provides `Watcher`, the directory-change wake source +`FilesystemBackend.watch()` is built on. Its job is narrow, and its +contract is explicit. It calls `onChange` whenever the watched +directory's contents may have changed. Spurious wakes are allowed, +because `drainAvailable()` is always the actual authority on what can be +claimed. A false-positive wake costs one wasted scan. It is never a +correctness problem. + +Three platform strategies exist, tried in order of preference. +`watchKQueue` runs only on Darwin platforms, meaning macOS and iOS. It +opens the directory with `O_EVTONLY`. It attaches a `DispatchSource` +watching kqueue VNODE +events: write, extend, delete, and rename. This strategy gives wake +latency in the microseconds, with no external dependency. + +`watchLinux` first attempts `openInotify`. `openInotify` sets up an +inotify watch for `IN_CREATE` and `IN_MOVED_TO`. The second of these is +the event the atomic tmp-to-new rename actually fires. `runInotifyLoop` +then polls that file descriptor with a five-hundred-millisecond timeout, +so that task cancellation is still checked promptly between events. It +reads and discards whatever event bytes arrive. The handler only needs to +know that something happened. It does not need to know what happened. If +`openInotify` cannot establish a watch, because the kernel's inotify +watch limit is exhausted or the filesystem does not support inotify, +`watchLinux` falls back automatically to `watchPoll`. + +Every other platform uses `watchPoll` directly. `watchPoll` is a plain +two-hundred-millisecond snapshot comparison of the directory's file +listing. It is correct everywhere, but it can miss a change for up to one +poll interval, about one hundred milliseconds on average. + +Every path calls `onChange` once immediately, before entering its +platform-specific wait loop. This first call means any work already +present when the watcher attaches is not stranded waiting for the next +actual filesystem event. `ContinuationBox` is the small bridge the Darwin +path uses. It lets an `async` caller wait on a `DispatchSource`'s cancel +handler. That handler fires on an ordinary `DispatchQueue` callback, +rather than inside Swift's structured concurrency. + +## QueueKitTelemetry.swift + +This file provides `QueueLatencyWindow` and `reportQueueStats`. These are +the self-report telemetry QueueKit emits through IntellectusLib, +MOOTx01's telemetry library, after every `drain()` call. + +The reporting path is additive, and it is deliberately cheap when +unused. The very first check inside `reportQueueStats` is a single +atomic flag read, `Intellectus.isEnabled`. The function returns +immediately when that flag is false. The file's header comment puts the +cost of this check at about one nanosecond. When telemetry is enabled, +the function makes one `pendingCount()` call. It then emits up to six +named metrics. Every metric is tagged with the estate identifier and the +kit name, so a fleet of estates can be compared metric by metric. + +`QueueLatencyWindow` keeps a fixed-capacity rolling list of recent drain +latencies, one hundred samples by default. It computes an arbitrary +percentile from that list on demand. `percentile(_:)` guards its input +explicitly. A percentile argument that is not a finite number between +zero and one hundred, for example `NaN` or a value produced by a +caller's arithmetic error, returns `0` rather than computing an array +index from it. Converting a non-finite floating-point value to an +integer index can crash the program outright on some inputs. + +`reportQueueStats` treats a failed `pendingCount()` read as its own +signal, rather than as a zero. Reporting `queue.depth = 0` when the read +actually failed would be indistinguishable from an honestly empty queue. +It would tell whoever is watching the telemetry that everything is +drained, when the truth is that the read itself did not work. Instead, a +failed read skips the `queue.depth` metric entirely, in favor of a +dedicated `queue.depth_unavailable` counter. Every metric that can only +be computed honestly from a known depth is skipped along with it. +`queue.idle_nonempty` is one such metric, as is part of +`queue.head_of_line_age_s`. `queue.drain_count` and the two latency +percentiles have no such dependency, and they are always reported. + +## Rust and Python Ports and Conformance + +The `rust/` directory contains a second, independent implementation of +the package. It has eight source files: `error.rs`, `job.rs`, +`backend.rs`, `filesystem.rs`, `facade.rs`, `drain_lease.rs`, +`persistencekit.rs`, and `lib.rs`. A `persistencekit` feature flag gates +`persistencekit.rs`. These files reimplement the facade, both backends, +and the drain lease, for non-Apple hosts. + +The `python/queuekit/` directory contains a third implementation. It +covers only `FilesystemBackend`. The package specification restricts +Python to that one backend, since Python tooling has no need of the +PersistenceKit database path. + +All three implementations are gated by the same conformance fixtures. +These fixtures live in `Tests/QueueKitTests/Fixtures/`, as recorded job +and completion-signal input and output pairs. Each fixture pins one +exact set of bytes. Rust's own `rust/tests/conformance.rs` loads the +identical fixture files the Swift test suite reads. It asserts that +Rust's encoder produces the same bytes Swift already produced. This +mechanism is what turns "the wire format is a contract" from an +assertion in this document into something a test suite actually checks. +Suppose a field name, a key order, or an encoding rule changes in any one +implementation. The shared fixtures catch the drift immediately. They +catch it in whichever implementation was not updated to match. diff --git a/packages/kits/QueueKit/docs/OVERVIEW.md b/packages/kits/QueueKit/docs/OVERVIEW.md new file mode 100644 index 0000000..25fa763 --- /dev/null +++ b/packages/kits/QueueKit/docs/OVERVIEW.md @@ -0,0 +1,182 @@ +--- +doc: OVERVIEW +package: QueueKit +repo: moot-system +authored_commit: 909513d0a8ecb1e9e903af9f4d25b3e4f2528242 +authored_date: 2026-07-04 +sources: + - path: Sources/QueueKit/DrainLease.swift + blob: 3f1df6aa1fab44c993bd49d96de179ff03450303 + - path: Sources/QueueKit/FilesystemBackend.swift + blob: c0e5f536e20b1def11875e57b587e8392b6f936d + - path: Sources/QueueKit/Job.swift + blob: 5f39e97a8113e47de2f2a18563446072de8ec6cf + - path: Sources/QueueKit/ObservationStatus.swift + blob: a32b0e8c93311ce498d12ffb4df864a2652c707e + - path: Sources/QueueKit/PersistenceKitBackend.swift + blob: 96175a8b27c8b6e929f417d95b141f727c5179d7 + - path: Sources/QueueKit/QueueBackend.swift + blob: eb67e0b821bc9bfdec8cfde621c5e70d4295d331 + - path: Sources/QueueKit/QueueError.swift + blob: 2697c4b7404b9e04259267cd8f4008030ebb6754 + - path: Sources/QueueKit/QueueKit.swift + blob: 60dfaa1e8f92ec051810b50d8b9cadc47388c02f + - path: Sources/QueueKit/QueueKitTelemetry.swift + blob: 29024f112bf133012283205175aa336b8d80d7c9 + - path: Sources/QueueKit/Watcher.swift + blob: cf2b270b9c60da34f7a25c016f8c18b6ba6149e4 +--- + +# QueueKit Overview + +## What This Library Does + +QueueKit is a general-purpose work queue. It lets one part of MOOTx01 hand a +unit of work to another part. The two parts need not run in the same +process. They need not even run at the same moment. + +MOOTx01 is an on-device AI memory system. It stores what an AI observes over +time. It helps the AI recall that record later. Many of its jobs are best +done off to the side, by a separate worker. Encoding a memory is one +example. Running a dreaming pass over the estate is another. Replaying a +signal is a third. Handing off this work means the part of the system that +noticed it does not have to wait for it to finish. + +QueueKit is a Kit. A Kit is a larger package that composes libraries into a +subsystem. A kit may depend on a library. A library never depends back on a +kit. QueueKit lives in moot-system, the repository that houses the kits that +hold MOOTx01 together at the system level. + +The unit of work is called a `Job`. A job carries an identifier, the name of +the stream it belongs to, and a submission timestamp. It also carries a +priority and an arbitrary payload of bytes. A producer calls `send` to hand +a job to the queue. A consumer calls `drain` to claim available jobs. The +consumer does the work, then calls `reply` to mark each job finished. +QueueKit exposes exactly four permanent methods for this cycle: `send`, +`drain`, `watch`, and `reply`. Every backend supports all four the same way, +no matter which storage mechanism sits underneath it. + +## The Problem It Solves + +Handing work across a process boundary is harder than it looks. Three +requirements make it hard. QueueKit exists to meet all three at once. + +First, the handoff must survive a crash. The producing process might die +partway through. The queue itself might die. The consuming process might +die. In every case, no job may be lost without a trace, and no job may be +claimed twice at once. A job is claimed exactly once at a time. A completed +job leaves a durable record of how it finished. + +Second, independent workloads must not interfere with each other while they +share one queue. An encode worker and a dreaming worker might both draw from +the same estate's queue. Neither should be able to claim, or block on, the +other's jobs. QueueKit calls a workload a stream. Every operation has a +stream-scoped form, so consumers can share one queue without stepping on +each other. + +Third, the format on disk or in the database must mean the same thing no +matter which programming language wrote it or read it. MOOTx01 ships Swift +code for Apple devices. It ships Rust code for cross-platform daemons and +services. It ships Python code for command-line tooling. A job enqueued by +one language's producer must be decodable, byte for byte, by another +language's consumer. QueueKit treats its wire format as a contract. The wire +format is the exact bytes a job or a completion signal is encoded as. +Conformance fixtures shared across all three implementations gate this +contract. + +## How It Works + +QueueKit separates the fixed four-method interface from the storage +mechanism underneath it. A backend is anything that conforms to the +`QueueBackend` protocol. A backend can write a job, claim the available +jobs, watch for new ones, and mark a job complete. The public `QueueKit` +class does not know or care which backend it holds. It forwards every call. +It adds only the telemetry and the await-drain convenience described below. + +Two backends ship today. `FilesystemBackend` needs no database. It lays out +four subdirectories under a root: `tmp`, `new`, `cur`, and `done`. This +follows the maildir layout that mail servers such as Postfix have used for +decades, to hand off messages between processes without a lock file. A job +is written into `tmp`, then renamed atomically into `new`. Claiming it is +another atomic rename, into `cur`. Finishing it renames it into `done`. +Every step relies on the filesystem's own atomicity guarantee. Two +processes racing to claim the same job can never both win. + +`PersistenceKitBackend` stores jobs as rows in a table inside PersistenceKit, +MOOTx01's storage kit. It suits an estate that already keeps a shared, +encrypted SQLite database open for other purposes. Claiming a batch of jobs +here is one guarded database transaction. That transaction flips every +available row from `new` to `cur`, under a single session identifier. It +then reads back exactly the rows that transaction claimed. This scales as +one pass over the table, rather than as one query per job. + +Every job carries a submission timestamp built from a Hybrid Logical Clock, +or HLC. An HLC is a clock design that combines wall-clock time with a small +counter. Events from different machines or processes still sort into one +gap-free order this way, even when the machines' clocks disagree slightly. +QueueKit uses the HLC ordering to decide which job is oldest. This matters +when a consumer wants first-in-first-out delivery. It also matters when +telemetry reports how long the oldest waiting job has been sitting in the +queue. + +Two supporting mechanisms keep multi-process operation honest. `DrainLease` +is a heartbeat lock, keyed by stream. It lets many processes share one +estate, while guaranteeing that only one of them actively drains a given +stream at a time. A stale lease, one whose holder has not renewed it +recently, is automatically taken over. `Watcher` gives the filesystem +backend an efficient way to notice new work without spinning a loop. It +uses the kqueue mechanism on Apple platforms. It uses inotify on Linux. +Everywhere else, it falls back to a plain poll every two hundred +milliseconds. It falls back automatically whenever the faster mechanism +cannot be set up. + +## How the Pieces Fit + +Figure 1 shows the library's topology: its major parts, and how a job moves +through them. + +![Figure 1. Topology of QueueKit](topology.svg) + +*Figure 1. Topology of QueueKit. A producer calls the `QueueKit` facade. The +facade delegates to whichever backend is mounted. Dashed regions mark +external kit boundaries. One boundary is the encrypted database that +PersistenceKit owns. Another is the telemetry sink that IntellectusLib +owns. A third dashed region marks the maildir directories, the filesystem +backend's own on-disk state.* + +A caller mounts `QueueKit` once, over either backend. Afterward, the caller +talks only to the facade. `send` and the batch form `send(batch:)` hand +jobs to the backend. `drain` and the stream-scoped `drain(stream:)` claim +available jobs. Both return the claimed jobs together with a session +identifier. That identifier groups everything claimed in one call. `watch` +subscribes a handler that fires for every job as it becomes available. On +the filesystem backend, `Watcher` drives this. On the PersistenceKit +backend, a database change observer drives it instead. `reply` marks +claimed jobs finished with a terminal status. It has a single-job form, a +per-session form, and a per-batch form. On the filesystem backend, `reply` +also writes a durable signal file recording how the job ended. + +`awaitDrain` is a convenience. It is built on top of the four permanent +methods, rather than added as a new backend capability. It polls the +pending count and the in-flight count until both reach zero. A bulk caller +can use it to wait for every enqueued job to finish, without writing its own +polling loop. An importer finishing a batch is one example. A test verifying +that a pipeline emptied is another. + +Telemetry is additive and off by default. When enabled, `QueueKitTelemetry` +reports queue depth, drain throughput, and claim latency percentiles. It +also reports the age of the oldest waiting job. It sends every report to +IntellectusLib, MOOTx01's self-report telemetry library. Each report is +tagged by estate, so a fleet of estates can be watched side by side. + +## What Ships in the Package + +The package ships ten Swift source files. These implement the facade, the +two backends, and their supporting infrastructure. A Rust port lives in +`rust/`. It has eight source files and roughly three thousand lines. It +reimplements both backends, for non-Apple hosts. A Python port lives in +`python/queuekit/`. It reimplements the filesystem backend only, since +command-line tooling has no need of a database backend. A shared set of +conformance fixtures lives under `Tests/QueueKitTests/Fixtures/`. These are +recorded job and completion-signal input and output pairs. They gate all +three implementations to the same wire format, byte for byte. diff --git a/packages/kits/QueueKit/docs/topology.svg b/packages/kits/QueueKit/docs/topology.svg new file mode 100644 index 0000000..a69eacb --- /dev/null +++ b/packages/kits/QueueKit/docs/topology.svg @@ -0,0 +1,128 @@ + + + + + + + + + + + + + QueueKit: a job crosses a process boundary + + + + Producer + send / send(batch:) + + + QueueKit facade + drain · watch · reply + awaitDrain + + + FilesystemBackend + POSIX maildir, no DB + + + PersistenceKitBackend + shared SQLite table + + + Consumer + drain worker / watch handler + + + + + + + + + + + reply(to:status:) / reply(session:) / reply(batch:) + + + + Watcher + kqueue · inotify · poll + + + DrainLease + per-stream heartbeat TTL + + + QueueKitTelemetry + queue.* metrics + + + + wake (spurious OK) + + + + + gates reclaimInFlight + + + + reportQueueStats after drain() + + + + FilesystemBackend's own on-disk state + + Maildir + tmp/ → new/ → cur/ → done/ + + + External kit boundary: PersistenceKit + + queue.sqlite + queuekit_jobs table + + + External kit boundary: IntellectusLib + + Telemetry sink + off by default, ~1 ns gate + + + + + + diff --git a/packages/libs/LoopbackHTTP/docs/AGENT_MAP.md b/packages/libs/LoopbackHTTP/docs/AGENT_MAP.md new file mode 100644 index 0000000..2bede22 --- /dev/null +++ b/packages/libs/LoopbackHTTP/docs/AGENT_MAP.md @@ -0,0 +1,79 @@ +--- +doc: AGENT_MAP +package: LoopbackHTTP +repo: moot-system +authored_commit: 909513d0a8ecb1e9e903af9f4d25b3e4f2528242 +authored_date: 2026-07-04 +sources: + - path: Sources/LoopbackHTTP/HTTPWire.swift + blob: c07525c2583c06aa8406287eef5cb17279b9b86a + - path: Sources/LoopbackHTTP/POSIXSocket.swift + blob: 6dfba4dee7ad3121d47c8bfc937854c6f13ad1b1 +--- + +# AGENT_MAP: LoopbackHTTP + +PURPOSE: zero-dependency, loopback-pinned HTTP/1.1 server primitive (transport + wire parsing/framing only; no auth/policy). Bind loopback socket → accept → recv bytes → HTTPRequest.read parses → caller builds HTTPResponse or drives SSEStream → send/sendAll writes bytes back. + +DEPS: imports Foundation (HTTPWire.swift); Glibc (Linux) / Darwin (Apple) via `#if canImport(Glibc)` guard (POSIXSocket.swift). No external SwiftPM packages (zero-dependency kit rule). No rust/ port; extraction target is OS transport glue, exempt from the Swift/Rust parity discipline (ADR-LOOPBACKHTTP-001); parity for the MCP transport is enforced at the JSON-RPC wire, not here. Imported by: moot-mgr (monitor daemon dashboard read-API + control listener), resident mootx01 MCP daemon (JSON-RPC transport, SSE notifications). + +ENTRY POINTS (most callers need only these): +- POSIXSocket.swift:43 `POSIXSocket.listenLoopbackTCP(port:) throws -> (fd, port)`; bind+listen, loopback-only +- HTTPWire.swift:95 `HTTPRequest.read(fd:maxHeaderBytes:maxBodyBytes:) -> HTTPRequest?`; parse one request off a connected fd +- HTTPWire.swift:201 `HTTPResponse.send(fd:)`; serialize + write one response +- HTTPWire.swift:252/:259 `SSEStream.writeHead()` / `.send(_:)`; SSE head + data frames + +## Symbol Table + +### Transport: POSIXSocket.swift +- :32 `enum POSIXSocket`: namespace; all calls synchronous/blocking, run off cooperative pool +- :43 `listenLoopbackTCP(port:) throws -> (fd: Int32, port: UInt16)`: binds hard-pinned to 127.0.0.1 (never INADDR_ANY); port 0 = OS-assigned, read back via getsockname; listen backlog 16 +- :91 `listenUnix(path:) throws -> Int32`: UDS listener; unlinks stale path first; chmod 0600 BEFORE listen (closes bindable-but-world-readable window); listen backlog 16 +- :134 `acceptOne(_ listenFD:) -> Int32?`: accept one conn; nil on failure, caller decides retry +- :141 `recv(_:max:) -> Data?`: up to `max` bytes; empty non-nil Data = clean EOF; nil = read error +- :150 `sendAll(_:_:) -> Bool`: loops write() until all bytes sent or a write fails; single source of truth for "fully flushed to socket" used by both HTTPResponse.send and SSEStream +- :166 `enum SocketError: Error, Sendable, Equatable`: structured errors (project rule: enums not bare optionals) +- :168 `case syscall(String, Int32)`: named syscall + errno +- :170 `case pathTooLong`: UDS path exceeds fixed sun_path buffer + +### Request: HTTPWire.swift +- :27 `struct HTTPRequest: Sendable`: method/path(no query)/query(raw, no '?')/headers(lowercased keys)/body +- :36 `init(method:path:query:headers:body:)`: public memberwise; used directly by tests to synthesize requests +- :46 `bearerToken: String?`: reads Authorization, case-insensitive "bearer " prefix strip + trim; CONVENIENCE ONLY, no validation +- :54 `origin: String?`: raw Origin header passthrough; CONVENIENCE ONLY, no allow-list check +- :69 `wantsEventStream: Bool`: Accept:text/event-stream OR exact query param `stream=1` (split on '&', exact match; NOT substring; see GOTCHAS) +- :95 `static func read(fd:maxHeaderBytes:maxBodyBytes:) -> HTTPRequest?`: buffers until "\r\n\r\n"; nil if header block exceeds maxHeaderBytes before terminator found, or on recv failure +- :114 `private static func parse(buffer:headerEnd:fd:maxBodyBytes:) -> HTTPRequest?`: request-line split, header lowercase-key parse, Content-Length-bounded body read (truncates silently at maxBodyBytes, does not error) + +### Response: HTTPWire.swift +- :164 `struct HTTPResponse: Sendable`: status/headers/body, caller-composed (not a closed consumer-specific case set; see GOTCHAS) +- :169 `init(status:headers:body:)`: headers/body default to empty +- :176 `static func json(status:body:) -> HTTPResponse`: Content-Type: application/json +- :182 `static func asset(contentType:body:) -> HTTPResponse`: 200 + Cache-Control: no-store (redeployed UI never served stale) +- :191 `static var notFound: HTTPResponse`: 404 + `{"error":"not_found"}` +- :201 `func send(fd:)`: computes Content-Length from actual body (overrides any caller value); header order: Content-Type, Content-Length, then rest sorted alphabetically, then Connection: close ALWAYS last (caller-supplied Connection does not suppress it) +- :221 `static func reason(_ status:) -> String`: reason phrase for 200/400/401/403/404/413/500; default "OK" + +### SSE: HTTPWire.swift +- :242 `struct SSEStream: Sendable`: long-lived push connection wrapper; caller owns source/cadence/lifetime/fd close +- :245 `init(fd:)` +- :252 `func writeHead() -> Bool`: call exactly once before any frame; false = peer already gone +- :259 `func send(_ payload:) -> Bool`: writes `"data: \(payload)\n\n"`; false = write failed, caller should stop +- :264 `static let responseHead: Data`: fixed head: 200 OK, text/event-stream, no-cache, keep-alive + +## INVARIANTS / GOTCHAS + +- AUTH-FREE INVARIANT (ADR-LOOPBACKHTTP-001 condition 3): no auth/policy logic anywhere in this package. `bearerToken`/`origin` are read-only conveniences; accept/reject is composed ABOVE this library by each consumer (none in Community Edition, bearer+Origin in moot-mgr, OAuth 2.1 in EE-only remote layer). Do not add auth enforcement here; it would break the "one binary, all editions" property. +- LOOPBACK-ONLY BY CONSTRUCTION: `listenLoopbackTCP` binds the literal `127.0.0.1` (0x7F000001 big-endian), never `INADDR_ANY`/0.0.0.0. Do not parameterize the bind address to accept a caller-supplied host. +- UDS PERMISSION WINDOW: `listenUnix` must chmod 0600 BEFORE `listen()`, not after; the current ordering closes the world-readable window; do not reorder. +- SSE CSRF NOTE: `wantsEventStream` via `?stream=1` is a plain GET, so browser CORS preflight does NOT run. A cross-origin page can open an SSE connection if the loopback port is reachable. Any consumer serving sensitive data over SSE MUST check `request.origin` itself before upgrading; this library does not and will not enforce it (see AUTH-FREE INVARIANT). +- `wantsEventStream` query check is EXACT-MATCH on a `&`-split parameter, not `query.contains("stream=1")`. The substring form was a real bug (matched "stream=10", "mystream=1", "x=stream=1"); regression-locked by Tests/LoopbackHTTPTests/WantsEventStreamTests.swift. Do not revert to substring matching. +- Header names are lowercased on parse (`parse(buffer:...)`) so all internal/consumer lookups (`headers["content-length"]`, `headers["authorization"]`) use lowercase keys only; HTTP header names are case-insensitive on the wire but this dictionary is not. +- Body truncation, not rejection: `HTTPRequest.read`/`parse` silently truncate a body at `maxBodyBytes` when Content-Length exceeds it; they do not return nil or an error. A listener that must reject oversize bodies (e.g. the MCP `tools/call` listener) needs to check Content-Length itself before/after calling, or size its cap so truncation cannot corrupt a valid request. +- `maxHeaderBytes`/`maxBodyBytes` are per-call parameters, not fixed constants (default 64 KiB each); deliberately, so a small dashboard control listener and a larger MCP listener each pick their own cap (ADR-LOOPBACKHTTP-001 condition 2). +- `HTTPResponse.send` always appends `Connection: close` last, even if the caller supplied their own `Connection` header (that value is emitted earlier, in the sorted section, and does not suppress the trailing one). Every response this library writes closes the connection; there is no keep-alive path for ordinary responses (SSE is the sole long-lived exception). +- `HTTPResponse`/`SSEStream` are open, caller-composed value types, not a closed enum of consumer-specific response shapes; this genericity (added in P1a) is what let a second consumer (resident MCP daemon) reuse the library without modifying it. Do not narrow them back to a fixed case set for one consumer's convenience. +- All `POSIXSocket` calls are synchronous/blocking. Callers must run them off a dedicated Thread or `Task.detached`, never inline on a thread that must stay responsive. +- Network.framework (`NWListener`) was tried and rejected: it returns POSIXErrorCode 22/EINVAL binding a listening socket in this project's command-line (non-app-bundle) build environment, on every configuration tested. Do not reintroduce it as "simpler" without re-verifying that constraint is gone. +- Swift-only, no Rust port: this library is OS-transport glue, not deterministic substrate compute, so it is exempt from the Swift/Rust byte-parity discipline that governs libraries like LatticeLib. ARIA_MCP-rust hand-rolls its own std::net transport under a no-FFI rule; the two sides agree only at the JSON-RPC message level. +- Zero external SwiftPM dependencies (Package.swift target has none); adding one requires re-justifying against the kit dependency rules that motivated this extraction in the first place. diff --git a/packages/libs/LoopbackHTTP/docs/DETAILS.md b/packages/libs/LoopbackHTTP/docs/DETAILS.md new file mode 100644 index 0000000..4a14340 --- /dev/null +++ b/packages/libs/LoopbackHTTP/docs/DETAILS.md @@ -0,0 +1,275 @@ +--- +doc: DETAILS +package: LoopbackHTTP +repo: moot-system +authored_commit: 909513d0a8ecb1e9e903af9f4d25b3e4f2528242 +authored_date: 2026-07-04 +sources: + - path: Sources/LoopbackHTTP/HTTPWire.swift + blob: c07525c2583c06aa8406287eef5cb17279b9b86a + - path: Sources/LoopbackHTTP/POSIXSocket.swift + blob: 6dfba4dee7ad3121d47c8bfc937854c6f13ad1b1 +--- + +# LoopbackHTTP Details + +This document walks through every source file in the package. Read +`OVERVIEW.md` first for the big picture. Files appear here in pipeline +order. The socket transport comes first, then the HTTP protocol layer +built on top of it. + +## POSIXSocket.swift + +This file provides `POSIXSocket`, a set of small blocking-socket helpers. +It also provides `SocketError`, the error type they throw. Everything in +this file is synchronous. A call blocks the calling thread until the +operating system answers. Callers should run these calls off a dedicated +thread or a detached task. They should never run on a thread that must stay +responsive. + +Two design choices explain the whole file. First, the code targets plain +POSIX sockets rather than Apple's `Network.framework`. That higher-level +framework refused to bind a listening server socket in this project's +command-line build environment. It returned a low-level POSIX error code on +every configuration tried. The restriction is tied to running outside an +app bundle. Plain sockets bind correctly in that same environment. The +library uses them directly, and the same security guarantees apply either +way. + +Second, the file imports `Glibc` on Linux and `Darwin` on Apple platforms +through a compile-time check. The same source file compiles on both without +any conditional logic beyond the import line. + +`listenLoopbackTCP(port:)` opens a TCP listening socket. It binds the +socket to `127.0.0.1`, never to `0.0.0.0`. Binding to `0.0.0.0` would accept +connections from any network interface, not just the local machine. The +bind address is a literal numeric value. It is not a name the operating +system could resolve differently, so this guarantee cannot be weakened by a +misconfigured host file or DNS entry. + +Passing port zero lets the operating system pick an unused port. The +function reads the actual bound port back with `getsockname` and returns +it. This matters because the caller often needs to tell another process +which port to connect to. This function is the one place in the library +that decides which network interface can reach the server at all. It +decides in favor of the strictest possible answer every time, regardless of +what the caller asks for. + +`listenUnix(path:)` opens a Unix-domain socket at a filesystem path instead +of a network port. A Unix-domain socket is a connection endpoint that lives +in the filesystem. Only other processes on the same machine can reach it. A +web browser has no way to speak to one at all. This makes it a natural +channel for the most sensitive operations, since browser-based attacks +cannot structurally reach it. + +The function removes any stale socket file left over from a previous +crashed run before binding. A dead file cannot then block a fresh start. +After binding, it changes the file's permissions to owner-read-write-only +before calling `listen`. That mode is 0600. There is no window during which the +socket file exists but is readable by other users on the machine. This chmod +step is what turns "only this user can connect" from a hope into a +guarantee the filesystem itself enforces. + +`acceptOne(_:)` accepts one pending connection on a listening socket. It +returns the new connection's descriptor, or `nil` if the accept call +failed. It is a thin wrapper. The decision of whether and how to retry +after a failure belongs to the caller, not to this function. + +`recv(_:max:)` reads up to `max` bytes from a connected socket. It returns +them as `Data`, or `nil` on a read error. An empty but non-nil result means +the peer closed the connection cleanly. A caller reading in a loop uses that +signal to know when to stop. + +`sendAll(_:_:)` writes every byte of `data` to a socket. It loops until the +whole buffer is sent or a write fails. A single call to the underlying +system write is not guaranteed to send the whole buffer at once. The kernel +is free to accept less. This function is therefore the one place other +files in the package go to guarantee that a full response, or a full SSE +frame, actually reaches the wire. It returns `false` on any failed write. A +caller uses that as its signal that the peer is gone and the connection +should be torn down. + +`SocketError` is the file's structured error type. The package's error +handling rule prefers enumerated error cases over an optional value with no +explanation. A caller, or a log line, can then say exactly which system +call failed. It can name the error number too, rather than only report that +something failed. `.syscall(String, Int32)` names the failing call and carries its +`errno` value. `.pathTooLong` covers the one Unix-domain-socket case where +the caller's path does not fit the fixed-size buffer the operating system's +socket address structure provides. + +## HTTPWire.swift + +This file provides three public types. `HTTPRequest` is a parsed incoming +request. `HTTPResponse` is an outgoing response the caller builds. +`SSEStream` is the narrower Server-Sent-Events framing used for a +long-lived push connection. All three depend on `POSIXSocket` for the +actual byte transfer. Nothing in this file opens a socket itself. + +The file's scope is deliberately narrow. It is not a general-purpose HTTP +server. It parses a request line, a set of headers, and an optional body +bounded by a `Content-Length` header. It writes exactly one response per +connection before that connection closes. It does not support chunked +transfer encoding, HTTP/1.1 keep-alive across many requests, or HTTP/2. +Every response this library writes closes the connection afterward. An SSE +stream is the one exception, since the caller keeps it open on purpose. + +### HTTPRequest + +`HTTPRequest` is a plain value. It holds the request `method`, for example +`"GET"`. It holds the `path`, with any query string already removed. It +also holds the raw `query` string that followed the `?`. The `query` string +is empty if there was none. It also holds a lowercase-keyed table of `headers` and the +request `body` as `Data`. Storing the path and query separately means every +consumer performs that split identically. No consumer has to write its own, +possibly buggy, parsing. + +`bearerToken` reads the `Authorization` header. It checks that the header +begins with the case-insensitive prefix `"bearer "`. It returns the +remaining token text with surrounding whitespace trimmed, or `nil` if the +header is absent or does not match. This is a convenience reader only. The +library extracts the token text so a caller does not have to. It never +decides whether that token is valid. The decision to accept or reject a +request based on this value belongs entirely to the code above this +library. This keeps with the package's edition-neutral, authentication-free +design. + +`origin` reads the `Origin` header directly, with no further processing. A +browser sets this header to say which web page's script initiated the +request. Like `bearerToken`, this reader exists purely so a caller can +inspect the value. The library does not compare it against an allow-list +itself. + +`wantsEventStream` decides whether the caller likely wants an SSE response +instead of an ordinary one. It returns `true` if the request's `Accept` +header mentions `text/event-stream`. It also returns `true` if the query +string carries an exact `stream=1` parameter. The exact-parameter check +matters for a specific reason. An earlier version of this logic used a +substring check, `query.contains("stream=1")`. That check incorrectly +matched `stream=10`, `mystream=1`, and `x=stream=1`. None of those mean what +`stream=1` means. + +The current implementation splits the query on `&`. It compares each +resulting parameter for exact equality to `"stream=1"`. A longer number or +an unrelated parameter name containing the same substring cannot fool it. +A dedicated regression test file, +`Tests/LoopbackHTTPTests/WantsEventStreamTests.swift`, exists specifically +to keep this fix from silently regressing. + +This same property carries a documented security note. An SSE request made +this way is an ordinary browser `GET` request. The browser's cross-origin +preflight check never runs for it. A page on an unrelated origin can +therefore open an SSE connection to a loopback server, if that server is +reachable at all. Any consumer that serves sensitive data over SSE must +check the `origin` value itself. It must refuse an unexpected origin before +upgrading to a stream. This library does not perform that check, again by +the authentication-free design. + +`HTTPRequest.read(fd:maxHeaderBytes:maxBodyBytes:)` is the entry point that +turns raw socket bytes into a parsed request. It reads repeatedly from the +socket, accumulating bytes into a buffer, until it finds the header +terminator. The terminator is `"\r\n\r\n"`, the blank line that ends an HTTP +header block. If the accumulated header block ever exceeds +`maxHeaderBytes` before that terminator appears, the function gives up and +returns `nil`. It will not keep buffering an unbounded amount of untrusted +network data. This cap protects the server's memory against a hostile or +broken client sending an endless header. + +The caps are parameters rather than fixed constants, because different +listeners have different legitimate needs. A small dashboard control +listener wants a small cap. A listener serving `tools/call` requests for the +Model Context Protocol wants a larger one. Each caller can choose the cap +that matches its own traffic, without every listener sharing one compromise +value. + +The private `parse(buffer:headerEnd:fd:maxBodyBytes:)` helper does the +actual parsing once the header terminator has been found. It reads the +request line and splits it into method and target. It splits the target on +the first `?` into path and query. It then walks the remaining header +lines. Each line is split on its first colon. The function trims whitespace +from each side and lowercases the header name. + +This normalization matters. Later lookups, such as +`headers["content-length"]`, do not have to guess whether a client sent +`Content-Length` or `content-length` or some other capitalization. HTTP +header names are case-insensitive on the wire. This normalization is what +lets the rest of the library treat them as plain lowercase string keys. + +If a `Content-Length` header is present, the function reads exactly that +many more bytes from the socket, up to `maxBodyBytes`. If the declared +length exceeds the cap, the body is silently truncated to the cap rather +than the request being rejected. A caller for whom truncation would be +dangerous should set its cap high enough that truncation cannot happen. It +can also check the `Content-Length` header itself before calling `read` at +all. + +### HTTPResponse + +`HTTPResponse` is a plain value: a `status` code, a table of `headers`, and +a `body`. The library builds `HTTPRequest` for the caller. This type works +the other way around. The caller builds it, and the library only +serializes it. This inversion is deliberate. An earlier version of this +code hard-coded a closed set of response shapes for one specific consumer. +Generalizing it to a plain value that any caller can construct is what let +a second consumer, the resident MCP transport, reuse the same library +without waiting for new cases to be added here every time its needs +changed. + +Three static convenience constructors cover the shapes most callers need, +without limiting them to only those shapes. `json(status:body:)` builds a +response with an `application/json` content type. `asset(contentType:body:)` +builds a `200 OK` response carrying `Cache-Control: no-store`. That header +tells the browser never to reuse a cached copy. This matters for a locally +served dashboard. A stale cached page after a new build would show outdated +controls, or worse, controls that no longer match the running server's API. +`notFound` is a ready-made `404` response with a small consistent JSON error +body. Every listener reports a missing route the same way. + +`send(fd:)` serializes the response and writes it to the socket through +`POSIXSocket.sendAll`. It computes `Content-Length` itself from the actual +body byte count. It overrides anything the caller supplied for that header. +A caller-supplied length that does not match the real body would produce a +response the receiving HTTP client cannot parse correctly. + +Headers are written in a fixed, deterministic order. `Content-Type` and +`Content-Length` come first, if present. Every other header follows, +sorted alphabetically. A `Connection: close` line is always appended last, +regardless of anything the caller set for that header. The HTTP +specification allows headers in any order, so fixing the order this way is +not required by the specification. It makes every response byte-for-byte +reproducible for testing and logging. That reproducibility matters more +here than flexibility no caller needs. + +`reason(_:)` maps a status code to its human-readable reason phrase, for +example `404` to `"Not Found"`, for the small fixed set of statuses this +library emits. It falls back to `"OK"` for anything else. The reason phrase +is advisory text. HTTP clients are not required to interpret it. + +### SSEStream + +`SSEStream` frames the Server-Sent-Events wire format for one open +connection, identified by its socket descriptor `fd`. `HTTPRequest` and +`HTTPResponse` each cover one request-response exchange. An `SSEStream` +value works differently. The caller holds it for the life of a +long-running push connection. The library plays no role in deciding how +long that is, what triggers each message, or when to close it. + +`writeHead()` writes the fixed SSE response head. That head is the status +line and three headers a client needs to recognize an event stream: +`Content-Type: text/event-stream`, `Cache-Control: no-cache`, and +`Connection: keep-alive`. It must be called exactly once, before any data +frame. It returns `false` if the write failed. The documentation notes that +a failed write means the peer is already gone. A caller seeing `false` +should close the connection rather than attempt to send frames into a +stream nobody is reading. + +`send(_:)` writes one SSE `data:` frame carrying the given payload string. +The wire format is `"data: \n\n"`, the payload text followed by +exactly one blank line, as the SSE specification requires. It also returns +`false` on a failed write, for the same reason. + +Both callers of this type in MOOTx01 decide their own cadence and lifetime +entirely outside this file. The moot-mgr dashboard drives the stream from +polling its own store. The resident MCP transport drives it from JSON-RPC +notifications. `SSEStream` only guarantees that whatever they send is +framed correctly on the wire. diff --git a/packages/libs/LoopbackHTTP/docs/OVERVIEW.md b/packages/libs/LoopbackHTTP/docs/OVERVIEW.md new file mode 100644 index 0000000..c051eb2 --- /dev/null +++ b/packages/libs/LoopbackHTTP/docs/OVERVIEW.md @@ -0,0 +1,121 @@ +--- +doc: OVERVIEW +package: LoopbackHTTP +repo: moot-system +authored_commit: 909513d0a8ecb1e9e903af9f4d25b3e4f2528242 +authored_date: 2026-07-04 +sources: + - path: Sources/LoopbackHTTP/HTTPWire.swift + blob: c07525c2583c06aa8406287eef5cb17279b9b86a + - path: Sources/LoopbackHTTP/POSIXSocket.swift + blob: 6dfba4dee7ad3121d47c8bfc937854c6f13ad1b1 +--- + +# LoopbackHTTP Overview + +## What This Library Does + +LoopbackHTTP is a small HTTP/1.1 server primitive. It listens on a socket. It +reads one request. It lets the caller write one response. + +The library also frames a Server-Sent-Events (SSE) stream. SSE is a way for a +server to push a series of small text messages to a browser. The messages +travel over one open connection. The library does not decide what the +traffic means. It only moves the bytes correctly. + +The library binds strictly to the loopback address, `127.0.0.1`. A loopback +address is one that only the same machine can reach. No other computer on the +network can connect to it. Every listener this library creates is reachable +only from processes on the same device. + +## The Problem It Solves + +Two programs inside MOOTx01 need a local HTTP server. The moot-mgr monitor +daemon serves a small dashboard. The resident mootx01 MCP daemon serves the +Model Context Protocol used by AI tool integrations. + +Before this library existed, each program hand-rolled its own socket and +parsing code. Two hand-rolled copies drift apart over time. One gets a +security fix and the other does not. One handles a header correctly and the +other does not. + +LoopbackHTTP was extracted from moot-mgr under decision record +ADR-LOOPBACKHTTP-001. Both programs now share one audited implementation +instead of maintaining two. + +The library adds no external package dependency. It wraps only the system C +socket API supplied by the operating system: `libc` on Linux and `Darwin` on +Apple platforms. A larger, general-purpose HTTP package such as SwiftNIO was +available. It was rejected because it would pull in a dependency graph +broader than this narrow need justifies. The moot-system kit rules ask each +library to justify every dependency it takes. + +Even Apple's own `Network.framework` was tried. It was rejected for a +concrete reason. Its listener type refused to bind a listening server socket +in this project's command-line build environment. It failed with a +low-level error code on every configuration tested. Plain POSIX sockets bind +correctly in that same environment. The library builds on them directly. + +The library is deliberately authentication-free. It exposes two convenience +readers on a parsed request, `bearerToken` and `origin`, so a caller can read +those header values easily. The library itself never decides whether a +request is authorized. That decision sits one layer above the transport, in +each consumer. The Community Edition of moot-mgr enforces no policy at all. +The full moot-mgr enforces a bearer token plus an Origin header check. An +Enterprise Edition-only remote layer enforces OAuth 2.1. Keeping the +transport policy-free lets the identical compiled library ship inside every +edition unchanged. Only the layer above it differs. + +The library is Swift-only. MOOTx01 keeps a byte-for-byte parity discipline +between a Swift implementation and a Rust implementation for deterministic +computation. Classification and fingerprinting are two examples. Both legs +must produce the exact same answer from the exact same input. + +Binding a socket and parsing HTTP headers is operating-system transport +glue. It is not that kind of deterministic computation, so it falls outside +the parity rule. The Rust side of MOOTx01's MCP transport, ARIA_MCP-rust, +hand-rolls its own separate networking code. A project rule forbids that +code from calling into Swift or C libraries across a foreign-function +boundary. The two sides only have to agree at the JSON-RPC message level, +not at the socket level. + +## How It Works + +The library is organized in two layers, one file each. + +`POSIXSocket.swift` is the transport layer. It opens a listening socket +bound to `127.0.0.1` on a chosen TCP port. It can also open a Unix-domain +socket at a filesystem path locked to owner-only access. It accepts one +incoming connection at a time. It reads or writes raw bytes on the +resulting connection. + +`HTTPWire.swift` is the protocol layer, built on top of the transport. +`HTTPRequest.read(fd:)` reads raw bytes from a connected socket. It parses +them into a structured request: a method, a path, a query string, a table +of headers, and a body. `HTTPResponse` is a value the caller builds: a +status code, headers, and a body. The library serializes that value into +correct HTTP/1.1 wire bytes and writes it back to the socket. `SSEStream` +frames the narrower SSE wire format. It writes one head line once, then any +number of `data:` frames over time, on a connection the caller keeps open. + +A typical request passes through the library in one direction only. Bytes +arrive from `POSIXSocket`. `HTTPRequest.read` turns them into a request +value. The caller inspects that value and decides what to do. The caller's +resulting `HTTPResponse` or `SSEStream` frames go back out through +`POSIXSocket`. Nothing loops back. No component here holds state between one +request and the next connection. + +That simple, one-way shape is why this library needs no larger topology +diagram. It is two files, a handful of value types, and a straight line from +incoming bytes to outgoing bytes. + +## What Ships in the Package + +The package ships two Swift source files and no bundled resources, +artifacts, or Rust port. `Package.swift` declares one library product, +`LoopbackHTTP`, with no external dependencies. It targets macOS 26 and iOS +26. + +The package also builds on Linux Swift. `POSIXSocket.swift` uses a +compile-time import guard that selects `Glibc` there and `Darwin` on Apple +platforms. diff --git a/packages/libs/ObserverSink/docs/AGENT_MAP.md b/packages/libs/ObserverSink/docs/AGENT_MAP.md new file mode 100644 index 0000000..dd6baa8 --- /dev/null +++ b/packages/libs/ObserverSink/docs/AGENT_MAP.md @@ -0,0 +1,104 @@ +--- +doc: AGENT_MAP +package: ObserverSink +repo: moot-system +authored_commit: 909513d0a8ecb1e9e903af9f4d25b3e4f2528242 +authored_date: 2026-07-04 +sources: + - path: Sources/ObserverSink/PersistenceStatsSink.swift + blob: e76c55599795f4bc85b860a4901dc9ab04dd201f + - path: Sources/ObserverSink/StatsStore.swift + blob: 483c736feffd82821a8d77030afde0e182d320fc +--- + +# AGENT_MAP: ObserverSink + +PURPOSE: reusable PersistenceKit-backed telemetry sink for the MOOTx01 manager pipeline (Manager 1.0, Phase 0.5). IntellectusLib.report(_:) → PersistenceStatsSink.receive(_:) → flag-gated SQLite write into StatsStore (metric_samples / event_samples). Also owns topology_snapshots (governor write, dashboard read) and store-level retention. + +DEPS: imports IntellectusLib (StatsSink protocol, StatSample/EventKind datum), PersistenceKit + PersistenceKitSQLite (Storage/SchemaDeclaration/RowStore, SQLiteStorage backend), Foundation, OSLog. Dependency hierarchy (no inversion, per MANAGER_1.0_PLAN.md §4): IntellectusLib (floor) → PersistenceKit (kit) → ObserverSink (this lib). Imported by: none within the moot-system repo at this commit: Package.swift comments name the intended consumers (Intellectus install site in AriaResident/aria-mcp, moot-mgr manager) as living outside this repo. Rust port in rust/ (crate `observer-sink`) mirrors schema + flag semantics exactly; conformance tests in Tests/ObserverSinkTests/ObserverSinkConformanceTests.swift and rust/tests/conformance.rs gate parity (16 scenarios). + +ENTRY POINTS (most callers need only these): +- PersistenceStatsSink.swift:102 `PersistenceStatsSink.init(store:dropboxID:)`: construct the sink; store must already be `.open()`ed +- PersistenceStatsSink.swift:123 `receive(_ sample: StatSample)`: StatsSink conformance; called by Intellectus.report(_:), never call directly +- StatsStore.swift:362 `StatsStore.init(url:) throws`: construct the store +- StatsStore.swift:384 `open() async throws`: apply schema/migrations + seed control rows (idempotent) + +## Symbol Table + +### Sink: PersistenceStatsSink.swift +- :77 `struct PersistenceStatsSink: StatsSink`: Sendable; all mutable state lives in StatsStore, not here +- :82 `store: StatsStore` (private) +- :89 `dropboxID: String` (private): tags every written row; identifies source consumer/process +- :91 `logger` (private): `Logger(subsystem: "com.mootx01.kit", category: "ObserverSink")` +- :102 `init(store:dropboxID:)`: see ENTRY POINTS +- :123 `receive(_:)`: synchronous, non-blocking; dispatches unstructured Task; inside: isMonitoringEnabled() gate → insertMetric/insertEvent by StatSample case → catch-and-log, never throws/propagates + +### Schema constants: StatsStore.swift `enum StatsStoreSchema` +- :49 `enum StatsStoreSchema`: table/column name constants; typo → compile error, not silent empty query +- :54 `metricSamplesTable = "metric_samples"` / :57 `eventSamplesTable = "event_samples"` / :60 `controlTable = "control"` / :117 `topologySnapshotsTable = "topology_snapshots"` +- :65 `tsColumn = "ts"` (TEXT ISO-8601 on every table) / :68 `dropboxIDColumn = "dropbox_id"` +- :73 `nameColumn`, :76 `valueColumn` (REAL), :79 `tagsColumn` (TEXT JSON): metric_samples +- :84 `kindColumn` (EventKind raw string), :87 `nounTypeColumn` (INTEGER ordinal), :91 `rowIDColumn = "estate_row_id"` (NOT the synthetic PK), :94 `estateColumn`: event_samples +- :99 `keyColumn` (PK), :102 `controlValueColumn`: control +- :107 `monitoringKey = "monitoring"` (value "1"/"0") / :112 `retentionCutoffKey = "retention_cutoff"` (ISO-8601, default epoch-zero) +- :120 `generatedAtColumn`, :123 `payloadColumn`, :132 `topologyFingerprintColumn` (nullable, v3): topology_snapshots + +### Store: StatsStore.swift `final class StatsStore: Sendable` +- :160 `final class StatsStore: Sendable`: wraps SQLiteStorage +- :164 `storage: SQLiteStorage` (private) / :165 `logger` (private) +- :175 `static let schemaVersion = 3`: v1: metric/event/control; v2: +topology_snapshots; v3: +topology_fingerprint (nullable) +- :193 `static let schema: SchemaDeclaration`: 4 tables, 2 indices (ts on metric_samples/event_samples), 2 migrations (v1→v2 createTable, v2→v3 addColumn) +- :317 migrations block: v1→v2 :321, v2→v3 :340; SQLite backend creates fresh DBs at latest version directly (migrations replay only matters for InMemory/upgrade path) +- :362 `init(url:) throws`: fresh random estateID per StatsStore instance; no I/O +- :384 `open() async throws`: applies schema, then :396/:398 `seedControlIfAbsent` for monitoring="0" and retention_cutoff=epoch-zero; SEED-IF-ABSENT not upsert +- :412 `seedControlIfAbsent(key:value:)` (private): existence check then insert; preserves operator-set values across reopen +- :433 `close() async`: idempotent +- :448 `isMonitoringEnabled() async throws -> Bool`: control["monitoring"]=="1"; missing row → false (safe default) +- :470 `setMonitoringEnabled(_:) async throws`: upsert on keyColumn; manager-only caller by convention (sink never writes this) +- :496 `insertMetric(name:value:tags:ts:dropboxID:) async throws`: assigns row_id UUID; tags→JSON via encodeTagsJSON; ts(Double epoch)→.timestamp(Date) at this boundary +- :532 `insertEvent(kind:nounType:rowID:estate:ts:dropboxID:) async throws`: rowID stored in estate_row_id column, NOT row_id (that's the synthetic PK) +- :576 `writeTopologySnapshot(estate:generatedAt:payload:fingerprint:) async throws`: upsert keyed on estate (latest-wins, no history); throws StorageError.invalidQuery if payload isn't valid UTF-8; fingerprint nil → .null column +- :615 `latestTopologySnapshot(estate:) async throws -> Data?`: estate=nil → newest generated_at across ALL estates (dashboard "all" view); uses :679 generatedAtInstant to tolerate both .timestamp and .text read-back reps (SQLite vs InMemory): DO NOT match only .timestamp, that was a real bug (all rows tie at .distantPast under SQLite) +- :655 `loadTopologyFingerprint(estate:) async throws -> String?`: nil if no snapshot / pre-v3 row / written without fingerprint +- :679 `generatedAtInstant(_:) -> Date` (private): see above +- :696 `queryMetrics(dropboxID:) async throws -> [MetricRow]`: full-table (optionally dropbox-filtered) read, ts ascending +- :725 `queryMetricsByNames(_:dropboxID:) async throws -> [MetricRow]`: `WHERE name IN (...)`; empty names → [] immediately; USE THIS not queryMetrics+filter on hot read paths +- :769 `countMetrics() async throws -> Int`: COUNT(*), no row decode +- :784 `queryEvents(dropboxID:) async throws -> [EventRow]`: full-table (optionally dropbox-filtered) read, ts ascending +- :818 `deleteMetricsBefore(cutoff:now:) async throws -> Int`: ts < cutoff; updates retention_cutoff control row to `cutoff` (not `now`) +- :842 `deleteEventsBefore(cutoff:now:) async throws -> Int`: same semantics, event_samples +- :873 `storageStats(now:) async throws -> StorageStats?`: store's OWN SQLite backend health (WAL frames, file size, page/freelist counts), not an observed estate's storage; via StorageIntrospection, SQLite backend always returns non-nil +- :883 `recordRetentionCutoff(_:now:) async throws` (private): upserts retention_cutoff as ISO-8601 +- :907 `static let iso8601Formatter: DateFormatter`: "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'", UTC, en_US_POSIX; matches SQLiteStorage's timestamp codec +- :924 `encodeTagsJSON(_:) -> String` (private): `.sortedKeys` for determinism; empty/unencodable → "{}" +- :939 `static func decodeTagsJSON(_:) -> [String: String]`: parse failure → [:] (forward-compatible) + +### Row types +- :951 `struct MetricRow: Sendable`: rowID/name/value/tags/ts/dropboxID; :966 `init?(storageRow:)` returns nil on any column type mismatch (compactMap-friendly) +- :986 `struct EventRow: Sendable`: rowID/kind/nounType/rowIDStr/estate/ts/dropboxID; :1003 `init?(storageRow:)` same nil-on-mismatch contract + +### Rust port (rust/, crate observer_sink): mirrors above 1:1 except where Rust semantics differ +- rust/src/sink.rs `PersistenceStatsSink::receive`: INLINE synchronous I/O (no Task/thread spawn); Rust StatsSink trait is sync and SqliteStorage is Mutex-backed +- rust/src/store.rs `StatsStore::SCHEMA_VERSION = 3`, `make_schema()`: line-by-line mirror of Swift `StatsStore.schema` +- rust/src/store.rs `epoch_to_iso8601` / `iso8601_to_epoch` (crate-internal): explicit ISO-8601 codec (no DateFormatter equivalent); clamps NaN/±inf/out-of-range to year 0001–9999 window, never panics (unit-tested at bottom of store.rs) +- rust/src/store.rs `write_topology_snapshot_bytes`: Rust-only convenience: lossy UTF-8 conversion of raw bytes; Swift has no equivalent because Foundation's `String(data:encoding:)` guard is a runtime check callers already handle inline + +## INVARIANTS / GOTCHAS + +- Seed-if-absent, NEVER upsert, for control-row defaults in `open()`. Upserting monitoring on every open would reset an operator's "on" back to "off" on every process restart. This exact bug was fixed in Swift commit 852821cc (see rust/src/store.rs comment); do not reintroduce it in either port. +- `isMonitoringEnabled()` fails safe: missing row → `false` (off), never assume on. +- `PersistenceStatsSink` only READS the monitoring flag; only the manager process calls `setMonitoringEnabled`. Do not add a write path to the sink. +- `receive(_:)` MUST stay synchronous and non-blocking (StatsSink protocol contract). All I/O happens off the calling thread/task (Swift: unstructured Task; Rust: inline sync, which is fine because Rust's I/O is not on an async hot path here). +- No in-process buffering at v1.0 by design: one Task per sample in Swift. Do not add batching without also updating OVERVIEW/DETAILS and the design comment in PersistenceStatsSink.swift. +- `event_samples.estate_row_id` is NOT the table's primary key: `row_id` (synthetic UUID) is the PK; `estate_row_id` carries the estate's own UUID for that row. Never conflate the two when reading/writing. +- Retention (`deleteMetricsBefore`/`deleteEventsBefore`) takes `cutoff` AND `now` as caller-supplied parameters. NO `Date()` / `SystemTime::now()` call inside either engine: determinism contract, both ports. +- `recordRetentionCutoff` stores `cutoff` (the boundary applied), not `now` (when the pass ran): intentional, dashboard-facing choice. +- `latestTopologySnapshot`/`latest_topology_snapshot` newest-across-estates comparison MUST tolerate both `.timestamp` and `.text` read-back representations for `generated_at` (SQLite returns text, InMemory returns timestamp). Matching only one representation reintroduces a real, previously-shipped bug where SQLite rows all tie at `.distantPast` / `i64::MIN` and an arbitrary row wins. +- `topology_snapshots` is latest-wins per estate (PRIMARY KEY = estate): no history retained. This is intentional; do not add versioning without a schema bump and a doc update. +- `topology_fingerprint` column is nullable (v3, additive). Absent/null means "no fingerprint computed yet," not an error: callers (autonomic governor) treat null as "recompute once." +- Timestamps are TEXT (ISO-8601 UTC, millisecond precision) on every table. No REAL/numeric timestamp column exists anywhere in this schema. Swift boundary: `Date(timeIntervalSince1970: ts)` at insert, `iso8601Formatter` for retention-cutoff string formatting. Rust boundary: `epoch_to_iso8601`/`iso8601_to_epoch`. +- Rust's `epoch_to_iso8601` clamps to the year 0001–9999 window and never panics on NaN/±infinity/out-of-range f64: required because a malformed 5-digit or negative year would break every lexicographic TEXT comparison retention and topology-snapshot ordering rely on. Six unit tests in rust/src/store.rs lock this behavior; do not remove them when refactoring. +- Tag maps: encoded with `.sortedKeys` (Swift) / serialized from a `BTreeMap` (Rust) for deterministic JSON: needed for reproducible test fixtures, not for correctness of the round trip itself. Decode failure → empty map in both ports (forward-compatible, never throws). +- `queryMetricsByNames` is the required hot-path read API: issues `WHERE name IN (...)`; do not reintroduce `queryMetrics` + Swift-side filtering on a read path that matters for latency. +- No pinned data artifacts in this package (unlike LatticeLib-style libs): every table starts empty; there is nothing to version-check against a build-time asset. +- Both-ports parity is enforced ONLY by the two conformance test suites (Tests/ObserverSinkTests + rust/tests/conformance.rs), not by any shared fixture file. Changing schema, flag semantics, or timestamp encoding in one leg requires updating the other leg AND both suites by hand. diff --git a/packages/libs/ObserverSink/docs/DETAILS.md b/packages/libs/ObserverSink/docs/DETAILS.md new file mode 100644 index 0000000..93d13a9 --- /dev/null +++ b/packages/libs/ObserverSink/docs/DETAILS.md @@ -0,0 +1,353 @@ +--- +doc: DETAILS +package: ObserverSink +repo: moot-system +authored_commit: 909513d0a8ecb1e9e903af9f4d25b3e4f2528242 +authored_date: 2026-07-04 +sources: + - path: Sources/ObserverSink/PersistenceStatsSink.swift + blob: e76c55599795f4bc85b860a4901dc9ab04dd201f + - path: Sources/ObserverSink/StatsStore.swift + blob: 483c736feffd82821a8d77030afde0e182d320fc +--- + +# ObserverSink Details + +This document walks through both source files in the package. Read +`OVERVIEW.md` first for the big picture. `PersistenceStatsSink.swift` +comes first, because it is the entry point every caller uses. +`StatsStore.swift` follows, because it is what the sink calls into. + +## PersistenceStatsSink.swift + +This file provides `PersistenceStatsSink`. It is the concrete +implementation of IntellectusLib's `StatsSink` protocol, and the only +type in the package a typical caller touches directly. + +A `StatsSink` is any type that can receive one `StatSample` at a time. +IntellectusLib defines the protocol and a global installation point. +`PersistenceStatsSink` is the implementation that actually saves +samples, by writing them into a `StatsStore`. A host program constructs +one sink. It installs the sink with `Intellectus.install(sink:)`. It then +turns monitoring on with `Intellectus.setEnabled(true)`. From that point +on, every call to `Intellectus.report(_:)` anywhere in the process +reaches this sink's `receive(_:)` method. + +The struct holds three pieces of state. It holds the `store` it writes +to, a `dropboxID` string that identifies this consumer, and a `Logger` +for diagnostics. All three are `let` constants. The struct conforms to +`Sendable`, a requirement because `StatsSink` values live in a +process-wide global and get called from any thread. No mutable state +lives in the sink itself. Anything that changes over time, such as the +monitoring flag or the stored rows, lives inside `StatsStore`. `StatsStore` +manages its own concurrency. + +`init(store:dropboxID:)` is a plain constructor. It does not open the +store. The caller must call `store.open()` first, because opening +involves async I/O that a synchronous initializer cannot perform. This +split between construction and opening matters when a host wants to +retry a failed open without recreating the sink. + +`receive(_:)` is the method that matters. The `StatsSink` protocol +documents a requirement: whatever calls `Intellectus.report(_:)` must +never stall waiting for a database write. So `receive(_:)` is declared as +a synchronous, non-blocking function. To honor that requirement, +`receive(_:)` captures three values first. It captures the store, the +dropbox ID, and the logger. Each value is `Sendable`. The method then +starts a background task and returns. The caller never waits. + +Inside the task, the real work happens in three steps. First, the code +calls `store.isMonitoringEnabled()`. This is the flag-row check described +in `OVERVIEW.md`: a cheap read of one row in the `control` table. If +monitoring is off, the sample is discarded, and a debug-level log line is +written. The level is debug specifically, because this check runs on +every single sample when monitoring is off. A louder log level would +flood the console. Second, if monitoring is on, the code switches on the +`StatSample` case. A `.metric` sample calls `store.insertMetric(...)`. An +`.event` sample calls `store.insertEvent(...)`. Third, the whole block +sits inside a `do`/`catch`. Any thrown error, whether from the flag read +or from either insert, gets logged at `.error` level and never rethrown. +A telemetry sink that could crash the process it monitors would defeat +its own purpose. So every failure mode here ends in a log line, never a +propagated error. + +The file's header comments record a deliberate scope limit for this +version. There is no in-process buffering. Each sample launches its own +task. SQLite's write-ahead log mode keeps concurrent writes from many +such tasks safe and reasonably efficient. The comments name a possible +future improvement: batching many samples behind a timer. They also +explain why that improvement was not built yet. Stats recording sits off +the hot substrate path. An occasional dropped sample under heavy load is +an acceptable cost there. The simpler design is also easier to reason +about. + +## StatsStore.swift + +This file provides `StatsStore`, the SQLite-backed component that owns +the telemetry schema. It also provides `StatsStoreSchema`, a namespace of +string constants that name the tables and columns. + +### Schema Constants: `StatsStoreSchema` + +Every table name and column name that this package touches gets declared +once, as a `public static let`, inside `StatsStoreSchema`. The file's own +comment explains the reason. Gathering the strings in one place turns a +typo into a compile error, instead of a query that silently matches zero +rows. `monitoringKey` and `retentionCutoffKey` additionally name the two +well-known rows inside the `control` table. That table is a generic +key-value store, rather than one column per concept. + +### Schema Declaration: `StatsStore.schema`. + +`StatsStore.schema` is a `SchemaDeclaration` value. This value comes from +PersistenceKit. It describes tables, columns, indices, and migrations. It +never runs any SQL directly. `StatsStore.schemaVersion` is currently +three. PersistenceKit checks this number against the declaration's +migrations. This check decides whether an existing database needs an +upgrade. + +The schema declares four tables. `metric_samples` and `event_samples` +hold the telemetry rows this library exists to record. Both use a +`.uuid("row_id")` synthetic primary key that the store assigns, so +callers never have to generate or track one. `control` is the key-value +table for the monitoring flag and the retention-cutoff timestamp. Its +primary key is the `key` column itself. That design is what lets +`setMonitoringEnabled` use an upsert, rather than a delete followed by an +insert. `topology_snapshots`, added in schema version two, holds one row +per estate. Its primary key is the `estate` column. Writing a new +snapshot for an estate automatically replaces the old one. History never +accumulates. + +Two indices exist purely to make retention fast: `idx_metric_samples_ts` +and `idx_event_samples_ts`. Retention deletes rows with `ts` below a +cutoff. Without an index on `ts`, each retention pass would scan the +entire table. + +Two migrations are declared, one per schema version bump. The migration +from version one to version two adds the `topology_snapshots` table. The +comment notes that this change is purely additive. No existing row is +touched, and the new table starts empty until the autonomic governor's +next duty cycle fills it. The migration from version two to version three +adds the nullable `topology_fingerprint` column to that same table, again +additively. Any row written before this version simply reads back with +that column set to `nil`. + +### Lifecycle: `init`, `open()`, `close()` + +`init(url:)` builds the underlying `SQLiteStorage` for a database file at +`url`. It generates a fresh random estate identifier for the storage +configuration. It performs no I/O beyond what opening a SQLite connection +requires. + +`open()` does two things. First, it asks PersistenceKit's storage layer +to apply `StatsStore.schema`, creating tables and running any pending +migrations. Second, it seeds two default rows in the `control` table. +`"monitoring"` gets set to `"0"`, and `"retention_cutoff"` gets set to the +epoch-zero ISO-8601 string. Both seeds go through +`seedControlIfAbsent(key:value:)`, a private helper that checks whether a +row with that key already exists before inserting. This seed-if-absent +approach, rather than an upsert, matters for one reason. The monitoring +flag is meant to be a persistent switch. Suppose `open()` blindly +overwrote it every time. An operator's choice to turn monitoring on +would then reset to off silently, on every process restart. Seeding only +when the row is absent avoids this. The first `open()` call installs the +defaults. Every later call on the same database becomes a no-op for those +two rows. + +`close()` closes the underlying storage. It is documented as idempotent, +meaning it is safe to call more than once. That property matters for +callers that close a store in both a normal shutdown path and a `defer` +block. + +### Monitoring Flag: `isMonitoringEnabled()`, `setMonitoringEnabled(_:)` + +`isMonitoringEnabled()` reads the single `control` row whose key is +`"monitoring"`. It returns `true` only if that row's value is the string +`"1"`. If the row is missing entirely, for instance if a caller queries a +store that was never opened, the function returns `false`. This +fail-safe default matters. A missing flag should never read as +"recording is on." + +`setMonitoringEnabled(_:)` writes `"1"` or `"0"` to that same row, using +an upsert keyed on the `key` column. So it works whether or not the row +already exists. The manager process is the only intended caller of this +method. `PersistenceStatsSink` only ever reads the flag; it never sets +it. This asymmetry is what lets one manager control every consumer's +recording behavior, without touching the consumers themselves. + +### Writing Samples: `insertMetric(...)`, `insertEvent(...)` + +`insertMetric(name:value:tags:ts:dropboxID:)` inserts one row into +`metric_samples`. It generates the row's UUID primary key itself. It +encodes the `tags` dictionary to a JSON string with `encodeTagsJSON(_:)`. +It converts the caller-supplied `ts`, epoch seconds as a `Double`, into a +`Date` for storage. The schema stores timestamps as ISO-8601 text, not as +a native numeric column. The conversion from epoch seconds happens here, +at the boundary between the caller's representation and the store's. So +every other part of the codebase can treat `ts` as a plain number. + +`insertEvent(kind:nounType:rowID:estate:ts:dropboxID:)` does the same job +for `event_samples`. Its `rowID` parameter is stored in the +`estate_row_id` column. It is not stored in a column literally named +`row_id`. That name is already taken by the table's own synthetic primary +key. The file's comments call this out explicitly. A future reader +should not confuse the estate's row identifier with the row's own +storage identifier. + +### Topology Snapshots: `writeTopologySnapshot(...)`, `latestTopologySnapshot(estate:)`, `loadTopologyFingerprint(estate:)`. + +`writeTopologySnapshot(estate:generatedAt:payload:fingerprint:)` upserts +one row per estate into `topology_snapshots`, keyed on the `estate` +column. So a new snapshot always replaces the previous one for that +estate, instead of accumulating history. The `payload` parameter is raw +`Data` that must decode as UTF-8 text. The function throws if it does +not, because the autonomic governor that calls this method always +produces valid UTF-8 JSON. A failure here signals a bug worth surfacing, +rather than silently storing garbage. The optional `fingerprint` +parameter is a stable, process-independent hash of the topology inputs. +When supplied, it lets a restarting governor compare against a freshly +computed fingerprint. The governor can then skip re-reading the estate's +full set of drawers, tunnels, and facts when nothing has changed. When +the parameter is omitted, the column is written as `nil`. + +`latestTopologySnapshot(estate:)` reads the payload back. Passing a +specific estate identifier is a primary-key lookup that returns at most +one row. Passing `nil` instead asks for the single newest snapshot across +every estate in the table. This is the behavior the dashboard's "all +estates" view relies on, since that view has no single estate identifier +to filter by. Comparing "newest" requires care. The column is declared as +a timestamp, but PersistenceKit's SQLite backend reads timestamp columns +back as plain text. Its in-memory backend, used in tests, reads them back +as native timestamp values instead. `generatedAtInstant(_:)` is a small +private helper that normalizes either representation to a `Date` before +comparing. An earlier version of this code matched only the timestamp +representation. Under that earlier version, every row tied under SQLite. +An arbitrary row won each time. This was a bug. The in-memory-backed +tests could not catch it, because they never exercised this mismatch. + +`loadTopologyFingerprint(estate:)` is the read side of the fingerprint +parameter described above. It is a primary-key lookup. The lookup returns +`nil` in three cases: no snapshot exists yet, the row predates schema +version three, or a snapshot was written without a fingerprint. + +### Reading Samples: `queryMetrics`, `queryMetricsByNames`, `queryEvents`, `countMetrics`. + +`queryMetrics(dropboxID:)` and `queryEvents(dropboxID:)` return every row +in their respective tables. Each call can optionally filter to one +consumer's dropbox, and results come back ordered by timestamp ascending. +Both decode raw storage rows into the public `MetricRow` and `EventRow` +structs. Both skip any row that fails to decode. Neither throws. This is +a defensive choice. It tolerates a malformed row without losing the rest +of the query's results. + +`queryMetricsByNames(_:dropboxID:)` exists for a specific reason. Reading +every metric row and filtering it on the client side does not scale. It +breaks down once the table holds many distinct metric names. This method +issues a `WHERE name IN (...)` predicate. The database itself then +narrows the result set. The method also short-circuits to an empty +array, with no query at all, when the caller passes an empty name set. +The method's documentation states explicitly that hot read paths should +use this, instead of `queryMetrics` plus a Swift-side filter. + +`countMetrics()` answers "how many metric rows exist" with a `COUNT(*)` +query. This query never decodes a single row. It backs a dashboard +total-count display. It never pays the cost of reading and parsing every +row just to throw away everything except the count. + +### Retention: `deleteMetricsBefore(cutoff:now:)`, `deleteEventsBefore(cutoff:now:)` + +Both methods delete rows whose `ts` sits strictly before a +caller-supplied `cutoff`. Both then update the `retention_cutoff` control +row through the private `recordRetentionCutoff(_:now:)` helper. Neither +method reads the system clock. `cutoff` and `now` both come from the +caller. This determinism rule matters. The engine never calls `Date()` +internally. A test can then build an exact, reproducible retention +scenario. The test never has to race against wall-clock time. The stored +cutoff records the boundary that was applied. It does not record the +moment retention ran. A dashboard can then show "data older than this +point was removed." That message is more useful to an operator than a +timestamp of when the deletion happened to execute. + +### DB-Layer Health: `storageStats(now:)` + +`storageStats(now:)` reports statistics about the stats store's own +SQLite file: page counts, WAL frame counts, file size. These are distinct +from any statistics about an observed estate's own storage. The method +delegates to PersistenceKit's `StorageIntrospection` capability, which +the SQLite backend implements directly. So the call always succeeds in +practice. The optional return type exists only to keep the API honest for +a hypothetical backend that does not support introspection. The `now` +parameter follows the same determinism rule as retention. The caller +supplies the timestamp to stamp on the snapshot. + +### JSON Tag Encoding: `encodeTagsJSON(_:)`, `decodeTagsJSON(_:)` + +Metric tags form a flat `[String: String]` map. The store keeps this map +as a JSON text column, not a separate table. The tag set is small and +simple. A second table would add complexity without adding value. `encodeTagsJSON(_:)` encodes with `.sortedKeys`, so +the same tag map always produces the same JSON string. This helps tests, +and it helps any future deduplication logic that might compare encoded +rows. `decodeTagsJSON(_:)` is forward-compatible by design. A string that +fails to parse decodes to an empty dictionary, rather than throwing. So a +future change to the tag encoding cannot break old rows written before +the change. + +### Row Result Types: `MetricRow`, `EventRow` + +Both are plain `Sendable` structs that mirror a decoded database row. +Their failable initializers, `init?(storageRow:)`, pattern-match every +expected column and its expected `TypedValue` case. If any column is +missing, or has an unexpected type, the initializer returns `nil` rather +than crashing. This is what lets `queryMetrics` and `queryEvents` use +`compactMap` to silently drop malformed rows, instead of failing the +whole query. + +## Rust Port and Conformance + +The `rust/` directory contains a second implementation of this library. +`src/lib.rs` re-exports the public surface. `src/sink.rs` implements +`PersistenceStatsSink`. `src/store.rs` implements `StatsStore` and +`StatsStoreSchema` against the Rust build of PersistenceKit. The two +ports share table names, column names, and the monitoring-flag semantics +exactly. The schema-declaration code in `store.rs`'s `make_schema()` +function is a line-by-line mirror of the Swift `StatsStore.schema` value. + +The two legs differ only where Rust and Swift semantics genuinely +differ, rather than translating one language's idioms into the other +mechanically. `PersistenceStatsSink::receive` in Rust performs its +database write inline and synchronously. The Rust `StatsSink` trait is +synchronous, and Rust's `SqliteStorage` is safely accessible without an +async runtime. The Swift version's task dispatch exists only because +Swift's `Storage` protocol is actor-isolated and asynchronous. The +monitoring-flag check and the discard-on-off behavior stay identical in +both languages. + +The Rust `store.rs` file also owns timestamp conversion explicitly, +through two free functions: `epoch_to_iso8601` and `iso8601_to_epoch`. +Rust has no equivalent to Foundation's `DateFormatter` built in. Some +inputs are unsafe on their own, such as `NaN`, infinities, and +out-of-range years. The valid year window runs from 0001 to 9999. These +functions clamp unsafe values to the nearest valid boundary in that +window. They never panic. They never produce malformed text. A block of +unit tests at the bottom of `store.rs` checks every one of +these edge cases directly. A malformed ISO-8601 string would break every +string comparison that retention and topology-snapshot ordering depend +on. The Rust store also exposes one convenience method the Swift store +does not need: `write_topology_snapshot_bytes`. This method accepts raw +bytes and performs a lossy UTF-8 conversion. It is useful because Rust's +type system otherwise forces a caller to prove UTF-8 validity before +calling the primary method. Swift's runtime guard does not require its +callers to construct that proof in advance. + +Conformance between the two ports is enforced by two independent but +matching test suites: `Tests/ObserverSinkTests/ObserverSinkConformanceTests.swift` +and `rust/tests/conformance.rs`. Both exercise the same scenarios over the +same table and column names. These scenarios include the schema version +and control-row seeding. They include the monitoring flag, plus its +persistence across a close-and-reopen cycle. They include metric and +event round-trips, retention roll-off for both tables, and tag JSON +round-tripping. They include the topology-snapshot read, write, and +latest-wins paths too. They include per-estate isolation and the +DB-layer health check as well. Changing either leg's schema or flag +semantics requires updating both files and both test suites by hand. +Nothing in either build enforces this automatically. diff --git a/packages/libs/ObserverSink/docs/OVERVIEW.md b/packages/libs/ObserverSink/docs/OVERVIEW.md new file mode 100644 index 0000000..c072be7 --- /dev/null +++ b/packages/libs/ObserverSink/docs/OVERVIEW.md @@ -0,0 +1,135 @@ +--- +doc: OVERVIEW +package: ObserverSink +repo: moot-system +authored_commit: 909513d0a8ecb1e9e903af9f4d25b3e4f2528242 +authored_date: 2026-07-04 +sources: + - path: Sources/ObserverSink/PersistenceStatsSink.swift + blob: e76c55599795f4bc85b860a4901dc9ab04dd201f + - path: Sources/ObserverSink/StatsStore.swift + blob: 483c736feffd82821a8d77030afde0e182d320fc +--- + +# ObserverSink Overview + +## What This Library Does + +ObserverSink records telemetry for the MOOTx01 manager pipeline. Telemetry +is data that describes what a running program does. It is separate from +the memories the program stores. A companion library, IntellectusLib, +defines two shapes of telemetry. A metric is a named number, such as a +latency measurement. An event is a record that one memory row was +captured or otherwise acted on. IntellectusLib defines only the shapes +and a plug-in point. It does not know how to save anything to disk. + +ObserverSink is the plug-in that saves them. It ships two pieces. The +first piece is `PersistenceStatsSink`. IntellectusLib calls it each time +new telemetry arrives. The second piece is `StatsStore`. It is a small +SQLite database, plus the code that reads and writes it. Together the two +pieces turn a stream of in-memory telemetry values into durable rows. A +separate manager process can later query those rows. + +## The Problem It Solves + +A running MOOTx01 process is a black box unless something records what +it does. An operator may want to know how fast captures run. An operator +may want to know how many think-cycles fired. An operator may want to +know whether a given estate behaves normally. An estate is one user's +complete memory store in MOOTx01. This kind of information needs a +durable home, because the process may restart or get inspected only +after the fact. + +Writing telemetry to disk on every observation is expensive if done +carelessly. It is also dangerous if it cannot be turned off. ObserverSink +solves both problems with one mechanism: a flag row in the database +itself. Before writing anything, the sink checks whether monitoring is +switched on. The manager owns that switch. The process being observed +does not. This lets the manager enable or disable recording for every +consumer at once, with no restart required. The source comments describe +this flag-row signal as Bob's confirmed choice for Manager 1.0. + +The library also has to work without slowing down the code it observes. +`receive(_:)` is the method IntellectusLib calls. It must return +immediately, because it may run on time-sensitive code paths. ObserverSink +meets this requirement in a simple way. It never performs file I/O on the +calling thread. The actual database write always happens elsewhere. + +## How It Works + +Each telemetry value follows the same path. IntellectusLib calls +`PersistenceStatsSink.receive(_:)` with one `StatSample`. That sample is +either a `.metric` case or an `.event` case. The sink captures what it +needs and starts a background task. It then returns right away. The +calling code never waits for the database write to finish. + +Inside that background task, the sink first asks the store whether +monitoring is enabled. This is a database read, but a cheap one. The +`control` table holds only a handful of rows. If the flag is off, the +sample is dropped and nothing else happens. If the flag is on, the sink +asks the store to save the sample in the correct table. A `.metric` +sample goes into `metric_samples`. A `.event` sample goes into +`event_samples`. If the write fails for any reason, the sink logs the +error and swallows it. A telemetry failure must never crash the program +it describes. + +`StatsStore` also owns two other kinds of data, beyond metrics and +events. A `control` table holds the monitoring flag itself. It also holds +a timestamp that records when data was last rolled off through +retention. A `topology_snapshots` table holds one row per estate: the +latest picture of that estate's structure. A separate background process, +the autonomic governor, writes this picture. Dashboards ask for it and +get it back exactly as written. This table is not telemetry in the +metric or event sense. Even so, it shares the same store and the same +open-and-migrate machinery. It follows the same guiding principle too. +One writer produces the data. Many readers consume it. No history +persists beyond the latest value. + +Retention keeps the two sample tables from growing without bound. The +manager periodically calls `deleteMetricsBefore(cutoff:now:)` and +`deleteEventsBefore(cutoff:now:)`. It supplies the cutoff timestamp +itself; the store never reads the system clock on its own. This keeps +the store's behavior fully determined by its caller's inputs. That +property matters for testing, and for reasoning about what a given +retention pass will do. + +## How the Pieces Fit + +Figure 1 shows the library's topology: its major parts and how data moves +between them. + +![Figure 1. Topology of ObserverSink](topology.svg) + +*Figure 1. Topology of ObserverSink. A telemetry sample flows from the +caller through the sink's background task, past the monitoring-flag +gate, into the matching SQLite table. A separate flow lets the autonomic +governor publish topology snapshots that a dashboard reads back later.* + +`Intellectus.report(_:)`, part of IntellectusLib, is the only caller of +`PersistenceStatsSink.receive(_:)`. The sink holds a reference to a +`StatsStore` and a `dropboxID` string. That string tags every row the +sink writes with the identity of the process that produced it. This lets +one shared store serve many concurrent consumers, for example several +`aria-mcp` process instances, without mixing up their rows. SQLite's +write-ahead log mode makes concurrent writers from separate consumers +safe, with no extra locking needed in this library. + +`StatsStore` wraps a `SQLiteStorage` value supplied by PersistenceKit. That +kit provides the schema-declaration and row-storage machinery this +library builds on. `StatsStore` never talks to SQLite directly. It +describes its four tables as a `SchemaDeclaration` and lets PersistenceKit +create, migrate, and query them. + +## What Ships in the Package + +The package ships two Swift source files, `PersistenceStatsSink.swift` +and `StatsStore.swift`, plus a Rust port in `rust/`. The Rust port +reimplements the same schema and the same sink logic for consumers that +are not written in Swift. Parallel conformance test suites exercise both +legs: `Tests/ObserverSinkTests` for Swift, `rust/tests/conformance.rs` for +Rust. Both suites check the same sixteen scenarios. These cover the +schema version, control-row seeding, the monitoring flag, and metric and +event round-trips. They also cover retention roll-off, tag encoding, and +topology-snapshot reads and writes. This package pins no data artifacts. +Every table starts empty and fills only as the processes that use it +run. diff --git a/packages/libs/ObserverSink/docs/topology.svg b/packages/libs/ObserverSink/docs/topology.svg new file mode 100644 index 0000000..975a8e8 --- /dev/null +++ b/packages/libs/ObserverSink/docs/topology.svg @@ -0,0 +1,104 @@ + + + + + + + + + + + + + ObserverSink: telemetry sample in, durable row out + + + + Intellectus.report + caller (IntellectusLib) + + + PersistenceStatsSink + receive() → Task, flag gate + + + StatsStore + insertMetric / insertEvent + + + + + + + + + SQLite stats store (StatsStore, schema v3) + + + control + monitoring flag + + + metric_samples + name, value, tags, ts + + + event_samples + kind, noun_type, estate + + + topology_snapshots + 1 row/estate, latest-wins + + + + + + + + + Autonomic Governor + writeTopologySnapshot + + + + + + Manager / moot-mgr + setMonitoringEnabled + deleteMetricsBefore / deleteEventsBefore + + + + + + + + Dashboard reads + queryMetrics(ByNames) · queryEvents · + countMetrics · latestTopologySnapshot + + + + +