diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 79184d4d..0c279808 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -192,6 +192,40 @@ jobs: workspaces: | . -> target + - name: Build genuine final-v8 CLI for the v8↔v9 format fence + if: needs.classify_changes.outputs.run_full_ci == 'true' + env: + # Last merged schema-v8 commit before RFC-026 Phase B2 activates v9. + # Keep this immutable: HEAD^ stops being v8 after the first v9 merge. + FINAL_INTERNAL_V8_COMMIT: 725793af83394235bf4b848b6c2c4454ac1f95e1 + run: | + set -euo pipefail + v8_source="$RUNNER_TEMP/omnigraph-final-v8" + v8_bin="$RUNNER_TEMP/omnigraph-final-v8-bin" + git fetch --no-tags --depth=1 origin "$FINAL_INTERNAL_V8_COMMIT" + git worktree add --detach "$v8_source" "$FINAL_INTERNAL_V8_COMMIT" + cargo build --locked \ + --manifest-path "$v8_source/Cargo.toml" \ + --package omnigraph-cli \ + --bin omnigraph \ + --target-dir "$GITHUB_WORKSPACE/target" + cp "$GITHUB_WORKSPACE/target/debug/omnigraph" "$v8_bin" + test -x "$v8_bin" + # The old and current workspaces intentionally share the expensive + # third-party dependency cache. Their path packages have the same + # names/versions, so remove only those local artifacts before Cargo + # builds the current workspace; otherwise it can reuse a v8 engine + # object behind the v9 test binary. + cargo clean --target-dir "$GITHUB_WORKSPACE/target" \ + --package omnigraph-compiler \ + --package omnigraph-policy \ + --package omnigraph-engine \ + --package omnigraph-api-types \ + --package omnigraph-cluster \ + --package omnigraph-server \ + --package omnigraph-cli + echo "OMNIGRAPH_V8_BIN=$v8_bin" >> "$GITHUB_ENV" + - name: Run workspace tests if: needs.classify_changes.outputs.run_full_ci == 'true' # On same-repo PRs, regenerate openapi.json as part of the drift test diff --git a/AGENTS.md b/AGENTS.md index 7c35472d..a3c14c41 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -28,12 +28,13 @@ Tools that support `@`-imports (Claude Code) auto-include all three files via th OmniGraph is a typed property-graph engine built as a coordination layer over many Lance datasets. Highlights: -- **Storage**: per node/edge table lifetime a separate Lance dataset; multi-dataset commits coordinated atomically through internal `__manifest` schema v8. The v5 stable-identity contract remains intact: manifest ownership, OCC, recovery, and identity-derived paths under `nodes/` and `edges/` use the immutable table pair, while `table_key` is only the current public alias. V6 creates every graph table with exact non-null physical `id` as Lance's unenforced PK and fences every production insert/upsert on that field; v7 added identity-keyed MemWAL enrollment/lifecycle authority; v8 activates stream-config v2 and recovery-v11 for the private one-generation B1 row/fold core. +- **Storage**: per node/edge table lifetime a separate Lance dataset; multi-dataset commits coordinated atomically through internal `__manifest` schema v9. The v5 stable-identity contract remains intact: manifest ownership, OCC, recovery, and identity-derived paths under `nodes/` and `edges/` use the immutable table pair, while `table_key` is only the current public alias. V6 creates every graph table with exact non-null physical `id` as Lance's unenforced PK and fences every production insert/upsert on that field; v7 added identity-keyed MemWAL enrollment/lifecycle authority; v8 added the private one-generation B1 row/fold core; v9 activates stream-config v3, lifecycle state-v2, manifest-selected graph-global `_stream_tokens.lance` authority, and recovery-v12 for the private B2 compare-and-chain/fold core. - **Languages**: a `.pg` schema language and a `.gq` query language, both Pest-based, with typed IR. Persisted accepted SchemaIR v2 owns one graph identity domain, one monotonic no-reuse allocator, rename-stable type/property IDs, and node/edge table-incarnation IDs. - **Multi-modal querying**: vector ANN (`nearest`), full-text (`search`/`fuzzy`/`match_text`/`bm25`), Reciprocal Rank Fusion (`rrf`), and graph traversal (`Expand`, anti-join `not { … }`) in one runtime. - **Branches and commits across the whole graph**: Git-style — every successful publish appends to a commit DAG; merges are three-way at the row level. - **Atomic per-query writes**: `mutate_as` and `load` accumulate insert/update batches into an in-memory `MutationStaging.pending` per touched table. Strict insert and upsert both route through the sealed exact-`id`, filter-bearing adapter; bare Lance Append is test-only. Their RFC-022 adapter resolves or rejects relevant recovery intents before base capture, captures `(native branch id, exact graph_head, schema identity)`, then rechecks recovery and authority under schema → branch → table gates. Mutation/Load keeps one keyed transaction per table and rejects more than 8,192 rows or 32 MiB before recovery arm; mutation update scans stream into the remaining table budget after pending-key shadowing, with blob sizes checked before payload reads. It then arms an identity-bearing recovery-v9 sidecar containing exact Lance transaction identities + pre-minted lineage, commits each table with zero transparent conflict retries, confirms the achieved effects, and publishes under the same token. Unrelated retryable pre-effect authority movement may fully reprepare without changing logical mode. A proven effect-free strict conflict returns `KeyConflict` only after a fresh exact-ID probe; no exact match triggers bounded full strict-mode reprepare, while an effect-free upsert conflict also fully reprepares. Any earlier effect or ambiguity returns `RecoveryRequired` with the sidecar retained. Strict read-set conflicts return `ReadSetChanged`. Deletes stage through the same path. D₂ at parse time remains the constructive (insert/update) XOR destructive (delete) boundary. -- **RFC-026 private streaming core**: Phase A's main-only adapter enrolls one exact-`id` table into one empty unsharded Lance MemWAL under recovery-v10, then publishes its physical binding and `OPEN` lifecycle. Phase B1 adds a root-singleflight worker for one no-roll generation (8,192 rows / 32 MiB), watcher success plus the same writer's post-durability `check_fenced()` before clean acknowledgement, conservative replay/fold-only recovery, and one recovery-v11 strict exact-`id` fold whose sole `__manifest` CAS advances the table pointer, lifecycle witness/epoch floor, and graph lineage together. Cheap raw bounds and exact post-tombstone validation run before recovery; put ownership follows charge → shared admission → same-key queue → worker mode; cold replay installs exact fold-only accounting; exclusive admission spans seal, drain proof, table effect, and publication. Fold now charges the scanner's logical slices against the same 32-MiB generation cap and copies each scanner emission into dense owned arrays before retaining it, so the 8,192-row high-entropy near-cap closure cell folds and publishes successfully; its isolated fold RSS delta was 284,934,144 bytes (about 272 MiB), below the 384-MiB remeasurement tripwire. Post-invocation ambiguity is `AckUnknown`; unresolved recovery blocks progress. The topology remains main-only, unsharded, one resident writer and one live writer process, with no fresh reads or generation GC. The private **B2a unbounded retain-all gate is implemented**: OmniGraph imposes no retained-byte, object-count, file-count, or history quota and never deletes a canonical durable `_mem_wal` object. Lance may clean only its losing `.binpb.tmp.` atomic-CAS staging. Complete and partial unreferenced generation residue stays non-authoritative and is never descended into, read, mutated, adopted, or deleted through recovery/reopen; local and configured-RustFS provider-failure cells prove loud typed outcomes. The 1/8/32/128 history instrument keeps warm-ack operation count flat while exposing growing serialized authority and combined fold/history cost; its LIST totals, wall time, and RSS are advisory, not quotas or SLOs. Provider exhaustion may halt admission, fold, or recovery progress; it never permits silent loss or publication of partial state. Row count, Arrow memory, deadlines, retries, and ambiguous outcomes remain bounded. RC.1's missing cross-open materialization-attempt receipt and complete physical-output envelope are therefore not blockers for this profile; they remain relevant to the future B2b managed-reclamation design. Compare-and-chain tokens, trusted attribution, lifecycle receipts, correction, authorization, and product parity remain required and inactive. The core remains crate-private behind one feature-gated, doc-hidden test seam; no schema-v9, SDK, CLI, HTTP, Cedar, or OpenAPI streaming contract ships from this result. +- **RFC-026 private streaming core**: Phase A's main-only adapter enrolls one exact-`id` table into one empty unsharded Lance MemWAL under recovery-v10, then publishes its physical binding and `OPEN` lifecycle. Phase B1 adds a root-singleflight worker for one no-roll generation (8,192 rows / 32 MiB of logical dense-slice Arrow data), watcher success plus the same writer's post-durability `check_fenced()` before clean acknowledgement, conservative replay/fold-only recovery, and the bounded fold mechanics. Cheap raw bounds and exact post-tombstone validation run before recovery; put ownership follows charge → shared admission → same-key queue → worker mode; cold replay installs exact fold-only accounting; exclusive admission spans seal, drain proof, both table effects, and publication. Admission, replay, and fold all charge `ArrayData::get_slice_memory_size`; fold densifies retained arrays. Backing-buffer capacity and physical allocation are not the limit, while isolated fold RSS remains a 384-MiB remeasurement tripwire. The 8,192-row high-entropy near-cap closure cell remains green. Internal schema v9 adds the private B2 compare-and-chain core: canonical payload and token digests, grammar-impossible trusted hidden row metadata (`__omnigraph_stream_v1$`), same-generation token overlays, and a manifest-selected graph-global `_stream_tokens.lance` authority. Exact base/token lookups and recovery validation materialize at most the requested rows plus one and cap both Arrow batch bytes and cumulative retained bytes. Admission recaptures lifecycle/binding/HEAD authority only after acquiring shared admission and the same-key queue, so a stale provisional capture cannot authorize a WAL put. Recovery-v12 owns exact pre-minted base and token transactions; only their exact joint outcome may drive the sole `__manifest` CAS that advances both pointers, lifecycle state-v2, graph lineage, and a durable fold-attribution summary. Recovery-v11 is historical v8 state and is refused under v9 because it lacks the token participant and complete state-v2 authority. Post-invocation ambiguity is `AckUnknown`; unresolved recovery blocks progress. The topology remains main-only, unsharded, one resident writer and one live writer process, with no fresh reads or generation GC. The private **B2a unbounded retain-all profile is active**: OmniGraph imposes no retained-byte, object-count, file-count, or history quota and never deletes a canonical durable `_mem_wal` object. Lance may clean only its losing `.binpb.tmp.` atomic-CAS staging. Complete and partial unreferenced generation residue stays non-authoritative and untouched through recovery/reopen; provider exhaustion is loud. The genuine v8↔v9 refusal/rebuild gate preserves logical rows—including an ordinary v8 `__omnigraph_stream_v1` user property—vectors, and PK metadata while omitting hidden trusted stream metadata from export. The core remains crate-private behind one feature-gated, doc-hidden test seam. Explicit production enrollment, lifecycle management/correction/status, authorization, and SDK/CLI/HTTP/OpenAPI surfaces remain inactive. +- **RFC-026 B2 preprocessing bound**: before blob materialization or canonical encoding, the private adapter reserves a 128-MiB worst-case envelope (original 32-MiB Arrow row + possible 32-MiB replacement + 64-MiB canonical payload) and an inflight slot. The private profile admits exactly two envelopes (256 MiB root-wide), preserving the two-caller stale-authority race without unbounded preprocessing. The slot transfers into queued/worker ownership; scratch releases after digest derivation. Pressure fails effect-free as typed `stream_b2_preprocessing_bytes`; this is process-memory admission, not retained-storage GC or quota. - **HTTP server**: Axum + utoipa OpenAPI, bearer auth (SHA-256 hashed, optional AWS Secrets Manager). Cedar policy enforcement is engine-wide — every `_as` writer calls `Omnigraph::enforce(action, scope, actor)`, so HTTP, CLI, and embedded SDK consumers all hit the same gate. **Cluster-only boot** (RFC-011): the server always boots from a cluster directory (`--cluster `, RFC-005) and serves N graphs (N ≥ 1) under multi-graph routes (`/graphs/{graph_id}/...` + read-only `GET /graphs` enumeration); there are no single-graph flat routes and no positional-URI boot. Per-graph + server-level Cedar policies. Runtime add/remove (`POST /graphs`, `DELETE /graphs/{id}`) is not exposed — operators run `cluster apply` and restart. - **CLI** with two-surface config (RFC-007/008): the team-owned cluster directory (`cluster.yaml`) plus the per-operator `~/.omnigraph/config.yaml` (servers, clusters, credentials, actor, profiles, aliases, defaults). Graphs are addressed via `--store`/`--server`/`--cluster`/`--profile`/operator defaults (RFC-011). Multi-format output (json/jsonl/csv/kv/table). @@ -272,15 +273,15 @@ omnigraph policy explain --cluster ./company-brain --graph knowledge --actor act |---|---|---| | Columnar storage on object store | ✅ Arrow/Lance | URI normalization, S3 env-var plumbing | | Per-dataset versioning + time travel | ✅ | `snapshot_at_version`, `entity_at`, snapshot-pinned reads across many tables | -| Stable schema + table identity | — | Persisted accepted SchemaIR v2 owns one graph identity domain and a shared monotonic no-reuse allocator for nonzero type, property, and table-incarnation IDs. Internal manifest schema v5 introduced identity-keyed table registration/version/tombstone rows, OCC, recovery ownership, and `nodes/{stable_table_id:016x}-{table_incarnation_id:016x}` / `edges/{stable_table_id:016x}-{table_incarnation_id:016x}` paths; v6 preserves that contract and adds keyed fencing; v7 adds identity-keyed stream lifecycle authority; v8 retains it while activating the private data-bearing stream-config-v2 capability. `table_key` is a mutable alias. A pure type rename changes only that alias and keeps the same dataset/path/version/history; drop/re-add mints a new type/property identity, incarnation, path, and independent Lance version sequence. This binary serves v8 only; v7 and older graphs use export/import rebuild. | +| Stable schema + table identity | — | Persisted accepted SchemaIR v2 owns one graph identity domain and a shared monotonic no-reuse allocator for nonzero type, property, and table-incarnation IDs. Internal manifest schema v5 introduced identity-keyed table registration/version/tombstone rows, OCC, recovery ownership, and `nodes/{stable_table_id:016x}-{table_incarnation_id:016x}` / `edges/{stable_table_id:016x}-{table_incarnation_id:016x}` paths; v6 preserves that contract and adds keyed fencing; v7 adds identity-keyed stream lifecycle authority; v8 adds the private B1 data path; v9 activates config-v3/state-v2 and graph-global token authority. `table_key` is a mutable alias. A pure type rename changes only that alias and keeps the same dataset/path/version/history; drop/re-add mints a new type/property identity, incarnation, path, and independent Lance version sequence. This binary serves v9 only; v8 and older graphs use export/import rebuild. | | Per-dataset branches | ✅ | **Graph-level** refs are logically atomic through authoritative `__manifest` `BranchContents`; native create/delete crash gaps are classified and reclaimed under a single-writer-process boundary; live names are path-prefix-disjoint; data-table forks are lazy; system branches are filtered | -| Atomic single-dataset commits | ✅ | **Multi-table publish uses three layers**, not one Lance primitive: (1) each table's Lance effect, (2) one identity-aware `__manifest` CAS for graph visibility, and (3) recovery for the gap between them. The five established graph-content/maintenance writers (`MutationStaging::commit_all`, SchemaApply, BranchMerge, EnsureIndices, Optimize) emit identity-bearing recovery-v9 `__recovery/{ulid}.json` intents before their first durable effect; RFC-026's private enrollment emits recovery-v10, and its private exact-`id` stream fold emits recovery-v11 with a pre-minted Update transaction, exact generation marker, fixed lineage, physical cut, and complete pointer/lifecycle outcome. Every pin, effect, registration, rename, tombstone, and output slot carries `(stable_table_id, table_incarnation_id)`; established payload names such as `protocol_v3`, `protocol_v4`, `protocol_v7`, `protocol_v8`, and `protocol_v10` are wire-field names, not a license to infer missing authority. Pre-v9 files are explicitly historical and are never upgraded or assigned identity from aliases, paths, matching versions, or serde defaults. Under final admission → schema → branch → table gates, Mutation/Load, SchemaApply, BranchMerge, EnsureIndices, and StreamFold prove the manifest/HEAD baseline, pre-mint exact Lance transactions + fixed lineage, and durably confirm complete outcomes before publication; `Armed` is rollback-only for the established writers, while StreamFold accepts only exact no-effect or its planned `N + 1` effect and rolls the latter forward under captured authority. Optimize uses the v9 identity envelope with bounded maintenance provenance because Lance does not expose caller-minted maintenance transactions. Read-write open recovers all-or-nothing by identity, publishing one batch forward or restoring owned effects then publishing the restored versions; stream enrollment is roll-forward-or-retire only and never restores/deletes ambiguous MemWAL state, and a confirmed StreamFold publishes its exact pointer/lifecycle/lineage result. Foreign or buried effects fail closed, and completed recovery is audited in `_graph_commit_recoveries.lance`. A pure SchemaApply type rename is metadata-only and keeps identity/path/version/index history; a simultaneous property change rewrites that same dataset, while only AddType is first-touch. Schema staging, the fixed manifest outcome, and catalog publication remain ordered; read capture binds one refreshed snapshot to one accepted SchemaIR-derived catalog under the schema gate, and read-only open refuses an incoherent fixed outcome. Enrolled write entry points and `refresh` also perform roll-forward-only in-process healing. These gates remain process-local; destructive recovery against a live writer in another process still requires a distributed fence. Engine writes stay behind sealed `TableStorage` `stage_*` + `commit_staged` APIs; deletes, StreamFold, and the pinned Lance one-segment full-table vector build use those staged adapters, while Lance issue [#6666](https://github.com/lance-format/lance/issues/6666) remains relevant only to generic multi-segment exact vector publication. | +| Atomic single-dataset commits | ✅ | **Multi-table publish uses three layers**, not one Lance primitive: (1) each Lance participant's effect, (2) one identity-aware `__manifest` CAS for graph visibility, and (3) recovery for the gap between them. The five established graph-content/maintenance writers (`MutationStaging::commit_all`, SchemaApply, BranchMerge, EnsureIndices, Optimize) emit identity-bearing recovery-v9 `__recovery/{ulid}.json` intents before their first durable effect; RFC-026's private enrollment emits recovery-v10, while the current private stream fold emits recovery-v12 with exact pre-minted base-table and `_stream_tokens.lance` transactions, an exact generation marker, fixed lineage, physical cut, planned token winners, complete lifecycle state-v2, and fold attribution. Recovery-v11 is retained only as historical v8 syntax and cannot recover under schema v9. Every pin, effect, registration, rename, tombstone, and output slot carries `(stable_table_id, table_incarnation_id)`; established payload names such as `protocol_v3`, `protocol_v4`, `protocol_v7`, `protocol_v8`, and `protocol_v10` are wire-field names, not a license to infer missing authority. Pre-v9 files are explicitly historical and are never upgraded or assigned identity from aliases, paths, matching versions, or serde defaults. Under final admission → schema → branch → stream-token → table gates where applicable, writers prove the manifest/HEAD baseline, pre-mint exact Lance transactions + fixed lineage, and durably confirm complete outcomes before publication. `Armed` is rollback-only for established writers; StreamFold retires only when both participants are proven effect-free, completes an exact missing token effect from the durable plan when the exact base effect landed, and publishes only the exact joint outcome. Optimize uses the v9 identity envelope with bounded maintenance provenance because Lance does not expose caller-minted maintenance transactions. Read-write open recovers all-or-nothing by identity; stream enrollment is roll-forward-or-retire only and never restores/deletes ambiguous MemWAL state. Foreign or buried effects fail closed, and completed recovery is audited in `_graph_commit_recoveries.lance`. A pure SchemaApply type rename is metadata-only and keeps identity/path/version/index history; a simultaneous property change rewrites that same dataset, while only AddType is first-touch. Schema staging, the fixed manifest outcome, and catalog publication remain ordered; read capture binds one refreshed snapshot to one accepted SchemaIR-derived catalog under the schema gate, and read-only open refuses an incoherent fixed outcome. Enrolled write entry points and `refresh` also perform roll-forward-only in-process healing. These gates remain process-local; destructive recovery against a live writer in another process still requires a distributed fence. Engine writes stay behind sealed `TableStorage` `stage_*` + `commit_staged` APIs; deletes, StreamFold, and the pinned Lance one-segment full-table vector build use those staged adapters, while Lance issue [#6666](https://github.com/lance-format/lance/issues/6666) remains relevant only to generic multi-segment exact vector publication. | | Compaction (`compact_files`) + reindex (`optimize_indices`) | ✅ | `omnigraph optimize` orchestrates over all node/edge tables with bounded physical concurrency and one graph visibility envelope. Under schema → main → every accepted-catalog table gate it loads one identity-bound catalog/snapshot, skips uncovered HEAD drift, and plans only productive tables. One identity-bearing recovery-v9 `SidecarKind::Optimize` envelope pins the complete set before any table HEAD movement; each productive task runs `compact_files`, Lance `optimize_indices` (incremental coverage merge, not retrain), and declared-missing index materialization, but no task publishes independently. After all tasks settle, one maintenance-class monotonic manifest CAS publishes every still-needed identity pointer plus one graph lineage commit; a pointer already at or beyond the achieved version is converged and omitted. Thus two changed tables become visible together, a no-work run creates no sidecar/lineage, and any post-arm error returns `RecoveryRequired`. Full v9 recovery rolls the complete set forward in one batch or compensates a partial set before visibility. Main remains held through final physical-only `__manifest` compaction. Optimize's bounded maintenance classifier carries stable table identity but has no exact caller-minted transaction/authority/fixed-lineage proof, so destructive recovery retains the documented single-writer-process boundary until Lance exposes a stable maintenance transaction API and OmniGraph has distributed fencing. It **commits even with no compaction work if index coverage is stale**; reports untrainable vector-only work as pending without pinning it; **refuses on an unrecovered graph**; **skips uncovered HEAD > manifest drift** with `DriftNeedsRepair`; and **compacts blob-bearing tables**. | | Repair uncovered drift | — | `omnigraph repair` explicitly classifies uncovered table `HEAD > manifest` drift: verified maintenance drift (`ReserveFragments`/`Rewrite`) can be published with `--confirm`; suspicious or unverifiable drift requires `--force --confirm`. Sidecar-covered crash residuals still recover automatically on open. | | Cleanup (`cleanup_old_versions`) | ✅ | `omnigraph cleanup` derives requested `--keep` / `--older-than` cutoffs from each table's available versions; Lance refs plus OmniGraph's live-lazy-branch and recovery floors may retain additional versions. It fails closed on unopenable pins, recovery intent, or uncovered main-table HEAD drift | | BTREE / inverted (FTS) / vector indexes | ✅ | `@index`/`@key` declares intent; the physical index is derived state that never fails a logical op. Built per column through one chokepoint (`build_indices_on_dataset_for_catalog`, type-dispatched by `node_prop_index_kind`: enum + orderable scalar → BTREE, free-text String → FTS, Vector → vector); idempotent; lazy across branches. **Schema apply and mutation/load build no indexes inline**: they publish only logical data/schema effects, leaving physical intent pending. `ensure_indices` first runs the roll-forward-only recovery barrier before base capture or planning, then materializes every declared-but-missing artifact for one table through one staged mixed CreateIndex transaction under its identity-bearing recovery-v9 authority/lineage/delta envelope; it continues to report untrainable Vector columns as pending. `Armed` is rollback-only, `EffectsConfirmed` rolls forward only while captured authority holds, and first-touch refs carry exact identity. `optimize` separately restores fragment coverage through its bounded identity-bearing v9 maintenance envelope. | -| Strict insert / upsert ingestion | ✅ transaction conflict filters + uncommitted fragment staging | Internal schema v6 introduced the explicit logical mode and v8 preserves it. General strict insert and upsert use the sealed exact-`id`, forced-v2 MergeInsert adapter; strict insert exact-probes its pinned parent before minting `omnigraph.insert_absence=v1`, while an all-new upsert may mint the same optional certificate only from its completed effect statistics. Mutation/Load remains one transaction per table, capped before arm at 8,192 rows / 32 MiB. BranchMerge's proven all-new route accepts only a complete certificate chain plus final source/target native-incarnation checks; it stages bounded fragments with `InsertBuilder`, commits them as exact-`id` filtered `Update` transactions, and performs zero target preflights, target merge joins, or committed Appends. Missing or malformed proof falls back to the general ordered diff. Raw Lance writers are outside the supported graph-writer topology, and the certificate is an internal, non-cryptographic capability. The final five-pair production gate passed at 10K (3.875× median; 24,297,472-byte max paired RSS overhead) and 100K (~3.886×; 32,604,160 bytes). | -| MemWAL streaming foundation | ✅ MemWAL system index, durable WAL generations, epoch-fenced shard writer | Internal schema v8 carries exact identity-keyed physical binding, current-HEAD witness, `OPEN`/`DRAINING`/`SEALED` state, per-shard epoch floor, and stream-config v2. Phase A's recovery-v10 adapter enrolls one empty unsharded shard. Private B1 adds one root-singleflight, no-roll generation capped at 8,192 rows / 32 MiB; watcher success plus same-writer post-durability epoch check; detached ownership; fold-only replay; seal/drain proof; and recovery-v11 exact-transaction fold followed by one graph-visibility CAS. The fold charges logical slices and densifies each scanner emission, so the deterministic 8,192-row/near-32-MiB high-entropy generation now closes; the measured isolated fold RSS delta is 284,934,144 bytes (about 272 MiB), below the 384-MiB remeasurement tripwire. It performs no generation GC or fresh reads. The private B2a gate implements unbounded retain-all: no OmniGraph byte/object/file/history quota; no canonical durable `_mem_wal` deletion; typed provider failure; and complete/partial orphan residue retained, non-authoritative, and untouched below its root through retry/reopen. Lance-owned losing manifest temp-file cleanup is allowed. The 1/8/32/128 local/RustFS instrument is advisory and exposes growing authority/history cost without creating a threshold. Row/logical-Arrow admission, deadline, retry, and ambiguity bounds remain. B2b retains the future Lance-owned reclamation design. Common schema-v9/config-v3/state-v2/recovery-v12 enrollment/token/attribution/lifecycle/correction contracts remain inactive. There is no public stream schema, enrollment, put/fold/status API, Cedar action, CLI, HTTP route, or OpenAPI contract. | +| Strict insert / upsert ingestion | ✅ transaction conflict filters + uncommitted fragment staging | Internal schema v6 introduced the explicit logical mode and v9 preserves it. General strict insert and upsert use the sealed exact-`id`, forced-v2 MergeInsert adapter; strict insert exact-probes its pinned parent before minting `omnigraph.insert_absence=v1`, while an all-new upsert may mint the same optional certificate only from its completed effect statistics. Mutation/Load remains one transaction per table, capped before arm at 8,192 rows / 32 MiB. BranchMerge's proven all-new route accepts only a complete certificate chain plus final source/target native-incarnation checks; it stages bounded fragments with `InsertBuilder`, commits them as exact-`id` filtered `Update` transactions, and performs zero target preflights, target merge joins, or committed Appends. Missing or malformed proof falls back to the general ordered diff. Raw Lance writers are outside the supported graph-writer topology, and the certificate is an internal, non-cryptographic capability. The final five-pair production gate passed at 10K (3.875× median; 24,297,472-byte max paired RSS overhead) and 100K (~3.886×; 32,604,160 bytes). | +| MemWAL streaming foundation | ✅ MemWAL system index, durable WAL generations, epoch-fenced shard writer | Internal schema v9 carries exact identity-keyed physical binding, current-HEAD witness, lifecycle state-v2, per-shard epoch floor, stream-config v3, and the manifest-selected `_stream_tokens.lance` pointer. Phase A's recovery-v10 adapter enrolls one empty unsharded shard. The private B1 worker remains one root-singleflight, no-roll generation capped at 8,192 rows / 32 MiB of logical dense-slice Arrow data with watcher-plus-post-fence acknowledgement, fold-only replay, and seal/drain proof; backing-buffer capacity and physical RSS are not admission authority. Private B2 admission adds canonical payload/token digests, trusted hidden metadata, same-key compare-and-chain/idempotency classification, same-generation overlays, and authority recapture after shared admission. Recovery-v12 folds one generation through exact base and token transactions; its sole manifest CAS advances both pointers, lifecycle, lineage, and durable attribution together. The private B2a profile remains unbounded retain-all: no OmniGraph byte/object/file/history quota, no canonical durable `_mem_wal` deletion, typed provider failure, and inert retained orphan residue. The 1/8/32/128 local/RustFS instrument remains advisory. It performs no generation GC or fresh reads. B2b retains the future Lance-owned reclamation design. The genuine v8↔v9 refusal/rebuild gate is active. Explicit production enrollment, revisioned lifecycle management, correction/status, Cedar, SDK, CLI, HTTP, and OpenAPI contracts remain inactive. | | Vector search | ✅ | `nearest()` query op; embedding pipeline (Gemini / OpenAI clients); `@embed` in schema | | Full-text search | ✅ | `search/fuzzy/match_text/bm25` query ops | | Hybrid ranking | — | `rrf(...)` Reciprocal Rank Fusion in one runtime | diff --git a/crates/omnigraph-cli/tests/crossversion_upgrade.rs b/crates/omnigraph-cli/tests/crossversion_upgrade.rs index ea0227dc..4edc95bb 100644 --- a/crates/omnigraph-cli/tests/crossversion_upgrade.rs +++ b/crates/omnigraph-cli/tests/crossversion_upgrade.rs @@ -14,11 +14,15 @@ //! `OMNIGRAPH_V5_BIN` (built from the final internal-v5 commit) and proves both //! directions of the v5/current format fence. Each case skips only when its variable //! is unset; a set but invalid path fails loudly. The v6 case uses -//! `OMNIGRAPH_V6_BIN` and proves the v6/v8 fence across the v7 foundation. -//! The immediate-predecessor v7 case uses `OMNIGRAPH_V7_BIN` and proves the -//! genuine v7 ↔ v8 format fence plus strict export/init/load rebuild. The v7 +//! `OMNIGRAPH_V6_BIN` and proves the v6/current fence across the v7 foundation. +//! The v7 case uses `OMNIGRAPH_V7_BIN` and proves the genuine v7 ↔ current +//! format fence plus strict export/init/load rebuild. The v7 //! image is intentionally unenrolled because that binary exposes no production -//! enrollment route; this does not claim retained physical config-v1 state. +//! enrollment route; this does not claim retained physical config-v1 state. The +//! immediate-predecessor v8 case uses `OMNIGRAPH_V8_BIN` and proves the genuine +//! v8 ↔ v9 fence, strict rebuild, non-exposure of v9's trusted physical stream +//! metadata, and preservation of a genuine v8 user property whose old +//! grammar-valid name motivated v9's grammar-impossible physical field. mod support; @@ -96,7 +100,20 @@ fn v7_bin() -> Option { Some(path) } -/// Run the OLD (0.7.2) binary hermetically (no developer `~/.omnigraph`). +/// Resolve the final internal-v8 binary (the last merged v8 implementation +/// before RFC-026 Phase-B2 format activation). +fn v8_bin() -> Option { + let path = PathBuf::from(std::env::var_os("OMNIGRAPH_V8_BIN")?); + assert!( + path.exists() && path.is_file(), + "OMNIGRAPH_V8_BIN is set but is not a binary file: {} \ + (unset it to skip, or point it at the omnigraph binary built from the final internal-v8 commit)", + path.display(), + ); + Some(path) +} + +/// Run any predecessor binary hermetically (no developer `~/.omnigraph`). fn run_old(bin: &Path, args: &[&str]) -> std::process::Output { Command::new(bin) .env("OMNIGRAPH_HOME", HERMETIC_OPERATOR_HOME) @@ -163,6 +180,22 @@ fn assert_export_fidelity(label: &str, original: &[u8], rebuilt: &[u8]) { ); } +fn assert_export_omits_trusted_stream_metadata(label: &str, export: &[u8]) { + for line in String::from_utf8_lossy(export) + .lines() + .filter(|line| !line.trim().is_empty()) + { + let row = serde_json::from_str::(line).expect("valid export JSONL"); + let data = row["data"] + .as_object() + .expect("export row data must be an object"); + assert!( + !data.contains_key("__omnigraph_stream_v1$"), + "{label} export must not expose the trusted physical stream metadata column; row={row}", + ); + } +} + fn assert_exported_blob_fidelity(label: &str, original: &[u8], rebuilt: &[u8]) { let original_blob = exported_row_with_data_value(original, "name", "blob-sentinel"); let rebuilt_blob = exported_row_with_data_value(rebuilt, "name", "blob-sentinel"); @@ -343,12 +376,12 @@ fn current_binary_refuses_and_rebuilds_a_genuine_v3_graph() { // 5. Round-trip fidelity: re-export with the current binary and compare. let reexport = output_success(cli().arg("export").arg(&new_graph)); - assert_export_fidelity("v3 → v8", &export.stdout, &reexport.stdout); + assert_export_fidelity("v3 → v9", &export.stdout, &reexport.stdout); assert_current_graph_tables_use_exact_id_pk(&new_graph); } #[test] -fn current_v8_refuses_and_rebuilds_genuine_v4_and_v4_refuses_v8() { +fn current_v9_refuses_and_rebuilds_genuine_v4_and_v4_refuses_v9() { let Some(previous) = previous_bin() else { eprintln!( "skipping immediate-predecessor upgrade test: OMNIGRAPH_PREVIOUS_BIN is not set to a 0.8.1 binary" @@ -398,7 +431,7 @@ fn current_v8_refuses_and_rebuilds_genuine_v4_and_v4_refuses_v8() { assert!(stderr.contains("0.8.x"), "got: {stderr}"); assert!(stderr.contains("export"), "got: {stderr}"); - let new_graph = temp.path().join("new-v8-from-v4.omni"); + let new_graph = temp.path().join("new-v9-from-v4.omni"); output_success( cli() .arg("init") @@ -416,13 +449,13 @@ fn current_v8_refuses_and_rebuilds_genuine_v4_and_v4_refuses_v8() { .arg(&new_graph), ); let reexport = output_success(cli().arg("export").arg(&new_graph)); - assert_export_fidelity("v4 → v8", &export.stdout, &reexport.stdout); + assert_export_fidelity("v4 → v9", &export.stdout, &reexport.stdout); assert_current_graph_tables_use_exact_id_pk(&new_graph); let reverse = run_old(&previous, &["snapshot", new_graph.to_str().unwrap()]); assert!( !reverse.status.success(), - "a v4 binary must refuse a genuine v8 graph" + "a v4 binary must refuse a genuine v9 graph" ); let reverse_stderr = String::from_utf8_lossy(&reverse.stderr); assert!( @@ -434,7 +467,7 @@ fn current_v8_refuses_and_rebuilds_genuine_v4_and_v4_refuses_v8() { } #[test] -fn current_v8_refuses_and_rebuilds_genuine_v5_and_v5_refuses_v8() { +fn current_v9_refuses_and_rebuilds_genuine_v5_and_v5_refuses_v9() { let Some(v5) = v5_bin() else { eprintln!( "skipping immediate-predecessor v5 upgrade test: OMNIGRAPH_V5_BIN is not set to a final internal-v5 binary" @@ -500,7 +533,7 @@ fn current_v8_refuses_and_rebuilds_genuine_v5_and_v5_refuses_v8() { let jsonl = temp.path().join("v5.jsonl"); std::fs::write(&jsonl, &export.stdout).unwrap(); - // The current v8 binary refuses before reading the predecessor image as if + // The current v9 binary refuses before reading the predecessor image as if // it already had RFC-023's physical PK contract. let refusal = output_failure(cli().arg("snapshot").arg(&v5_graph)); let stderr = String::from_utf8_lossy(&refusal.stderr); @@ -533,7 +566,7 @@ fn current_v8_refuses_and_rebuilds_genuine_v5_and_v5_refuses_v8() { let duplicate_jsonl = temp.path().join("v5-duplicate-id.jsonl"); std::fs::write(&duplicate_jsonl, duplicate_export).unwrap(); - let rejected_graph = temp.path().join("rejected-v8-from-v5.omni"); + let rejected_graph = temp.path().join("rejected-v9-from-v5.omni"); output_success( cli() .arg("init") @@ -567,7 +600,7 @@ fn current_v8_refuses_and_rebuilds_genuine_v5_and_v5_refuses_v8() { "a rejected target import must leave the old source root untouched", ); - let current_graph = temp.path().join("new-v8-from-v5.omni"); + let current_graph = temp.path().join("new-v9-from-v5.omni"); output_success( cli() .arg("init") @@ -585,8 +618,8 @@ fn current_v8_refuses_and_rebuilds_genuine_v5_and_v5_refuses_v8() { .arg(¤t_graph), ); let reexport = output_success(cli().arg("export").arg(¤t_graph)); - assert_export_fidelity("v5 → v8", &export.stdout, &reexport.stdout); - assert_exported_blob_fidelity("v5 → v8", &export.stdout, &reexport.stdout); + assert_export_fidelity("v5 → v9", &export.stdout, &reexport.stdout); + assert_exported_blob_fidelity("v5 → v9", &export.stdout, &reexport.stdout); assert_current_graph_tables_use_exact_id_pk(¤t_graph); assert_current_blob_bytes(¤t_graph, &[0, 1, 2, 3, 255]); @@ -595,19 +628,19 @@ fn current_v8_refuses_and_rebuilds_genuine_v5_and_v5_refuses_v8() { let reverse = run_old(&v5, &["snapshot", current_graph.to_str().unwrap()]); assert!( !reverse.status.success(), - "a v5 binary must refuse a genuine v8 graph", + "a v5 binary must refuse a genuine v9 graph", ); let reverse_stderr = String::from_utf8_lossy(&reverse.stderr); assert!( reverse_stderr.contains("upgrade omnigraph") || reverse_stderr.contains("newer") || reverse_stderr.contains("expects v5"), - "unexpected v5→v8 reverse-refusal message: {reverse_stderr}", + "unexpected v5→v9 reverse-refusal message: {reverse_stderr}", ); } #[test] -fn current_v8_refuses_and_rebuilds_genuine_v6_and_v6_refuses_v8() { +fn current_v9_refuses_and_rebuilds_genuine_v6_and_v6_refuses_v9() { let Some(v6) = v6_bin() else { eprintln!( "skipping immediate-predecessor v6 upgrade test: OMNIGRAPH_V6_BIN is not set to a final internal-v6 binary" @@ -658,13 +691,13 @@ fn current_v8_refuses_and_rebuilds_genuine_v6_and_v6_refuses_v8() { let jsonl = temp.path().join("v6.jsonl"); std::fs::write(&jsonl, &export.stdout).unwrap(); - let v8_graph = temp.path().join("new-v8-from-v6.omni"); + let v9_graph = temp.path().join("new-v9-from-v6.omni"); output_success( cli() .arg("init") .arg("--schema") .arg(&schema) - .arg(&v8_graph), + .arg(&v9_graph), ); output_success( cli() @@ -673,28 +706,28 @@ fn current_v8_refuses_and_rebuilds_genuine_v6_and_v6_refuses_v8() { .arg("overwrite") .arg("--data") .arg(&jsonl) - .arg(&v8_graph), + .arg(&v9_graph), ); - let reexport = output_success(cli().arg("export").arg(&v8_graph)); - assert_export_fidelity("v6 → v8", &export.stdout, &reexport.stdout); - assert_current_graph_tables_use_exact_id_pk(&v8_graph); + let reexport = output_success(cli().arg("export").arg(&v9_graph)); + assert_export_fidelity("v6 → v9", &export.stdout, &reexport.stdout); + assert_current_graph_tables_use_exact_id_pk(&v9_graph); - let reverse = run_old(&v6, &["snapshot", v8_graph.to_str().unwrap()]); + let reverse = run_old(&v6, &["snapshot", v9_graph.to_str().unwrap()]); assert!( !reverse.status.success(), - "a v6 binary must refuse a genuine v8 graph", + "a v6 binary must refuse a genuine v9 graph", ); let reverse_stderr = String::from_utf8_lossy(&reverse.stderr); assert!( reverse_stderr.contains("upgrade omnigraph") || reverse_stderr.contains("newer") || reverse_stderr.contains("expects v6"), - "unexpected v6→v8 reverse-refusal message: {reverse_stderr}", + "unexpected v6→v9 reverse-refusal message: {reverse_stderr}", ); } #[test] -fn current_v8_refuses_and_rebuilds_genuine_v7_and_v7_refuses_v8() { +fn current_v9_refuses_and_rebuilds_genuine_v7_and_v7_refuses_v9() { let Some(v7) = v7_bin() else { eprintln!( "skipping immediate-predecessor v7 upgrade test: OMNIGRAPH_V7_BIN is not set to a final internal-v7 binary" @@ -748,13 +781,13 @@ fn current_v8_refuses_and_rebuilds_genuine_v7_and_v7_refuses_v8() { let jsonl = temp.path().join("v7.jsonl"); std::fs::write(&jsonl, &export.stdout).unwrap(); - let v8_graph = temp.path().join("new-v8-config-v2-from-v7.omni"); + let v9_graph = temp.path().join("new-v9-config-v3-from-v7.omni"); output_success( cli() .arg("init") .arg("--schema") .arg(&schema) - .arg(&v8_graph), + .arg(&v9_graph), ); output_success( cli() @@ -763,26 +796,154 @@ fn current_v8_refuses_and_rebuilds_genuine_v7_and_v7_refuses_v8() { .arg("overwrite") .arg("--data") .arg(&jsonl) - .arg(&v8_graph), + .arg(&v9_graph), ); - let reexport = output_success(cli().arg("export").arg(&v8_graph)); + let reexport = output_success(cli().arg("export").arg(&v9_graph)); assert_export_fidelity( - "unenrolled v7 format → v8/config-v2", + "unenrolled v7 format → v9/config-v3", &export.stdout, &reexport.stdout, ); - assert_current_graph_tables_use_exact_id_pk(&v8_graph); + assert_current_graph_tables_use_exact_id_pk(&v9_graph); - let reverse = run_old(&v7, &["snapshot", v8_graph.to_str().unwrap()]); + let reverse = run_old(&v7, &["snapshot", v9_graph.to_str().unwrap()]); assert!( !reverse.status.success(), - "a v7 binary must refuse a genuine v8 graph", + "a v7 binary must refuse a genuine v9 graph", ); let reverse_stderr = String::from_utf8_lossy(&reverse.stderr); assert!( reverse_stderr.contains("upgrade omnigraph") || reverse_stderr.contains("newer") || reverse_stderr.contains("expects v7"), - "unexpected v7→v8 reverse-refusal message: {reverse_stderr}", + "unexpected v7→v9 reverse-refusal message: {reverse_stderr}", + ); +} + +#[test] +fn current_v9_refuses_and_rebuilds_genuine_v8_and_v8_refuses_v9() { + let Some(v8) = v8_bin() else { + eprintln!( + "skipping immediate-predecessor v8 upgrade test: OMNIGRAPH_V8_BIN is not set to a final internal-v8 binary" + ); + return; + }; + + let temp = tempdir().unwrap(); + let schema = temp.path().join("v8-stream-name-collision.pg"); + let data = temp.path().join("v8-stream-name-collision.jsonl"); + let search_schema = std::fs::read_to_string(fixture("search.pg")).unwrap(); + std::fs::write( + &schema, + format!( + "{search_schema}\nnode LegacyCollision {{\n slug: String @key\n __omnigraph_stream_v1: String\n}}\n" + ), + ) + .unwrap(); + let mut search_data = std::fs::read_to_string(fixture("search.jsonl")).unwrap(); + if !search_data.ends_with('\n') { + search_data.push('\n'); + } + search_data.push_str( + r#"{"type":"LegacyCollision","data":{"slug":"legacy-name","__omnigraph_stream_v1":"user-owned-v8-value"}} +"#, + ); + std::fs::write(&data, search_data).unwrap(); + let v8_graph = temp.path().join("old-v8-config-v2.omni"); + let v8_uri = v8_graph.to_str().unwrap(); + + // Mint the genuine immediate-predecessor image with the final v8 binary. + // This is the real config-v2/schema-v8 layout, not a v9 manifest whose + // internal-schema stamp was edited after creation. + assert_ok( + "v8 init", + &run_old(&v8, &["init", "--schema", schema.to_str().unwrap(), v8_uri]), + ); + assert_ok( + "v8 load", + &run_old( + &v8, + &[ + "load", + "--mode", + "overwrite", + "--data", + data.to_str().unwrap(), + v8_uri, + ], + ), + ); + let export = run_old(&v8, &["export", v8_uri]); + assert_ok("v8 export", &export); + assert!(!export.stdout.is_empty(), "v8 export produced no rows"); + let legacy = exported_row_with_data_value( + &export.stdout, + "__omnigraph_stream_v1", + "user-owned-v8-value", + ); + assert_eq!(legacy["type"], "LegacyCollision"); + + // The current binary must fail before interpreting a v8 graph as if it had + // v9's trusted physical metadata and manifest-selected token authority. + let refusal = output_failure(cli().arg("snapshot").arg(&v8_graph)); + let stderr = String::from_utf8_lossy(&refusal.stderr); + assert!( + stderr.contains("0.12.x"), + "v8 refusal must name the release line that wrote internal schema v8, got: {stderr}", + ); + assert!( + stderr.contains("export"), + "v8 refusal must direct the operator to export/import rebuild, got: {stderr}", + ); + + let jsonl = temp.path().join("v8.jsonl"); + std::fs::write(&jsonl, &export.stdout).unwrap(); + let v9_graph = temp.path().join("new-v9-config-v3-from-v8.omni"); + output_success( + cli() + .arg("init") + .arg("--schema") + .arg(&schema) + .arg(&v9_graph), + ); + output_success( + cli() + .arg("load") + .arg("--mode") + .arg("overwrite") + .arg("--data") + .arg(&jsonl) + .arg(&v9_graph), + ); + let reexport = output_success(cli().arg("export").arg(&v9_graph)); + assert_export_fidelity( + "v8/config-v2 → v9/config-v3", + &export.stdout, + &reexport.stdout, + ); + assert_export_omits_trusted_stream_metadata("rebuilt v9", &reexport.stdout); + let rebuilt_legacy = exported_row_with_data_value( + &reexport.stdout, + "__omnigraph_stream_v1", + "user-owned-v8-value", + ); + assert_eq!( + rebuilt_legacy["data"]["__omnigraph_stream_v1"], + legacy["data"]["__omnigraph_stream_v1"], + "v8's grammar-valid user property must not be mistaken for v9 protocol metadata", + ); + assert_current_graph_tables_use_exact_id_pk(&v9_graph); + + let reverse = run_old(&v8, &["snapshot", v9_graph.to_str().unwrap()]); + assert!( + !reverse.status.success(), + "a v8 binary must refuse a genuine v9 graph", + ); + let reverse_stderr = String::from_utf8_lossy(&reverse.stderr); + assert!( + reverse_stderr.contains("upgrade omnigraph") + || reverse_stderr.contains("newer") + || reverse_stderr.contains("expects v8"), + "unexpected v8→v9 reverse-refusal message: {reverse_stderr}", ); } diff --git a/crates/omnigraph-compiler/src/catalog/schema_ir.rs b/crates/omnigraph-compiler/src/catalog/schema_ir.rs index e6aa266c..7799c429 100644 --- a/crates/omnigraph-compiler/src/catalog/schema_ir.rs +++ b/crates/omnigraph-compiler/src/catalog/schema_ir.rs @@ -1748,12 +1748,13 @@ edge Relates: Human -> Human @rename_from("Knows") { @unique(src, dst) } #[test] fn validation_rejects_reserved_storage_system_column_names() { - const RESERVED: [&str; 5] = [ + const RESERVED: [&str; 6] = [ "_rowid", "_rowaddr", "_rowoffset", "_row_created_at_version", "_row_last_updated_at_version", + "__omnigraph_stream_v1$", ]; let accepted = initialize( "interface I { ip: String } node N { np: String } edge E: N -> N { ep: String }", @@ -1774,6 +1775,11 @@ edge Relates: Human -> Human @rename_from("Knows") { @unique(src, dst) } assert!(error.contains("exported"), "unexpected error: {error}"); } } + + let mut malformed = accepted; + malformed.nodes[0].properties[0].name = "__OMNIGRAPH_STREAM_V1$".to_string(); + let error = validate_schema_ir(&malformed).unwrap_err().to_string(); + assert!(error.contains("reserved"), "unexpected error: {error}"); } #[test] diff --git a/crates/omnigraph-compiler/src/schema/mod.rs b/crates/omnigraph-compiler/src/schema/mod.rs index 134bc2de..ea765971 100644 --- a/crates/omnigraph-compiler/src/schema/mod.rs +++ b/crates/omnigraph-compiler/src/schema/mod.rs @@ -1,19 +1,23 @@ pub mod ast; pub mod parser; -/// Names owned by Lance's virtual row-address and row-version columns. +/// Names owned by Lance's virtual columns or by OmniGraph's physical protocol. /// /// Keep this compiler-owned list exact rather than depending on Lance here: /// `omnigraph-compiler` deliberately has no storage-substrate dependency. The -/// engine's Lance surface guards pin these five surveyed upstream constants; -/// every Lance bump still requires a source audit for newly added names. +/// The engine's Lance surface guards pin the five surveyed upstream constants; +/// every Lance bump still requires a source audit for newly added names. The +/// final name is OmniGraph's nullable RFC-026 trusted-attribution envelope. Its +/// trailing `$` is deliberately outside the `.pg` identifier grammar; this +/// check still protects hand-authored accepted SchemaIR. pub(crate) fn is_reserved_storage_system_column(name: &str) -> bool { - matches!( - name, - "_rowid" + name.eq_ignore_ascii_case("__omnigraph_stream_v1$") + || matches!( + name, + "_rowid" | "_rowaddr" | "_rowoffset" | "_row_created_at_version" | "_row_last_updated_at_version" - ) + ) } diff --git a/crates/omnigraph-compiler/src/schema/parser_tests.rs b/crates/omnigraph-compiler/src/schema/parser_tests.rs index 971626c2..e585f93e 100644 --- a/crates/omnigraph-compiler/src/schema/parser_tests.rs +++ b/crates/omnigraph-compiler/src/schema/parser_tests.rs @@ -1040,7 +1040,7 @@ fn test_parse_error_diagnostic_has_span() { } #[test] -fn test_reject_lance_virtual_system_column_property_names() { +fn test_reject_storage_system_column_property_names() { const RESERVED: [&str; 5] = [ "_rowid", "_rowaddr", @@ -1063,6 +1063,13 @@ fn test_reject_lance_virtual_system_column_property_names() { } } - parse_schema("node N { _row_id: String _rowid2: String _ROWID: String }") - .expect("only the five exact, case-sensitive Lance names are reserved"); + parse_schema( + "node N { _row_id: String _rowid2: String _ROWID: String __omnigraph_stream_v1: String __omnigraph_stream_v2: String }", + ) + .expect("grammar-valid historical names remain ordinary user properties"); + + assert!( + parse_schema("node N { __omnigraph_stream_v1$: String }").is_err(), + "the protocol-private v9 field must remain outside the .pg identifier grammar" + ); } diff --git a/crates/omnigraph-server/src/lib.rs b/crates/omnigraph-server/src/lib.rs index f4541a80..e533182d 100644 --- a/crates/omnigraph-server/src/lib.rs +++ b/crates/omnigraph-server/src/lib.rs @@ -901,6 +901,9 @@ impl ApiError { // fields: a fold request is a retryable logical conflict, while an // invoked-but-unconfirmed append is unavailable/ambiguous. err @ OmniError::FoldRequired { .. } => Self::conflict(err.to_string()), + err @ (OmniError::StreamBindingChanged { .. } + | OmniError::StreamSequenceConflict { .. } + | OmniError::StreamIdempotencyConflict { .. }) => Self::conflict(err.to_string()), err @ OmniError::AckUnknown { .. } => Self::internal(err.to_string()), OmniError::RecoveryRequired { operation_id, diff --git a/crates/omnigraph/src/db/graph_coordinator.rs b/crates/omnigraph/src/db/graph_coordinator.rs index c0b9ec81..79e26043 100644 --- a/crates/omnigraph/src/db/graph_coordinator.rs +++ b/crates/omnigraph/src/db/graph_coordinator.rs @@ -531,6 +531,7 @@ impl GraphCoordinator { actor_id: actor_id.map(str::to_string), merged_parent_commit_id, created_at: crate::db::now_micros()?, + stream_fold_attribution: None, }) } diff --git a/crates/omnigraph/src/db/manifest.rs b/crates/omnigraph/src/db/manifest.rs index 82a48dbd..d71805da 100644 --- a/crates/omnigraph/src/db/manifest.rs +++ b/crates/omnigraph/src/db/manifest.rs @@ -33,6 +33,10 @@ mod recovery; mod state; #[path = "manifest/stream.rs"] pub(crate) mod stream; +#[path = "manifest/stream_token.rs"] +pub(crate) mod stream_token; +#[path = "manifest/token_store.rs"] +pub(crate) mod token_store; use graph::{ init_manifest_graph, open_manifest_graph, open_manifest_graph_with_lineage, snapshot_state_at, @@ -54,17 +58,19 @@ pub(crate) use recovery::{ HealPendingOutcome, MAX_BRANCH_MERGE_DATA_TRANSACTIONS, RecoveryAuthorityToken, RecoveryBranchMergeEffect, RecoveryBranchMergeEffectKind, RecoveryLineageIntent, RecoveryManifestDelta, RecoveryMode, RecoverySchemaApplyEffect, RecoverySchemaApplyEffectKind, - RecoverySidecar, RecoverySidecarHandle, RecoveryStreamFoldCut, RecoveryTableUpdateSlot, - SidecarKind, SidecarTablePin, SidecarTableRegistration, - SidecarTableRename, SidecarTombstone, complete_stream_enrollment_sidecar_v10, - complete_stream_fold_sidecar_v11, confirm_branch_merge_sidecar_v9, + RecoverySidecar, RecoverySidecarHandle, + RecoveryStreamFoldCut, RecoveryTableUpdateSlot, SidecarKind, SidecarTablePin, + SidecarTableRegistration, SidecarTableRename, SidecarTombstone, + complete_stream_enrollment_sidecar_v10, complete_stream_fold_sidecar_v12, + confirm_branch_merge_sidecar_v9, confirm_ensure_indices_sidecar_v9, confirm_occ_sidecar_v9, confirm_schema_apply_sidecar_v9, - delete_sidecar, ensure_read_only_schema_coherent, finalize_effect_free_occ_sidecar, - finalize_effect_free_stream_fold_sidecar_v11, - heal_pending_sidecars_roll_forward, list_sidecars, new_branch_merge_sidecar_v9, - new_ensure_indices_sidecar_v9, new_occ_sidecar_v9, new_optimize_sidecar_v9, - new_schema_apply_sidecar_v9, new_stream_enrollment_sidecar_v10, new_stream_fold_sidecar_v11, - recover_manifest_drift, schema_apply_serial_queue_key, write_sidecar, + confirm_stream_fold_sidecar_v12, delete_sidecar, ensure_read_only_schema_coherent, + finalize_effect_free_occ_sidecar, finalize_effect_free_stream_fold_sidecar_v12, + heal_pending_sidecars_roll_forward, + list_sidecars, new_branch_merge_sidecar_v9, new_ensure_indices_sidecar_v9, new_occ_sidecar_v9, + new_optimize_sidecar_v9, new_schema_apply_sidecar_v9, new_stream_enrollment_sidecar_v10, + new_stream_fold_sidecar_v12, recover_manifest_drift, + schema_apply_serial_queue_key, write_sidecar, }; pub use state::SubTableEntry; #[cfg(test)] @@ -72,9 +78,10 @@ use state::string_column; pub(crate) use state::{GraphLineageRow, read_graph_lineage}; use state::{ManifestState, read_manifest_state}; pub(crate) use stream::{ - CurrentHeadWitness, STREAM_CONFIG_VERSION, StreamLifecycle, StreamLifecycleEntry, - StreamPhysicalBinding, + CurrentHeadWitness, EnrollmentReceipt, STREAM_CONFIG_VERSION, StreamLifecycle, + StreamLifecycleEntry, StreamPhysicalBinding, stream_enrollment_intent_digest_v1, }; +pub(crate) use token_store::{StreamTokenAuthorityEntry, open_stream_token_authority_at}; /// The internal-schema (storage-format) version this binary writes and reads. /// A graph's on-disk per-branch stamp is read via [`internal_schema_stamp_at`]; @@ -94,6 +101,7 @@ const OBJECT_TYPE_GRAPH_COMMIT: &str = "graph_commit"; /// `object_id` is `graph_head:` (`graph_head:main` for the main branch). const OBJECT_TYPE_GRAPH_HEAD: &str = "graph_head"; const OBJECT_TYPE_STREAM_STATE: &str = "stream_state"; +const OBJECT_TYPE_STREAM_TOKEN_AUTHORITY: &str = "stream_token_authority"; const TABLE_VERSION_MANAGEMENT_KEY: &str = "table_version_management"; /// Stable head-key segment for the main branch in `graph_head:` rows. @@ -156,6 +164,9 @@ pub struct Snapshot { /// table identity. `table_key` inside each value is diagnostic and is never /// used to select the authority row. stream_lifecycles: HashMap, + /// Exact graph-global sequencing participant selected by this manifest + /// snapshot. Its raw Lance HEAD is never authority. + stream_token_authority: StreamTokenAuthorityEntry, /// Per-graph read caches (shared `Session` + held-handle cache), injected by /// `Omnigraph::resolved_target` for live Branch reads so table opens reuse /// handles (0 IO on a warm repeat) and one `Session`. `None` for write-prelude @@ -173,6 +184,11 @@ pub struct Snapshot { #[derive(Debug, Clone)] pub struct SnapshotTable { dataset: Dataset, + /// Public reflection view. The physical dataset additionally carries the + /// reserved RFC-026 attribution struct; keeping a separately owned schema + /// prevents a caller from recovering that protocol column through + /// `SnapshotTable::schema`. + public_schema: LanceSchema, } /// Read-only scan builder for a [`SnapshotTable`]. @@ -183,19 +199,116 @@ pub struct SnapshotTable { /// writable handle and bypass graph publication. pub struct SnapshotScanner { scanner: Scanner, + physical_schema: arrow_schema::SchemaRef, + public_projection: Vec, + forbidden_protocol_column: Option, +} + +fn is_protocol_column_path(reference: &str) -> bool { + reference.split('.').any(|segment| { + segment + .trim_matches(|character| matches!(character, '`' | '"' | '[' | ']')) + .eq_ignore_ascii_case(crate::db::STREAM_METADATA_COLUMN) + }) +} + +fn ensure_expr_omits_protocol_column(filter: &Expr) -> Result<()> { + if filter.column_refs().iter().any(|column| { + column + .name + .eq_ignore_ascii_case(crate::db::STREAM_METADATA_COLUMN) + }) { + return Err(OmniError::manifest(format!( + "column '{}' is reserved for OmniGraph storage protocol metadata", + crate::db::STREAM_METADATA_COLUMN + ))); + } + Ok(()) +} + +fn filter_text_references_protocol_column(filter: &str) -> bool { + let bytes = filter.as_bytes(); + let needle = crate::db::STREAM_METADATA_COLUMN.as_bytes(); + let is_identifier_byte = |byte: u8| byte.is_ascii_alphanumeric() || matches!(byte, b'_' | b'$'); + let mut index = 0; + let mut in_string_literal = false; + while index < bytes.len() { + if bytes[index] == b'\'' { + if in_string_literal && bytes.get(index + 1) == Some(&b'\'') { + index += 2; + continue; + } + in_string_literal = !in_string_literal; + index += 1; + continue; + } + if !in_string_literal && index + needle.len() <= bytes.len() { + let candidate = &bytes[index..index + needle.len()]; + let left_boundary = index == 0 || !is_identifier_byte(bytes[index - 1]); + let right_boundary = index + needle.len() == bytes.len() + || !is_identifier_byte(bytes[index + needle.len()]); + if left_boundary + && right_boundary + && candidate.eq_ignore_ascii_case(needle) + { + return true; + } + } + index += 1; + } + false +} + +fn parse_public_filter(schema: arrow_schema::SchemaRef, filter: &str) -> Result { + // The protocol field is deliberately outside the public schema grammar. + // Reject its literal spelling before SQL parsing as well: some parser + // paths treat `$` as placeholder syntax instead of a column reference. + if filter_text_references_protocol_column(filter) { + return Err(OmniError::manifest(format!( + "column '{}' is reserved for OmniGraph storage protocol metadata", + crate::db::STREAM_METADATA_COLUMN + ))); + } + let expression = lance_datafusion::planner::Planner::new(schema) + .parse_filter(filter) + .map_err(|error| OmniError::Lance(error.to_string()))?; + ensure_expr_omits_protocol_column(&expression)?; + Ok(expression) } impl SnapshotScanner { /// Select the output columns. pub fn project>(&mut self, columns: &[T]) -> Result<&mut Self> { + if columns + .iter() + .any(|column| is_protocol_column_path(column.as_ref())) + { + return Err(OmniError::manifest(format!( + "column '{}' is reserved for OmniGraph storage protocol metadata", + crate::db::STREAM_METADATA_COLUMN + ))); + } + // Lance expands `*` against the scanner's physical schema. Expand it + // here against the already-sealed public schema so callers cannot + // recover protocol metadata through wildcard projection. + let mut public_columns = Vec::with_capacity(columns.len()); + for column in columns { + let column = column.as_ref(); + if column.trim() == "*" { + public_columns.extend(self.public_projection.iter().cloned()); + } else { + public_columns.push(column.to_string()); + } + } self.scanner - .project(columns) + .project(&public_columns) .map_err(|error| OmniError::Lance(error.to_string()))?; Ok(self) } /// Apply a SQL filter expression. pub fn filter(&mut self, filter: &str) -> Result<&mut Self> { + parse_public_filter(Arc::clone(&self.physical_schema), filter)?; self.scanner .filter(filter) .map_err(|error| OmniError::Lance(error.to_string()))?; @@ -204,6 +317,10 @@ impl SnapshotScanner { /// Apply a structured DataFusion filter expression. pub fn filter_expr(&mut self, filter: Expr) -> &mut Self { + if ensure_expr_omits_protocol_column(&filter).is_err() { + self.forbidden_protocol_column = Some(crate::db::STREAM_METADATA_COLUMN.to_string()); + return self; + } self.scanner.filter_expr(filter); self } @@ -254,6 +371,11 @@ impl SnapshotScanner { /// Execute the configured read without exposing its physical plan. pub async fn try_into_stream(&self) -> Result { + if let Some(column) = &self.forbidden_protocol_column { + return Err(OmniError::manifest(format!( + "column '{column}' is reserved for OmniGraph storage protocol metadata" + ))); + } self.scanner .try_into_stream() .await @@ -263,18 +385,44 @@ impl SnapshotScanner { impl SnapshotTable { fn new(dataset: Dataset) -> Self { - Self { dataset } + let mut public_schema = dataset.schema().clone(); + public_schema + .fields + .retain(|field| field.name != crate::db::STREAM_METADATA_COLUMN); + Self { + dataset, + public_schema, + } } /// Build a read-only scanner over this pinned table version. pub fn scan(&self) -> SnapshotScanner { + let mut scanner = self.dataset.scan(); + let public_projection = self + .public_schema + .fields + .iter() + .map(|field| field.name.clone()) + .collect::>(); + scanner + .project(&public_projection) + .expect("manifest-validated public fields must project from their physical dataset"); SnapshotScanner { - scanner: self.dataset.scan(), + scanner, + physical_schema: Arc::new(arrow_schema::Schema::from(self.dataset.schema())), + public_projection, + forbidden_protocol_column: None, } } /// Count rows in this pinned table version, optionally with a filter. pub async fn count_rows(&self, filter: Option) -> Result { + if let Some(filter) = filter.as_deref() { + parse_public_filter( + Arc::new(arrow_schema::Schema::from(self.dataset.schema())), + filter, + )?; + } self.dataset .count_rows(filter) .await @@ -283,7 +431,7 @@ impl SnapshotTable { /// Lance schema of this pinned table version. pub fn schema(&self) -> &LanceSchema { - self.dataset.schema() + &self.public_schema } /// Lance manifest version of this pinned table. @@ -291,16 +439,27 @@ impl SnapshotTable { self.dataset.version().version } - /// Read-only physical index metadata for this pinned table version. + /// Read-only user-index metadata for this pinned table version. Lance's + /// MemWAL system index is part of OmniGraph's private write protocol and + /// is deliberately absent from SDK reflection. pub async fn load_indices(&self) -> Result>> { - self.dataset + let indices = self + .dataset .load_indices() .await - .map_err(|error| OmniError::Lance(error.to_string())) + .map_err(|error| OmniError::Lance(error.to_string()))?; + Ok(Arc::new( + indices + .iter() + .filter(|index| index.name != lance_index::mem_wal::MEM_WAL_INDEX_NAME) + .cloned() + .collect(), + )) } /// Whether `column` has complete usable BTREE coverage. pub async fn index_coverage(&self, column: &str) -> Result { + self.ensure_public_column(column)?; crate::table_store::TableStore::key_column_index_coverage(&self.dataset, column).await } @@ -311,18 +470,31 @@ impl SnapshotTable { /// Whether this table has a user BTREE index on `column`. pub async fn has_btree_index(&self, column: &str) -> Result { + self.ensure_public_column(column)?; crate::table_store::TableStore::has_btree_index_on(&self.dataset, column).await } /// Whether this table has a user full-text index on `column`. pub async fn has_fts_index(&self, column: &str) -> Result { + self.ensure_public_column(column)?; crate::table_store::TableStore::has_fts_index_on(&self.dataset, column).await } /// Whether this table has a user vector index on `column`. pub async fn has_vector_index(&self, column: &str) -> Result { + self.ensure_public_column(column)?; crate::table_store::TableStore::has_vector_index_on(&self.dataset, column).await } + + fn ensure_public_column(&self, column: &str) -> Result<()> { + if is_protocol_column_path(column) { + return Err(OmniError::manifest(format!( + "column '{}' is reserved for OmniGraph storage protocol metadata", + crate::db::STREAM_METADATA_COLUMN + ))); + } + Ok(()) + } } impl Snapshot { @@ -460,6 +632,21 @@ impl Snapshot { self.stream_lifecycles.iter() } + /// Exact durable pointer to the graph-global RFC-026 token participant. + pub(crate) fn stream_token_authority(&self) -> &StreamTokenAuthorityEntry { + &self.stream_token_authority + } + + /// Open `_stream_tokens.lance` only at the exact manifest-selected witness. + pub(crate) async fn open_stream_token_authority(&self) -> Result { + let session = self + .read_caches + .as_ref() + .map(|caches| Arc::clone(&caches.session)) + .unwrap_or_else(crate::lance_access::control_session); + open_stream_token_authority_at(&self.root_uri, &self.stream_token_authority, &session).await + } + /// Refuse any physical table/schema effect that would bypass RFC-026 /// lifecycle authority. Phase A has no drain operation that can atomically /// advance `CurrentHeadWitness`, so even SEALED is fenced here. Later @@ -700,6 +887,13 @@ pub(crate) enum ManifestChange { expected: Option, next: StreamLifecycleEntry, }, + /// Advance the one graph-global stream-token participant pointer under an + /// exact manifest-row CAS. Genesis always provisions the row, so there is + /// no absent-pointer/bootstrap publish mode. + SetStreamTokenAuthority { + expected: StreamTokenAuthorityEntry, + next: StreamTokenAuthorityEntry, + }, } /// One table-version authority assertion supplied to a publish attempt. @@ -845,6 +1039,7 @@ impl ManifestCoordinator { .map(|entry| (entry.table_key.clone(), entry)) .collect(), stream_lifecycles: state.stream_lifecycles, + stream_token_authority: state.stream_token_authority, read_caches: None, } } diff --git a/crates/omnigraph/src/db/manifest/graph.rs b/crates/omnigraph/src/db/manifest/graph.rs index 8e05d09d..95410f4e 100644 --- a/crates/omnigraph/src/db/manifest/graph.rs +++ b/crates/omnigraph/src/db/manifest/graph.rs @@ -19,6 +19,7 @@ use super::state::{ GraphLineageRow, ManifestState, SubTableEntry, entries_to_batch, graph_lineage_row_parts, manifest_schema, read_manifest_state, read_manifest_state_and_lineage, }; +use super::token_store::initialize_stream_token_authority; use super::{TableIdentity, table_path_for_identity}; /// The manifest version the init `Dataset::write` produces (Lance datasets start @@ -34,6 +35,7 @@ pub(super) async fn init_manifest_graph( control_session: &Arc, ) -> Result<(Dataset, ManifestState, Vec)> { let root = root_uri.trim_end_matches('/'); + let stream_token_authority = initialize_stream_token_authority(root, control_session).await?; let (entries, version_metadata) = build_initial_entries(root, catalog, control_session).await?; // Genesis graph commit: parentless, actorless, minted once and folded into @@ -47,10 +49,16 @@ pub(super) async fn init_manifest_graph( merged_parent_commit_id: None, actor_id: None, created_at: crate::db::now_micros()?, + stream_fold_attribution: None, }; let genesis_lineage = graph_lineage_row_parts(&genesis, None)?; - let manifest_batch = entries_to_batch(&entries, &version_metadata, &genesis_lineage)?; + let manifest_batch = entries_to_batch( + &entries, + &version_metadata, + &genesis_lineage, + &stream_token_authority, + )?; let schema = manifest_schema(); let reader = RecordBatchIterator::new(vec![Ok(manifest_batch)], schema); let params = WriteParams { @@ -225,9 +233,33 @@ async fn create_empty_dataset( .map_err(|e| OmniError::Lance(e.to_string())) } -fn keyed_graph_table_schema(schema: &SchemaRef) -> Result { +pub(super) fn keyed_graph_table_schema(schema: &SchemaRef) -> Result { + // Engine catalogs have already crossed `fixup_physical_schemas` and carry + // the exact internal field. Manifest-level tests and older call sites may + // still supply the logical schema. Accept those two representations only: + // a caller-defined lookalike must never be interpreted as trusted metadata. + let mut has_stream_metadata = false; + for field in schema + .fields() + .iter() + .filter(|field| field.name() == crate::db::STREAM_METADATA_COLUMN) + { + if has_stream_metadata { + return Err(OmniError::manifest_internal(format!( + "graph table schema supplies reserved physical field '{}' more than once", + crate::db::STREAM_METADATA_COLUMN + ))); + } + super::stream_token::validate_trusted_stream_metadata_field(field).map_err(|error| { + OmniError::manifest_internal(format!( + "graph table schema supplies a non-canonical reserved physical field '{}': {error}", + crate::db::STREAM_METADATA_COLUMN + )) + })?; + has_stream_metadata = true; + } let mut id_count = 0; - let fields = schema + let mut fields = schema .fields() .iter() .map(|field| { @@ -260,6 +292,13 @@ fn keyed_graph_table_schema(schema: &SchemaRef) -> Result { )); } + // Internal schema v9 provisions the exact nullable trusted-attribution + // envelope on every graph table from creation. Pre-stream/direct rows use + // a null top-level struct; no caller-supplied lookalike is interpreted. + if !has_stream_metadata { + fields.push(super::stream_token::trusted_stream_metadata_field()); + } + Ok(std::sync::Arc::new(Schema::new_with_metadata( fields, schema.metadata.clone(), diff --git a/crates/omnigraph/src/db/manifest/layout.rs b/crates/omnigraph/src/db/manifest/layout.rs index 4eae6082..98100aa5 100644 --- a/crates/omnigraph/src/db/manifest/layout.rs +++ b/crates/omnigraph/src/db/manifest/layout.rs @@ -7,6 +7,7 @@ use crate::error::{OmniError, Result}; use crate::storage::{StorageKind, join_uri, storage_kind_for_uri}; use super::TableIdentity; +use super::token_store::STREAM_TOKEN_DATASET_PATH; const MANIFEST_DIR: &str = "__manifest"; @@ -14,6 +15,11 @@ pub(crate) fn manifest_uri(root: &str) -> String { format!("{}/{}", root.trim_end_matches('/'), MANIFEST_DIR) } +/// Physical location of the graph-global RFC-026 token participant. +pub(super) fn stream_token_uri(root_uri: &str) -> String { + table_uri_for_path(root_uri, STREAM_TOKEN_DATASET_PATH, None) +} + #[cfg(test)] pub(super) async fn open_manifest_dataset(root_uri: &str, branch: Option<&str>) -> Result { let control_session = crate::lance_access::control_session(); @@ -82,6 +88,14 @@ pub(super) fn stream_state_object_id(identity: TableIdentity) -> String { ) } +/// Fixed graph-global authority row for `_stream_tokens.lance`. +/// +/// It deliberately carries no fake/zero table identity: the token participant +/// is graph protocol state, not a user table lifetime. +pub(super) const fn stream_token_authority_object_id() -> &'static str { + "stream_token_authority" +} + pub(super) fn table_id_to_key(request_id: Option<&Vec>) -> lance_namespace::Result { match request_id { Some(request_id) if request_id.len() == 1 && !request_id[0].is_empty() => { diff --git a/crates/omnigraph/src/db/manifest/metadata.rs b/crates/omnigraph/src/db/manifest/metadata.rs index 5e7f8a49..3aeb297f 100644 --- a/crates/omnigraph/src/db/manifest/metadata.rs +++ b/crates/omnigraph/src/db/manifest/metadata.rs @@ -198,12 +198,28 @@ fn full_manifest_object_store_path( table_path: &str, manifest_path: &str, ) -> Result { - if manifest_path.contains("://") { - return object_store_path_from_uri(manifest_path); + // Lance may spell the same local object-store root through different + // filesystem aliases across a commit handle and a later reopen (notably + // `/var` versus `/private/var` on macOS). Once the canonical graph-relative + // table path is present, discard that unstable raw prefix and rebuild the + // object path from the graph root plus the suffix below the dataset. + if let Some((_, suffix)) = manifest_path.rsplit_once(table_path) { + let dataset_uri = join_uri(root_uri, table_path); + let dataset_path = object_store_path_from_uri(&dataset_uri)?; + let suffix = suffix.trim_start_matches('/'); + return if suffix.is_empty() { + Ok(dataset_path) + } else { + Ok(format!( + "{}/{}", + dataset_path.trim_end_matches('/'), + suffix + )) + }; } - if manifest_path.contains(table_path) { - return Ok(manifest_path.to_string()); + if manifest_path.contains("://") { + return object_store_path_from_uri(manifest_path); } let dataset_uri = join_uri(root_uri, table_path); diff --git a/crates/omnigraph/src/db/manifest/migrations.rs b/crates/omnigraph/src/db/manifest/migrations.rs index d3d28122..8522e3b7 100644 --- a/crates/omnigraph/src/db/manifest/migrations.rs +++ b/crates/omnigraph/src/db/manifest/migrations.rs @@ -67,10 +67,14 @@ use crate::error::{OmniError, Result}; /// per-shard epoch floor. /// - v8 — RFC-026 Phase B1 activates data-bearing MemWAL state with the exact /// persisted config-v2 writer profile and recovery-v11 `StreamFold` intents. +/// - v9 — RFC-026 Phase B2 provisions the reserved trusted-row metadata and +/// manifest-selected token authority, and upgrades enrolled streams to +/// config-v3/state-v2/recovery-v12 authority. V8 graphs cross this immutable +/// format boundary by export/init/load rebuild. /// -/// v1–v7 graphs are not served by this binary (see `MIN_SUPPORTED`); the history +/// v1–v8 graphs are not served by this binary (see `MIN_SUPPORTED`); the history /// is kept for provenance and to document what each stamp value meant. -pub(crate) const INTERNAL_MANIFEST_SCHEMA_VERSION: u32 = 8; +pub(crate) const INTERNAL_MANIFEST_SCHEMA_VERSION: u32 = 9; /// The oldest on-disk internal-schema stamp this binary will open. With no /// in-place migration, this equals `INTERNAL_MANIFEST_SCHEMA_VERSION`: a graph @@ -89,7 +93,7 @@ pub(crate) const MIN_SUPPORTED_INTERNAL_SCHEMA_VERSION: u32 = INTERNAL_MANIFEST_ /// stamped each version (verify with /// `git show vX.Y.Z:crates/omnigraph/src/db/manifest/migrations.rs`): /// v1 ≤ 0.3.1, v2 0.4.1–0.6.1, v3 0.6.2–0.7.2, v4 0.8.x, v5 0.9.x, -/// v6 0.10.x, v7 0.11.x, v8 0.12.x. +/// v6 0.10.x, v7 0.11.x, v8 0.12.x, v9 0.13.x. pub(crate) fn release_for_internal_schema_version(stamp: u32) -> &'static str { match stamp { 1 => "0.3.1 or earlier", @@ -100,7 +104,8 @@ pub(crate) fn release_for_internal_schema_version(stamp: u32) -> &'static str { 6 => "0.10.x", 7 => "0.11.x", 8 => "0.12.x", - // Unreachable today (1–8 are mapped; > CURRENT is caught by the ceiling + 9 => "0.13.x", + // Unreachable today (1–9 are mapped; > CURRENT is caught by the ceiling // guard before this is consulted). Worded to read naturally after // "created by omnigraph " if a future bump ever leaves a gap. _ => "an unrecognized older release", @@ -180,12 +185,12 @@ mod tests { use super::*; /// The guard accepts exactly the single served version and refuses anything - /// below the floor or above the ceiling. With `MIN == CURRENT == 8` the live - /// range is exactly `[8, 8]`. + /// below the floor or above the ceiling. With `MIN == CURRENT == 9` the live + /// range is exactly `[9, 9]`. #[test] fn unsupported_guard_accepts_exactly_the_supported_range() { - assert_eq!(INTERNAL_MANIFEST_SCHEMA_VERSION, 8); - assert_eq!(MIN_SUPPORTED_INTERNAL_SCHEMA_VERSION, 8); + assert_eq!(INTERNAL_MANIFEST_SCHEMA_VERSION, 9); + assert_eq!(MIN_SUPPORTED_INTERNAL_SCHEMA_VERSION, 9); for stamp in MIN_SUPPORTED_INTERNAL_SCHEMA_VERSION..=INTERNAL_MANIFEST_SCHEMA_VERSION { assert!( refuse_if_stamp_unsupported(stamp).is_ok(), @@ -215,6 +220,7 @@ mod tests { assert_eq!(release_for_internal_schema_version(6), "0.10.x"); assert_eq!(release_for_internal_schema_version(7), "0.11.x"); assert_eq!(release_for_internal_schema_version(8), "0.12.x"); + assert_eq!(release_for_internal_schema_version(9), "0.13.x"); assert_eq!( release_for_internal_schema_version(99), "an unrecognized older release" diff --git a/crates/omnigraph/src/db/manifest/publisher.rs b/crates/omnigraph/src/db/manifest/publisher.rs index 49432294..682568b7 100644 --- a/crates/omnigraph/src/db/manifest/publisher.rs +++ b/crates/omnigraph/src/db/manifest/publisher.rs @@ -32,8 +32,8 @@ use crate::error::{OmniError, Result}; #[cfg(test)] use super::SubTableUpdate; use super::layout::{ - open_manifest_dataset_with_session, stream_state_object_id, table_object_id, - tombstone_object_id, version_object_id, + open_manifest_dataset_with_session, stream_state_object_id, stream_token_authority_object_id, + table_object_id, tombstone_object_id, version_object_id, }; use super::metadata::{TableVersionMetadata, parse_namespace_version_request}; use super::migrations::{read_stamp, refuse_if_stamp_unsupported}; @@ -42,11 +42,12 @@ use super::state::{ graph_head_object_id, graph_lineage_row_parts, head_lineage_row, manifest_rows_batch, manifest_schema, read_manifest_state, read_publish_scan, }; +use super::stream_token::StreamFoldAttributionSummary; use super::{ ExpectedTableVersions, MAIN_BRANCH_HEAD_KEY, ManifestChange, OBJECT_TYPE_STREAM_STATE, - OBJECT_TYPE_TABLE, OBJECT_TYPE_TABLE_TOMBSTONE, OBJECT_TYPE_TABLE_VERSION, - StreamLifecycleEntry, SubTableEntry, TableIdentity, TableRegistration, TableRename, - TableTombstone, + OBJECT_TYPE_STREAM_TOKEN_AUTHORITY, OBJECT_TYPE_TABLE, OBJECT_TYPE_TABLE_TOMBSTONE, + OBJECT_TYPE_TABLE_VERSION, StreamLifecycleEntry, StreamTokenAuthorityEntry, SubTableEntry, + TableIdentity, TableRegistration, TableRename, TableTombstone, }; /// Bound on the publisher-level retry loop that wraps Lance's row-level CAS @@ -75,6 +76,10 @@ pub(crate) struct LineageIntent { pub merged_parent_commit_id: Option, /// Commit timestamp (microseconds since the UNIX epoch). pub created_at: i64, + /// Fixed RFC-026 winner commitment for a system stream fold. Every other + /// writer leaves this absent. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub stream_fold_attribution: Option, } /// The exact mutable graph-head authority a prepared write observed. A missing @@ -204,6 +209,7 @@ struct LoadedPublishState { lineage_rows: Vec, graph_heads: HashMap, stream_lifecycles: HashMap, + stream_token_authority: StreamTokenAuthorityEntry, } impl GraphNamespacePublisher { @@ -270,6 +276,7 @@ impl GraphNamespacePublisher { lineage_rows: scan.lineage_rows, graph_heads: scan.graph_heads, stream_lifecycles: scan.stream_lifecycles, + stream_token_authority: scan.stream_token_authority, }) } @@ -279,12 +286,14 @@ impl GraphNamespacePublisher { existing_versions: &HashMap<(TableIdentity, u64), SubTableEntry>, existing_tombstones: &HashMap<(TableIdentity, u64), ()>, existing_stream_lifecycles: &HashMap, + existing_stream_token_authority: &StreamTokenAuthorityEntry, ) -> Result> { let mut request_versions = HashMap::<(TableIdentity, u64), ()>::new(); let mut binding_changes = HashMap::::new(); let mut known_tables = known_tables.clone(); let mut rows = Vec::with_capacity(changes.len()); let mut lifecycle_changes = HashMap::::new(); + let mut token_authority_changed = false; let mut max_tombstones = HashMap::::new(); for (identity, version) in existing_tombstones.keys() { max_tombstones @@ -462,7 +471,8 @@ impl GraphNamespacePublisher { } ManifestChange::Update(_) | ManifestChange::Tombstone(_) - | ManifestChange::SetStreamLifecycle { .. } => {} + | ManifestChange::SetStreamLifecycle { .. } + | ManifestChange::SetStreamTokenAuthority { .. } => {} } } @@ -674,6 +684,46 @@ impl GraphNamespacePublisher { row_count: None, }); } + ManifestChange::SetStreamTokenAuthority { expected, next } => { + expected.validate()?; + next.validate()?; + if existing_stream_token_authority != expected { + return Err(OmniError::manifest_read_set_changed( + stream_token_authority_object_id(), + Some(expected.to_metadata_json()?), + Some(existing_stream_token_authority.to_metadata_json()?), + )); + } + if token_authority_changed { + return Err(OmniError::manifest( + "manifest batch changes stream-token authority more than once", + )); + } + token_authority_changed = true; + if next == existing_stream_token_authority { + continue; + } + if next.current_head_witness.table_version + <= expected.current_head_witness.table_version + { + return Err(OmniError::manifest(format!( + "stream-token authority must advance monotonically from version {} to a newer exact version, got {}", + expected.current_head_witness.table_version, + next.current_head_witness.table_version + ))); + } + rows.push(PendingVersionRow { + object_id: stream_token_authority_object_id().to_string(), + object_type: OBJECT_TYPE_STREAM_TOKEN_AUTHORITY.to_string(), + location: Some(next.location.clone()), + metadata: Some(next.to_metadata_json()?), + table_key: String::new(), + identity: None, + table_version: Some(next.current_head_witness.table_version), + table_branch: None, + row_count: None, + }); + } } } @@ -716,6 +766,7 @@ impl GraphNamespacePublisher { merged_parent_commit_id: intent.merged_parent_commit_id.clone(), actor_id: intent.actor_id.clone(), created_at: intent.created_at, + stream_fold_attribution: intent.stream_fold_attribution.clone(), }; let parts = graph_lineage_row_parts(&commit, intent.branch.as_deref())?; Ok(( @@ -844,11 +895,13 @@ impl GraphNamespacePublisher { rows: &[PendingVersionRow], registered_tables: &HashMap, existing_stream_lifecycles: &HashMap, + existing_stream_token_authority: &StreamTokenAuthorityEntry, ) -> Result<( HashMap, Vec, Vec<(TableIdentity, u64)>, HashMap, + StreamTokenAuthorityEntry, )> { let mut registrations = registered_tables.clone(); for row in rows { @@ -892,6 +945,7 @@ impl GraphNamespacePublisher { .map(|(identity, version)| (*identity, *version)) .collect(); let mut stream_lifecycles = existing_stream_lifecycles.clone(); + let mut stream_token_authority = existing_stream_token_authority.clone(); for row in rows { match row.object_type.as_str() { @@ -969,6 +1023,20 @@ impl GraphNamespacePublisher { )?; stream_lifecycles.insert(identity, lifecycle); } + OBJECT_TYPE_STREAM_TOKEN_AUTHORITY => { + let metadata = row.metadata.as_deref().ok_or_else(|| { + OmniError::manifest_internal( + "post-publish fold: stream_token_authority row missing metadata", + ) + })?; + stream_token_authority = StreamTokenAuthorityEntry::from_manifest_row( + &row.object_id, + row.location.as_deref(), + row.table_version, + row.table_branch.as_deref(), + metadata, + )?; + } _ => {} } } @@ -978,6 +1046,7 @@ impl GraphNamespacePublisher { version_map.into_values().collect(), tombstones, stream_lifecycles, + stream_token_authority, )) } @@ -1243,6 +1312,7 @@ impl ManifestBatchPublisher for GraphNamespacePublisher { lineage_rows, graph_heads, stream_lifecycles, + stream_token_authority, } = loaded; // Exact logical authority is checked on EVERY attempt from this @@ -1269,6 +1339,7 @@ impl ManifestBatchPublisher for GraphNamespacePublisher { &existing_versions, &existing_tombstones, &stream_lifecycles, + &stream_token_authority, )?; // Fold the graph commit into the SAME batch so table-version rows @@ -1303,6 +1374,7 @@ impl ManifestBatchPublisher for GraphNamespacePublisher { .map(|(identity, version)| (*identity, *version)), graph_heads, stream_lifecycles, + stream_token_authority, )?; return Ok(PublishOutcome { dataset, @@ -1314,14 +1386,20 @@ impl ManifestBatchPublisher for GraphNamespacePublisher { // Build the post-publish fold inputs from the pre-publish state ∪ the // rows we are about to commit, BEFORE `rows` is moved into merge_rows // (RFC-013 PR2 #1b). Recomputed per attempt from freshly-loaded state. - let (fold_registrations, fold_entries, fold_tombstones, fold_stream_lifecycles) = - Self::fold_inputs( - &existing_versions, - &existing_tombstones, - &rows, - &known_tables, - &stream_lifecycles, - )?; + let ( + fold_registrations, + fold_entries, + fold_tombstones, + fold_stream_lifecycles, + fold_stream_token_authority, + ) = Self::fold_inputs( + &existing_versions, + &existing_tombstones, + &rows, + &known_tables, + &stream_lifecycles, + &stream_token_authority, + )?; let mut fold_graph_heads = graph_heads; if let Some(intent) = lineage { fold_graph_heads.insert( @@ -1344,6 +1422,7 @@ impl ManifestBatchPublisher for GraphNamespacePublisher { fold_tombstones, fold_graph_heads, fold_stream_lifecycles, + fold_stream_token_authority, )?; match self.merge_rows(dataset, rows).await { diff --git a/crates/omnigraph/src/db/manifest/recovery.rs b/crates/omnigraph/src/db/manifest/recovery.rs index 8c7458c9..56c674ac 100644 --- a/crates/omnigraph/src/db/manifest/recovery.rs +++ b/crates/omnigraph/src/db/manifest/recovery.rs @@ -53,8 +53,8 @@ use crate::storage::StorageAdapter; use crate::table_store::StagedTransactionIdentity; use crate::table_store::mem_wal::{ MemWalEnrollmentPlan, MemWalEnrollmentState, capture_current_head_witness, classify_enrollment, - provision_shard_from_exact_index_only, stream_config_v2_hash, - validate_stream_config_v2_binding, + provision_shard_from_exact_index_only, stream_config_v2_hash, stream_config_v3_hash, + validate_stream_config_v3_binding, }; use super::Snapshot; @@ -62,6 +62,7 @@ use super::publisher::{ GraphHeadExpectation, GraphNamespacePublisher, LineageIntent, ManifestBatchPublisher, PublishPrecondition, }; +use super::stream_token::STREAM_FOLD_ACTOR; use super::{ ExpectedTableVersions, ManifestChange, ManifestCoordinator, SubTableUpdate, TableIdentity, TableRegistration, TableRename, TableTombstone, TableVersionExpectation, @@ -131,6 +132,12 @@ async fn publish_recovery_commit( .protocol_v11 .as_ref() .map(|protocol| &protocol.lineage) + }) + .or_else(|| { + sidecar + .protocol_v12 + .as_ref() + .map(|protocol| &protocol.lineage) }); let exact_rollback_id = sidecar .protocol_v3 @@ -164,7 +171,7 @@ async fn publish_recovery_commit( }) .flatten() }); - let intent = match (exact_lineage, exact_rollback_id, kind) { + let mut intent = match (exact_lineage, exact_rollback_id, kind) { (Some(lineage), _, RecoveryKind::RolledForward) => LineageIntent::from(lineage), (_, Some(rollback_graph_commit_id), RecoveryKind::RolledBack) => LineageIntent { graph_commit_id: rollback_graph_commit_id.to_string(), @@ -175,6 +182,7 @@ async fn publish_recovery_commit( .started_at .parse::() .unwrap_or(crate::db::now_micros()?), + stream_fold_attribution: None, }, (Some(_), _, RecoveryKind::OrphanedBranchDiscarded) | (_, Some(_), RecoveryKind::OrphanedBranchDiscarded) => { @@ -195,6 +203,7 @@ async fn publish_recovery_commit( actor_id: Some(RECOVERY_ACTOR.to_string()), merged_parent_commit_id, created_at: crate::db::now_micros()?, + stream_fold_attribution: None, } } _ => { @@ -203,6 +212,11 @@ async fn publish_recovery_commit( )); } }; + if matches!(kind, RecoveryKind::RolledForward) { + if let Some(protocol) = sidecar.protocol_v12.as_ref() { + intent.stream_fold_attribution = Some(protocol.token.attribution_summary.clone()); + } + } let publisher = GraphNamespacePublisher::new_with_session( root_uri, sidecar.branch.as_deref(), @@ -241,6 +255,12 @@ async fn publish_recovery_commit( .protocol_v11 .as_ref() .map(|protocol| &protocol.authority) + }) + .or_else(|| { + sidecar + .protocol_v12 + .as_ref() + .map(|protocol| &protocol.authority) }); let precondition = match (exact_authority, kind) { (Some(authority), RecoveryKind::RolledForward) => { @@ -327,15 +347,26 @@ pub(crate) const RECOVERY_DIR_NAME: &str = "__recovery"; /// exact empty enrollment, but it never restores the base table or deletes /// MemWAL objects. /// -/// v10 → v11: RFC-026 bounded MemWAL fold. Schema v11 is reserved for one +/// v10 → v11: RFC-026 bounded MemWAL fold. Schema v11 was reserved for one /// exact, roll-forward-only base-table transaction that atomically applies one /// immutable flushed generation and advances Lance's merged-generation -/// watermark. The payload fixes the stream binding/configuration, prior table +/// watermark. The payload fixed the stream binding/configuration, prior table /// witness and merge progress, claimed shard epoch, immutable generation cut, /// Lance transaction identity, graph lineage, and exact confirmed manifest -/// outcome. Recovery may finalize exact no-effect or publish an exact owned -/// effect; every other observation fails closed without Restore. -pub(crate) const SIDECAR_SCHEMA_VERSION: u32 = 11; +/// outcome. It is a historical internal-schema-v8 envelope and is never +/// emitted for an internal-schema-v9 graph. +/// +/// v11 → v12: RFC-026 B2 compare-and-chain fold. Schema v12 keeps exactly +/// one base table in `tables` and adds the graph-global `_stream_tokens.lance` +/// participant as a separate typed effect. The Armed payload fixes both Lance +/// transaction identities, the manifest-selected token-table prestate, the +/// complete prior lifecycle, and a digest/count summary of the winning trusted +/// attribution. EffectsConfirmed binds both achieved HEAD witnesses, the +/// exact base-table manifest update, the next token-table authority pointer, +/// and the complete next lifecycle. Recovery may publish only exact/exact or +/// discard exact-no-effect/no-effect; either partial or any foreign movement +/// remains `RecoveryRequired`. +pub(crate) const SIDECAR_SCHEMA_VERSION: u32 = 12; /// The only recovery generation emitted by the manifest-v5 write paths. pub(crate) const IDENTITY_AWARE_SIDECAR_SCHEMA_VERSION: u32 = 9; @@ -343,8 +374,11 @@ pub(crate) const IDENTITY_AWARE_SIDECAR_SCHEMA_VERSION: u32 = 9; /// Exact roll-forward-only RFC-026 MemWAL enrollment generation. pub(crate) const STREAM_ENROLLMENT_SIDECAR_SCHEMA_VERSION: u32 = 10; -/// Exact roll-forward-only RFC-026 MemWAL fold generation. -pub(crate) const STREAM_FOLD_SIDECAR_SCHEMA_VERSION: u32 = 11; +/// Historical base-only RFC-026 MemWAL fold generation. +pub(crate) const LEGACY_STREAM_FOLD_SIDECAR_SCHEMA_VERSION: u32 = 11; + +/// Exact two-participant RFC-026 MemWAL fold generation. +pub(crate) const STREAM_FOLD_SIDECAR_SCHEMA_VERSION: u32 = 12; /// Schema v11 is the first sidecar allowed to describe data-bearing MemWAL /// state, which is bound to stream-config v2 rather than Phase A's config-v1. @@ -490,6 +524,16 @@ pub(crate) enum ClassificationMode { } impl SidecarKind { + /// Recovery intents whose unpublished physical state can change authority + /// observed by every graph writer, even on another branch or table. + /// + /// SchemaApply owns the accepted schema domain. StreamFold v12 owns the + /// graph-global token-table HEAD. Treating all StreamFold generations as + /// global is deliberately conservative for historical sidecars. + pub(crate) fn is_graph_global_barrier(self) -> bool { + matches!(self, Self::SchemaApply | Self::StreamFold) + } + /// Resolve the classification mode for this writer at a given sidecar /// `schema_version`. Exhaustive over `SidecarKind`, so adding a variant is a /// compile error here until its recovery semantics are declared. @@ -657,6 +701,7 @@ impl From<&RecoveryLineageIntent> for LineageIntent { actor_id: intent.actor_id.clone(), merged_parent_commit_id: intent.merged_parent_commit_id.clone(), created_at: intent.created_at, + stream_fold_attribution: None, } } } @@ -964,6 +1009,9 @@ pub(crate) struct RecoveryProtocolV10 { pub baseline_head: super::CurrentHeadWitness, pub enrollment_plan: MemWalEnrollmentPlan, pub intended_binding: super::StreamPhysicalBinding, + /// Complete logical result fixed before the first enrollment effect. It is + /// never reconstructed from the Lance index or shard created later. + pub enrollment_receipt: super::EnrollmentReceipt, } /// Immutable, post-drain fresh-tier cut owned by one schema-v11 fold. @@ -1017,6 +1065,82 @@ pub(crate) struct RecoveryProtocolV11 { pub confirmed_update: Option, } +/// Canonical pre-effect commitment to the complete set of winning stream +/// attribution rows staged for the token-table participant. +/// +/// The digest is computed by the fold planner over its versioned canonical +/// winner encoding. Recovery does not reinterpret or rebuild the winner set; +/// the pre-minted Lance transaction identity owns the staged physical effect, +/// while this summary prevents an Armed intent from being rebound to another +/// winner set with the same row count. +pub(crate) use super::stream_token::StreamFoldAttributionSummary as RecoveryStreamFoldAttributionSummary; + +/// Exact base-table participant in a schema-v12 StreamFold. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(crate) struct RecoveryStreamFoldBaseEffect { + pub planned_transaction: StagedTransactionIdentity, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub confirmed_transaction: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub confirmed_merged_generation: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub confirmed_head: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub confirmed_update: Option, +} + +/// Exact graph-global `_stream_tokens.lance` participant in a schema-v12 +/// StreamFold. `prior_authority` is the only logical prestate: the raw token +/// dataset HEAD is inspected solely to classify whether this exact planned +/// transaction happened. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(crate) struct RecoveryStreamFoldTokenEffect { + pub prior_authority: super::StreamTokenAuthorityEntry, + pub planned_transaction: StagedTransactionIdentity, + /// Complete bounded current-token rows required to finish the only + /// writer-reachable partial state (base exact, token not invoked/effect- + /// free) after a crash between sequential Lance commits. + pub planned_rows: Vec, + pub attribution_summary: RecoveryStreamFoldAttributionSummary, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub confirmed_transaction: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub confirmed_head: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub next_authority: Option, +} + +/// Schema-v12 exact two-participant MemWAL fold payload. +/// +/// Armed is durable before either Lance commit. The base-table participant and +/// the graph-global token participant are classified independently. Only the +/// `(exact effect, exact effect)` matrix cell may roll forward through the one +/// `__manifest` CAS; `(no effect, no effect)` may retire effect-free. Partial, +/// foreign, and ambiguous cells stay owned by this sidecar and fail closed. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(crate) struct RecoveryProtocolV12 { + pub authority: RecoveryAuthorityToken, + pub lineage: RecoveryLineageIntent, + pub binding: super::StreamPhysicalBinding, + pub prior_merged_generation: Option, + pub generation_cut: RecoveryStreamFoldCut, + pub merged_generation: MergedGeneration, + pub effect_phase: RecoveryEffectPhase, + pub prior_lifecycle: super::StreamLifecycleEntry, + /// Complete logical post-fold state fixed before either Lance effect. Its + /// HEAD witness carries the planned `(main, N+1, transaction UUID)` and a + /// null e-tag output slot. Confirmation/recovery substitutes only the exact + /// achieved e-tag-bearing witness; every other field is immutable. + pub planned_next_lifecycle: super::StreamLifecycleEntry, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub next_lifecycle: Option, + pub base: RecoveryStreamFoldBaseEffect, + pub token: RecoveryStreamFoldTokenEffect, +} + /// Schema-v6 EnsureIndices rollback identity retained for compatibility. /// Recovery must still be able to prove that a previously published /// compensation was a rollback rather than infer the outcome from aligned @@ -1098,6 +1222,9 @@ pub(crate) struct RecoverySidecar { /// Exact RFC-026 one-generation fold payload (schema v11 only). #[serde(default, skip_serializing_if = "Option::is_none")] pub protocol_v11: Option, + /// Exact RFC-026 B2 two-participant fold payload (schema v12 only). + #[serde(default, skip_serializing_if = "Option::is_none")] + pub protocol_v12: Option>, /// EnsureIndices-only fixed rollback identity. It does not make the /// physical index effects exact; it only makes compensation retry-safe. #[serde(default, skip_serializing_if = "Option::is_none")] @@ -1563,6 +1690,7 @@ fn validate_sidecar_shape(sidecar_uri: &str, sidecar: &RecoverySidecar) -> Resul || sidecar.protocol_v8.is_some() || sidecar.protocol_v10.is_some() || sidecar.protocol_v11.is_some() + || sidecar.protocol_v12.is_some() { return Err(malformed( "an exact-effect protocol is present on a pre-v3 sidecar".to_string(), @@ -1575,10 +1703,14 @@ fn validate_sidecar_shape(sidecar_uri: &str, sidecar: &RecoverySidecar) -> Resul return validate_stream_enrollment_v10_shape(sidecar_uri, sidecar); } - if sidecar.schema_version == STREAM_FOLD_SIDECAR_SCHEMA_VERSION { + if sidecar.schema_version == LEGACY_STREAM_FOLD_SIDECAR_SCHEMA_VERSION { return validate_stream_fold_v11_shape(sidecar_uri, sidecar); } + if sidecar.schema_version == STREAM_FOLD_SIDECAR_SCHEMA_VERSION { + return validate_stream_fold_v12_shape(sidecar_uri, sidecar); + } + if sidecar.schema_version == IDENTITY_AWARE_SIDECAR_SCHEMA_VERSION { return match sidecar.writer_kind { SidecarKind::Mutation | SidecarKind::Load => { @@ -1592,7 +1724,7 @@ fn validate_sidecar_shape(sidecar_uri: &str, sidecar: &RecoverySidecar) -> Resul "StreamEnrollment requires the dedicated schema-v10 envelope".to_string(), )), SidecarKind::StreamFold => Err(malformed( - "StreamFold requires the dedicated schema-v11 envelope".to_string(), + "StreamFold requires the dedicated schema-v12 envelope".to_string(), )), }; } @@ -1615,6 +1747,7 @@ fn validate_sidecar_shape(sidecar_uri: &str, sidecar: &RecoverySidecar) -> Resul || sidecar.protocol_v8.is_some() || sidecar.protocol_v10.is_some() || sidecar.protocol_v11.is_some() + || sidecar.protocol_v12.is_some() { return Err(malformed( "schema-v5 SchemaApply must target main and cannot carry v3/v4 protocols" @@ -1646,6 +1779,7 @@ fn validate_sidecar_shape(sidecar_uri: &str, sidecar: &RecoverySidecar) -> Resul || sidecar.protocol_v8.is_some() || sidecar.protocol_v10.is_some() || sidecar.protocol_v11.is_some() + || sidecar.protocol_v12.is_some() { return Err(malformed( "a writer-specific exact protocol is present on the wrong sidecar generation" @@ -1956,6 +2090,7 @@ fn validate_stream_enrollment_v10_shape( || sidecar.protocol_v7.is_some() || sidecar.protocol_v8.is_some() || sidecar.protocol_v11.is_some() + || sidecar.protocol_v12.is_some() || sidecar.ensure_indices_rollback_v6.is_some() || sidecar.merge_source_commit_id.is_some() || !sidecar.additional_registrations.is_empty() @@ -1977,9 +2112,14 @@ fn validate_stream_enrollment_v10_shape( || protocol.lineage.merged_parent_commit_id.is_some() || protocol.lineage.graph_commit_id.is_empty() || sidecar.actor_id != protocol.lineage.actor_id + || protocol + .authority + .graph_head + .as_ref() + .is_some_and(|head| head == &protocol.lineage.graph_commit_id) { return Err(malformed( - "StreamEnrollment lineage must be a fixed canonical-main commit owned by the sidecar actor" + "StreamEnrollment lineage must be a distinct fixed canonical-main commit owned by the sidecar actor" .to_string(), )); } @@ -2031,6 +2171,32 @@ fn validate_stream_enrollment_v10_shape( .to_string(), )); } + protocol + .enrollment_receipt + .validate() + .map_err(|error| malformed(format!("invalid enrollment receipt: {error}")))?; + if protocol.enrollment_receipt.physical_binding != *binding { + return Err(malformed( + "StreamEnrollment receipt physical binding differs from its intended binding" + .to_string(), + )); + } + let expected_intent_digest = super::stream_enrollment_intent_digest_v1( + pin.identity, + &binding.table_location, + &protocol.authority.schema_identity_domain, + &protocol.authority.schema_ir_hash, + protocol.authority.schema_identity_version, + &protocol.baseline_head, + &binding.stream_config_hash, + ) + .map_err(|error| malformed(format!("invalid enrollment intent: {error}")))?; + if protocol.enrollment_receipt.enrollment_intent_digest != expected_intent_digest { + return Err(malformed( + "StreamEnrollment receipt digest differs from the fixed canonical caller intent" + .to_string(), + )); + } MemWalEnrollmentPlan::new( protocol.enrollment_plan.enrollment_id, protocol.enrollment_plan.shard_id, @@ -2053,6 +2219,7 @@ fn validate_stream_fold_v11_shape(sidecar_uri: &str, sidecar: &RecoverySidecar) || sidecar.protocol_v7.is_some() || sidecar.protocol_v8.is_some() || sidecar.protocol_v10.is_some() + || sidecar.protocol_v12.is_some() || sidecar.ensure_indices_rollback_v6.is_some() || sidecar.merge_source_commit_id.is_some() || !sidecar.additional_registrations.is_empty() @@ -2081,7 +2248,7 @@ fn validate_stream_fold_v11_shape(sidecar_uri: &str, sidecar: &RecoverySidecar) .is_some_and(|head| head == &protocol.lineage.graph_commit_id) { return Err(malformed( - "StreamFold lineage must be a distinct fixed canonical-main commit owned by the sidecar actor" + "StreamFold lineage must be a distinct fixed canonical-main commit owned by omnigraph:stream-fold" .to_string(), )); } @@ -2278,36 +2445,15 @@ fn validate_stream_fold_v11_shape(sidecar_uri: &str, sidecar: &RecoverySidecar) Ok(()) } -fn validate_canonical_uuid_text( - malformed: &F, - field: &str, - value: &str, - require_v4: bool, -) -> Result -where - F: Fn(String) -> OmniError, -{ - let parsed = ShardId::parse_str(value) - .map_err(|error| malformed(format!("{field} is not a UUID: {error}")))?; - if parsed.is_nil() || parsed.to_string() != value { - return Err(malformed(format!( - "{field} must be canonical lowercase hyphenated non-nil UUID text" - ))); - } - if require_v4 && parsed.get_version_num() != 4 { - return Err(malformed(format!("{field} must be a UUID v4 value"))); - } - Ok(parsed) -} - -fn validate_optimize_v9_shape(sidecar_uri: &str, sidecar: &RecoverySidecar) -> Result<()> { +fn validate_stream_fold_v12_shape(sidecar_uri: &str, sidecar: &RecoverySidecar) -> Result<()> { let malformed = |reason: String| { OmniError::manifest_internal(format!( "recovery sidecar at '{}' has an invalid schema-v{} shape: {}", sidecar_uri, sidecar.schema_version, reason )) }; - if sidecar.writer_kind != SidecarKind::Optimize + if sidecar.writer_kind != SidecarKind::StreamFold + || sidecar.branch.is_some() || sidecar.protocol_v3.is_some() || sidecar.protocol_v4.is_some() || sidecar.protocol_v7.is_some() @@ -2315,413 +2461,905 @@ fn validate_optimize_v9_shape(sidecar_uri: &str, sidecar: &RecoverySidecar) -> R || sidecar.protocol_v10.is_some() || sidecar.protocol_v11.is_some() || sidecar.ensure_indices_rollback_v6.is_some() + || sidecar.merge_source_commit_id.is_some() || !sidecar.additional_registrations.is_empty() || !sidecar.tombstones.is_empty() + || sidecar.schema_apply_manifest_published + || sidecar.schema_apply_target_schema_ir_hash.is_some() { return Err(malformed( - "schema-v9 Optimize must carry only identity-bearing table pins".to_string(), + "schema-v12 StreamFold must target canonical main and carry only protocol_v12" + .to_string(), + )); + } + let protocol = sidecar + .protocol_v12 + .as_ref() + .ok_or_else(|| malformed("missing required protocol_v12 payload".to_string()))?; + validate_authority_identity(&malformed, &protocol.authority)?; + if protocol.lineage.branch.is_some() + || protocol.lineage.merged_parent_commit_id.is_some() + || protocol.lineage.graph_commit_id.is_empty() + || sidecar.actor_id.as_deref() != Some(STREAM_FOLD_ACTOR) + || protocol.lineage.actor_id.as_deref() != Some(STREAM_FOLD_ACTOR) + || sidecar.actor_id != protocol.lineage.actor_id + || protocol + .authority + .graph_head + .as_ref() + .is_some_and(|head| head == &protocol.lineage.graph_commit_id) + { + return Err(malformed( + "StreamFold lineage must be a distinct fixed canonical-main commit owned by omnigraph:stream-fold" + .to_string(), )); } - validate_unique_pin_identities(&malformed, &sidecar.tables, true)?; - Ok(()) -} -fn validate_ensure_indices_v6_shape(sidecar_uri: &str, sidecar: &RecoverySidecar) -> Result<()> { - let malformed = |reason: String| { - OmniError::manifest_internal(format!( - "recovery sidecar at '{}' has an invalid schema-v{} shape: {}", - sidecar_uri, sidecar.schema_version, reason - )) - }; - if !matches!(sidecar.writer_kind, SidecarKind::EnsureIndices) { + validate_unique_pin_identities(&malformed, &sidecar.tables, true)?; + if sidecar.tables.len() != 1 { return Err(malformed(format!( - "schema-v6 is reserved for EnsureIndices, found {:?}", - sidecar.writer_kind, + "bounded StreamFold requires exactly one base-table pin, got {}", + sidecar.tables.len() ))); } - if sidecar.protocol_v3.is_some() - || sidecar.protocol_v4.is_some() - || sidecar.protocol_v7.is_some() - || sidecar.protocol_v8.is_some() - || sidecar.protocol_v10.is_some() - || sidecar.protocol_v11.is_some() - || sidecar.merge_source_commit_id.is_some() - || !sidecar.additional_registrations.is_empty() - || !sidecar.tombstones.is_empty() + let pin = &sidecar.tables[0]; + let expected_post = pin.expected_version.checked_add(1).ok_or_else(|| { + malformed("StreamFold base-table version overflows its exact N+1 outcome".to_string()) + })?; + let prior = &protocol.prior_lifecycle; + prior + .validate() + .map_err(|error| malformed(format!("invalid prior StreamFold lifecycle: {error}")))?; + if pin.expected_version == 0 + || pin.table_branch.is_some() + || pin.post_commit_pin != expected_post + || prior.identity != pin.identity + || prior.diagnostic_table_key != pin.table_key + || prior.lifecycle != super::StreamLifecycle::Open + || prior.binding != protocol.binding + || prior.current_head_witness.table_version != pin.expected_version + || prior.current_head_witness.branch_identifier + != lance::dataset::refs::BranchIdentifier::main() { return Err(malformed( - "schema-v6 EnsureIndices cannot carry exact, merge, registration, or tombstone fields" + "StreamFold base pin, physical binding, and complete prior lifecycle disagree" .to_string(), )); } - let protocol = sidecar.ensure_indices_rollback_v6.as_ref().ok_or_else(|| { - malformed("schema-v6 EnsureIndices is missing its fixed rollback payload".to_string()) - })?; - if protocol.rollback_graph_commit_id.is_empty() { + + let binding = &protocol.binding; + let canonical_path = super::table_path_for_identity(&pin.table_key, pin.identity) + .map_err(|error| malformed(format!("invalid StreamFold binding path: {error}")))?; + if binding + .identity() + .map_err(|error| malformed(format!("invalid StreamFold binding identity: {error}")))? + != pin.identity + || binding.table_location != canonical_path + || binding.table_branch.is_some() + || binding.shard_ids.len() != 1 + || binding.shard_ids[0] != protocol.generation_cut.shard_id.to_string() + || binding.stream_config_version != super::stream::STREAM_CONFIG_VERSION + || binding.stream_config_hash != stream_config_v3_hash() + { return Err(malformed( - "schema-v6 EnsureIndices has an empty rollback commit id".to_string(), + "StreamFold physical binding differs from its base pin, shard cut, or config-v3 contract" + .to_string(), )); } - let pin_keys: HashSet<&str> = sidecar - .tables - .iter() - .map(|pin| pin.table_key.as_str()) - .collect(); - if sidecar.tables.is_empty() || pin_keys.len() != sidecar.tables.len() { + validate_canonical_uuid_text( + &malformed, + "StreamFold enrollment UUID", + &binding.enrollment_id, + true, + )?; + validate_canonical_uuid_text( + &malformed, + "StreamFold shard UUID", + &binding.shard_ids[0], + true, + )?; + if binding.enrollment_id == binding.shard_ids[0] { return Err(malformed( - "schema-v6 EnsureIndices requires unique non-empty table pins".to_string(), + "StreamFold enrollment and shard UUIDs must be distinct".to_string(), )); } - if let Some(outcomes) = protocol.rollback_audit_outcomes.as_ref() { - let outcome_keys: HashSet<&str> = outcomes - .iter() - .map(|outcome| outcome.table_key.as_str()) - .collect(); - if outcome_keys.len() != outcomes.len() || !outcome_keys.is_subset(&pin_keys) { - return Err(malformed( - "rollback audit outcomes must name a unique subset of table pins".to_string(), - )); - } - } - Ok(()) -} -fn validate_ensure_indices_v8_shape(sidecar_uri: &str, sidecar: &RecoverySidecar) -> Result<()> { - let malformed = |reason: String| { - OmniError::manifest_internal(format!( - "recovery sidecar at '{}' has an invalid schema-v{} shape: {}", - sidecar_uri, sidecar.schema_version, reason - )) - }; - if sidecar.writer_kind != SidecarKind::EnsureIndices { - return Err(malformed(format!( - "schema-v8 is reserved for EnsureIndices, found {:?}", - sidecar.writer_kind - ))); - } - if sidecar.protocol_v3.is_some() - || sidecar.protocol_v4.is_some() - || sidecar.protocol_v7.is_some() - || sidecar.protocol_v10.is_some() - || sidecar.protocol_v11.is_some() - || sidecar.ensure_indices_rollback_v6.is_some() - || sidecar.merge_source_commit_id.is_some() - || !sidecar.additional_registrations.is_empty() - || !sidecar.tombstones.is_empty() - || sidecar.schema_apply_manifest_published - || sidecar.schema_apply_target_schema_ir_hash.is_some() + let cut = &protocol.generation_cut; + let prior_epoch_floor = prior + .epoch_floor_by_shard + .get(&cut.shard_id.to_string()) + .copied() + .ok_or_else(|| malformed("prior lifecycle is missing the bound shard epoch".to_string()))?; + if prior_epoch_floor == 0 + || cut.writer_epoch <= prior_epoch_floor + || cut.shard_manifest_version == 0 + || cut.replay_after_wal_entry_position == 0 + || cut.generation == 0 + || cut.generation_path.is_empty() + || cut.generation_path.trim() != cut.generation_path { return Err(malformed( - "schema-v8 EnsureIndices must carry authority/delta only in protocol_v8".to_string(), + "StreamFold requires a higher claimed epoch and a non-empty exact flushed-generation cut" + .to_string(), )); } - let protocol = sidecar - .protocol_v8 - .as_ref() - .ok_or_else(|| malformed("missing required protocol_v8 payload".to_string()))?; - validate_authority_identity(&malformed, &protocol.authority)?; - if !protocol.intended_delta.registrations.is_empty() - || !protocol.intended_delta.renames.is_empty() - || !protocol.intended_delta.tombstones.is_empty() + if protocol.merged_generation.shard_id != cut.shard_id + || protocol.merged_generation.generation != cut.generation { return Err(malformed( - "schema-v8 EnsureIndices cannot register or tombstone tables".to_string(), + "StreamFold merged generation differs from its immutable shard cut".to_string(), )); } - - let sidecar_branch = sidecar.branch.as_deref().filter(|branch| *branch != "main"); - let lineage_branch = protocol - .lineage - .branch - .as_deref() - .filter(|branch| *branch != "main"); - if sidecar_branch != lineage_branch - || sidecar.actor_id != protocol.lineage.actor_id - || protocol.lineage.merged_parent_commit_id.is_some() - { - return Err(malformed( - "schema-v8 sidecar branch/actor does not match its original lineage".to_string(), - )); + match protocol.prior_merged_generation.as_ref() { + Some(prior_progress) + if prior_progress.shard_id == cut.shard_id + && prior_progress.generation > 0 + && prior_progress + .generation + .checked_add(1) + .is_some_and(|next| next == cut.generation) => {} + None if cut.generation == 1 => {} + _ => { + return Err(malformed( + "StreamFold cut must be the one exact successor of prior merge progress" + .to_string(), + )); + } } - if protocol.lineage.graph_commit_id.is_empty() - || protocol.rollback_graph_commit_id.is_empty() - || protocol.lineage.graph_commit_id == protocol.rollback_graph_commit_id + + if protocol.base.planned_transaction.read_version != pin.expected_version + || protocol.base.planned_transaction.uuid == prior.current_head_witness.transaction_uuid { return Err(malformed( - "schema-v8 original and rollback commit ids must be non-empty and distinct".to_string(), + "StreamFold base transaction must read the prior pin and have a fresh UUID".to_string(), )); } + validate_canonical_uuid_text( + &malformed, + "StreamFold base planned transaction UUID", + &protocol.base.planned_transaction.uuid, + false, + )?; - validate_unique_pin_identities(&malformed, &sidecar.tables, true)?; - let pin_ids: HashSet = sidecar.tables.iter().map(|pin| pin.identity).collect(); - let pin_aliases: HashSet<&str> = sidecar - .tables - .iter() - .map(|pin| pin.table_key.as_str()) - .collect(); - let effect_ids: HashSet = protocol - .effects - .iter() - .map(|effect| effect.identity) - .collect(); - let effect_uuids: HashSet<&str> = protocol - .effects - .iter() - .map(|effect| effect.planned_transaction.uuid.as_str()) - .collect(); - let delta_ids: HashSet = protocol - .intended_delta - .table_updates - .iter() - .map(|slot| slot.identity) - .collect(); - if effect_ids.len() != protocol.effects.len() - || effect_uuids.len() != protocol.effects.len() - || effect_uuids.iter().any(|uuid| uuid.is_empty()) - || delta_ids.len() != protocol.intended_delta.table_updates.len() - || effect_ids != pin_ids - || delta_ids != pin_ids + protocol + .token + .prior_authority + .validate() + .map_err(|error| malformed(format!("invalid prior stream-token authority: {error}")))?; + if protocol.token.planned_transaction.read_version + != protocol + .token + .prior_authority + .current_head_witness + .table_version + || protocol.token.planned_transaction.uuid + == protocol + .token + .prior_authority + .current_head_witness + .transaction_uuid { return Err(malformed( - "schema-v8 pins, effects, transaction UUIDs, and delta slots must be unique and one-to-one" + "StreamFold token transaction must read the manifest-selected prior token version and have a fresh UUID" .to_string(), )); } - if let Some(outcomes) = protocol.rollback_audit_outcomes.as_ref() { - let outcome_keys: HashSet<&str> = outcomes - .iter() - .map(|outcome| outcome.table_key.as_str()) - .collect(); - if outcome_keys.len() != outcomes.len() || !outcome_keys.is_subset(&pin_aliases) { + validate_canonical_uuid_text( + &malformed, + "StreamFold token planned transaction UUID", + &protocol.token.planned_transaction.uuid, + false, + )?; + let expected_token_post = protocol + .token + .planned_transaction + .read_version + .checked_add(1) + .ok_or_else(|| malformed("StreamFold token table version overflows".to_string()))?; + validate_stream_fold_attribution_summary(&malformed, &protocol.token.attribution_summary)?; + if protocol.token.planned_rows.is_empty() + || protocol.token.planned_rows.len() + > crate::table_store::mem_wal::B1_MAX_GENERATION_ROWS as usize + { + return Err(malformed(format!( + "StreamFold planned token rows must remain within the 1..={} generation bound", + crate::table_store::mem_wal::B1_MAX_GENERATION_ROWS + ))); + } + super::token_store::validate_stream_token_plan_bounds(&protocol.token.planned_rows).map_err( + |error| { + malformed(format!( + "StreamFold planned token rows exceed their fixed config-v3 recovery bounds: {error}" + )) + }, + )?; + let mut prior_logical_id = None; + for row in &protocol.token.planned_rows { + row.validate() + .map_err(|error| malformed(format!("invalid planned stream-token row: {error}")))?; + if row.identity != pin.identity + || row.origin_enrollment_id != protocol.binding.enrollment_id + || row.stream_incarnation_id + != protocol + .prior_lifecycle + .enrollment_receipt + .stream_incarnation_id + || prior_logical_id.is_some_and(|prior| prior >= row.logical_id.as_str()) + { return Err(malformed( - "schema-v8 rollback audit outcomes must name a unique subset of table pins" + "StreamFold planned token rows must match the base identity/enrollment and be strictly sorted by logical id" .to_string(), )); } + prior_logical_id = Some(row.logical_id.as_str()); + } + let planned_attribution = super::stream_token::stream_fold_attribution_commitment( + &protocol.token.planned_rows, + ) + .map_err(|error| malformed(format!("invalid planned StreamFold attribution: {error}")))?; + if planned_attribution.visible_contributor_count + != protocol.token.attribution_summary.visible_contributor_count + || planned_attribution.visible_write_count + != protocol.token.attribution_summary.visible_write_count + || planned_attribution.winning_attribution_digest + != protocol.token.attribution_summary.winning_attribution_digest + { + return Err(malformed( + "StreamFold planned token rows differ from their fixed attribution summary" + .to_string(), + )); } + let planned_base_head = super::CurrentHeadWitness { + branch_identifier: lance::dataset::refs::BranchIdentifier::main(), + table_version: expected_post, + transaction_uuid: protocol.base.planned_transaction.uuid.clone(), + manifest_e_tag: None, + }; + validate_stream_fold_next_lifecycle( + &malformed, + sidecar, + protocol, + &protocol.planned_next_lifecycle, + &planned_base_head, + )?; - for pin in &sidecar.tables { - let effect = protocol - .effects - .iter() - .find(|effect| effect.identity == pin.identity) - .expect("schema-v8 key sets checked above"); - let slot = protocol - .intended_delta - .table_updates - .iter() - .find(|slot| slot.identity == pin.identity) - .expect("schema-v8 key sets checked above"); - if effect.table_key != pin.table_key || slot.table_key != pin.table_key { - return Err(malformed(format!( - "schema-v8 identity {} has inconsistent diagnostic aliases", - pin.identity - ))); - } - let exact_output = pin.expected_version.checked_add(1).ok_or_else(|| { - malformed(format!( - "schema-v8 effect '{}' overflows its output version", - pin.table_key - )) - })?; - if effect.planned_transaction.read_version != pin.expected_version - || pin.post_commit_pin != exact_output - || slot.expected_version != pin.expected_version - || slot.table_branch != pin.table_branch - { - return Err(malformed(format!( - "schema-v8 effect '{}' does not match its pin/delta pre-state", - pin.table_key - ))); - } - let is_first_touch = effect.source_fork_version.is_some(); - if (is_first_touch - && !pin - .table_branch - .as_deref() - .is_some_and(|branch| branch != "main" && Some(branch) == sidecar_branch)) - || effect - .source_fork_version - .is_some_and(|version| version != pin.expected_version) - { - return Err(malformed(format!( - "schema-v8 effect '{}' has inconsistent first-touch fork identity", - pin.table_key - ))); - } - match protocol.effect_phase { - RecoveryEffectPhase::Armed => { - if effect.confirmed_transaction.is_some() - || effect.confirmed_branch_identifier.is_some() - || slot.confirmed.is_some() - || pin.confirmed_version.is_some() - { - return Err(malformed(format!( - "Armed schema-v8 effect '{}' already carries confirmation", - pin.table_key - ))); - } + let base_confirmation = ( + protocol.base.confirmed_transaction.as_ref(), + protocol.base.confirmed_merged_generation.as_ref(), + protocol.base.confirmed_head.as_ref(), + protocol.base.confirmed_update.as_ref(), + ); + let token_confirmation = ( + protocol.token.confirmed_transaction.as_ref(), + protocol.token.confirmed_head.as_ref(), + protocol.token.next_authority.as_ref(), + ); + match ( + protocol.effect_phase, + base_confirmation, + token_confirmation, + protocol.next_lifecycle.as_ref(), + ) { + (RecoveryEffectPhase::Armed, (None, None, None, None), (None, None, None), None) + if pin.confirmed_version.is_none() => {} + ( + RecoveryEffectPhase::EffectsConfirmed, + ( + Some(base_transaction), + Some(confirmed_merged_generation), + Some(base_head), + Some(base_update), + ), + (Some(token_transaction), Some(token_head), Some(next_token_authority)), + Some(next_lifecycle), + ) => { + if base_transaction != &protocol.base.planned_transaction + || confirmed_merged_generation != &protocol.merged_generation + || base_head.branch_identifier != lance::dataset::refs::BranchIdentifier::main() + || base_head.table_version != expected_post + || base_head.transaction_uuid != protocol.base.planned_transaction.uuid + || protocol + .prior_lifecycle + .current_head_witness + .manifest_e_tag + .as_ref() + .is_some_and(|prior_e_tag| { + base_head + .manifest_e_tag + .as_ref() + .is_none_or(|achieved_e_tag| achieved_e_tag == prior_e_tag) + }) + || base_update.table_version != expected_post + || base_update.table_branch.is_some() + || pin.confirmed_version != Some(expected_post) + { + return Err(malformed( + "EffectsConfirmed StreamFold base output differs from its exact planned N+1 effect" + .to_string(), + )); } - RecoveryEffectPhase::EffectsConfirmed => { - let confirmed_transaction = - effect.confirmed_transaction.as_ref().ok_or_else(|| { - malformed(format!( - "EffectsConfirmed schema-v8 effect '{}' lacks transaction confirmation", - pin.table_key - )) - })?; - let confirmed_update = slot.confirmed.as_ref().ok_or_else(|| { - malformed(format!( - "EffectsConfirmed schema-v8 effect '{}' lacks a manifest output", - pin.table_key - )) - })?; - if confirmed_transaction != &effect.planned_transaction - || confirmed_update.table_version != exact_output - || confirmed_update.table_branch != pin.table_branch - || pin.confirmed_version != Some(exact_output) - || is_first_touch != effect.confirmed_branch_identifier.is_some() - { - return Err(malformed(format!( - "schema-v8 effect '{}' confirmation differs from its exact plan", - pin.table_key - ))); - } + if token_transaction != &protocol.token.planned_transaction + || token_head.branch_identifier != lance::dataset::refs::BranchIdentifier::main() + || token_head.table_version != expected_token_post + || token_head.transaction_uuid != protocol.token.planned_transaction.uuid + || protocol + .token + .prior_authority + .current_head_witness + .manifest_e_tag + .as_ref() + .is_some_and(|prior_e_tag| { + token_head + .manifest_e_tag + .as_ref() + .is_none_or(|achieved_e_tag| achieved_e_tag == prior_e_tag) + }) + || next_token_authority.current_head_witness != *token_head + || next_token_authority.location != protocol.token.prior_authority.location + || next_token_authority.schema_version + != protocol.token.prior_authority.schema_version + || next_token_authority.schema_hash != protocol.token.prior_authority.schema_hash + { + return Err(malformed( + "EffectsConfirmed StreamFold token output differs from its exact planned N+1 effect" + .to_string(), + )); } + next_token_authority.validate().map_err(|error| { + malformed(format!("invalid next stream-token authority: {error}")) + })?; + let mut expected_next_lifecycle = protocol.planned_next_lifecycle.clone(); + expected_next_lifecycle.current_head_witness = base_head.clone(); + if next_lifecycle != &expected_next_lifecycle { + return Err(malformed( + "EffectsConfirmed StreamFold next lifecycle differs from its pre-effect plan outside the achieved base HEAD witness" + .to_string(), + )); + } + next_lifecycle.validate().map_err(|error| { + malformed(format!( + "invalid confirmed next StreamFold lifecycle: {error}" + )) + })?; + } + _ => { + return Err(malformed( + "StreamFold confirmation fields must be all absent while Armed and both participants plus the full next lifecycle must be exact while EffectsConfirmed" + .to_string(), + )); } } Ok(()) } -fn validate_schema_apply_v7_shape(sidecar_uri: &str, sidecar: &RecoverySidecar) -> Result<()> { - let malformed = |reason: String| { - OmniError::manifest_internal(format!( - "recovery sidecar at '{}' has an invalid schema-v{} shape: {}", - sidecar_uri, sidecar.schema_version, reason +fn validate_stream_fold_attribution_summary( + malformed: &F, + summary: &RecoveryStreamFoldAttributionSummary, +) -> Result<()> +where + F: Fn(String) -> OmniError, +{ + summary.validate().map_err(|error| { + malformed(format!( + "invalid StreamFold winning attribution summary: {error}" )) - }; - if !matches!(sidecar.writer_kind, SidecarKind::SchemaApply) { - return Err(malformed(format!( - "schema-v7 is reserved for SchemaApply, found {:?}", - sidecar.writer_kind - ))); - } - if sidecar.branch.is_some() - || sidecar.protocol_v3.is_some() - || sidecar.protocol_v4.is_some() - || sidecar.protocol_v8.is_some() - || sidecar.protocol_v10.is_some() - || sidecar.protocol_v11.is_some() - || sidecar.ensure_indices_rollback_v6.is_some() - || sidecar.merge_source_commit_id.is_some() - || !sidecar.additional_registrations.is_empty() + }) +} + +fn validate_stream_fold_next_lifecycle( + malformed: &F, + sidecar: &RecoverySidecar, + protocol: &RecoveryProtocolV12, + next: &super::StreamLifecycleEntry, + base_head: &super::CurrentHeadWitness, +) -> Result<()> +where + F: Fn(String) -> OmniError, +{ + next.validate() + .map_err(|error| malformed(format!("invalid next StreamFold lifecycle: {error}")))?; + let prior = &protocol.prior_lifecycle; + let expected_revision = prior + .lifecycle_revision + .checked_add(1) + .ok_or_else(|| malformed("StreamFold lifecycle revision overflows".to_string()))?; + let mut expected_epoch_floors = prior.epoch_floor_by_shard.clone(); + expected_epoch_floors.insert( + protocol.generation_cut.shard_id.to_string(), + protocol.generation_cut.writer_epoch, + ); + if next.identity != prior.identity + || next.diagnostic_table_key != prior.diagnostic_table_key + || next.lifecycle != prior.lifecycle + || next.binding != prior.binding + || next.current_head_witness != *base_head + || next.epoch_floor_by_shard != expected_epoch_floors + || next.lifecycle_revision != expected_revision + || next.enrollment_receipt != prior.enrollment_receipt + || next.management_receipts != prior.management_receipts + || next.claim_receipts != prior.claim_receipts + || next.current_claim_receipt_id != prior.current_claim_receipt_id + || next.drain != prior.drain + || next.strict_block != prior.strict_block + || next.sealed_proof != prior.sealed_proof + { + return Err(malformed( + "StreamFold next lifecycle changes authority outside the exact HEAD/epoch/revision/fold-summary transition" + .to_string(), + )); + } + let summary = next.last_fold_summary.as_ref().ok_or_else(|| { + malformed("EffectsConfirmed StreamFold next lifecycle has no fold summary".to_string()) + })?; + if summary.operation_id != sidecar.operation_id + || summary.graph_commit_id.as_deref() != Some(protocol.lineage.graph_commit_id.as_str()) + || summary.outcome != super::stream::LastFoldOutcome::Published + || summary.exact_generation_cut.shard_id != protocol.generation_cut.shard_id.to_string() + || summary.exact_generation_cut.writer_epoch != protocol.generation_cut.writer_epoch + || summary.exact_generation_cut.shard_manifest_version + != protocol.generation_cut.shard_manifest_version + || summary.exact_generation_cut.replay_after_wal_entry_position + != protocol.generation_cut.replay_after_wal_entry_position + || summary.exact_generation_cut.generation != protocol.generation_cut.generation + || summary.exact_generation_cut.generation_path != protocol.generation_cut.generation_path + || summary.visible_rows != protocol.token.attribution_summary.visible_write_count + { + return Err(malformed( + "StreamFold next lifecycle summary differs from the fixed lineage, generation cut, or winning attribution" + .to_string(), + )); + } + Ok(()) +} + +fn validate_canonical_uuid_text( + malformed: &F, + field: &str, + value: &str, + require_v4: bool, +) -> Result +where + F: Fn(String) -> OmniError, +{ + let parsed = ShardId::parse_str(value) + .map_err(|error| malformed(format!("{field} is not a UUID: {error}")))?; + if parsed.is_nil() || parsed.to_string() != value { + return Err(malformed(format!( + "{field} must be canonical lowercase hyphenated non-nil UUID text" + ))); + } + if require_v4 && parsed.get_version_num() != 4 { + return Err(malformed(format!("{field} must be a UUID v4 value"))); + } + Ok(parsed) +} + +fn validate_optimize_v9_shape(sidecar_uri: &str, sidecar: &RecoverySidecar) -> Result<()> { + let malformed = |reason: String| { + OmniError::manifest_internal(format!( + "recovery sidecar at '{}' has an invalid schema-v{} shape: {}", + sidecar_uri, sidecar.schema_version, reason + )) + }; + if sidecar.writer_kind != SidecarKind::Optimize + || sidecar.protocol_v3.is_some() + || sidecar.protocol_v4.is_some() + || sidecar.protocol_v7.is_some() + || sidecar.protocol_v8.is_some() + || sidecar.protocol_v10.is_some() + || sidecar.protocol_v11.is_some() + || sidecar.protocol_v12.is_some() + || sidecar.ensure_indices_rollback_v6.is_some() + || !sidecar.additional_registrations.is_empty() + || !sidecar.tombstones.is_empty() + { + return Err(malformed( + "schema-v9 Optimize must carry only identity-bearing table pins".to_string(), + )); + } + validate_unique_pin_identities(&malformed, &sidecar.tables, true)?; + Ok(()) +} + +fn validate_ensure_indices_v6_shape(sidecar_uri: &str, sidecar: &RecoverySidecar) -> Result<()> { + let malformed = |reason: String| { + OmniError::manifest_internal(format!( + "recovery sidecar at '{}' has an invalid schema-v{} shape: {}", + sidecar_uri, sidecar.schema_version, reason + )) + }; + if !matches!(sidecar.writer_kind, SidecarKind::EnsureIndices) { + return Err(malformed(format!( + "schema-v6 is reserved for EnsureIndices, found {:?}", + sidecar.writer_kind, + ))); + } + if sidecar.protocol_v3.is_some() + || sidecar.protocol_v4.is_some() + || sidecar.protocol_v7.is_some() + || sidecar.protocol_v8.is_some() + || sidecar.protocol_v10.is_some() + || sidecar.protocol_v11.is_some() + || sidecar.protocol_v12.is_some() + || sidecar.merge_source_commit_id.is_some() + || !sidecar.additional_registrations.is_empty() + || !sidecar.tombstones.is_empty() + { + return Err(malformed( + "schema-v6 EnsureIndices cannot carry exact, merge, registration, or tombstone fields" + .to_string(), + )); + } + let protocol = sidecar.ensure_indices_rollback_v6.as_ref().ok_or_else(|| { + malformed("schema-v6 EnsureIndices is missing its fixed rollback payload".to_string()) + })?; + if protocol.rollback_graph_commit_id.is_empty() { + return Err(malformed( + "schema-v6 EnsureIndices has an empty rollback commit id".to_string(), + )); + } + let pin_keys: HashSet<&str> = sidecar + .tables + .iter() + .map(|pin| pin.table_key.as_str()) + .collect(); + if sidecar.tables.is_empty() || pin_keys.len() != sidecar.tables.len() { + return Err(malformed( + "schema-v6 EnsureIndices requires unique non-empty table pins".to_string(), + )); + } + if let Some(outcomes) = protocol.rollback_audit_outcomes.as_ref() { + let outcome_keys: HashSet<&str> = outcomes + .iter() + .map(|outcome| outcome.table_key.as_str()) + .collect(); + if outcome_keys.len() != outcomes.len() || !outcome_keys.is_subset(&pin_keys) { + return Err(malformed( + "rollback audit outcomes must name a unique subset of table pins".to_string(), + )); + } + } + Ok(()) +} + +fn validate_ensure_indices_v8_shape(sidecar_uri: &str, sidecar: &RecoverySidecar) -> Result<()> { + let malformed = |reason: String| { + OmniError::manifest_internal(format!( + "recovery sidecar at '{}' has an invalid schema-v{} shape: {}", + sidecar_uri, sidecar.schema_version, reason + )) + }; + if sidecar.writer_kind != SidecarKind::EnsureIndices { + return Err(malformed(format!( + "schema-v8 is reserved for EnsureIndices, found {:?}", + sidecar.writer_kind + ))); + } + if sidecar.protocol_v3.is_some() + || sidecar.protocol_v4.is_some() + || sidecar.protocol_v7.is_some() + || sidecar.protocol_v10.is_some() + || sidecar.protocol_v11.is_some() + || sidecar.protocol_v12.is_some() + || sidecar.ensure_indices_rollback_v6.is_some() + || sidecar.merge_source_commit_id.is_some() + || !sidecar.additional_registrations.is_empty() || !sidecar.tombstones.is_empty() || sidecar.schema_apply_manifest_published || sidecar.schema_apply_target_schema_ir_hash.is_some() { return Err(malformed( - "schema-v7 SchemaApply must target main and carry authority/delta only in protocol_v7" - .to_string(), + "schema-v8 EnsureIndices must carry authority/delta only in protocol_v8".to_string(), )); } let protocol = sidecar - .protocol_v7 + .protocol_v8 .as_ref() - .ok_or_else(|| malformed("missing required protocol_v7 payload".to_string()))?; + .ok_or_else(|| malformed("missing required protocol_v8 payload".to_string()))?; validate_authority_identity(&malformed, &protocol.authority)?; - if protocol.target_schema_ir_hash.is_empty() { + if !protocol.intended_delta.registrations.is_empty() + || !protocol.intended_delta.renames.is_empty() + || !protocol.intended_delta.tombstones.is_empty() + { return Err(malformed( - "schema-v7 SchemaApply has an empty target schema identity".to_string(), + "schema-v8 EnsureIndices cannot register or tombstone tables".to_string(), )); } - if sidecar.actor_id != protocol.lineage.actor_id - || protocol.lineage.branch.is_some() + + let sidecar_branch = sidecar.branch.as_deref().filter(|branch| *branch != "main"); + let lineage_branch = protocol + .lineage + .branch + .as_deref() + .filter(|branch| *branch != "main"); + if sidecar_branch != lineage_branch + || sidecar.actor_id != protocol.lineage.actor_id || protocol.lineage.merged_parent_commit_id.is_some() { return Err(malformed( - "schema-v7 sidecar actor/main lineage fields do not match".to_string(), + "schema-v8 sidecar branch/actor does not match its original lineage".to_string(), )); } if protocol.lineage.graph_commit_id.is_empty() || protocol.rollback_graph_commit_id.is_empty() - || protocol.rollback_graph_commit_id == protocol.lineage.graph_commit_id + || protocol.lineage.graph_commit_id == protocol.rollback_graph_commit_id { return Err(malformed( - "schema-v7 original and rollback commit ids must be non-empty and distinct".to_string(), + "schema-v8 original and rollback commit ids must be non-empty and distinct".to_string(), )); } - validate_unique_pin_identities(&malformed, &sidecar.tables, false)?; + validate_unique_pin_identities(&malformed, &sidecar.tables, true)?; let pin_ids: HashSet = sidecar.tables.iter().map(|pin| pin.identity).collect(); - let rollback_aliases: HashSet<&str> = sidecar + let pin_aliases: HashSet<&str> = sidecar .tables .iter() - .map(|pin| schema_apply_rollback_table_key(protocol, pin)) + .map(|pin| pin.table_key.as_str()) .collect(); let effect_ids: HashSet = protocol .effects .iter() .map(|effect| effect.identity) .collect(); + let effect_uuids: HashSet<&str> = protocol + .effects + .iter() + .map(|effect| effect.planned_transaction.uuid.as_str()) + .collect(); let delta_ids: HashSet = protocol .intended_delta .table_updates .iter() .map(|slot| slot.identity) .collect(); - let effect_uuids: HashSet<&str> = protocol - .effects - .iter() - .map(|effect| effect.kind.planned_transaction().uuid.as_str()) - .collect(); if effect_ids.len() != protocol.effects.len() - || delta_ids.len() != protocol.intended_delta.table_updates.len() || effect_uuids.len() != protocol.effects.len() || effect_uuids.iter().any(|uuid| uuid.is_empty()) - || pin_ids != effect_ids - || pin_ids != delta_ids + || delta_ids.len() != protocol.intended_delta.table_updates.len() + || effect_ids != pin_ids + || delta_ids != pin_ids { return Err(malformed( - "schema-v7 pins, effects, transaction UUIDs, and delta slots must be unique and one-to-one" + "schema-v8 pins, effects, transaction UUIDs, and delta slots must be unique and one-to-one" .to_string(), )); } + if let Some(outcomes) = protocol.rollback_audit_outcomes.as_ref() { + let outcome_keys: HashSet<&str> = outcomes + .iter() + .map(|outcome| outcome.table_key.as_str()) + .collect(); + if outcome_keys.len() != outcomes.len() || !outcome_keys.is_subset(&pin_aliases) { + return Err(malformed( + "schema-v8 rollback audit outcomes must name a unique subset of table pins" + .to_string(), + )); + } + } - let registration_ids: HashSet = protocol - .intended_delta - .registrations - .iter() - .map(|registration| registration.identity) - .collect(); - let tombstone_ids: HashSet = protocol - .intended_delta - .tombstones - .iter() - .map(|tombstone| tombstone.identity) - .collect(); - let rename_ids: HashSet = protocol - .intended_delta - .renames - .iter() - .map(|rename| rename.identity) - .collect(); - let first_touch_ids: HashSet = protocol - .effects - .iter() - .filter(|effect| effect.kind.is_first_touch()) - .map(|effect| effect.identity) - .collect(); - if registration_ids.len() != protocol.intended_delta.registrations.len() - || tombstone_ids.len() != protocol.intended_delta.tombstones.len() - || rename_ids.len() != protocol.intended_delta.renames.len() - || registration_ids != first_touch_ids - || !registration_ids.is_disjoint(&tombstone_ids) - || !registration_ids.is_disjoint(&rename_ids) - || !tombstone_ids.is_disjoint(&rename_ids) - { - return Err(malformed( - "SchemaApply first-touch effects must match unique registration identities, while registrations, renames, and tombstones remain identity-disjoint" - .to_string(), + for pin in &sidecar.tables { + let effect = protocol + .effects + .iter() + .find(|effect| effect.identity == pin.identity) + .expect("schema-v8 key sets checked above"); + let slot = protocol + .intended_delta + .table_updates + .iter() + .find(|slot| slot.identity == pin.identity) + .expect("schema-v8 key sets checked above"); + if effect.table_key != pin.table_key || slot.table_key != pin.table_key { + return Err(malformed(format!( + "schema-v8 identity {} has inconsistent diagnostic aliases", + pin.identity + ))); + } + let exact_output = pin.expected_version.checked_add(1).ok_or_else(|| { + malformed(format!( + "schema-v8 effect '{}' overflows its output version", + pin.table_key + )) + })?; + if effect.planned_transaction.read_version != pin.expected_version + || pin.post_commit_pin != exact_output + || slot.expected_version != pin.expected_version + || slot.table_branch != pin.table_branch + { + return Err(malformed(format!( + "schema-v8 effect '{}' does not match its pin/delta pre-state", + pin.table_key + ))); + } + let is_first_touch = effect.source_fork_version.is_some(); + if (is_first_touch + && !pin + .table_branch + .as_deref() + .is_some_and(|branch| branch != "main" && Some(branch) == sidecar_branch)) + || effect + .source_fork_version + .is_some_and(|version| version != pin.expected_version) + { + return Err(malformed(format!( + "schema-v8 effect '{}' has inconsistent first-touch fork identity", + pin.table_key + ))); + } + match protocol.effect_phase { + RecoveryEffectPhase::Armed => { + if effect.confirmed_transaction.is_some() + || effect.confirmed_branch_identifier.is_some() + || slot.confirmed.is_some() + || pin.confirmed_version.is_some() + { + return Err(malformed(format!( + "Armed schema-v8 effect '{}' already carries confirmation", + pin.table_key + ))); + } + } + RecoveryEffectPhase::EffectsConfirmed => { + let confirmed_transaction = + effect.confirmed_transaction.as_ref().ok_or_else(|| { + malformed(format!( + "EffectsConfirmed schema-v8 effect '{}' lacks transaction confirmation", + pin.table_key + )) + })?; + let confirmed_update = slot.confirmed.as_ref().ok_or_else(|| { + malformed(format!( + "EffectsConfirmed schema-v8 effect '{}' lacks a manifest output", + pin.table_key + )) + })?; + if confirmed_transaction != &effect.planned_transaction + || confirmed_update.table_version != exact_output + || confirmed_update.table_branch != pin.table_branch + || pin.confirmed_version != Some(exact_output) + || is_first_touch != effect.confirmed_branch_identifier.is_some() + { + return Err(malformed(format!( + "schema-v8 effect '{}' confirmation differs from its exact plan", + pin.table_key + ))); + } + } + } + } + Ok(()) +} + +fn validate_schema_apply_v7_shape(sidecar_uri: &str, sidecar: &RecoverySidecar) -> Result<()> { + let malformed = |reason: String| { + OmniError::manifest_internal(format!( + "recovery sidecar at '{}' has an invalid schema-v{} shape: {}", + sidecar_uri, sidecar.schema_version, reason + )) + }; + if !matches!(sidecar.writer_kind, SidecarKind::SchemaApply) { + return Err(malformed(format!( + "schema-v7 is reserved for SchemaApply, found {:?}", + sidecar.writer_kind + ))); + } + if sidecar.branch.is_some() + || sidecar.protocol_v3.is_some() + || sidecar.protocol_v4.is_some() + || sidecar.protocol_v8.is_some() + || sidecar.protocol_v10.is_some() + || sidecar.protocol_v11.is_some() + || sidecar.protocol_v12.is_some() + || sidecar.ensure_indices_rollback_v6.is_some() + || sidecar.merge_source_commit_id.is_some() + || !sidecar.additional_registrations.is_empty() + || !sidecar.tombstones.is_empty() + || sidecar.schema_apply_manifest_published + || sidecar.schema_apply_target_schema_ir_hash.is_some() + { + return Err(malformed( + "schema-v7 SchemaApply must target main and carry authority/delta only in protocol_v7" + .to_string(), + )); + } + let protocol = sidecar + .protocol_v7 + .as_ref() + .ok_or_else(|| malformed("missing required protocol_v7 payload".to_string()))?; + validate_authority_identity(&malformed, &protocol.authority)?; + if protocol.target_schema_ir_hash.is_empty() { + return Err(malformed( + "schema-v7 SchemaApply has an empty target schema identity".to_string(), + )); + } + if sidecar.actor_id != protocol.lineage.actor_id + || protocol.lineage.branch.is_some() + || protocol.lineage.merged_parent_commit_id.is_some() + { + return Err(malformed( + "schema-v7 sidecar actor/main lineage fields do not match".to_string(), + )); + } + if protocol.lineage.graph_commit_id.is_empty() + || protocol.rollback_graph_commit_id.is_empty() + || protocol.rollback_graph_commit_id == protocol.lineage.graph_commit_id + { + return Err(malformed( + "schema-v7 original and rollback commit ids must be non-empty and distinct".to_string(), + )); + } + + validate_unique_pin_identities(&malformed, &sidecar.tables, false)?; + let pin_ids: HashSet = sidecar.tables.iter().map(|pin| pin.identity).collect(); + let rollback_aliases: HashSet<&str> = sidecar + .tables + .iter() + .map(|pin| schema_apply_rollback_table_key(protocol, pin)) + .collect(); + let effect_ids: HashSet = protocol + .effects + .iter() + .map(|effect| effect.identity) + .collect(); + let delta_ids: HashSet = protocol + .intended_delta + .table_updates + .iter() + .map(|slot| slot.identity) + .collect(); + let effect_uuids: HashSet<&str> = protocol + .effects + .iter() + .map(|effect| effect.kind.planned_transaction().uuid.as_str()) + .collect(); + if effect_ids.len() != protocol.effects.len() + || delta_ids.len() != protocol.intended_delta.table_updates.len() + || effect_uuids.len() != protocol.effects.len() + || effect_uuids.iter().any(|uuid| uuid.is_empty()) + || pin_ids != effect_ids + || pin_ids != delta_ids + { + return Err(malformed( + "schema-v7 pins, effects, transaction UUIDs, and delta slots must be unique and one-to-one" + .to_string(), + )); + } + + let registration_ids: HashSet = protocol + .intended_delta + .registrations + .iter() + .map(|registration| registration.identity) + .collect(); + let tombstone_ids: HashSet = protocol + .intended_delta + .tombstones + .iter() + .map(|tombstone| tombstone.identity) + .collect(); + let rename_ids: HashSet = protocol + .intended_delta + .renames + .iter() + .map(|rename| rename.identity) + .collect(); + let first_touch_ids: HashSet = protocol + .effects + .iter() + .filter(|effect| effect.kind.is_first_touch()) + .map(|effect| effect.identity) + .collect(); + if registration_ids.len() != protocol.intended_delta.registrations.len() + || tombstone_ids.len() != protocol.intended_delta.tombstones.len() + || rename_ids.len() != protocol.intended_delta.renames.len() + || registration_ids != first_touch_ids + || !registration_ids.is_disjoint(&tombstone_ids) + || !registration_ids.is_disjoint(&rename_ids) + || !tombstone_ids.is_disjoint(&rename_ids) + { + return Err(malformed( + "SchemaApply first-touch effects must match unique registration identities, while registrations, renames, and tombstones remain identity-disjoint" + .to_string(), )); } for registration in &protocol.intended_delta.registrations { @@ -2934,6 +3572,7 @@ fn validate_branch_merge_v4_shape(sidecar_uri: &str, sidecar: &RecoverySidecar) || sidecar.protocol_v8.is_some() || sidecar.protocol_v10.is_some() || sidecar.protocol_v11.is_some() + || sidecar.protocol_v12.is_some() || sidecar.ensure_indices_rollback_v6.is_some() { return Err(malformed( @@ -3648,6 +4287,7 @@ pub(crate) async fn heal_pending_sidecars_roll_forward( } let _schema_guard = write_queue.acquire(&schema_apply_serial_queue_key()).await; let _branch_guard = write_queue.acquire_branch(sidecar.branch.as_deref()).await; + let _stream_token_guard = write_queue.acquire_stream_token().await; let queue_keys: Vec = sidecar .tables .iter() @@ -3808,6 +4448,7 @@ async fn discard_orphaned_branch_sidecar( actor_id: Some(RECOVERY_ACTOR.to_string()), merged_parent_commit_id: None, created_at: crate::db::now_micros()?, + stream_fold_attribution: None, }; let publisher = GraphNamespacePublisher::new_with_session( root_uri, @@ -3907,6 +4548,7 @@ pub(crate) async fn recover_manifest_drift( // concurrent open/refresh cannot Restore or delete a ref underneath a // live writer from another `Omnigraph` handle. let _branch_guard = write_queue.acquire_branch(sidecar.branch.as_deref()).await; + let _stream_token_guard = write_queue.acquire_stream_token().await; let table_keys = sidecar .tables .iter() @@ -4160,531 +4802,1185 @@ pub(crate) async fn finalize_effect_free_occ_sidecar( { return Ok(false); } - - match cleanup_unpublished_no_effect_forks(root_uri, storage, sidecar, &states).await? { - NoEffectForkCleanup::Complete => {} - NoEffectForkCleanup::DeferredPathChild { .. } => return Ok(false), + + match cleanup_unpublished_no_effect_forks(root_uri, storage, sidecar, &states).await? { + NoEffectForkCleanup::Complete => {} + NoEffectForkCleanup::DeferredPathChild { .. } => return Ok(false), + } + delete_sidecar_by_operation_id(root_uri, storage, &sidecar.operation_id).await?; + Ok(true) +} + +/// A writer has crossed its acknowledgement boundary once the fixed outcome +/// and recovery audit are durable, so its Phase-D cleanup is best-effort. +/// Open/write-entry recovery has no user operation to acknowledge and keeps +/// cleanup required before declaring its sweep complete. +#[derive(Clone, Copy)] +enum StreamEnrollmentCleanup { + Required, + BestEffortAfterVisible, +} + +/// Recover one bounded RFC-026 enrollment intent. +/// +/// This adapter deliberately has no rollback branch. The only destructive +/// action available here is deleting a sidecar after exact classification has +/// proved that neither enrollment effect exists. Once the singleton index is +/// present, recovery either completes the pre-minted empty shard and publishes +/// the fixed lifecycle outcome, or retains the sidecar and fails closed. +async fn process_stream_enrollment_sidecar_v10( + root_uri: &str, + storage: &std::sync::Arc, + snapshot: &Snapshot, + sidecar: &RecoverySidecar, + cleanup_policy: StreamEnrollmentCleanup, +) -> Result { + let protocol = sidecar.protocol_v10.as_ref().ok_or_else(|| { + OmniError::manifest_internal("schema-v10 StreamEnrollment is missing protocol_v10") + })?; + let pin = sidecar.tables.first().ok_or_else(|| { + OmniError::manifest_internal("schema-v10 StreamEnrollment has no table pin") + })?; + let manifest_entry = snapshot_entry_for_pin(snapshot, pin) + .map_err(|error| stream_enrollment_effect_error(sidecar, error))? + .ok_or_else(|| { + OmniError::recovery_required( + sidecar.operation_id.clone(), + format!( + "stream enrollment table identity {} is no longer live in the manifest", + pin.identity + ), + ) + })?; + if manifest_entry.table_path != protocol.intended_binding.table_location + || manifest_entry.table_branch.is_some() + { + return Err(OmniError::recovery_required( + sidecar.operation_id.clone(), + format!( + "stream enrollment manifest authority changed for identity {}: expected path '{}' on main, found path '{}' ref {:?} version {}", + pin.identity, + protocol.intended_binding.table_location, + manifest_entry.table_path, + manifest_entry.table_branch, + manifest_entry.table_version, + ), + )); + } + + let anchor = crate::instrumentation::open_dataset( + &pin.table_path, + crate::instrumentation::VersionResolution::At(pin.expected_version), + None, + crate::instrumentation::table_wrapper(), + ) + .await + .map_err(|error| stream_enrollment_effect_error(sidecar, error))?; + let mut state = classify_enrollment( + &anchor, + &protocol.baseline_head, + &protocol.enrollment_plan, + ) + .await + .map_err(|error| { + OmniError::recovery_required(sidecar.operation_id.clone(), error.to_string()) + })?; + + if matches!(state, MemWalEnrollmentState::ExactNoEffect) { + if snapshot.stream_lifecycle(pin.identity).is_some() + || manifest_entry.table_version != pin.expected_version + { + return Err(OmniError::recovery_required( + sidecar.operation_id.clone(), + "manifest contains stream lifecycle authority but the exact enrollment effects are absent", + )); + } + delete_sidecar_by_operation_id(root_uri, storage.as_ref(), &sidecar.operation_id).await?; + return Ok(true); + } + + // Publication may have succeeded before audit/sidecar cleanup. In that + // state the live graph head is intentionally the fixed lineage id rather + // than the pre-effect authority, so recognize the exact visible outcome + // before comparing against arm-time authority. + if let MemWalEnrollmentState::ExactIndexAndExpectedEmptyShard { + receipt, + shard_manifest, + } = &state + { + let lifecycle = stream_enrollment_lifecycle(sidecar, receipt, shard_manifest) + .map_err(|error| stream_enrollment_effect_error(sidecar, error))?; + if stream_enrollment_original_visible(root_uri, sidecar, &lifecycle) + .await + .map_err(|error| stream_enrollment_effect_error(sidecar, error))? + { + return finalize_visible_stream_enrollment( + root_uri, + storage.as_ref(), + sidecar, + cleanup_policy, + ) + .await + .map_err(|error| stream_enrollment_effect_error(sidecar, error)); + } + } + + let live_authority = read_live_recovery_authority(root_uri, storage, None) + .await + .map_err(|error| stream_enrollment_effect_error(sidecar, error))?; + if live_authority != protocol.authority { + return Err(OmniError::recovery_required( + sidecar.operation_id.clone(), + "stream enrollment graph/schema authority changed after recovery was armed", + )); + } + + if matches!(state, MemWalEnrollmentState::ExactIndexOnly(_)) { + provision_shard_from_exact_index_only( + &anchor, + &protocol.baseline_head, + &protocol.enrollment_plan, + ) + .await + .map_err(|error| { + OmniError::recovery_required(sidecar.operation_id.clone(), error.to_string()) + })?; + state = classify_enrollment( + &anchor, + &protocol.baseline_head, + &protocol.enrollment_plan, + ) + .await + .map_err(|error| { + OmniError::recovery_required(sidecar.operation_id.clone(), error.to_string()) + })?; + } + + let MemWalEnrollmentState::ExactIndexAndExpectedEmptyShard { + receipt, + shard_manifest, + } = state + else { + return Err(OmniError::recovery_required( + sidecar.operation_id.clone(), + "stream enrollment did not converge to the exact index-plus-empty-shard state", + )); + }; + let lifecycle = stream_enrollment_lifecycle(sidecar, &receipt, &shard_manifest) + .map_err(|error| stream_enrollment_effect_error(sidecar, error))?; + + if stream_enrollment_original_visible(root_uri, sidecar, &lifecycle) + .await + .map_err(|error| stream_enrollment_effect_error(sidecar, error))? + { + return finalize_visible_stream_enrollment( + root_uri, + storage.as_ref(), + sidecar, + cleanup_policy, + ) + .await + .map_err(|error| stream_enrollment_effect_error(sidecar, error)); + } + if snapshot.stream_lifecycle(pin.identity).is_some() + || manifest_entry.table_version != pin.expected_version + { + return Err(OmniError::recovery_required( + sidecar.operation_id.clone(), + "a different stream lifecycle row is already visible for the enrollment identity", + )); + } + crate::failpoints::maybe_fail( + crate::failpoints::names::STREAM_ENROLLMENT_POST_SHARD_PRE_MANIFEST, + ) + .map_err(|error| stream_enrollment_effect_error(sidecar, error))?; + + let successor = anchor + .checkout_version(receipt.head.table_version) + .await + .map_err(|error| { + stream_enrollment_effect_error(sidecar, OmniError::Lance(error.to_string())) + })?; + let version_metadata = super::TableVersionMetadata::from_dataset( + root_uri, + &manifest_entry.table_path, + &successor, + ) + .map_err(|error| stream_enrollment_effect_error(sidecar, error))?; + let updates = vec![ + ManifestChange::Update(SubTableUpdate { + identity: pin.identity, + table_key: pin.table_key.clone(), + table_version: receipt.head.table_version, + table_branch: None, + row_count: manifest_entry.row_count, + version_metadata, + }), + ManifestChange::SetStreamLifecycle { + expected: None, + next: lifecycle, + }, + ]; + let expected = HashMap::from([( + pin.identity, + TableVersionExpectation { + table_key: pin.table_key.clone(), + table_version: pin.expected_version, + }, + )]); + let (_, graph_commit_id) = publish_recovery_commit( + root_uri, + sidecar, + RecoveryKind::RolledForward, + &updates, + &expected, + ) + .await + .map_err(|error| stream_enrollment_effect_error(sidecar, error))?; + record_audit( + root_uri, + sidecar, + graph_commit_id, + RecoveryKind::RolledForward, + vec![TableOutcome { + table_key: pin.table_key.clone(), + from_version: pin.expected_version, + to_version: receipt.head.table_version, + }], + ) + .await + .map_err(|error| stream_enrollment_effect_error(sidecar, error))?; + cleanup_visible_stream_enrollment_sidecar(root_uri, storage.as_ref(), sidecar, cleanup_policy) + .await + .map_err(|error| stream_enrollment_effect_error(sidecar, error))?; + Ok(true) +} + +async fn cleanup_visible_stream_enrollment_sidecar( + root_uri: &str, + storage: &dyn StorageAdapter, + sidecar: &RecoverySidecar, + cleanup_policy: StreamEnrollmentCleanup, +) -> Result<()> { + let handle = RecoverySidecarHandle { + operation_id: sidecar.operation_id.clone(), + sidecar_uri: sidecar_uri(root_uri, &sidecar.operation_id), + }; + match delete_sidecar(&handle, storage).await { + Ok(()) => Ok(()), + Err(error) => match cleanup_policy { + StreamEnrollmentCleanup::Required => Err(error), + StreamEnrollmentCleanup::BestEffortAfterVisible => { + tracing::warn!( + error = %error, + operation_id = sidecar.operation_id.as_str(), + "stream enrollment recovery sidecar cleanup failed; the next open will resolve it" + ); + Ok(()) + } + }, + } +} + +/// Once exact classification observes either enrollment effect, every failure +/// is an unresolved durable operation. Preserve an already-typed recovery +/// error and convert lower-level I/O, failpoint, audit, or publication errors +/// so callers never mistake partial format state for an ordinary retry. +fn stream_enrollment_effect_error(sidecar: &RecoverySidecar, error: OmniError) -> OmniError { + match error { + OmniError::RecoveryRequired { .. } => error, + other => OmniError::recovery_required( + sidecar.operation_id.clone(), + format!("stream enrollment completion failed while a durable effect may exist: {other}"), + ), + } +} + +/// Writer-side entry into the same exact recovery adapter used on reopen. +/// Callers retain exclusive stream admission and the normal write gates while +/// this completes; exposing one implementation prevents the success path and +/// crash path from learning different enrollment semantics. +#[allow(dead_code)] // Reached by the intentionally dormant Phase-A orchestrator. +pub(crate) async fn complete_stream_enrollment_sidecar_v10( + root_uri: &str, + storage: std::sync::Arc, + snapshot: &Snapshot, + sidecar: &RecoverySidecar, +) -> Result<()> { + process_stream_enrollment_sidecar_v10( + root_uri, + &storage, + snapshot, + sidecar, + StreamEnrollmentCleanup::BestEffortAfterVisible, + ) + .await + .map(|_| ()) +} + +fn stream_enrollment_lifecycle( + sidecar: &RecoverySidecar, + receipt: &crate::table_store::mem_wal::MemWalEnrollmentReceipt, + shard_manifest: &lance_index::mem_wal::ShardManifest, +) -> Result { + let protocol = sidecar + .protocol_v10 + .as_ref() + .expect("validated StreamEnrollment protocol"); + let pin = &sidecar.tables[0]; + let lifecycle = super::StreamLifecycleEntry::new_open_enrollment( + pin.identity, + pin.table_key.clone(), + protocol.intended_binding.clone(), + receipt.head.clone(), + BTreeMap::from([( + protocol.enrollment_plan.shard_id.to_string(), + shard_manifest.writer_epoch, + )]), + protocol.enrollment_receipt.clone(), + )?; + Ok(lifecycle) +} + +async fn stream_enrollment_original_visible( + root_uri: &str, + sidecar: &RecoverySidecar, + expected_lifecycle: &super::StreamLifecycleEntry, +) -> Result { + let protocol = sidecar + .protocol_v10 + .as_ref() + .expect("validated StreamEnrollment protocol"); + let pin = &sidecar.tables[0]; + let (commits, _) = ManifestCoordinator::read_graph_lineage_at(root_uri, None).await?; + let Some(commit) = commits + .iter() + .find(|commit| commit.graph_commit_id == protocol.lineage.graph_commit_id) + else { + return Ok(false); + }; + if commit.manifest_branch.is_some() + || commit.parent_commit_id.as_ref() != protocol.authority.graph_head.as_ref() + || commit.merged_parent_commit_id.is_some() + || commit.actor_id != protocol.lineage.actor_id + || commit.created_at != protocol.lineage.created_at + { + return Err(OmniError::manifest_internal(format!( + "StreamEnrollment sidecar '{}' found fixed commit '{}' with mismatched lineage", + sidecar.operation_id, protocol.lineage.graph_commit_id + ))); + } + let committed = ManifestCoordinator::snapshot_at(root_uri, None, commit.manifest_version).await?; + let entry_matches = snapshot_entry_by_identity(&committed, pin.identity).is_some_and(|entry| { + entry.table_key == pin.table_key + && entry.table_path == protocol.intended_binding.table_location + && entry.table_version == expected_lifecycle.current_head_witness.table_version + && entry.table_branch.is_none() + }); + if !entry_matches + || committed.stream_lifecycle(pin.identity) != Some(expected_lifecycle) + { + return Err(OmniError::manifest_internal(format!( + "StreamEnrollment sidecar '{}' found fixed commit '{}' but its table/lifecycle outcome differs", + sidecar.operation_id, protocol.lineage.graph_commit_id + ))); + } + Ok(true) +} + +async fn finalize_visible_stream_enrollment( + root_uri: &str, + storage: &dyn StorageAdapter, + sidecar: &RecoverySidecar, + cleanup_policy: StreamEnrollmentCleanup, +) -> Result { + let protocol = sidecar + .protocol_v10 + .as_ref() + .expect("validated StreamEnrollment protocol"); + let pin = &sidecar.tables[0]; + let mut audit = RecoveryAudit::open(root_uri).await?; + let already_recorded = audit.list().await?.iter().any(|record| { + record.operation_id == sidecar.operation_id + && record.recovery_kind == RecoveryKind::RolledForward + }); + if !already_recorded { + crate::failpoints::maybe_fail(crate::failpoints::names::RECOVERY_RECORD_AUDIT)?; + audit + .append(RecoveryAuditRecord { + graph_commit_id: protocol.lineage.graph_commit_id.clone(), + recovery_kind: RecoveryKind::RolledForward, + recovery_for_actor: sidecar.actor_id.clone(), + operation_id: sidecar.operation_id.clone(), + sidecar_writer_kind: format!("{:?}", sidecar.writer_kind), + per_table_outcomes: vec![TableOutcome { + table_key: pin.table_key.clone(), + from_version: pin.expected_version, + to_version: pin.post_commit_pin, + }], + created_at: crate::db::now_micros()?, + }) + .await?; + } + cleanup_visible_stream_enrollment_sidecar(root_uri, storage, sidecar, cleanup_policy).await?; + Ok(true) +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum StreamFoldEffectState { + ExactNoEffect, + ExactEffect, +} + +/// Classify the base-table evidence for one schema-v11 fold without consulting +/// mutable "latest" state a second time. The caller supplies one coherent HEAD +/// witness, its transaction, the transaction's embedded merged-generation +/// list, and the MemWAL index progress read from that same Dataset version. +fn classify_stream_fold_observation( + pin: &SidecarTablePin, + protocol: &RecoveryProtocolV11, + observed_head: &super::CurrentHeadWitness, + observed_transaction: Option<&StagedTransactionIdentity>, + transaction_merged_generations: Option<&[MergedGeneration]>, + observed_merged_generation: Option<&MergedGeneration>, +) -> std::result::Result { + if observed_head.branch_identifier != lance::dataset::refs::BranchIdentifier::main() { + return Err("observed StreamFold table HEAD is not canonical main".to_string()); + } + if observed_head.table_version == pin.expected_version { + if protocol.effect_phase != RecoveryEffectPhase::Armed { + return Err( + "EffectsConfirmed StreamFold has no corresponding base-table movement".to_string(), + ); + } + if observed_head != &protocol.prior_head + || observed_merged_generation != protocol.prior_merged_generation.as_ref() + { + return Err( + "StreamFold numeric no-movement state disagrees with its prior HEAD or merge-progress witness" + .to_string(), + ); + } + return Ok(StreamFoldEffectState::ExactNoEffect); + } + if observed_head.table_version != pin.post_commit_pin { + return Err(format!( + "StreamFold observed table version {}, expected exact N={} or N+1={}", + observed_head.table_version, pin.expected_version, pin.post_commit_pin + )); + } + if observed_transaction != Some(&protocol.planned_transaction) + || observed_head.transaction_uuid != protocol.planned_transaction.uuid + { + return Err( + "StreamFold N+1 transaction identity differs from its pre-minted effect".to_string(), + ); + } + let Some(transaction_merged_generations) = transaction_merged_generations else { + return Err("StreamFold N+1 operation is not a Lance Update transaction".to_string()); + }; + if transaction_merged_generations != std::slice::from_ref(&protocol.merged_generation) { + return Err( + "StreamFold N+1 transaction does not embed exactly its one merged generation" + .to_string(), + ); + } + if observed_merged_generation != Some(&protocol.merged_generation) { + return Err( + "StreamFold N+1 MemWAL merge progress differs from its transaction cut".to_string(), + ); + } + if protocol.effect_phase == RecoveryEffectPhase::EffectsConfirmed { + let exact_confirmation = protocol.confirmed_transaction.as_ref() + == Some(&protocol.planned_transaction) + && protocol.confirmed_merged_generation.as_ref() == Some(&protocol.merged_generation) + && protocol.confirmed_head.as_ref() == Some(observed_head) + && protocol.confirmed_update.as_ref().is_some_and(|update| { + update.table_version == observed_head.table_version && update.table_branch.is_none() + }) + && pin.confirmed_version == Some(observed_head.table_version); + if !exact_confirmation { + return Err( + "StreamFold durable confirmation differs from the exact observed N+1 effect" + .to_string(), + ); + } + } + Ok(StreamFoldEffectState::ExactEffect) +} + +struct ObservedStreamFoldEffect { + state: StreamFoldEffectState, + head: super::CurrentHeadWitness, + exact_update: Option, +} + +async fn observe_stream_fold_effect( + root_uri: &str, + sidecar: &RecoverySidecar, +) -> Result { + let protocol = sidecar.protocol_v11.as_ref().ok_or_else(|| { + OmniError::manifest_internal("schema-v11 StreamFold is missing protocol_v11") + })?; + let pin = sidecar + .tables + .first() + .ok_or_else(|| OmniError::manifest_internal("schema-v11 StreamFold has no table pin"))?; + let dataset = crate::instrumentation::open_dataset( + &pin.table_path, + crate::instrumentation::VersionResolution::Latest, + None, + crate::instrumentation::table_wrapper(), + ) + .await + .map_err(|error| stream_fold_effect_error(sidecar, error))?; + let head = capture_current_head_witness(&dataset) + .await + .map_err(|error| { + stream_fold_effect_error(sidecar, OmniError::manifest_internal(error.to_string())) + })?; + let transaction = dataset + .read_transaction() + .await + .map_err(|error| stream_fold_effect_error(sidecar, OmniError::Lance(error.to_string())))? + .ok_or_else(|| { + stream_fold_effect_error( + sidecar, + OmniError::manifest_internal("StreamFold table HEAD has no transaction"), + ) + })?; + let transaction_identity = StagedTransactionIdentity::from(&transaction); + let transaction_merged_generations = match &transaction.operation { + lance::dataset::transaction::Operation::Update { + merged_generations, .. + } => Some(merged_generations.as_slice()), + _ => None, + }; + let details = dataset + .mem_wal_index_details() + .await + .map_err(|error| stream_fold_effect_error(sidecar, OmniError::Lance(error.to_string())))? + .ok_or_else(|| { + stream_fold_effect_error( + sidecar, + OmniError::manifest_internal( + "StreamFold base table no longer contains the bound MemWAL index", + ), + ) + })?; + validate_stream_config_v3_binding(&details, &protocol.binding).map_err(|error| { + stream_fold_effect_error( + sidecar, + OmniError::manifest_internal(format!( + "StreamFold config-v2 physical validation failed: {error}" + )), + ) + })?; + if details + .merged_generations + .iter() + .any(|progress| progress.shard_id != protocol.generation_cut.shard_id) + { + return Err(stream_fold_effect_error( + sidecar, + OmniError::manifest_internal( + "bounded StreamFold MemWAL index contains merge progress for a foreign shard", + ), + )); + } + let mut matching_progress = details + .merged_generations + .iter() + .filter(|progress| progress.shard_id == protocol.generation_cut.shard_id); + let observed_merged_generation = matching_progress.next(); + if matching_progress.next().is_some() { + return Err(stream_fold_effect_error( + sidecar, + OmniError::manifest_internal( + "StreamFold MemWAL index contains duplicate progress for its bound shard", + ), + )); + } + let state = classify_stream_fold_observation( + pin, + protocol, + &head, + Some(&transaction_identity), + transaction_merged_generations, + observed_merged_generation, + ) + .map_err(|reason| OmniError::recovery_required(sidecar.operation_id.clone(), reason))?; + let exact_update = if state == StreamFoldEffectState::ExactEffect { + let row_count = dataset.count_rows(None).await.map_err(|error| { + stream_fold_effect_error(sidecar, OmniError::Lance(error.to_string())) + })? as u64; + let version_metadata = super::TableVersionMetadata::from_dataset( + root_uri, + &protocol.binding.table_location, + &dataset, + ) + .map_err(|error| stream_fold_effect_error(sidecar, error))?; + Some(RecoveryConfirmedTableUpdate { + table_version: head.table_version, + table_branch: None, + row_count, + version_metadata, + }) + } else { + None + }; + if protocol.effect_phase == RecoveryEffectPhase::EffectsConfirmed + && protocol.confirmed_update.as_ref() != exact_update.as_ref() + { + return Err(OmniError::recovery_required( + sidecar.operation_id.clone(), + "StreamFold confirmed manifest update differs from the exact observed Lance output", + )); + } + Ok(ObservedStreamFoldEffect { + state, + head, + exact_update, + }) +} + +fn stream_fold_lifecycle( + sidecar: &RecoverySidecar, + _head: super::CurrentHeadWitness, + _epoch_floor: u64, +) -> Result { + Err(OmniError::recovery_required( + sidecar.operation_id.clone(), + "historical schema-v11 StreamFold does not carry the complete state-v2 lifecycle authority required by internal schema v9", + )) +} + +fn stream_fold_prior_lifecycle(sidecar: &RecoverySidecar) -> Result { + let protocol = sidecar + .protocol_v11 + .as_ref() + .expect("validated StreamFold protocol"); + stream_fold_lifecycle( + sidecar, + protocol.prior_head.clone(), + protocol.prior_epoch_floor, + ) +} + +fn stream_fold_next_lifecycle(sidecar: &RecoverySidecar) -> Result { + let protocol = sidecar + .protocol_v11 + .as_ref() + .expect("validated StreamFold protocol"); + let head = protocol.confirmed_head.clone().ok_or_else(|| { + OmniError::manifest_internal("confirmed StreamFold has no achieved HEAD witness") + })?; + stream_fold_lifecycle(sidecar, head, protocol.generation_cut.writer_epoch) +} + +fn validate_stream_fold_manifest_prestate( + snapshot: &Snapshot, + sidecar: &RecoverySidecar, +) -> Result<(super::SubTableEntry, super::StreamLifecycleEntry)> { + let protocol = sidecar + .protocol_v11 + .as_ref() + .expect("validated StreamFold protocol"); + let pin = &sidecar.tables[0]; + let entry = snapshot_entry_for_pin(snapshot, pin) + .map_err(|error| stream_fold_effect_error(sidecar, error))? + .cloned() + .ok_or_else(|| { + OmniError::recovery_required( + sidecar.operation_id.clone(), + format!( + "StreamFold table identity {} is no longer live in the manifest", + pin.identity + ), + ) + })?; + let expected_lifecycle = stream_fold_prior_lifecycle(sidecar) + .map_err(|error| stream_fold_effect_error(sidecar, error))?; + if entry.table_path != protocol.binding.table_location + || entry.table_branch.is_some() + || entry.table_version != pin.expected_version + || snapshot.stream_lifecycle(pin.identity) != Some(&expected_lifecycle) + { + return Err(OmniError::recovery_required( + sidecar.operation_id.clone(), + "StreamFold manifest pointer or exact prior lifecycle authority changed after recovery was armed", + )); + } + Ok((entry, expected_lifecycle)) +} + +#[derive(Clone, Copy)] +enum StreamFoldCleanup { + Required, + #[allow(dead_code)] + BestEffortAfterVisible, +} + +async fn cleanup_visible_stream_fold_sidecar( + root_uri: &str, + storage: &dyn StorageAdapter, + sidecar: &RecoverySidecar, + cleanup_policy: StreamFoldCleanup, +) -> Result<()> { + let handle = RecoverySidecarHandle { + operation_id: sidecar.operation_id.clone(), + sidecar_uri: sidecar_uri(root_uri, &sidecar.operation_id), + }; + match delete_sidecar(&handle, storage).await { + Ok(()) => Ok(()), + Err(error) => match cleanup_policy { + StreamFoldCleanup::Required => Err(error), + StreamFoldCleanup::BestEffortAfterVisible => { + tracing::warn!( + error = %error, + operation_id = sidecar.operation_id.as_str(), + "StreamFold recovery sidecar cleanup failed; the next recovery barrier will resolve it" + ); + Ok(()) + } + }, + } +} + +fn stream_fold_effect_error(sidecar: &RecoverySidecar, error: OmniError) -> OmniError { + match error { + OmniError::RecoveryRequired { .. } => error, + other => OmniError::recovery_required( + sidecar.operation_id.clone(), + format!("StreamFold completion failed while a durable effect may exist: {other}"), + ), + } +} + +async fn stream_fold_original_visible(root_uri: &str, sidecar: &RecoverySidecar) -> Result { + let protocol = sidecar + .protocol_v11 + .as_ref() + .expect("validated StreamFold protocol"); + if protocol.effect_phase != RecoveryEffectPhase::EffectsConfirmed { + return Ok(false); + } + let pin = &sidecar.tables[0]; + let update = protocol + .confirmed_update + .as_ref() + .expect("validated confirmed StreamFold update"); + let expected_lifecycle = stream_fold_next_lifecycle(sidecar)?; + let (commits, _) = ManifestCoordinator::read_graph_lineage_at(root_uri, None).await?; + let Some(commit) = commits + .iter() + .find(|commit| commit.graph_commit_id == protocol.lineage.graph_commit_id) + else { + return Ok(false); + }; + if commit.manifest_branch.is_some() + || commit.parent_commit_id.as_ref() != protocol.authority.graph_head.as_ref() + || commit.merged_parent_commit_id.is_some() + || commit.actor_id != protocol.lineage.actor_id + || commit.created_at != protocol.lineage.created_at + { + return Err(OmniError::manifest_internal(format!( + "StreamFold sidecar '{}' found fixed commit '{}' with mismatched lineage", + sidecar.operation_id, protocol.lineage.graph_commit_id + ))); + } + let committed = + ManifestCoordinator::snapshot_at(root_uri, None, commit.manifest_version).await?; + let entry_matches = snapshot_entry_by_identity(&committed, pin.identity).is_some_and(|entry| { + entry.table_key == pin.table_key + && entry.table_path == protocol.binding.table_location + && entry.table_version == update.table_version + && entry.table_branch.is_none() + && entry.row_count == update.row_count + && entry.version_metadata == update.version_metadata + }); + if !entry_matches || committed.stream_lifecycle(pin.identity) != Some(&expected_lifecycle) { + return Err(OmniError::manifest_internal(format!( + "StreamFold sidecar '{}' found fixed commit '{}' but its table/lifecycle outcome differs", + sidecar.operation_id, protocol.lineage.graph_commit_id + ))); + } + Ok(true) +} + +async fn finalize_visible_stream_fold( + root_uri: &str, + storage: &dyn StorageAdapter, + sidecar: &RecoverySidecar, + cleanup_policy: StreamFoldCleanup, +) -> Result { + let protocol = sidecar + .protocol_v11 + .as_ref() + .expect("validated StreamFold protocol"); + let pin = &sidecar.tables[0]; + let update = protocol + .confirmed_update + .as_ref() + .expect("validated confirmed StreamFold update"); + let mut audit = RecoveryAudit::open(root_uri).await?; + let expected_outcomes = vec![TableOutcome { + table_key: pin.table_key.clone(), + from_version: pin.expected_version, + to_version: update.table_version, + }]; + let expected_writer_kind = format!("{:?}", sidecar.writer_kind); + let records = audit.list().await?; + let operation_records: Vec<_> = records + .iter() + .filter(|record| record.operation_id == sidecar.operation_id) + .collect(); + if operation_records.iter().any(|record| { + record.graph_commit_id != protocol.lineage.graph_commit_id + || record.recovery_kind != RecoveryKind::RolledForward + || record.recovery_for_actor != sidecar.actor_id + || record.sidecar_writer_kind != expected_writer_kind + || record.per_table_outcomes != expected_outcomes + }) { + return Err(OmniError::recovery_required( + sidecar.operation_id.clone(), + "StreamFold recovery audit row differs from the exact fixed outcome", + )); + } + let already_recorded = !operation_records.is_empty(); + if !already_recorded { + crate::failpoints::maybe_fail(crate::failpoints::names::RECOVERY_RECORD_AUDIT)?; + audit + .append(RecoveryAuditRecord { + graph_commit_id: protocol.lineage.graph_commit_id.clone(), + recovery_kind: RecoveryKind::RolledForward, + recovery_for_actor: sidecar.actor_id.clone(), + operation_id: sidecar.operation_id.clone(), + sidecar_writer_kind: expected_writer_kind, + per_table_outcomes: expected_outcomes, + created_at: crate::db::now_micros()?, + }) + .await?; } - delete_sidecar_by_operation_id(root_uri, storage, &sidecar.operation_id).await?; + cleanup_visible_stream_fold_sidecar(root_uri, storage, sidecar, cleanup_policy).await?; Ok(true) } -/// A writer has crossed its acknowledgement boundary once the fixed outcome -/// and recovery audit are durable, so its Phase-D cleanup is best-effort. -/// Open/write-entry recovery has no user operation to acknowledge and keeps -/// cleanup required before declaring its sweep complete. -#[derive(Clone, Copy)] -enum StreamEnrollmentCleanup { - Required, - BestEffortAfterVisible, -} - -/// Recover one bounded RFC-026 enrollment intent. -/// -/// This adapter deliberately has no rollback branch. The only destructive -/// action available here is deleting a sidecar after exact classification has -/// proved that neither enrollment effect exists. Once the singleton index is -/// present, recovery either completes the pre-minted empty shard and publishes -/// the fixed lifecycle outcome, or retains the sidecar and fails closed. -async fn process_stream_enrollment_sidecar_v10( +async fn process_stream_fold_sidecar_v11( root_uri: &str, storage: &std::sync::Arc, snapshot: &Snapshot, sidecar: &RecoverySidecar, - cleanup_policy: StreamEnrollmentCleanup, + cleanup_policy: StreamFoldCleanup, ) -> Result { - let protocol = sidecar.protocol_v10.as_ref().ok_or_else(|| { - OmniError::manifest_internal("schema-v10 StreamEnrollment is missing protocol_v10") - })?; - let pin = sidecar.tables.first().ok_or_else(|| { - OmniError::manifest_internal("schema-v10 StreamEnrollment has no table pin") - })?; - let manifest_entry = snapshot_entry_for_pin(snapshot, pin) - .map_err(|error| stream_enrollment_effect_error(sidecar, error))? - .ok_or_else(|| { - OmniError::recovery_required( - sidecar.operation_id.clone(), - format!( - "stream enrollment table identity {} is no longer live in the manifest", - pin.identity - ), - ) - })?; - if manifest_entry.table_path != protocol.intended_binding.table_location - || manifest_entry.table_branch.is_some() - { - return Err(OmniError::recovery_required( - sidecar.operation_id.clone(), - format!( - "stream enrollment manifest authority changed for identity {}: expected path '{}' on main, found path '{}' ref {:?} version {}", - pin.identity, - protocol.intended_binding.table_location, - manifest_entry.table_path, - manifest_entry.table_branch, - manifest_entry.table_version, - ), - )); - } - - let anchor = crate::instrumentation::open_dataset( - &pin.table_path, - crate::instrumentation::VersionResolution::At(pin.expected_version), - None, - crate::instrumentation::table_wrapper(), - ) - .await - .map_err(|error| stream_enrollment_effect_error(sidecar, error))?; - let mut state = classify_enrollment( - &anchor, - &protocol.baseline_head, - &protocol.enrollment_plan, - ) - .await - .map_err(|error| { - OmniError::recovery_required(sidecar.operation_id.clone(), error.to_string()) - })?; + let observed = observe_stream_fold_effect(root_uri, sidecar).await?; - if matches!(state, MemWalEnrollmentState::ExactNoEffect) { - if snapshot.stream_lifecycle(pin.identity).is_some() - || manifest_entry.table_version != pin.expected_version - { + if stream_fold_original_visible(root_uri, sidecar) + .await + .map_err(|error| stream_fold_effect_error(sidecar, error))? + { + if observed.state != StreamFoldEffectState::ExactEffect { return Err(OmniError::recovery_required( sidecar.operation_id.clone(), - "manifest contains stream lifecycle authority but the exact enrollment effects are absent", + "fixed StreamFold graph commit is visible without its exact base-table effect", )); } - delete_sidecar_by_operation_id(root_uri, storage.as_ref(), &sidecar.operation_id).await?; - return Ok(true); - } - - // Publication may have succeeded before audit/sidecar cleanup. In that - // state the live graph head is intentionally the fixed lineage id rather - // than the pre-effect authority, so recognize the exact visible outcome - // before comparing against arm-time authority. - if let MemWalEnrollmentState::ExactIndexAndExpectedEmptyShard { - receipt, - shard_manifest, - } = &state - { - let lifecycle = stream_enrollment_lifecycle(sidecar, receipt, shard_manifest) - .map_err(|error| stream_enrollment_effect_error(sidecar, error))?; - if stream_enrollment_original_visible(root_uri, sidecar, &lifecycle) - .await - .map_err(|error| stream_enrollment_effect_error(sidecar, error))? - { - return finalize_visible_stream_enrollment( - root_uri, - storage.as_ref(), - sidecar, - cleanup_policy, - ) + return finalize_visible_stream_fold(root_uri, storage.as_ref(), sidecar, cleanup_policy) .await - .map_err(|error| stream_enrollment_effect_error(sidecar, error)); - } + .map_err(|error| stream_fold_effect_error(sidecar, error)); } + let (manifest_entry, prior_lifecycle) = + validate_stream_fold_manifest_prestate(snapshot, sidecar)?; + let protocol = sidecar + .protocol_v11 + .as_ref() + .expect("validated StreamFold protocol"); let live_authority = read_live_recovery_authority(root_uri, storage, None) .await - .map_err(|error| stream_enrollment_effect_error(sidecar, error))?; + .map_err(|error| stream_fold_effect_error(sidecar, error))?; if live_authority != protocol.authority { return Err(OmniError::recovery_required( sidecar.operation_id.clone(), - "stream enrollment graph/schema authority changed after recovery was armed", + "StreamFold graph/schema authority changed after recovery was armed", )); } - if matches!(state, MemWalEnrollmentState::ExactIndexOnly(_)) { - provision_shard_from_exact_index_only( - &anchor, - &protocol.baseline_head, - &protocol.enrollment_plan, - ) - .await - .map_err(|error| { - OmniError::recovery_required(sidecar.operation_id.clone(), error.to_string()) - })?; - state = classify_enrollment( - &anchor, - &protocol.baseline_head, - &protocol.enrollment_plan, - ) - .await - .map_err(|error| { - OmniError::recovery_required(sidecar.operation_id.clone(), error.to_string()) - })?; + if observed.state == StreamFoldEffectState::ExactNoEffect { + delete_sidecar_by_operation_id(root_uri, storage.as_ref(), &sidecar.operation_id).await?; + return Ok(true); } - let MemWalEnrollmentState::ExactIndexAndExpectedEmptyShard { - receipt, - shard_manifest, - } = state - else { - return Err(OmniError::recovery_required( - sidecar.operation_id.clone(), - "stream enrollment did not converge to the exact index-plus-empty-shard state", - )); - }; - let lifecycle = stream_enrollment_lifecycle(sidecar, &receipt, &shard_manifest) - .map_err(|error| stream_enrollment_effect_error(sidecar, error))?; - - if stream_enrollment_original_visible(root_uri, sidecar, &lifecycle) - .await - .map_err(|error| stream_enrollment_effect_error(sidecar, error))? - { - return finalize_visible_stream_enrollment( + let mut confirmed = sidecar.clone(); + if protocol.effect_phase == RecoveryEffectPhase::Armed { + let exact_update = observed + .exact_update + .as_ref() + .expect("exact StreamFold effect has a reconstructed update"); + confirm_stream_fold_sidecar_v11( root_uri, storage.as_ref(), - sidecar, - cleanup_policy, + &mut confirmed, + protocol.planned_transaction.clone(), + protocol.merged_generation.clone(), + observed.head.clone(), + SubTableUpdate { + identity: pin_identity(sidecar), + table_key: sidecar.tables[0].table_key.clone(), + table_version: observed.head.table_version, + table_branch: None, + row_count: exact_update.row_count, + version_metadata: exact_update.version_metadata.clone(), + }, ) .await - .map_err(|error| stream_enrollment_effect_error(sidecar, error)); - } - if snapshot.stream_lifecycle(pin.identity).is_some() - || manifest_entry.table_version != pin.expected_version - { - return Err(OmniError::recovery_required( - sidecar.operation_id.clone(), - "a different stream lifecycle row is already visible for the enrollment identity", - )); + .map_err(|error| stream_fold_effect_error(sidecar, error))?; } - crate::failpoints::maybe_fail( - crate::failpoints::names::STREAM_ENROLLMENT_POST_SHARD_PRE_MANIFEST, - ) - .map_err(|error| stream_enrollment_effect_error(sidecar, error))?; - - let successor = anchor - .checkout_version(receipt.head.table_version) - .await - .map_err(|error| { - stream_enrollment_effect_error(sidecar, OmniError::Lance(error.to_string())) - })?; - let version_metadata = super::TableVersionMetadata::from_dataset( - root_uri, - &manifest_entry.table_path, - &successor, - ) - .map_err(|error| stream_enrollment_effect_error(sidecar, error))?; - let updates = vec![ - ManifestChange::Update(SubTableUpdate { - identity: pin.identity, - table_key: pin.table_key.clone(), - table_version: receipt.head.table_version, - table_branch: None, - row_count: manifest_entry.row_count, - version_metadata, + let confirmed_protocol = confirmed + .protocol_v11 + .as_ref() + .expect("confirmed StreamFold protocol"); + let confirmed_update = confirmed_protocol + .confirmed_update + .as_ref() + .expect("confirmed StreamFold update"); + let next_lifecycle = stream_fold_next_lifecycle(&confirmed) + .map_err(|error| stream_fold_effect_error(&confirmed, error))?; + let updates = vec![ + ManifestChange::Update(SubTableUpdate { + identity: pin_identity(&confirmed), + table_key: confirmed.tables[0].table_key.clone(), + table_version: confirmed_update.table_version, + table_branch: confirmed_update.table_branch.clone(), + row_count: confirmed_update.row_count, + version_metadata: confirmed_update.version_metadata.clone(), }), ManifestChange::SetStreamLifecycle { - expected: None, - next: lifecycle, + expected: Some(prior_lifecycle), + next: next_lifecycle, }, ]; let expected = HashMap::from([( - pin.identity, + pin_identity(&confirmed), TableVersionExpectation { - table_key: pin.table_key.clone(), - table_version: pin.expected_version, + table_key: confirmed.tables[0].table_key.clone(), + table_version: manifest_entry.table_version, }, )]); + crate::failpoints::maybe_fail(crate::failpoints::names::RECOVERY_BEFORE_ROLL_FORWARD_PUBLISH)?; let (_, graph_commit_id) = publish_recovery_commit( root_uri, - sidecar, + &confirmed, RecoveryKind::RolledForward, &updates, &expected, ) .await - .map_err(|error| stream_enrollment_effect_error(sidecar, error))?; + .map_err(|error| stream_fold_effect_error(&confirmed, error))?; record_audit( root_uri, - sidecar, + &confirmed, graph_commit_id, RecoveryKind::RolledForward, vec![TableOutcome { - table_key: pin.table_key.clone(), - from_version: pin.expected_version, - to_version: receipt.head.table_version, + table_key: confirmed.tables[0].table_key.clone(), + from_version: confirmed.tables[0].expected_version, + to_version: confirmed_update.table_version, }], ) .await - .map_err(|error| stream_enrollment_effect_error(sidecar, error))?; - cleanup_visible_stream_enrollment_sidecar(root_uri, storage.as_ref(), sidecar, cleanup_policy) + .map_err(|error| stream_fold_effect_error(&confirmed, error))?; + cleanup_visible_stream_fold_sidecar(root_uri, storage.as_ref(), &confirmed, cleanup_policy) .await - .map_err(|error| stream_enrollment_effect_error(sidecar, error))?; + .map_err(|error| stream_fold_effect_error(&confirmed, error))?; Ok(true) } -async fn cleanup_visible_stream_enrollment_sidecar( +fn pin_identity(sidecar: &RecoverySidecar) -> TableIdentity { + sidecar.tables[0].identity +} + +/// Retire one Armed StreamFold only after exact N/no-merge-progress evidence. +/// An exact effect, foreign movement, or unreadable evidence leaves the sidecar +/// in place so the caller returns `RecoveryRequired` rather than replanning +/// around an unresolved fold. +#[allow(dead_code)] +pub(crate) async fn finalize_effect_free_stream_fold_sidecar_v11( root_uri: &str, - storage: &dyn StorageAdapter, + storage: &std::sync::Arc, + snapshot: &Snapshot, sidecar: &RecoverySidecar, - cleanup_policy: StreamEnrollmentCleanup, -) -> Result<()> { - let handle = RecoverySidecarHandle { - operation_id: sidecar.operation_id.clone(), - sidecar_uri: sidecar_uri(root_uri, &sidecar.operation_id), - }; - match delete_sidecar(&handle, storage).await { - Ok(()) => Ok(()), - Err(error) => match cleanup_policy { - StreamEnrollmentCleanup::Required => Err(error), - StreamEnrollmentCleanup::BestEffortAfterVisible => { - tracing::warn!( - error = %error, - operation_id = sidecar.operation_id.as_str(), - "stream enrollment recovery sidecar cleanup failed; the next open will resolve it" - ); - Ok(()) - } - }, +) -> Result { + validate_sidecar_shape(&sidecar_uri(root_uri, &sidecar.operation_id), sidecar)?; + let protocol = sidecar.protocol_v11.as_ref().ok_or_else(|| { + OmniError::manifest_internal( + "effect-free StreamFold finalization requires a schema-v11 payload", + ) + })?; + if protocol.effect_phase != RecoveryEffectPhase::Armed { + return Ok(false); } -} - -/// Once exact classification observes either enrollment effect, every failure -/// is an unresolved durable operation. Preserve an already-typed recovery -/// error and convert lower-level I/O, failpoint, audit, or publication errors -/// so callers never mistake partial format state for an ordinary retry. -fn stream_enrollment_effect_error(sidecar: &RecoverySidecar, error: OmniError) -> OmniError { - match error { - OmniError::RecoveryRequired { .. } => error, - other => OmniError::recovery_required( + validate_stream_fold_manifest_prestate(snapshot, sidecar)?; + let live_authority = read_live_recovery_authority(root_uri, storage, None) + .await + .map_err(|error| stream_fold_effect_error(sidecar, error))?; + if live_authority != protocol.authority { + return Err(OmniError::recovery_required( sidecar.operation_id.clone(), - format!("stream enrollment completion failed while a durable effect may exist: {other}"), - ), + "StreamFold graph/schema authority changed before effect-free finalization", + )); + } + let observed = observe_stream_fold_effect(root_uri, sidecar).await?; + if observed.state != StreamFoldEffectState::ExactNoEffect { + return Ok(false); } + delete_sidecar_by_operation_id(root_uri, storage.as_ref(), &sidecar.operation_id).await?; + Ok(true) } -/// Writer-side entry into the same exact recovery adapter used on reopen. -/// Callers retain exclusive stream admission and the normal write gates while -/// this completes; exposing one implementation prevents the success path and -/// crash path from learning different enrollment semantics. -#[allow(dead_code)] // Reached by the intentionally dormant Phase-A orchestrator. -pub(crate) async fn complete_stream_enrollment_sidecar_v10( +/// Writer-side entry into the same exact roll-forward adapter used by recovery. +/// Callers retain exclusive stream admission and normal write gates while this +/// completes; cleanup after a graph-visible outcome is best effort because the +/// fixed lineage makes re-entry idempotent. +#[allow(dead_code)] +pub(crate) async fn complete_stream_fold_sidecar_v11( root_uri: &str, storage: std::sync::Arc, snapshot: &Snapshot, sidecar: &RecoverySidecar, ) -> Result<()> { - process_stream_enrollment_sidecar_v10( + process_stream_fold_sidecar_v11( root_uri, &storage, snapshot, sidecar, - StreamEnrollmentCleanup::BestEffortAfterVisible, + StreamFoldCleanup::BestEffortAfterVisible, ) .await .map(|_| ()) } -fn stream_enrollment_lifecycle( - sidecar: &RecoverySidecar, - receipt: &crate::table_store::mem_wal::MemWalEnrollmentReceipt, - shard_manifest: &lance_index::mem_wal::ShardManifest, -) -> Result { - let protocol = sidecar - .protocol_v10 - .as_ref() - .expect("validated StreamEnrollment protocol"); - let pin = &sidecar.tables[0]; - let lifecycle = super::StreamLifecycleEntry { - identity: pin.identity, - diagnostic_table_key: pin.table_key.clone(), - lifecycle: super::StreamLifecycle::Open, - binding: protocol.intended_binding.clone(), - current_head_witness: receipt.head.clone(), - epoch_floor_by_shard: BTreeMap::from([( - protocol.enrollment_plan.shard_id.to_string(), - shard_manifest.writer_epoch, - )]), - }; - lifecycle.validate()?; - Ok(lifecycle) +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum StreamFoldParticipantState { + ExactNoEffect, + ExactEffect, } -async fn stream_enrollment_original_visible( - root_uri: &str, - sidecar: &RecoverySidecar, - expected_lifecycle: &super::StreamLifecycleEntry, -) -> Result { - let protocol = sidecar - .protocol_v10 - .as_ref() - .expect("validated StreamEnrollment protocol"); - let pin = &sidecar.tables[0]; - let (commits, _) = ManifestCoordinator::read_graph_lineage_at(root_uri, None).await?; - let Some(commit) = commits - .iter() - .find(|commit| commit.graph_commit_id == protocol.lineage.graph_commit_id) - else { - return Ok(false); - }; - if commit.manifest_branch.is_some() - || commit.parent_commit_id.as_ref() != protocol.authority.graph_head.as_ref() - || commit.merged_parent_commit_id.is_some() - || commit.actor_id != protocol.lineage.actor_id - || commit.created_at != protocol.lineage.created_at - { - return Err(OmniError::manifest_internal(format!( - "StreamEnrollment sidecar '{}' found fixed commit '{}' with mismatched lineage", - sidecar.operation_id, protocol.lineage.graph_commit_id - ))); - } - let committed = ManifestCoordinator::snapshot_at(root_uri, None, commit.manifest_version).await?; - let entry_matches = snapshot_entry_by_identity(&committed, pin.identity).is_some_and(|entry| { - entry.table_key == pin.table_key - && entry.table_path == protocol.intended_binding.table_location - && entry.table_version == expected_lifecycle.current_head_witness.table_version - && entry.table_branch.is_none() - }); - if !entry_matches - || committed.stream_lifecycle(pin.identity) != Some(expected_lifecycle) - { - return Err(OmniError::manifest_internal(format!( - "StreamEnrollment sidecar '{}' found fixed commit '{}' but its table/lifecycle outcome differs", - sidecar.operation_id, protocol.lineage.graph_commit_id - ))); - } - Ok(true) +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum StreamFoldV12EffectMatrix { + EffectFree, + Complete, + Partial, } -async fn finalize_visible_stream_enrollment( - root_uri: &str, - storage: &dyn StorageAdapter, - sidecar: &RecoverySidecar, - cleanup_policy: StreamEnrollmentCleanup, -) -> Result { - let protocol = sidecar - .protocol_v10 - .as_ref() - .expect("validated StreamEnrollment protocol"); - let pin = &sidecar.tables[0]; - let mut audit = RecoveryAudit::open(root_uri).await?; - let already_recorded = audit.list().await?.iter().any(|record| { - record.operation_id == sidecar.operation_id - && record.recovery_kind == RecoveryKind::RolledForward - }); - if !already_recorded { - crate::failpoints::maybe_fail(crate::failpoints::names::RECOVERY_RECORD_AUDIT)?; - audit - .append(RecoveryAuditRecord { - graph_commit_id: protocol.lineage.graph_commit_id.clone(), - recovery_kind: RecoveryKind::RolledForward, - recovery_for_actor: sidecar.actor_id.clone(), - operation_id: sidecar.operation_id.clone(), - sidecar_writer_kind: format!("{:?}", sidecar.writer_kind), - per_table_outcomes: vec![TableOutcome { - table_key: pin.table_key.clone(), - from_version: pin.expected_version, - to_version: pin.post_commit_pin, - }], - created_at: crate::db::now_micros()?, - }) - .await?; +fn classify_stream_fold_v12_effect_matrix( + base: StreamFoldParticipantState, + token: StreamFoldParticipantState, +) -> StreamFoldV12EffectMatrix { + match (base, token) { + (StreamFoldParticipantState::ExactNoEffect, StreamFoldParticipantState::ExactNoEffect) => { + StreamFoldV12EffectMatrix::EffectFree + } + (StreamFoldParticipantState::ExactEffect, StreamFoldParticipantState::ExactEffect) => { + StreamFoldV12EffectMatrix::Complete + } + _ => StreamFoldV12EffectMatrix::Partial, } - cleanup_visible_stream_enrollment_sidecar(root_uri, storage, sidecar, cleanup_policy).await?; - Ok(true) } -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -enum StreamFoldEffectState { - ExactNoEffect, - ExactEffect, +struct ObservedStreamFoldBaseV12 { + state: StreamFoldParticipantState, + head: super::CurrentHeadWitness, + transaction: StagedTransactionIdentity, + exact_update: Option, } -/// Classify the base-table evidence for one schema-v11 fold without consulting -/// mutable "latest" state a second time. The caller supplies one coherent HEAD -/// witness, its transaction, the transaction's embedded merged-generation -/// list, and the MemWAL index progress read from that same Dataset version. -fn classify_stream_fold_observation( +fn classify_stream_fold_base_v12_observation( pin: &SidecarTablePin, - protocol: &RecoveryProtocolV11, + protocol: &RecoveryProtocolV12, observed_head: &super::CurrentHeadWitness, - observed_transaction: Option<&StagedTransactionIdentity>, + observed_transaction: &StagedTransactionIdentity, transaction_merged_generations: Option<&[MergedGeneration]>, observed_merged_generation: Option<&MergedGeneration>, -) -> std::result::Result { +) -> std::result::Result { if observed_head.branch_identifier != lance::dataset::refs::BranchIdentifier::main() { - return Err("observed StreamFold table HEAD is not canonical main".to_string()); + return Err("observed StreamFold base-table HEAD is not canonical main".to_string()); } if observed_head.table_version == pin.expected_version { - if protocol.effect_phase != RecoveryEffectPhase::Armed { - return Err( - "EffectsConfirmed StreamFold has no corresponding base-table movement".to_string(), - ); - } - if observed_head != &protocol.prior_head + if observed_head != &protocol.prior_lifecycle.current_head_witness || observed_merged_generation != protocol.prior_merged_generation.as_ref() { return Err( - "StreamFold numeric no-movement state disagrees with its prior HEAD or merge-progress witness" + "StreamFold numeric base no-movement state disagrees with its complete prior lifecycle or merge-progress witness" .to_string(), ); } - return Ok(StreamFoldEffectState::ExactNoEffect); + return Ok(StreamFoldParticipantState::ExactNoEffect); } if observed_head.table_version != pin.post_commit_pin { return Err(format!( - "StreamFold observed table version {}, expected exact N={} or N+1={}", + "StreamFold observed base-table version {}, expected exact N={} or N+1={}", observed_head.table_version, pin.expected_version, pin.post_commit_pin )); } - if observed_transaction != Some(&protocol.planned_transaction) - || observed_head.transaction_uuid != protocol.planned_transaction.uuid + if observed_transaction != &protocol.base.planned_transaction + || observed_head.transaction_uuid != protocol.base.planned_transaction.uuid { return Err( - "StreamFold N+1 transaction identity differs from its pre-minted effect".to_string(), + "StreamFold base N+1 transaction identity differs from its pre-minted effect" + .to_string(), ); } let Some(transaction_merged_generations) = transaction_merged_generations else { - return Err("StreamFold N+1 operation is not a Lance Update transaction".to_string()); + return Err("StreamFold base N+1 operation is not a Lance Update transaction".to_string()); }; if transaction_merged_generations != std::slice::from_ref(&protocol.merged_generation) { return Err( - "StreamFold N+1 transaction does not embed exactly its one merged generation" + "StreamFold base N+1 transaction does not embed exactly its one merged generation" .to_string(), ); } if observed_merged_generation != Some(&protocol.merged_generation) { return Err( - "StreamFold N+1 MemWAL merge progress differs from its transaction cut".to_string(), + "StreamFold base N+1 MemWAL merge progress differs from its transaction cut" + .to_string(), ); } if protocol.effect_phase == RecoveryEffectPhase::EffectsConfirmed { - let exact_confirmation = protocol.confirmed_transaction.as_ref() - == Some(&protocol.planned_transaction) - && protocol.confirmed_merged_generation.as_ref() == Some(&protocol.merged_generation) - && protocol.confirmed_head.as_ref() == Some(observed_head) - && protocol.confirmed_update.as_ref().is_some_and(|update| { - update.table_version == observed_head.table_version && update.table_branch.is_none() - }) + let exact_confirmation = protocol.base.confirmed_transaction.as_ref() + == Some(&protocol.base.planned_transaction) + && protocol.base.confirmed_merged_generation.as_ref() + == Some(&protocol.merged_generation) + && protocol.base.confirmed_head.as_ref() == Some(observed_head) + && protocol + .base + .confirmed_update + .as_ref() + .is_some_and(|update| { + update.table_version == observed_head.table_version + && update.table_branch.is_none() + }) && pin.confirmed_version == Some(observed_head.table_version); if !exact_confirmation { return Err( - "StreamFold durable confirmation differs from the exact observed N+1 effect" + "StreamFold durable base confirmation differs from the exact observed N+1 effect" .to_string(), ); } } - Ok(StreamFoldEffectState::ExactEffect) -} - -struct ObservedStreamFoldEffect { - state: StreamFoldEffectState, - head: super::CurrentHeadWitness, - exact_update: Option, + Ok(StreamFoldParticipantState::ExactEffect) } -async fn observe_stream_fold_effect( +async fn observe_stream_fold_base_v12( root_uri: &str, sidecar: &RecoverySidecar, -) -> Result { - let protocol = sidecar.protocol_v11.as_ref().ok_or_else(|| { - OmniError::manifest_internal("schema-v11 StreamFold is missing protocol_v11") +) -> Result { + let protocol = sidecar.protocol_v12.as_ref().ok_or_else(|| { + OmniError::manifest_internal("schema-v12 StreamFold is missing protocol_v12") + })?; + let pin = sidecar.tables.first().ok_or_else(|| { + OmniError::manifest_internal("schema-v12 StreamFold has no base-table pin") })?; - let pin = sidecar - .tables - .first() - .ok_or_else(|| OmniError::manifest_internal("schema-v11 StreamFold has no table pin"))?; let dataset = crate::instrumentation::open_dataset( &pin.table_path, crate::instrumentation::VersionResolution::Latest, @@ -4705,7 +6001,7 @@ async fn observe_stream_fold_effect( .ok_or_else(|| { stream_fold_effect_error( sidecar, - OmniError::manifest_internal("StreamFold table HEAD has no transaction"), + OmniError::manifest_internal("StreamFold base-table HEAD has no transaction"), ) })?; let transaction_identity = StagedTransactionIdentity::from(&transaction); @@ -4727,11 +6023,11 @@ async fn observe_stream_fold_effect( ), ) })?; - validate_stream_config_v2_binding(&details, &protocol.binding).map_err(|error| { + validate_stream_config_v3_binding(&details, &protocol.binding).map_err(|error| { stream_fold_effect_error( sidecar, OmniError::manifest_internal(format!( - "StreamFold config-v2 physical validation failed: {error}" + "StreamFold config-v3 physical validation failed: {error}" )), ) })?; @@ -4760,16 +6056,16 @@ async fn observe_stream_fold_effect( ), )); } - let state = classify_stream_fold_observation( + let state = classify_stream_fold_base_v12_observation( pin, protocol, &head, - Some(&transaction_identity), + &transaction_identity, transaction_merged_generations, observed_merged_generation, ) .map_err(|reason| OmniError::recovery_required(sidecar.operation_id.clone(), reason))?; - let exact_update = if state == StreamFoldEffectState::ExactEffect { + let exact_update = if state == StreamFoldParticipantState::ExactEffect { let row_count = dataset.count_rows(None).await.map_err(|error| { stream_fold_effect_error(sidecar, OmniError::Lance(error.to_string())) })? as u64; @@ -4789,161 +6085,455 @@ async fn observe_stream_fold_effect( None }; if protocol.effect_phase == RecoveryEffectPhase::EffectsConfirmed - && protocol.confirmed_update.as_ref() != exact_update.as_ref() + && protocol.base.confirmed_update.as_ref() != exact_update.as_ref() { return Err(OmniError::recovery_required( sidecar.operation_id.clone(), - "StreamFold confirmed manifest update differs from the exact observed Lance output", + format!( + "StreamFold confirmed base manifest update differs from the exact observed Lance output: confirmed={:?}, observed={exact_update:?}", + protocol.base.confirmed_update + ), )); } - Ok(ObservedStreamFoldEffect { + Ok(ObservedStreamFoldBaseV12 { state, head, + transaction: transaction_identity, exact_update, }) } -fn stream_fold_lifecycle( - sidecar: &RecoverySidecar, +struct ObservedStreamFoldTokenV12 { + state: StreamFoldParticipantState, head: super::CurrentHeadWitness, - epoch_floor: u64, -) -> Result { - let protocol = sidecar - .protocol_v11 - .as_ref() - .expect("validated StreamFold protocol"); - let pin = &sidecar.tables[0]; - let lifecycle = super::StreamLifecycleEntry { - identity: pin.identity, - diagnostic_table_key: pin.table_key.clone(), - lifecycle: super::StreamLifecycle::Open, - binding: protocol.binding.clone(), - current_head_witness: head, - epoch_floor_by_shard: BTreeMap::from([( - protocol.generation_cut.shard_id.to_string(), - epoch_floor, - )]), - }; - lifecycle.validate()?; - Ok(lifecycle) -} - -fn stream_fold_prior_lifecycle(sidecar: &RecoverySidecar) -> Result { - let protocol = sidecar - .protocol_v11 - .as_ref() - .expect("validated StreamFold protocol"); - stream_fold_lifecycle( - sidecar, - protocol.prior_head.clone(), - protocol.prior_epoch_floor, - ) + transaction: StagedTransactionIdentity, + authority: super::StreamTokenAuthorityEntry, +} + +fn classify_stream_fold_token_v12_observation( + protocol: &RecoveryProtocolV12, + observed_authority: &super::StreamTokenAuthorityEntry, + observed_transaction: &StagedTransactionIdentity, +) -> std::result::Result { + let effect = &protocol.token; + if observed_authority == &effect.prior_authority { + return Ok(StreamFoldParticipantState::ExactNoEffect); + } + let expected_version = effect + .prior_authority + .current_head_witness + .table_version + .checked_add(1) + .ok_or_else(|| "StreamFold token-table prior version overflows".to_string())?; + if observed_authority.location != effect.prior_authority.location + || observed_authority.schema_version != effect.prior_authority.schema_version + || observed_authority.schema_hash != effect.prior_authority.schema_hash + || observed_authority.current_head_witness.table_version != expected_version + || observed_authority.current_head_witness.transaction_uuid + != effect.planned_transaction.uuid + || observed_transaction != &effect.planned_transaction + { + return Err( + "StreamFold token-table HEAD is neither the manifest-selected prior witness nor the exact planned N+1 effect" + .to_string(), + ); + } + if protocol.effect_phase == RecoveryEffectPhase::EffectsConfirmed + && (effect.confirmed_transaction.as_ref() != Some(&effect.planned_transaction) + || effect.confirmed_head.as_ref() != Some(&observed_authority.current_head_witness) + || effect.next_authority.as_ref() != Some(observed_authority)) + { + return Err( + "StreamFold durable token confirmation differs from the exact observed N+1 effect" + .to_string(), + ); + } + Ok(StreamFoldParticipantState::ExactEffect) } -fn stream_fold_next_lifecycle(sidecar: &RecoverySidecar) -> Result { - let protocol = sidecar - .protocol_v11 - .as_ref() - .expect("validated StreamFold protocol"); - let head = protocol.confirmed_head.clone().ok_or_else(|| { - OmniError::manifest_internal("confirmed StreamFold has no achieved HEAD witness") +async fn observe_stream_fold_token_v12( + root_uri: &str, + sidecar: &RecoverySidecar, +) -> Result { + let protocol = sidecar.protocol_v12.as_ref().ok_or_else(|| { + OmniError::manifest_internal("schema-v12 StreamFold is missing protocol_v12") })?; - stream_fold_lifecycle(sidecar, head, protocol.generation_cut.writer_epoch) + let token_uri = super::layout::stream_token_uri(root_uri); + let control_session = crate::lance_access::control_session(); + let dataset = crate::instrumentation::open_dataset( + &token_uri, + crate::instrumentation::VersionResolution::Latest, + Some(&control_session), + crate::instrumentation::table_wrapper(), + ) + .await + .map_err(|error| stream_fold_effect_error(sidecar, error))?; + let authority = super::token_store::stream_token_authority_entry_for_dataset(&dataset) + .await + .map_err(|error| stream_fold_effect_error(sidecar, error))?; + let transaction = dataset + .read_transaction() + .await + .map_err(|error| stream_fold_effect_error(sidecar, OmniError::Lance(error.to_string())))? + .ok_or_else(|| { + stream_fold_effect_error( + sidecar, + OmniError::manifest_internal("StreamFold token-table HEAD has no transaction"), + ) + })?; + let transaction = StagedTransactionIdentity::from(&transaction); + let state = classify_stream_fold_token_v12_observation(protocol, &authority, &transaction) + .map_err(|reason| OmniError::recovery_required(sidecar.operation_id.clone(), reason))?; + Ok(ObservedStreamFoldTokenV12 { + state, + head: authority.current_head_witness.clone(), + transaction, + authority, + }) } -fn validate_stream_fold_manifest_prestate( - snapshot: &Snapshot, +/// Finish the only partial prefix emitted by the writer's fixed commit order: +/// base exact, token still at the manifest-selected prior witness. The bounded +/// token rows and transaction UUID were durable before the base commit, so +/// recovery can re-stage exactly that logical participant rather than adopt or +/// infer anything from raw HEAD. +async fn finish_missing_stream_fold_token_v12( + root_uri: &str, sidecar: &RecoverySidecar, -) -> Result<(super::SubTableEntry, super::StreamLifecycleEntry)> { - let protocol = sidecar - .protocol_v11 - .as_ref() - .expect("validated StreamFold protocol"); - let pin = &sidecar.tables[0]; - let entry = snapshot_entry_for_pin(snapshot, pin) - .map_err(|error| stream_fold_effect_error(sidecar, error))? - .cloned() +) -> Result { + let protocol = sidecar.protocol_v12.as_ref().ok_or_else(|| { + OmniError::manifest_internal("schema-v12 StreamFold is missing protocol_v12") + })?; + if protocol.effect_phase != RecoveryEffectPhase::Armed { + return Err(OmniError::recovery_required( + sidecar.operation_id.clone(), + "cannot finish a missing StreamFold token effect after confirmation", + )); + } + let session = crate::lance_access::control_session(); + let dataset = super::token_store::open_stream_token_authority_at( + root_uri, + &protocol.token.prior_authority, + &session, + ) + .await + .map_err(|error| stream_fold_effect_error(sidecar, error))?; + let mut staged = super::token_store::stage_stream_token_upsert( + dataset.clone(), + &protocol.token.prior_authority, + &protocol.token.planned_rows, + ) + .await + .map_err(|error| stream_fold_effect_error(sidecar, error))?; + staged + .bind_transaction_identity(&protocol.token.planned_transaction) + .map_err(|error| stream_fold_effect_error(sidecar, error))?; + let table_store = crate::table_store::TableStore::new(root_uri, session); + let (achieved, committed) = table_store + .commit_staged_exact(std::sync::Arc::new(dataset), staged) + .await + .map_err(|error| stream_fold_effect_error(sidecar, error))?; + let expected_version = protocol + .token + .planned_transaction + .read_version + .checked_add(1) .ok_or_else(|| { - OmniError::recovery_required( - sidecar.operation_id.clone(), - format!( - "StreamFold table identity {} is no longer live in the manifest", - pin.identity - ), + stream_fold_effect_error( + sidecar, + OmniError::manifest_internal("StreamFold token version overflows"), ) })?; - let expected_lifecycle = stream_fold_prior_lifecycle(sidecar) - .map_err(|error| stream_fold_effect_error(sidecar, error))?; - if entry.table_path != protocol.binding.table_location - || entry.table_branch.is_some() - || entry.table_version != pin.expected_version - || snapshot.stream_lifecycle(pin.identity) != Some(&expected_lifecycle) + if committed != protocol.token.planned_transaction || achieved.version().version != expected_version { return Err(OmniError::recovery_required( sidecar.operation_id.clone(), - "StreamFold manifest pointer or exact prior lifecycle authority changed after recovery was armed", + "recovery committed a non-exact StreamFold token transaction", )); } - Ok((entry, expected_lifecycle)) -} - -#[derive(Clone, Copy)] -enum StreamFoldCleanup { - Required, - #[allow(dead_code)] - BestEffortAfterVisible, + let authority = super::token_store::stream_token_authority_entry_for_dataset(&achieved) + .await + .map_err(|error| stream_fold_effect_error(sidecar, error))?; + let state = classify_stream_fold_token_v12_observation(protocol, &authority, &committed) + .map_err(|reason| OmniError::recovery_required(sidecar.operation_id.clone(), reason))?; + if state != StreamFoldParticipantState::ExactEffect { + return Err(OmniError::recovery_required( + sidecar.operation_id.clone(), + "recovery did not achieve the exact planned StreamFold token effect", + )); + } + Ok(ObservedStreamFoldTokenV12 { + state, + head: authority.current_head_witness.clone(), + transaction: committed, + authority, + }) } -async fn cleanup_visible_stream_fold_sidecar( +async fn validate_stream_fold_token_rows_against_base_v12( root_uri: &str, - storage: &dyn StorageAdapter, sidecar: &RecoverySidecar, - cleanup_policy: StreamFoldCleanup, ) -> Result<()> { - let handle = RecoverySidecarHandle { - operation_id: sidecar.operation_id.clone(), - sidecar_uri: sidecar_uri(root_uri, &sidecar.operation_id), - }; - match delete_sidecar(&handle, storage).await { - Ok(()) => Ok(()), - Err(error) => match cleanup_policy { - StreamFoldCleanup::Required => Err(error), - StreamFoldCleanup::BestEffortAfterVisible => { - tracing::warn!( - error = %error, - operation_id = sidecar.operation_id.as_str(), - "StreamFold recovery sidecar cleanup failed; the next recovery barrier will resolve it" - ); - Ok(()) + let protocol = sidecar + .protocol_v12 + .as_ref() + .expect("validated schema-v12 StreamFold protocol"); + let pin = &sidecar.tables[0]; + let uri = format!( + "{}/{}", + root_uri.trim_end_matches('/'), + protocol.binding.table_location.trim_start_matches('/') + ); + let session = crate::lance_access::control_session(); + let dataset = crate::instrumentation::open_dataset( + &uri, + crate::instrumentation::VersionResolution::At(pin.post_commit_pin), + Some(&session), + crate::instrumentation::table_wrapper(), + ) + .await + .map_err(|error| stream_fold_effect_error(sidecar, error))?; + let exact_ids = protocol + .token + .planned_rows + .iter() + .map(|row| row.logical_id.clone()) + .collect::>(); + let mut scanner = dataset.scan(); + scanner.filter_expr(datafusion::prelude::col("id").in_list( + exact_ids + .iter() + .cloned() + .map(datafusion::prelude::lit) + .collect(), + false, + )); + let row_limit = exact_ids.len().saturating_add(1); + scanner.batch_size(row_limit); + scanner.batch_size_bytes( + crate::table_store::mem_wal::B2_MAX_TOKEN_PROJECTION_ARROW_BYTES, + ); + scanner + .limit( + Some(i64::try_from(row_limit).map_err(|_| { + stream_fold_effect_error( + sidecar, + OmniError::manifest_internal( + "recovery base-token probe row limit exceeds i64", + ), + ) + })?), + None, + ) + .map_err(|error| { + stream_fold_effect_error(sidecar, OmniError::Lance(error.to_string())) + })?; + scanner + .project(&["id", crate::db::STREAM_METADATA_COLUMN]) + .map_err(|error| stream_fold_effect_error(sidecar, OmniError::Lance(error.to_string())))?; + let mut stream = scanner.try_into_stream().await.map_err(|error| { + stream_fold_effect_error(sidecar, OmniError::Lance(error.to_string())) + })?; + let planned = protocol + .token + .planned_rows + .iter() + .map(|row| (row.logical_id.as_str(), row)) + .collect::>(); + let mut observed = std::collections::BTreeSet::new(); + let mut observed_rows = 0_usize; + let mut retained_bytes = 0_u64; + while let Some(batch) = futures::TryStreamExt::try_next(&mut stream) + .await + .map_err(|error| stream_fold_effect_error(sidecar, OmniError::Lance(error.to_string())))? + { + observed_rows = observed_rows + .checked_add(batch.num_rows()) + .ok_or_else(|| { + stream_fold_effect_error( + sidecar, + OmniError::manifest_internal("recovery base-token probe row-count overflow"), + ) + })?; + if observed_rows > planned.len() { + return Err(OmniError::recovery_required( + sidecar.operation_id.clone(), + "recovery base-token probe returned more than one row per planned key", + )); + } + let batch_bytes = u64::try_from(batch.get_array_memory_size()).map_err(|_| { + stream_fold_effect_error( + sidecar, + OmniError::manifest_internal( + "recovery base-token probe batch Arrow size exceeds u64", + ), + ) + })?; + if batch_bytes > crate::table_store::mem_wal::B2_MAX_TOKEN_PROJECTION_ARROW_BYTES { + return Err(stream_fold_effect_error( + sidecar, + OmniError::resource_limit( + "stream_fold_recovery_probe_batch_arrow_bytes", + crate::table_store::mem_wal::B2_MAX_TOKEN_PROJECTION_ARROW_BYTES, + batch_bytes, + ), + )); + } + let ids = batch + .column_by_name("id") + .and_then(|array| { + array + .as_any() + .downcast_ref::() + }) + .ok_or_else(|| { + stream_fold_effect_error( + sidecar, + OmniError::manifest_internal( + "recovery base-token probe has no exact Utf8 id column", + ), + ) + })?; + let metadata = batch + .column_by_name(crate::db::STREAM_METADATA_COLUMN) + .ok_or_else(|| { + stream_fold_effect_error( + sidecar, + OmniError::manifest_internal( + "recovery base-token probe has no trusted metadata column", + ), + ) + })?; + for row_index in 0..batch.num_rows() { + if arrow_array::Array::is_null(ids, row_index) { + return Err(OmniError::recovery_required( + sidecar.operation_id.clone(), + "recovery base-token probe found a null logical id", + )); } - }, + let logical_id = ids.value(row_index); + let expected = planned.get(logical_id).ok_or_else(|| { + OmniError::recovery_required( + sidecar.operation_id.clone(), + "recovery base-token probe returned an unplanned logical id", + ) + })?; + let decoded = super::stream_token::decode_trusted_stream_metadata( + metadata.as_ref(), + row_index, + ) + .map_err(|error| { + OmniError::recovery_required(sidecar.operation_id.clone(), error.to_string()) + })? + .ok_or_else(|| { + OmniError::recovery_required( + sidecar.operation_id.clone(), + "planned StreamFold token row has null base attribution", + ) + })?; + retained_bytes = super::token_store::add_stream_lookup_retained_bytes( + "stream_fold_recovery_probe_retained_bytes", + retained_bytes, + decoded.lookup_retained_bytes(logical_id).map_err(|error| { + stream_fold_effect_error( + sidecar, + OmniError::manifest_internal(error.to_string()), + ) + })?, + crate::table_store::mem_wal::B2_MAX_TOKEN_PROJECTION_ARROW_BYTES, + ) + .map_err(|error| stream_fold_effect_error(sidecar, error))?; + if !decoded.agrees_with_authority(expected) || !observed.insert(logical_id.to_string()) + { + return Err(OmniError::recovery_required( + sidecar.operation_id.clone(), + "planned StreamFold token authority differs from the achieved base winner", + )); + } + } + } + if observed.len() != planned.len() { + return Err(OmniError::recovery_required( + sidecar.operation_id.clone(), + "achieved StreamFold base version is missing one or more planned token winners", + )); } + Ok(()) } -fn stream_fold_effect_error(sidecar: &RecoverySidecar, error: OmniError) -> OmniError { - match error { - OmniError::RecoveryRequired { .. } => error, - other => OmniError::recovery_required( +fn validate_stream_fold_v12_manifest_prestate( + snapshot: &Snapshot, + sidecar: &RecoverySidecar, +) -> Result { + let protocol = sidecar + .protocol_v12 + .as_ref() + .expect("validated schema-v12 StreamFold protocol"); + let pin = &sidecar.tables[0]; + let entry = snapshot_entry_for_pin(snapshot, pin) + .map_err(|error| stream_fold_effect_error(sidecar, error))? + .cloned() + .ok_or_else(|| { + OmniError::recovery_required( + sidecar.operation_id.clone(), + format!( + "StreamFold table identity {} is no longer live in the manifest", + pin.identity + ), + ) + })?; + if entry.table_path != protocol.binding.table_location + || entry.table_branch.is_some() + || entry.table_version != pin.expected_version + || snapshot.stream_lifecycle(pin.identity) != Some(&protocol.prior_lifecycle) + || snapshot.stream_token_authority() != &protocol.token.prior_authority + { + return Err(OmniError::recovery_required( sidecar.operation_id.clone(), - format!("StreamFold completion failed while a durable effect may exist: {other}"), - ), + "StreamFold manifest base pointer, complete prior lifecycle, or manifest-selected token authority changed after recovery was armed", + )); } + Ok(entry) } -async fn stream_fold_original_visible(root_uri: &str, sidecar: &RecoverySidecar) -> Result { +fn stream_fold_v12_lineage_matches( + commit: &super::GraphLineageRow, + protocol: &RecoveryProtocolV12, +) -> bool { + commit.manifest_branch.is_none() + && commit.parent_commit_id.as_ref() == protocol.authority.graph_head.as_ref() + && commit.merged_parent_commit_id.is_none() + && commit.actor_id == protocol.lineage.actor_id + && commit.created_at == protocol.lineage.created_at + && commit.stream_fold_attribution.as_ref() == Some(&protocol.token.attribution_summary) +} + +async fn stream_fold_v12_original_visible( + root_uri: &str, + sidecar: &RecoverySidecar, +) -> Result { let protocol = sidecar - .protocol_v11 + .protocol_v12 .as_ref() - .expect("validated StreamFold protocol"); + .expect("validated schema-v12 StreamFold protocol"); if protocol.effect_phase != RecoveryEffectPhase::EffectsConfirmed { return Ok(false); } let pin = &sidecar.tables[0]; - let update = protocol + let base_update = protocol + .base .confirmed_update .as_ref() - .expect("validated confirmed StreamFold update"); - let expected_lifecycle = stream_fold_next_lifecycle(sidecar)?; + .expect("validated confirmed base update"); + let next_lifecycle = protocol + .next_lifecycle + .as_ref() + .expect("validated confirmed next lifecycle"); + let next_token_authority = protocol + .token + .next_authority + .as_ref() + .expect("validated confirmed token authority"); let (commits, _) = ManifestCoordinator::read_graph_lineage_at(root_uri, None).await?; let Some(commit) = commits .iter() @@ -4951,12 +6541,7 @@ async fn stream_fold_original_visible(root_uri: &str, sidecar: &RecoverySidecar) else { return Ok(false); }; - if commit.manifest_branch.is_some() - || commit.parent_commit_id.as_ref() != protocol.authority.graph_head.as_ref() - || commit.merged_parent_commit_id.is_some() - || commit.actor_id != protocol.lineage.actor_id - || commit.created_at != protocol.lineage.created_at - { + if !stream_fold_v12_lineage_matches(commit, protocol) { return Err(OmniError::manifest_internal(format!( "StreamFold sidecar '{}' found fixed commit '{}' with mismatched lineage", sidecar.operation_id, protocol.lineage.graph_commit_id @@ -4967,35 +6552,39 @@ async fn stream_fold_original_visible(root_uri: &str, sidecar: &RecoverySidecar) let entry_matches = snapshot_entry_by_identity(&committed, pin.identity).is_some_and(|entry| { entry.table_key == pin.table_key && entry.table_path == protocol.binding.table_location - && entry.table_version == update.table_version + && entry.table_version == base_update.table_version && entry.table_branch.is_none() - && entry.row_count == update.row_count - && entry.version_metadata == update.version_metadata + && entry.row_count == base_update.row_count + && entry.version_metadata == base_update.version_metadata }); - if !entry_matches || committed.stream_lifecycle(pin.identity) != Some(&expected_lifecycle) { + if !entry_matches + || committed.stream_lifecycle(pin.identity) != Some(next_lifecycle) + || committed.stream_token_authority() != next_token_authority + { return Err(OmniError::manifest_internal(format!( - "StreamFold sidecar '{}' found fixed commit '{}' but its table/lifecycle outcome differs", + "StreamFold sidecar '{}' found fixed commit '{}' but its base/lifecycle/token outcome differs", sidecar.operation_id, protocol.lineage.graph_commit_id ))); } Ok(true) } -async fn finalize_visible_stream_fold( +async fn finalize_visible_stream_fold_v12( root_uri: &str, storage: &dyn StorageAdapter, sidecar: &RecoverySidecar, cleanup_policy: StreamFoldCleanup, ) -> Result { let protocol = sidecar - .protocol_v11 + .protocol_v12 .as_ref() - .expect("validated StreamFold protocol"); + .expect("validated schema-v12 StreamFold protocol"); let pin = &sidecar.tables[0]; let update = protocol + .base .confirmed_update .as_ref() - .expect("validated confirmed StreamFold update"); + .expect("validated confirmed base update"); let mut audit = RecoveryAudit::open(root_uri).await?; let expected_outcomes = vec![TableOutcome { table_key: pin.table_key.clone(), @@ -5020,8 +6609,7 @@ async fn finalize_visible_stream_fold( "StreamFold recovery audit row differs from the exact fixed outcome", )); } - let already_recorded = !operation_records.is_empty(); - if !already_recorded { + if operation_records.is_empty() { crate::failpoints::maybe_fail(crate::failpoints::names::RECOVERY_RECORD_AUDIT)?; audit .append(RecoveryAuditRecord { @@ -5039,36 +6627,43 @@ async fn finalize_visible_stream_fold( Ok(true) } -async fn process_stream_fold_sidecar_v11( +async fn process_stream_fold_sidecar_v12( root_uri: &str, storage: &std::sync::Arc, snapshot: &Snapshot, sidecar: &RecoverySidecar, cleanup_policy: StreamFoldCleanup, ) -> Result { - let observed = observe_stream_fold_effect(root_uri, sidecar).await?; + let observed_base = observe_stream_fold_base_v12(root_uri, sidecar).await?; + let mut observed_token = observe_stream_fold_token_v12(root_uri, sidecar).await?; + let mut matrix = + classify_stream_fold_v12_effect_matrix(observed_base.state, observed_token.state); - if stream_fold_original_visible(root_uri, sidecar) + if stream_fold_v12_original_visible(root_uri, sidecar) .await .map_err(|error| stream_fold_effect_error(sidecar, error))? { - if observed.state != StreamFoldEffectState::ExactEffect { + if matrix != StreamFoldV12EffectMatrix::Complete { return Err(OmniError::recovery_required( sidecar.operation_id.clone(), - "fixed StreamFold graph commit is visible without its exact base-table effect", + "fixed StreamFold graph commit is visible without both exact physical effects", )); } - return finalize_visible_stream_fold(root_uri, storage.as_ref(), sidecar, cleanup_policy) - .await - .map_err(|error| stream_fold_effect_error(sidecar, error)); + return finalize_visible_stream_fold_v12( + root_uri, + storage.as_ref(), + sidecar, + cleanup_policy, + ) + .await + .map_err(|error| stream_fold_effect_error(sidecar, error)); } - let (manifest_entry, prior_lifecycle) = - validate_stream_fold_manifest_prestate(snapshot, sidecar)?; + let manifest_entry = validate_stream_fold_v12_manifest_prestate(snapshot, sidecar)?; let protocol = sidecar - .protocol_v11 + .protocol_v12 .as_ref() - .expect("validated StreamFold protocol"); + .expect("validated schema-v12 StreamFold protocol"); let live_authority = read_live_recovery_authority(root_uri, storage, None) .await .map_err(|error| stream_fold_effect_error(sidecar, error))?; @@ -5078,47 +6673,88 @@ async fn process_stream_fold_sidecar_v11( "StreamFold graph/schema authority changed after recovery was armed", )); } + if observed_base.state == StreamFoldParticipantState::ExactEffect { + validate_stream_fold_token_rows_against_base_v12(root_uri, sidecar).await?; + } - if observed.state == StreamFoldEffectState::ExactNoEffect { - delete_sidecar_by_operation_id(root_uri, storage.as_ref(), &sidecar.operation_id).await?; - return Ok(true); + if matrix == StreamFoldV12EffectMatrix::Partial + && observed_base.state == StreamFoldParticipantState::ExactEffect + && observed_token.state == StreamFoldParticipantState::ExactNoEffect + { + observed_token = finish_missing_stream_fold_token_v12(root_uri, sidecar).await?; + matrix = + classify_stream_fold_v12_effect_matrix(observed_base.state, observed_token.state); + } + + match matrix { + StreamFoldV12EffectMatrix::EffectFree => { + if protocol.effect_phase != RecoveryEffectPhase::Armed { + return Err(OmniError::recovery_required( + sidecar.operation_id.clone(), + "EffectsConfirmed StreamFold has no corresponding physical effects", + )); + } + delete_sidecar_by_operation_id(root_uri, storage.as_ref(), &sidecar.operation_id) + .await?; + return Ok(true); + } + StreamFoldV12EffectMatrix::Partial => { + return Err(OmniError::recovery_required( + sidecar.operation_id.clone(), + "StreamFold has an unreachable token-only/base-missing prefix; it remains owned and cannot be published or adopted", + )); + } + StreamFoldV12EffectMatrix::Complete => {} } let mut confirmed = sidecar.clone(); if protocol.effect_phase == RecoveryEffectPhase::Armed { - let exact_update = observed + let update = observed_base .exact_update .as_ref() - .expect("exact StreamFold effect has a reconstructed update"); - confirm_stream_fold_sidecar_v11( + .expect("exact base effect has a reconstructed update"); + confirm_stream_fold_sidecar_v12( root_uri, storage.as_ref(), &mut confirmed, - protocol.planned_transaction.clone(), + observed_base.transaction.clone(), protocol.merged_generation.clone(), - observed.head.clone(), + observed_base.head.clone(), SubTableUpdate { identity: pin_identity(sidecar), table_key: sidecar.tables[0].table_key.clone(), - table_version: observed.head.table_version, - table_branch: None, - row_count: exact_update.row_count, - version_metadata: exact_update.version_metadata.clone(), + table_version: update.table_version, + table_branch: update.table_branch.clone(), + row_count: update.row_count, + version_metadata: update.version_metadata.clone(), }, + observed_token.transaction.clone(), + observed_token.head.clone(), + observed_token.authority.clone(), ) .await .map_err(|error| stream_fold_effect_error(sidecar, error))?; } let confirmed_protocol = confirmed - .protocol_v11 + .protocol_v12 .as_ref() - .expect("confirmed StreamFold protocol"); + .expect("confirmed schema-v12 StreamFold protocol"); let confirmed_update = confirmed_protocol + .base .confirmed_update .as_ref() - .expect("confirmed StreamFold update"); - let next_lifecycle = stream_fold_next_lifecycle(&confirmed) - .map_err(|error| stream_fold_effect_error(&confirmed, error))?; + .expect("confirmed StreamFold base update"); + let next_lifecycle = confirmed_protocol + .next_lifecycle + .as_ref() + .expect("confirmed StreamFold next lifecycle") + .clone(); + let next_token_authority = confirmed_protocol + .token + .next_authority + .as_ref() + .expect("confirmed StreamFold token authority") + .clone(); let updates = vec![ ManifestChange::Update(SubTableUpdate { identity: pin_identity(&confirmed), @@ -5128,8 +6764,12 @@ async fn process_stream_fold_sidecar_v11( row_count: confirmed_update.row_count, version_metadata: confirmed_update.version_metadata.clone(), }), + ManifestChange::SetStreamTokenAuthority { + expected: confirmed_protocol.token.prior_authority.clone(), + next: next_token_authority, + }, ManifestChange::SetStreamLifecycle { - expected: Some(prior_lifecycle), + expected: Some(confirmed_protocol.prior_lifecycle.clone()), next: next_lifecycle, }, ]; @@ -5169,31 +6809,23 @@ async fn process_stream_fold_sidecar_v11( Ok(true) } -fn pin_identity(sidecar: &RecoverySidecar) -> TableIdentity { - sidecar.tables[0].identity -} - -/// Retire one Armed StreamFold only after exact N/no-merge-progress evidence. -/// An exact effect, foreign movement, or unreadable evidence leaves the sidecar -/// in place so the caller returns `RecoveryRequired` rather than replanning -/// around an unresolved fold. #[allow(dead_code)] -pub(crate) async fn finalize_effect_free_stream_fold_sidecar_v11( +pub(crate) async fn finalize_effect_free_stream_fold_sidecar_v12( root_uri: &str, storage: &std::sync::Arc, snapshot: &Snapshot, sidecar: &RecoverySidecar, ) -> Result { validate_sidecar_shape(&sidecar_uri(root_uri, &sidecar.operation_id), sidecar)?; - let protocol = sidecar.protocol_v11.as_ref().ok_or_else(|| { + let protocol = sidecar.protocol_v12.as_ref().ok_or_else(|| { OmniError::manifest_internal( - "effect-free StreamFold finalization requires a schema-v11 payload", + "effect-free StreamFold finalization requires a schema-v12 payload", ) })?; if protocol.effect_phase != RecoveryEffectPhase::Armed { return Ok(false); } - validate_stream_fold_manifest_prestate(snapshot, sidecar)?; + validate_stream_fold_v12_manifest_prestate(snapshot, sidecar)?; let live_authority = read_live_recovery_authority(root_uri, storage, None) .await .map_err(|error| stream_fold_effect_error(sidecar, error))?; @@ -5203,26 +6835,25 @@ pub(crate) async fn finalize_effect_free_stream_fold_sidecar_v11( "StreamFold graph/schema authority changed before effect-free finalization", )); } - let observed = observe_stream_fold_effect(root_uri, sidecar).await?; - if observed.state != StreamFoldEffectState::ExactNoEffect { + let base = observe_stream_fold_base_v12(root_uri, sidecar).await?; + let token = observe_stream_fold_token_v12(root_uri, sidecar).await?; + if classify_stream_fold_v12_effect_matrix(base.state, token.state) + != StreamFoldV12EffectMatrix::EffectFree + { return Ok(false); } delete_sidecar_by_operation_id(root_uri, storage.as_ref(), &sidecar.operation_id).await?; Ok(true) } -/// Writer-side entry into the same exact roll-forward adapter used by recovery. -/// Callers retain exclusive stream admission and normal write gates while this -/// completes; cleanup after a graph-visible outcome is best effort because the -/// fixed lineage makes re-entry idempotent. #[allow(dead_code)] -pub(crate) async fn complete_stream_fold_sidecar_v11( +pub(crate) async fn complete_stream_fold_sidecar_v12( root_uri: &str, storage: std::sync::Arc, snapshot: &Snapshot, sidecar: &RecoverySidecar, ) -> Result<()> { - process_stream_fold_sidecar_v11( + process_stream_fold_sidecar_v12( root_uri, &storage, snapshot, @@ -5260,8 +6891,14 @@ async fn process_sidecar( )) .await; } + if sidecar.schema_version == LEGACY_STREAM_FOLD_SIDECAR_SCHEMA_VERSION { + return Err(OmniError::recovery_required( + sidecar.operation_id.clone(), + "historical schema-v11 StreamFold cannot be recovered under internal schema v9 because it lacks the token participant and complete state-v2 lifecycle authority", + )); + } if sidecar.schema_version == STREAM_FOLD_SIDECAR_SCHEMA_VERSION { - return Box::pin(process_stream_fold_sidecar_v11( + return Box::pin(process_stream_fold_sidecar_v12( root_uri, storage, snapshot, @@ -5331,7 +6968,7 @@ async fn process_sidecar( } SidecarKind::StreamFold => { return Err(OmniError::manifest_internal( - "StreamFold appeared outside its schema-v11 recovery envelope", + "StreamFold appeared outside its schema-v12 recovery envelope", )); } } @@ -7797,6 +9434,7 @@ fn has_exact_protocol(sidecar: &RecoverySidecar) -> bool { || sidecar.protocol_v8.is_some() || sidecar.protocol_v10.is_some() || sidecar.protocol_v11.is_some() + || sidecar.protocol_v12.is_some() } fn has_fixed_rollback_identity(sidecar: &RecoverySidecar) -> bool { @@ -9754,6 +11392,7 @@ fn new_unvalidated_sidecar( protocol_v8: None, protocol_v10: None, protocol_v11: None, + protocol_v12: None, ensure_indices_rollback_v6: None, } } @@ -9774,6 +11413,7 @@ pub(crate) fn new_stream_enrollment_sidecar_v10( baseline_head: super::CurrentHeadWitness, enrollment_plan: MemWalEnrollmentPlan, intended_binding: super::StreamPhysicalBinding, + enrollment_receipt: super::EnrollmentReceipt, ) -> Result { let mut sidecar = new_unvalidated_sidecar( STREAM_ENROLLMENT_SIDECAR_SCHEMA_VERSION, @@ -9788,6 +11428,7 @@ pub(crate) fn new_stream_enrollment_sidecar_v10( baseline_head, enrollment_plan, intended_binding, + enrollment_receipt, }); validate_sidecar_shape("", &sidecar)?; Ok(sidecar) @@ -9817,7 +11458,7 @@ pub(crate) fn new_stream_fold_sidecar_v11( let merged_generation = MergedGeneration::new(generation_cut.shard_id, generation_cut.generation); let mut sidecar = new_unvalidated_sidecar( - STREAM_FOLD_SIDECAR_SCHEMA_VERSION, + LEGACY_STREAM_FOLD_SIDECAR_SCHEMA_VERSION, SidecarKind::StreamFold, None, actor_id, @@ -9843,30 +11484,175 @@ pub(crate) fn new_stream_fold_sidecar_v11( Ok(sidecar) } -/// Bind the exact achieved output of a schema-v11 StreamFold and durably move -/// its sidecar from Armed to EffectsConfirmed. +/// Arm one exact RFC-026 B2 two-participant fold before either Lance HEAD may +/// move. `tables` remains exactly the single base table; the graph-global token +/// participant is carried by `protocol_v12.token` and is never represented by +/// a sentinel graph-table identity. +#[allow(clippy::too_many_arguments)] +#[allow(dead_code)] +pub(crate) fn new_stream_fold_sidecar_v12( + actor_id: Option, + table: SidecarTablePin, + authority: RecoveryAuthorityToken, + lineage: RecoveryLineageIntent, + binding: super::StreamPhysicalBinding, + prior_lifecycle: super::StreamLifecycleEntry, + mut planned_next_lifecycle: super::StreamLifecycleEntry, + prior_merged_generation: Option, + generation_cut: RecoveryStreamFoldCut, + base_planned_transaction: StagedTransactionIdentity, + prior_token_authority: super::StreamTokenAuthorityEntry, + token_planned_transaction: StagedTransactionIdentity, + token_planned_rows: Vec, + attribution_summary: RecoveryStreamFoldAttributionSummary, +) -> Result { + let merged_generation = + MergedGeneration::new(generation_cut.shard_id, generation_cut.generation); + let mut sidecar = new_unvalidated_sidecar( + STREAM_FOLD_SIDECAR_SCHEMA_VERSION, + SidecarKind::StreamFold, + None, + actor_id, + vec![table], + ); + if let Some(summary) = planned_next_lifecycle.last_fold_summary.as_mut() { + summary.operation_id = sidecar.operation_id.clone(); + } + sidecar.protocol_v12 = Some(Box::new(RecoveryProtocolV12 { + authority, + lineage, + binding, + prior_merged_generation, + generation_cut, + merged_generation, + effect_phase: RecoveryEffectPhase::Armed, + prior_lifecycle, + planned_next_lifecycle, + next_lifecycle: None, + base: RecoveryStreamFoldBaseEffect { + planned_transaction: base_planned_transaction, + confirmed_transaction: None, + confirmed_merged_generation: None, + confirmed_head: None, + confirmed_update: None, + }, + token: RecoveryStreamFoldTokenEffect { + prior_authority: prior_token_authority, + planned_transaction: token_planned_transaction, + planned_rows: token_planned_rows, + attribution_summary, + confirmed_transaction: None, + confirmed_head: None, + next_authority: None, + }, + })); + validate_sidecar_shape("", &sidecar)?; + Ok(sidecar) +} + +/// Bind the exact achieved output of a schema-v11 StreamFold and durably move +/// its sidecar from Armed to EffectsConfirmed. +/// +/// Validation happens on a clone before the single-object rewrite. A mismatch +/// leaves the durable sidecar Armed; fold recovery will independently classify +/// exact N/N+1 physical state and either reconstruct this confirmation or fail +/// closed. The helper never accepts a rebased/later version, a different merge +/// watermark, or a lifecycle witness detached from the planned transaction. +#[allow(clippy::too_many_arguments)] +pub(crate) async fn confirm_stream_fold_sidecar_v11( + root_uri: &str, + storage: &dyn StorageAdapter, + sidecar: &mut RecoverySidecar, + committed_transaction: StagedTransactionIdentity, + confirmed_merged_generation: MergedGeneration, + achieved_head: super::CurrentHeadWitness, + update: SubTableUpdate, +) -> Result<()> { + crate::failpoints::maybe_fail(crate::failpoints::names::RECOVERY_SIDECAR_CONFIRM)?; + let uri = sidecar_uri(root_uri, &sidecar.operation_id); + validate_sidecar_shape(&uri, sidecar)?; + let protocol = sidecar.protocol_v11.as_ref().ok_or_else(|| { + OmniError::manifest_internal( + "confirm_stream_fold_sidecar_v11 requires a schema-v11 StreamFold sidecar", + ) + })?; + if protocol.effect_phase != RecoveryEffectPhase::Armed { + return Err(OmniError::manifest_internal(format!( + "StreamFold sidecar '{}' is already EffectsConfirmed", + sidecar.operation_id + ))); + } + let pin = sidecar.tables.first().expect("validated StreamFold pin"); + if committed_transaction != protocol.planned_transaction + || confirmed_merged_generation != protocol.merged_generation + || achieved_head.branch_identifier != lance::dataset::refs::BranchIdentifier::main() + || achieved_head.table_version != pin.post_commit_pin + || achieved_head.transaction_uuid != committed_transaction.uuid + || update.identity != pin.identity + || update.table_key != pin.table_key + || update.table_version != pin.post_commit_pin + || update.table_branch.is_some() + { + return Err(OmniError::manifest_internal(format!( + "StreamFold sidecar '{}' confirmation differs from its exact planned N+1 effect; leaving recovery Armed", + sidecar.operation_id + ))); + } + + let mut confirmed = sidecar.clone(); + let confirmed_protocol = confirmed + .protocol_v11 + .as_mut() + .expect("validated StreamFold protocol"); + confirmed_protocol.effect_phase = RecoveryEffectPhase::EffectsConfirmed; + confirmed_protocol.confirmed_transaction = Some(committed_transaction); + confirmed_protocol.confirmed_merged_generation = Some(confirmed_merged_generation); + confirmed_protocol.confirmed_head = Some(achieved_head); + confirmed_protocol.confirmed_update = Some(RecoveryConfirmedTableUpdate { + table_version: update.table_version, + table_branch: update.table_branch, + row_count: update.row_count, + version_metadata: update.version_metadata, + }); + confirmed.tables[0].confirmed_version = Some(update.table_version); + + validate_sidecar_shape(&uri, &confirmed)?; + let json = serde_json::to_string_pretty(&confirmed).map_err(|error| { + OmniError::manifest_internal(format!( + "failed to serialize confirmed StreamFold recovery sidecar: {error}" + )) + })?; + storage.write_text(&uri, &json).await?; + *sidecar = confirmed; + Ok(()) +} + +/// Bind both exact schema-v12 participant outputs and durably move one +/// StreamFold from Armed to EffectsConfirmed in a single sidecar rewrite. /// -/// Validation happens on a clone before the single-object rewrite. A mismatch -/// leaves the durable sidecar Armed; fold recovery will independently classify -/// exact N/N+1 physical state and either reconstruct this confirmation or fail -/// closed. The helper never accepts a rebased/later version, a different merge -/// watermark, or a lifecycle witness detached from the planned transaction. +/// Callers invoke this only after both Lance commits have succeeded. If the +/// process dies after either commit, the durable Armed envelope remains and +/// recovery classifies the two participants independently; it never treats a +/// partial cell as publishable. #[allow(clippy::too_many_arguments)] -pub(crate) async fn confirm_stream_fold_sidecar_v11( +pub(crate) async fn confirm_stream_fold_sidecar_v12( root_uri: &str, storage: &dyn StorageAdapter, sidecar: &mut RecoverySidecar, - committed_transaction: StagedTransactionIdentity, + base_committed_transaction: StagedTransactionIdentity, confirmed_merged_generation: MergedGeneration, - achieved_head: super::CurrentHeadWitness, - update: SubTableUpdate, + achieved_base_head: super::CurrentHeadWitness, + base_update: SubTableUpdate, + token_committed_transaction: StagedTransactionIdentity, + achieved_token_head: super::CurrentHeadWitness, + next_token_authority: super::StreamTokenAuthorityEntry, ) -> Result<()> { crate::failpoints::maybe_fail(crate::failpoints::names::RECOVERY_SIDECAR_CONFIRM)?; let uri = sidecar_uri(root_uri, &sidecar.operation_id); validate_sidecar_shape(&uri, sidecar)?; - let protocol = sidecar.protocol_v11.as_ref().ok_or_else(|| { + let protocol = sidecar.protocol_v12.as_ref().ok_or_else(|| { OmniError::manifest_internal( - "confirm_stream_fold_sidecar_v11 requires a schema-v11 StreamFold sidecar", + "confirm_stream_fold_sidecar_v12 requires a schema-v12 StreamFold sidecar", ) })?; if protocol.effect_phase != RecoveryEffectPhase::Armed { @@ -9875,39 +11661,64 @@ pub(crate) async fn confirm_stream_fold_sidecar_v11( sidecar.operation_id ))); } - let pin = sidecar.tables.first().expect("validated StreamFold pin"); - if committed_transaction != protocol.planned_transaction + let pin = sidecar + .tables + .first() + .expect("validated StreamFold base pin"); + if base_committed_transaction != protocol.base.planned_transaction || confirmed_merged_generation != protocol.merged_generation - || achieved_head.branch_identifier != lance::dataset::refs::BranchIdentifier::main() - || achieved_head.table_version != pin.post_commit_pin - || achieved_head.transaction_uuid != committed_transaction.uuid - || update.identity != pin.identity - || update.table_key != pin.table_key - || update.table_version != pin.post_commit_pin - || update.table_branch.is_some() + || achieved_base_head.branch_identifier != lance::dataset::refs::BranchIdentifier::main() + || achieved_base_head.table_version != pin.post_commit_pin + || achieved_base_head.transaction_uuid != base_committed_transaction.uuid + || base_update.identity != pin.identity + || base_update.table_key != pin.table_key + || base_update.table_version != pin.post_commit_pin + || base_update.table_branch.is_some() + || token_committed_transaction != protocol.token.planned_transaction + || achieved_token_head.branch_identifier != lance::dataset::refs::BranchIdentifier::main() + || achieved_token_head.table_version + != protocol + .token + .planned_transaction + .read_version + .checked_add(1) + .ok_or_else(|| { + OmniError::manifest_internal("StreamFold token table version overflows") + })? + || achieved_token_head.transaction_uuid != token_committed_transaction.uuid + || next_token_authority.current_head_witness != achieved_token_head + || next_token_authority.location != protocol.token.prior_authority.location + || next_token_authority.schema_version != protocol.token.prior_authority.schema_version + || next_token_authority.schema_hash != protocol.token.prior_authority.schema_hash { return Err(OmniError::manifest_internal(format!( - "StreamFold sidecar '{}' confirmation differs from its exact planned N+1 effect; leaving recovery Armed", + "StreamFold sidecar '{}' confirmation differs from one of its two exact planned N+1 effects; leaving recovery Armed", sidecar.operation_id ))); } let mut confirmed = sidecar.clone(); let confirmed_protocol = confirmed - .protocol_v11 + .protocol_v12 .as_mut() .expect("validated StreamFold protocol"); confirmed_protocol.effect_phase = RecoveryEffectPhase::EffectsConfirmed; - confirmed_protocol.confirmed_transaction = Some(committed_transaction); - confirmed_protocol.confirmed_merged_generation = Some(confirmed_merged_generation); - confirmed_protocol.confirmed_head = Some(achieved_head); - confirmed_protocol.confirmed_update = Some(RecoveryConfirmedTableUpdate { - table_version: update.table_version, - table_branch: update.table_branch, - row_count: update.row_count, - version_metadata: update.version_metadata, + confirmed_protocol.base.confirmed_transaction = Some(base_committed_transaction); + confirmed_protocol.base.confirmed_merged_generation = Some(confirmed_merged_generation); + confirmed_protocol.base.confirmed_head = Some(achieved_base_head.clone()); + confirmed_protocol.base.confirmed_update = Some(RecoveryConfirmedTableUpdate { + table_version: base_update.table_version, + table_branch: base_update.table_branch, + row_count: base_update.row_count, + version_metadata: base_update.version_metadata, }); - confirmed.tables[0].confirmed_version = Some(update.table_version); + confirmed_protocol.token.confirmed_transaction = Some(token_committed_transaction); + confirmed_protocol.token.confirmed_head = Some(achieved_token_head); + confirmed_protocol.token.next_authority = Some(next_token_authority); + let mut next_lifecycle = confirmed_protocol.planned_next_lifecycle.clone(); + next_lifecycle.current_head_witness = achieved_base_head; + confirmed_protocol.next_lifecycle = Some(next_lifecycle); + confirmed.tables[0].confirmed_version = Some(base_update.table_version); validate_sidecar_shape(&uri, &confirmed)?; let json = serde_json::to_string_pretty(&confirmed).map_err(|error| { @@ -10032,6 +11843,7 @@ pub(crate) fn new_ensure_indices_sidecar_v9( }), protocol_v10: None, protocol_v11: None, + protocol_v12: None, ensure_indices_rollback_v6: None, }; validate_sidecar_shape("", &sidecar)?; @@ -10288,6 +12100,7 @@ pub(crate) fn new_occ_sidecar_v9( protocol_v8: None, protocol_v10: None, protocol_v11: None, + protocol_v12: None, ensure_indices_rollback_v6: None, }; validate_sidecar_shape("", &sidecar)?; @@ -10474,6 +12287,7 @@ pub(crate) fn new_schema_apply_sidecar_v9( protocol_v8: None, protocol_v10: None, protocol_v11: None, + protocol_v12: None, ensure_indices_rollback_v6: None, }; validate_sidecar_shape("", &sidecar)?; @@ -10641,6 +12455,7 @@ pub(crate) fn new_branch_merge_sidecar_v9( protocol_v8: None, protocol_v10: None, protocol_v11: None, + protocol_v12: None, ensure_indices_rollback_v6: None, }; validate_sidecar_shape("", &sidecar)?; @@ -10956,6 +12771,28 @@ mod tests { } } + #[test] + fn stream_fold_v12_effect_matrix_publishes_only_exact_exact() { + use StreamFoldParticipantState::{ExactEffect, ExactNoEffect}; + + assert_eq!( + classify_stream_fold_v12_effect_matrix(ExactNoEffect, ExactNoEffect), + StreamFoldV12EffectMatrix::EffectFree + ); + assert_eq!( + classify_stream_fold_v12_effect_matrix(ExactEffect, ExactEffect), + StreamFoldV12EffectMatrix::Complete + ); + assert_eq!( + classify_stream_fold_v12_effect_matrix(ExactEffect, ExactNoEffect), + StreamFoldV12EffectMatrix::Partial + ); + assert_eq!( + classify_stream_fold_v12_effect_matrix(ExactNoEffect, ExactEffect), + StreamFoldV12EffectMatrix::Partial + ); + } + fn occ_sidecar() -> RecoverySidecar { let planned = transaction(5, "planned-transaction"); let identity = test_identity("node:Person"); @@ -10989,7 +12826,7 @@ mod tests { crate::db::manifest::table_path_for_identity("node:Person", identity).unwrap(); let shard_id = ShardId::parse_str("22222222-2222-4222-8222-222222222222").unwrap(); new_stream_fold_sidecar_v11( - Some("omnigraph:stream-fold".to_string()), + Some(STREAM_FOLD_ACTOR.to_string()), make_pin("node:Person", "memory://test-graph/nodes/person", 5, 6), RecoveryAuthorityToken { branch_identifier: lance::dataset::refs::BranchIdentifier::main(), @@ -11001,7 +12838,7 @@ mod tests { RecoveryLineageIntent { graph_commit_id: "01H000000000000000000000F1".to_string(), branch: None, - actor_id: Some("omnigraph:stream-fold".to_string()), + actor_id: Some(STREAM_FOLD_ACTOR.to_string()), merged_parent_commit_id: None, created_at: 456, }, @@ -11036,6 +12873,350 @@ mod tests { .unwrap() } + fn stream_fold_sidecar_v12() -> RecoverySidecar { + use crate::db::manifest::stream::{LastFoldOutcome, LastFoldSummary, StreamGenerationCut}; + use crate::db::manifest::{ + CurrentHeadWitness, EnrollmentReceipt, STREAM_CONFIG_VERSION, StreamLifecycleEntry, + StreamPhysicalBinding, + }; + + let identity = test_identity("node:Person"); + let table_location = + crate::db::manifest::table_path_for_identity("node:Person", identity).unwrap(); + let enrollment_id = "11111111-1111-4111-8111-111111111111"; + let shard_id = ShardId::parse_str("22222222-2222-4222-8222-222222222222").unwrap(); + let binding = StreamPhysicalBinding { + stable_table_id: identity.stable_table_id, + table_incarnation_id: identity.table_incarnation_id, + table_location, + table_branch: None, + enrollment_id: enrollment_id.to_string(), + shard_ids: vec![shard_id.to_string()], + stream_config_version: STREAM_CONFIG_VERSION, + stream_config_hash: stream_config_v3_hash(), + }; + let prior_head = CurrentHeadWitness { + branch_identifier: lance::dataset::refs::BranchIdentifier::main(), + table_version: 5, + transaction_uuid: "33333333-3333-4333-8333-333333333333".to_string(), + manifest_e_tag: None, + }; + let receipt = EnrollmentReceipt::new( + "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa".to_string(), + format!("sha256:{}", "1".repeat(64)), + "44444444-4444-4444-8444-444444444444".to_string(), + binding.clone(), + ) + .unwrap(); + let prior_lifecycle = StreamLifecycleEntry::new_open_enrollment( + identity, + "node:Person".to_string(), + binding.clone(), + prior_head, + BTreeMap::from([(shard_id.to_string(), 1)]), + receipt, + ) + .unwrap(); + let base_transaction = transaction(5, "55555555-5555-4555-8555-555555555555"); + let generation_cut = RecoveryStreamFoldCut { + shard_id, + writer_epoch: 2, + shard_manifest_version: 7, + replay_after_wal_entry_position: 9, + generation: 1, + generation_path: "_mem_wal/shard/gen_1".to_string(), + }; + let lineage = RecoveryLineageIntent { + graph_commit_id: "01H000000000000000000000F1".to_string(), + branch: None, + actor_id: Some(STREAM_FOLD_ACTOR.to_string()), + merged_parent_commit_id: None, + created_at: 456, + }; + let mut planned_next_lifecycle = prior_lifecycle.clone(); + planned_next_lifecycle.current_head_witness = CurrentHeadWitness { + branch_identifier: lance::dataset::refs::BranchIdentifier::main(), + table_version: 6, + transaction_uuid: base_transaction.uuid.clone(), + manifest_e_tag: None, + }; + planned_next_lifecycle + .epoch_floor_by_shard + .insert(shard_id.to_string(), 2); + planned_next_lifecycle.lifecycle_revision = 2; + planned_next_lifecycle.last_fold_summary = Some(LastFoldSummary { + operation_id: "constructor-replaces-this".to_string(), + graph_commit_id: Some(lineage.graph_commit_id.clone()), + exact_generation_cut: StreamGenerationCut { + shard_id: shard_id.to_string(), + writer_epoch: 2, + shard_manifest_version: 7, + replay_after_wal_entry_position: 9, + generation: 1, + generation_path: "_mem_wal/shard/gen_1".to_string(), + }, + outcome: LastFoldOutcome::Published, + input_rows: 1, + input_bytes: 1, + visible_rows: 1, + visible_bytes: 1, + recorded_at: 456, + }); + let prior_token_authority = super::super::token_store::StreamTokenAuthorityEntry { + location: super::super::token_store::STREAM_TOKEN_DATASET_PATH.to_string(), + schema_version: super::super::token_store::STREAM_TOKEN_AUTHORITY_SCHEMA_VERSION, + schema_hash: super::super::token_store::stream_token_schema_hash(), + current_head_witness: CurrentHeadWitness { + branch_identifier: lance::dataset::refs::BranchIdentifier::main(), + table_version: 1, + transaction_uuid: "66666666-6666-4666-8666-666666666666".to_string(), + manifest_e_tag: None, + }, + }; + let contributor = super::super::stream_token::TrustedContributorId::new("actor:test") + .unwrap(); + let request = super::super::stream_token::AdmissionRequest { + identity, + logical_id: "person-1".to_string(), + envelope: super::super::stream_token::StreamWriteEnvelope { + stream_incarnation_id: "44444444-4444-4444-8444-444444444444".to_string(), + write_id: "88888888-8888-4888-8888-888888888888".to_string(), + predecessor_token: None, + }, + contributor_id: contributor, + payload_digest: super::super::stream_token::PayloadDigest::from_bytes([9; 32]), + }; + let candidate = request.candidate_token().unwrap(); + let metadata = super::super::stream_token::TrustedStreamRowMetadata::new_admission( + &request, + candidate, + None, + 1, + "99999999-9999-4999-8999-999999999999".to_string(), + 0, + ) + .unwrap(); + let token_rows = vec![ + super::super::stream_token::StreamTokenAuthorityRow::from_present_metadata( + identity, + "person-1".to_string(), + enrollment_id.to_string(), + &metadata, + ) + .unwrap(), + ]; + let attribution = super::super::stream_token::stream_fold_attribution_commitment( + &token_rows, + ) + .unwrap(); + new_stream_fold_sidecar_v12( + Some(STREAM_FOLD_ACTOR.to_string()), + make_pin("node:Person", "memory://test-graph/nodes/person", 5, 6), + RecoveryAuthorityToken { + branch_identifier: lance::dataset::refs::BranchIdentifier::main(), + graph_head: Some("01H000000000000000000000F0".to_string()), + schema_identity_domain: "domain-a".to_string(), + schema_ir_hash: "schema-hash".to_string(), + schema_identity_version: 1, + }, + lineage, + binding, + prior_lifecycle, + planned_next_lifecycle, + None, + generation_cut, + base_transaction, + prior_token_authority, + transaction(1, "77777777-7777-4777-8777-777777777777"), + token_rows, + attribution, + ) + .unwrap() + } + + #[tokio::test] + async fn stream_fold_v12_base_token_probe_refuses_duplicate_exact_id_rows() { + use crate::db::manifest::stream_token::{ + TrustedStreamRowMetadata, build_trusted_stream_metadata_array, + trusted_stream_metadata_field, + }; + + fn physical_batch(rows: &[(&str, Option)]) -> RecordBatch { + let schema = Arc::new(Schema::new(vec![ + Field::new("id", DataType::Utf8, false), + Field::new("age", DataType::Int32, true), + trusted_stream_metadata_field(), + ])); + let metadata = rows + .iter() + .map(|(_, metadata)| metadata.clone()) + .collect::>(); + RecordBatch::try_new( + schema, + vec![ + Arc::new(StringArray::from( + rows.iter().map(|(id, _)| *id).collect::>(), + )), + Arc::new(Int32Array::from(vec![None::; rows.len()])), + build_trusted_stream_metadata_array(&metadata).unwrap(), + ], + ) + .unwrap() + } + + let dir = tempfile::tempdir().unwrap(); + let root = dir.path().to_str().unwrap(); + let sidecar = stream_fold_sidecar_v12(); + let protocol = sidecar.protocol_v12.as_ref().unwrap(); + let planned = protocol.token.planned_rows.first().unwrap().clone(); + let metadata = TrustedStreamRowMetadata { + stream_incarnation_id: planned.stream_incarnation_id.clone(), + contributor_id: planned.contributor_id.clone(), + write_id: planned.write_id.clone(), + predecessor_token: planned.predecessor_token, + stream_token: planned.current_token, + fold_base_token: planned.fold_base_token, + chain_depth: planned.chain_depth, + origin: planned.origin.clone(), + payload_digest: planned.payload_digest, + }; + metadata + .validate_for(planned.identity, &planned.logical_id) + .unwrap(); + + let uri = format!( + "{}/{}", + root.trim_end_matches('/'), + protocol.binding.table_location.trim_start_matches('/') + ); + let store = TableStore::new(root, Arc::new(lance::session::Session::default())); + let mut dataset = TableStore::write_dataset( + &uri, + physical_batch(&[(&planned.logical_id, Some(metadata.clone()))]), + ) + .await + .unwrap(); + for logical_id in ["other-2", "other-3", "other-4", "other-5"] { + store + .append_batch(&uri, &mut dataset, physical_batch(&[(logical_id, None)])) + .await + .unwrap(); + } + store + .append_batch( + &uri, + &mut dataset, + physical_batch(&[(&planned.logical_id, Some(metadata))]), + ) + .await + .unwrap(); + assert_eq!(dataset.version().version, sidecar.tables[0].post_commit_pin); + + let error = validate_stream_fold_token_rows_against_base_v12(root, &sidecar) + .await + .expect_err("duplicate achieved base rows must block StreamFold publication"); + assert!( + matches!( + &error, + OmniError::RecoveryRequired { operation_id, .. } + if operation_id == &sidecar.operation_id + ), + "{error:?}" + ); + assert!( + error + .to_string() + .contains("more than one row per planned key"), + "{error:?}" + ); + } + + #[test] + fn stream_fold_v12_shape_keeps_token_participant_out_of_base_table_pins() { + let sidecar = stream_fold_sidecar_v12(); + assert_eq!(sidecar.schema_version, STREAM_FOLD_SIDECAR_SCHEMA_VERSION); + assert_eq!(sidecar.tables.len(), 1); + assert!(sidecar.protocol_v11.is_none()); + let protocol = sidecar.protocol_v12.as_ref().unwrap(); + assert_eq!( + protocol.token.planned_transaction.read_version, + protocol + .token + .prior_authority + .current_head_witness + .table_version + ); + assert_eq!( + protocol + .planned_next_lifecycle + .last_fold_summary + .as_ref() + .unwrap() + .operation_id, + sidecar.operation_id + ); + let body = serde_json::to_string(&sidecar).unwrap(); + let json: serde_json::Value = serde_json::from_str(&body).unwrap(); + assert!( + json["protocol_v12"].is_object(), + "boxing the in-memory payload must not change the recovery JSON wire shape" + ); + let reparsed = parse_sidecar("", &body).unwrap(); + assert!(reparsed.protocol_v12.is_some()); + + let mut commit = crate::db::manifest::GraphLineageRow { + graph_commit_id: protocol.lineage.graph_commit_id.clone(), + manifest_branch: None, + manifest_version: 17, + parent_commit_id: protocol.authority.graph_head.clone(), + merged_parent_commit_id: None, + actor_id: protocol.lineage.actor_id.clone(), + created_at: protocol.lineage.created_at, + stream_fold_attribution: Some(protocol.token.attribution_summary.clone()), + }; + assert!(stream_fold_v12_lineage_matches(&commit, protocol)); + commit.stream_fold_attribution = None; + assert!( + !stream_fold_v12_lineage_matches(&commit, protocol), + "visible-commit recovery must not accept a missing winner commitment" + ); + let mut different = protocol.token.attribution_summary.clone(); + different.winning_attribution_digest = format!("sha256:{}", "cd".repeat(32)); + commit.stream_fold_attribution = Some(different); + assert!( + !stream_fold_v12_lineage_matches(&commit, protocol), + "visible-commit recovery must require the exact fixed winner commitment" + ); + } + + #[test] + fn stream_fold_v12_shape_rejects_a_second_base_pin_and_malformed_winner_digest() { + let mut extra_pin = stream_fold_sidecar_v12(); + extra_pin + .tables + .push(make_pin("node:Company", "ignored", 5, 6)); + let error = validate_sidecar_shape("", &extra_pin).unwrap_err(); + assert!(error.to_string().contains("exactly one base-table pin")); + + let mut bad_digest = stream_fold_sidecar_v12(); + bad_digest + .protocol_v12 + .as_mut() + .unwrap() + .token + .attribution_summary + .winning_attribution_digest = "sha256:ABC".to_string(); + let error = validate_sidecar_shape("", &bad_digest).unwrap_err(); + assert!(error.to_string().contains("winning attribution summary")); + + let mut wrong_actor = stream_fold_sidecar_v12(); + wrong_actor.actor_id = Some("act-not-system".to_string()); + wrong_actor.protocol_v12.as_mut().unwrap().lineage.actor_id = + Some("act-not-system".to_string()); + let error = validate_sidecar_shape("", &wrong_actor).unwrap_err(); + assert!(error.to_string().contains(STREAM_FOLD_ACTOR)); + } + fn test_version_metadata() -> super::super::TableVersionMetadata { serde_json::from_value(serde_json::json!({ "manifest_path": "nodes/person/_versions/6.manifest", @@ -11080,7 +13261,10 @@ mod tests { let sidecar = stream_fold_sidecar(); let json = serde_json::to_string(&sidecar).unwrap(); let parsed = parse_sidecar("memory://graph/__recovery/fold-v11.json", &json).unwrap(); - assert_eq!(parsed.schema_version, STREAM_FOLD_SIDECAR_SCHEMA_VERSION); + assert_eq!( + parsed.schema_version, + LEGACY_STREAM_FOLD_SIDECAR_SCHEMA_VERSION + ); assert_eq!(parsed.writer_kind, SidecarKind::StreamFold); let protocol = parsed.protocol_v11.as_ref().unwrap(); assert_eq!(protocol.effect_phase, RecoveryEffectPhase::Armed); @@ -11339,10 +13523,111 @@ mod tests { ); } + #[tokio::test] + async fn stream_fold_v12_confirmation_binds_both_participants_in_one_rewrite() { + let dir = tempfile::tempdir().unwrap(); + let root = dir.path().to_str().unwrap(); + let storage: Arc = Arc::new(ObjectStorageAdapter::local()); + let mut sidecar = stream_fold_sidecar_v12(); + bind_sidecar_pins_to_root(&mut sidecar, root); + write_sidecar(root, storage.as_ref(), &sidecar) + .await + .unwrap(); + let protocol = sidecar.protocol_v12.as_ref().unwrap().clone(); + let achieved_base_head = crate::db::manifest::CurrentHeadWitness { + branch_identifier: lance::dataset::refs::BranchIdentifier::main(), + table_version: 6, + transaction_uuid: protocol.base.planned_transaction.uuid.clone(), + manifest_e_tag: Some("base-next-etag".to_string()), + }; + let achieved_token_head = crate::db::manifest::CurrentHeadWitness { + branch_identifier: lance::dataset::refs::BranchIdentifier::main(), + table_version: 2, + transaction_uuid: protocol.token.planned_transaction.uuid.clone(), + manifest_e_tag: None, + }; + let next_token_authority = crate::db::manifest::StreamTokenAuthorityEntry { + location: protocol.token.prior_authority.location.clone(), + schema_version: protocol.token.prior_authority.schema_version, + schema_hash: protocol.token.prior_authority.schema_hash.clone(), + current_head_witness: achieved_token_head.clone(), + }; + let update = SubTableUpdate { + identity: sidecar.tables[0].identity, + table_key: sidecar.tables[0].table_key.clone(), + table_version: 6, + table_branch: None, + row_count: 9, + version_metadata: test_version_metadata(), + }; + + confirm_stream_fold_sidecar_v12( + root, + storage.as_ref(), + &mut sidecar, + protocol.base.planned_transaction.clone(), + protocol.merged_generation.clone(), + achieved_base_head.clone(), + update.clone(), + transaction(1, "88888888-8888-4888-8888-888888888888"), + achieved_token_head.clone(), + next_token_authority.clone(), + ) + .await + .expect_err("a foreign token transaction must leave the whole sidecar Armed"); + assert_eq!( + sidecar.protocol_v12.as_ref().unwrap().effect_phase, + RecoveryEffectPhase::Armed + ); + + confirm_stream_fold_sidecar_v12( + root, + storage.as_ref(), + &mut sidecar, + protocol.base.planned_transaction.clone(), + protocol.merged_generation.clone(), + achieved_base_head.clone(), + update, + protocol.token.planned_transaction.clone(), + achieved_token_head, + next_token_authority.clone(), + ) + .await + .unwrap(); + let confirmed = sidecar.protocol_v12.as_ref().unwrap(); + assert_eq!( + confirmed.effect_phase, + RecoveryEffectPhase::EffectsConfirmed + ); + assert_eq!( + confirmed.token.next_authority.as_ref(), + Some(&next_token_authority) + ); + assert_eq!( + confirmed + .next_lifecycle + .as_ref() + .unwrap() + .current_head_witness, + achieved_base_head + ); + + let body = storage + .read_text_if_exists(&sidecar_uri(root, &sidecar.operation_id)) + .await + .unwrap() + .unwrap(); + let persisted = parse_sidecar("", &body).unwrap(); + assert_eq!( + persisted.protocol_v12.unwrap().token.next_authority, + Some(next_token_authority) + ); + } + #[tokio::test] async fn ordinary_recovery_refuses_open_stream_before_physical_classification() { use crate::db::manifest::{ - CurrentHeadWitness, STREAM_CONFIG_VERSION, StreamLifecycle, StreamLifecycleEntry, + CurrentHeadWitness, EnrollmentReceipt, STREAM_CONFIG_VERSION, StreamLifecycleEntry, StreamPhysicalBinding, }; use lance_index::mem_wal::ShardId; @@ -11357,29 +13642,42 @@ mod tests { let mut manifest = ManifestCoordinator::open(root).await.unwrap(); let entry = manifest.snapshot().entry("node:Person").unwrap().clone(); - let plan = MemWalEnrollmentPlan::new(ShardId::new_v4(), ShardId::new_v4()).unwrap(); - let lifecycle = StreamLifecycleEntry { - identity: entry.identity, - diagnostic_table_key: entry.table_key.clone(), - lifecycle: StreamLifecycle::Open, - binding: StreamPhysicalBinding { - stable_table_id: entry.identity.stable_table_id, - table_incarnation_id: entry.identity.table_incarnation_id, - table_location: entry.table_path.clone(), - table_branch: None, - enrollment_id: plan.enrollment_id.to_string(), - shard_ids: vec![plan.shard_id.to_string()], - stream_config_version: STREAM_CONFIG_VERSION, - stream_config_hash: plan.stream_config_hash(), - }, - current_head_witness: CurrentHeadWitness { + let plan = MemWalEnrollmentPlan::new( + ShardId::parse_str("11111111-1111-4111-8111-111111111111").unwrap(), + ShardId::parse_str("22222222-2222-4222-8222-222222222222").unwrap(), + ) + .unwrap(); + let binding = StreamPhysicalBinding { + stable_table_id: entry.identity.stable_table_id, + table_incarnation_id: entry.identity.table_incarnation_id, + table_location: entry.table_path.clone(), + table_branch: None, + enrollment_id: plan.enrollment_id.to_string(), + shard_ids: vec![plan.shard_id.to_string()], + stream_config_version: STREAM_CONFIG_VERSION, + stream_config_hash: plan.stream_config_hash(), + }; + let receipt = EnrollmentReceipt::new( + "33333333-3333-4333-8333-333333333333".to_string(), + format!("sha256:{}", "0".repeat(64)), + "44444444-4444-4444-8444-444444444444".to_string(), + binding.clone(), + ) + .unwrap(); + let lifecycle = StreamLifecycleEntry::new_open_enrollment( + entry.identity, + entry.table_key.clone(), + binding, + CurrentHeadWitness { branch_identifier: lance::dataset::refs::BranchIdentifier::main(), table_version: entry.table_version, transaction_uuid: ShardId::new_v4().to_string(), manifest_e_tag: None, }, - epoch_floor_by_shard: BTreeMap::from([(plan.shard_id.to_string(), 1)]), - }; + BTreeMap::from([(plan.shard_id.to_string(), 1)]), + receipt, + ) + .unwrap(); manifest .commit_changes(&[ManifestChange::SetStreamLifecycle { expected: None, diff --git a/crates/omnigraph/src/db/manifest/state.rs b/crates/omnigraph/src/db/manifest/state.rs index 5db039c5..c3b587eb 100644 --- a/crates/omnigraph/src/db/manifest/state.rs +++ b/crates/omnigraph/src/db/manifest/state.rs @@ -10,10 +10,12 @@ use crate::error::{OmniError, Result}; use super::layout::{stream_state_object_id, table_object_id, version_object_id}; use super::metadata::TableVersionMetadata; +use super::stream_token::{STREAM_FOLD_ACTOR, StreamFoldAttributionSummary}; use super::{ MAIN_BRANCH_HEAD_KEY, OBJECT_TYPE_GRAPH_COMMIT, OBJECT_TYPE_GRAPH_HEAD, - OBJECT_TYPE_STREAM_STATE, OBJECT_TYPE_TABLE, OBJECT_TYPE_TABLE_TOMBSTONE, - OBJECT_TYPE_TABLE_VERSION, StreamLifecycleEntry, TableIdentity, TableRegistration, + OBJECT_TYPE_STREAM_STATE, OBJECT_TYPE_STREAM_TOKEN_AUTHORITY, OBJECT_TYPE_TABLE, + OBJECT_TYPE_TABLE_TOMBSTONE, OBJECT_TYPE_TABLE_VERSION, StreamLifecycleEntry, + StreamTokenAuthorityEntry, TableIdentity, TableRegistration, }; #[derive(Debug, Clone)] @@ -37,6 +39,7 @@ pub(super) struct ManifestState { /// a fresh table view and the commit graph's older derived cache. pub(super) graph_heads: HashMap, pub(super) stream_lifecycles: HashMap, + pub(super) stream_token_authority: StreamTokenAuthorityEntry, } #[derive(Debug, Clone)] @@ -47,10 +50,10 @@ struct TableTombstoneEntry { } /// A graph-lineage commit projected out of the `__manifest` `graph_commit` -/// rows (RFC-013 step 4). Field-for-field identical to `commit_graph::GraphCommit` -/// so the commit-graph cache can be sourced from the manifest projection without -/// touching any reader above that boundary. Kept as a separate struct here to -/// keep `state.rs` free of the `commit_graph` module dependency. +/// rows (RFC-013 step 4). Its public-lineage fields project directly into +/// `commit_graph::GraphCommit`; protocol-private immutable metadata may remain +/// here until its product surface is deliberately activated. Kept as a separate +/// struct so `state.rs` stays free of the `commit_graph` module dependency. #[derive(Debug, Clone, PartialEq, Eq)] pub(crate) struct GraphLineageRow { pub(crate) graph_commit_id: String, @@ -60,6 +63,10 @@ pub(crate) struct GraphLineageRow { pub(crate) merged_parent_commit_id: Option, pub(crate) actor_id: Option, pub(crate) created_at: i64, + /// Compact RFC-026 commitment to the exact graph-visible fold winners. + /// Ordinary graph commits carry `None`; the complete winning rows remain + /// in the base and stream-token datasets. + pub(crate) stream_fold_attribution: Option, } /// JSON payload of a `graph_commit` row's `metadata` column. The immutable @@ -75,6 +82,33 @@ struct GraphCommitMetadata { #[serde(default, skip_serializing_if = "Option::is_none")] actor_id: Option, created_at: i64, + #[serde(default, skip_serializing_if = "Option::is_none")] + stream_fold_attribution: Option, +} + +fn validate_graph_commit_attribution(commit: &GraphLineageRow) -> Result<()> { + let fixed_actor = commit.actor_id.as_deref() == Some(STREAM_FOLD_ACTOR); + let Some(summary) = commit.stream_fold_attribution.as_ref() else { + // Config-v2/v11 folds predate this metadata field and used the same + // fixed actor. Their immutable lineage must remain readable and their + // recovery must remain publishable. Recovery-v12 separately requires + // an exact summary when deciding whether its original commit is visible. + return Ok(()); + }; + summary.validate().map_err(|error| { + OmniError::manifest_internal(format!( + "graph commit '{}' has invalid stream-fold attribution: {error}", + commit.graph_commit_id + )) + })?; + if commit.manifest_branch.is_some() || commit.merged_parent_commit_id.is_some() || !fixed_actor + { + return Err(OmniError::manifest_internal(format!( + "graph commit '{}' carries stream-fold attribution without fixed canonical-main actor '{}'", + commit.graph_commit_id, STREAM_FOLD_ACTOR + ))); + } + Ok(()) } /// JSON payload of a `graph_head` row's `metadata` column. @@ -108,6 +142,7 @@ struct ManifestScan { /// a present head from an absent one (notably on a fresh named branch). graph_heads: HashMap, stream_lifecycles: HashMap, + stream_token_authority: Option, } pub(super) fn manifest_schema() -> SchemaRef { @@ -166,6 +201,7 @@ pub(super) async fn read_manifest_state_and_lineage( lineage_rows, graph_heads, stream_lifecycles, + stream_token_authority, } = read_manifest_scan(dataset, true).await?; let state = assemble_manifest_state( version, @@ -176,6 +212,7 @@ pub(super) async fn read_manifest_state_and_lineage( .map(|t| (t.identity, t.tombstone_version)), graph_heads, stream_lifecycles, + required_stream_token_authority(stream_token_authority)?, )?; Ok((state, lineage_rows)) } @@ -190,6 +227,7 @@ fn manifest_state_from_scan(version: u64, scan: ManifestScan) -> Result, graph_heads: HashMap, stream_lifecycles: HashMap, + stream_token_authority: StreamTokenAuthorityEntry, ) -> Result { let mut latest_versions = HashMap::::new(); for entry in version_entries { @@ -292,6 +331,7 @@ pub(super) fn assemble_manifest_state( entries, graph_heads, stream_lifecycles, + stream_token_authority, }) } @@ -336,6 +376,7 @@ pub(super) struct PublishScan { /// main). Absence is meaningful and is preserved by a missing map entry. pub(super) graph_heads: HashMap, pub(super) stream_lifecycles: HashMap, + pub(super) stream_token_authority: StreamTokenAuthorityEntry, } /// One-scan read of everything the publish path needs. `collect_lineage` is @@ -354,6 +395,7 @@ pub(super) async fn read_publish_scan(dataset: &Dataset) -> Result lineage_rows: scan.lineage_rows, graph_heads: scan.graph_heads, stream_lifecycles: scan.stream_lifecycles, + stream_token_authority: required_stream_token_authority(scan.stream_token_authority)?, }) } @@ -379,7 +421,7 @@ fn decode_graph_commit_row( serde_json::from_str(metadata.value(row)).map_err(|e| { OmniError::manifest_internal(format!("failed to decode graph_commit metadata: {e}")) })?; - Ok(GraphLineageRow { + let commit = GraphLineageRow { graph_commit_id: object_ids.value(row).to_string(), manifest_branch: if branches.is_null(row) { None @@ -391,7 +433,10 @@ fn decode_graph_commit_row( merged_parent_commit_id: commit_meta.merged_parent_commit_id, actor_id: commit_meta.actor_id, created_at: commit_meta.created_at, - }) + stream_fold_attribution: commit_meta.stream_fold_attribution, + }; + validate_graph_commit_attribution(&commit)?; + Ok(commit) } /// Decode one `graph_head` row into its exact branch-key / commit-id pair. @@ -460,6 +505,7 @@ async fn read_manifest_scan(dataset: &Dataset, collect_lineage: bool) -> Result< let mut lineage_rows = Vec::new(); let mut graph_heads = HashMap::new(); let mut stream_lifecycles = HashMap::new(); + let mut stream_token_authority = None; for batch in &batches { let object_types = string_column(batch, "object_type")?; @@ -633,6 +679,36 @@ async fn read_manifest_scan(dataset: &Dataset, collect_lineage: bool) -> Result< ))); } } + OBJECT_TYPE_STREAM_TOKEN_AUTHORITY => { + require_null_table_identity( + stable_table_ids, + table_incarnation_ids, + row, + OBJECT_TYPE_STREAM_TOKEN_AUTHORITY, + )?; + if metadata.is_null(row) { + return Err(OmniError::manifest_internal( + "manifest stream_token_authority row is missing metadata", + )); + } + if !table_key.is_empty() || !row_counts.is_null(row) { + return Err(OmniError::manifest_internal( + "manifest stream_token_authority row must have an empty table_key and null row_count", + )); + } + let authority = StreamTokenAuthorityEntry::from_manifest_row( + object_ids.value(row), + (!locations.is_null(row)).then(|| locations.value(row)), + (!versions.is_null(row)).then(|| versions.value(row)), + (!branches.is_null(row)).then(|| branches.value(row)), + metadata.value(row), + )?; + if stream_token_authority.replace(authority).is_some() { + return Err(OmniError::manifest_internal( + "manifest contains duplicate stream_token_authority rows", + )); + } + } // Commit rows are skipped on the table-state path; unknown future // object types are skipped on every path. _ => {} @@ -669,6 +745,17 @@ async fn read_manifest_scan(dataset: &Dataset, collect_lineage: bool) -> Result< lineage_rows, graph_heads, stream_lifecycles, + stream_token_authority, + }) +} + +fn required_stream_token_authority( + authority: Option, +) -> Result { + authority.ok_or_else(|| { + OmniError::manifest_internal( + "manifest is missing the graph-global stream_token_authority row", + ) }) } @@ -776,11 +863,13 @@ pub(crate) fn graph_lineage_row_parts( commit: &GraphLineageRow, branch: Option<&str>, ) -> Result<[GraphLineageRowPart; 2]> { + validate_graph_commit_attribution(commit)?; let commit_metadata = serde_json::to_string(&GraphCommitMetadata { parent_commit_id: commit.parent_commit_id.clone(), merged_parent_commit_id: commit.merged_parent_commit_id.clone(), actor_id: commit.actor_id.clone(), created_at: commit.created_at, + stream_fold_attribution: commit.stream_fold_attribution.clone(), }) .map_err(|e| { OmniError::manifest_internal(format!("failed to encode graph_commit metadata: {e}")) @@ -817,8 +906,10 @@ pub(super) fn entries_to_batch( entries: &[SubTableEntry], version_metadata: &HashMap, genesis_lineage: &[GraphLineageRowPart], + stream_token_authority: &StreamTokenAuthorityEntry, ) -> Result { - let cap = entries.len() * 2 + genesis_lineage.len(); + stream_token_authority.validate()?; + let cap = entries.len() * 2 + genesis_lineage.len() + 1; let mut object_ids = Vec::with_capacity(cap); let mut object_types = Vec::with_capacity(cap); let mut locations = Vec::with_capacity(cap); @@ -878,6 +969,21 @@ pub(super) fn entries_to_batch( row_counts.push(None); } + // The graph-global token participant exists from genesis. It has no table + // identity or diagnostic alias; its exact version witness is selected by + // this singleton row and may move only through the shared publisher. + object_ids.push(stream_token_authority.object_id().to_string()); + object_types.push(OBJECT_TYPE_STREAM_TOKEN_AUTHORITY.to_string()); + locations.push(Some(stream_token_authority.location.clone())); + metadata.push(Some(stream_token_authority.to_metadata_json()?)); + table_keys.push(String::new()); + table_identities.push(None); + table_versions.push(Some( + stream_token_authority.current_head_witness.table_version, + )); + table_branches.push(None); + row_counts.push(None); + manifest_rows_batch( object_ids, object_types, @@ -978,6 +1084,35 @@ pub(super) fn manifest_rows_batch( "manifest {object_type} row at index {row} must not carry table identity" ))); } + OBJECT_TYPE_STREAM_TOKEN_AUTHORITY => { + if identity.is_some() { + return Err(OmniError::manifest_internal(format!( + "manifest {object_type} row at index {row} must not carry table identity" + ))); + } + let metadata = metadata + .get(row) + .and_then(Option::as_deref) + .ok_or_else(|| { + OmniError::manifest_internal(format!( + "manifest {object_type} row at index {row} is missing metadata" + )) + })?; + if table_keys.get(row).is_some_and(|key| !key.is_empty()) + || row_counts.get(row).copied().flatten().is_some() + { + return Err(OmniError::manifest_internal(format!( + "manifest {object_type} row at index {row} must have an empty table_key and null row_count" + ))); + } + StreamTokenAuthorityEntry::from_manifest_row( + object_ids.get(row).map(String::as_str).unwrap_or_default(), + locations.get(row).and_then(Option::as_deref), + table_versions.get(row).copied().flatten(), + table_branches.get(row).and_then(Option::as_deref), + metadata, + )?; + } _ => {} } } diff --git a/crates/omnigraph/src/db/manifest/stream.rs b/crates/omnigraph/src/db/manifest/stream.rs index cbda4d53..e9759059 100644 --- a/crates/omnigraph/src/db/manifest/stream.rs +++ b/crates/omnigraph/src/db/manifest/stream.rs @@ -1,23 +1,25 @@ //! Durable RFC-026 stream lifecycle authority. //! -//! Internal schema v8 preserves the v7 lifecycle-row representation while -//! requiring the data-bearing RFC-026 config-v2 physical profile. Public -//! streaming remains inactive; Phase B1 is reachable only through private test -//! seams. +//! Internal schema v9 cuts enrolled streams over to the RFC-026 state-v2 and +//! config-v3 wire contracts. The public lifecycle operations remain inactive, +//! but their complete durable slots are present now so a later activation does +//! not reinterpret an already-stamped graph through serde defaults. use std::collections::BTreeMap; use lance::dataset::refs::BranchIdentifier; use lance_index::mem_wal::ShardId; -use serde::{Deserialize, Serialize}; +use serde::{Deserialize, Deserializer, Serialize}; +use sha2::{Digest, Sha256}; use crate::error::{OmniError, Result}; use super::layout::stream_state_object_id; use super::{TableIdentity, TableRegistration}; -const STREAM_STATE_PROTOCOL_VERSION: u32 = 1; -pub(crate) const STREAM_CONFIG_VERSION: u32 = 2; +pub(crate) const STREAM_STATE_PROTOCOL_VERSION: u32 = 2; +pub(crate) const STREAM_CONFIG_VERSION: u32 = 3; +pub(crate) const INITIAL_LIFECYCLE_REVISION: u64 = 1; /// Stable physical enrollment binding for the bounded RFC-026 profile. /// @@ -25,15 +27,17 @@ pub(crate) const STREAM_CONFIG_VERSION: u32 = 2; /// decoder requires all copies to agree; it never derives authority from the /// diagnostic alias or from a compatible-looking path. #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] pub(crate) struct StreamPhysicalBinding { pub(crate) stable_table_id: u64, pub(crate) table_incarnation_id: u64, pub(crate) table_location: String, /// Main is represented canonically as `None`. Named refs are not supported - /// by the v8 bounded profile. + /// by the initial v9 retain-all profile. + #[serde(deserialize_with = "deserialize_present_option")] pub(crate) table_branch: Option, pub(crate) enrollment_id: String, - /// Sorted, unique UUID namespace. V8 Phase B1 permits exactly one shard. + /// Sorted, unique UUID namespace. Initial v9 activation permits one shard. pub(crate) shard_ids: Vec, pub(crate) stream_config_version: u32, pub(crate) stream_config_hash: String, @@ -58,7 +62,7 @@ impl StreamPhysicalBinding { } if self.table_branch.is_some() { return Err(OmniError::manifest_internal( - "internal schema v8 stream bindings support only canonical main (table_branch = null)", + "internal schema v9 stream bindings support only canonical main (table_branch = null)", )); } let enrollment_id = validate_uuid("enrollment_id", &self.enrollment_id)?; @@ -69,7 +73,7 @@ impl StreamPhysicalBinding { } if self.shard_ids.len() != 1 { return Err(OmniError::manifest_internal(format!( - "internal schema v8 requires exactly one stream shard, got {}", + "internal schema v9 requires exactly one stream shard, got {}", self.shard_ids.len() ))); } @@ -120,10 +124,12 @@ impl StreamPhysicalBinding { /// Exact public Lance witness for the currently accepted physical table HEAD. /// This changes on every base-table commit and is not enrollment identity. #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] pub(crate) struct CurrentHeadWitness { pub(crate) branch_identifier: BranchIdentifier, pub(crate) table_version: u64, pub(crate) transaction_uuid: String, + #[serde(deserialize_with = "deserialize_present_option")] pub(crate) manifest_e_tag: Option, } @@ -131,7 +137,7 @@ impl CurrentHeadWitness { fn validate(&self) -> Result<()> { if self.branch_identifier != BranchIdentifier::main() { return Err(OmniError::manifest_internal( - "internal schema v8 stream HEAD witness must name the main branch", + "internal schema v9 stream HEAD witness must name the main branch", )); } if self.table_version == 0 { @@ -171,8 +177,265 @@ impl StreamLifecycle { } } +/// Immutable lost-result receipt for the one enrollment that created this +/// stream incarnation. It remains unchanged across physical rebinds. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(crate) struct EnrollmentReceipt { + pub(crate) enrollment_request_id: String, + pub(crate) enrollment_intent_digest: String, + pub(crate) stream_incarnation_id: String, + pub(crate) physical_binding: StreamPhysicalBinding, + pub(crate) initial_lifecycle_revision: u64, +} + +impl EnrollmentReceipt { + pub(crate) fn new( + enrollment_request_id: String, + enrollment_intent_digest: String, + stream_incarnation_id: String, + physical_binding: StreamPhysicalBinding, + ) -> Result { + let receipt = Self { + enrollment_request_id, + enrollment_intent_digest, + stream_incarnation_id, + physical_binding, + initial_lifecycle_revision: INITIAL_LIFECYCLE_REVISION, + }; + receipt.validate()?; + Ok(receipt) + } + + pub(super) fn validate(&self) -> Result<()> { + let binding_identity = self.physical_binding.identity()?; + self.physical_binding.validate(binding_identity)?; + let request_id = validate_uuid("enrollment_request_id", &self.enrollment_request_id)?; + if request_id.get_version_num() != 4 { + return Err(OmniError::manifest_internal( + "stream enrollment_request_id must be a UUID v4 value", + )); + } + validate_digest("enrollment_intent_digest", &self.enrollment_intent_digest)?; + let stream_incarnation_id = + validate_uuid("stream_incarnation_id", &self.stream_incarnation_id)?; + if stream_incarnation_id.get_version_num() != 4 { + return Err(OmniError::manifest_internal( + "stream stream_incarnation_id must be a UUID v4 value", + )); + } + let enrollment_id = validate_uuid("enrollment_id", &self.physical_binding.enrollment_id)?; + if request_id == enrollment_id || stream_incarnation_id == enrollment_id { + return Err(OmniError::manifest_internal( + "stream request, incarnation, and physical enrollment identities must be distinct", + )); + } + if request_id == stream_incarnation_id { + return Err(OmniError::manifest_internal( + "stream enrollment_request_id and stream_incarnation_id must be distinct", + )); + } + for shard_id in &self.physical_binding.shard_ids { + let shard_id = validate_uuid("shard_id", shard_id)?; + if shard_id == request_id || shard_id == stream_incarnation_id { + return Err(OmniError::manifest_internal( + "stream logical enrollment identities must be distinct from shard identities", + )); + } + } + if self.initial_lifecycle_revision != INITIAL_LIFECYCLE_REVISION { + return Err(OmniError::manifest_internal(format!( + "stream enrollment receipt initial lifecycle revision must be {INITIAL_LIFECYCLE_REVISION}" + ))); + } + Ok(()) + } +} + +/// Terminal receipt for a successful externally initiated lifecycle request. +/// `result_payload` is the complete bounded result object; serde's ordered map +/// representation keeps the enclosing state payload deterministic. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(crate) struct ManagementReceipt { + pub(crate) operation_id: String, + pub(crate) operation_kind: String, + pub(crate) request_digest: String, + pub(crate) from_revision: u64, + pub(crate) to_revision: u64, + pub(crate) actor_id: String, + pub(crate) result_payload: serde_json::Value, + pub(crate) result_digest: String, +} + +/// The selected durable-retention contract under which a writer claim ran. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub(crate) enum ClaimProfile { + RetainAll, + ManagedReclamation, +} + +/// Exact terminal classification of one caller-visible claim invocation. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub(crate) enum ClaimAttemptClassification { + NoEffect, + AbortedNoEffect, + StockManifestOnly, + StockManifestPlusSentinel, + PatchedSentinelOnly, + PatchedSentinelPlusNamingManifest, +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(crate) struct ClaimAttemptEffect { + pub(crate) ordinal: u32, + pub(crate) attempt_id: String, + pub(crate) attempt_plan_digest: String, + pub(crate) bound_prestate_digest: String, + #[serde(deserialize_with = "deserialize_present_option")] + pub(crate) storage_envelope_digest: Option, + pub(crate) planned_sentinel_position: u64, + pub(crate) planned_sentinel_digest: String, + #[serde(deserialize_with = "deserialize_present_option")] + pub(crate) achieved_shard_manifest_version: Option, + #[serde(deserialize_with = "deserialize_present_option")] + pub(crate) achieved_writer_epoch: Option, + #[serde(deserialize_with = "deserialize_present_option")] + pub(crate) observed_sentinel_position: Option, + #[serde(deserialize_with = "deserialize_present_option")] + pub(crate) observed_sentinel_digest: Option, + pub(crate) attempt_terminal_effect_digest: String, + pub(crate) classification: ClaimAttemptClassification, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub(crate) enum ClaimTerminalClassification { + StockManifestPlusSentinel, + PatchedSentinelPlusNamingManifest, +} + +/// Complete graph-manifest-authoritative projection of one effectful claim. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(crate) struct ClaimReceipt { + pub(crate) claim_id: String, + pub(crate) recovery_operation_id: String, + pub(crate) claim_kind: String, + pub(crate) profile: ClaimProfile, + pub(crate) claim_operation_digest: String, + pub(crate) attempt_count: u32, + pub(crate) attempt_effect_chain: Vec, + pub(crate) attempt_effect_chain_digest: String, + pub(crate) terminal_attempt_id: String, + pub(crate) terminal_pre_shard_manifest_version: u64, + pub(crate) achieved_shard_manifest_version: u64, + pub(crate) achieved_writer_epoch: u64, + pub(crate) sentinel_position: u64, + pub(crate) sentinel_digest: String, + pub(crate) replay_cursor: u64, + pub(crate) terminal_effect_digest: String, + pub(crate) terminal_classification: ClaimTerminalClassification, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub(crate) enum DrainGoal { + Sealed, + OpenAfterFold, +} + +/// Durable restart plan for a revision-fenced drain. Config-v3 requires the +/// Phase-D `guarded_operation` slot to be explicit JSON null. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(crate) struct DrainDescriptor { + pub(crate) drain_id: String, + pub(crate) operation_expected_revision: u64, + pub(crate) operation_request_digest: String, + pub(crate) goal: DrainGoal, + pub(crate) initiating_actor: String, + pub(crate) initiated_at: i64, + pub(crate) expected_binding: StreamPhysicalBinding, + pub(crate) expected_current_head_witness: CurrentHeadWitness, + pub(crate) target_epoch_floor_by_shard: BTreeMap, + #[serde(deserialize_with = "deserialize_present_option")] + pub(crate) guarded_operation: Option, +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(crate) struct StrictBlock { + pub(crate) block_token: String, + pub(crate) enrollment_id: String, + pub(crate) shard_id: String, + pub(crate) generation: u64, + pub(crate) generation_path: String, + pub(crate) shard_manifest_version: u64, + pub(crate) replay_cursor: u64, + pub(crate) base_current_head_witness: CurrentHeadWitness, + pub(crate) validation_contract_version: u32, + pub(crate) violation_code: String, + pub(crate) violation_digest: String, + pub(crate) correction_view_digest: String, + pub(crate) offending_key_count: u64, + pub(crate) correction_revision: u64, +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(crate) struct SealedProof { + pub(crate) drain_id: String, + pub(crate) shard_manifest_version: u64, + pub(crate) writer_epoch: u64, + pub(crate) replay_cursor: u64, + pub(crate) current_generation: u64, + pub(crate) base_merged_generation: u64, + pub(crate) base_current_head_witness: CurrentHeadWitness, + pub(crate) current_claim_receipt_id: String, + pub(crate) claim_receipt_set_digest: String, + pub(crate) verified_empty_digest: String, +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(crate) struct StreamGenerationCut { + pub(crate) shard_id: String, + pub(crate) writer_epoch: u64, + pub(crate) shard_manifest_version: u64, + pub(crate) replay_after_wal_entry_position: u64, + pub(crate) generation: u64, + pub(crate) generation_path: String, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub(crate) enum LastFoldOutcome { + Published, + StrictBlocked, +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(crate) struct LastFoldSummary { + pub(crate) operation_id: String, + #[serde(deserialize_with = "deserialize_present_option")] + pub(crate) graph_commit_id: Option, + pub(crate) exact_generation_cut: StreamGenerationCut, + pub(crate) outcome: LastFoldOutcome, + pub(crate) input_rows: u64, + pub(crate) input_bytes: u64, + pub(crate) visible_rows: u64, + pub(crate) visible_bytes: u64, + pub(crate) recorded_at: i64, +} + /// One materialized `stream_state::` authority row. -#[derive(Debug, Clone, PartialEq, Eq)] +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] pub(crate) struct StreamLifecycleEntry { pub(crate) identity: TableIdentity, /// Human-readable only. It may lag a metadata-only rename and must never be @@ -183,9 +446,26 @@ pub(crate) struct StreamLifecycleEntry { pub(crate) current_head_witness: CurrentHeadWitness, /// Epochs are scoped to the binding's never-reused shard IDs. pub(crate) epoch_floor_by_shard: BTreeMap, + /// Monotonic state-row CAS revision. Every successful publication of this + /// row advances it exactly once, including witness-only publications. + pub(crate) lifecycle_revision: u64, + pub(crate) enrollment_receipt: EnrollmentReceipt, + pub(crate) management_receipts: Vec, + pub(crate) claim_receipts: Vec, + #[serde(deserialize_with = "deserialize_present_option")] + pub(crate) current_claim_receipt_id: Option, + #[serde(deserialize_with = "deserialize_present_option")] + pub(crate) drain: Option, + #[serde(deserialize_with = "deserialize_present_option")] + pub(crate) strict_block: Option, + #[serde(deserialize_with = "deserialize_present_option")] + pub(crate) sealed_proof: Option, + #[serde(deserialize_with = "deserialize_present_option")] + pub(crate) last_fold_summary: Option, } #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] struct StreamStatePayload { protocol_version: u32, stable_table_id: u64, @@ -194,9 +474,456 @@ struct StreamStatePayload { binding: StreamPhysicalBinding, current_head_witness: CurrentHeadWitness, epoch_floor_by_shard: BTreeMap, + lifecycle_revision: u64, + enrollment_receipt: EnrollmentReceipt, + management_receipts: Vec, + claim_receipts: Vec, + #[serde(deserialize_with = "deserialize_present_option")] + current_claim_receipt_id: Option, + #[serde(deserialize_with = "deserialize_present_option")] + drain: Option, + #[serde(deserialize_with = "deserialize_present_option")] + strict_block: Option, + #[serde(deserialize_with = "deserialize_present_option")] + sealed_proof: Option, + #[serde(deserialize_with = "deserialize_present_option")] + last_fold_summary: Option, +} + +impl ManagementReceipt { + fn validate(&self, current_revision: u64) -> Result<()> { + validate_uuid("management operation_id", &self.operation_id)?; + validate_protocol_label("management operation_kind", &self.operation_kind)?; + validate_digest("management request_digest", &self.request_digest)?; + if self.from_revision == 0 + || self.to_revision <= self.from_revision + || self.to_revision > current_revision + { + return Err(OmniError::manifest_internal(format!( + "stream management receipt revision range {}..{} is invalid at lifecycle revision {current_revision}", + self.from_revision, self.to_revision + ))); + } + validate_canonical_text("management actor_id", &self.actor_id)?; + if !self.result_payload.is_object() { + return Err(OmniError::manifest_internal( + "stream management receipt result_payload must be a JSON object", + )); + } + validate_digest("management result_digest", &self.result_digest) + } +} + +impl ClaimAttemptEffect { + fn validate(&self, profile: ClaimProfile, expected_ordinal: u32) -> Result<()> { + if self.ordinal != expected_ordinal { + return Err(OmniError::manifest_internal(format!( + "stream claim attempt ordinal {} is not the expected contiguous ordinal {expected_ordinal}", + self.ordinal + ))); + } + let attempt_id = validate_uuid("claim attempt_id", &self.attempt_id)?; + if attempt_id.get_version_num() != 4 { + return Err(OmniError::manifest_internal( + "stream claim attempt_id must be a UUID v4 value", + )); + } + validate_digest("claim attempt_plan_digest", &self.attempt_plan_digest)?; + validate_digest("claim bound_prestate_digest", &self.bound_prestate_digest)?; + match (profile, self.storage_envelope_digest.as_deref()) { + (ClaimProfile::RetainAll, None) => {} + (ClaimProfile::RetainAll, Some(_)) => { + return Err(OmniError::manifest_internal( + "retain-all claim attempts cannot carry a managed-reclamation storage envelope", + )); + } + (ClaimProfile::ManagedReclamation, Some(digest)) => { + validate_digest("claim storage_envelope_digest", digest)?; + } + (ClaimProfile::ManagedReclamation, None) => { + return Err(OmniError::manifest_internal( + "managed-reclamation claim attempts require a storage envelope digest", + )); + } + } + if self.planned_sentinel_position == 0 { + return Err(OmniError::manifest_internal( + "stream claim planned sentinel position must be non-zero", + )); + } + validate_digest( + "claim planned_sentinel_digest", + &self.planned_sentinel_digest, + )?; + validate_digest( + "claim attempt_terminal_effect_digest", + &self.attempt_terminal_effect_digest, + )?; + + let achieved_manifest = match ( + self.achieved_shard_manifest_version, + self.achieved_writer_epoch, + ) { + (Some(version), Some(epoch)) if version > 0 && epoch > 0 => true, + (None, None) => false, + _ => { + return Err(OmniError::manifest_internal( + "stream claim attempt must carry achieved manifest version and writer epoch together and non-zero", + )); + } + }; + let observed_sentinel = match ( + self.observed_sentinel_position, + self.observed_sentinel_digest.as_deref(), + ) { + (Some(position), Some(digest)) if position > 0 => { + validate_digest("claim observed_sentinel_digest", digest)?; + if position != self.planned_sentinel_position + || digest != self.planned_sentinel_digest + { + return Err(OmniError::manifest_internal( + "stream claim observed sentinel differs from its pre-armed plan", + )); + } + true + } + (None, None) => false, + _ => { + return Err(OmniError::manifest_internal( + "stream claim attempt must carry observed sentinel position and digest together", + )); + } + }; + let expected_effects = match self.classification { + ClaimAttemptClassification::NoEffect | ClaimAttemptClassification::AbortedNoEffect => { + (false, false) + } + ClaimAttemptClassification::StockManifestOnly => (true, false), + ClaimAttemptClassification::StockManifestPlusSentinel + | ClaimAttemptClassification::PatchedSentinelPlusNamingManifest => (true, true), + ClaimAttemptClassification::PatchedSentinelOnly => (false, true), + }; + if (achieved_manifest, observed_sentinel) != expected_effects { + return Err(OmniError::manifest_internal(format!( + "stream claim attempt effect fields disagree with classification {:?}", + self.classification + ))); + } + Ok(()) + } +} + +impl ClaimReceipt { + fn validate(&self) -> Result<()> { + let claim_id = validate_uuid("claim_id", &self.claim_id)?; + if claim_id.get_version_num() != 4 { + return Err(OmniError::manifest_internal( + "stream claim_id must be a UUID v4 value", + )); + } + validate_canonical_text("claim recovery_operation_id", &self.recovery_operation_id)?; + validate_protocol_label("claim_kind", &self.claim_kind)?; + validate_digest("claim_operation_digest", &self.claim_operation_digest)?; + validate_digest( + "claim attempt_effect_chain_digest", + &self.attempt_effect_chain_digest, + )?; + validate_digest("claim sentinel_digest", &self.sentinel_digest)?; + validate_digest("claim terminal_effect_digest", &self.terminal_effect_digest)?; + if self.attempt_count == 0 + || usize::try_from(self.attempt_count).ok() != Some(self.attempt_effect_chain.len()) + { + return Err(OmniError::manifest_internal( + "stream claim attempt_count must exactly match a non-empty attempt_effect_chain", + )); + } + let mut attempt_ids = std::collections::BTreeSet::new(); + for (index, attempt) in self.attempt_effect_chain.iter().enumerate() { + let ordinal = u32::try_from(index + 1).map_err(|_| { + OmniError::manifest_internal("stream claim attempt ordinal overflow") + })?; + attempt.validate(self.profile, ordinal)?; + if !attempt_ids.insert(attempt.attempt_id.as_str()) { + return Err(OmniError::manifest_internal( + "stream claim attempt IDs must be unique within a receipt", + )); + } + } + let terminal = self + .attempt_effect_chain + .last() + .expect("non-empty chain checked above"); + let expected_terminal_classification = match self.terminal_classification { + ClaimTerminalClassification::StockManifestPlusSentinel => { + ClaimAttemptClassification::StockManifestPlusSentinel + } + ClaimTerminalClassification::PatchedSentinelPlusNamingManifest => { + ClaimAttemptClassification::PatchedSentinelPlusNamingManifest + } + }; + if terminal.attempt_id != self.terminal_attempt_id + || terminal.classification != expected_terminal_classification + || terminal.achieved_shard_manifest_version + != Some(self.achieved_shard_manifest_version) + || terminal.achieved_writer_epoch != Some(self.achieved_writer_epoch) + || terminal.observed_sentinel_position != Some(self.sentinel_position) + || terminal.observed_sentinel_digest.as_deref() != Some(self.sentinel_digest.as_str()) + || terminal.attempt_terminal_effect_digest != self.terminal_effect_digest + { + return Err(OmniError::manifest_internal( + "stream claim terminal receipt fields do not match the final classified attempt", + )); + } + if self.terminal_pre_shard_manifest_version == 0 + || self.achieved_shard_manifest_version <= self.terminal_pre_shard_manifest_version + || self.achieved_writer_epoch == 0 + || self.sentinel_position == 0 + || self.replay_cursor > self.sentinel_position + { + return Err(OmniError::manifest_internal( + "stream claim terminal manifest, epoch, sentinel, or replay cursor is invalid", + )); + } + Ok(()) + } +} + +impl DrainDescriptor { + fn validate(&self, entry: &StreamLifecycleEntry) -> Result<()> { + validate_uuid("drain_id", &self.drain_id)?; + if self.operation_expected_revision == 0 + || self.operation_expected_revision >= entry.lifecycle_revision + { + return Err(OmniError::manifest_internal( + "stream drain operation_expected_revision must precede the current lifecycle revision", + )); + } + validate_digest( + "drain operation_request_digest", + &self.operation_request_digest, + )?; + validate_canonical_text("drain initiating_actor", &self.initiating_actor)?; + if self.initiated_at <= 0 { + return Err(OmniError::manifest_internal( + "stream drain initiated_at must be a positive timestamp", + )); + } + self.expected_binding.validate(entry.identity)?; + self.expected_current_head_witness.validate()?; + validate_epoch_floors( + &self.expected_binding, + &self.target_epoch_floor_by_shard, + "drain target", + )?; + for (shard_id, target_epoch) in &self.target_epoch_floor_by_shard { + if target_epoch + < entry + .epoch_floor_by_shard + .get(shard_id) + .expect("both maps exactly match the binding") + { + return Err(OmniError::manifest_internal( + "stream drain target epoch cannot move behind current shard authority", + )); + } + } + if self.guarded_operation.is_some() { + return Err(OmniError::manifest_internal( + "stream config-v3 drain guarded_operation must be null", + )); + } + Ok(()) + } +} + +impl StrictBlock { + fn validate(&self, entry: &StreamLifecycleEntry) -> Result<()> { + validate_digest("strict block_token", &self.block_token)?; + if self.enrollment_id != entry.binding.enrollment_id { + return Err(OmniError::manifest_internal( + "stream strict block enrollment_id differs from the current binding", + )); + } + validate_uuid("strict block enrollment_id", &self.enrollment_id)?; + let shard_id = validate_uuid("strict block shard_id", &self.shard_id)?; + if !entry + .binding + .shard_ids + .iter() + .any(|bound| bound == &shard_id.to_string()) + { + return Err(OmniError::manifest_internal( + "stream strict block shard_id is not present in the current binding", + )); + } + validate_canonical_text("strict block generation_path", &self.generation_path)?; + self.base_current_head_witness.validate()?; + if self.generation == 0 + || self.shard_manifest_version == 0 + || self.validation_contract_version == 0 + || self.correction_revision > entry.lifecycle_revision + { + return Err(OmniError::manifest_internal( + "stream strict block carries an invalid generation, contract, or correction revision", + )); + } + validate_protocol_label("strict block violation_code", &self.violation_code)?; + validate_digest("strict block violation_digest", &self.violation_digest)?; + validate_digest( + "strict block correction_view_digest", + &self.correction_view_digest, + )?; + Ok(()) + } +} + +impl SealedProof { + fn validate(&self, entry: &StreamLifecycleEntry) -> Result<()> { + validate_uuid("sealed proof drain_id", &self.drain_id)?; + validate_uuid( + "sealed proof current_claim_receipt_id", + &self.current_claim_receipt_id, + )?; + if self.shard_manifest_version == 0 + || self.writer_epoch == 0 + || self.current_generation < self.base_merged_generation + { + return Err(OmniError::manifest_internal( + "stream sealed proof carries an invalid manifest, epoch, or generation cut", + )); + } + self.base_current_head_witness.validate()?; + if self.base_current_head_witness != entry.current_head_witness + || entry.current_claim_receipt_id.as_deref() + != Some(self.current_claim_receipt_id.as_str()) + || entry + .epoch_floor_by_shard + .values() + .any(|epoch| *epoch != self.writer_epoch) + { + return Err(OmniError::manifest_internal( + "stream sealed proof does not match the current table or claim authority", + )); + } + let current_claim = entry + .claim_receipts + .iter() + .find(|receipt| receipt.claim_id == self.current_claim_receipt_id) + .ok_or_else(|| { + OmniError::manifest_internal( + "stream sealed proof current claim receipt is not retained", + ) + })?; + if current_claim.achieved_shard_manifest_version != self.shard_manifest_version + || current_claim.achieved_writer_epoch != self.writer_epoch + { + return Err(OmniError::manifest_internal( + "stream sealed proof shard authority differs from its current claim receipt", + )); + } + validate_digest( + "sealed proof claim_receipt_set_digest", + &self.claim_receipt_set_digest, + )?; + validate_digest( + "sealed proof verified_empty_digest", + &self.verified_empty_digest, + ) + } +} + +impl StreamGenerationCut { + fn validate(&self) -> Result<()> { + validate_uuid("fold cut shard_id", &self.shard_id)?; + validate_canonical_text("fold cut generation_path", &self.generation_path)?; + if self.writer_epoch == 0 || self.shard_manifest_version == 0 || self.generation == 0 { + return Err(OmniError::manifest_internal( + "stream fold cut epoch, shard-manifest version, and generation must be non-zero", + )); + } + Ok(()) + } +} + +impl LastFoldSummary { + fn validate(&self, entry: &StreamLifecycleEntry) -> Result<()> { + validate_canonical_text("last fold operation_id", &self.operation_id)?; + self.exact_generation_cut.validate()?; + if !entry + .binding + .shard_ids + .contains(&self.exact_generation_cut.shard_id) + { + return Err(OmniError::manifest_internal( + "stream last-fold cut shard is not present in the current binding", + )); + } + if self.visible_rows > self.input_rows || self.recorded_at <= 0 { + return Err(OmniError::manifest_internal( + "stream last-fold row counts or timestamp are invalid", + )); + } + match (self.outcome, self.graph_commit_id.as_deref()) { + (LastFoldOutcome::Published, Some(commit_id)) => { + validate_canonical_text("last fold graph_commit_id", commit_id)?; + } + (LastFoldOutcome::StrictBlocked, None) + if self.visible_rows == 0 && self.visible_bytes == 0 => {} + (LastFoldOutcome::Published, None) => { + return Err(OmniError::manifest_internal( + "a published stream fold summary requires graph_commit_id", + )); + } + (LastFoldOutcome::StrictBlocked, _) => { + return Err(OmniError::manifest_internal( + "a strict-blocked stream fold must have no graph commit and zero visible output", + )); + } + } + Ok(()) + } } impl StreamLifecycleEntry { + /// Build the first OPEN state-v2 row. Every value is fixed before the + /// enrollment recovery sidecar is armed; crash recovery publishes this + /// exact receipt rather than inventing logical identity from a physical + /// effect observed later. + #[allow(clippy::too_many_arguments)] + pub(crate) fn new_open_enrollment( + identity: TableIdentity, + diagnostic_table_key: String, + binding: StreamPhysicalBinding, + current_head_witness: CurrentHeadWitness, + epoch_floor_by_shard: BTreeMap, + enrollment_receipt: EnrollmentReceipt, + ) -> Result { + if enrollment_receipt.physical_binding != binding { + return Err(OmniError::manifest_internal( + "initial stream lifecycle binding must exactly match its enrollment receipt", + )); + } + let entry = Self { + identity, + diagnostic_table_key, + lifecycle: StreamLifecycle::Open, + binding, + current_head_witness, + epoch_floor_by_shard, + lifecycle_revision: INITIAL_LIFECYCLE_REVISION, + enrollment_receipt, + management_receipts: Vec::new(), + claim_receipts: Vec::new(), + current_claim_receipt_id: None, + drain: None, + strict_block: None, + sealed_proof: None, + last_fold_summary: None, + }; + entry.validate()?; + Ok(entry) + } + pub(crate) fn object_id(&self) -> String { stream_state_object_id(self.identity) } @@ -210,24 +937,154 @@ impl StreamLifecycleEntry { } self.binding.validate(self.identity)?; self.current_head_witness.validate()?; - - if self.epoch_floor_by_shard.len() != self.binding.shard_ids.len() { + validate_epoch_floors(&self.binding, &self.epoch_floor_by_shard, "current")?; + if self.lifecycle_revision < INITIAL_LIFECYCLE_REVISION { return Err(OmniError::manifest_internal( - "stream epoch-floor keys must exactly match the physical shard binding", + "stream lifecycle_revision must be non-zero", )); } - for shard_id in &self.binding.shard_ids { - let epoch = self.epoch_floor_by_shard.get(shard_id).ok_or_else(|| { - OmniError::manifest_internal(format!( - "stream epoch floor missing bound shard {shard_id}" - )) - })?; - if *epoch == 0 { - return Err(OmniError::manifest_internal(format!( - "stream epoch floor for shard {shard_id} must record a claimed non-zero epoch" - ))); + self.enrollment_receipt.validate()?; + if self.enrollment_receipt.physical_binding.identity()? != self.identity { + return Err(OmniError::manifest_internal( + "stream enrollment receipt binding identity differs from its lifecycle row", + )); + } + if self.enrollment_receipt.initial_lifecycle_revision > self.lifecycle_revision { + return Err(OmniError::manifest_internal( + "stream enrollment receipt begins after the current lifecycle revision", + )); + } + + let mut management_occurrences = std::collections::BTreeSet::new(); + let mut prior_management_terminal_revision = 0; + for receipt in &self.management_receipts { + receipt.validate(self.lifecycle_revision)?; + if !management_occurrences.insert(( + receipt.operation_kind.as_str(), + receipt.operation_id.as_str(), + )) { + return Err(OmniError::manifest_internal( + "stream management receipt occurrences must be unique", + )); + } + if receipt.to_revision <= prior_management_terminal_revision { + return Err(OmniError::manifest_internal( + "stream management receipt history must be ordered by increasing terminal revision", + )); + } + prior_management_terminal_revision = receipt.to_revision; + } + + let mut claim_ids = std::collections::BTreeSet::new(); + let mut greatest_claim_epoch = None; + for receipt in &self.claim_receipts { + receipt.validate()?; + if !claim_ids.insert(receipt.claim_id.as_str()) { + return Err(OmniError::manifest_internal( + "stream claim receipt IDs must be unique", + )); + } + if greatest_claim_epoch + .is_some_and(|prior_epoch| prior_epoch >= receipt.achieved_writer_epoch) + { + return Err(OmniError::manifest_internal( + "stream claim receipt history must be ordered by strictly increasing writer epoch", + )); + } + greatest_claim_epoch = Some(receipt.achieved_writer_epoch); + } + match ( + self.current_claim_receipt_id.as_deref(), + greatest_claim_epoch, + ) { + (None, None) => {} + (Some(current_id), Some(greatest_epoch)) => { + validate_uuid("current_claim_receipt_id", current_id)?; + let current = self + .claim_receipts + .iter() + .find(|receipt| receipt.claim_id == current_id) + .ok_or_else(|| { + OmniError::manifest_internal( + "stream current_claim_receipt_id does not name retained claim history", + ) + })?; + if current.achieved_writer_epoch != greatest_epoch { + return Err(OmniError::manifest_internal( + "stream current claim receipt does not carry the greatest achieved epoch", + )); + } + if self + .epoch_floor_by_shard + .values() + .any(|epoch| *epoch != current.achieved_writer_epoch) + { + return Err(OmniError::manifest_internal( + "stream current claim receipt epoch differs from the current shard epoch floor", + )); + } + } + _ => { + return Err(OmniError::manifest_internal( + "stream claim history and current_claim_receipt_id must be absent or present together", + )); } } + + match self.lifecycle { + StreamLifecycle::Open => { + if self.drain.is_some() + || self.strict_block.is_some() + || self.sealed_proof.is_some() + { + return Err(OmniError::manifest_internal( + "OPEN stream lifecycle cannot carry drain, strict-block, or sealed-proof state", + )); + } + } + StreamLifecycle::Draining => { + let drain = self.drain.as_ref().ok_or_else(|| { + OmniError::manifest_internal( + "DRAINING stream lifecycle requires one drain descriptor", + ) + })?; + drain.validate(self)?; + if drain.expected_binding != self.binding + || drain.expected_current_head_witness != self.current_head_witness + || self.sealed_proof.is_some() + { + return Err(OmniError::manifest_internal( + "DRAINING stream authority disagrees with its current binding/HEAD or carries a sealed proof", + )); + } + if let Some(block) = &self.strict_block { + block.validate(self)?; + if block.base_current_head_witness != self.current_head_witness { + return Err(OmniError::manifest_internal( + "stream strict block base witness differs from current DRAINING authority", + )); + } + } + } + StreamLifecycle::Sealed => { + if self.drain.is_some() || self.strict_block.is_some() { + return Err(OmniError::manifest_internal( + "SEALED stream lifecycle cannot retain drain or strict-block state", + )); + } + self.sealed_proof + .as_ref() + .ok_or_else(|| { + OmniError::manifest_internal( + "SEALED stream lifecycle requires one exact empty proof", + ) + })? + .validate(self)?; + } + } + if let Some(summary) = &self.last_fold_summary { + summary.validate(self)?; + } Ok(()) } @@ -261,6 +1118,15 @@ impl StreamLifecycleEntry { binding: self.binding.clone(), current_head_witness: self.current_head_witness.clone(), epoch_floor_by_shard: self.epoch_floor_by_shard.clone(), + lifecycle_revision: self.lifecycle_revision, + enrollment_receipt: self.enrollment_receipt.clone(), + management_receipts: self.management_receipts.clone(), + claim_receipts: self.claim_receipts.clone(), + current_claim_receipt_id: self.current_claim_receipt_id.clone(), + drain: self.drain.clone(), + strict_block: self.strict_block.clone(), + sealed_proof: self.sealed_proof.clone(), + last_fold_summary: self.last_fold_summary.clone(), }) .map_err(|error| { OmniError::manifest_internal(format!( @@ -310,6 +1176,15 @@ impl StreamLifecycleEntry { binding: payload.binding, current_head_witness: payload.current_head_witness, epoch_floor_by_shard: payload.epoch_floor_by_shard, + lifecycle_revision: payload.lifecycle_revision, + enrollment_receipt: payload.enrollment_receipt, + management_receipts: payload.management_receipts, + claim_receipts: payload.claim_receipts, + current_claim_receipt_id: payload.current_claim_receipt_id, + drain: payload.drain, + strict_block: payload.strict_block, + sealed_proof: payload.sealed_proof, + last_fold_summary: payload.last_fold_summary, }; entry.validate()?; if location != Some(entry.binding.table_location.as_str()) { @@ -331,6 +1206,228 @@ impl StreamLifecycleEntry { } } +/// Canonical digest of the caller intent fixed before enrollment has any +/// physical effect. Engine-minted enrollment, shard, and stream-incarnation +/// identities are results and deliberately do not participate. Every text +/// field is length-prefixed and every integer is fixed-width big endian. +#[allow(clippy::too_many_arguments)] +pub(crate) fn stream_enrollment_intent_digest_v1( + identity: TableIdentity, + table_location: &str, + schema_identity_domain: &str, + schema_ir_hash: &str, + schema_identity_version: u32, + expected_unenrolled_head: &CurrentHeadWitness, + stream_config_hash: &str, +) -> Result { + identity.validate()?; + validate_canonical_text("enrollment intent table_location", table_location)?; + validate_canonical_text( + "enrollment intent schema_identity_domain", + schema_identity_domain, + )?; + validate_canonical_text("enrollment intent schema_ir_hash", schema_ir_hash)?; + if schema_identity_version == 0 { + return Err(OmniError::manifest_internal( + "stream enrollment intent schema_identity_version must be non-zero", + )); + } + expected_unenrolled_head.validate()?; + validate_digest("enrollment intent stream_config_hash", stream_config_hash)?; + + let mut hasher = Sha256::new(); + hash_bytes(&mut hasher, b"omnigraph.stream-enrollment-intent.v1"); + hasher.update(identity.stable_table_id.to_be_bytes()); + hasher.update(identity.table_incarnation_id.to_be_bytes()); + hash_bytes(&mut hasher, table_location.as_bytes()); + // Config-v3 currently permits canonical main only; bind that fact rather + // than serializing Lance's implementation-owned branch identifier. + hash_bytes(&mut hasher, b"main"); + hash_bytes(&mut hasher, schema_identity_domain.as_bytes()); + hash_bytes(&mut hasher, schema_ir_hash.as_bytes()); + hasher.update(schema_identity_version.to_be_bytes()); + hasher.update(expected_unenrolled_head.table_version.to_be_bytes()); + hash_bytes( + &mut hasher, + expected_unenrolled_head.transaction_uuid.as_bytes(), + ); + match expected_unenrolled_head.manifest_e_tag.as_deref() { + Some(e_tag) => { + hasher.update([1]); + hash_bytes(&mut hasher, e_tag.as_bytes()); + } + None => hasher.update([0]), + } + hasher.update(STREAM_CONFIG_VERSION.to_be_bytes()); + hash_bytes(&mut hasher, stream_config_hash.as_bytes()); + Ok(format!("sha256:{:x}", hasher.finalize())) +} + +fn hash_bytes(hasher: &mut Sha256, bytes: &[u8]) { + hasher.update((bytes.len() as u64).to_be_bytes()); + hasher.update(bytes); +} + +/// `Option` normally treats an absent serde field as `None`. State-v2 needs +/// every nullable slot to be physically present so a truncated or older row is +/// never reinterpreted by a default. +fn deserialize_present_option<'de, D, T>( + deserializer: D, +) -> std::result::Result, D::Error> +where + D: Deserializer<'de>, + T: Deserialize<'de>, +{ + Option::::deserialize(deserializer) +} + +fn validate_epoch_floors( + binding: &StreamPhysicalBinding, + floors: &BTreeMap, + context: &str, +) -> Result<()> { + if floors.len() != binding.shard_ids.len() { + return Err(OmniError::manifest_internal(format!( + "stream {context} epoch-floor keys must exactly match the physical shard binding" + ))); + } + for shard_id in &binding.shard_ids { + let epoch = floors.get(shard_id).ok_or_else(|| { + OmniError::manifest_internal(format!( + "stream {context} epoch floor missing bound shard {shard_id}" + )) + })?; + if *epoch == 0 { + return Err(OmniError::manifest_internal(format!( + "stream {context} epoch floor for shard {shard_id} must be non-zero" + ))); + } + } + Ok(()) +} + +/// Build the smallest state-v2-valid SEALED authority used by manifest and +/// branch-control unit fixtures. Production lifecycle management remains +/// inactive; keeping this constructor test-only prevents fixtures from +/// bypassing the exact claim and empty-proof requirements by mutating only the +/// lifecycle enum. +#[cfg(test)] +pub(crate) fn test_sealed_lifecycle_from( + current: &StreamLifecycleEntry, +) -> Result { + let claim_id = "77777777-7777-4777-8777-777777777777".to_string(); + let attempt_id = "88888888-8888-4888-8888-888888888888".to_string(); + let sentinel_digest = format!("sha256:{}", "c".repeat(64)); + let terminal_effect_digest = format!("sha256:{}", "d".repeat(64)); + let attempt = ClaimAttemptEffect { + ordinal: 1, + attempt_id: attempt_id.clone(), + attempt_plan_digest: format!("sha256:{}", "a".repeat(64)), + bound_prestate_digest: format!("sha256:{}", "b".repeat(64)), + storage_envelope_digest: None, + planned_sentinel_position: 1, + planned_sentinel_digest: sentinel_digest.clone(), + achieved_shard_manifest_version: Some(2), + achieved_writer_epoch: Some(1), + observed_sentinel_position: Some(1), + observed_sentinel_digest: Some(sentinel_digest.clone()), + attempt_terminal_effect_digest: terminal_effect_digest.clone(), + classification: ClaimAttemptClassification::StockManifestPlusSentinel, + }; + let claim = ClaimReceipt { + claim_id: claim_id.clone(), + recovery_operation_id: "test-seal".to_string(), + claim_kind: "TEST_SEAL".to_string(), + profile: ClaimProfile::RetainAll, + claim_operation_digest: format!("sha256:{}", "e".repeat(64)), + attempt_count: 1, + attempt_effect_chain: vec![attempt], + attempt_effect_chain_digest: format!("sha256:{}", "f".repeat(64)), + terminal_attempt_id: attempt_id, + terminal_pre_shard_manifest_version: 1, + achieved_shard_manifest_version: 2, + achieved_writer_epoch: 1, + sentinel_position: 1, + sentinel_digest, + replay_cursor: 1, + terminal_effect_digest, + terminal_classification: ClaimTerminalClassification::StockManifestPlusSentinel, + }; + + let mut sealed = current.clone(); + sealed.lifecycle = StreamLifecycle::Sealed; + sealed.lifecycle_revision = sealed.lifecycle_revision.checked_add(1).ok_or_else(|| { + OmniError::manifest_internal("test sealed lifecycle revision overflow") + })?; + sealed.claim_receipts = vec![claim]; + sealed.current_claim_receipt_id = Some(claim_id.clone()); + for epoch in sealed.epoch_floor_by_shard.values_mut() { + *epoch = 1; + } + let drain_id = sealed + .drain + .as_ref() + .map(|drain| drain.drain_id.clone()) + .unwrap_or_else(|| "99999999-9999-4999-8999-999999999999".to_string()); + sealed.drain = None; + sealed.strict_block = None; + sealed.sealed_proof = Some(SealedProof { + drain_id, + shard_manifest_version: 2, + writer_epoch: 1, + replay_cursor: 1, + current_generation: 1, + base_merged_generation: 0, + base_current_head_witness: sealed.current_head_witness.clone(), + current_claim_receipt_id: claim_id, + claim_receipt_set_digest: format!("sha256:{}", "a".repeat(64)), + verified_empty_digest: format!("sha256:{}", "b".repeat(64)), + }); + sealed.validate()?; + Ok(sealed) +} + +fn validate_digest(field: &str, value: &str) -> Result<()> { + let Some(digest) = value.strip_prefix("sha256:") else { + return Err(OmniError::manifest_internal(format!( + "stream {field} must use canonical sha256: form" + ))); + }; + if digest.len() != 64 + || !digest + .bytes() + .all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte)) + { + return Err(OmniError::manifest_internal(format!( + "stream {field} must contain exactly 64 lowercase hexadecimal digits" + ))); + } + Ok(()) +} + +fn validate_canonical_text(field: &str, value: &str) -> Result<()> { + if value.is_empty() || value.trim() != value { + return Err(OmniError::manifest_internal(format!( + "stream {field} must be non-empty canonical text" + ))); + } + Ok(()) +} + +fn validate_protocol_label(field: &str, value: &str) -> Result<()> { + validate_canonical_text(field, value)?; + if !value.bytes().enumerate().all(|(index, byte)| match byte { + b'A'..=b'Z' => true, + b'0'..=b'9' | b'_' => index > 0, + _ => false, + }) { + return Err(OmniError::manifest_internal(format!( + "stream {field} must use canonical SCREAMING_SNAKE_CASE text" + ))); + } + Ok(()) +} + fn validate_uuid(field: &str, value: &str) -> Result { let parsed = ShardId::parse_str(value).map_err(|error| { OmniError::manifest_internal(format!("stream {field} is not a UUID: {error}")) @@ -354,20 +1451,28 @@ mod tests { fn entry() -> StreamLifecycleEntry { let shard_id = "22222222-2222-4222-8222-222222222222".to_string(); + let binding = StreamPhysicalBinding { + stable_table_id: 7, + table_incarnation_id: 9, + table_location: "nodes/0000000000000007-0000000000000009".to_string(), + table_branch: None, + enrollment_id: "11111111-1111-4111-8111-111111111111".to_string(), + shard_ids: vec![shard_id.clone()], + stream_config_version: STREAM_CONFIG_VERSION, + stream_config_hash: format!("sha256:{}", "a".repeat(64)), + }; + let enrollment_receipt = EnrollmentReceipt::new( + "44444444-4444-4444-8444-444444444444".to_string(), + format!("sha256:{}", "b".repeat(64)), + "55555555-5555-4555-8555-555555555555".to_string(), + binding.clone(), + ) + .unwrap(); StreamLifecycleEntry { identity: TableIdentity::new(7, 9).unwrap(), diagnostic_table_key: "node:Person".to_string(), lifecycle: StreamLifecycle::Open, - binding: StreamPhysicalBinding { - stable_table_id: 7, - table_incarnation_id: 9, - table_location: "nodes/0000000000000007-0000000000000009".to_string(), - table_branch: None, - enrollment_id: "11111111-1111-4111-8111-111111111111".to_string(), - shard_ids: vec![shard_id.clone()], - stream_config_version: STREAM_CONFIG_VERSION, - stream_config_hash: format!("sha256:{}", "a".repeat(64)), - }, + binding, current_head_witness: CurrentHeadWitness { branch_identifier: BranchIdentifier::main(), table_version: 4, @@ -375,6 +1480,15 @@ mod tests { manifest_e_tag: None, }, epoch_floor_by_shard: BTreeMap::from([(shard_id, 1)]), + lifecycle_revision: INITIAL_LIFECYCLE_REVISION, + enrollment_receipt, + management_receipts: Vec::new(), + claim_receipts: Vec::new(), + current_claim_receipt_id: None, + drain: None, + strict_block: None, + sealed_proof: None, + last_fold_summary: None, } } @@ -396,6 +1510,8 @@ mod tests { assert_eq!(decoded.identity, entry.identity); assert_eq!(decoded.diagnostic_table_key, "node:RenamedPerson"); assert_eq!(decoded.binding, entry.binding); + assert_eq!(decoded.lifecycle_revision, INITIAL_LIFECYCLE_REVISION); + assert_eq!(decoded.enrollment_receipt, entry.enrollment_receipt); assert!( StreamLifecycleEntry::from_manifest_row( @@ -423,6 +1539,139 @@ mod tests { ); } + #[test] + fn state_v2_serializes_every_inert_slot_and_rejects_omission_or_v1() { + let entry = entry(); + let json = entry.to_metadata_json().unwrap(); + let payload: serde_json::Value = serde_json::from_str(&json).unwrap(); + assert_eq!(payload["protocol_version"], 2); + assert_eq!(payload["lifecycle_revision"], 1); + assert!( + payload["management_receipts"] + .as_array() + .unwrap() + .is_empty() + ); + assert!(payload["claim_receipts"].as_array().unwrap().is_empty()); + for field in [ + "current_claim_receipt_id", + "drain", + "strict_block", + "sealed_proof", + "last_fold_summary", + ] { + assert!( + payload.get(field).is_some(), + "missing state-v2 slot {field}" + ); + assert!( + payload[field].is_null(), + "state-v2 slot {field} is not null" + ); + } + + let mut missing_slot = payload.clone(); + missing_slot.as_object_mut().unwrap().remove("drain"); + let missing_slot = serde_json::to_string(&missing_slot).unwrap(); + assert!( + StreamLifecycleEntry::from_manifest_row( + &entry.object_id(), + &entry.diagnostic_table_key, + entry.identity, + Some(&entry.binding.table_location), + Some(entry.current_head_witness.table_version), + None, + &missing_slot, + ) + .is_err(), + "a nullable state-v2 field must be present explicitly rather than supplied by serde" + ); + + let mut v1 = payload; + v1["protocol_version"] = serde_json::json!(1); + let v1 = serde_json::to_string(&v1).unwrap(); + assert!( + StreamLifecycleEntry::from_manifest_row( + &entry.object_id(), + &entry.diagnostic_table_key, + entry.identity, + Some(&entry.binding.table_location), + Some(entry.current_head_witness.table_version), + None, + &v1, + ) + .is_err() + ); + } + + #[test] + fn state_v2_open_requires_receipt_revision_and_null_lifecycle_slots() { + let mut missing_receipt_revision = entry(); + missing_receipt_revision + .enrollment_receipt + .initial_lifecycle_revision = 2; + assert!(missing_receipt_revision.validate().is_err()); + + let mut zero_revision = entry(); + zero_revision.lifecycle_revision = 0; + assert!(zero_revision.validate().is_err()); + + let mut draining_without_descriptor = entry(); + draining_without_descriptor.lifecycle = StreamLifecycle::Draining; + assert!(draining_without_descriptor.validate().is_err()); + + let mut sealed_without_proof = entry(); + sealed_without_proof.lifecycle = StreamLifecycle::Sealed; + assert!(sealed_without_proof.validate().is_err()); + } + + #[test] + fn enrollment_intent_digest_is_stable_and_binds_authority() { + let entry = entry(); + let digest = stream_enrollment_intent_digest_v1( + entry.identity, + &entry.binding.table_location, + "66666666-6666-4666-8666-666666666666", + &format!("sha256:{}", "c".repeat(64)), + 2, + &entry.current_head_witness, + &entry.binding.stream_config_hash, + ) + .unwrap(); + assert_eq!( + digest, + "sha256:0717d7ffecb791046c7a269bf767a1309cf3c35df9f721c7796ebbe060f66c14" + ); + assert_eq!( + digest, + stream_enrollment_intent_digest_v1( + entry.identity, + &entry.binding.table_location, + "66666666-6666-4666-8666-666666666666", + &format!("sha256:{}", "c".repeat(64)), + 2, + &entry.current_head_witness, + &entry.binding.stream_config_hash, + ) + .unwrap() + ); + let mut moved = entry.current_head_witness.clone(); + moved.table_version += 1; + assert_ne!( + digest, + stream_enrollment_intent_digest_v1( + entry.identity, + &entry.binding.table_location, + "66666666-6666-4666-8666-666666666666", + &format!("sha256:{}", "c".repeat(64)), + 2, + &moved, + &entry.binding.stream_config_hash, + ) + .unwrap() + ); + } + #[test] fn lifecycle_payload_rejects_identity_branch_and_epoch_ambiguity() { let mut wrong_identity = entry(); diff --git a/crates/omnigraph/src/db/manifest/stream_token.rs b/crates/omnigraph/src/db/manifest/stream_token.rs new file mode 100644 index 00000000..72480f6b --- /dev/null +++ b/crates/omnigraph/src/db/manifest/stream_token.rs @@ -0,0 +1,2085 @@ +//! RFC-026 B2 compare-and-chain token and trusted-attribution primitives. +//! +//! This module is deliberately storage-agnostic. It owns the canonical token +//! preimages and the pure admission classifier; the MemWAL adapter, token-table +//! participant, and recovery publisher consume these results but do not get to +//! reinterpret them. + +use std::fmt; +use std::str::FromStr; +use std::sync::Arc; + +use arrow_array::builder::{StringBuilder, StructBuilder, UInt32Builder, UInt64Builder}; +use arrow_array::{Array, ArrayRef, StringArray, StructArray, UInt32Array, UInt64Array}; +use arrow_schema::{DataType, Field, Schema}; +use lance_index::mem_wal::ShardId; +use serde::{Deserialize, Deserializer, Serialize, Serializer}; +use sha2::{Digest, Sha256}; + +use super::TableIdentity; + +const SHA256_WIRE_PREFIX: &str = "sha256:"; +const SHA256_HEX_LEN: usize = 64; +pub(crate) const STREAM_TOKEN_DERIVATION_VERSION: u32 = 1; +pub(crate) const STREAM_PAYLOAD_DIGEST_VERSION: u32 = 1; +pub(crate) const STREAM_PAYLOAD_ENCODING_VERSION: u32 = 1; +pub(crate) const STREAM_TOKEN_WIRE_VERSION: &str = "sha256-lowerhex-v1"; +const STREAM_TOKEN_DOMAIN_V1: &[u8] = b"omnigraph.stream-token.v1\0"; +const PAYLOAD_DIGEST_DOMAIN_V1: &[u8] = b"omnigraph.stream-payload.v1\0"; +const FOLD_ATTRIBUTION_DOMAIN_V1: &[u8] = b"omnigraph.stream-fold-attribution.v1\0"; +/// Conservative per-entry allowance for the BTree node, allocator headers, +/// and the duplicated logical-id key retained by exact authority lookups. +const STREAM_LOOKUP_ENTRY_OVERHEAD_BYTES: usize = 256; + +fn retained_string_bytes(total: &mut u64, value: &str) -> ProtocolResult<()> { + *total = total + .checked_add(u64::try_from(value.len()).map_err(|_| { + StreamTokenProtocolError::invalid("stream_lookup_bytes", "string length exceeds u64") + })?) + .ok_or_else(|| { + StreamTokenProtocolError::invalid("stream_lookup_bytes", "retained-byte sum overflow") + })?; + Ok(()) +} + +fn retained_origin_bytes(total: &mut u64, origin: &StreamRowOrigin) -> ProtocolResult<()> { + match origin { + StreamRowOrigin::Admission { + admission_attempt_id, + .. + } => retained_string_bytes(total, admission_attempt_id), + StreamRowOrigin::Correction { correction_id, .. } => { + retained_string_bytes(total, correction_id) + } + } +} + +/// Fixed system actor that owns every RFC-026 graph-visible stream fold. +/// +/// Keep this beside the attribution protocol rather than duplicating a string +/// literal across admission, recovery, and lineage validation. A commit that +/// carries a fold-attribution summary must be authored by this actor. +pub(crate) const STREAM_FOLD_ACTOR: &str = "omnigraph:stream-fold"; + +/// A failure in the trusted stream-token protocol layer. +/// +/// Admission conflicts are values returned by [`classify_admission`]. Errors +/// are reserved for malformed input or durable state that cannot be trusted. +#[derive(Debug, Clone, PartialEq, Eq, thiserror::Error)] +pub(crate) enum StreamTokenProtocolError { + #[error("invalid stream protocol field '{field}': {reason}")] + Invalid { field: &'static str, reason: String }, + #[error("stream token authority is corrupt: {0}")] + Corruption(String), +} + +type ProtocolResult = std::result::Result; + +impl StreamTokenProtocolError { + fn invalid(field: &'static str, reason: impl Into) -> Self { + Self::Invalid { + field, + reason: reason.into(), + } + } +} + +/// The exact v1 opaque compare-and-chain token. +/// +/// Its only accepted textual form is `sha256:` followed by 64 lowercase hex +/// digits. Callers echo this value; they never construct its preimage. +#[derive(Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)] +pub(crate) struct StreamToken([u8; 32]); + +impl StreamToken { + #[cfg(test)] + pub(crate) fn from_bytes(bytes: [u8; 32]) -> Self { + Self(bytes) + } + + pub(crate) fn as_bytes(&self) -> &[u8; 32] { + &self.0 + } + + pub(crate) fn derive(input: &StreamTokenInput<'_>) -> ProtocolResult { + input.validate()?; + + let stream_incarnation = + canonical_uuid_bytes("stream_incarnation_id", input.stream_incarnation_id)?; + let write_id = canonical_uuid_bytes("write_id", input.write_id)?; + let mut hasher = Sha256::new(); + hasher.update(STREAM_TOKEN_DOMAIN_V1); + hash_u64(&mut hasher, input.identity.stable_table_id); + hash_u64(&mut hasher, input.identity.table_incarnation_id); + hash_bytes(&mut hasher, input.logical_id.as_bytes()); + hasher.update(stream_incarnation); + hash_optional_digest(&mut hasher, input.predecessor_token); + hasher.update(write_id); + hash_bytes(&mut hasher, input.contributor_id.as_str().as_bytes()); + hasher.update(input.payload_digest.as_bytes()); + Ok(Self(hasher.finalize().into())) + } +} + +impl fmt::Display for StreamToken { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(formatter, "{SHA256_WIRE_PREFIX}")?; + write_lower_hex(formatter, &self.0) + } +} + +impl fmt::Debug for StreamToken { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + fmt::Display::fmt(self, formatter) + } +} + +impl FromStr for StreamToken { + type Err = StreamTokenProtocolError; + + fn from_str(value: &str) -> ProtocolResult { + decode_sha256("stream_token", value).map(Self) + } +} + +impl Serialize for StreamToken { + fn serialize(&self, serializer: S) -> std::result::Result + where + S: Serializer, + { + serializer.serialize_str(&self.to_string()) + } +} + +impl<'de> Deserialize<'de> for StreamToken { + fn deserialize(deserializer: D) -> std::result::Result + where + D: Deserializer<'de>, + { + let value = String::deserialize(deserializer)?; + Self::from_str(&value).map_err(serde::de::Error::custom) + } +} + +/// SHA-256 over one completely normalized logical payload. +/// +/// This is a distinct type from [`StreamToken`] so a token can never be passed +/// accidentally where the trusted payload digest is required. +#[derive(Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)] +pub(crate) struct PayloadDigest([u8; 32]); + +impl PayloadDigest { + #[cfg(test)] + pub(crate) fn from_bytes(bytes: [u8; 32]) -> Self { + Self(bytes) + } + + pub(crate) fn as_bytes(&self) -> &[u8; 32] { + &self.0 + } + + /// Hash deterministic, type-aware logical row bytes after normalization. + /// + /// The caller owns the canonical field encoding. This function binds those + /// bytes to the exact table lifetime and accepted schema before the result + /// can participate in a stream token. + pub(crate) fn derive(input: &PayloadDigestInput<'_>) -> ProtocolResult { + input.identity.validate().map_err(|error| { + StreamTokenProtocolError::invalid("table_identity", error.to_string()) + })?; + decode_sha256("accepted_schema_hash", input.accepted_schema_hash)?; + + let mut hasher = Sha256::new(); + hasher.update(PAYLOAD_DIGEST_DOMAIN_V1); + hash_u64(&mut hasher, input.identity.stable_table_id); + hash_u64(&mut hasher, input.identity.table_incarnation_id); + hash_bytes(&mut hasher, input.accepted_schema_hash.as_bytes()); + hash_bytes(&mut hasher, input.canonical_payload); + Ok(Self(hasher.finalize().into())) + } +} + +impl fmt::Display for PayloadDigest { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(formatter, "{SHA256_WIRE_PREFIX}")?; + write_lower_hex(formatter, &self.0) + } +} + +impl fmt::Debug for PayloadDigest { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + fmt::Display::fmt(self, formatter) + } +} + +impl FromStr for PayloadDigest { + type Err = StreamTokenProtocolError; + + fn from_str(value: &str) -> ProtocolResult { + decode_sha256("payload_digest", value).map(Self) + } +} + +impl Serialize for PayloadDigest { + fn serialize(&self, serializer: S) -> std::result::Result + where + S: Serializer, + { + serializer.serialize_str(&self.to_string()) + } +} + +impl<'de> Deserialize<'de> for PayloadDigest { + fn deserialize(deserializer: D) -> std::result::Result + where + D: Deserializer<'de>, + { + let value = String::deserialize(deserializer)?; + Self::from_str(&value).map_err(serde::de::Error::custom) + } +} + +/// Inputs to the versioned payload digest after all row normalization. +pub(crate) struct PayloadDigestInput<'a> { + pub(crate) identity: TableIdentity, + pub(crate) accepted_schema_hash: &'a str, + pub(crate) canonical_payload: &'a [u8], +} + +/// A contributor identity resolved at the trusted engine boundary. +#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)] +pub(crate) struct TrustedContributorId(String); + +impl TrustedContributorId { + pub(crate) fn new(value: impl Into) -> ProtocolResult { + let value = value.into(); + if value.is_empty() { + return Err(StreamTokenProtocolError::invalid( + "contributor_id", + "must be non-empty", + )); + } + Ok(Self(value)) + } + + pub(crate) fn as_str(&self) -> &str { + &self.0 + } +} + +impl Serialize for TrustedContributorId { + fn serialize(&self, serializer: S) -> std::result::Result + where + S: Serializer, + { + serializer.serialize_str(&self.0) + } +} + +impl<'de> Deserialize<'de> for TrustedContributorId { + fn deserialize(deserializer: D) -> std::result::Result + where + D: Deserializer<'de>, + { + let value = String::deserialize(deserializer)?; + Self::new(value).map_err(serde::de::Error::custom) + } +} + +/// Caller-owned idempotency and predecessor envelope for one logical row. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(crate) struct StreamWriteEnvelope { + pub(crate) stream_incarnation_id: String, + pub(crate) write_id: String, + #[serde(deserialize_with = "deserialize_present_option")] + pub(crate) predecessor_token: Option, +} + +impl StreamWriteEnvelope { + pub(crate) fn validate(&self) -> ProtocolResult<()> { + canonical_uuid_bytes("stream_incarnation_id", &self.stream_incarnation_id)?; + canonical_uuid_bytes("write_id", &self.write_id)?; + Ok(()) + } +} + +/// Exact durable origin of one winning stream row. +/// +/// A tagged enum makes the RFC's variant-specific nullability structural: +/// there is exactly one origin and only its selected fields can be present. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(tag = "kind", rename_all = "snake_case", deny_unknown_fields)] +pub(crate) enum StreamRowOrigin { + Admission { + admission_attempt_id: String, + caller_ordinal: u64, + }, + Correction { + correction_id: String, + plan_ordinal: u64, + }, +} + +impl StreamRowOrigin { + pub(crate) fn validate(&self) -> ProtocolResult<()> { + match self { + Self::Admission { + admission_attempt_id, + .. + } => { + canonical_uuid_bytes("admission_attempt_id", admission_attempt_id)?; + } + Self::Correction { correction_id, .. } => { + canonical_uuid_bytes("correction_id", correction_id)?; + } + } + Ok(()) + } +} + +/// Reserved `__omnigraph_stream_v1$` metadata stored atomically with a row. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(crate) struct TrustedStreamRowMetadata { + pub(crate) stream_incarnation_id: String, + pub(crate) contributor_id: TrustedContributorId, + pub(crate) write_id: String, + #[serde(deserialize_with = "deserialize_present_option")] + pub(crate) predecessor_token: Option, + pub(crate) stream_token: StreamToken, + #[serde(deserialize_with = "deserialize_present_option")] + pub(crate) fold_base_token: Option, + pub(crate) chain_depth: u32, + pub(crate) origin: StreamRowOrigin, + pub(crate) payload_digest: PayloadDigest, +} + +/// Exact physical field added to every internal-schema-v9 graph table. +/// +/// The top-level struct and every physical child are nullable. Lance may +/// synthesize null children while taking/projecting a null parent struct, so a +/// physically non-null child would make an otherwise valid unattributed row +/// unreadable. The decoder supplies the stricter logical contract: once the +/// parent is present, every required value must be present and the origin tag +/// controls the nullable variant children. +pub(crate) fn trusted_stream_metadata_field() -> Field { + let origin = DataType::Struct( + vec![ + Field::new("kind", DataType::Utf8, true), + Field::new("admission_attempt_id", DataType::Utf8, true), + Field::new("caller_ordinal", DataType::UInt64, true), + Field::new("correction_id", DataType::Utf8, true), + Field::new("plan_ordinal", DataType::UInt64, true), + ] + .into(), + ); + Field::new( + crate::db::STREAM_METADATA_COLUMN, + DataType::Struct( + vec![ + Field::new("stream_incarnation_id", DataType::Utf8, true), + Field::new("contributor_id", DataType::Utf8, true), + Field::new("write_id", DataType::Utf8, true), + Field::new("predecessor_token", DataType::Utf8, true), + Field::new("stream_token", DataType::Utf8, true), + Field::new("fold_base_token", DataType::Utf8, true), + Field::new("chain_depth", DataType::UInt32, true), + Field::new("origin", origin, true), + Field::new("payload_digest", DataType::Utf8, true), + ] + .into(), + ), + true, + ) +} + +/// Refuse a compatible-looking hidden column instead of interpreting it as +/// RFC-026 metadata. +pub(crate) fn validate_trusted_stream_metadata_field(field: &Field) -> ProtocolResult<()> { + let expected = trusted_stream_metadata_field(); + if field != &expected { + return Err(StreamTokenProtocolError::invalid( + "stream_metadata_field", + format!( + "must exactly match the internal {} schema", + crate::db::STREAM_METADATA_COLUMN + ), + )); + } + Ok(()) +} + +/// Require one exact canonical trusted-attribution field in a physical v9 +/// graph-table schema. Case-insensitive lookalikes are rejected rather than +/// ignored so a malformed storage format cannot masquerade as a user column. +pub(crate) fn validate_trusted_stream_metadata_schema(schema: &Schema) -> ProtocolResult<()> { + let mut matches = schema.fields().iter().filter(|field| { + field + .name() + .eq_ignore_ascii_case(crate::db::STREAM_METADATA_COLUMN) + }); + let field = matches.next().ok_or_else(|| { + StreamTokenProtocolError::invalid( + "stream_metadata_field", + format!( + "physical schema is missing exact internal field '{}'", + crate::db::STREAM_METADATA_COLUMN + ), + ) + })?; + if matches.next().is_some() || field.name() != crate::db::STREAM_METADATA_COLUMN { + return Err(StreamTokenProtocolError::invalid( + "stream_metadata_field", + format!( + "physical schema must contain exactly one canonical internal field '{}'", + crate::db::STREAM_METADATA_COLUMN + ), + )); + } + validate_trusted_stream_metadata_field(field) +} + +/// Build the reserved physical struct without accepting caller-supplied raw +/// origin or token text. +pub(crate) fn build_trusted_stream_metadata_array( + values: &[Option], +) -> ProtocolResult { + let field = trusted_stream_metadata_field(); + // `StructBuilder::from_fields` eagerly gives every Utf8 child a 1-KiB + // value buffer, even when every parent cell is null. Direct mutations can + // retain thousands of one-row batches, so that representation would spend + // the keyed-write budget on empty protocol children. Arrow's canonical + // null constructor preserves the exact datatype and parent nulls without + // allocating those unused child capacities. + if values.iter().all(Option::is_none) { + return Ok(arrow_array::new_null_array(field.data_type(), values.len())); + } + let DataType::Struct(fields) = field.data_type() else { + unreachable!("trusted stream metadata field is always a struct") + }; + let mut builder = StructBuilder::from_fields(fields.clone(), values.len()); + for value in values { + match value { + Some(metadata) => { + metadata.validate_structure()?; + append_metadata(&mut builder, metadata); + } + None => append_null_metadata(&mut builder), + } + } + Ok(Arc::new(builder.finish())) +} + +/// Decode one physical metadata cell with exact type, tag, and nullability +/// checks. [`TrustedStreamRowMetadata::validate_for`] additionally binds the +/// result to its containing table/key and verifies the token preimage. +pub(crate) fn decode_trusted_stream_metadata( + array: &dyn Array, + row: usize, +) -> ProtocolResult> { + if array.data_type() != trusted_stream_metadata_field().data_type() { + return Err(StreamTokenProtocolError::invalid( + "stream_metadata_array", + "has a non-canonical physical type", + )); + } + let array = array + .as_any() + .downcast_ref::() + .ok_or_else(|| { + StreamTokenProtocolError::invalid( + "stream_metadata_array", + "must be an Arrow StructArray", + ) + })?; + if row >= array.len() { + return Err(StreamTokenProtocolError::invalid( + "stream_metadata_row", + format!("row {row} is outside array length {}", array.len()), + )); + } + // Lance can materialize a null struct parent as a valid struct whose + // children are all null/default-valued while projecting/taking rows. Treat + // those representations as the same canonical absence. Any partially + // populated struct continues into the strict decoder and fails loudly. + if array.is_null(row) || is_absent_metadata_sentinel(array, row)? { + return Ok(None); + } + + let origin_array = array + .column(7) + .as_any() + .downcast_ref::() + .ok_or_else(|| { + StreamTokenProtocolError::invalid("origin", "must be an Arrow StructArray") + })?; + if origin_array.is_null(row) { + return Err(StreamTokenProtocolError::invalid( + "origin", + "must be non-null when stream metadata is present", + )); + } + let kind = required_string(origin_array, 0, row, "origin.kind")?; + let admission_attempt_id = optional_string(origin_array, 1, row, "admission_attempt_id")?; + let caller_ordinal = optional_u64(origin_array, 2, row, "caller_ordinal")?; + let correction_id = optional_string(origin_array, 3, row, "correction_id")?; + let plan_ordinal = optional_u64(origin_array, 4, row, "plan_ordinal")?; + let origin = match ( + kind.as_str(), + admission_attempt_id, + caller_ordinal, + correction_id, + plan_ordinal, + ) { + ("admission", Some(admission_attempt_id), Some(caller_ordinal), None, None) => { + StreamRowOrigin::Admission { + admission_attempt_id, + caller_ordinal, + } + } + ("correction", None, None, Some(correction_id), Some(plan_ordinal)) => { + StreamRowOrigin::Correction { + correction_id, + plan_ordinal, + } + } + _ => { + return Err(StreamTokenProtocolError::invalid( + "origin", + "tag and variant-specific nullable children do not agree", + )); + } + }; + + let metadata = TrustedStreamRowMetadata { + stream_incarnation_id: required_string(array, 0, row, "stream_incarnation_id")?, + contributor_id: TrustedContributorId::new(required_string( + array, + 1, + row, + "contributor_id", + )?)?, + write_id: required_string(array, 2, row, "write_id")?, + predecessor_token: optional_string(array, 3, row, "predecessor_token")? + .map(|value| StreamToken::from_str(&value)) + .transpose()?, + stream_token: StreamToken::from_str(&required_string(array, 4, row, "stream_token")?)?, + fold_base_token: optional_string(array, 5, row, "fold_base_token")? + .map(|value| StreamToken::from_str(&value)) + .transpose()?, + chain_depth: required_u32(array, 6, row, "chain_depth")?, + origin, + payload_digest: PayloadDigest::from_str(&required_string( + array, + 8, + row, + "payload_digest", + )?)?, + }; + metadata.validate_structure()?; + Ok(Some(metadata)) +} + +fn is_absent_metadata_sentinel(array: &StructArray, row: usize) -> ProtocolResult { + let empty_string = |column: usize, field: &'static str| -> ProtocolResult { + let values = array + .column(column) + .as_any() + .downcast_ref::() + .ok_or_else(|| { + StreamTokenProtocolError::invalid(field, "must be an Arrow StringArray") + })?; + Ok(values.is_null(row) || values.value(row).is_empty()) + }; + if !(empty_string(0, "stream_incarnation_id")? + && empty_string(1, "contributor_id")? + && empty_string(2, "write_id")? + && empty_string(3, "predecessor_token")? + && empty_string(4, "stream_token")? + && empty_string(5, "fold_base_token")? + && empty_string(8, "payload_digest")?) + { + return Ok(false); + } + let chain_depth = array + .column(6) + .as_any() + .downcast_ref::() + .ok_or_else(|| { + StreamTokenProtocolError::invalid("chain_depth", "must be an Arrow UInt32Array") + })?; + if !chain_depth.is_null(row) && chain_depth.value(row) != 0 { + return Ok(false); + } + let origin = array + .column(7) + .as_any() + .downcast_ref::() + .ok_or_else(|| { + StreamTokenProtocolError::invalid("origin", "must be an Arrow StructArray") + })?; + if origin.is_null(row) { + return Ok(true); + } + let origin_empty_string = |column: usize, field: &'static str| -> ProtocolResult { + let values = origin + .column(column) + .as_any() + .downcast_ref::() + .ok_or_else(|| { + StreamTokenProtocolError::invalid(field, "must be an Arrow StringArray") + })?; + Ok(values.is_null(row) || values.value(row).is_empty()) + }; + let origin_empty_u64 = |column: usize, field: &'static str| -> ProtocolResult { + let values = origin + .column(column) + .as_any() + .downcast_ref::() + .ok_or_else(|| { + StreamTokenProtocolError::invalid(field, "must be an Arrow UInt64Array") + })?; + Ok(values.is_null(row) || values.value(row) == 0) + }; + Ok(origin_empty_string(0, "origin.kind")? + && origin_empty_string(1, "admission_attempt_id")? + && origin_empty_u64(2, "caller_ordinal")? + && origin_empty_string(3, "correction_id")? + && origin_empty_u64(4, "plan_ordinal")?) +} + +impl TrustedStreamRowMetadata { + /// Conservative logical bytes retained when this metadata is held in the + /// bounded exact-key lookup map. Variable strings are counted exactly; + /// fixed struct/key/node overhead is charged explicitly. + pub(crate) fn lookup_retained_bytes(&self, logical_id: &str) -> ProtocolResult { + let fixed = std::mem::size_of::() + .checked_add(std::mem::size_of::()) + .and_then(|value| value.checked_add(STREAM_LOOKUP_ENTRY_OVERHEAD_BYTES)) + .ok_or_else(|| { + StreamTokenProtocolError::invalid( + "stream_lookup_bytes", + "fixed retained-byte sum overflow", + ) + })?; + let mut total = u64::try_from(fixed).map_err(|_| { + StreamTokenProtocolError::invalid("stream_lookup_bytes", "fixed bytes exceed u64") + })?; + retained_string_bytes(&mut total, logical_id)?; + retained_string_bytes(&mut total, &self.stream_incarnation_id)?; + retained_string_bytes(&mut total, self.contributor_id.as_str())?; + retained_string_bytes(&mut total, &self.write_id)?; + retained_origin_bytes(&mut total, &self.origin)?; + Ok(total) + } + + /// Mint trusted admission metadata only after [`classify_admission`] has + /// returned this exact candidate as `New`. + #[allow(clippy::too_many_arguments)] + pub(crate) fn new_admission( + request: &AdmissionRequest, + candidate_token: StreamToken, + fold_base_token: Option, + chain_depth: u32, + admission_attempt_id: String, + caller_ordinal: u64, + ) -> ProtocolResult { + request.validate()?; + if request.candidate_token()? != candidate_token { + return Err(StreamTokenProtocolError::Corruption(format!( + "candidate token for '{}' changed between classification and metadata construction", + request.logical_id + ))); + } + let metadata = Self { + stream_incarnation_id: request.envelope.stream_incarnation_id.clone(), + contributor_id: request.contributor_id.clone(), + write_id: request.envelope.write_id.clone(), + predecessor_token: request.envelope.predecessor_token, + stream_token: candidate_token, + fold_base_token, + chain_depth, + origin: StreamRowOrigin::Admission { + admission_attempt_id, + caller_ordinal, + }, + payload_digest: request.payload_digest, + }; + metadata.validate_for(request.identity, &request.logical_id)?; + Ok(metadata) + } + + fn validate_structure(&self) -> ProtocolResult<()> { + canonical_uuid_bytes("stream_incarnation_id", &self.stream_incarnation_id)?; + canonical_uuid_bytes("write_id", &self.write_id)?; + self.origin.validate()?; + if self.chain_depth == 0 { + return Err(StreamTokenProtocolError::invalid( + "chain_depth", + "must be non-zero", + )); + } + if self.chain_depth == 1 && self.fold_base_token != self.predecessor_token { + return Err(StreamTokenProtocolError::invalid( + "fold_base_token", + "depth-one metadata must start at its predecessor token", + )); + } + Ok(()) + } + + pub(crate) fn validate_for( + &self, + identity: TableIdentity, + logical_id: &str, + ) -> ProtocolResult<()> { + self.validate_structure()?; + let derived = StreamToken::derive(&StreamTokenInput { + identity, + logical_id, + stream_incarnation_id: &self.stream_incarnation_id, + predecessor_token: self.predecessor_token, + write_id: &self.write_id, + contributor_id: &self.contributor_id, + payload_digest: self.payload_digest, + })?; + if derived != self.stream_token { + return Err(StreamTokenProtocolError::Corruption(format!( + "row '{logical_id}' embeds a stream token that does not match its trusted preimage" + ))); + } + Ok(()) + } + + pub(crate) fn agrees_with_authority(&self, authority: &StreamTokenAuthorityRow) -> bool { + self.stream_incarnation_id == authority.stream_incarnation_id + && self.contributor_id == authority.contributor_id + && self.write_id == authority.write_id + && self.predecessor_token == authority.predecessor_token + && self.stream_token == authority.current_token + && self.fold_base_token == authority.fold_base_token + && self.chain_depth == authority.chain_depth + && self.origin == authority.origin + && self.payload_digest == authority.payload_digest + } +} + +/// Complete, unambiguous input to stream-token derivation. +pub(crate) struct StreamTokenInput<'a> { + pub(crate) identity: TableIdentity, + pub(crate) logical_id: &'a str, + pub(crate) stream_incarnation_id: &'a str, + pub(crate) predecessor_token: Option, + pub(crate) write_id: &'a str, + pub(crate) contributor_id: &'a TrustedContributorId, + pub(crate) payload_digest: PayloadDigest, +} + +impl StreamTokenInput<'_> { + fn validate(&self) -> ProtocolResult<()> { + self.identity.validate().map_err(|error| { + StreamTokenProtocolError::invalid("table_identity", error.to_string()) + })?; + canonical_uuid_bytes("stream_incarnation_id", self.stream_incarnation_id)?; + canonical_uuid_bytes("write_id", self.write_id)?; + if self.contributor_id.as_str().is_empty() { + return Err(StreamTokenProtocolError::invalid( + "contributor_id", + "must be non-empty", + )); + } + Ok(()) + } +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub(crate) enum StreamTokenDisposition { + Present, + Withdrawn, +} + +/// Terminal evidence required when the current token has been withdrawn. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(crate) struct StreamTerminalCorrection { + pub(crate) actor: TrustedContributorId, + pub(crate) correction_id: String, +} + +impl StreamTerminalCorrection { + fn validate(&self) -> ProtocolResult<()> { + canonical_uuid_bytes("terminal_correction_id", &self.correction_id)?; + Ok(()) + } +} + +/// The one current `_stream_tokens.lance` row for a logical graph key. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(crate) struct StreamTokenAuthorityRow { + pub(crate) identity: TableIdentity, + pub(crate) logical_id: String, + pub(crate) origin_enrollment_id: String, + pub(crate) stream_incarnation_id: String, + pub(crate) current_token: StreamToken, + pub(crate) write_id: String, + #[serde(deserialize_with = "deserialize_present_option")] + pub(crate) predecessor_token: Option, + #[serde(deserialize_with = "deserialize_present_option")] + pub(crate) fold_base_token: Option, + pub(crate) chain_depth: u32, + pub(crate) disposition: StreamTokenDisposition, + pub(crate) contributor_id: TrustedContributorId, + pub(crate) payload_digest: PayloadDigest, + pub(crate) origin: StreamRowOrigin, + #[serde(deserialize_with = "deserialize_present_option")] + pub(crate) terminal_correction: Option, +} + +impl StreamTokenAuthorityRow { + /// Conservative logical bytes retained by the bounded current-token map. + /// The logical id is charged twice because both the row and BTree key own + /// it independently. + pub(crate) fn lookup_retained_bytes(&self) -> ProtocolResult { + let fixed = std::mem::size_of::() + .checked_add(std::mem::size_of::()) + .and_then(|value| value.checked_add(STREAM_LOOKUP_ENTRY_OVERHEAD_BYTES)) + .ok_or_else(|| { + StreamTokenProtocolError::invalid( + "stream_lookup_bytes", + "fixed retained-byte sum overflow", + ) + })?; + let mut total = u64::try_from(fixed).map_err(|_| { + StreamTokenProtocolError::invalid("stream_lookup_bytes", "fixed bytes exceed u64") + })?; + retained_string_bytes(&mut total, &self.logical_id)?; + retained_string_bytes(&mut total, &self.logical_id)?; + retained_string_bytes(&mut total, &self.origin_enrollment_id)?; + retained_string_bytes(&mut total, &self.stream_incarnation_id)?; + retained_string_bytes(&mut total, &self.write_id)?; + retained_string_bytes(&mut total, self.contributor_id.as_str())?; + retained_origin_bytes(&mut total, &self.origin)?; + if let Some(correction) = &self.terminal_correction { + retained_string_bytes(&mut total, correction.actor.as_str())?; + retained_string_bytes(&mut total, &correction.correction_id)?; + } + Ok(total) + } + + /// Materialize the post-fold PRESENT authority from the exact winning base + /// metadata. The fold must stage this row and the base effect together. + pub(crate) fn from_present_metadata( + identity: TableIdentity, + logical_id: String, + origin_enrollment_id: String, + metadata: &TrustedStreamRowMetadata, + ) -> ProtocolResult { + metadata.validate_for(identity, &logical_id)?; + let row = Self { + identity, + logical_id, + origin_enrollment_id, + stream_incarnation_id: metadata.stream_incarnation_id.clone(), + current_token: metadata.stream_token, + write_id: metadata.write_id.clone(), + predecessor_token: metadata.predecessor_token, + fold_base_token: metadata.fold_base_token, + chain_depth: metadata.chain_depth, + disposition: StreamTokenDisposition::Present, + contributor_id: metadata.contributor_id.clone(), + payload_digest: metadata.payload_digest, + origin: metadata.origin.clone(), + terminal_correction: None, + }; + row.validate()?; + Ok(row) + } + + pub(crate) fn validate(&self) -> ProtocolResult<()> { + self.identity.validate().map_err(|error| { + StreamTokenProtocolError::invalid("table_identity", error.to_string()) + })?; + let enrollment = canonical_uuid("origin_enrollment_id", &self.origin_enrollment_id)?; + if enrollment.get_version_num() != 4 { + return Err(StreamTokenProtocolError::invalid( + "origin_enrollment_id", + "must be a UUID v4 value", + )); + } + canonical_uuid_bytes("stream_incarnation_id", &self.stream_incarnation_id)?; + canonical_uuid_bytes("write_id", &self.write_id)?; + self.origin.validate()?; + if self.chain_depth == 0 { + return Err(StreamTokenProtocolError::invalid( + "chain_depth", + "must be non-zero", + )); + } + match (self.disposition, &self.terminal_correction) { + (StreamTokenDisposition::Present, None) => {} + (StreamTokenDisposition::Present, Some(_)) => { + return Err(StreamTokenProtocolError::invalid( + "terminal_correction", + "must be null for PRESENT token authority", + )); + } + (StreamTokenDisposition::Withdrawn, Some(correction)) => correction.validate()?, + (StreamTokenDisposition::Withdrawn, None) => { + return Err(StreamTokenProtocolError::invalid( + "terminal_correction", + "must be present for WITHDRAWN token authority", + )); + } + } + let derived = StreamToken::derive(&StreamTokenInput { + identity: self.identity, + logical_id: &self.logical_id, + stream_incarnation_id: &self.stream_incarnation_id, + predecessor_token: self.predecessor_token, + write_id: &self.write_id, + contributor_id: &self.contributor_id, + payload_digest: self.payload_digest, + })?; + if derived != self.current_token { + return Err(StreamTokenProtocolError::Corruption(format!( + "token row for '{}' does not match its canonical preimage", + self.logical_id + ))); + } + Ok(()) + } +} + +/// Durable representation shared by recovery-v12 and the immutable +/// `graph_commit` lineage row. +/// +/// The enclosing `Option` on graph lineage is the discriminator: ordinary +/// commits carry no value, while a stream fold carries all three fields as one +/// indivisible commitment. Keeping the exact recovery wire field names here +/// avoids a second representation that can drift from the committed audit +/// payload. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(crate) struct StreamFoldAttributionSummary { + pub(crate) visible_contributor_count: u64, + pub(crate) visible_write_count: u64, + pub(crate) winning_attribution_digest: String, +} + +impl StreamFoldAttributionSummary { + pub(crate) fn validate(&self) -> ProtocolResult<()> { + if self.visible_contributor_count == 0 { + return Err(StreamTokenProtocolError::invalid( + "visible_contributor_count", + "must be non-zero", + )); + } + if self.visible_write_count == 0 { + return Err(StreamTokenProtocolError::invalid( + "visible_write_count", + "must be non-zero", + )); + } + if self.visible_contributor_count > self.visible_write_count { + return Err(StreamTokenProtocolError::invalid( + "visible_contributor_count", + "must not exceed visible_write_count", + )); + } + decode_sha256( + "winning_attribution_digest", + &self.winning_attribution_digest, + )?; + Ok(()) + } +} + +/// Hash the RFC-026 sorted winning +/// `(contributor_id, stream_token, write_id, tagged_origin)` tuples. +pub(crate) fn stream_fold_attribution_commitment( + winners: &[StreamTokenAuthorityRow], +) -> ProtocolResult { + if winners.is_empty() { + return Err(StreamTokenProtocolError::invalid( + "fold_winners", + "must contain at least one attributed winner", + )); + } + + let mut contributors = Vec::with_capacity(winners.len()); + let mut tuples = Vec::with_capacity(winners.len()); + for winner in winners { + winner.validate()?; + if winner.disposition != StreamTokenDisposition::Present { + return Err(StreamTokenProtocolError::Corruption(format!( + "fold winner '{}' is not PRESENT", + winner.logical_id + ))); + } + contributors.push(winner.contributor_id.as_str().as_bytes().to_vec()); + + let mut tuple = Vec::new(); + tuple.extend_from_slice( + &u64::try_from(winner.contributor_id.as_str().len()) + .map_err(|_| { + StreamTokenProtocolError::invalid( + "contributor_id", + "length exceeds the canonical u64 envelope", + ) + })? + .to_be_bytes(), + ); + tuple.extend_from_slice(winner.contributor_id.as_str().as_bytes()); + tuple.extend_from_slice(winner.current_token.as_bytes()); + tuple.extend_from_slice(&canonical_uuid_bytes("write_id", &winner.write_id)?); + match &winner.origin { + StreamRowOrigin::Admission { + admission_attempt_id, + caller_ordinal, + } => { + tuple.push(0); + tuple.extend_from_slice(&canonical_uuid_bytes( + "admission_attempt_id", + admission_attempt_id, + )?); + tuple.extend_from_slice(&caller_ordinal.to_be_bytes()); + } + StreamRowOrigin::Correction { + correction_id, + plan_ordinal, + } => { + tuple.push(1); + tuple.extend_from_slice(&canonical_uuid_bytes("correction_id", correction_id)?); + tuple.extend_from_slice(&plan_ordinal.to_be_bytes()); + } + } + tuples.push(tuple); + } + contributors.sort(); + contributors.dedup(); + tuples.sort(); + + let mut hasher = Sha256::new(); + hasher.update(FOLD_ATTRIBUTION_DOMAIN_V1); + hash_u64( + &mut hasher, + u64::try_from(tuples.len()) + .map_err(|_| StreamTokenProtocolError::invalid("fold_winners", "count exceeds u64"))?, + ); + for tuple in tuples { + hash_bytes(&mut hasher, &tuple); + } + Ok(StreamFoldAttributionSummary { + visible_contributor_count: u64::try_from(contributors.len()).map_err(|_| { + StreamTokenProtocolError::invalid("fold_contributors", "count exceeds u64") + })?, + visible_write_count: u64::try_from(winners.len()) + .map_err(|_| StreamTokenProtocolError::invalid("fold_winners", "count exceeds u64"))?, + winning_attribution_digest: format!("sha256:{:x}", hasher.finalize()), + }) +} + +/// Fully normalized trusted request presented to the admission classifier. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct AdmissionRequest { + pub(crate) identity: TableIdentity, + pub(crate) logical_id: String, + pub(crate) envelope: StreamWriteEnvelope, + pub(crate) contributor_id: TrustedContributorId, + pub(crate) payload_digest: PayloadDigest, +} + +impl AdmissionRequest { + pub(crate) fn validate(&self) -> ProtocolResult<()> { + self.identity.validate().map_err(|error| { + StreamTokenProtocolError::invalid("table_identity", error.to_string()) + })?; + self.envelope.validate() + } + + pub(crate) fn candidate_token(&self) -> ProtocolResult { + StreamToken::derive(&StreamTokenInput { + identity: self.identity, + logical_id: &self.logical_id, + stream_incarnation_id: &self.envelope.stream_incarnation_id, + predecessor_token: self.envelope.predecessor_token, + write_id: &self.envelope.write_id, + contributor_id: &self.contributor_id, + payload_digest: self.payload_digest, + }) + } +} + +/// Effect-free result of comparing a normalized row with durable authority. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) enum AdmissionClassification { + New { + candidate_token: StreamToken, + }, + AlreadyDurable { + authority: StreamTokenAuthorityRow, + /// Required for PRESENT; absent for WITHDRAWN because the visible base + /// row may legitimately be absent or contain the older value. + present_metadata: Option, + }, + SequenceConflict { + current_token: Option, + }, + IdempotencyConflict { + current_token: StreamToken, + }, + BindingChanged { + current_stream_incarnation_id: String, + }, +} + +/// Validate the redundant base-row copy against the manifest-selected token +/// authority for one logical key. +/// +/// The base field is not a sequencing authority, but it is durable evidence +/// that must agree whenever the selected token is PRESENT. A non-null base +/// copy with no selected token row is likewise corruption: treating it as an +/// empty chain would let a later write silently reset sequencing. WITHDRAWN +/// deliberately permits an absent or older visible base row. +pub(crate) fn validate_authority_base_pair( + identity: TableIdentity, + logical_id: &str, + current_authority: Option<&StreamTokenAuthorityRow>, + current_base_metadata: Option<&TrustedStreamRowMetadata>, +) -> ProtocolResult> { + let Some(current) = current_authority else { + if current_base_metadata.is_some() { + return Err(StreamTokenProtocolError::Corruption(format!( + "base row for '{logical_id}' carries trusted stream metadata but the manifest-selected token authority has no row" + ))); + } + return Ok(None); + }; + + current.validate()?; + if current.identity != identity || current.logical_id != logical_id { + return Err(StreamTokenProtocolError::Corruption(format!( + "selected token row ({}, '{}') does not match requested key ({identity}, '{logical_id}')", + current.identity, current.logical_id + ))); + } + match current.disposition { + StreamTokenDisposition::Present => { + let metadata = current_base_metadata.ok_or_else(|| { + StreamTokenProtocolError::Corruption(format!( + "PRESENT token row for '{logical_id}' has no matching base-row stream metadata" + )) + })?; + metadata.validate_for(identity, logical_id)?; + if !metadata.agrees_with_authority(current) { + return Err(StreamTokenProtocolError::Corruption(format!( + "PRESENT token row for '{logical_id}' disagrees with base-row stream metadata" + ))); + } + Ok(Some(metadata.clone())) + } + StreamTokenDisposition::Withdrawn => Ok(None), + } +} + +/// Classify one occurrence before minting an origin or calling Lance. +/// +/// `current_stream_incarnation_id` comes from the revalidated lifecycle/token +/// authority. `current_authority` comes only from the manifest-selected token +/// dataset version. Every PRESENT authority is checked against its base-row +/// copy before either an idempotent retry or a successor may proceed. This +/// prevents a later occurrence from silently healing divergent authority. +pub(crate) fn classify_admission( + current_stream_incarnation_id: &str, + request: &AdmissionRequest, + current_authority: Option<&StreamTokenAuthorityRow>, + current_base_metadata: Option<&TrustedStreamRowMetadata>, +) -> ProtocolResult { + request.validate()?; + canonical_uuid_bytes( + "current_stream_incarnation_id", + current_stream_incarnation_id, + )?; + if request.envelope.stream_incarnation_id != current_stream_incarnation_id { + return Ok(AdmissionClassification::BindingChanged { + current_stream_incarnation_id: current_stream_incarnation_id.to_string(), + }); + } + + let candidate = request.candidate_token()?; + let present_metadata = validate_authority_base_pair( + request.identity, + &request.logical_id, + current_authority, + current_base_metadata, + )?; + let Some(current) = current_authority else { + return if request.envelope.predecessor_token.is_none() { + Ok(AdmissionClassification::New { + candidate_token: candidate, + }) + } else { + Ok(AdmissionClassification::SequenceConflict { + current_token: None, + }) + }; + }; + + if current.stream_incarnation_id != current_stream_incarnation_id { + return Err(StreamTokenProtocolError::Corruption(format!( + "token row for '{}' names stream incarnation '{}' but lifecycle authority names '{}'", + current.logical_id, current.stream_incarnation_id, current_stream_incarnation_id + ))); + } + + let same_occurrence = current.write_id == request.envelope.write_id + && current.predecessor_token == request.envelope.predecessor_token; + if same_occurrence + && (current.contributor_id != request.contributor_id + || current.payload_digest != request.payload_digest) + { + return Ok(AdmissionClassification::IdempotencyConflict { + current_token: current.current_token, + }); + } + + if candidate == current.current_token { + let all_token_fields_match = same_occurrence + && current.stream_incarnation_id == request.envelope.stream_incarnation_id + && current.contributor_id == request.contributor_id + && current.payload_digest == request.payload_digest; + if !all_token_fields_match { + return Err(StreamTokenProtocolError::Corruption(format!( + "candidate token for '{}' equals current token but its canonical preimage differs", + request.logical_id + ))); + } + + return Ok(AdmissionClassification::AlreadyDurable { + authority: current.clone(), + present_metadata, + }); + } + + if request.envelope.predecessor_token != Some(current.current_token) { + return Ok(AdmissionClassification::SequenceConflict { + current_token: Some(current.current_token), + }); + } + + Ok(AdmissionClassification::New { + candidate_token: candidate, + }) +} + +/// Require nullable protocol fields to be present as either an explicit JSON +/// null or a value. Serde's default `Option` behavior accepts a missing +/// field as `None`, which would silently reinterpret an older payload. +fn deserialize_present_option<'de, D, T>( + deserializer: D, +) -> std::result::Result, D::Error> +where + D: Deserializer<'de>, + T: Deserialize<'de>, +{ + Option::::deserialize(deserializer) +} + +fn canonical_uuid(field: &'static str, value: &str) -> ProtocolResult { + let parsed = ShardId::parse_str(value).map_err(|error| { + StreamTokenProtocolError::invalid(field, format!("must be a UUID: {error}")) + })?; + if parsed.is_nil() { + return Err(StreamTokenProtocolError::invalid(field, "must be non-nil")); + } + if parsed.to_string() != value { + return Err(StreamTokenProtocolError::invalid( + field, + "must use canonical lowercase hyphenated UUID text", + )); + } + Ok(parsed) +} + +fn canonical_uuid_bytes(field: &'static str, value: &str) -> ProtocolResult<[u8; 16]> { + Ok(*canonical_uuid(field, value)?.as_bytes()) +} + +fn decode_sha256(field: &'static str, value: &str) -> ProtocolResult<[u8; 32]> { + let Some(hex) = value.strip_prefix(SHA256_WIRE_PREFIX) else { + return Err(StreamTokenProtocolError::invalid( + field, + "must use the exact 'sha256:' prefix", + )); + }; + if hex.len() != SHA256_HEX_LEN { + return Err(StreamTokenProtocolError::invalid( + field, + "must contain exactly 64 lowercase hexadecimal digits", + )); + } + if !hex + .bytes() + .all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte)) + { + return Err(StreamTokenProtocolError::invalid( + field, + "must contain lowercase hexadecimal digits only", + )); + } + + let mut bytes = [0_u8; 32]; + for (index, pair) in hex.as_bytes().chunks_exact(2).enumerate() { + bytes[index] = (hex_nibble(pair[0]) << 4) | hex_nibble(pair[1]); + } + let canonical = format!("{SHA256_WIRE_PREFIX}{}", lower_hex(&bytes)); + if canonical != value { + return Err(StreamTokenProtocolError::invalid( + field, + "must round-trip through the canonical sha256-lowerhex-v1 representation", + )); + } + Ok(bytes) +} + +fn hex_nibble(byte: u8) -> u8 { + match byte { + b'0'..=b'9' => byte - b'0', + b'a'..=b'f' => byte - b'a' + 10, + _ => unreachable!("decode_sha256 validates the alphabet first"), + } +} + +fn lower_hex(bytes: &[u8]) -> String { + const ALPHABET: &[u8; 16] = b"0123456789abcdef"; + let mut output = String::with_capacity(bytes.len() * 2); + for byte in bytes { + output.push(ALPHABET[(byte >> 4) as usize] as char); + output.push(ALPHABET[(byte & 0x0f) as usize] as char); + } + output +} + +fn write_lower_hex(formatter: &mut fmt::Formatter<'_>, bytes: &[u8]) -> fmt::Result { + for byte in bytes { + write!(formatter, "{byte:02x}")?; + } + Ok(()) +} + +fn hash_u64(hasher: &mut Sha256, value: u64) { + hasher.update(value.to_be_bytes()); +} + +fn hash_bytes(hasher: &mut Sha256, value: &[u8]) { + hash_u64(hasher, value.len() as u64); + hasher.update(value); +} + +fn hash_optional_digest(hasher: &mut Sha256, value: Option) { + match value { + None => hasher.update([0]), + Some(value) => { + hasher.update([1]); + hasher.update(value.as_bytes()); + } + } +} + +fn append_metadata(builder: &mut StructBuilder, metadata: &TrustedStreamRowMetadata) { + append_string(builder, 0, Some(&metadata.stream_incarnation_id)); + append_string(builder, 1, Some(metadata.contributor_id.as_str())); + append_string(builder, 2, Some(&metadata.write_id)); + let predecessor = metadata.predecessor_token.map(|token| token.to_string()); + append_string(builder, 3, predecessor.as_deref()); + let token = metadata.stream_token.to_string(); + append_string(builder, 4, Some(&token)); + let fold_base = metadata.fold_base_token.map(|token| token.to_string()); + append_string(builder, 5, fold_base.as_deref()); + append_u32(builder, 6, Some(metadata.chain_depth)); + append_origin(builder, Some(&metadata.origin)); + let payload = metadata.payload_digest.to_string(); + append_string(builder, 8, Some(&payload)); + builder.append(true); +} + +fn append_null_metadata(builder: &mut StructBuilder) { + // Keep every child nullable because Lance may materialize a null parent as + // a valid all-null struct during take/projection. The decoder recognizes + // only that complete all-null shape as absence; partial metadata is never + // accepted as an unattributed row. + append_string(builder, 0, None); + append_string(builder, 1, None); + append_string(builder, 2, None); + append_string(builder, 3, None); + append_string(builder, 4, None); + append_string(builder, 5, None); + append_u32(builder, 6, None); + append_origin(builder, None); + append_string(builder, 8, None); + builder.append(false); +} + +fn append_origin(builder: &mut StructBuilder, origin: Option<&StreamRowOrigin>) { + let origin_builder = builder + .field_builder::(7) + .expect("trusted stream origin uses a StructBuilder"); + match origin { + Some(StreamRowOrigin::Admission { + admission_attempt_id, + caller_ordinal, + }) => { + append_string(origin_builder, 0, Some("admission")); + append_string(origin_builder, 1, Some(admission_attempt_id)); + append_u64(origin_builder, 2, Some(*caller_ordinal)); + append_string(origin_builder, 3, None); + append_u64(origin_builder, 4, None); + origin_builder.append(true); + } + Some(StreamRowOrigin::Correction { + correction_id, + plan_ordinal, + }) => { + append_string(origin_builder, 0, Some("correction")); + append_string(origin_builder, 1, None); + append_u64(origin_builder, 2, None); + append_string(origin_builder, 3, Some(correction_id)); + append_u64(origin_builder, 4, Some(*plan_ordinal)); + origin_builder.append(true); + } + None => { + append_string(origin_builder, 0, None); + append_string(origin_builder, 1, None); + append_u64(origin_builder, 2, None); + append_string(origin_builder, 3, None); + append_u64(origin_builder, 4, None); + origin_builder.append(false); + } + } +} + +fn append_string(builder: &mut StructBuilder, field: usize, value: Option<&str>) { + let builder = builder + .field_builder::(field) + .expect("trusted stream schema string field uses StringBuilder"); + match value { + Some(value) => builder.append_value(value), + None => builder.append_null(), + } +} + +fn append_u32(builder: &mut StructBuilder, field: usize, value: Option) { + let builder = builder + .field_builder::(field) + .expect("trusted stream schema u32 field uses UInt32Builder"); + match value { + Some(value) => builder.append_value(value), + None => builder.append_null(), + } +} + +fn append_u64(builder: &mut StructBuilder, field: usize, value: Option) { + let builder = builder + .field_builder::(field) + .expect("trusted stream schema u64 field uses UInt64Builder"); + match value { + Some(value) => builder.append_value(value), + None => builder.append_null(), + } +} + +fn required_string( + array: &StructArray, + field: usize, + row: usize, + name: &'static str, +) -> ProtocolResult { + optional_string(array, field, row, name)?.ok_or_else(|| { + StreamTokenProtocolError::invalid(name, "must be non-null when metadata is present") + }) +} + +fn optional_string( + array: &StructArray, + field: usize, + row: usize, + name: &'static str, +) -> ProtocolResult> { + let values = array + .column(field) + .as_any() + .downcast_ref::() + .ok_or_else(|| StreamTokenProtocolError::invalid(name, "must have Arrow Utf8 type"))?; + Ok((!values.is_null(row)).then(|| values.value(row).to_string())) +} + +fn required_u32( + array: &StructArray, + field: usize, + row: usize, + name: &'static str, +) -> ProtocolResult { + let values = array + .column(field) + .as_any() + .downcast_ref::() + .ok_or_else(|| StreamTokenProtocolError::invalid(name, "must have Arrow UInt32 type"))?; + if values.is_null(row) { + return Err(StreamTokenProtocolError::invalid( + name, + "must be non-null when metadata is present", + )); + } + Ok(values.value(row)) +} + +fn optional_u64( + array: &StructArray, + field: usize, + row: usize, + name: &'static str, +) -> ProtocolResult> { + let values = array + .column(field) + .as_any() + .downcast_ref::() + .ok_or_else(|| StreamTokenProtocolError::invalid(name, "must have Arrow UInt64 type"))?; + Ok((!values.is_null(row)).then(|| values.value(row))) +} + +#[cfg(test)] +mod tests { + use super::*; + + const STREAM_INCARNATION: &str = "11111111-1111-4111-8111-111111111111"; + const NEXT_STREAM_INCARNATION: &str = "22222222-2222-4222-8222-222222222222"; + const ENROLLMENT_ID: &str = "33333333-3333-4333-8333-333333333333"; + const WRITE_X: &str = "44444444-4444-4444-8444-444444444444"; + const WRITE_Y: &str = "55555555-5555-4555-8555-555555555555"; + const ATTEMPT_X: &str = "66666666-6666-4666-8666-666666666666"; + const ATTEMPT_Y: &str = "77777777-7777-4777-8777-777777777777"; + + fn identity() -> TableIdentity { + TableIdentity::new(7, 9).unwrap() + } + + fn contributor(value: &str) -> TrustedContributorId { + TrustedContributorId::new(value).unwrap() + } + + fn payload(byte: u8) -> PayloadDigest { + PayloadDigest::from_bytes([byte; 32]) + } + + fn request( + write_id: &str, + predecessor_token: Option, + actor: &str, + digest: PayloadDigest, + ) -> AdmissionRequest { + AdmissionRequest { + identity: identity(), + logical_id: "person-17".to_string(), + envelope: StreamWriteEnvelope { + stream_incarnation_id: STREAM_INCARNATION.to_string(), + write_id: write_id.to_string(), + predecessor_token, + }, + contributor_id: contributor(actor), + payload_digest: digest, + } + } + + fn present_authority( + request: &AdmissionRequest, + token: StreamToken, + attempt_id: &str, + ordinal: u64, + ) -> (StreamTokenAuthorityRow, TrustedStreamRowMetadata) { + let metadata = TrustedStreamRowMetadata::new_admission( + request, + token, + request.envelope.predecessor_token, + 1, + attempt_id.to_string(), + ordinal, + ) + .unwrap(); + let authority = StreamTokenAuthorityRow::from_present_metadata( + request.identity, + request.logical_id.clone(), + ENROLLMENT_ID.to_string(), + &metadata, + ) + .unwrap(); + (authority, metadata) + } + + #[test] + fn sha256_wire_form_is_exact_and_serde_uses_only_that_form() { + let token = StreamToken::from_bytes([0xab; 32]); + let expected = format!("sha256:{}", "ab".repeat(32)); + assert_eq!(token.to_string(), expected); + assert_eq!(StreamToken::from_str(&expected).unwrap(), token); + assert_eq!( + serde_json::to_string(&token).unwrap(), + format!("\"{expected}\"") + ); + assert_eq!( + serde_json::from_str::(&format!("\"{expected}\"")).unwrap(), + token + ); + + for malformed in [ + format!("SHA256:{}", "ab".repeat(32)), + format!("sha256:{}", "AB".repeat(32)), + format!(" sha256:{}", "ab".repeat(32)), + format!("sha256:{} ", "ab".repeat(32)), + format!("sha256:{}", "ab".repeat(31)), + format!("sha256:{}g", "ab".repeat(31)), + "q6urq6urq6urq6urq6urq6urq6urq6urq6urq6s=".to_string(), + ] { + assert!(StreamToken::from_str(&malformed).is_err(), "{malformed}"); + } + } + + #[test] + fn token_and_payload_derivations_are_versioned_deterministic_and_bound() { + let schema_hash = format!("sha256:{}", "0a".repeat(32)); + let digest = PayloadDigest::derive(&PayloadDigestInput { + identity: identity(), + accepted_schema_hash: &schema_hash, + canonical_payload: b"typed-row-v1", + }) + .unwrap(); + assert_eq!( + digest.to_string(), + "sha256:e2221e8a802628e78feb85edea4664cef2bf251118c41ba445daaa138d531af1" + ); + assert_eq!( + digest, + PayloadDigest::derive(&PayloadDigestInput { + identity: identity(), + accepted_schema_hash: &schema_hash, + canonical_payload: b"typed-row-v1", + }) + .unwrap() + ); + assert_ne!( + digest, + PayloadDigest::derive(&PayloadDigestInput { + identity: identity(), + accepted_schema_hash: &schema_hash, + canonical_payload: b"typed-row-v2", + }) + .unwrap() + ); + + let first_request = request(WRITE_X, None, "actor:alice", digest); + let token = first_request.candidate_token().unwrap(); + assert_eq!( + token.to_string(), + "sha256:6b665af0a10fec538e32469a44454af0ee213dcf5cc2cedd0d472a10cf65b758" + ); + assert_eq!(token, first_request.candidate_token().unwrap()); + let other_actor = request(WRITE_X, None, "actor:bob", digest); + assert_ne!(token, other_actor.candidate_token().unwrap()); + let other_table = AdmissionRequest { + identity: TableIdentity::new(8, 9).unwrap(), + ..first_request.clone() + }; + assert_ne!(token, other_table.candidate_token().unwrap()); + } + + #[test] + fn new_absent_key_requires_null_predecessor_and_matching_incarnation() { + let first = request(WRITE_X, None, "actor:alice", payload(1)); + let candidate = first.candidate_token().unwrap(); + assert_eq!( + classify_admission(STREAM_INCARNATION, &first, None, None).unwrap(), + AdmissionClassification::New { + candidate_token: candidate + } + ); + + let stale_predecessor = request( + WRITE_X, + Some(StreamToken::from_bytes([9; 32])), + "actor:alice", + payload(1), + ); + assert_eq!( + classify_admission(STREAM_INCARNATION, &stale_predecessor, None, None).unwrap(), + AdmissionClassification::SequenceConflict { + current_token: None + } + ); + + let mut stale_incarnation = first; + stale_incarnation.envelope.stream_incarnation_id = NEXT_STREAM_INCARNATION.to_string(); + assert_eq!( + classify_admission(STREAM_INCARNATION, &stale_incarnation, None, None).unwrap(), + AdmissionClassification::BindingChanged { + current_stream_incarnation_id: STREAM_INCARNATION.to_string() + } + ); + } + + #[test] + fn exact_current_retry_returns_persisted_origin_and_certificate() { + let first = request(WRITE_X, None, "actor:alice", payload(1)); + let token = first.candidate_token().unwrap(); + let (authority, metadata) = present_authority(&first, token, ATTEMPT_X, 17); + assert_eq!( + classify_admission( + STREAM_INCARNATION, + &first, + Some(&authority), + Some(&metadata), + ) + .unwrap(), + AdmissionClassification::AlreadyDurable { + authority, + present_metadata: Some(metadata), + } + ); + } + + #[test] + fn same_occurrence_reused_by_another_actor_or_payload_is_idempotency_conflict() { + let first = request(WRITE_X, None, "actor:alice", payload(1)); + let token = first.candidate_token().unwrap(); + let (authority, metadata) = present_authority(&first, token, ATTEMPT_X, 0); + + for reused in [ + request(WRITE_X, None, "actor:bob", payload(1)), + request(WRITE_X, None, "actor:alice", payload(2)), + ] { + assert_eq!( + classify_admission( + STREAM_INCARNATION, + &reused, + Some(&authority), + Some(&metadata), + ) + .unwrap(), + AdmissionClassification::IdempotencyConflict { + current_token: token + } + ); + } + } + + #[test] + fn x_then_y_then_retry_x_is_a_sequence_conflict_not_a_stale_write() { + let x = request(WRITE_X, None, "actor:alice", payload(1)); + let token_x = x.candidate_token().unwrap(); + let y = request(WRITE_Y, Some(token_x), "actor:alice", payload(2)); + let token_y = y.candidate_token().unwrap(); + let (authority_y, metadata_y) = present_authority(&y, token_y, ATTEMPT_Y, 1); + + assert_eq!( + classify_admission( + STREAM_INCARNATION, + &x, + Some(&authority_y), + Some(&metadata_y), + ) + .unwrap(), + AdmissionClassification::SequenceConflict { + current_token: Some(token_y) + } + ); + } + + #[test] + fn successor_must_name_the_complete_current_token() { + let x = request(WRITE_X, None, "actor:alice", payload(1)); + let token_x = x.candidate_token().unwrap(); + let (authority_x, metadata_x) = present_authority(&x, token_x, ATTEMPT_X, 0); + let y = request(WRITE_Y, Some(token_x), "actor:alice", payload(2)); + let token_y = y.candidate_token().unwrap(); + assert_eq!( + classify_admission( + STREAM_INCARNATION, + &y, + Some(&authority_x), + Some(&metadata_x), + ) + .unwrap(), + AdmissionClassification::New { + candidate_token: token_y + } + ); + + // The UUID alone is not the occurrence key. Reusing it against the + // newly current predecessor is a distinct (though SDK-discouraged) + // occurrence and remains safe because its full token changes. + let reused_uuid = request(WRITE_X, Some(token_x), "actor:alice", payload(2)); + assert!(matches!( + classify_admission( + STREAM_INCARNATION, + &reused_uuid, + Some(&authority_x), + Some(&metadata_x), + ) + .unwrap(), + AdmissionClassification::New { .. } + )); + } + + #[test] + fn present_authority_requires_exact_base_metadata_but_withdrawn_does_not() { + let first = request(WRITE_X, None, "actor:alice", payload(1)); + let token = first.candidate_token().unwrap(); + let (authority, mut metadata) = present_authority(&first, token, ATTEMPT_X, 0); + assert!( + classify_admission( + STREAM_INCARNATION, + &first, + None, + Some(&metadata), + ) + .is_err(), + "a base attribution without selected token authority must not reset to an empty chain" + ); + assert!( + validate_authority_base_pair(first.identity, &first.logical_id, None, Some(&metadata)) + .is_err(), + "the shared admission/fold validator must reject orphaned base attribution" + ); + assert!(classify_admission(STREAM_INCARNATION, &first, Some(&authority), None).is_err()); + metadata.origin = StreamRowOrigin::Admission { + admission_attempt_id: ATTEMPT_Y.to_string(), + caller_ordinal: 0, + }; + assert!( + classify_admission( + STREAM_INCARNATION, + &first, + Some(&authority), + Some(&metadata), + ) + .is_err() + ); + + let withdrawn = StreamTokenAuthorityRow { + disposition: StreamTokenDisposition::Withdrawn, + terminal_correction: Some(StreamTerminalCorrection { + actor: contributor("actor:operator"), + correction_id: ATTEMPT_Y.to_string(), + }), + ..authority + }; + assert!(matches!( + classify_admission(STREAM_INCARNATION, &first, Some(&withdrawn), None).unwrap(), + AdmissionClassification::AlreadyDurable { + present_metadata: None, + .. + } + )); + } + + #[test] + fn successor_cannot_advance_over_corrupt_present_authority() { + let first = request(WRITE_X, None, "actor:alice", payload(1)); + let token = first.candidate_token().unwrap(); + let (authority, mut metadata) = present_authority(&first, token, ATTEMPT_X, 0); + let successor = request(WRITE_Y, Some(token), "actor:alice", payload(2)); + + assert!( + classify_admission( + STREAM_INCARNATION, + &successor, + Some(&authority), + None, + ) + .is_err() + ); + metadata.stream_token = successor.candidate_token().unwrap(); + assert!( + classify_admission( + STREAM_INCARNATION, + &successor, + Some(&authority), + Some(&metadata), + ) + .is_err() + ); + } + + #[test] + fn row_metadata_enforces_nonzero_chain_and_depth_one_fold_base() { + let first = request(WRITE_X, None, "actor:alice", payload(1)); + let token = first.candidate_token().unwrap(); + let (_, mut metadata) = present_authority(&first, token, ATTEMPT_X, 0); + metadata.chain_depth = 0; + assert!(metadata.validate_for(identity(), "person-17").is_err()); + + metadata.chain_depth = 1; + metadata.fold_base_token = Some(StreamToken::from_bytes([9; 32])); + assert!(metadata.validate_for(identity(), "person-17").is_err()); + + metadata.chain_depth = 2; + metadata.fold_base_token = None; + assert!(metadata.validate_for(identity(), "person-17").is_ok()); + } + + #[test] + fn tagged_origin_and_disposition_enforce_variant_nullability() { + let origin = StreamRowOrigin::Admission { + admission_attempt_id: ATTEMPT_X.to_string(), + caller_ordinal: 3, + }; + let json = serde_json::to_value(&origin).unwrap(); + assert_eq!(json["kind"], "admission"); + assert!(json.get("correction_id").is_none()); + assert!( + serde_json::from_value::(serde_json::json!({ + "kind": "admission", + "admission_attempt_id": ATTEMPT_X, + "caller_ordinal": 3, + "correction_id": ATTEMPT_Y + })) + .is_err() + ); + + let first = request(WRITE_X, None, "actor:alice", payload(1)); + let token = first.candidate_token().unwrap(); + let (mut authority, _) = present_authority(&first, token, ATTEMPT_X, 0); + authority.terminal_correction = Some(StreamTerminalCorrection { + actor: contributor("actor:operator"), + correction_id: ATTEMPT_Y.to_string(), + }); + assert!(authority.validate().is_err()); + authority.disposition = StreamTokenDisposition::Withdrawn; + assert!(authority.validate().is_ok()); + authority.terminal_correction = None; + assert!(authority.validate().is_err()); + } + + #[test] + fn fold_attribution_summary_is_complete_canonical_and_serde_exact() { + let summary = StreamFoldAttributionSummary { + visible_contributor_count: 2, + visible_write_count: 3, + winning_attribution_digest: format!("sha256:{}", "ab".repeat(32)), + }; + summary.validate().unwrap(); + let encoded = serde_json::to_string(&summary).unwrap(); + assert_eq!( + serde_json::from_str::(&encoded).unwrap(), + summary + ); + assert!( + serde_json::from_value::(serde_json::json!({ + "visible_contributor_count": 2, + "visible_write_count": 3, + "winning_attribution_digest": format!("sha256:{}", "ab".repeat(32)), + "unexpected": true + })) + .is_err() + ); + + for invalid in [ + StreamFoldAttributionSummary { + visible_contributor_count: 0, + ..summary.clone() + }, + StreamFoldAttributionSummary { + visible_contributor_count: 4, + ..summary.clone() + }, + StreamFoldAttributionSummary { + visible_write_count: 0, + ..summary.clone() + }, + StreamFoldAttributionSummary { + winning_attribution_digest: format!("sha256:{}", "AB".repeat(32)), + ..summary.clone() + }, + ] { + assert!(invalid.validate().is_err(), "{invalid:?}"); + } + } + + #[test] + fn hidden_arrow_struct_is_exact_nullable_and_round_trips_trusted_metadata() { + let field = trusted_stream_metadata_field(); + assert_eq!(field.name(), crate::db::STREAM_METADATA_COLUMN); + assert!(field.is_nullable()); + validate_trusted_stream_metadata_field(&field).unwrap(); + + let all_null_values = vec![None; 8192]; + let all_null = build_trusted_stream_metadata_array(&all_null_values).unwrap(); + assert_eq!(all_null.data_type(), field.data_type()); + assert_eq!(all_null.len(), all_null_values.len()); + assert_eq!(all_null.null_count(), all_null_values.len()); + let all_null_bytes = all_null.get_array_memory_size(); + assert!( + all_null_bytes < 1024 * 1024, + "all-null trusted metadata must not retain eager Utf8 child capacity: {all_null_bytes} bytes" + ); + assert_eq!( + decode_trusted_stream_metadata(all_null.as_ref(), all_null_values.len() - 1).unwrap(), + None + ); + + let first = request(WRITE_X, None, "actor:alice", payload(1)); + let token = first.candidate_token().unwrap(); + let (_, metadata) = present_authority(&first, token, ATTEMPT_X, 9); + let array = build_trusted_stream_metadata_array(&[None, Some(metadata.clone())]).unwrap(); + assert_eq!( + decode_trusted_stream_metadata(array.as_ref(), 0).unwrap(), + None + ); + assert_eq!( + decode_trusted_stream_metadata(array.as_ref(), 1).unwrap(), + Some(metadata) + ); + assert!(decode_trusted_stream_metadata(array.as_ref(), 2).is_err()); + + let wrong = Field::new(crate::db::STREAM_METADATA_COLUMN, DataType::Utf8, true); + assert!(validate_trusted_stream_metadata_field(&wrong).is_err()); + + let missing = Schema::new(vec![Field::new("id", DataType::Utf8, false)]); + assert!(validate_trusted_stream_metadata_schema(&missing).is_err()); + let case_lookalike = Schema::new(vec![Field::new( + "__OMNIGRAPH_STREAM_V1$", + trusted_stream_metadata_field().data_type().clone(), + true, + )]); + assert!(validate_trusted_stream_metadata_schema(&case_lookalike).is_err()); + let canonical = Schema::new(vec![trusted_stream_metadata_field()]); + validate_trusted_stream_metadata_schema(&canonical).unwrap(); + } + + #[test] + fn uuid_fields_refuse_nil_and_noncanonical_forms() { + let mut invalid = request(WRITE_X, None, "actor:alice", payload(1)); + invalid.envelope.write_id = "00000000-0000-0000-0000-000000000000".to_string(); + assert!(invalid.validate().is_err()); + invalid.envelope.write_id = "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa".to_uppercase(); + assert!(invalid.validate().is_err()); + + assert!( + serde_json::from_value::(serde_json::json!({ + "stream_incarnation_id": STREAM_INCARNATION, + "write_id": WRITE_X + })) + .is_err(), + "nullable protocol fields must be explicit rather than serde-defaulted" + ); + assert!( + serde_json::from_value::(serde_json::json!({ + "stream_incarnation_id": STREAM_INCARNATION, + "write_id": WRITE_X, + "predecessor_token": null + })) + .is_ok() + ); + } +} diff --git a/crates/omnigraph/src/db/manifest/tests.rs b/crates/omnigraph/src/db/manifest/tests.rs index d87bbd6a..6039ab6a 100644 --- a/crates/omnigraph/src/db/manifest/tests.rs +++ b/crates/omnigraph/src/db/manifest/tests.rs @@ -1,7 +1,7 @@ use std::collections::{BTreeMap, HashMap}; use std::sync::Arc; -use arrow_array::{Int32Array, RecordBatch, RecordBatchIterator, StringArray, UInt64Array}; +use arrow_array::{Array, Int32Array, RecordBatch, RecordBatchIterator, StringArray, UInt64Array}; use arrow_schema::{DataType, Field, Schema}; use async_trait::async_trait; use futures::TryStreamExt; @@ -21,6 +21,7 @@ use super::publisher::{ PublishPrecondition, }; use super::state::read_publish_scan; +use super::stream_token::{STREAM_FOLD_ACTOR, StreamFoldAttributionSummary}; use super::*; use omnigraph_compiler::schema::parser::parse_schema; use omnigraph_compiler::{ @@ -120,6 +121,41 @@ fn table_identity_rejects_zero_and_drives_paths_and_object_ids() { ); } +#[test] +fn manifest_graph_table_schema_adds_exact_hidden_metadata_and_rejects_lookalikes() { + let catalog = build_test_catalog(); + let input = &catalog.node_types["Person"].arrow_schema; + assert!( + input + .field_with_name(crate::db::STREAM_METADATA_COLUMN) + .is_err() + ); + + let physical = super::graph::keyed_graph_table_schema(input).unwrap(); + let hidden = physical + .field_with_name(crate::db::STREAM_METADATA_COLUMN) + .unwrap(); + assert_eq!( + hidden, + &super::stream_token::trusted_stream_metadata_field() + ); + assert!(hidden.is_nullable()); + + let mut supplied = input.fields().iter().cloned().collect::>(); + supplied.push(Arc::new(Field::new( + crate::db::STREAM_METADATA_COLUMN, + DataType::Utf8, + true, + ))); + let supplied = Arc::new(Schema::new_with_metadata(supplied, input.metadata.clone())); + let error = super::graph::keyed_graph_table_schema(&supplied).unwrap_err(); + assert!( + error + .to_string() + .contains("non-canonical reserved physical field") + ); +} + #[tokio::test] async fn historical_alias_binding_keeps_same_name_node_and_edge_identities_distinct() { let dir = tempfile::tempdir().unwrap(); @@ -161,6 +197,105 @@ async fn test_init_creates_manifest_and_sub_tables() { } } +#[tokio::test] +async fn init_materializes_one_exact_manifest_selected_stream_token_dataset() { + let dir = tempfile::tempdir().unwrap(); + let uri = dir.path().to_str().unwrap(); + let catalog = build_test_catalog(); + + let mc = ManifestCoordinator::init(uri, &catalog).await.unwrap(); + let snapshot = mc.snapshot(); + let authority = snapshot.stream_token_authority(); + assert_eq!(authority.location, token_store::STREAM_TOKEN_DATASET_PATH); + assert_eq!( + authority.schema_version, + token_store::STREAM_TOKEN_AUTHORITY_SCHEMA_VERSION + ); + assert_eq!( + authority.schema_hash, + token_store::stream_token_schema_hash() + ); + assert_eq!( + authority.current_head_witness.branch_identifier, + BranchIdentifier::main() + ); + + let token_dataset = snapshot.open_stream_token_authority().await.unwrap(); + assert_eq!( + token_dataset.version().version, + authority.current_head_witness.table_version + ); + assert_eq!(token_dataset.count_rows(None).await.unwrap(), 0); + let actual_schema: Schema = token_dataset.schema().into(); + assert_eq!(&actual_schema, token_store::stream_token_schema().as_ref()); + assert_eq!( + actual_schema + .field_with_name("id") + .unwrap() + .metadata() + .get(lance::datatypes::LANCE_UNENFORCED_PRIMARY_KEY) + .map(String::as_str), + Some("true") + ); + + let manifest = open_manifest_dataset(uri, None).await.unwrap(); + let batches: Vec = manifest + .scan() + .try_into_stream() + .await + .unwrap() + .try_collect() + .await + .unwrap(); + let mut authority_rows = 0; + for batch in batches { + let object_types = super::state::string_column(&batch, "object_type").unwrap(); + let stable_ids = batch + .column_by_name("stable_table_id") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + let incarnation_ids = batch + .column_by_name("table_incarnation_id") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + for row in 0..batch.num_rows() { + if object_types.value(row) == OBJECT_TYPE_STREAM_TOKEN_AUTHORITY { + authority_rows += 1; + assert!(stable_ids.is_null(row)); + assert!(incarnation_ids.is_null(row)); + } + } + } + assert_eq!(authority_rows, 1); +} + +#[test] +fn stream_token_internal_primary_key_is_canonical_and_collision_free() { + let identity = TableIdentity::new(7, 9).unwrap(); + assert_eq!( + token_store::stream_token_row_id(identity, "a:b").unwrap(), + "stream-token-v1:0000000000000007:0000000000000009:a:b" + ); + assert_ne!( + token_store::stream_token_row_id(identity, "a:b").unwrap(), + token_store::stream_token_row_id(identity, "a3a62").unwrap() + ); + assert!( + token_store::stream_token_row_id( + TableIdentity { + stable_table_id: 0, + table_incarnation_id: 1, + }, + "id" + ) + .is_err() + ); +} + #[tokio::test] async fn test_open_reads_existing_manifest() { let dir = tempfile::tempdir().unwrap(); @@ -1249,7 +1384,8 @@ impl ManifestBatchPublisher for RecordingPublisher { ManifestChange::RegisterTable(_) | ManifestChange::RenameTable(_) | ManifestChange::Tombstone(_) - | ManifestChange::SetStreamLifecycle { .. } => None, + | ManifestChange::SetStreamLifecycle { .. } + | ManifestChange::SetStreamTokenAuthority { .. } => None, }) .collect(); self.requests.lock().await.extend_from_slice(&requests); @@ -1433,34 +1569,147 @@ async fn append_person_and_make_update( } } +#[tokio::test] +async fn table_and_stream_token_pointers_publish_in_one_manifest_cas() { + let dir = tempfile::tempdir().unwrap(); + let uri = dir.path().to_str().unwrap(); + let catalog = build_test_catalog(); + let mut mc = ManifestCoordinator::init(uri, &catalog).await.unwrap(); + let before = mc.snapshot(); + let person = before.entry("node:Person").unwrap().clone(); + let expected_token = before.stream_token_authority().clone(); + + // Move only the physical token HEAD. Until the manifest row changes, the + // old snapshot must continue opening the exact selected version. + let mut token_dataset = before.open_stream_token_authority().await.unwrap(); + token_dataset + .update_config([("omnigraph.test.token_effect", Some("1"))]) + .await + .unwrap(); + let next_token = token_store::stream_token_authority_entry_for_dataset(&token_dataset) + .await + .unwrap(); + assert!( + next_token.current_head_witness.table_version + > expected_token.current_head_witness.table_version + ); + assert_eq!( + before + .open_stream_token_authority() + .await + .unwrap() + .version() + .version, + expected_token.current_head_witness.table_version + ); + + let table_update = append_person_and_make_update(uri, &person, "TokenAtomic").await; + let manifest_before = mc.version(); + mc.commit_changes(&[ + ManifestChange::Update(table_update.clone()), + ManifestChange::SetStreamTokenAuthority { + expected: expected_token.clone(), + next: next_token.clone(), + }, + ]) + .await + .unwrap(); + + assert_eq!(mc.version(), manifest_before + 1); + let after = mc.snapshot(); + assert_eq!( + after.entry("node:Person").unwrap().table_version, + table_update.table_version + ); + assert_eq!(after.stream_token_authority(), &next_token); + assert_eq!( + after + .open_stream_token_authority() + .await + .unwrap() + .version() + .version, + next_token.current_head_witness.table_version + ); + + let stale = mc + .commit_changes(&[ManifestChange::SetStreamTokenAuthority { + expected: expected_token, + next: next_token.clone(), + }]) + .await + .unwrap_err(); + assert!(matches!( + stale, + OmniError::Manifest(ManifestError { + details: Some(ManifestConflictDetails::ReadSetChanged { .. }), + .. + }) + )); + + let reopened = ManifestCoordinator::open(uri).await.unwrap().snapshot(); + assert_eq!(reopened.stream_token_authority(), &next_token); + assert_eq!( + reopened.entry("node:Person").unwrap().table_version, + table_update.table_version + ); +} + fn stream_lifecycle_for_person( person_entry: &SubTableEntry, table_version: u64, lifecycle: StreamLifecycle, ) -> StreamLifecycleEntry { let shard_id = "22222222-2222-4222-8222-222222222222".to_string(); - StreamLifecycleEntry { - identity: person_entry.identity, - diagnostic_table_key: person_entry.table_key.clone(), - lifecycle, - binding: StreamPhysicalBinding { - stable_table_id: person_entry.identity.stable_table_id, - table_incarnation_id: person_entry.identity.table_incarnation_id, - table_location: person_entry.table_path.clone(), - table_branch: None, - enrollment_id: "11111111-1111-4111-8111-111111111111".to_string(), - shard_ids: vec![shard_id.clone()], - stream_config_version: STREAM_CONFIG_VERSION, - stream_config_hash: format!("sha256:{}", "a".repeat(64)), - }, - current_head_witness: CurrentHeadWitness { - branch_identifier: BranchIdentifier::main(), - table_version, - transaction_uuid: "33333333-3333-4333-8333-333333333333".to_string(), - manifest_e_tag: None, - }, - epoch_floor_by_shard: BTreeMap::from([(shard_id, 1)]), + let binding = StreamPhysicalBinding { + stable_table_id: person_entry.identity.stable_table_id, + table_incarnation_id: person_entry.identity.table_incarnation_id, + table_location: person_entry.table_path.clone(), + table_branch: None, + enrollment_id: "11111111-1111-4111-8111-111111111111".to_string(), + shard_ids: vec![shard_id.clone()], + stream_config_version: STREAM_CONFIG_VERSION, + stream_config_hash: format!("sha256:{}", "a".repeat(64)), + }; + let head = CurrentHeadWitness { + branch_identifier: BranchIdentifier::main(), + table_version, + transaction_uuid: "33333333-3333-4333-8333-333333333333".to_string(), + manifest_e_tag: None, + }; + let mut entry = StreamLifecycleEntry::new_open_enrollment( + person_entry.identity, + person_entry.table_key.clone(), + binding.clone(), + head.clone(), + BTreeMap::from([(shard_id.clone(), 1)]), + stream::EnrollmentReceipt::new( + "44444444-4444-4444-8444-444444444444".to_string(), + format!("sha256:{}", "b".repeat(64)), + "55555555-5555-4555-8555-555555555555".to_string(), + binding.clone(), + ) + .unwrap(), + ) + .unwrap(); + if lifecycle == StreamLifecycle::Draining { + entry.lifecycle = StreamLifecycle::Draining; + entry.lifecycle_revision = 2; + entry.drain = Some(stream::DrainDescriptor { + drain_id: "66666666-6666-4666-8666-666666666666".to_string(), + operation_expected_revision: 1, + operation_request_digest: format!("sha256:{}", "c".repeat(64)), + goal: stream::DrainGoal::OpenAfterFold, + initiating_actor: "actor:test".to_string(), + initiated_at: 1, + expected_binding: binding, + expected_current_head_witness: head, + target_epoch_floor_by_shard: BTreeMap::from([(shard_id, 1)]), + guarded_operation: None, + }); } + entry.validate().unwrap(); + entry } #[tokio::test] @@ -1595,8 +1844,7 @@ async fn stream_lifecycle_and_table_pointer_publish_in_one_manifest_cas() { }) if lifecycle == "DRAINING" && operation == "schema apply" )); - let mut sealed = draining; - sealed.lifecycle = StreamLifecycle::Sealed; + let sealed = super::stream::test_sealed_lifecycle_from(&draining).unwrap(); let expected_draining = reopened .snapshot() .stream_lifecycle(person_entry.identity) @@ -1918,7 +2166,7 @@ async fn test_init_stamps_internal_schema_version() { ManifestCoordinator::init(uri, &catalog).await.unwrap(); let ds = open_manifest_dataset(uri, None).await.unwrap(); - assert_eq!(super::migrations::INTERNAL_MANIFEST_SCHEMA_VERSION, 8); + assert_eq!(super::migrations::INTERNAL_MANIFEST_SCHEMA_VERSION, 9); assert_eq!( super::migrations::read_stamp(&ds), super::migrations::INTERNAL_MANIFEST_SCHEMA_VERSION, @@ -2199,6 +2447,109 @@ fn lineage_now_micros() -> i64 { .as_micros() as i64 } +#[tokio::test] +async fn graph_commit_metadata_round_trips_optional_stream_fold_attribution() { + let dir = tempfile::tempdir().unwrap(); + let uri = dir.path().to_str().unwrap(); + let catalog = build_test_catalog(); + let _coordinator = ManifestCoordinator::init(uri, &catalog).await.unwrap(); + let publisher = GraphNamespacePublisher::new(uri, None); + let empty = HashMap::new(); + + let ordinary = LineageIntent { + graph_commit_id: ulid::Ulid::new().to_string(), + branch: None, + actor_id: Some("act-alice".to_string()), + merged_parent_commit_id: None, + created_at: lineage_now_micros(), + stream_fold_attribution: None, + }; + publisher + .publish(&[], &empty, Some(&ordinary)) + .await + .unwrap(); + + let summary = StreamFoldAttributionSummary { + visible_contributor_count: 2, + visible_write_count: 3, + winning_attribution_digest: format!("sha256:{}", "ab".repeat(32)), + }; + let historical_fold = LineageIntent { + graph_commit_id: ulid::Ulid::new().to_string(), + branch: None, + actor_id: Some(STREAM_FOLD_ACTOR.to_string()), + merged_parent_commit_id: None, + created_at: lineage_now_micros(), + stream_fold_attribution: None, + }; + publisher + .publish(&[], &empty, Some(&historical_fold)) + .await + .expect("pre-commitment v11 folds with the fixed actor remain readable"); + + let wrong_actor = LineageIntent { + graph_commit_id: ulid::Ulid::new().to_string(), + branch: None, + actor_id: Some("act-not-system".to_string()), + merged_parent_commit_id: None, + created_at: lineage_now_micros(), + stream_fold_attribution: Some(summary.clone()), + }; + let error = publisher + .publish(&[], &empty, Some(&wrong_actor)) + .await + .expect_err("ordinary actors must not publish a stream-fold commitment"); + assert!(error.to_string().contains(STREAM_FOLD_ACTOR)); + + let fold = LineageIntent { + graph_commit_id: ulid::Ulid::new().to_string(), + branch: None, + actor_id: Some(STREAM_FOLD_ACTOR.to_string()), + merged_parent_commit_id: None, + created_at: lineage_now_micros(), + stream_fold_attribution: Some(summary.clone()), + }; + publisher.publish(&[], &empty, Some(&fold)).await.unwrap(); + + let dataset = open_manifest_dataset(uri, None).await.unwrap(); + let (rows, _) = read_graph_lineage(&dataset).await.unwrap(); + assert_eq!( + rows.iter() + .find(|row| row.graph_commit_id == ordinary.graph_commit_id) + .unwrap() + .stream_fold_attribution + .as_ref(), + None + ); + assert_eq!( + rows.iter() + .find(|row| row.graph_commit_id == historical_fold.graph_commit_id) + .unwrap() + .stream_fold_attribution + .as_ref(), + None + ); + assert_eq!( + rows.iter() + .find(|row| row.graph_commit_id == fold.graph_commit_id) + .unwrap() + .stream_fold_attribution + .as_ref(), + Some(&summary) + ); + assert!( + rows.iter() + .filter(|row| row.graph_commit_id != fold.graph_commit_id) + .all(|row| row.stream_fold_attribution.is_none()), + "genesis and every ordinary commit must keep attribution logically null" + ); + assert!( + rows.iter() + .all(|row| row.graph_commit_id != wrong_actor.graph_commit_id), + "rejected attribution intents must not leave immutable graph_commit rows" + ); +} + /// Race two lineage-only publishes against the same exact named-branch head. /// Returns after proving exactly one committed and the loser surfaced a typed /// read-set change. `establish_head=false` exercises the load-bearing absent-row @@ -2221,6 +2572,7 @@ async fn assert_exact_named_head_race(establish_head: bool) { actor_id: None, merged_parent_commit_id: None, created_at: lineage_now_micros(), + stream_fold_attribution: None, }; publisher .publish(&[], &empty, Some(&intent)) @@ -2250,6 +2602,7 @@ async fn assert_exact_named_head_race(establish_head: bool) { actor_id: Some("act-a".to_string()), merged_parent_commit_id: None, created_at: lineage_now_micros(), + stream_fold_attribution: None, }; let intent_b = LineageIntent { graph_commit_id: ulid::Ulid::new().to_string(), @@ -2257,6 +2610,7 @@ async fn assert_exact_named_head_race(establish_head: bool) { actor_id: Some("act-b".to_string()), merged_parent_commit_id: None, created_at: lineage_now_micros(), + stream_fold_attribution: None, }; let publisher_a = GraphNamespacePublisher::new(uri, Some("feature")); let publisher_b = GraphNamespacePublisher::new(uri, Some("feature")); @@ -2523,6 +2877,7 @@ async fn concurrent_disjoint_writes_share_head_and_form_linear_chain() { actor_id: Some("act-a".to_string()), merged_parent_commit_id: None, created_at: lineage_now_micros(), + stream_fold_attribution: None, }; let intent_b = LineageIntent { graph_commit_id: ulid::Ulid::new().to_string(), @@ -2530,6 +2885,7 @@ async fn concurrent_disjoint_writes_share_head_and_form_linear_chain() { actor_id: Some("act-b".to_string()), merged_parent_commit_id: None, created_at: lineage_now_micros(), + stream_fold_attribution: None, }; // Empty expected-versions: the two writers are disjoint, so neither asserts a // version on the other's table; contention is purely the shared head row. @@ -2609,6 +2965,7 @@ async fn concurrent_disjoint_writes_form_linear_chain_on_s3() { actor_id: Some("act-a".to_string()), merged_parent_commit_id: None, created_at: lineage_now_micros(), + stream_fold_attribution: None, }; let intent_b = LineageIntent { graph_commit_id: ulid::Ulid::new().to_string(), @@ -2616,6 +2973,7 @@ async fn concurrent_disjoint_writes_form_linear_chain_on_s3() { actor_id: Some("act-b".to_string()), merged_parent_commit_id: None, created_at: lineage_now_micros(), + stream_fold_attribution: None, }; let empty = HashMap::new(); let (res_a, res_b) = tokio::join!( @@ -2705,6 +3063,7 @@ async fn n_concurrent_disjoint_writers_converge_to_one_linear_chain() { actor_id: None, merged_parent_commit_id: None, created_at: lineage_now_micros(), + stream_fold_attribution: None, }; let publisher = GraphNamespacePublisher::new(&uri, None); match publisher.publish(&changes, &empty, Some(&intent)).await { diff --git a/crates/omnigraph/src/db/manifest/token_store.rs b/crates/omnigraph/src/db/manifest/token_store.rs new file mode 100644 index 00000000..e53dd4f3 --- /dev/null +++ b/crates/omnigraph/src/db/manifest/token_store.rs @@ -0,0 +1,1203 @@ +//! Manifest-selected RFC-026 stream-token dataset authority. +//! +//! `_stream_tokens.lance` is a graph-global physical participant. Its raw +//! Lance HEAD is never logical authority: `__manifest` stores one exact +//! [`CurrentHeadWitness`], and every reader opens that selected version and +//! verifies the complete witness before using it. + +use std::collections::{BTreeMap, HashMap}; +use std::sync::Arc; + +use std::str::FromStr; + +use arrow_array::{Array, RecordBatch, RecordBatchIterator, StringArray, UInt32Array, UInt64Array}; +use arrow_schema::{DataType, Field, Schema, SchemaRef}; +use datafusion::prelude::{col, lit}; +use futures::TryStreamExt; +use lance::Dataset; +use lance::dataset::refs::BranchIdentifier; +use lance::dataset::write::merge_insert::SourceDedupeBehavior; +use lance::dataset::{MergeInsertBuilder, WhenMatched, WhenNotMatched, WriteMode, WriteParams}; +use lance::datatypes::LANCE_UNENFORCED_PRIMARY_KEY; +use lance_file::version::LanceFileVersion; +use lance_index::mem_wal::ShardId; +use serde::{Deserialize, Serialize}; +use sha2::{Digest, Sha256}; + +use crate::error::{OmniError, Result}; + +use super::layout::{stream_token_authority_object_id, stream_token_uri}; +use super::stream_token::{ + PayloadDigest, StreamRowOrigin, StreamTerminalCorrection, StreamToken, StreamTokenAuthorityRow, + StreamTokenDisposition, TrustedContributorId, +}; +use super::{CurrentHeadWitness, TableIdentity}; + +pub(crate) const STREAM_TOKEN_DATASET_PATH: &str = "_stream_tokens.lance"; +pub(crate) const STREAM_TOKEN_AUTHORITY_SCHEMA_VERSION: u32 = 1; +const STREAM_TOKEN_AUTHORITY_PROTOCOL_VERSION: u32 = 1; + +/// Canonical schema descriptor hashed into the manifest authority row. +/// +/// Keep this in the same order and nullability as [`stream_token_schema`]. A +/// physical schema change requires a new descriptor, schema version, and graph +/// format strand; it must never be accepted through permissive field matching. +const STREAM_TOKEN_SCHEMA_DESCRIPTOR_V1: &str = concat!( + "omnigraph.stream-token-authority.schema.v1\n", + "id:utf8:required:unenforced-primary-key\n", + "stable_table_id:uint64:required\n", + "table_incarnation_id:uint64:required\n", + "logical_id:utf8:required\n", + "origin_enrollment_id:utf8:required\n", + "stream_incarnation_id:utf8:required\n", + "current_token:utf8:required\n", + "write_id:utf8:required\n", + "predecessor_token:utf8:nullable\n", + "disposition:utf8:required\n", + "contributor_id:utf8:required\n", + "payload_digest:utf8:required\n", + "origin_kind:utf8:required\n", + "origin_id:utf8:required\n", + "origin_ordinal:uint64:required\n", + "fold_base_token:utf8:nullable\n", + "chain_depth:uint32:required\n", + "terminal_correction_actor:utf8:nullable\n", + "terminal_correction_operation_id:utf8:nullable\n", +); + +/// The single graph-global token-table pointer materialized in `__manifest`. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(crate) struct StreamTokenAuthorityEntry { + /// Canonical graph-relative dataset path. This is never a user table path. + pub(crate) location: String, + pub(crate) schema_version: u32, + pub(crate) schema_hash: String, + /// Exact main-branch version selected by the same manifest snapshot. + /// Its provider-local `manifest_e_tag` is always absent: a local graph + /// copy changes inode-derived ETags without changing the immutable Lance + /// version or transaction identity. + pub(crate) current_head_witness: CurrentHeadWitness, +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct StreamTokenAuthorityPayload { + protocol_version: u32, + schema_version: u32, + schema_hash: String, + current_head_witness: CurrentHeadWitness, +} + +impl StreamTokenAuthorityEntry { + pub(crate) fn object_id(&self) -> &'static str { + stream_token_authority_object_id() + } + + pub(crate) fn validate(&self) -> Result<()> { + if self.location != STREAM_TOKEN_DATASET_PATH { + return Err(OmniError::manifest_internal(format!( + "stream-token authority location '{}' is not canonical '{}'", + self.location, STREAM_TOKEN_DATASET_PATH + ))); + } + if self.schema_version != STREAM_TOKEN_AUTHORITY_SCHEMA_VERSION { + return Err(OmniError::manifest_internal(format!( + "unsupported stream-token authority schema version {}, expected {}", + self.schema_version, STREAM_TOKEN_AUTHORITY_SCHEMA_VERSION + ))); + } + let expected_hash = stream_token_schema_hash(); + if self.schema_hash != expected_hash { + return Err(OmniError::manifest_internal(format!( + "stream-token authority schema hash '{}' does not match '{}'", + self.schema_hash, expected_hash + ))); + } + validate_head_witness(&self.current_head_witness) + } + + pub(super) fn to_metadata_json(&self) -> Result { + self.validate()?; + serde_json::to_string(&StreamTokenAuthorityPayload { + protocol_version: STREAM_TOKEN_AUTHORITY_PROTOCOL_VERSION, + schema_version: self.schema_version, + schema_hash: self.schema_hash.clone(), + current_head_witness: self.current_head_witness.clone(), + }) + .map_err(|error| { + OmniError::manifest_internal(format!( + "failed to encode stream-token authority metadata: {error}" + )) + }) + } + + pub(super) fn from_manifest_row( + object_id: &str, + location: Option<&str>, + table_version: Option, + table_branch: Option<&str>, + metadata_json: &str, + ) -> Result { + let expected_object_id = stream_token_authority_object_id(); + if object_id != expected_object_id { + return Err(OmniError::manifest_internal(format!( + "manifest stream_token_authority row has object_id '{object_id}', expected '{expected_object_id}'" + ))); + } + let payload: StreamTokenAuthorityPayload = + serde_json::from_str(metadata_json).map_err(|error| { + OmniError::manifest_internal(format!( + "failed to decode stream-token authority metadata: {error}" + )) + })?; + if payload.protocol_version != STREAM_TOKEN_AUTHORITY_PROTOCOL_VERSION { + return Err(OmniError::manifest_internal(format!( + "unsupported stream-token authority payload version {}, expected {}", + payload.protocol_version, STREAM_TOKEN_AUTHORITY_PROTOCOL_VERSION + ))); + } + let location = location.ok_or_else(|| { + OmniError::manifest_internal("manifest stream_token_authority row is missing location") + })?; + let entry = Self { + location: location.to_string(), + schema_version: payload.schema_version, + schema_hash: payload.schema_hash, + current_head_witness: payload.current_head_witness, + }; + entry.validate()?; + if table_version != Some(entry.current_head_witness.table_version) { + return Err(OmniError::manifest_internal( + "manifest stream_token_authority row table_version does not match its exact HEAD witness", + )); + } + if table_branch.is_some() { + return Err(OmniError::manifest_internal( + "manifest stream_token_authority row must select canonical main (table_branch = null)", + )); + } + Ok(entry) + } +} + +pub(crate) fn stream_token_schema() -> SchemaRef { + let primary_key_metadata: HashMap = + [(LANCE_UNENFORCED_PRIMARY_KEY.to_string(), "true".to_string())] + .into_iter() + .collect(); + Arc::new(Schema::new(vec![ + Field::new("id", DataType::Utf8, false).with_metadata(primary_key_metadata), + Field::new("stable_table_id", DataType::UInt64, false), + Field::new("table_incarnation_id", DataType::UInt64, false), + Field::new("logical_id", DataType::Utf8, false), + Field::new("origin_enrollment_id", DataType::Utf8, false), + Field::new("stream_incarnation_id", DataType::Utf8, false), + Field::new("current_token", DataType::Utf8, false), + Field::new("write_id", DataType::Utf8, false), + Field::new("predecessor_token", DataType::Utf8, true), + Field::new("disposition", DataType::Utf8, false), + Field::new("contributor_id", DataType::Utf8, false), + Field::new("payload_digest", DataType::Utf8, false), + Field::new("origin_kind", DataType::Utf8, false), + Field::new("origin_id", DataType::Utf8, false), + Field::new("origin_ordinal", DataType::UInt64, false), + Field::new("fold_base_token", DataType::Utf8, true), + Field::new("chain_depth", DataType::UInt32, false), + Field::new("terminal_correction_actor", DataType::Utf8, true), + Field::new("terminal_correction_operation_id", DataType::Utf8, true), + ])) +} + +pub(crate) fn stream_token_schema_hash() -> String { + let digest = Sha256::digest(STREAM_TOKEN_SCHEMA_DESCRIPTOR_V1.as_bytes()); + format!("sha256:{digest:x}") +} + +/// Collision-free canonical PK for `(table lifetime, logical id)`. +/// +/// The table components are fixed-width hexadecimal and the complete UTF-8 +/// logical id is the terminal component. Because it consumes the remainder of +/// the key (the identifier is never parsed by splitting from the right), even +/// embedded separators cannot alias another tuple. No zero/sentinel +/// [`TableIdentity`] is ever constructed for this graph-global dataset. +pub(crate) fn stream_token_row_id(identity: TableIdentity, logical_id: &str) -> Result { + identity.validate()?; + Ok(format!( + "stream-token-v1:{:016x}:{:016x}:{logical_id}", + identity.stable_table_id, identity.table_incarnation_id, + )) +} + +/// Encode complete current-token rows using the exact v1 physical schema. +pub(crate) fn stream_token_rows_to_batch(rows: &[StreamTokenAuthorityRow]) -> Result { + if rows.is_empty() { + return Err(OmniError::manifest_internal( + "stream-token upsert requires at least one authority row", + )); + } + + let mut ids = Vec::with_capacity(rows.len()); + let mut stable_table_ids = Vec::with_capacity(rows.len()); + let mut table_incarnation_ids = Vec::with_capacity(rows.len()); + let mut logical_ids = Vec::with_capacity(rows.len()); + let mut origin_enrollment_ids = Vec::with_capacity(rows.len()); + let mut stream_incarnation_ids = Vec::with_capacity(rows.len()); + let mut current_tokens = Vec::with_capacity(rows.len()); + let mut write_ids = Vec::with_capacity(rows.len()); + let mut predecessor_tokens = Vec::with_capacity(rows.len()); + let mut dispositions = Vec::with_capacity(rows.len()); + let mut contributors = Vec::with_capacity(rows.len()); + let mut payload_digests = Vec::with_capacity(rows.len()); + let mut origin_kinds = Vec::with_capacity(rows.len()); + let mut origin_ids = Vec::with_capacity(rows.len()); + let mut origin_ordinals = Vec::with_capacity(rows.len()); + let mut fold_base_tokens = Vec::with_capacity(rows.len()); + let mut chain_depths = Vec::with_capacity(rows.len()); + let mut terminal_actors = Vec::with_capacity(rows.len()); + let mut terminal_operation_ids = Vec::with_capacity(rows.len()); + let mut seen = std::collections::HashSet::with_capacity(rows.len()); + + for row in rows { + row.validate().map_err(stream_token_protocol_error)?; + let id = stream_token_row_id(row.identity, &row.logical_id)?; + if !seen.insert(id.clone()) { + return Err(OmniError::manifest(format!( + "stream-token upsert contains duplicate logical key ({}, '{}')", + row.identity, row.logical_id + ))); + } + let (origin_kind, origin_id, origin_ordinal) = match &row.origin { + StreamRowOrigin::Admission { + admission_attempt_id, + caller_ordinal, + } => ("ADMISSION", admission_attempt_id.clone(), *caller_ordinal), + StreamRowOrigin::Correction { + correction_id, + plan_ordinal, + } => ("CORRECTION", correction_id.clone(), *plan_ordinal), + }; + let (terminal_actor, terminal_operation_id) = row + .terminal_correction + .as_ref() + .map(|correction| { + ( + Some(correction.actor.as_str().to_string()), + Some(correction.correction_id.clone()), + ) + }) + .unwrap_or((None, None)); + + ids.push(id); + stable_table_ids.push(row.identity.stable_table_id); + table_incarnation_ids.push(row.identity.table_incarnation_id); + logical_ids.push(row.logical_id.clone()); + origin_enrollment_ids.push(row.origin_enrollment_id.clone()); + stream_incarnation_ids.push(row.stream_incarnation_id.clone()); + current_tokens.push(row.current_token.to_string()); + write_ids.push(row.write_id.clone()); + predecessor_tokens.push(row.predecessor_token.map(|token| token.to_string())); + dispositions.push(match row.disposition { + StreamTokenDisposition::Present => "PRESENT", + StreamTokenDisposition::Withdrawn => "WITHDRAWN", + }); + contributors.push(row.contributor_id.as_str().to_string()); + payload_digests.push(row.payload_digest.to_string()); + origin_kinds.push(origin_kind); + origin_ids.push(origin_id); + origin_ordinals.push(origin_ordinal); + fold_base_tokens.push(row.fold_base_token.map(|token| token.to_string())); + chain_depths.push(row.chain_depth); + terminal_actors.push(terminal_actor); + terminal_operation_ids.push(terminal_operation_id); + } + + RecordBatch::try_new( + stream_token_schema(), + vec![ + Arc::new(StringArray::from(ids)), + Arc::new(UInt64Array::from(stable_table_ids)), + Arc::new(UInt64Array::from(table_incarnation_ids)), + Arc::new(StringArray::from(logical_ids)), + Arc::new(StringArray::from(origin_enrollment_ids)), + Arc::new(StringArray::from(stream_incarnation_ids)), + Arc::new(StringArray::from(current_tokens)), + Arc::new(StringArray::from(write_ids)), + Arc::new(StringArray::from(predecessor_tokens)), + Arc::new(StringArray::from(dispositions)), + Arc::new(StringArray::from(contributors)), + Arc::new(StringArray::from(payload_digests)), + Arc::new(StringArray::from(origin_kinds)), + Arc::new(StringArray::from(origin_ids)), + Arc::new(UInt64Array::from(origin_ordinals)), + Arc::new(StringArray::from(fold_base_tokens)), + Arc::new(UInt32Array::from(chain_depths)), + Arc::new(StringArray::from(terminal_actors)), + Arc::new(StringArray::from(terminal_operation_ids)), + ], + ) + .map_err(|error| { + OmniError::manifest_internal(format!( + "failed to build stream-token authority batch: {error}" + )) + }) +} + +/// Enforce the config-v3 bounds for the exact winner projection which can +/// enter recovery-v12. This runs before acknowledgement for every projected +/// warm generation and again at the staging/recovery boundary. +pub(crate) fn validate_stream_token_plan_bounds( + rows: &[StreamTokenAuthorityRow], +) -> Result<()> { + validate_stream_token_plan_bounds_with_limits( + rows, + crate::table_store::mem_wal::B2_MAX_TOKEN_PROJECTION_ARROW_BYTES, + crate::table_store::mem_wal::B2_MAX_TOKEN_RECOVERY_JSON_BYTES, + ) +} + +pub(crate) fn add_stream_lookup_retained_bytes( + resource: &'static str, + current: u64, + additional: u64, + limit: u64, +) -> Result { + let total = current.checked_add(additional).ok_or_else(|| { + OmniError::manifest_internal(format!("{resource} retained-byte accounting overflow")) + })?; + if total > limit { + return Err(OmniError::resource_limit(resource, limit, total)); + } + Ok(total) +} + +fn validate_stream_token_plan_bounds_with_limits( + rows: &[StreamTokenAuthorityRow], + arrow_limit: u64, + json_limit: u64, +) -> Result<()> { + let batch = stream_token_rows_to_batch(rows)?; + let arrow_bytes = u64::try_from(batch.get_array_memory_size()).map_err(|_| { + OmniError::manifest_internal("stream-token projection Arrow size exceeds u64") + })?; + if arrow_bytes > arrow_limit { + return Err(OmniError::resource_limit( + "stream_token_projection_arrow_bytes", + arrow_limit, + arrow_bytes, + )); + } + + let json_limit_usize = usize::try_from(json_limit) + .map_err(|_| OmniError::manifest_internal("stream-token JSON cap exceeds usize"))?; + let mut writer = BoundedCountWriter::new(json_limit_usize); + let result = serde_json::to_writer(&mut writer, rows); + if writer.exceeded { + return Err(OmniError::resource_limit( + "stream_token_recovery_json_bytes", + json_limit, + u64::try_from(writer.attempted).unwrap_or(u64::MAX), + )); + } + result.map_err(|error| { + OmniError::manifest_internal(format!( + "failed to size stream-token recovery projection: {error}" + )) + })?; + Ok(()) +} + +struct BoundedCountWriter { + written: usize, + limit: usize, + attempted: usize, + exceeded: bool, +} + +impl BoundedCountWriter { + fn new(limit: usize) -> Self { + Self { + written: 0, + limit, + attempted: 0, + exceeded: false, + } + } +} + +impl std::io::Write for BoundedCountWriter { + fn write(&mut self, bytes: &[u8]) -> std::io::Result { + self.attempted = self.written.saturating_add(bytes.len()); + if self.attempted > self.limit { + self.exceeded = true; + return Err(std::io::Error::other( + "stream-token recovery projection exceeds its configured byte cap", + )); + } + self.written = self.attempted; + Ok(bytes.len()) + } + + fn flush(&mut self) -> std::io::Result<()> { + Ok(()) + } +} + +/// Decode and validate every current-token row from one exact-schema batch. +pub(crate) fn stream_token_rows_from_batch( + batch: &RecordBatch, +) -> Result> { + if batch.schema().as_ref() != stream_token_schema().as_ref() { + return Err(OmniError::manifest_internal( + "stream-token scan returned a non-v1 physical schema", + )); + } + let ids = required_string_array(batch, "id")?; + let stable_table_ids = required_u64_array(batch, "stable_table_id")?; + let table_incarnation_ids = required_u64_array(batch, "table_incarnation_id")?; + let logical_ids = required_string_array(batch, "logical_id")?; + let origin_enrollment_ids = required_string_array(batch, "origin_enrollment_id")?; + let stream_incarnation_ids = required_string_array(batch, "stream_incarnation_id")?; + let current_tokens = required_string_array(batch, "current_token")?; + let write_ids = required_string_array(batch, "write_id")?; + let predecessor_tokens = required_string_array(batch, "predecessor_token")?; + let dispositions = required_string_array(batch, "disposition")?; + let contributors = required_string_array(batch, "contributor_id")?; + let payload_digests = required_string_array(batch, "payload_digest")?; + let origin_kinds = required_string_array(batch, "origin_kind")?; + let origin_ids = required_string_array(batch, "origin_id")?; + let origin_ordinals = required_u64_array(batch, "origin_ordinal")?; + let fold_base_tokens = required_string_array(batch, "fold_base_token")?; + let chain_depths = required_u32_array(batch, "chain_depth")?; + let terminal_actors = required_string_array(batch, "terminal_correction_actor")?; + let terminal_operation_ids = required_string_array(batch, "terminal_correction_operation_id")?; + + let mut rows = Vec::with_capacity(batch.num_rows()); + let mut seen = std::collections::HashSet::with_capacity(batch.num_rows()); + for index in 0..batch.num_rows() { + require_non_null(ids, index, "id")?; + require_non_null(stable_table_ids, index, "stable_table_id")?; + require_non_null(table_incarnation_ids, index, "table_incarnation_id")?; + require_non_null(logical_ids, index, "logical_id")?; + require_non_null(origin_enrollment_ids, index, "origin_enrollment_id")?; + require_non_null(stream_incarnation_ids, index, "stream_incarnation_id")?; + require_non_null(current_tokens, index, "current_token")?; + require_non_null(write_ids, index, "write_id")?; + require_non_null(dispositions, index, "disposition")?; + require_non_null(contributors, index, "contributor_id")?; + require_non_null(payload_digests, index, "payload_digest")?; + require_non_null(origin_kinds, index, "origin_kind")?; + require_non_null(origin_ids, index, "origin_id")?; + require_non_null(origin_ordinals, index, "origin_ordinal")?; + require_non_null(chain_depths, index, "chain_depth")?; + + let identity = TableIdentity::new( + stable_table_ids.value(index), + table_incarnation_ids.value(index), + )?; + let logical_id = logical_ids.value(index).to_string(); + let expected_id = stream_token_row_id(identity, &logical_id)?; + if ids.value(index) != expected_id { + return Err(OmniError::manifest_internal(format!( + "stream-token row id '{}' does not match canonical key '{}'", + ids.value(index), + expected_id + ))); + } + if !seen.insert(expected_id) { + return Err(OmniError::manifest_internal(format!( + "stream-token batch contains duplicate logical key ({identity}, '{logical_id}')" + ))); + } + + let origin = match origin_kinds.value(index) { + "ADMISSION" => StreamRowOrigin::Admission { + admission_attempt_id: origin_ids.value(index).to_string(), + caller_ordinal: origin_ordinals.value(index), + }, + "CORRECTION" => StreamRowOrigin::Correction { + correction_id: origin_ids.value(index).to_string(), + plan_ordinal: origin_ordinals.value(index), + }, + other => { + return Err(OmniError::manifest_internal(format!( + "stream-token row has unsupported origin_kind '{other}'" + ))); + } + }; + let disposition = match dispositions.value(index) { + "PRESENT" => StreamTokenDisposition::Present, + "WITHDRAWN" => StreamTokenDisposition::Withdrawn, + other => { + return Err(OmniError::manifest_internal(format!( + "stream-token row has unsupported disposition '{other}'" + ))); + } + }; + let terminal_correction = match ( + terminal_actors.is_null(index), + terminal_operation_ids.is_null(index), + ) { + (true, true) => None, + (false, false) => Some(StreamTerminalCorrection { + actor: TrustedContributorId::new(terminal_actors.value(index).to_string()) + .map_err(stream_token_protocol_error)?, + correction_id: terminal_operation_ids.value(index).to_string(), + }), + _ => { + return Err(OmniError::manifest_internal( + "stream-token terminal correction actor and operation must be both null or both present", + )); + } + }; + let row = StreamTokenAuthorityRow { + identity, + logical_id, + origin_enrollment_id: origin_enrollment_ids.value(index).to_string(), + stream_incarnation_id: stream_incarnation_ids.value(index).to_string(), + current_token: StreamToken::from_str(current_tokens.value(index)) + .map_err(stream_token_protocol_error)?, + write_id: write_ids.value(index).to_string(), + predecessor_token: optional_stream_token(predecessor_tokens, index)?, + disposition, + contributor_id: TrustedContributorId::new(contributors.value(index).to_string()) + .map_err(stream_token_protocol_error)?, + payload_digest: PayloadDigest::from_str(payload_digests.value(index)) + .map_err(stream_token_protocol_error)?, + origin, + fold_base_token: optional_stream_token(fold_base_tokens, index)?, + chain_depth: chain_depths.value(index), + terminal_correction, + }; + row.validate().map_err(stream_token_protocol_error)?; + rows.push(row); + } + Ok(rows) +} + +/// Look up one logical graph key from an already exact-pinned token dataset. +pub(crate) async fn lookup_stream_token_row( + dataset: &Dataset, + authority: &StreamTokenAuthorityEntry, + identity: TableIdentity, + logical_id: &str, +) -> Result> { + validate_exact_dataset(dataset, authority).await?; + let id = stream_token_row_id(identity, logical_id)?; + let mut scanner = dataset.scan(); + scanner.filter_expr(col("id").eq(lit(id))); + scanner.batch_size(2); + scanner.batch_size_bytes( + crate::table_store::mem_wal::B2_MAX_TOKEN_PROJECTION_ARROW_BYTES, + ); + scanner + .limit(Some(2), None) + .map_err(|error| OmniError::Lance(error.to_string()))?; + let mut stream = scanner + .try_into_stream() + .await + .map_err(|error| OmniError::Lance(error.to_string()))?; + let mut selected = None; + while let Some(batch) = stream + .try_next() + .await + .map_err(|error| OmniError::Lance(error.to_string()))? + { + let batch_bytes = u64::try_from(batch.get_array_memory_size()).map_err(|_| { + OmniError::manifest_internal("stream-token lookup batch Arrow size exceeds u64") + })?; + if batch_bytes > crate::table_store::mem_wal::B2_MAX_TOKEN_PROJECTION_ARROW_BYTES { + return Err(OmniError::resource_limit( + "stream_token_lookup_batch_arrow_bytes", + crate::table_store::mem_wal::B2_MAX_TOKEN_PROJECTION_ARROW_BYTES, + batch_bytes, + )); + } + for row in stream_token_rows_from_batch(&batch)? { + let retained_bytes = row + .lookup_retained_bytes() + .map_err(|error| OmniError::manifest_internal(error.to_string()))?; + if retained_bytes + > crate::table_store::mem_wal::B2_MAX_TOKEN_PROJECTION_ARROW_BYTES + { + return Err(OmniError::resource_limit( + "stream_token_lookup_retained_bytes", + crate::table_store::mem_wal::B2_MAX_TOKEN_PROJECTION_ARROW_BYTES, + retained_bytes, + )); + } + if selected.replace(row).is_some() { + return Err(OmniError::manifest_internal(format!( + "manifest-selected stream-token dataset contains duplicate current rows for ({identity}, '{logical_id}')" + ))); + } + } + } + if selected + .as_ref() + .is_some_and(|row| row.identity != identity || row.logical_id != logical_id) + { + return Err(OmniError::manifest_internal( + "stream-token exact-id lookup returned a row for a different logical key", + )); + } + Ok(selected) +} + +/// Read only the manifest-selected current-token rows named by one bounded +/// generation. The structured exact-id predicate keeps materialized output +/// bounded by the generation instead of by all retained token authority. +pub(crate) async fn stream_token_rows_for_keys( + dataset: &Dataset, + authority: &StreamTokenAuthorityEntry, + identity: TableIdentity, + logical_ids: &std::collections::BTreeSet, +) -> Result> { + validate_exact_dataset(dataset, authority).await?; + if logical_ids.is_empty() + || logical_ids.len() + > crate::table_store::mem_wal::B1_MAX_GENERATION_ROWS as usize + { + return Err(OmniError::manifest_internal(format!( + "stream-token fold lookup requires 1..={} exact keys, got {}", + crate::table_store::mem_wal::B1_MAX_GENERATION_ROWS, + logical_ids.len() + ))); + } + let exact_ids = logical_ids + .iter() + .map(|logical_id| stream_token_row_id(identity, logical_id)) + .collect::>>()?; + let mut scanner = dataset.scan(); + scanner.filter_expr(col("id").in_list(exact_ids.into_iter().map(lit).collect(), false)); + scanner.batch_size(logical_ids.len().saturating_add(1)); + scanner.batch_size_bytes( + crate::table_store::mem_wal::B2_MAX_TOKEN_PROJECTION_ARROW_BYTES, + ); + scanner + .limit( + Some(i64::try_from(logical_ids.len().saturating_add(1)).map_err(|_| { + OmniError::manifest_internal("stream-token lookup row limit exceeds i64") + })?), + None, + ) + .map_err(|error| OmniError::Lance(error.to_string()))?; + let mut stream = scanner + .try_into_stream() + .await + .map_err(|error| OmniError::Lance(error.to_string()))?; + let mut selected = BTreeMap::new(); + let mut observed_rows = 0_usize; + let mut retained_bytes = 0_u64; + while let Some(batch) = stream + .try_next() + .await + .map_err(|error| OmniError::Lance(error.to_string()))? + { + observed_rows = observed_rows + .checked_add(batch.num_rows()) + .ok_or_else(|| OmniError::manifest_internal("stream-token lookup row overflow"))?; + if observed_rows > logical_ids.len() { + return Err(OmniError::manifest_internal(format!( + "manifest-selected stream-token dataset returned more than one row per requested key for table {identity}" + ))); + } + let batch_bytes = u64::try_from(batch.get_array_memory_size()).map_err(|_| { + OmniError::manifest_internal("stream-token lookup batch Arrow size exceeds u64") + })?; + if batch_bytes > crate::table_store::mem_wal::B2_MAX_TOKEN_PROJECTION_ARROW_BYTES { + return Err(OmniError::resource_limit( + "stream_token_lookup_batch_arrow_bytes", + crate::table_store::mem_wal::B2_MAX_TOKEN_PROJECTION_ARROW_BYTES, + batch_bytes, + )); + } + for row in stream_token_rows_from_batch(&batch)? { + if row.identity != identity || !logical_ids.contains(&row.logical_id) { + return Err(OmniError::manifest_internal( + "stream-token exact-key scan returned a row outside its requested key set", + )); + } + retained_bytes = add_stream_lookup_retained_bytes( + "stream_token_lookup_retained_bytes", + retained_bytes, + row.lookup_retained_bytes() + .map_err(|error| OmniError::manifest_internal(error.to_string()))?, + crate::table_store::mem_wal::B2_MAX_TOKEN_PROJECTION_ARROW_BYTES, + )?; + if selected.insert(row.logical_id.clone(), row).is_some() { + return Err(OmniError::manifest_internal(format!( + "manifest-selected stream-token dataset contains duplicate current rows for table {identity}" + ))); + } + } + } + Ok(selected) +} + +/// Stage one exact-`id` current-token upsert without advancing Lance HEAD. +/// +/// `dataset` must be the exact handle selected by `authority`; the helper +/// validates that witness before producing any staged files. The returned +/// [`crate::table_store::StagedWrite`] must enter recovery-v12 before its one +/// strict `commit_staged_exact` invocation. +pub(crate) async fn stage_stream_token_upsert( + dataset: Dataset, + authority: &StreamTokenAuthorityEntry, + rows: &[StreamTokenAuthorityRow], +) -> Result { + validate_exact_dataset(&dataset, authority).await?; + validate_stream_token_plan_bounds(rows)?; + let batch = stream_token_rows_to_batch(rows)?; + let row_count = u64::try_from(batch.num_rows()) + .map_err(|_| OmniError::manifest_internal("stream-token upsert row count exceeds u64"))?; + let schema = batch.schema(); + let reader = RecordBatchIterator::new(vec![Ok(batch)], schema); + let stream = lance_datafusion::utils::reader_to_stream(Box::new(reader)); + let mut builder = + MergeInsertBuilder::try_new(Arc::new(dataset.clone()), vec!["id".to_string()]) + .map_err(|error| OmniError::Lance(error.to_string()))?; + builder + .when_matched(WhenMatched::UpdateAll) + .when_not_matched(WhenNotMatched::InsertAll) + .use_index(false) + .conflict_retries(0) + .source_dedupe_behavior(SourceDedupeBehavior::FirstSeen); + let uncommitted = builder + .try_build() + .map_err(|error| OmniError::Lance(error.to_string()))? + .execute_uncommitted(stream) + .await + .map_err(|error| OmniError::Lance(error.to_string()))?; + if uncommitted.transaction.read_version != authority.current_head_witness.table_version { + return Err(OmniError::manifest_internal(format!( + "stream-token staged transaction read version {} does not match manifest-selected version {}", + uncommitted.transaction.read_version, authority.current_head_witness.table_version + ))); + } + crate::table_store::staged_exact_id_upsert_result( + &dataset, + uncommitted, + row_count, + "stage_stream_token_upsert", + ) +} + +fn required_string_array<'a>(batch: &'a RecordBatch, name: &str) -> Result<&'a StringArray> { + batch + .column_by_name(name) + .and_then(|array| array.as_any().downcast_ref::()) + .ok_or_else(|| { + OmniError::manifest_internal(format!( + "stream-token batch column '{name}' is missing or not Utf8" + )) + }) +} + +fn required_u64_array<'a>(batch: &'a RecordBatch, name: &str) -> Result<&'a UInt64Array> { + batch + .column_by_name(name) + .and_then(|array| array.as_any().downcast_ref::()) + .ok_or_else(|| { + OmniError::manifest_internal(format!( + "stream-token batch column '{name}' is missing or not UInt64" + )) + }) +} + +fn required_u32_array<'a>(batch: &'a RecordBatch, name: &str) -> Result<&'a UInt32Array> { + batch + .column_by_name(name) + .and_then(|array| array.as_any().downcast_ref::()) + .ok_or_else(|| { + OmniError::manifest_internal(format!( + "stream-token batch column '{name}' is missing or not UInt32" + )) + }) +} + +fn require_non_null(array: &dyn Array, index: usize, name: &str) -> Result<()> { + if array.is_null(index) { + return Err(OmniError::manifest_internal(format!( + "stream-token required column '{name}' is null at row {index}" + ))); + } + Ok(()) +} + +fn optional_stream_token(array: &StringArray, index: usize) -> Result> { + (!array.is_null(index)) + .then(|| StreamToken::from_str(array.value(index)).map_err(stream_token_protocol_error)) + .transpose() +} + +fn stream_token_protocol_error(error: impl std::fmt::Display) -> OmniError { + OmniError::manifest_internal(format!("invalid stream-token authority row: {error}")) +} + +pub(super) async fn initialize_stream_token_authority( + root_uri: &str, + control_session: &Arc, +) -> Result { + let schema = stream_token_schema(); + let batch = RecordBatch::new_empty(schema.clone()); + let reader = RecordBatchIterator::new(vec![Ok(batch)], schema); + let params = WriteParams { + mode: WriteMode::Create, + enable_stable_row_ids: true, + data_storage_version: Some(LanceFileVersion::V2_2), + auto_cleanup: None, + skip_auto_cleanup: true, + session: Some(Arc::clone(control_session)), + ..Default::default() + }; + let dataset = Dataset::write(reader, &stream_token_uri(root_uri), Some(params)) + .await + .map_err(|error| OmniError::Lance(error.to_string()))?; + stream_token_authority_entry_for_dataset(&dataset).await +} + +/// Build the only valid manifest pointer for an already-achieved exact token +/// dataset version. Callers still need recovery ownership before any effect; +/// this helper only captures and validates the physical witness. +pub(crate) async fn stream_token_authority_entry_for_dataset( + dataset: &Dataset, +) -> Result { + let actual_schema: Schema = dataset.schema().into(); + if &actual_schema != stream_token_schema().as_ref() { + return Err(OmniError::manifest_internal( + "cannot publish a stream-token dataset with a non-v1 schema", + )); + } + let entry = StreamTokenAuthorityEntry { + location: STREAM_TOKEN_DATASET_PATH.to_string(), + schema_version: STREAM_TOKEN_AUTHORITY_SCHEMA_VERSION, + schema_hash: stream_token_schema_hash(), + current_head_witness: capture_exact_head_witness(dataset).await?, + }; + entry.validate()?; + Ok(entry) +} + +/// Open only the exact token-table version selected by `__manifest`. +/// +/// This helper intentionally has no latest-HEAD fallback. A moved raw HEAD is +/// invisible until its exact witness is published, and any mismatch at the +/// selected version is corruption rather than an adoption opportunity. +pub(crate) async fn open_stream_token_authority_at( + root_uri: &str, + authority: &StreamTokenAuthorityEntry, + control_session: &Arc, +) -> Result { + authority.validate()?; + let dataset = crate::instrumentation::open_dataset( + &stream_token_uri(root_uri), + crate::instrumentation::VersionResolution::At(authority.current_head_witness.table_version), + Some(control_session), + crate::instrumentation::table_wrapper(), + ) + .await?; + validate_exact_dataset(&dataset, authority).await?; + Ok(dataset) +} + +/// Open raw token HEAD only as a final uncovered-drift check. The caller must +/// already hold the graph-global stream-token gate and compare this complete +/// witness with manifest authority before arming any other participant. +pub(crate) async fn open_stream_token_authority_head( + root_uri: &str, + expected: &StreamTokenAuthorityEntry, + control_session: &Arc, +) -> Result { + expected.validate()?; + let dataset = crate::instrumentation::open_dataset( + &stream_token_uri(root_uri), + crate::instrumentation::VersionResolution::Latest, + Some(control_session), + crate::instrumentation::table_wrapper(), + ) + .await?; + let observed = stream_token_authority_entry_for_dataset(&dataset).await?; + if &observed != expected { + return Err(OmniError::manifest_conflict(format!( + "stream-token raw HEAD {:?} differs from manifest-selected authority {:?}; explicit recovery is required", + observed.current_head_witness, expected.current_head_witness + ))); + } + Ok(dataset) +} + +async fn validate_exact_dataset( + dataset: &Dataset, + authority: &StreamTokenAuthorityEntry, +) -> Result<()> { + let actual_schema: Schema = dataset.schema().into(); + if &actual_schema != stream_token_schema().as_ref() { + return Err(OmniError::manifest_internal( + "manifest-selected stream-token dataset has a schema different from its v1 authority", + )); + } + let actual = capture_exact_head_witness(dataset).await?; + if actual != authority.current_head_witness { + return Err(OmniError::manifest_read_set_changed( + stream_token_authority_object_id(), + Some(authority.to_metadata_json()?), + Some( + StreamTokenAuthorityEntry { + location: authority.location.clone(), + schema_version: authority.schema_version, + schema_hash: authority.schema_hash.clone(), + current_head_witness: actual, + } + .to_metadata_json()?, + ), + )); + } + Ok(()) +} + +async fn capture_exact_head_witness(dataset: &Dataset) -> Result { + let branch_before = dataset + .branch_identifier() + .await + .map_err(|error| OmniError::Lance(error.to_string()))?; + let table_version = dataset.version().version; + let transaction = dataset + .read_transaction() + .await + .map_err(|error| OmniError::Lance(error.to_string()))? + .ok_or_else(|| { + OmniError::manifest_internal( + "manifest-selected stream-token dataset version has no transaction", + ) + })?; + let branch_after = dataset + .branch_identifier() + .await + .map_err(|error| OmniError::Lance(error.to_string()))?; + if branch_before != branch_after || dataset.version().version != table_version { + return Err(OmniError::manifest_internal( + "stream-token exact-version witness is internally incoherent", + )); + } + let witness = CurrentHeadWitness { + branch_identifier: branch_before, + table_version, + transaction_uuid: transaction.uuid, + // Object-store ETags are useful within one provider commit attempt, + // but they are not durable graph identity. In particular, LocalFileSystem + // derives them partly from the inode, so copying an otherwise exact graph + // changes the value. Version + Lance transaction UUID is the stable exact + // token-table witness; strict commit/recovery still fences every effect. + manifest_e_tag: None, + }; + validate_head_witness(&witness)?; + Ok(witness) +} + +fn validate_head_witness(witness: &CurrentHeadWitness) -> Result<()> { + if witness.branch_identifier != BranchIdentifier::main() { + return Err(OmniError::manifest_internal( + "stream-token authority must select the main Lance branch", + )); + } + if witness.table_version == 0 { + return Err(OmniError::manifest_internal( + "stream-token authority table_version must be non-zero", + )); + } + let transaction_uuid = ShardId::parse_str(&witness.transaction_uuid).map_err(|error| { + OmniError::manifest_internal(format!( + "stream-token authority transaction_uuid is not a UUID: {error}" + )) + })?; + if transaction_uuid.is_nil() || transaction_uuid.to_string() != witness.transaction_uuid { + return Err(OmniError::manifest_internal( + "stream-token authority transaction_uuid must be non-nil canonical lowercase UUID text", + )); + } + if witness.manifest_e_tag.is_some() { + return Err(OmniError::manifest_internal( + "stream-token authority manifest_e_tag must be absent because provider-local ETags are not durable graph identity", + )); + } + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::db::manifest::stream_token::{ + PayloadDigest, StreamRowOrigin, StreamTokenInput, TrustedContributorId, + }; + + fn authority_row_for(logical_id: &str, contributor: &str) -> StreamTokenAuthorityRow { + let identity = TableIdentity::new(7, 9).unwrap(); + let contributor_id = TrustedContributorId::new(contributor).unwrap(); + let payload_digest = PayloadDigest::from_bytes([0x5a; 32]); + let stream_incarnation_id = "11111111-1111-4111-8111-111111111111"; + let write_id = "22222222-2222-4222-8222-222222222222"; + let current_token = StreamToken::derive(&StreamTokenInput { + identity, + logical_id, + stream_incarnation_id, + predecessor_token: None, + write_id, + contributor_id: &contributor_id, + payload_digest, + }) + .unwrap(); + StreamTokenAuthorityRow { + identity, + logical_id: logical_id.to_string(), + origin_enrollment_id: "33333333-3333-4333-8333-333333333333".to_string(), + stream_incarnation_id: stream_incarnation_id.to_string(), + current_token, + write_id: write_id.to_string(), + predecessor_token: None, + disposition: StreamTokenDisposition::Present, + contributor_id, + payload_digest, + origin: StreamRowOrigin::Admission { + admission_attempt_id: "44444444-4444-4444-8444-444444444444".to_string(), + caller_ordinal: 17, + }, + fold_base_token: None, + chain_depth: 1, + terminal_correction: None, + } + } + + fn authority_row() -> StreamTokenAuthorityRow { + authority_row_for("person:17", "actor:alice") + } + + #[test] + fn token_authority_row_batch_round_trips_exactly() { + let row = authority_row(); + let batch = stream_token_rows_to_batch(std::slice::from_ref(&row)).unwrap(); + assert_eq!(stream_token_rows_from_batch(&batch).unwrap(), vec![row]); + } + + #[test] + fn token_plan_bounds_fail_loudly_for_arrow_and_recovery_json() { + let row = authority_row(); + let arrow_error = validate_stream_token_plan_bounds_with_limits( + std::slice::from_ref(&row), + 1, + u64::MAX, + ) + .unwrap_err(); + assert!( + matches!( + arrow_error, + OmniError::ResourceLimitExceeded { + ref resource, + limit: 1, + actual, + } if resource == "stream_token_projection_arrow_bytes" && actual > 1 + ), + "{arrow_error:?}" + ); + + let json_error = validate_stream_token_plan_bounds_with_limits( + std::slice::from_ref(&row), + u64::MAX, + 1, + ) + .unwrap_err(); + assert!( + matches!( + json_error, + OmniError::ResourceLimitExceeded { + ref resource, + limit: 1, + actual, + } if resource == "stream_token_recovery_json_bytes" && actual > 1 + ), + "{json_error:?}" + ); + } + + #[test] + fn exact_lookup_retained_bytes_are_cumulative_and_fail_loudly() { + let first = authority_row_for("person:17", "actor:alice"); + let second = authority_row_for("person:18", "actor:bob"); + first.validate().unwrap(); + second.validate().unwrap(); + let first_bytes = first.lookup_retained_bytes().unwrap(); + let second_bytes = second.lookup_retained_bytes().unwrap(); + let limit = first_bytes + .checked_add(second_bytes) + .unwrap() + .checked_sub(1) + .unwrap(); + let retained = add_stream_lookup_retained_bytes( + "stream_token_lookup_retained_bytes", + 0, + first_bytes, + limit, + ) + .unwrap(); + let error = add_stream_lookup_retained_bytes( + "stream_token_lookup_retained_bytes", + retained, + second_bytes, + limit, + ) + .unwrap_err(); + assert!(matches!( + error, + OmniError::ResourceLimitExceeded { + ref resource, + limit: actual_limit, + actual, + } if resource == "stream_token_lookup_retained_bytes" + && actual_limit == limit + && actual == first_bytes + second_bytes + )); + } + + #[tokio::test] + async fn staged_token_upsert_is_invisible_until_commit_and_lookup_is_manifest_pinned() { + let dir = tempfile::tempdir().unwrap(); + let root = dir.path().to_str().unwrap(); + let session = crate::lance_access::control_session(); + let authority = initialize_stream_token_authority(root, &session) + .await + .unwrap(); + assert!( + authority.current_head_witness.manifest_e_tag.is_none(), + "manifest-selected token authority must not persist provider-local ETags" + ); + let dataset = open_stream_token_authority_at(root, &authority, &session) + .await + .unwrap(); + let row = authority_row(); + let staged = + stage_stream_token_upsert(dataset.clone(), &authority, std::slice::from_ref(&row)) + .await + .unwrap(); + assert_eq!(dataset.count_rows(None).await.unwrap(), 0); + + let store = crate::table_store::TableStore::new(root, Arc::clone(&session)); + let (achieved, committed) = store + .commit_staged_exact(Arc::new(dataset), staged) + .await + .unwrap(); + assert_eq!( + committed.read_version, + authority.current_head_witness.table_version + ); + let next = stream_token_authority_entry_for_dataset(&achieved) + .await + .unwrap(); + assert!(next.current_head_witness.manifest_e_tag.is_none()); + assert_eq!( + lookup_stream_token_row(&achieved, &next, row.identity, &row.logical_id) + .await + .unwrap(), + Some(row) + ); + } +} diff --git a/crates/omnigraph/src/db/mod.rs b/crates/omnigraph/src/db/mod.rs index a9f673c7..b373ff26 100644 --- a/crates/omnigraph/src/db/mod.rs +++ b/crates/omnigraph/src/db/mod.rs @@ -21,6 +21,15 @@ use crate::error::{OmniError, Result}; pub(crate) const SCHEMA_APPLY_LOCK_BRANCH: &str = "__schema_apply_lock__"; +/// Physical-only RFC-026 trusted row metadata. The compiler reserves this +/// exact name; logical reads, reflection, export, and user index selection must +/// never expose it. A different wire shape requires a differently versioned +/// name rather than permissive reinterpretation. +/// Grammar-impossible physical protocol field. The trailing `$` prevents any +/// historical or future `.pg` property from colliding with this v9-only row +/// envelope while remaining a valid Arrow/Lance field name. +pub(crate) const STREAM_METADATA_COLUMN: &str = "__omnigraph_stream_v1$"; + /// Mutation kind, threaded through the early table-version checks so the /// engine can apply an op-kind-aware staging policy. This check is not the /// RFC-022 publish authority: enrolled mutation/load attempts additionally diff --git a/crates/omnigraph/src/db/omnigraph.rs b/crates/omnigraph/src/db/omnigraph.rs index 72d8f1d1..77f95e49 100644 --- a/crates/omnigraph/src/db/omnigraph.rs +++ b/crates/omnigraph/src/db/omnigraph.rs @@ -161,7 +161,11 @@ pub(crate) struct WriteTxn { /// combine an old source with a new identity-bearing catalog (or vice versa). #[derive(Debug)] struct HandleSchemaView { + /// Physical engine catalog, including storage-only fields and metadata. catalog: Arc, + /// Public reflection view derived atomically from `catalog`; it differs + /// only by omitting protocol-private fields. + public_catalog: Arc, source: Arc, schema_ir_hash: String, schema_identity_domain: String, @@ -326,6 +330,13 @@ fn private_b1_worker_limits() -> crate::table_store::mem_wal::B1WorkerLimits { max_resident_writers_root: 1, max_resident_writers_per_table: 1, max_reserved_arrow_bytes: 32 * 1024 * 1024, + // A B2 caller reserves the full worst-case preprocessing envelope + // before blob materialization or canonical encoding. Keeping one such + // envelopes root-wide is the minimum qualified concurrency contract: + // one stale provisional caller may wait while another becomes the + // winner whose authority it must later observe. + max_b2_preprocessing_bytes: + crate::table_store::mem_wal::B2_MAX_PREPROCESSING_BYTES_ROOT, // Every queued input is charged against the same aggregate Arrow // reservation synchronously, before detachment or cold claim. This // count therefore bounds scheduling/control overhead; it cannot admit @@ -418,6 +429,7 @@ impl Omnigraph { let schema_identity_domain = schema_ir.schema_identity_domain.as_str().to_string(); let mut catalog = build_catalog_from_ir(&schema_ir)?; fixup_physical_schemas(&mut catalog)?; + let public_catalog = public_catalog_view(&catalog)?; // Establish an atomic ownership claim on `_schema.pg` before // writing the remaining init artifacts. A check-then-write preflight @@ -497,6 +509,7 @@ impl Omnigraph { }), schema_view: Arc::new(ArcSwap::from_pointee(HandleSchemaView { catalog: Arc::new(catalog), + public_catalog: Arc::new(public_catalog), source: Arc::new(schema_source.to_string()), schema_ir_hash: accepted_schema_ir_hash, schema_identity_domain, @@ -668,6 +681,7 @@ impl Omnigraph { let schema_identity_domain = accepted_ir.schema_identity_domain.as_str().to_string(); let mut catalog = build_catalog_from_ir(&accepted_ir)?; fixup_physical_schemas(&mut catalog)?; + let public_catalog = public_catalog_view(&catalog)?; let session = lance_access.data_session(); let db = Self { @@ -687,6 +701,7 @@ impl Omnigraph { }), schema_view: Arc::new(ArcSwap::from_pointee(HandleSchemaView { catalog: Arc::new(catalog), + public_catalog: Arc::new(public_catalog), source: Arc::new(schema_source), schema_ir_hash: accepted_state.schema_ir_hash, schema_identity_domain, @@ -712,7 +727,7 @@ impl Omnigraph { /// catalog pointer; callers can hold the returned `Arc` across awaits /// without blocking concurrent `apply_schema`. pub fn catalog(&self) -> Arc { - Arc::clone(&self.schema_view.load().catalog) + Arc::clone(&self.schema_view.load().public_catalog) } /// Returns an `Arc` snapshot of the schema source. @@ -744,8 +759,10 @@ impl Omnigraph { .to_string(), )); } + let public_catalog = public_catalog_view(&catalog)?; self.schema_view.store(Arc::new(HandleSchemaView { catalog: Arc::new(catalog), + public_catalog: Arc::new(public_catalog), source: Arc::new(schema_source), schema_ir_hash, schema_identity_domain: accepted_ir.schema_identity_domain.as_str().to_string(), @@ -1760,8 +1777,9 @@ impl Omnigraph { /// RFC-022 synchronous write/control recovery barrier. Run the live-safe /// healer, then reject any guarded unresolved intent on a relevant graph branch. A - /// SchemaApply intent is graph-global because it changes the accepted schema - /// identity used by every branch. + /// SchemaApply is graph-global because it changes accepted schema identity; + /// StreamFold is graph-global because recovery-v12 owns an unpublished + /// `_stream_tokens.lance` HEAD shared by every table and branch. /// /// `relevant_branches` must use the engine convention (`None` = main), but /// this helper defensively folds `Some("main")` as well so load's explicit @@ -1772,7 +1790,7 @@ impl Omnigraph { ) -> Result<()> { let outcome = self.heal_pending_recovery_sidecars_outcome().await?; let blocking = outcome.unresolved.iter().find(|intent| { - intent.writer_kind == crate::db::manifest::SidecarKind::SchemaApply + intent.writer_kind.is_graph_global_barrier() || relevant_branches .iter() .any(|branch| branch.filter(|name| *name != "main") == intent.branch.as_deref()) @@ -1804,15 +1822,16 @@ impl Omnigraph { /// sidecar's table effects are unreachable and the recovery sweep records an /// orphan-discard audit. Safety comes from branch_delete subsequently taking /// schema -> target branch -> every accepted-catalog table gate before the - /// ref mutation, which waits out any live in-process owner. SchemaApply - /// remains graph-global and bounded main StreamEnrollment can establish the - /// topology-closing lifecycle authority, so both still block deletion. + /// ref mutation, which waits out any live in-process owner. Graph-global + /// recovery and bounded main StreamEnrollment can establish authority that + /// deletion must not bypass, so both still block deletion. async fn heal_pending_recovery_sidecars_for_branch_delete(&self, branch: &str) -> Result<()> { let outcome = self.heal_pending_recovery_sidecars_outcome().await?; if let Some(intent) = outcome.unresolved.iter().find(|intent| { matches!( intent.writer_kind, crate::db::manifest::SidecarKind::SchemaApply + | crate::db::manifest::SidecarKind::StreamFold | crate::db::manifest::SidecarKind::StreamEnrollment ) }) { @@ -1845,7 +1864,7 @@ impl Omnigraph { crate::db::manifest::list_sidecars(&self.root_uri, self.storage.as_ref()).await?; let blocking = sidecars.iter().find(|sidecar| { let sidecar_branch = sidecar.branch.as_deref().filter(|branch| *branch != "main"); - sidecar.writer_kind == crate::db::manifest::SidecarKind::SchemaApply + sidecar.writer_kind.is_graph_global_barrier() || relevant_branches .iter() .any(|branch| branch.filter(|name| *name != "main") == sidecar_branch) @@ -2791,6 +2810,7 @@ impl Omnigraph { .acquire(&crate::db::manifest::schema_apply_serial_queue_key()) .await; let _branch_guards = self.write_queue().acquire_branches(&control_branches).await; + let _stream_token_guard = self.write_queue().acquire_stream_token().await; self.ensure_schema_apply_not_locked("branch_create").await?; let control_catalog = self.build_accepted_catalog_with_schema_gate_held().await?; let table_queue_keys = @@ -2892,6 +2912,7 @@ impl Omnigraph { .acquire(&crate::db::manifest::schema_apply_serial_queue_key()) .await; let _branch_guards = self.write_queue().acquire_branches(&control_branches).await; + let _stream_token_guard = self.write_queue().acquire_stream_token().await; self.ensure_schema_apply_not_locked("branch_create_from") .await?; let control_catalog = self.build_accepted_catalog_with_schema_gate_held().await?; @@ -2978,6 +2999,7 @@ impl Omnigraph { .acquire(&crate::db::manifest::schema_apply_serial_queue_key()) .await; let _branch_guards = self.write_queue().acquire_branches(&control_branches).await; + let _stream_token_guard = self.write_queue().acquire_stream_token().await; self.ensure_schema_apply_not_locked("branch_delete").await?; let control_catalog = self.build_accepted_catalog_with_schema_gate_held().await?; let table_queue_keys = @@ -3333,7 +3355,8 @@ fn blob_description_is_null(descriptions: &StructArray, row: usize) -> Result Result<()> { Ok(()) } +/// Derive the warm public reflection catalog from one validated physical +/// catalog. Keeping both views in the same `HandleSchemaView` publication +/// preserves cheap `catalog()` snapshots without letting storage-protocol +/// columns become SDK schema. +fn public_catalog_view(physical: &Catalog) -> Result { + fn strip(schema: &Arc, table_key: &str) -> Result> { + let mut hidden_count = 0; + let fields = schema + .fields() + .iter() + .filter_map(|field| { + if field.name() == crate::db::STREAM_METADATA_COLUMN { + hidden_count += 1; + None + } else { + Some(field.as_ref().clone()) + } + }) + .collect::>(); + if hidden_count != 1 { + return Err(OmniError::manifest_internal(format!( + "physical catalog for '{table_key}' must contain exactly one reserved field '{}'; found {hidden_count}", + crate::db::STREAM_METADATA_COLUMN + ))); + } + Ok(Arc::new(Schema::new_with_metadata( + fields, + schema.metadata().clone(), + ))) + } + + let mut public = physical.clone(); + for (name, node_type) in &mut public.node_types { + node_type.arrow_schema = strip(&node_type.arrow_schema, &format!("node:{name}"))?; + } + for (name, edge_type) in &mut public.edge_types { + edge_type.arrow_schema = strip(&edge_type.arrow_schema, &format!("edge:{name}"))?; + } + Ok(public) +} + fn physical_table_schema( schema: &Arc, blob_properties: &HashSet, table_key: &str, ) -> Result> { let mut id_count = 0; - let fields = schema + if schema + .fields() + .iter() + .any(|field| field.name() == crate::db::STREAM_METADATA_COLUMN) + { + return Err(OmniError::manifest_internal(format!( + "logical schema for '{table_key}' illegally supplies reserved field '{}'", + crate::db::STREAM_METADATA_COLUMN + ))); + } + + let mut fields = schema .fields() .iter() .map(|field| { @@ -3410,6 +3485,8 @@ fn physical_table_schema( ))); } + fields.push(crate::db::manifest::stream_token::trusted_stream_metadata_field()); + Ok(Arc::new(Schema::new_with_metadata( fields, schema.metadata.clone(), @@ -3548,6 +3625,9 @@ fn schema_for_table_key(catalog: &Catalog, table_key: &str) -> Result Result { let mut obj = serde_json::Map::new(); for (i, field) in batch.schema().fields().iter().enumerate() { + if field.name() == crate::db::STREAM_METADATA_COLUMN { + continue; + } obj.insert( field.name().clone(), json_value_from_array(batch.column(i).as_ref(), row)?, @@ -3556,6 +3636,187 @@ fn record_batch_row_to_json(batch: &RecordBatch, row: usize) -> Result Result> { + canonical_stream_payload_v1_with_limit( + batch, + row, + crate::table_store::mem_wal::B2_MAX_CANONICAL_PAYLOAD_BYTES, + ) +} + +fn canonical_stream_payload_v1_with_limit( + batch: &RecordBatch, + row: usize, + byte_limit: u64, +) -> Result> { + if row >= batch.num_rows() { + return Err(OmniError::manifest_internal(format!( + "canonical stream payload row {row} is outside a {}-row batch", + batch.num_rows() + ))); + } + let limit = usize::try_from(byte_limit) + .map_err(|_| OmniError::manifest_internal("canonical stream payload cap exceeds usize"))?; + let mut writer = BoundedCanonicalWriter::new(limit); + let encoded = (|| -> Result<()> { + use std::io::Write as _; + + writer.write_all(b"{").map_err(canonical_writer_error)?; + let mut first = true; + for (index, field) in batch.schema().fields().iter().enumerate() { + if field.name() == crate::db::STREAM_METADATA_COLUMN { + continue; + } + if !first { + writer.write_all(b",").map_err(canonical_writer_error)?; + } + first = false; + serde_json::to_writer(&mut writer, field.name()) + .map_err(canonical_json_error)?; + writer.write_all(b":").map_err(canonical_writer_error)?; + write_canonical_json_from_array(&mut writer, batch.column(index).as_ref(), row)?; + } + writer.write_all(b"}").map_err(canonical_writer_error)?; + Ok(()) + })(); + if writer.exceeded { + return Err(OmniError::resource_limit( + "stream_canonical_payload_bytes", + byte_limit, + u64::try_from(writer.attempted).unwrap_or(u64::MAX), + )); + } + encoded?; + Ok(writer.bytes) +} + +struct BoundedCanonicalWriter { + bytes: Vec, + limit: usize, + attempted: usize, + exceeded: bool, +} + +impl BoundedCanonicalWriter { + fn new(limit: usize) -> Self { + Self { + bytes: Vec::new(), + limit, + attempted: 0, + exceeded: false, + } + } +} + +impl std::io::Write for BoundedCanonicalWriter { + fn write(&mut self, bytes: &[u8]) -> std::io::Result { + self.attempted = self.bytes.len().saturating_add(bytes.len()); + if self.attempted > self.limit { + self.exceeded = true; + return Err(std::io::Error::other( + "canonical stream payload exceeds its configured byte cap", + )); + } + self.bytes.extend_from_slice(bytes); + Ok(bytes.len()) + } + + fn flush(&mut self) -> std::io::Result<()> { + Ok(()) + } +} + +fn canonical_writer_error(error: std::io::Error) -> OmniError { + OmniError::manifest_internal(format!("failed to encode canonical stream payload: {error}")) +} + +fn canonical_json_error(error: serde_json::Error) -> OmniError { + OmniError::manifest_internal(format!("failed to encode canonical stream payload: {error}")) +} + +fn write_canonical_json_from_array( + writer: &mut BoundedCanonicalWriter, + array: &dyn Array, + row: usize, +) -> Result<()> { + use std::io::Write as _; + + if array.is_null(row) { + writer.write_all(b"null").map_err(canonical_writer_error)?; + return Ok(()); + } + match array.data_type() { + DataType::List(_) => { + let list = array + .as_any() + .downcast_ref::() + .ok_or_else(|| OmniError::Lance("expected ListArray".to_string()))?; + write_canonical_json_list(writer, list.value(row).as_ref()) + } + DataType::LargeList(_) => { + let list = array + .as_any() + .downcast_ref::() + .ok_or_else(|| OmniError::Lance("expected LargeListArray".to_string()))?; + write_canonical_json_list(writer, list.value(row).as_ref()) + } + DataType::FixedSizeList(_, _) => { + let list = array + .as_any() + .downcast_ref::() + .ok_or_else(|| OmniError::Lance("expected FixedSizeListArray".to_string()))?; + write_canonical_json_list(writer, list.value(row).as_ref()) + } + DataType::Struct(fields) => { + let values = array + .as_any() + .downcast_ref::() + .ok_or_else(|| OmniError::Lance("expected StructArray".to_string()))?; + writer.write_all(b"{").map_err(canonical_writer_error)?; + for (index, field) in fields.iter().enumerate() { + if index != 0 { + writer.write_all(b",").map_err(canonical_writer_error)?; + } + serde_json::to_writer(&mut *writer, field.name()) + .map_err(canonical_json_error)?; + writer.write_all(b":").map_err(canonical_writer_error)?; + write_canonical_json_from_array(writer, values.column(index).as_ref(), row)?; + } + writer.write_all(b"}").map_err(canonical_writer_error)?; + Ok(()) + } + _ => { + let value = json_value_from_array(array, row)?; + serde_json::to_writer(writer, &value).map_err(canonical_json_error) + } + } +} + +fn write_canonical_json_list( + writer: &mut BoundedCanonicalWriter, + values: &dyn Array, +) -> Result<()> { + use std::io::Write as _; + + writer.write_all(b"[").map_err(canonical_writer_error)?; + for index in 0..values.len() { + if index != 0 { + writer.write_all(b",").map_err(canonical_writer_error)?; + } + write_canonical_json_from_array(writer, values, index)?; + } + writer.write_all(b"]").map_err(canonical_writer_error)?; + Ok(()) +} + fn json_value_from_array(array: &dyn Array, row: usize) -> Result { if array.is_null(row) { return Ok(serde_json::Value::Null); @@ -3728,6 +3989,7 @@ fn json_value_from_array(array: &dyn Array, row: usize) -> Result Person { edge WorksAt: Person -> Company "#; + #[test] + fn canonical_stream_payload_v1_has_schema_ordered_known_answer() { + let nested_fields = vec![ + Arc::new(Field::new("z_child", DataType::Utf8, false)), + Arc::new(Field::new("a_child", DataType::Int32, false)), + ]; + let nested = StructArray::from(vec![ + ( + Arc::clone(&nested_fields[0]), + Arc::new(StringArray::from(vec!["nested"])) as Arc, + ), + ( + Arc::clone(&nested_fields[1]), + Arc::new(Int32Array::from(vec![9])) as Arc, + ), + ]); + let list = ListArray::from_iter_primitive::(vec![Some(vec![ + Some(1), + None, + Some(3), + ])]); + let schema = Arc::new(Schema::new(vec![ + Field::new("z", DataType::Utf8, false), + Field::new("nil", DataType::Int32, true), + Field::new("integer", DataType::Int64, false), + Field::new("float", DataType::Float64, false), + Field::new("date", DataType::Date32, false), + Field::new("binary", DataType::Binary, false), + Field::new("list", list.data_type().clone(), false), + Field::new("struct", DataType::Struct(nested_fields.into()), false), + ])); + let batch = RecordBatch::try_new( + schema, + vec![ + Arc::new(StringArray::from(vec!["a\n\"b"])), + Arc::new(Int32Array::from(vec![None])), + Arc::new(Int64Array::from(vec![-4])), + Arc::new(Float64Array::from(vec![1.5])), + Arc::new(Date32Array::from(vec![7])), + Arc::new(BinaryArray::from_vec(vec![b"\x00\xff".as_slice()])), + Arc::new(list), + Arc::new(nested), + ], + ) + .unwrap(); + + let encoded = canonical_stream_payload_v1(&batch, 0).unwrap(); + assert_eq!( + String::from_utf8(encoded).unwrap(), + r#"{"z":"a\n\"b","nil":null,"integer":-4,"float":1.5,"date":7,"binary":"AP8=","list":[1,null,3],"struct":{"z_child":"nested","a_child":9}}"# + ); + } + + #[test] + fn canonical_stream_payload_v1_fails_with_typed_bound_before_retaining_excess() { + let batch = RecordBatch::try_new( + Arc::new(Schema::new(vec![Field::new("id", DataType::Utf8, false)])), + vec![Arc::new(StringArray::from(vec!["payload"]))], + ) + .unwrap(); + let error = canonical_stream_payload_v1_with_limit(&batch, 0, 8).unwrap_err(); + assert!( + matches!( + error, + OmniError::ResourceLimitExceeded { + ref resource, + limit: 8, + actual, + } if resource == "stream_canonical_payload_bytes" && actual > 8 + ), + "{error:?}" + ); + } + #[derive(Debug)] struct RecordingStorageAdapter { inner: ObjectStorageAdapter, @@ -4360,8 +4696,7 @@ edge WorksAt: Person -> Company .stream_lifecycle(manifest.snapshot().entry("node:Person").unwrap().identity) .cloned() .unwrap(); - let mut sealed = current.clone(); - sealed.lifecycle = crate::db::manifest::StreamLifecycle::Sealed; + let sealed = crate::db::manifest::stream::test_sealed_lifecycle_from(¤t).unwrap(); manifest .commit_changes(&[ManifestChange::SetStreamLifecycle { expected: Some(current), diff --git a/crates/omnigraph/src/db/omnigraph/export.rs b/crates/omnigraph/src/db/omnigraph/export.rs index fc37a262..3cb3ca11 100644 --- a/crates/omnigraph/src/db/omnigraph/export.rs +++ b/crates/omnigraph/src/db/omnigraph/export.rs @@ -238,7 +238,13 @@ fn write_export_rows_from_batch( "id".to_string(), json_value_from_named_column(batch, "id", row)?, ); - for field in node_type.arrow_schema.fields().iter().skip(1) { + for field in node_type + .arrow_schema + .fields() + .iter() + .skip(1) + .filter(|field| field.name() != crate::db::STREAM_METADATA_COLUMN) + { data.insert( field.name().clone(), export_value_for_field( @@ -274,7 +280,13 @@ fn write_export_rows_from_batch( "id".to_string(), json_value_from_named_column(batch, "id", row)?, ); - for field in edge_type.arrow_schema.fields().iter().skip(3) { + for field in edge_type + .arrow_schema + .fields() + .iter() + .skip(3) + .filter(|field| field.name() != crate::db::STREAM_METADATA_COLUMN) + { data.insert( field.name().clone(), export_value_for_field( diff --git a/crates/omnigraph/src/db/omnigraph/optimize.rs b/crates/omnigraph/src/db/omnigraph/optimize.rs index 5ec7e705..645c1c1a 100644 --- a/crates/omnigraph/src/db/omnigraph/optimize.rs +++ b/crates/omnigraph/src/db/omnigraph/optimize.rs @@ -256,6 +256,7 @@ pub async fn optimize_all_tables(db: &Omnigraph) -> Result Result<()> { let sidecars = crate::db::manifest::list_sidecars(db.root_uri(), db.storage_adapter()).await?; let blocking = sidecars.iter().find(|sidecar| { - sidecar.writer_kind == crate::db::manifest::SidecarKind::SchemaApply + sidecar.writer_kind.is_graph_global_barrier() || sidecar .branch .as_deref() @@ -1107,6 +1108,7 @@ pub async fn cleanup_all_tables( graph_branches.sort(); graph_branches.dedup(); let _cleanup_branch_guards = db.write_queue().acquire_branches(&graph_branches).await; + let _cleanup_stream_token_guard = db.write_queue().acquire_stream_token().await; let gc_queue_keys = db.table_queue_keys_for_branches(&graph_branches, &cleanup_catalog); let _cleanup_table_guards = db.write_queue().acquire_many(&gc_queue_keys).await; diff --git a/crates/omnigraph/src/db/omnigraph/repair.rs b/crates/omnigraph/src/db/omnigraph/repair.rs index 99797f13..51c8e2d1 100644 --- a/crates/omnigraph/src/db/omnigraph/repair.rs +++ b/crates/omnigraph/src/db/omnigraph/repair.rs @@ -159,6 +159,7 @@ pub async fn repair_all_tables(db: &Omnigraph, options: RepairOptions) -> Result db.ensure_schema_apply_not_locked("repair").await?; let catalog = db.load_accepted_catalog_with_schema_gate_held().await?; let _main_branch_guard = db.write_queue().acquire_branch(None).await; + let _stream_token_guard = db.write_queue().acquire_stream_token().await; let table_keys = optimize::all_table_keys(&catalog); let queue_keys = table_keys diff --git a/crates/omnigraph/src/db/omnigraph/schema_apply.rs b/crates/omnigraph/src/db/omnigraph/schema_apply.rs index ac910e26..7212a6c4 100644 --- a/crates/omnigraph/src/db/omnigraph/schema_apply.rs +++ b/crates/omnigraph/src/db/omnigraph/schema_apply.rs @@ -189,7 +189,7 @@ pub(super) async fn preview_schema_apply( let planned = plan_schema_for_apply(db, desired_schema_source, options).await?; Ok(SchemaApplyPreview { plan: planned.plan, - catalog: planned.desired_catalog, + catalog: super::public_catalog_view(&planned.desired_catalog)?, }) } @@ -712,6 +712,7 @@ where // therefore cover only the concrete table effects; acquiring the schema key // again would deadlock because these queues are intentionally non-reentrant. let _main_branch_guard = db.write_queue().acquire_branch(None).await; + let _stream_token_guard = db.write_queue().acquire_stream_token().await; let _schema_apply_queue_guards = db .write_queue() .acquire_many(&schema_apply_queue_keys) diff --git a/crates/omnigraph/src/db/omnigraph/stream_enrollment.rs b/crates/omnigraph/src/db/omnigraph/stream_enrollment.rs index 0748727b..e8ea5a56 100644 --- a/crates/omnigraph/src/db/omnigraph/stream_enrollment.rs +++ b/crates/omnigraph/src/db/omnigraph/stream_enrollment.rs @@ -1,17 +1,18 @@ //! RFC-026 bounded stream enrollment orchestration. //! //! This module intentionally exposes no production SDK/API surface. It joins -//! the exact MemWAL adapter, recovery-v10 sidecar, manifest-v7 lifecycle row, +//! the exact MemWAL adapter, enrollment recovery sidecar, manifest lifecycle row, //! and process-local admission lease so the crash path and success path share -//! one roll-forward implementation. Internal schema v8 enrolls directly into -//! the exact config-v2 profile consumed by the private Phase-B1 worker. +//! one roll-forward implementation. Internal schema v9 fixes the complete +//! config-v3/state-v2 enrollment receipt before the recovery intent is armed. use lance_index::mem_wal::ShardId; use crate::db::manifest::{ - RecoveryAuthorityToken, RecoveryLineageIntent, STREAM_CONFIG_VERSION, SidecarTablePin, - StreamLifecycle, StreamPhysicalBinding, complete_stream_enrollment_sidecar_v10, - new_stream_enrollment_sidecar_v10, write_sidecar, + EnrollmentReceipt, RecoveryAuthorityToken, RecoveryLineageIntent, STREAM_CONFIG_VERSION, + SidecarTablePin, StreamLifecycle, StreamPhysicalBinding, + complete_stream_enrollment_sidecar_v10, new_stream_enrollment_sidecar_v10, + stream_enrollment_intent_digest_v1, write_sidecar, }; use crate::db::write_queue::StreamAdmissionKey; use crate::error::{OmniError, Result}; @@ -23,13 +24,21 @@ use crate::table_store::mem_wal::{ use super::Omnigraph; impl Omnigraph { - /// Validate the complete internal-v8 stream capability before a handle is + /// Validate the complete internal-v9 stream capability before a handle is /// served. Recovery has already consumed every parseable enrollment intent /// on read-write open; therefore a MemWAL index without a lifecycle row is /// uncovered partial format, while every lifecycle must match its exact - /// current table witness and bounded config-v2 generation topology. + /// current table witness and bounded config-v3 generation topology. pub(super) async fn validate_stream_format_consistency(&self) -> Result<()> { let snapshot = self.coordinator.read().await.snapshot(); + snapshot + .open_stream_token_authority() + .await + .map_err(|error| { + OmniError::manifest_internal(format!( + "internal-v9 stream-token authority consistency failed: {error}" + )) + })?; let has_active_stream = snapshot.stream_lifecycles().any(|(_, lifecycle)| { matches!( lifecycle.lifecycle, @@ -40,12 +49,22 @@ impl Omnigraph { let branches = self.coordinator.read().await.branch_list().await?; if branches.iter().any(|branch| branch != "main") { return Err(OmniError::manifest_internal( - "internal-v8 OPEN/DRAINING stream lifecycle cannot coexist with named graph branches", + "internal-v9 OPEN/DRAINING stream lifecycle cannot coexist with named graph branches", )); } } for entry in snapshot.entries() { let table = self.storage().open_snapshot_at_entry(entry).await?; + let physical_schema = arrow_schema::Schema::from(table.dataset().schema()); + crate::db::manifest::stream_token::validate_trusted_stream_metadata_schema( + &physical_schema, + ) + .map_err(|error| { + OmniError::manifest_internal(format!( + "internal-v9 stream format consistency failed for '{}' ({}): {error}", + entry.table_key, entry.identity + )) + })?; let full_path = format!( "{}/{}", self.root_uri().trim_end_matches('/'), @@ -102,7 +121,7 @@ impl Omnigraph { }; validation.map_err(|error| { OmniError::manifest_internal(format!( - "internal-v8 stream format consistency failed for '{}' ({}): {error}", + "internal-v9 stream format consistency failed for '{}' ({}): {error}", entry.table_key, entry.identity )) })?; @@ -110,7 +129,7 @@ impl Omnigraph { Ok(()) } - /// Enroll one existing main-branch table into the bounded config-v2 + /// Enroll one existing main-branch table into the bounded config-v3 /// physical profile. Kept crate-private until Phase B2 supplies the public /// stream contract. #[allow(dead_code)] @@ -153,6 +172,16 @@ impl Omnigraph { .map_err(|error| OmniError::manifest_conflict(error.to_string()))?; let enrollment_plan = MemWalEnrollmentPlan::new(ShardId::new_v4(), ShardId::new_v4()) .map_err(|error| OmniError::manifest_internal(error.to_string()))?; + // The public B2 API will accept the request ID from its caller. This + // still-private seam mints that id on the caller's behalf, but fixes it + // together with a separate logical incarnation before sidecar arm. + let enrollment_request_id = + mint_distinct_stream_uuid(&[enrollment_plan.enrollment_id, enrollment_plan.shard_id]); + let stream_incarnation_id = mint_distinct_stream_uuid(&[ + enrollment_plan.enrollment_id, + enrollment_plan.shard_id, + enrollment_request_id, + ]); let binding = StreamPhysicalBinding { stable_table_id: prepared_entry.identity.stable_table_id, table_incarnation_id: prepared_entry.identity.table_incarnation_id, @@ -174,6 +203,7 @@ impl Omnigraph { let schema_gate_key = crate::db::manifest::schema_apply_serial_queue_key(); let _schema_guard = write_queue.acquire(&schema_gate_key).await; let _branch_guard = write_queue.acquire_branch(None).await; + let _stream_token_guard = write_queue.acquire_stream_token().await; let _table_guards = write_queue .acquire_many(&[(prepared_entry.table_key.clone(), None)]) .await; @@ -247,6 +277,22 @@ impl Omnigraph { )); } + let enrollment_intent_digest = stream_enrollment_intent_digest_v1( + live_entry.identity, + &live_entry.table_path, + &txn.authority.schema_identity_domain, + &txn.authority.schema_ir_hash, + txn.authority.schema_identity_version, + &final_head, + &binding.stream_config_hash, + )?; + let enrollment_receipt = EnrollmentReceipt::new( + enrollment_request_id.to_string(), + enrollment_intent_digest, + stream_incarnation_id.to_string(), + binding.clone(), + )?; + let authority = RecoveryAuthorityToken { branch_identifier: txn.authority.branch_identifier.clone(), graph_head: txn.authority.graph_head.clone(), @@ -280,6 +326,7 @@ impl Omnigraph { final_head, enrollment_plan, binding.clone(), + enrollment_receipt, )?; write_sidecar(self.root_uri(), self.storage_adapter(), &sidecar).await?; crate::failpoints::maybe_fail( @@ -344,3 +391,12 @@ impl Omnigraph { self.enroll_stream_table_b1(table_key, None).await } } + +fn mint_distinct_stream_uuid(excluded: &[ShardId]) -> ShardId { + loop { + let candidate = ShardId::new_v4(); + if !excluded.contains(&candidate) { + return candidate; + } + } +} diff --git a/crates/omnigraph/src/db/omnigraph/stream_ingest.rs b/crates/omnigraph/src/db/omnigraph/stream_ingest.rs index fa5c9a45..e87391cf 100644 --- a/crates/omnigraph/src/db/omnigraph/stream_ingest.rs +++ b/crates/omnigraph/src/db/omnigraph/stream_ingest.rs @@ -7,32 +7,55 @@ //! the graph authority checks around those primitives and the one //! `__manifest` publication which makes a folded generation visible. +#[cfg(feature = "failpoints")] +use std::collections::{BTreeMap, BTreeSet}; use std::sync::Arc; -use arrow_array::{Array, RecordBatch, UInt32Array}; +use arrow_array::{Array, RecordBatch, StringArray, UInt32Array}; use arrow_schema::Schema as ArrowSchema; use arrow_select::take::take; +use datafusion::prelude::{col, lit}; use futures::TryStreamExt; use lance::dataset::mem_wal::scanner::LsmScanner; use lance::dataset::mem_wal::{DatasetMemWalExt, ShardWriter}; use lance_index::mem_wal::{MemWalIndexDetails, MergedGeneration, ShardId, ShardStatus}; use crate::db::manifest::{ - RecoveryAuthorityToken, RecoveryLineageIntent, RecoveryStreamFoldCut, SidecarKind, - SidecarTablePin, StreamLifecycle, StreamLifecycleEntry, StreamPhysicalBinding, TableIdentity, - complete_stream_fold_sidecar_v11, finalize_effect_free_stream_fold_sidecar_v11, list_sidecars, - new_stream_fold_sidecar_v11, write_sidecar, + CurrentHeadWitness, RecoveryAuthorityToken, RecoveryLineageIntent, RecoveryStreamFoldCut, + SidecarTablePin, + StreamLifecycle, StreamLifecycleEntry, StreamPhysicalBinding, TableIdentity, + complete_stream_fold_sidecar_v12, confirm_stream_fold_sidecar_v12, + finalize_effect_free_stream_fold_sidecar_v12, list_sidecars, new_stream_fold_sidecar_v12, + write_sidecar, +}; +use crate::db::manifest::stream_token::{ + AdmissionClassification, AdmissionRequest, PayloadDigest, PayloadDigestInput, + StreamFoldAttributionSummary, StreamRowOrigin, StreamToken, StreamTokenAuthorityRow, + StreamWriteEnvelope, TrustedContributorId, TrustedStreamRowMetadata, + build_trusted_stream_metadata_array, classify_admission, decode_trusted_stream_metadata, + stream_fold_attribution_commitment, validate_authority_base_pair, +}; +use crate::db::manifest::stream::{ + LastFoldOutcome, LastFoldSummary, StreamGenerationCut, +}; +use crate::db::manifest::token_store::{ + add_stream_lookup_retained_bytes, lookup_stream_token_row, open_stream_token_authority_head, + stage_stream_token_upsert, stream_token_authority_entry_for_dataset, stream_token_rows_for_keys, + validate_stream_token_plan_bounds, }; use crate::db::write_queue::StreamAdmissionKey; use crate::error::{OmniError, Result}; use crate::storage_layer::SnapshotHandle; use crate::table_store::mem_wal::{ - B1_MAX_GENERATION_ARROW_BYTES, B1_MAX_GENERATION_ROWS, CallerOrdinalRange, - CheckedExclusiveStreamAuthority, CheckedStreamAuthority, ClaimedMemWalWorker, DurableBatchAck, + B1_MAX_GENERATION_ARROW_BYTES, B1_MAX_GENERATION_ROWS, + B2_MAX_TOKEN_PROJECTION_ARROW_BYTES, CallerOrdinalRange, + CheckedExclusiveStreamAuthority, CheckedStreamAuthority, ClaimedMemWalWorker, + ConfirmedStreamTokenOverlay, ConfirmedStreamTokenOverlayRow, DurableBatchAck, IdleAuthorityCheck, IdleAuthorityFailure, MemWalWorkerError, OpenedMemWalWorker, PreparedPut, - PreparedPutFailure, SealedGenerationCut, StreamWorkerKey, WorkerOpenFailure, - b1_input_accounting, capture_current_head_witness, reconstruct_b1_writer_config, - validate_stream_config_v2_binding, + PreparedPutFailure, QueuedBatchPermit, SealedGenerationCut, StreamWorkerKey, + WorkerOpenFailure, + b1_input_accounting, b1_logical_batch_bytes, capture_current_head_witness, + reconstruct_b1_writer_config, validate_stream_config_v3_binding, }; use crate::validate::{ChangeSet, CommittedState, TableChange}; @@ -40,6 +63,16 @@ use super::{Omnigraph, WriteTxn}; const B1_MAX_FOLD_ATTEMPTS: usize = 2; +/// Private B2 result for one caller occurrence. Public response shaping stays +/// deliberately inactive; this value exists so crash/race tests can prove the +/// sequencing contract without exposing a product API. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct StreamTokenAdmissionAck { + pub(crate) stream_token: StreamToken, + pub(crate) origin: StreamRowOrigin, + pub(crate) already_durable: bool, +} + /// One coherent main-branch stream authority capture. /// /// `head` is opened at the exact physical HEAD proven equal to the lifecycle @@ -60,14 +93,22 @@ struct StreamAuthorityCapture { details: MemWalIndexDetails, } +#[derive(Debug, Clone, PartialEq, Eq)] +struct AttributedFoldPlan { + token_rows: Vec, + summary: StreamFoldAttributionSummary, +} + impl Omnigraph { - /// Admit one already-normalized, non-empty physical batch into the private - /// B1 MemWAL profile and acknowledge only after Lance's durability watcher - /// succeeds and the same worker observes no successor shard epoch. + /// Admit one already-normalized, non-empty physical batch through the + /// feature-gated B1 substrate seam. Synthetic trusted envelopes make these + /// older worker/capacity tests exercise the active B2 fold and recovery-v12 + /// path without exposing an unattributed writer in production. /// /// The `Arc` receiver is intentional. The worker owns a detached task so /// dropping the requesting future cannot abandon an invoked put, its /// watcher, or quiesced retirement. + #[cfg(feature = "failpoints")] pub(crate) async fn stream_put_phase_b1( self: &Arc, table_key: &str, @@ -77,21 +118,42 @@ impl Omnigraph { // Refuse an oversized caller buffer synchronously before recovery IO, // authority capture, detached ownership, or a cold epoch claim. validate_stream_input_bounds(table_key, &batch)?; + let batch = self + .storage() + .prepare_keyed_write_batch(table_key, batch) + .await?; + validate_stream_input_bounds(table_key, &batch)?; self.heal_pending_recovery_sidecars_for_write(&[None]) .await?; - let prepared = self + let provisional = self .capture_stream_authority(table_key, "stream put") .await?; - self.validate_stream_admission_batch(&prepared, &batch)?; + self.validate_stream_logical_admission_batch(&provisional, &batch)?; - let key = prepared.worker_key; - let admission_key = prepared.admission_key.clone(); + let ids = batch + .column_by_name("id") + .and_then(|array| array.as_any().downcast_ref::()) + .ok_or_else(|| { + OmniError::manifest_internal( + "validated B1 test batch has no exact Utf8 id column", + ) + })?; + let mut logical_ids = BTreeSet::new(); + for row in 0..batch.num_rows() { + if ids.is_null(row) { + return Err(OmniError::manifest("stream row id must be non-null")); + } + logical_ids.insert(ids.value(row).to_string()); + } + + let key = provisional.worker_key; + let admission_key = provisional.admission_key.clone(); let authority_db = Arc::clone(self); let authority_key = admission_key.clone(); // Inflight and Arrow budgets are acquired before any same-key queue or // shared-admission wait. The returned token is already inside that // bounded corridor and moves into the detached prepare closure. - let (queued, put_authority) = self + let (mut queued, put_authority) = self .stream_workers .reserve_put_input(key, table_key, &batch, move || async move { let shared = authority_db @@ -101,12 +163,494 @@ impl Omnigraph { CheckedStreamAuthority::from_shared_admission(shared) }) .await?; - let table_key_owned = table_key.to_string(); + + // The provisional capture exists only to choose the immutable + // admission domain. A fold can publish while this caller is waiting for + // the shared lease, so every read and every effect-free result below is + // based on a fresh capture made after that lease and the same-key queue + // are both owned. + self.ensure_no_relevant_stream_sidecar(key.identity, "stream put") + .await?; + let prepared = self + .capture_stream_authority(table_key, "stream put final admission") + .await?; + ensure_same_binding(key, &prepared, "stream put final admission authority")?; + self.validate_stream_logical_admission_batch(&prepared, &batch)?; + + // The table-wide input queue makes the warm overlay stable while the + // synthetic batch is chained. Durable authority is read once with one + // structured, generation-bounded exact-id predicate. + let mut generation_current = BTreeMap::new(); + for logical_id in &logical_ids { + if let Some(current) = self + .stream_workers + .confirmed_token_for_key(&queued, table_key, logical_id) + .await? + { + generation_current.insert(logical_id.clone(), current); + } + } + let token_dataset = prepared.txn.base.open_stream_token_authority().await?; + let durable_current = stream_token_rows_for_keys( + &token_dataset, + prepared.txn.base.stream_token_authority(), + prepared.entry.identity, + &logical_ids, + ) + .await?; + let durable_base_metadata = lookup_base_stream_metadata_for_keys( + prepared.head.dataset(), + prepared.entry.identity, + &logical_ids, + ) + .await?; + for logical_id in &logical_ids { + if generation_current.contains_key(logical_id) { + continue; + } + validate_authority_base_pair( + prepared.entry.identity, + logical_id, + durable_current.get(logical_id), + durable_base_metadata.get(logical_id), + ) + .map_err(|error| OmniError::manifest_internal(error.to_string()))?; + } + let contributor = TrustedContributorId::new("omnigraph:test-b1") + .map_err(|error| OmniError::manifest_internal(error.to_string()))?; + let stream_incarnation_id = prepared + .lifecycle + .enrollment_receipt + .stream_incarnation_id + .clone(); + let mut metadata_rows = Vec::with_capacity(batch.num_rows()); + let mut confirmed_token_updates = ConfirmedStreamTokenOverlay::new(); + for row in 0..batch.num_rows() { + let logical_id = ids.value(row).to_string(); + let (predecessor_token, fold_base_token, chain_depth) = + if let Some(current) = generation_current.get(&logical_id) { + ( + Some(current.authority.current_token), + current.metadata.fold_base_token, + current.metadata.chain_depth.checked_add(1).ok_or_else(|| { + OmniError::resource_limit( + format!("stream chain depth for {table_key}/{logical_id}"), + u32::MAX as u64, + u32::MAX as u64 + 1, + ) + })?, + ) + } else if let Some(current) = durable_current.get(&logical_id) { + ( + Some(current.current_token), + Some(current.current_token), + 1, + ) + } else { + (None, None, 1) + }; + let canonical_payload = super::canonical_stream_payload_v1(&batch, row)?; + let payload_digest = PayloadDigest::derive(&PayloadDigestInput { + identity: prepared.entry.identity, + accepted_schema_hash: &prepared.txn.authority.schema_ir_hash, + canonical_payload: &canonical_payload, + }) + .map_err(|error| OmniError::manifest_internal(error.to_string()))?; + let request = AdmissionRequest { + identity: prepared.entry.identity, + logical_id: logical_id.clone(), + envelope: StreamWriteEnvelope { + stream_incarnation_id: stream_incarnation_id.clone(), + write_id: ShardId::new_v4().to_string(), + predecessor_token, + }, + contributor_id: contributor.clone(), + payload_digest, + }; + let candidate = request + .candidate_token() + .map_err(|error| OmniError::manifest_internal(error.to_string()))?; + let caller_ordinal = caller_ordinals + .start + .checked_add(u64::try_from(row).map_err(|_| { + OmniError::manifest_internal("B1 test row ordinal exceeds u64") + })?) + .ok_or_else(|| OmniError::manifest_internal("B1 test ordinal overflow"))?; + let metadata = TrustedStreamRowMetadata::new_admission( + &request, + candidate, + fold_base_token, + chain_depth, + ShardId::new_v4().to_string(), + caller_ordinal, + ) + .map_err(|error| OmniError::manifest_internal(error.to_string()))?; + let authority = StreamTokenAuthorityRow::from_present_metadata( + request.identity, + logical_id.clone(), + prepared.binding.enrollment_id.clone(), + &metadata, + ) + .map_err(|error| OmniError::manifest_internal(error.to_string()))?; + let overlay = ConfirmedStreamTokenOverlayRow { + authority, + metadata: metadata.clone(), + }; + generation_current.insert(logical_id.clone(), overlay.clone()); + confirmed_token_updates.insert(logical_id, overlay); + metadata_rows.push(Some(metadata)); + } + let projected_token_rows = self + .stream_workers + .projected_token_authority_rows( + &queued, + table_key, + &confirmed_token_updates, + ) + .await?; + validate_generation_token_plan(table_key, &projected_token_rows)?; + let batch = append_trusted_stream_metadata(batch, metadata_rows)?; + self.validate_stream_admission_batch(&prepared, &batch)?; + queued.reprice_for_exact_batch(table_key, &batch)?; + self.finish_reserved_stream_put( + table_key.to_string(), + batch, + caller_ordinals, + key, + admission_key, + queued, + put_authority, + confirmed_token_updates, + ) + .await + } + + /// Admit one fully normalized logical row through RFC-026's private B2 + /// compare-and-chain boundary. The caller supplies only its idempotency + /// envelope; contributor identity is already resolved by the trusted + /// engine boundary. This remains crate-private and is exposed to tests + /// only through the feature-gated seam below. + pub(crate) async fn stream_put_phase_b2_one( + self: &Arc, + table_key: &str, + batch: RecordBatch, + caller_ordinal: u64, + envelope: StreamWriteEnvelope, + contributor_id: TrustedContributorId, + ) -> Result { + if batch.num_rows() != 1 { + return Err(OmniError::manifest(format!( + "private B2 admission requires exactly one row, got {}", + batch.num_rows() + ))); + } + validate_stream_input_bounds(table_key, &batch)?; + let mut preprocessing = self.stream_workers.reserve_b2_preprocessing()?; + // Blob bytes participate in the payload digest. Resolve them before + // recovery/authority and never hash an external descriptor which may + // change after acknowledgement. + let batch = self + .storage() + .prepare_keyed_write_batch(table_key, batch) + .await?; + validate_stream_input_bounds(table_key, &batch)?; + + let id_array = batch + .column_by_name("id") + .and_then(|array| array.as_any().downcast_ref::()) + .ok_or_else(|| { + OmniError::manifest_internal( + "validated stream admission batch has no exact Utf8 id column", + ) + })?; + if id_array.is_null(0) { + return Err(OmniError::manifest("stream row id must be non-null")); + } + let logical_id = id_array.value(0).to_string(); + let canonical_payload = super::canonical_stream_payload_v1(&batch, 0)?; + envelope + .validate() + .map_err(|error| OmniError::manifest(error.to_string()))?; + + self.heal_pending_recovery_sidecars_for_write(&[None]) + .await?; + let provisional = self + .capture_stream_authority(table_key, "stream token admission") + .await?; + self.validate_stream_logical_admission_batch(&provisional, &batch)?; + let key = provisional.worker_key; + let admission_key = provisional.admission_key.clone(); + crate::failpoints::maybe_fail( + crate::failpoints::names::STREAM_B2_AFTER_PROVISIONAL_AUTHORITY, + )?; + let authority_db = Arc::clone(self); + let authority_key = admission_key.clone(); + let (mut queued, put_authority) = self + .stream_workers + .reserve_b2_put_input( + key, + table_key, + &batch, + &mut preprocessing, + move || async move { + let shared = authority_db + .write_queue() + .acquire_stream_shared(&authority_key) + .await; + CheckedStreamAuthority::from_shared_admission(shared) + }, + ) + .await?; + + self.ensure_no_relevant_stream_sidecar(key.identity, "stream token admission") + .await?; + let prepared = self + .capture_stream_authority(table_key, "stream token final admission") + .await?; + ensure_same_binding( + key, + &prepared, + "stream token final admission authority", + )?; + self.validate_stream_logical_admission_batch(&prepared, &batch)?; + let payload_digest = PayloadDigest::derive(&PayloadDigestInput { + identity: prepared.entry.identity, + accepted_schema_hash: &prepared.txn.authority.schema_ir_hash, + canonical_payload: &canonical_payload, + }) + .map_err(|error| OmniError::manifest_internal(error.to_string()))?; + drop(canonical_payload); + drop(preprocessing); + let request = AdmissionRequest { + identity: prepared.entry.identity, + logical_id: logical_id.clone(), + envelope, + contributor_id, + payload_digest, + }; + request + .validate() + .map_err(|error| OmniError::manifest(error.to_string()))?; + + // Owning the same-key queue makes this overlay snapshot stable until + // the permit transfers into the worker. The shared admission lease + // simultaneously excludes a fold/token-table publication. + let overlay_current = self + .stream_workers + .confirmed_token_for_key(&queued, table_key, &logical_id) + .await?; + + let (durable_authority, durable_metadata) = if overlay_current.is_none() { + let token_dataset = prepared.txn.base.open_stream_token_authority().await?; + let authority = lookup_stream_token_row( + &token_dataset, + prepared.txn.base.stream_token_authority(), + prepared.entry.identity, + &logical_id, + ) + .await?; + // A missing token row plus a non-null base copy is corruption, so + // the base probe is unconditional whenever no confirmed overlay + // owns the key. + let metadata = lookup_base_stream_metadata( + prepared.head.dataset(), + prepared.entry.identity, + &logical_id, + ) + .await?; + (authority, metadata) + } else { + (None, None) + }; + let current_authority = overlay_current + .as_ref() + .map(|row| &row.authority) + .or(durable_authority.as_ref()); + let current_metadata = overlay_current + .as_ref() + .map(|row| &row.metadata) + .or(durable_metadata.as_ref()); + let stream_incarnation_id = prepared + .lifecycle + .enrollment_receipt + .stream_incarnation_id + .as_str(); + let classification = classify_admission( + stream_incarnation_id, + &request, + current_authority, + current_metadata, + ) + .map_err(|error| OmniError::manifest_internal(error.to_string()))?; + + let candidate = match classification { + AdmissionClassification::AlreadyDurable { authority, .. } => { + return Ok(StreamTokenAdmissionAck { + stream_token: authority.current_token, + origin: authority.origin, + already_durable: true, + }); + } + AdmissionClassification::BindingChanged { + current_stream_incarnation_id, + } => { + return Err(OmniError::StreamBindingChanged { + stable_table_id: request.identity.stable_table_id, + table_incarnation_id: request.identity.table_incarnation_id, + current_stream_incarnation_id, + }); + } + AdmissionClassification::SequenceConflict { current_token } => { + return Err(OmniError::StreamSequenceConflict { + stable_table_id: request.identity.stable_table_id, + table_incarnation_id: request.identity.table_incarnation_id, + logical_id, + current_token: current_token.map(|token| token.to_string()), + }); + } + AdmissionClassification::IdempotencyConflict { current_token } => { + return Err(OmniError::StreamIdempotencyConflict { + stable_table_id: request.identity.stable_table_id, + table_incarnation_id: request.identity.table_incarnation_id, + logical_id, + current_token: current_token.to_string(), + }); + } + AdmissionClassification::New { candidate_token } => candidate_token, + }; + + let (fold_base_token, chain_depth) = match overlay_current.as_ref() { + Some(current) => ( + current.metadata.fold_base_token, + current.metadata.chain_depth.checked_add(1).ok_or_else(|| { + OmniError::resource_limit( + format!("stream chain depth for {table_key}/{logical_id}"), + u32::MAX as u64, + u32::MAX as u64 + 1, + ) + })?, + ), + None => (request.envelope.predecessor_token, 1), + }; + let metadata = TrustedStreamRowMetadata::new_admission( + &request, + candidate, + fold_base_token, + chain_depth, + ShardId::new_v4().to_string(), + caller_ordinal, + ) + .map_err(|error| OmniError::manifest_internal(error.to_string()))?; + let authority = crate::db::manifest::stream_token::StreamTokenAuthorityRow::from_present_metadata( + request.identity, + logical_id.clone(), + prepared.binding.enrollment_id.clone(), + &metadata, + ) + .map_err(|error| OmniError::manifest_internal(error.to_string()))?; + // A row which cannot fit an otherwise-empty token/recovery projection + // is terminal for this occurrence; asking the caller to fold would + // create an endless retry loop. + validate_stream_token_plan_bounds(std::slice::from_ref(&authority))?; + let origin = metadata.origin.clone(); + let admission_attempt_id = match &origin { + StreamRowOrigin::Admission { + admission_attempt_id, + .. + } => admission_attempt_id.clone(), + StreamRowOrigin::Correction { .. } => { + return Err(OmniError::manifest_internal( + "stream admission minted a correction origin", + )); + } + }; + let logical_write_id = request.envelope.write_id.clone(); + let batch = append_trusted_stream_metadata(batch, vec![Some(metadata.clone())])?; + self.validate_stream_admission_batch(&prepared, &batch)?; + queued.reprice_for_exact_batch(table_key, &batch)?; + let mut confirmed_token_updates = ConfirmedStreamTokenOverlay::new(); + confirmed_token_updates.insert( + logical_id, + ConfirmedStreamTokenOverlayRow { + authority, + metadata, + }, + ); + let projected_token_rows = self + .stream_workers + .projected_token_authority_rows( + &queued, + table_key, + &confirmed_token_updates, + ) + .await?; + validate_generation_token_plan(table_key, &projected_token_rows)?; + + if let Err(error) = self + .finish_reserved_stream_put( + table_key.to_string(), + batch, + CallerOrdinalRange::new(caller_ordinal, caller_ordinal).map_err(worker_error)?, + key, + admission_key, + queued, + put_authority, + confirmed_token_updates, + ) + .await + { + return Err(match error { + OmniError::AckUnknown { + stable_table_id, + table_incarnation_id, + enrollment_id, + shard_id, + caller_ordinal_start, + caller_ordinal_end, + reason, + .. + } => OmniError::AckUnknown { + stable_table_id, + table_incarnation_id, + enrollment_id, + shard_id, + caller_ordinal_start, + caller_ordinal_end, + admission_attempt_id: Some(admission_attempt_id), + logical_write_ids: vec![logical_write_id], + unconfirmed_candidate_token: Some(candidate.to_string()), + reason, + }, + other => other, + }); + } + Ok(StreamTokenAdmissionAck { + stream_token: candidate, + origin, + already_durable: false, + }) + } + + /// Finish one already-queued stream append. B1 supplies an empty token + /// projection; B2 supplies the exact watcher-confirmed updates which must + /// become warm only after the post-durability fence check. + #[allow(clippy::too_many_arguments)] + async fn finish_reserved_stream_put( + self: &Arc, + table_key: String, + batch: RecordBatch, + caller_ordinals: CallerOrdinalRange, + key: StreamWorkerKey, + admission_key: StreamAdmissionKey, + queued: QueuedBatchPermit, + put_authority: CheckedStreamAuthority, + confirmed_token_updates: ConfirmedStreamTokenOverlay, + ) -> Result { let admitted_batch = batch.clone(); let idle_db = Arc::clone(self); let idle_key = key; let idle_admission_key = admission_key.clone(); - let idle_table_key = table_key_owned.clone(); + let idle_table_key = table_key.clone(); let idle_authority: IdleAuthorityCheck = Arc::new(move |writer: Arc| { let db = Arc::clone(&idle_db); let admission_key = idle_admission_key.clone(); @@ -139,6 +683,7 @@ impl Omnigraph { }) }); let db = Arc::clone(self); + let prepare_table_key = table_key.clone(); let prepare = Box::new(move |warm_writer: Option>| { Box::pin(async move { // Admission is outermost and remains inside the detached worker @@ -151,18 +696,16 @@ impl Omnigraph { db.ensure_no_relevant_stream_sidecar(key.identity, "stream put") .await?; let before = db - .capture_stream_authority(&table_key_owned, "stream put") + .capture_stream_authority(&prepare_table_key, "stream put") .await?; ensure_same_binding(key, &before, "stream put final authority")?; db.validate_stream_admission_batch(&before, &admitted_batch)?; validate_claimed_writer(&writer, key, before.epoch_floor).await?; - // A warm put repeats every authority and physical - // check immediately before the worker invokes it. db.ensure_no_relevant_stream_sidecar(key.identity, "stream put") .await?; let after = db - .capture_stream_authority(&table_key_owned, "stream put") + .capture_stream_authority(&prepare_table_key, "stream put") .await?; ensure_same_capture( &before, @@ -183,7 +726,7 @@ impl Omnigraph { db.ensure_no_relevant_stream_sidecar(key.identity, "stream put") .await?; let before = db - .capture_stream_authority(&table_key_owned, "stream put") + .capture_stream_authority(&prepare_table_key, "stream put") .await?; ensure_same_binding(key, &before, "stream put final authority")?; db.validate_stream_admission_batch(&before, &admitted_batch)?; @@ -254,15 +797,11 @@ impl Omnigraph { )); } - // Claiming advances the shard epoch. Recheck graph, - // lifecycle, schema, base HEAD, and the claimed shard - // immediately before handing the still-held lease to - // the serialized put worker. let checked = async { db.ensure_no_relevant_stream_sidecar(key.identity, "stream put") .await?; let after = db - .capture_stream_authority(&table_key_owned, "stream put") + .capture_stream_authority(&prepare_table_key, "stream put") .await?; ensure_same_capture( &before, @@ -287,9 +826,10 @@ impl Omnigraph { self.stream_workers .put( key, - table_key.to_string(), + table_key, batch, caller_ordinals, + confirmed_token_updates, queued, prepare, idle_authority, @@ -372,10 +912,11 @@ impl Omnigraph { let mut last_effect_free_error = None; for attempt in 0..B1_MAX_FOLD_ATTEMPTS { - match self - .stream_fold_attempt(&table_key, key, &before_cut, &cut) - .await - { + // Keep the large closed recovery-v12 fold future off this Tokio + // worker's stack. Debug failpoint builds otherwise compose it with + // the surrounding recovery/barrier future and can exceed the + // default worker stack in multi-table crash tests. + match Box::pin(self.stream_fold_attempt(&table_key, key, &before_cut, &cut)).await { Ok(FoldAttempt::Published) => return Ok(()), Ok(FoldAttempt::EffectFree(error)) if attempt + 1 < B1_MAX_FOLD_ATTEMPTS => { last_effect_free_error = Some(error); @@ -412,6 +953,14 @@ impl Omnigraph { let batches = scan_fresh_generation(&prepared, cut).await?; validate_fold_output_bounds(table_key, &batches)?; + let attribution = plan_fold_attribution( + &prepared.txn.base, + key.identity, + &prepared.lifecycle, + &prepared.binding, + &batches, + ) + .await?; let mut changeset = ChangeSet::new(); changeset.insert( table_key.to_string(), @@ -431,12 +980,23 @@ impl Omnigraph { .stage_stream_fold( prepared.head.clone(), table_key, - batches, + batches.clone(), cut.key.shard_id, cut.generation, ) .await?; let planned_transaction = staged.transaction_identity(); + let token_dataset = prepared.txn.base.open_stream_token_authority().await?; + let token_staged = stage_stream_token_upsert( + token_dataset.clone(), + prepared.txn.base.stream_token_authority(), + &attribution.token_rows, + ) + .await?; + let token_stage = ( + token_staged.transaction_identity(), + crate::storage_layer::StagedHandle::new(token_staged), + ); // Admission remains exclusively held inside `cut`. Enter the normal // RFC-022 inner order only for final authority, sidecar arm, effect, and @@ -446,6 +1006,7 @@ impl Omnigraph { .acquire(&crate::db::manifest::schema_apply_serial_queue_key()) .await; let _branch_guard = write_queue.acquire_branch(None).await; + let _stream_token_guard = write_queue.acquire_stream_token().await; let _table_guards = write_queue .acquire_many(&[(table_key.to_string(), None)]) .await; @@ -479,6 +1040,31 @@ impl Omnigraph { ) })?; ensure_live_stream_prestate(&prepared, &live_entry, &live_lifecycle)?; + if live.stream_token_authority() != prepared.txn.base.stream_token_authority() { + return Err(OmniError::manifest_read_set_changed( + "stream_fold_token_authority", + Some(format!( + "{:?}", + prepared.txn.base.stream_token_authority() + )), + Some(format!("{:?}", live.stream_token_authority())), + )); + } + let revalidated = plan_fold_attribution( + &live, + key.identity, + &live_lifecycle, + &prepared.binding, + &batches, + ) + .await?; + if revalidated != attribution { + return Err(OmniError::manifest_read_set_changed( + "stream_fold_attribution", + Some("prepared attributed winner set".to_string()), + Some("changed attributed winner set".to_string()), + )); + } let final_head = self .storage() @@ -508,7 +1094,7 @@ impl Omnigraph { .map_err(|error| OmniError::Lance(error.to_string()))? .ok_or_else(|| OmniError::manifest_internal("stream fold lost its MemWAL index"))?; let (enrollment_id, shard_id) = - validate_stream_config_v2_binding(&final_details, &prepared.binding) + validate_stream_config_v3_binding(&final_details, &prepared.binding) .map_err(worker_error)?; if enrollment_id != key.enrollment_id || shard_id != key.shard_id { return Err(OmniError::manifest_read_set_changed( @@ -519,7 +1105,9 @@ impl Omnigraph { } let prior_merged = exact_merged_generation(&final_details, key.shard_id)?; - let lineage = self.new_lineage_intent_for_branch(None, None).await?; + let lineage = self + .new_lineage_intent_for_branch(None, Some("omnigraph:stream-fold")) + .await?; let authority = RecoveryAuthorityToken { branch_identifier: prepared.txn.authority.branch_identifier.clone(), graph_head: prepared.txn.authority.graph_head.clone(), @@ -555,89 +1143,240 @@ impl Omnigraph { generation: cut.generation, generation_path: cut.path.clone(), }; - let sidecar = new_stream_fold_sidecar_v11( - None, - pin, - authority, - recovery_lineage, - prepared.binding.clone(), - prepared.lifecycle.current_head_witness.clone(), - prepared.epoch_floor, - prior_merged, - generation_cut, - planned_transaction, - )?; - let handle = write_sidecar(self.root_uri(), self.storage_adapter(), &sidecar).await?; - let outcome = match self.storage().commit_staged_exact(final_head, staged).await { - Ok(outcome) => outcome, - Err(error) => { - if !error.is_retryable_commit_conflict() { + Box::pin(async move { + let (token_planned_transaction, token_staged) = token_stage; + let token_head = SnapshotHandle::new( + open_stream_token_authority_head( + self.root_uri(), + live.stream_token_authority(), + &crate::lance_access::control_session(), + ) + .await?, + ); + let mut next_lifecycle = prepared.lifecycle.clone(); + next_lifecycle.current_head_witness = CurrentHeadWitness { + branch_identifier: lance::dataset::refs::BranchIdentifier::main(), + table_version: post_commit_pin, + transaction_uuid: planned_transaction.uuid.clone(), + manifest_e_tag: None, + }; + next_lifecycle + .epoch_floor_by_shard + .insert(key.shard_id.to_string(), cut.writer_epoch); + next_lifecycle.lifecycle_revision = next_lifecycle + .lifecycle_revision + .checked_add(1) + .ok_or_else(|| { + OmniError::manifest_internal("stream lifecycle revision overflow") + })?; + let (fold_rows, fold_bytes) = fold_output_size(&batches)?; + next_lifecycle.last_fold_summary = Some(LastFoldSummary { + operation_id: "pending-stream-fold-operation".to_string(), + graph_commit_id: Some(lineage.graph_commit_id.clone()), + exact_generation_cut: StreamGenerationCut { + shard_id: key.shard_id.to_string(), + writer_epoch: cut.writer_epoch, + shard_manifest_version: cut.shard_manifest_version, + replay_after_wal_entry_position: cut.replay_after_wal_entry_position, + generation: cut.generation, + generation_path: cut.path.clone(), + }, + outcome: LastFoldOutcome::Published, + input_rows: fold_rows, + input_bytes: fold_bytes, + visible_rows: fold_rows, + visible_bytes: fold_bytes, + recorded_at: lineage.created_at, + }); + let mut sidecar = new_stream_fold_sidecar_v12( + Some("omnigraph:stream-fold".to_string()), + pin, + authority, + recovery_lineage, + prepared.binding.clone(), + prepared.lifecycle.clone(), + next_lifecycle, + prior_merged, + generation_cut, + planned_transaction, + prepared.txn.base.stream_token_authority().clone(), + token_planned_transaction, + attribution.token_rows.clone(), + attribution.summary.clone(), + )?; + let handle = write_sidecar(self.root_uri(), self.storage_adapter(), &sidecar).await?; + + let base_outcome = match self.storage().commit_staged_exact(final_head, staged).await { + Ok(outcome) => outcome, + Err(error) => { + if error.is_retryable_commit_conflict() { + let effect_free = finalize_effect_free_stream_fold_sidecar_v12( + self.root_uri(), + &self.storage, + &live, + &sidecar, + ) + .await + .map_err(|classification_error| { + OmniError::recovery_required( + handle.operation_id.clone(), + format!( + "stream fold base commit failed ({error}); exact two-participant effect-free classification failed: {classification_error}" + ), + ) + })?; + if effect_free { + return Ok(FoldAttempt::EffectFree(error)); + } + } return Err(OmniError::recovery_required( handle.operation_id, - format!("stream fold commit failed with an ambiguous effect: {error}"), + format!("stream fold base commit requires recovery: {error}"), )); } - let effect_free = finalize_effect_free_stream_fold_sidecar_v11( - self.root_uri(), - &self.storage, - &live, - &sidecar, + }; + if !base_outcome.is_exact() { + return Err(OmniError::recovery_required( + handle.operation_id, + "stream fold base participant committed a non-exact transaction", + )); + } + let base_state = self + .storage() + .table_state(&prepared.full_path, base_outcome.snapshot()) + .await + .map_err(|error| { + OmniError::recovery_required(handle.operation_id.clone(), error.to_string()) + })?; + // Build the fixed manifest metadata from the coordinator's + // canonical root/table pair. TableStore may retain the caller's + // symlinked local root (`/var` vs `/private/var` on macOS), which + // would make the confirmation differ after recovery reopens the + // same manifest through the canonical root. + let base_version_metadata = crate::db::manifest::TableVersionMetadata::from_dataset( + self.root_uri(), + &prepared.entry.table_path, + base_outcome.snapshot().dataset(), + ) + .map_err(|error| { + OmniError::recovery_required(handle.operation_id.clone(), error.to_string()) + })?; + + crate::failpoints::maybe_fail( + crate::failpoints::names::STREAM_FOLD_POST_BASE_COMMIT_PRE_TOKEN_COMMIT, + ) + .map_err(|error| { + OmniError::recovery_required( + handle.operation_id.clone(), + format!("stream fold stopped after its exact base effect: {error}"), ) + })?; + + let token_outcome = match self + .storage() + .commit_staged_exact(token_head, token_staged) .await - .map_err(|classification_error| { - OmniError::recovery_required( - handle.operation_id.clone(), + { + Ok(outcome) => outcome, + Err(error) => { + let recovered = complete_stream_fold_sidecar_v12( + self.root_uri(), + Arc::clone(&self.storage), + &live, + &sidecar, + ) + .await; + if recovered.is_ok() { + self.refresh_coordinator_only().await?; + return Ok(FoldAttempt::Published); + } + return Err(OmniError::recovery_required( + handle.operation_id, format!( - "stream fold commit failed ({error}); exact effect-free classification failed: {classification_error}" + "stream fold token commit failed ({error}) and synchronous recovery did not complete: {}", + recovered.expect_err("checked as error") ), - ) - })?; - if effect_free { - return Ok(FoldAttempt::EffectFree(error)); + )); } + }; + if !token_outcome.is_exact() { return Err(OmniError::recovery_required( handle.operation_id, - format!("stream fold commit failed with an unclassified effect: {error}"), + "stream fold token participant committed a non-exact transaction", )); } - }; - if !outcome.is_exact() { - return Err(OmniError::recovery_required( - handle.operation_id, - format!( - "stream fold committed a non-exact transaction: planned={:?}, committed={:?}, version={}", - outcome.planned_transaction(), - outcome.committed_transaction(), - outcome.committed_version(), - ), - )); - } - crate::failpoints::maybe_fail( - crate::failpoints::names::STREAM_FOLD_POST_TABLE_COMMIT_PRE_CONFIRM, - ) - .map_err(|error| { - OmniError::recovery_required( - handle.operation_id.clone(), - format!("stream fold stopped after its exact table effect: {error}"), + crate::failpoints::maybe_fail( + crate::failpoints::names::STREAM_FOLD_POST_TOKEN_COMMIT_PRE_CONFIRM, ) - })?; - complete_stream_fold_sidecar_v11( - self.root_uri(), - Arc::clone(&self.storage), - &live, - &sidecar, - ) - .await - .map_err(|error| { - OmniError::recovery_required( - handle.operation_id, - format!("stream fold completion requires recovery: {error}"), + .map_err(|error| { + OmniError::recovery_required( + handle.operation_id.clone(), + format!("stream fold stopped after both exact effects: {error}"), + ) + })?; + + let achieved_base_head = + capture_current_head_witness(base_outcome.snapshot().dataset()) + .await + .map_err(|error| { + OmniError::recovery_required( + handle.operation_id.clone(), + error.to_string(), + ) + })?; + let next_token_authority = stream_token_authority_entry_for_dataset( + token_outcome.snapshot().dataset(), ) - })?; - self.refresh_coordinator_only().await?; - Ok(FoldAttempt::Published) + .await + .map_err(|error| { + OmniError::recovery_required(handle.operation_id.clone(), error.to_string()) + })?; + let achieved_token_head = next_token_authority.current_head_witness.clone(); + confirm_stream_fold_sidecar_v12( + self.root_uri(), + self.storage_adapter(), + &mut sidecar, + base_outcome.committed_transaction().clone(), + MergedGeneration::new(key.shard_id, cut.generation), + achieved_base_head, + crate::db::SubTableUpdate { + identity: key.identity, + table_key: table_key.to_string(), + table_version: base_state.version, + table_branch: None, + row_count: base_state.row_count, + version_metadata: base_version_metadata, + }, + token_outcome.committed_transaction().clone(), + achieved_token_head, + next_token_authority, + ) + .await + .map_err(|error| { + OmniError::recovery_required( + handle.operation_id.clone(), + format!("stream fold confirmation requires recovery: {error}"), + ) + })?; + complete_stream_fold_sidecar_v12( + self.root_uri(), + Arc::clone(&self.storage), + &live, + &sidecar, + ) + .await + .map_err(|error| { + OmniError::recovery_required( + handle.operation_id, + format!("stream fold completion requires recovery: {error}"), + ) + })?; + self.refresh_coordinator_only().await?; + Ok(FoldAttempt::Published) + }) + .await } async fn capture_stream_authority( @@ -718,7 +1457,7 @@ impl Omnigraph { })?; let binding = lifecycle.binding.clone(); let (enrollment_id, shard_id) = - validate_stream_config_v2_binding(&details, &binding).map_err(worker_error)?; + validate_stream_config_v3_binding(&details, &binding).map_err(worker_error)?; let worker_key = StreamWorkerKey::new(entry.identity, enrollment_id, shard_id).map_err(worker_error)?; let epoch_floor = lifecycle @@ -752,7 +1491,7 @@ impl Omnigraph { capture: &StreamAuthorityCapture, batch: &RecordBatch, ) -> Result<()> { - validate_stream_input_bounds(&capture.entry.table_key, batch)?; + validate_stream_stored_bounds(&capture.entry.table_key, batch)?; let expected: ArrowSchema = capture.head.dataset().schema().into(); if batch.schema().as_ref() != &expected { return Err(OmniError::manifest(format!( @@ -790,6 +1529,59 @@ impl Omnigraph { Ok(()) } + fn validate_stream_logical_admission_batch( + &self, + capture: &StreamAuthorityCapture, + batch: &RecordBatch, + ) -> Result<()> { + validate_stream_input_bounds(&capture.entry.table_key, batch)?; + let expected_physical: ArrowSchema = capture.head.dataset().schema().into(); + let expected_fields = expected_physical + .fields() + .iter() + .filter(|field| field.name() != crate::db::STREAM_METADATA_COLUMN) + .map(|field| field.as_ref().clone()) + .collect::>(); + let expected = ArrowSchema::new_with_metadata( + expected_fields, + expected_physical.metadata().clone(), + ); + if batch.schema().as_ref() != &expected { + return Err(OmniError::manifest(format!( + "stream batch schema for '{}' does not exactly match its accepted logical schema", + capture.entry.table_key + ))); + } + for (field, column) in batch.schema().fields().iter().zip(batch.columns()) { + if !field.is_nullable() && column.null_count() != 0 { + return Err(OmniError::manifest(format!( + "stream batch has nulls in required field '{}'", + field.name() + ))); + } + } + self.storage() + .validate_keyed_write_batch(&capture.entry.table_key, batch)?; + + let mut changeset = ChangeSet::new(); + changeset.insert( + capture.entry.table_key.clone(), + TableChange { + added: vec![batch.clone()], + changed: Vec::new(), + deleted_ids: Vec::new(), + }, + ); + if let Some(violation) = + crate::validate::evaluate_value_constraints(&changeset, &capture.txn.catalog) + .into_iter() + .next() + { + return Err(violation.into_omni_error()); + } + Ok(()) + } + async fn ensure_no_relevant_stream_sidecar( &self, identity: TableIdentity, @@ -797,7 +1589,7 @@ impl Omnigraph { ) -> Result<()> { let sidecars = list_sidecars(self.root_uri(), self.storage_adapter()).await?; if let Some(sidecar) = sidecars.iter().find(|sidecar| { - sidecar.writer_kind == SidecarKind::SchemaApply + sidecar.writer_kind.is_graph_global_barrier() || sidecar.tables.iter().any(|pin| pin.identity == identity) }) { return Err(OmniError::recovery_required( @@ -845,6 +1637,60 @@ impl Omnigraph { None => self.stream_fold_phase_b1(table_key).await, } } + + /// Feature-gated proof seam for one private B2 compare-and-chain row. + /// It intentionally accepts/returns only wire strings so protocol types do + /// not become public SDK surface. + #[cfg(feature = "failpoints")] + #[doc(hidden)] + #[allow(clippy::too_many_arguments)] + pub async fn failpoint_stream_b2_for_test( + self: &Arc, + table_key: &str, + batch: RecordBatch, + caller_ordinal: u64, + stream_incarnation_id: &str, + write_id: &str, + predecessor_token: Option<&str>, + contributor_id: &str, + ) -> Result { + let predecessor_token = predecessor_token + .map(str::parse::) + .transpose() + .map_err(|error| OmniError::manifest(error.to_string()))?; + let contributor_id = TrustedContributorId::new(contributor_id.to_string()) + .map_err(|error| OmniError::manifest(error.to_string()))?; + let ack = self + .stream_put_phase_b2_one( + table_key, + batch, + caller_ordinal, + StreamWriteEnvelope { + stream_incarnation_id: stream_incarnation_id.to_string(), + write_id: write_id.to_string(), + predecessor_token, + }, + contributor_id, + ) + .await?; + Ok(ack.stream_token.to_string()) + } + + /// Return the exact logical stream incarnation for private protocol tests. + #[cfg(feature = "failpoints")] + #[doc(hidden)] + pub async fn failpoint_stream_incarnation_for_test( + &self, + table_key: &str, + ) -> Result { + let capture = self + .capture_stream_authority(table_key, "stream incarnation test probe") + .await?; + Ok(capture + .lifecycle + .enrollment_receipt + .stream_incarnation_id) + } } enum FoldAttempt { @@ -852,11 +1698,165 @@ enum FoldAttempt { EffectFree(OmniError), } +async fn lookup_base_stream_metadata( + dataset: &lance::Dataset, + identity: TableIdentity, + logical_id: &str, +) -> Result> { + let logical_ids = std::collections::BTreeSet::from([logical_id.to_string()]); + let mut selected = lookup_base_stream_metadata_for_keys(dataset, identity, &logical_ids).await?; + Ok(selected.remove(logical_id)) +} + +async fn lookup_base_stream_metadata_for_keys( + dataset: &lance::Dataset, + identity: TableIdentity, + logical_ids: &std::collections::BTreeSet, +) -> Result> { + if logical_ids.is_empty() || logical_ids.len() > B1_MAX_GENERATION_ROWS as usize { + return Err(OmniError::manifest_internal(format!( + "base stream-metadata lookup requires 1..={B1_MAX_GENERATION_ROWS} exact keys, got {}", + logical_ids.len() + ))); + } + let mut scanner = dataset.scan(); + scanner + .project(&["id", crate::db::STREAM_METADATA_COLUMN]) + .map_err(|error| OmniError::Lance(error.to_string()))?; + scanner.filter_expr(col("id").in_list(logical_ids.iter().cloned().map(lit).collect(), false)); + scanner.batch_size(logical_ids.len().saturating_add(1)); + scanner.batch_size_bytes(B2_MAX_TOKEN_PROJECTION_ARROW_BYTES); + scanner + .limit( + Some(i64::try_from(logical_ids.len().saturating_add(1)).map_err(|_| { + OmniError::manifest_internal("base stream-metadata lookup row limit exceeds i64") + })?), + None, + ) + .map_err(|error| OmniError::Lance(error.to_string()))?; + let mut stream = scanner + .try_into_stream() + .await + .map_err(|error| OmniError::Lance(error.to_string()))?; + let mut seen = std::collections::BTreeSet::new(); + let mut selected = std::collections::BTreeMap::new(); + let mut retained_bytes = 0_u64; + let mut observed_rows = 0_usize; + while let Some(batch) = stream + .try_next() + .await + .map_err(|error| OmniError::Lance(error.to_string()))? + { + observed_rows = observed_rows + .checked_add(batch.num_rows()) + .ok_or_else(|| OmniError::manifest_internal("base stream lookup row overflow"))?; + if observed_rows > logical_ids.len() { + return Err(OmniError::manifest_internal( + "base stream-metadata lookup returned more than one row per requested key", + )); + } + let batch_bytes = u64::try_from(batch.get_array_memory_size()).map_err(|_| { + OmniError::manifest_internal("base stream lookup batch Arrow size exceeds u64") + })?; + if batch_bytes > B2_MAX_TOKEN_PROJECTION_ARROW_BYTES { + return Err(OmniError::resource_limit( + "base_stream_lookup_batch_arrow_bytes", + B2_MAX_TOKEN_PROJECTION_ARROW_BYTES, + batch_bytes, + )); + } + let ids = batch + .column_by_name("id") + .and_then(|array| array.as_any().downcast_ref::()) + .ok_or_else(|| { + OmniError::manifest_internal( + "base stream-metadata probe returned no exact Utf8 id column", + ) + })?; + let metadata = batch + .column_by_name(crate::db::STREAM_METADATA_COLUMN) + .ok_or_else(|| { + OmniError::manifest_internal(format!( + "base stream-metadata probe omitted reserved column '{}'", + crate::db::STREAM_METADATA_COLUMN + )) + })?; + for row in 0..batch.num_rows() { + if ids.is_null(row) || !logical_ids.contains(ids.value(row)) { + return Err(OmniError::manifest_internal( + "base stream-metadata exact-id probe returned a foreign row", + )); + } + let logical_id = ids.value(row); + if !seen.insert(logical_id.to_string()) { + return Err(OmniError::manifest_internal(format!( + "base table contains duplicate exact-id rows for '{logical_id}'" + ))); + } + retained_bytes = add_stream_lookup_retained_bytes( + "base_stream_lookup_retained_bytes", + retained_bytes, + u64::try_from( + std::mem::size_of::() + .saturating_add(logical_id.len()) + .saturating_add(256), + ) + .map_err(|_| OmniError::manifest_internal("base stream key bytes exceed u64"))?, + B2_MAX_TOKEN_PROJECTION_ARROW_BYTES, + )?; + let decoded = decode_trusted_stream_metadata(metadata.as_ref(), row) + .map_err(|error| OmniError::manifest_internal(error.to_string()))?; + if let Some(decoded) = &decoded { + decoded + .validate_for(identity, logical_id) + .map_err(|error| OmniError::manifest_internal(error.to_string()))?; + } + if let Some(decoded) = decoded { + retained_bytes = add_stream_lookup_retained_bytes( + "base_stream_lookup_retained_bytes", + retained_bytes, + decoded + .lookup_retained_bytes(logical_id) + .map_err(|error| OmniError::manifest_internal(error.to_string()))?, + B2_MAX_TOKEN_PROJECTION_ARROW_BYTES, + )?; + selected.insert(logical_id.to_string(), decoded); + } + } + } + Ok(selected) +} + fn worker_error(error: MemWalWorkerError) -> OmniError { OmniError::Lance(error.to_string()) } fn validate_stream_input_bounds(table_key: &str, batch: &RecordBatch) -> Result<()> { + if batch.num_rows() == 0 { + return Err(OmniError::manifest("stream batch must be non-empty")); + } + if batch + .column_by_name(lance::dataset::mem_wal::TOMBSTONE) + .is_some() + { + return Err(OmniError::manifest(format!( + "stream batch may not supply reserved column '{}'", + lance::dataset::mem_wal::TOMBSTONE + ))); + } + if batch + .column_by_name(crate::db::STREAM_METADATA_COLUMN) + .is_some() + { + return Err(OmniError::manifest(format!( + "stream caller may not supply reserved column '{}'", + crate::db::STREAM_METADATA_COLUMN + ))); + } + validate_stream_stored_bounds(table_key, batch) +} + +fn validate_stream_stored_bounds(table_key: &str, batch: &RecordBatch) -> Result<()> { if batch.num_rows() == 0 { return Err(OmniError::manifest("stream batch must be non-empty")); } @@ -871,6 +1871,13 @@ fn validate_stream_input_bounds(table_key: &str, batch: &RecordBatch) -> Result< } let charge = b1_input_accounting(batch).map_err(worker_error)?; if !charge.fits() { + if batch.num_rows() == 1 { + return Err(OmniError::resource_limit( + "stream_input_arrow_bytes", + B1_MAX_GENERATION_ARROW_BYTES, + charge.arrow_bytes, + )); + } return Err(OmniError::FoldRequired { table_key: table_key.to_string(), rows: charge.rows, @@ -880,6 +1887,61 @@ fn validate_stream_input_bounds(table_key: &str, batch: &RecordBatch) -> Result< Ok(()) } +fn validate_generation_token_plan( + table_key: &str, + rows: &[StreamTokenAuthorityRow], +) -> Result<()> { + match validate_stream_token_plan_bounds(rows) { + Ok(()) => Ok(()), + Err(OmniError::ResourceLimitExceeded { actual, .. }) => { + Err(OmniError::FoldRequired { + table_key: table_key.to_string(), + rows: u64::try_from(rows.len()).unwrap_or(u64::MAX), + bytes: actual, + }) + } + Err(error) => Err(error), + } +} + +fn append_trusted_stream_metadata( + batch: RecordBatch, + metadata: Vec>, +) -> Result { + if metadata.len() != batch.num_rows() { + return Err(OmniError::manifest_internal(format!( + "trusted stream metadata row count {} differs from batch row count {}", + metadata.len(), + batch.num_rows() + ))); + } + if batch + .column_by_name(crate::db::STREAM_METADATA_COLUMN) + .is_some() + { + return Err(OmniError::manifest(format!( + "stream caller may not supply reserved column '{}'", + crate::db::STREAM_METADATA_COLUMN + ))); + } + let hidden = build_trusted_stream_metadata_array(&metadata) + .map_err(|error| OmniError::manifest_internal(error.to_string()))?; + let source_schema = batch.schema(); + let mut fields = source_schema + .fields() + .iter() + .map(|field| field.as_ref().clone()) + .collect::>(); + fields.push(crate::db::manifest::stream_token::trusted_stream_metadata_field()); + let schema = Arc::new(ArrowSchema::new_with_metadata( + fields, + source_schema.metadata().clone(), + )); + let mut columns = batch.columns().to_vec(); + columns.push(hidden); + RecordBatch::try_new(schema, columns).map_err(|error| OmniError::Lance(error.to_string())) +} + /// Once the exclusive fold worker begins a cold claim or cut, every /// non-capacity failure requires the retained worker to settle and durable /// state to be reclassified before another attempt. There is deliberately no @@ -1057,19 +2119,11 @@ async fn scan_fresh_generation( rows, )); } - for column in batch.columns() { - let bytes = column - .to_data() - .get_slice_memory_size() - .map_err(|error| OmniError::Lance(error.to_string()))?; - logical_bytes = logical_bytes - .checked_add(u64::try_from(bytes).map_err(|_| { - OmniError::manifest_internal("stream fold logical bytes exceed u64") - })?) - .ok_or_else(|| { - OmniError::manifest_internal("stream fold logical byte count overflow") - })?; - } + logical_bytes = logical_bytes + .checked_add(b1_logical_batch_bytes(&batch).map_err(worker_error)?) + .ok_or_else(|| { + OmniError::manifest_internal("stream fold logical byte count overflow") + })?; if logical_bytes > B1_MAX_GENERATION_ARROW_BYTES { return Err(OmniError::resource_limit( format!("stream fold bytes for {}", capture.entry.table_key), @@ -1107,24 +2161,148 @@ async fn scan_fresh_generation( Ok(batches) } -fn validate_fold_output_bounds(table_key: &str, batches: &[RecordBatch]) -> Result<()> { - let mut rows = 0_u64; - let mut bytes = 0_u64; +async fn plan_fold_attribution( + snapshot: &crate::db::manifest::Snapshot, + identity: TableIdentity, + lifecycle: &StreamLifecycleEntry, + binding: &StreamPhysicalBinding, + batches: &[RecordBatch], +) -> Result { + let mut winners = Vec::new(); + let mut saw_null = false; + let mut saw_present = false; + let mut logical_ids = std::collections::BTreeSet::new(); + for batch in batches { - rows = rows - .checked_add( - u64::try_from(batch.num_rows()).map_err(|_| { - OmniError::manifest_internal("stream fold row count exceeds u64") - })?, + let ids = batch + .column_by_name("id") + .and_then(|array| array.as_any().downcast_ref::()) + .ok_or_else(|| { + OmniError::manifest_internal( + "stream fold output has no exact non-null Utf8 id column", + ) + })?; + let metadata = batch + .column_by_name(crate::db::STREAM_METADATA_COLUMN) + .ok_or_else(|| { + OmniError::manifest_internal(format!( + "stream fold output is missing reserved '{}' metadata", + crate::db::STREAM_METADATA_COLUMN + )) + })?; + for row in 0..batch.num_rows() { + if ids.is_null(row) { + return Err(OmniError::manifest_internal( + "stream fold output contains a null logical id", + )); + } + let logical_id = ids.value(row).to_string(); + if !logical_ids.insert(logical_id.clone()) { + return Err(OmniError::manifest_internal(format!( + "stream fold scanner returned duplicate winner id '{logical_id}'" + ))); + } + match decode_trusted_stream_metadata(metadata.as_ref(), row) + .map_err(|error| OmniError::manifest_internal(error.to_string()))? + { + Some(metadata) => { + saw_present = true; + metadata + .validate_for(identity, &logical_id) + .map_err(|error| OmniError::manifest_internal(error.to_string()))?; + winners.push((logical_id, metadata)); + } + None => saw_null = true, + } + } + } + + if saw_null { + return Err(OmniError::manifest_internal(if saw_present { + "stream fold generation mixes attributed and unattributed winners" + } else { + "internal schema v9 refuses an unattributed stream generation" + })); + } + if !saw_present { + return Err(OmniError::manifest_internal( + "stream fold has no attribution state for its non-empty generation", + )); + } + + let stream_incarnation_id = &lifecycle.enrollment_receipt.stream_incarnation_id; + let token_dataset = snapshot.open_stream_token_authority().await?; + let current = stream_token_rows_for_keys( + &token_dataset, + snapshot.stream_token_authority(), + identity, + &logical_ids, + ) + .await?; + let base_entry = snapshot + .entries() + .find(|entry| entry.identity == identity) + .ok_or_else(|| { + OmniError::manifest_internal(format!( + "stream fold cannot find manifest-selected base table for identity {identity}" + )) + })?; + let base_dataset = snapshot.open_dataset(&base_entry.table_key).await?; + let base_metadata = + lookup_base_stream_metadata_for_keys(&base_dataset, identity, &logical_ids).await?; + let mut token_rows = Vec::with_capacity(winners.len()); + for (logical_id, metadata) in winners { + if &metadata.stream_incarnation_id != stream_incarnation_id { + return Err(OmniError::manifest_internal(format!( + "stream fold winner '{logical_id}' names stream incarnation '{}' but lifecycle authority names '{stream_incarnation_id}'", + metadata.stream_incarnation_id + ))); + } + let current_row = current.get(&logical_id); + validate_authority_base_pair( + identity, + &logical_id, + current_row, + base_metadata.get(&logical_id), + ) + .map_err(|error| OmniError::manifest_internal(error.to_string()))?; + if current_row.is_some_and(|row| row.stream_incarnation_id != *stream_incarnation_id) { + return Err(OmniError::manifest_internal(format!( + "stream token authority for '{logical_id}' belongs to another stream incarnation" + ))); + } + let expected_fold_base = current_row.map(|row| row.current_token); + if metadata.fold_base_token != expected_fold_base { + return Err(OmniError::manifest_internal(format!( + "stream fold winner '{logical_id}' does not chain from the manifest-selected token authority" + ))); + } + if current_row.is_some_and(|row| row.current_token == metadata.stream_token) { + return Err(OmniError::manifest_internal(format!( + "stream fold winner '{logical_id}' does not advance its current token" + ))); + } + token_rows.push( + StreamTokenAuthorityRow::from_present_metadata( + identity, + logical_id, + binding.enrollment_id.clone(), + &metadata, ) - .ok_or_else(|| OmniError::manifest_internal("stream fold row count overflow"))?; - bytes = - bytes - .checked_add(u64::try_from(batch.get_array_memory_size()).map_err(|_| { - OmniError::manifest_internal("stream fold byte count exceeds u64") - })?) - .ok_or_else(|| OmniError::manifest_internal("stream fold byte count overflow"))?; + .map_err(|error| OmniError::manifest_internal(error.to_string()))?, + ); } + token_rows.sort_by(|left, right| left.logical_id.cmp(&right.logical_id)); + let summary = stream_fold_attribution_commitment(&token_rows) + .map_err(|error| OmniError::manifest_internal(error.to_string()))?; + Ok(AttributedFoldPlan { + token_rows, + summary, + }) +} + +fn validate_fold_output_bounds(table_key: &str, batches: &[RecordBatch]) -> Result<()> { + let (rows, bytes) = fold_output_size(batches)?; if rows == 0 { return Err(OmniError::manifest_internal( "stream fold cannot stage an empty generation", @@ -1147,6 +2325,24 @@ fn validate_fold_output_bounds(table_key: &str, batches: &[RecordBatch]) -> Resu Ok(()) } +fn fold_output_size(batches: &[RecordBatch]) -> Result<(u64, u64)> { + let mut rows = 0_u64; + let mut bytes = 0_u64; + for batch in batches { + rows = rows + .checked_add( + u64::try_from(batch.num_rows()).map_err(|_| { + OmniError::manifest_internal("stream fold row count exceeds u64") + })?, + ) + .ok_or_else(|| OmniError::manifest_internal("stream fold row count overflow"))?; + bytes = bytes + .checked_add(b1_logical_batch_bytes(batch).map_err(worker_error)?) + .ok_or_else(|| OmniError::manifest_internal("stream fold byte count overflow"))?; + } + Ok((rows, bytes)) +} + fn exact_merged_generation( details: &MemWalIndexDetails, shard_id: ShardId, diff --git a/crates/omnigraph/src/db/omnigraph/table_ops.rs b/crates/omnigraph/src/db/omnigraph/table_ops.rs index 0a06a779..fb45cd83 100644 --- a/crates/omnigraph/src/db/omnigraph/table_ops.rs +++ b/crates/omnigraph/src/db/omnigraph/table_ops.rs @@ -273,6 +273,7 @@ pub(super) async fn ensure_indices_for_branch( .write_queue() .acquire_branch(active_branch.as_deref()) .await; + let _stream_token_guard = db.write_queue().acquire_stream_token().await; let _queue_guards = db.write_queue().acquire_many(&queue_keys).await; db.ensure_no_pending_recovery_sidecars_under_gates( diff --git a/crates/omnigraph/src/db/write_queue.rs b/crates/omnigraph/src/db/write_queue.rs index f71b9cce..ce54dcfb 100644 --- a/crates/omnigraph/src/db/write_queue.rs +++ b/crates/omnigraph/src/db/write_queue.rs @@ -24,7 +24,9 @@ //! //! The admission lease is the **outermost** process-local gate. A caller that //! composes it with existing gates acquires admission key(s) first, then keeps -//! the established schema -> branch -> sorted-table order. This prevents an +//! the established schema -> branch -> token-authority -> sorted-table order. +//! The token-authority gate is graph-global because every B2 fold advances the +//! same manifest-selected `_stream_tokens.lance` participant. This prevents an //! append (which needs only a shared admission lease) from deadlocking with a //! drain or enrollment that also needs the existing gates. These locks are //! neither durable lifecycle authority nor distributed fencing. Callers must @@ -121,6 +123,15 @@ pub(crate) struct WriteQueueManager { /// publication; explicit authority/physical exceptions follow their own /// registered ordering contracts. branch_queues: Mutex, Arc>>>, + /// Graph-global RFC-026 sequencing-participant gate. + /// + /// This is intentionally a singleton per root rather than a synthetic + /// table queue: `_stream_tokens.lance` is graph protocol authority, not a + /// user table and not a member of the stable table-identity domain. A B2 + /// fold acquires it after the main-branch gate and before any graph-table + /// gate, then holds it through exact participant confirmation and the one + /// manifest visibility CAS. + stream_token_gate: Arc>, /// RFC-026 final-check-through-effect admission domains. /// /// Tokio's fair, write-preferring `RwLock` lets ordinary writers/appends @@ -258,6 +269,17 @@ impl WriteQueueManager { self.branch_slot(&key).lock_owned().await } + /// Acquire the graph-global RFC-026 token-participant gate. + /// + /// Callers compose this only in the canonical order: sorted relevant + /// stream admission -> schema -> main branch -> token -> sorted graph + /// tables. The mutex is process-local contention control; the exact + /// manifest-selected token witness and recovery-v12 remain durable + /// authority. + pub(crate) async fn acquire_stream_token(&self) -> OwnedMutexGuard<()> { + Arc::clone(&self.stream_token_gate).lock_owned().await + } + /// Acquire several graph-branch control gates in one deterministic order. /// /// Native branch create-from reads a source ref and mutates a target ref, so @@ -510,6 +532,36 @@ mod tests { renamed_task.await.unwrap(); } + #[tokio::test] + async fn separately_opened_handles_share_one_graph_token_gate() { + let root = format!("memory://stream-token-gate/{}", ulid::Ulid::new()); + let first = WriteQueueManager::for_root(&root); + let second = WriteQueueManager::for_root(&root); + + let held = first.acquire_stream_token().await; + let (started_tx, started_rx) = oneshot::channel(); + let (acquired_tx, mut acquired_rx) = oneshot::channel(); + let task = tokio::spawn(async move { + started_tx.send(()).unwrap(); + let _guard = second.acquire_stream_token().await; + acquired_tx.send(()).unwrap(); + }); + started_rx.await.unwrap(); + assert!( + timeout(Duration::from_millis(50), &mut acquired_rx) + .await + .is_err(), + "the token participant must serialize across independently opened handles" + ); + + drop(held); + timeout(Duration::from_secs(2), &mut acquired_rx) + .await + .expect("token gate did not acquire after release") + .expect("token gate task exited before acquisition"); + task.await.unwrap(); + } + #[tokio::test] async fn acquire_many_empty_returns_empty() { let qm = WriteQueueManager::new(); diff --git a/crates/omnigraph/src/error.rs b/crates/omnigraph/src/error.rs index 38046b11..c7fdcff3 100644 --- a/crates/omnigraph/src/error.rs +++ b/crates/omnigraph/src/error.rs @@ -145,6 +145,38 @@ pub enum OmniError { rows: u64, bytes: u64, }, + /// RFC-026 compare-and-chain request names a stream incarnation which is + /// no longer current. This is proven before Lance is invoked. + #[error( + "stream binding changed for table {stable_table_id:016x}:{table_incarnation_id:016x}: current stream incarnation is {current_stream_incarnation_id}" + )] + StreamBindingChanged { + stable_table_id: u64, + table_incarnation_id: u64, + current_stream_incarnation_id: String, + }, + /// RFC-026 compare-and-chain predecessor differs from the complete current + /// token. `None` means this key has no current stream token. + #[error( + "stream sequence conflict for table {stable_table_id:016x}:{table_incarnation_id:016x}, key '{logical_id}' (current token: {current_token:?})" + )] + StreamSequenceConflict { + stable_table_id: u64, + table_incarnation_id: u64, + logical_id: String, + current_token: Option, + }, + /// The same occurrence key was reused with a different trusted actor or + /// normalized payload. It can never be reinterpreted as a new change. + #[error( + "stream idempotency conflict for table {stable_table_id:016x}:{table_incarnation_id:016x}, key '{logical_id}' (current token: {current_token})" + )] + StreamIdempotencyConflict { + stable_table_id: u64, + table_incarnation_id: u64, + logical_id: String, + current_token: String, + }, /// RFC-026 invoked Lance's MemWAL append but could not prove the complete /// acknowledgement boundary: watcher durability plus no observed successor /// writer epoch. The attempt may be durable and is intentionally never @@ -161,6 +193,14 @@ pub enum OmniError { shard_id: String, caller_ordinal_start: u64, caller_ordinal_end: u64, + /// B2-only server-minted physical invocation identity. B1's private + /// substrate seam leaves this absent. + admission_attempt_id: Option, + /// B2 logical idempotency labels covered by this physical invocation. + logical_write_ids: Vec, + /// Deterministic but explicitly unconfirmed candidate returned only as + /// retry correlation; it is never valid predecessor authority. + unconfirmed_candidate_token: Option, reason: String, }, /// A durable recovery intent or retained pre-sidecar physical owner diff --git a/crates/omnigraph/src/exec/merge.rs b/crates/omnigraph/src/exec/merge.rs index 0831969e..bd50aadd 100644 --- a/crates/omnigraph/src/exec/merge.rs +++ b/crates/omnigraph/src/exec/merge.rs @@ -3131,6 +3131,7 @@ impl Omnigraph { _stream_admission, _schema_guard, _branch_guards, + _stream_token_guard, source_txn, target_txn, source_commits, @@ -3144,6 +3145,7 @@ impl Omnigraph { let branches = write_queue .acquire_branches(&[source_branch.clone(), target_branch.clone()]) .await; + let token = write_queue.acquire_stream_token().await; self.ensure_no_pending_recovery_sidecars_under_gates( &relevant_branches, @@ -3168,6 +3170,7 @@ impl Omnigraph { admission, schema, branches, + token, source_txn, target_txn, source_commits, diff --git a/crates/omnigraph/src/exec/mutation.rs b/crates/omnigraph/src/exec/mutation.rs index e3706c5a..8a6be167 100644 --- a/crates/omnigraph/src/exec/mutation.rs +++ b/crates/omnigraph/src/exec/mutation.rs @@ -353,6 +353,11 @@ fn build_insert_batch( for field in schema.fields() { if field.name() == "id" { columns.push(Arc::new(StringArray::from(vec![id]))); + } else if field.name() == crate::db::STREAM_METADATA_COLUMN { + columns.push( + crate::db::manifest::stream_token::build_trusted_stream_metadata_array(&[None]) + .map_err(|error| OmniError::manifest_internal(error.to_string()))?, + ); } else if blob_properties.contains(field.name()) { if let Some(Literal::String(uri)) = assignments.get(field.name()) { columns.push(build_blob_array_from_value(uri)?); diff --git a/crates/omnigraph/src/exec/staging.rs b/crates/omnigraph/src/exec/staging.rs index 4d6a8815..03b1a3f7 100644 --- a/crates/omnigraph/src/exec/staging.rs +++ b/crates/omnigraph/src/exec/staging.rs @@ -918,13 +918,16 @@ async fn acquire_mutation_commit_guards( // avoids an admission <-> table-queue inversion between those shapes. let stream_admission = write_queue.acquire_stream_shared_many(admission_keys).await; - // Preserve RFC-022's established order inside the admission window: - // schema -> branch -> sorted physical table queues. + // Preserve RFC-022's established order inside the admission window, with + // RFC-026's graph-global sequencing participant between graph authority + // and physical table effects: + // schema -> branch -> stream token -> sorted physical table queues. let schema_guard = write_queue .acquire(&crate::db::manifest::schema_apply_serial_queue_key()) .await; let branch_guard = write_queue.acquire_branch(branch).await; - let mut write_queues = vec![schema_guard, branch_guard]; + let token_guard = write_queue.acquire_stream_token().await; + let mut write_queues = vec![schema_guard, branch_guard, token_guard]; write_queues.extend(write_queue.acquire_many(table_queue_keys).await); MutationCommitGuards { @@ -1084,7 +1087,7 @@ impl StagedMutation { crate::db::manifest::list_sidecars(db.root_uri(), db.storage_adapter()).await?; if let Some(owner) = pending_sidecars.iter().find(|sidecar| { let sidecar_branch = sidecar.branch.as_deref().filter(|name| *name != "main"); - sidecar.writer_kind == SidecarKind::SchemaApply || sidecar_branch == enrolled_branch + sidecar.writer_kind.is_graph_global_barrier() || sidecar_branch == enrolled_branch }) { return Err(OmniError::recovery_required( owner.operation_id.clone(), diff --git a/crates/omnigraph/src/failpoints.rs b/crates/omnigraph/src/failpoints.rs index 0b587622..64ffd367 100644 --- a/crates/omnigraph/src/failpoints.rs +++ b/crates/omnigraph/src/failpoints.rs @@ -202,6 +202,12 @@ pub mod names { /// it waits for the current input corridor owner to transfer its charge. pub const STREAM_B1_AFTER_SHARED_BEFORE_QUEUE_WAIT: &str = "stream_b1.after_shared_before_queue_wait"; + /// B2 has captured provisional stream authority but has not acquired the + /// shared admission lease. Tests publish a fold in this gap to prove that + /// token lookup and every effect-free classification use the later gated + /// authority capture rather than this stale snapshot. + pub const STREAM_B2_AFTER_PROVISIONAL_AUTHORITY: &str = + "stream_b2.after_provisional_authority"; /// A put owns its exact Arrow charge, shared admission, and the same-key /// input queue, but has not begun warm validation or a cold writer claim. /// Tests park a cold replay opener here so already-charged waiters exist @@ -237,12 +243,18 @@ pub mod names { /// witness for deterministic replay-open races. pub const STREAM_B1_AFTER_RETIREMENT_RELEASE: &str = "stream_b1.after_retirement_release"; /// The immutable generation cut is proven and the worker retired, before - /// the schema-v11 recovery intent is armed. + /// the schema-v12 recovery intent is armed. pub const STREAM_FOLD_POST_DRAIN_PRE_SIDECAR: &str = "stream_fold.post_drain_pre_sidecar"; - /// The exact Lance Update (rows plus merged-generation marker) committed, - /// before its achieved identity and output are durably confirmed. - pub const STREAM_FOLD_POST_TABLE_COMMIT_PRE_CONFIRM: &str = - "stream_fold.post_table_commit_pre_confirm"; + /// Schema-v12: the exact base-table fold committed, while the separate + /// `_stream_tokens.lance` participant is still at its manifest-selected + /// prior witness. Recovery must classify this partial cell as sticky. + pub const STREAM_FOLD_POST_BASE_COMMIT_PRE_TOKEN_COMMIT: &str = + "stream_fold.post_base_commit_pre_token_commit"; + /// Schema-v12: both exact Lance participants committed, while the Armed + /// sidecar has not yet been rewritten with their achieved witnesses. + /// Recovery may reconstruct the exact/exact confirmation and roll forward. + pub const STREAM_FOLD_POST_TOKEN_COMMIT_PRE_CONFIRM: &str = + "stream_fold.post_token_commit_pre_confirm"; /// Reload owns the schema gate and is about to read/publish one contract view. pub const SCHEMA_RELOAD_BEFORE_CONTRACT_READ: &str = "schema_reload.before_contract_read"; /// Injects a retryable `RowLevelCasContention` from `load_publish_state` so a diff --git a/crates/omnigraph/src/loader/mod.rs b/crates/omnigraph/src/loader/mod.rs index 8d556533..cc899c5c 100644 --- a/crates/omnigraph/src/loader/mod.rs +++ b/crates/omnigraph/src/loader/mod.rs @@ -825,7 +825,14 @@ fn build_node_batch( // U64 range), not an independent rendering of its input JSON token. let mut property_columns: Vec = Vec::with_capacity(schema.fields().len() - 1); for field in schema.fields().iter().skip(1) { - if node_type.blob_properties.contains(field.name()) { + if field.name() == crate::db::STREAM_METADATA_COLUMN { + property_columns.push( + crate::db::manifest::stream_token::build_trusted_stream_metadata_array( + &vec![None; rows.len()], + ) + .map_err(|error| OmniError::manifest_internal(error.to_string()))?, + ); + } else if node_type.blob_properties.contains(field.name()) { let col = build_blob_column(field.name(), field.is_nullable(), rows)?; property_columns.push(col); } else { @@ -978,7 +985,14 @@ fn build_edge_batch( // Build edge property columns (skip id, src, dst at indices 0-2) let data_values: Vec = rows.iter().map(|(_, _, data)| data.clone()).collect(); for field in schema.fields().iter().skip(3) { - if edge_type.blob_properties.contains(field.name()) { + if field.name() == crate::db::STREAM_METADATA_COLUMN { + columns.push( + crate::db::manifest::stream_token::build_trusted_stream_metadata_array( + &vec![None; rows.len()], + ) + .map_err(|error| OmniError::manifest_internal(error.to_string()))?, + ); + } else if edge_type.blob_properties.contains(field.name()) { let col = build_blob_column(field.name(), field.is_nullable(), &data_values)?; columns.push(col); } else { diff --git a/crates/omnigraph/src/table_store.rs b/crates/omnigraph/src/table_store.rs index ba3f05be..d76bd6d6 100644 --- a/crates/omnigraph/src/table_store.rs +++ b/crates/omnigraph/src/table_store.rs @@ -4568,6 +4568,46 @@ fn staged_keyed_merge_result( )) } +/// Validate and package one protocol-internal exact-`id` upsert staged through +/// Lance's filter-bearing MergeInsert path. +/// +/// RFC-026's `_stream_tokens.lance` adapter shares the same substrate proof as +/// graph keyed writes but remains a separate graph-global participant. Keeping +/// this narrow bridge here lets that adapter preserve Lance's affected-row +/// metadata without exposing `StagedWrite` internals or inventing another +/// commit path. +pub(crate) fn staged_exact_id_upsert_result( + dataset: &Dataset, + uncommitted: UncommittedMergeInsert, + expected_rows: u64, + context: &'static str, +) -> Result { + let id_field_id = exact_id_primary_key_field_id(dataset, context)?; + validate_exact_id_filter(&uncommitted, id_field_id, context)?; + let stats = &uncommitted.stats; + let affected_rows = stats + .num_inserted_rows + .checked_add(stats.num_updated_rows) + .ok_or_else(|| { + OmniError::manifest_internal(format!("{context}: affected-row count overflow")) + })?; + if affected_rows != expected_rows + || stats.num_deleted_rows != 0 + || stats.num_skipped_duplicates != 0 + || stats.num_attempts != 1 + { + return Err(OmniError::manifest_internal(format!( + "{context}: merge stats were inserted={}, updated={}, deleted={}, skipped={}, attempts={}; expected inserted+updated={expected_rows}, deleted=0, skipped=0, attempts=1", + stats.num_inserted_rows, + stats.num_updated_rows, + stats.num_deleted_rows, + stats.num_skipped_duplicates, + stats.num_attempts, + ))); + } + staged_keyed_merge_result(uncommitted, context) +} + /// Precondition guard for `stage_merge_insert`. /// Both opt into `SourceDedupeBehavior::FirstSeen` to suppress the Lance /// `processed_row_ids` bug (MR-957). FirstSeen would *also* silently diff --git a/crates/omnigraph/src/table_store/mem_wal.rs b/crates/omnigraph/src/table_store/mem_wal.rs index 0c4de142..2d3428df 100644 --- a/crates/omnigraph/src/table_store/mem_wal.rs +++ b/crates/omnigraph/src/table_store/mem_wal.rs @@ -1,12 +1,14 @@ -//! Narrow RFC-026 adapters for bounded MemWAL enrollment and private Phase B1. +//! Narrow RFC-026 adapters for bounded MemWAL enrollment and the private +//! B1/common-B2 core. //! //! The parent module captures the exact main-branch witness, initializes the //! singleton unsharded MemWAL index, provisions one pre-minted empty shard, and //! classifies those enrollment effects after a lost result. The private //! [`worker`] submodule owns B1's one-generation admission, watcher, //! post-durability successor-epoch check, replay, seal/drain, and quiesced -//! retirement mechanics. Graph authority, recovery-v11 fold ownership, and the -//! sole `__manifest` visibility publication remain in +//! retirement mechanics. Current schema v9 adds common-B2 token/attribution; +//! graph authority, recovery-v12 base-plus-token fold ownership, and the sole +//! `__manifest` visibility publication remain in //! `db::omnigraph::stream_ingest`; no production streaming API is exposed. //! //! The caller must hold RFC-026's exclusive base-HEAD and cleanup/GC gates from @@ -136,7 +138,7 @@ impl MemWalEnrollmentPlan { /// bounded profile. Enrollment and shard identities are deliberately not /// included: they are carried separately in the physical binding. pub(crate) fn stream_config_hash(&self) -> String { - stream_config_v2_hash() + stream_config_v3_hash() } fn expected_writer_defaults(&self) -> BTreeMap { @@ -1051,7 +1053,7 @@ mod tests { let plan = plan(); assert_eq!( plan.stream_config_hash(), - "sha256:1885f9b7d28ffc12266e75e3d6d0448c3289dee152674aadb8cca2be865d8e9d" + "sha256:5cc5116580e888cbf1b39e8f6be9c514a9a4b8eb6b8da804d05e40678d24f51a" ); let mut details = MemWalIndexDetails { num_shards: 1, diff --git a/crates/omnigraph/src/table_store/mem_wal/worker.rs b/crates/omnigraph/src/table_store/mem_wal/worker.rs index 36cb638e..0272e970 100644 --- a/crates/omnigraph/src/table_store/mem_wal/worker.rs +++ b/crates/omnigraph/src/table_store/mem_wal/worker.rs @@ -32,6 +32,11 @@ use sha2::{Digest, Sha256}; use thiserror::Error; use tokio::sync::{OwnedRwLockReadGuard, OwnedRwLockWriteGuard, watch}; +use crate::db::manifest::stream_token::{ + STREAM_PAYLOAD_DIGEST_VERSION, STREAM_PAYLOAD_ENCODING_VERSION, + STREAM_TOKEN_DERIVATION_VERSION, STREAM_TOKEN_WIRE_VERSION, StreamTokenAuthorityRow, + TrustedStreamRowMetadata, +}; use crate::db::manifest::{ STREAM_CONFIG_VERSION, StreamLifecycleEntry, StreamPhysicalBinding, TableIdentity, }; @@ -69,6 +74,26 @@ const ENABLE_MEMTABLE_KEY: &str = "enable_memtable"; pub(crate) const B1_MAX_GENERATION_ROWS: u64 = 8_192; pub(crate) const B1_MAX_GENERATION_ARROW_BYTES: u64 = 32 * 1024 * 1024; pub(crate) const B1_MAX_GENERATION_BATCHES: usize = 8_192; +/// Maximum deterministic payload bytes hashed for one normalized B2 row. +/// This is intentionally larger than the Arrow generation cap because JSON +/// escaping/base64 can expand an otherwise legal physical value. +pub(crate) const B2_MAX_CANONICAL_PAYLOAD_BYTES: u64 = 64 * 1024 * 1024; +/// Conservative root reservation held while one B2 row is normalized and +/// canonically encoded: the caller Arrow batch, a possible fully materialized +/// replacement batch, and the maximum canonical payload can coexist briefly. +pub(crate) const B2_PREPROCESSING_RESERVATION_BYTES: u64 = + 2 * B1_MAX_GENERATION_ARROW_BYTES + B2_MAX_CANONICAL_PAYLOAD_BYTES; +/// The private profile preserves the minimum two-caller overlap needed to +/// revalidate a provisional authority capture while keeping preprocessing +/// memory finite root-wide. +pub(crate) const B2_MAX_PREPROCESSING_CALLS_ROOT: u64 = 2; +pub(crate) const B2_MAX_PREPROCESSING_BYTES_ROOT: u64 = + B2_MAX_PREPROCESSING_CALLS_ROOT * B2_PREPROCESSING_RESERVATION_BYTES; +/// Maximum Arrow memory occupied by the exact current-token winner projection +/// for one generation. +pub(crate) const B2_MAX_TOKEN_PROJECTION_ARROW_BYTES: u64 = 32 * 1024 * 1024; +/// Maximum canonical JSON bytes occupied by recovery-v12's planned token rows. +pub(crate) const B2_MAX_TOKEN_RECOVERY_JSON_BYTES: u64 = 32 * 1024 * 1024; const B1_WAL_BUFFER_BYTES: usize = 10 * 1024 * 1024; const B1_WAL_FLUSH_INTERVAL: Duration = Duration::from_millis(100); @@ -112,6 +137,7 @@ pub(crate) struct B1WorkerLimits { pub(crate) max_resident_writers_root: usize, pub(crate) max_resident_writers_per_table: usize, pub(crate) max_reserved_arrow_bytes: u64, + pub(crate) max_b2_preprocessing_bytes: u64, pub(crate) max_inflight_calls: usize, pub(crate) max_pending_generations: usize, pub(crate) put_deadline: Duration, @@ -146,6 +172,11 @@ impl B1WorkerLimits { "aggregate Arrow reservation must fit at least one legal generation ({B1_MAX_GENERATION_ARROW_BYTES} bytes)" ))); } + if self.max_b2_preprocessing_bytes < B2_PREPROCESSING_RESERVATION_BYTES { + return Err(MemWalWorkerError::config(format!( + "B2 preprocessing reservation must fit one legal row's bounded scratch ({B2_PREPROCESSING_RESERVATION_BYTES} bytes)" + ))); + } for (field, duration) in [ ("put_deadline", self.put_deadline), ("seal_deadline", self.seal_deadline), @@ -277,6 +308,39 @@ pub(crate) struct DurableBatchAck { pub(crate) row_count: u64, } +/// Watcher-confirmed same-generation sequencing state for one logical key. +/// +/// This is deliberately a warm projection, never restart authority. It is +/// installed only after the durability watcher and the same writer's final +/// epoch check both succeed, and it disappears with the resident worker. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct ConfirmedStreamTokenOverlayRow { + pub(crate) authority: StreamTokenAuthorityRow, + pub(crate) metadata: TrustedStreamRowMetadata, +} + +impl ConfirmedStreamTokenOverlayRow { + pub(crate) fn validate(&self, identity: TableIdentity, logical_id: &str) -> OmniResult<()> { + self.authority + .validate() + .map_err(|error| OmniError::manifest_internal(error.to_string()))?; + self.metadata + .validate_for(identity, logical_id) + .map_err(|error| OmniError::manifest_internal(error.to_string()))?; + if self.authority.identity != identity + || self.authority.logical_id != logical_id + || !self.metadata.agrees_with_authority(&self.authority) + { + return Err(OmniError::manifest_internal(format!( + "confirmed stream-token overlay for '{logical_id}' disagrees with its table identity or trusted metadata" + ))); + } + Ok(()) + } +} + +pub(crate) type ConfirmedStreamTokenOverlay = BTreeMap; + /// Opaque proof that the background-owned append task owns the shared /// admission lease after completing the caller's fresh authority check. pub(crate) struct CheckedStreamAuthority { @@ -338,9 +402,9 @@ pub(super) fn expected_b1_writer_defaults(enrollment_id: ShardId) -> BTreeMap String { +fn canonical_b1_stream_config_v2() -> String { format!( - "omnigraph.stream_config_version={STREAM_CONFIG_VERSION}\n\ + "omnigraph.stream_config_version=2\n\ sharding=unsharded\n\ num_shards=1\n\ shard_spec_id={UNSHARDED_SPEC_ID}\n\ @@ -359,8 +423,43 @@ fn canonical_b1_stream_config() -> String { ) } +/// Historical config-v2 digest used only to parse/refuse recovery-v11 files. +/// Active enrollment and fold code always emits config v3. pub(crate) fn stream_config_v2_hash() -> String { - let digest = Sha256::digest(canonical_b1_stream_config().as_bytes()); + let digest = Sha256::digest(canonical_b1_stream_config_v2().as_bytes()); + format!("sha256:{digest:x}") +} + +fn canonical_b2_stream_config() -> String { + format!( + "omnigraph.stream_config_version={STREAM_CONFIG_VERSION}\n\ + sharding=unsharded\n\ + num_shards=1\n\ + shard_spec_id={UNSHARDED_SPEC_ID}\n\ + shard_field_id=bucket\n\ + shard_result_type=int32\n\ + maintained_indexes=\n\ + durable_write=true\n\ + max_wal_buffer_size={B1_WAL_BUFFER_BYTES}\n\ + max_wal_flush_interval_ms={}\n\ + max_memtable_size={B1_MEMTABLE_BYTES}\n\ + max_memtable_rows={B1_MEMTABLE_ROWS}\n\ + max_memtable_batches={B1_MEMTABLE_BATCHES}\n\ + max_unflushed_memtable_bytes={B1_MEMTABLE_BYTES}\n\ + enable_memtable=true\n\ + stream_token_derivation_version={STREAM_TOKEN_DERIVATION_VERSION}\n\ + stream_token_wire={STREAM_TOKEN_WIRE_VERSION}\n\ + payload_digest_version={STREAM_PAYLOAD_DIGEST_VERSION}\n\ + payload_encoding_version={STREAM_PAYLOAD_ENCODING_VERSION}\n\ + max_canonical_payload_bytes={B2_MAX_CANONICAL_PAYLOAD_BYTES}\n\ + max_token_projection_arrow_bytes={B2_MAX_TOKEN_PROJECTION_ARROW_BYTES}\n\ + max_token_recovery_json_bytes={B2_MAX_TOKEN_RECOVERY_JSON_BYTES}\n", + B1_WAL_FLUSH_INTERVAL.as_millis(), + ) +} + +pub(crate) fn stream_config_v3_hash() -> String { + let digest = Sha256::digest(canonical_b2_stream_config().as_bytes()); format!("sha256:{digest:x}") } @@ -405,7 +504,7 @@ fn validate_b1_topology( let expected = expected_b1_writer_defaults(enrollment_id); if actual != expected { return Err(MemWalWorkerError::config(format!( - "persisted config-v2 defaults differ: expected={expected:?}, actual={actual:?}" + "persisted config-v3 defaults differ: expected={expected:?}, actual={actual:?}" ))); } Ok(()) @@ -432,10 +531,10 @@ fn validate_bound_merge_progress( Ok(()) } -/// Validate that durable graph binding identity and Lance's persisted config-v2 +/// Validate that durable graph binding identity and Lance's persisted config-v3 /// describe exactly the same physical enrollment. The enrollment UUID is /// never inferred from the replaceable MemWAL index UUID. -pub(crate) fn validate_stream_config_v2_binding( +pub(crate) fn validate_stream_config_v3_binding( details: &MemWalIndexDetails, binding: &StreamPhysicalBinding, ) -> Result<(ShardId, ShardId), MemWalWorkerError> { @@ -445,11 +544,11 @@ pub(crate) fn validate_stream_config_v2_binding( binding.stream_config_version ))); } - if binding.stream_config_hash != stream_config_v2_hash() { + if binding.stream_config_hash != stream_config_v3_hash() { return Err(MemWalWorkerError::config(format!( "binding config hash is {}, expected {}", binding.stream_config_hash, - stream_config_v2_hash() + stream_config_v3_hash() ))); } if binding.table_branch.is_some() || binding.shard_ids.len() != 1 { @@ -591,7 +690,7 @@ fn exact_unmerged_generations( Ok((merged, unmerged)) } -/// Passive read/open-time validation for a data-bearing config-v2 lifecycle. +/// Passive read/open-time validation for a data-bearing config-v3 lifecycle. /// /// Unlike [`classify_active_state`], this never calls `mem_wal_writer` and /// therefore never claims an epoch. It validates only durable/public state; @@ -625,7 +724,7 @@ pub(crate) async fn validate_b1_lifecycle_physical_state( .await .map_err(|error| MemWalWorkerError::lance("passive config read", error))? .ok_or_else(|| MemWalWorkerError::state("lifecycle-bound table has no MemWAL index"))?; - let (_, shard_id) = validate_stream_config_v2_binding(&details, &lifecycle.binding)?; + let (_, shard_id) = validate_stream_config_v3_binding(&details, &lifecycle.binding)?; let shard_ids = dataset .list_mem_wal_latest_shard_ids() .await @@ -792,8 +891,7 @@ impl GenerationAccounting { fn account_batch(batch: &RecordBatch) -> Result { let rows = u64::try_from(batch.num_rows()) .map_err(|_| MemWalWorkerError::state("batch row count does not fit u64"))?; - let arrow_bytes = u64::try_from(batch.get_array_memory_size()) - .map_err(|_| MemWalWorkerError::state("batch Arrow memory size does not fit u64"))?; + let arrow_bytes = b1_logical_batch_bytes(batch)?; Ok(GenerationAccounting { rows, arrow_bytes, @@ -801,6 +899,31 @@ fn account_batch(batch: &RecordBatch) -> Result Result { + batch.columns().iter().try_fold(0_u64, |total, column| { + let bytes = column + .to_data() + .get_slice_memory_size() + .map_err(|error| MemWalWorkerError::state(format!( + "logical Arrow slice accounting failed: {error}" + )))?; + total + .checked_add(u64::try_from(bytes).map_err(|_| { + MemWalWorkerError::state("logical Arrow slice size does not fit u64") + })?) + .ok_or_else(|| MemWalWorkerError::state("logical Arrow slice size overflow")) + }) +} + fn account_store(store: &BatchStore) -> Result { store .iter() @@ -1310,6 +1433,9 @@ struct MemWalWorker { table_key: String, writer: Arc, mode: tokio::sync::Mutex, + /// Warm watcher-confirmed projection for the one live generation. Cold + /// replay/fold-only workers always start empty and never reconstruct it. + confirmed_token_overlay: Mutex, usage: Mutex, last_used: Mutex, idle_task_armed: AtomicBool, @@ -1358,6 +1484,7 @@ impl MemWalWorker { table_key, writer: opened.claimed.writer, mode: tokio::sync::Mutex::new(mode), + confirmed_token_overlay: Mutex::new(BTreeMap::new()), usage: Mutex::new(usage), last_used: Mutex::new(std::time::Instant::now()), idle_task_armed: AtomicBool::new(false), @@ -1376,6 +1503,7 @@ impl MemWalWorker { table_key, writer: claimed.writer, mode: tokio::sync::Mutex::new(WorkerMode::Retiring), + confirmed_token_overlay: Mutex::new(BTreeMap::new()), usage: Mutex::new(WorkerUsage { resident: true, accounting: GenerationAccounting { @@ -1399,6 +1527,9 @@ impl MemWalWorker { shard_id: self.key.shard_id.to_string(), caller_ordinal_start: ordinals.start, caller_ordinal_end: ordinals.end, + admission_attempt_id: None, + logical_write_ids: Vec::new(), + unconfirmed_candidate_token: None, reason: reason.into(), } } @@ -1469,6 +1600,9 @@ struct RegistryUsage { resident_writers_root: usize, resident_writers_by_table: HashMap, reserved_arrow_bytes: u64, + /// Conservative scratch reservation held before a B2 row can materialize + /// external blobs or allocate its canonical payload. + b2_preprocessing_bytes: u64, /// Exact resident plus queued generation accounting by physical worker /// key. This is the decomposition of `reserved_arrow_bytes`, not a second /// lifecycle source of truth; it exists so same-key cap crossings retain @@ -1554,6 +1688,37 @@ struct InFlightPermit { registry: Arc, } +/// Root-scoped ownership for B2 preprocessing scratch. It is acquired before +/// blob materialization or canonical encoding and retains the ordinary +/// inflight slot until that ownership transfers into [`QueuedBatchPermit`]. +pub(crate) struct B2PreprocessingPermit { + registry: Arc, + bytes: u64, + inflight: Option, +} + +impl B2PreprocessingPermit { + fn take_inflight(&mut self) -> InFlightPermit { + self.inflight + .take() + .expect("B2 preprocessing inflight permit transfers exactly once") + } +} + +impl Drop for B2PreprocessingPermit { + fn drop(&mut self) { + let mut usage = self + .registry + .usage + .lock() + .expect("MemWAL registry usage poisoned"); + usage.b2_preprocessing_bytes = usage + .b2_preprocessing_bytes + .checked_sub(self.bytes) + .expect("B2 preprocessing reservation accounting underflow"); + } +} + impl Drop for InFlightPermit { fn drop(&mut self) { let mut usage = self @@ -1589,6 +1754,80 @@ impl QueuedBatchPermit { .expect("queued batch inflight permit transfers exactly once") } + /// Replace the preliminary logical-batch charge with the exact stored + /// batch after trusted stream metadata has been classified and appended. + /// + /// The preliminary charge is acquired before shared admission and the + /// same-key queue, so caller-owned row buffers are never unaccounted. This + /// adjustment is immediate/non-waiting and happens while that queue + /// position is held, before worker-mode inspection or any Lance call. + pub(crate) fn reprice_for_exact_batch( + &mut self, + table_key: &str, + batch: &RecordBatch, + ) -> OmniResult<()> { + if self.transferred { + return Err(OmniError::manifest_internal( + "cannot reprice a stream batch after its charge transferred to a worker", + )); + } + let exact = + b1_input_accounting(batch).map_err(|error| OmniError::Lance(error.to_string()))?; + let mut usage = self + .registry + .usage + .lock() + .expect("MemWAL registry usage poisoned"); + let current = usage + .reserved_by_key + .get(&self.key) + .copied() + .ok_or_else(|| { + OmniError::manifest_internal( + "queued stream charge disappeared before exact repricing", + ) + })?; + let without_preliminary = current + .checked_sub(self.charge) + .map_err(|error| OmniError::manifest_internal(error.to_string()))?; + let projected = without_preliminary + .checked_add(exact) + .map_err(|error| OmniError::Lance(error.to_string()))?; + if !projected.fits() { + return Err(OmniError::FoldRequired { + table_key: table_key.to_string(), + rows: projected.rows, + bytes: projected.arrow_bytes, + }); + } + let root_without_preliminary = usage + .reserved_arrow_bytes + .checked_sub(self.charge.arrow_bytes) + .ok_or_else(|| { + OmniError::manifest_internal("root stream charge underflow during exact repricing") + })?; + let root_projected = root_without_preliminary + .checked_add(exact.arrow_bytes) + .ok_or_else(|| { + MemWalWorkerRegistry::resource_error( + "stream_reserved_arrow_bytes", + self.registry.limits.max_reserved_arrow_bytes, + u64::MAX, + ) + })?; + if root_projected > self.registry.limits.max_reserved_arrow_bytes { + return Err(MemWalWorkerRegistry::resource_error( + "stream_reserved_arrow_bytes", + self.registry.limits.max_reserved_arrow_bytes, + root_projected, + )); + } + usage.reserved_arrow_bytes = root_projected; + usage.reserved_by_key.insert(self.key, projected); + self.charge = exact; + Ok(()) + } + fn transfer_to_worker( mut self, worker: &MemWalWorker, @@ -1725,6 +1964,44 @@ impl MemWalWorkerRegistry { }) } + /// Reserve the complete worst-case scratch envelope before B2 can resolve + /// blobs or allocate canonical payload bytes. The ordinary inflight slot + /// moves from this permit into the queued/worker corridor; scratch remains + /// reserved until canonical hashing is complete. + pub(crate) fn reserve_b2_preprocessing( + self: &Arc, + ) -> OmniResult { + let inflight = self + .reserve_inflight_core() + .map_err(Self::worker_error_to_omni)?; + let bytes = B2_PREPROCESSING_RESERVATION_BYTES; + let mut usage = self.usage.lock().expect("MemWAL registry usage poisoned"); + let actual = usage + .b2_preprocessing_bytes + .checked_add(bytes) + .ok_or_else(|| { + Self::resource_error( + "stream_b2_preprocessing_bytes", + self.limits.max_b2_preprocessing_bytes, + u64::MAX, + ) + })?; + if actual > self.limits.max_b2_preprocessing_bytes { + return Err(Self::resource_error( + "stream_b2_preprocessing_bytes", + self.limits.max_b2_preprocessing_bytes, + actual, + )); + } + usage.b2_preprocessing_bytes = actual; + drop(usage); + Ok(B2PreprocessingPermit { + registry: Arc::clone(self), + bytes, + inflight: Some(inflight), + }) + } + fn ensure_input_projection_fits( table_key: &str, mode: &WorkerMode, @@ -1848,6 +2125,59 @@ impl MemWalWorkerRegistry { let inflight = self .reserve_inflight_core() .map_err(Self::worker_error_to_omni)?; + self.reserve_put_input_with_inflight( + key, + table_key, + batch, + acquire_authority, + inflight, + ) + .await + } + + /// B2 sibling of [`Self::reserve_put_input`]. Preprocessing already owns + /// the inflight slot, so this transfers it without opening an unbounded + /// interval between scratch allocation and root accounting. + pub(crate) async fn reserve_b2_put_input( + self: &Arc, + key: StreamWorkerKey, + table_key: &str, + batch: &RecordBatch, + preprocessing: &mut B2PreprocessingPermit, + acquire_authority: F, + ) -> OmniResult<(QueuedBatchPermit, CheckedStreamAuthority)> + where + F: FnOnce() -> Fut + Send, + Fut: Future + Send, + { + if !Arc::ptr_eq(self, &preprocessing.registry) { + return Err(OmniError::manifest_internal( + "B2 preprocessing permit belongs to another root registry", + )); + } + let inflight = preprocessing.take_inflight(); + self.reserve_put_input_with_inflight( + key, + table_key, + batch, + acquire_authority, + inflight, + ) + .await + } + + async fn reserve_put_input_with_inflight( + self: &Arc, + key: StreamWorkerKey, + table_key: &str, + batch: &RecordBatch, + acquire_authority: F, + inflight: InFlightPermit, + ) -> OmniResult<(QueuedBatchPermit, CheckedStreamAuthority)> + where + F: FnOnce() -> Fut + Send, + Fut: Future + Send, + { let charge = b1_input_accounting(batch).map_err(|error| OmniError::Lance(error.to_string()))?; if !charge.fits() { @@ -1871,6 +2201,159 @@ impl MemWalWorkerRegistry { Ok((queued, authority)) } + /// Read one watcher-confirmed token while the caller owns this key's input- + /// queue position. A vacant worker means an empty live + /// generation. Replayed/flushed state is fold-only and therefore cannot + /// be treated as a reconstructed sequencing overlay. + pub(crate) async fn confirmed_token_for_key( + self: &Arc, + queued: &QueuedBatchPermit, + table_key: &str, + logical_id: &str, + ) -> OmniResult> { + if !Arc::ptr_eq(self, &queued.registry) { + return Err(OmniError::manifest_internal( + "stream token overlay permit belongs to another root registry", + )); + } + let slot = self.slot(queued.key); + let worker = { + let state = slot.state.lock().await; + match &*state { + RegistrySlotState::Vacant => return Ok(None), + RegistrySlotState::Active(worker) => Arc::clone(worker), + RegistrySlotState::Opening(_) => { + return Err(OmniError::manifest_internal( + "stream token overlay observed an opener despite owning the same-key input queue", + )); + } + RegistrySlotState::Retiring(_) => { + return Err(OmniError::Lance( + MemWalWorkerError::Retiring { + key: queued.key, + reason: "original abort completion has not settled".to_string(), + } + .to_string(), + )); + } + } + }; + let mode = worker.mode.lock().await; + match &*mode { + WorkerMode::Admit(_) => {} + WorkerMode::FoldReplay(replay) => { + return Err(OmniError::FoldRequired { + table_key: table_key.to_string(), + rows: replay.accounting.rows, + bytes: replay.accounting.arrow_bytes, + }); + } + WorkerMode::FoldFlushed(_) => { + return Err(OmniError::FoldRequired { + table_key: table_key.to_string(), + rows: 0, + bytes: 0, + }); + } + WorkerMode::Retiring => { + return Err(OmniError::Lance( + MemWalWorkerError::Retiring { + key: queued.key, + reason: "token overlay is unavailable while the worker retires".to_string(), + } + .to_string(), + )); + } + } + let current = worker + .confirmed_token_overlay + .lock() + .expect("MemWAL token overlay poisoned") + .get(logical_id) + .cloned(); + Ok(current) + } + + /// Return the exact current-generation token winners after applying one + /// queued update set. The caller owns the same-key input queue, so this + /// projection cannot race a watcher-confirmed overlay installation. It is + /// used to enforce recovery-v12/token-table byte bounds before Lance is + /// invoked and an acknowledgement can become possible. + pub(crate) async fn projected_token_authority_rows( + self: &Arc, + queued: &QueuedBatchPermit, + table_key: &str, + updates: &ConfirmedStreamTokenOverlay, + ) -> OmniResult> { + if !Arc::ptr_eq(self, &queued.registry) { + return Err(OmniError::manifest_internal( + "stream token projection permit belongs to another root registry", + )); + } + for (logical_id, update) in updates { + update.validate(queued.key.identity, logical_id)?; + } + let slot = self.slot(queued.key); + let worker = { + let state = slot.state.lock().await; + match &*state { + RegistrySlotState::Vacant => None, + RegistrySlotState::Active(worker) => Some(Arc::clone(worker)), + RegistrySlotState::Opening(_) => { + return Err(OmniError::manifest_internal( + "stream token projection observed an opener despite owning the same-key input queue", + )); + } + RegistrySlotState::Retiring(_) => { + return Err(OmniError::Lance( + MemWalWorkerError::Retiring { + key: queued.key, + reason: "original abort completion has not settled".to_string(), + } + .to_string(), + )); + } + } + }; + let mut projected = ConfirmedStreamTokenOverlay::new(); + if let Some(worker) = worker { + let mode = worker.mode.lock().await; + match &*mode { + WorkerMode::Admit(_) => {} + WorkerMode::FoldReplay(replay) => { + return Err(OmniError::FoldRequired { + table_key: table_key.to_string(), + rows: replay.accounting.rows, + bytes: replay.accounting.arrow_bytes, + }); + } + WorkerMode::FoldFlushed(_) => { + return Err(OmniError::FoldRequired { + table_key: table_key.to_string(), + rows: 0, + bytes: 0, + }); + } + WorkerMode::Retiring => { + return Err(OmniError::Lance(format!( + "MemWAL worker {} is retiring", + queued.key + ))); + } + } + projected = worker + .confirmed_token_overlay + .lock() + .expect("MemWAL token overlay poisoned") + .clone(); + } + projected.extend(updates.clone()); + Ok(projected + .into_values() + .map(|row| row.authority) + .collect()) + } + fn release_queued_charge(&self, key: StreamWorkerKey, charge: GenerationAccounting) { let mut usage = self.usage.lock().expect("MemWAL registry usage poisoned"); let current = usage.reserved_by_key.get(&key).copied().unwrap_or_default(); @@ -2336,6 +2819,7 @@ impl MemWalWorkerRegistry { table_key: String, batch: RecordBatch, caller_ordinals: CallerOrdinalRange, + confirmed_token_updates: ConfirmedStreamTokenOverlay, mut queued: QueuedBatchPermit, prepare: PreparePut, idle_authority: IdleAuthorityCheck, @@ -2358,6 +2842,7 @@ impl MemWalWorkerRegistry { table_key, batch, caller_ordinals, + confirmed_token_updates, queued, prepare, idle_authority, @@ -2375,6 +2860,7 @@ impl MemWalWorkerRegistry { table_key: String, batch: RecordBatch, caller_ordinals: CallerOrdinalRange, + confirmed_token_updates: ConfirmedStreamTokenOverlay, queued: QueuedBatchPermit, prepare: PreparePut, idle_authority: IdleAuthorityCheck, @@ -2604,8 +3090,16 @@ impl MemWalWorkerRegistry { }; self.ensure_idle_task(&slot, &worker, idle_authority); - self.invoke_put(&slot, &worker, batch, caller_ordinals, authority, queued) - .await + self.invoke_put( + &slot, + &worker, + batch, + caller_ordinals, + confirmed_token_updates, + authority, + queued, + ) + .await } async fn invoke_put( @@ -2614,6 +3108,7 @@ impl MemWalWorkerRegistry { worker: &Arc, batch: RecordBatch, caller_ordinals: CallerOrdinalRange, + confirmed_token_updates: ConfirmedStreamTokenOverlay, authority: CheckedStreamAuthority, queued: QueuedBatchPermit, ) -> OmniResult { @@ -2636,6 +3131,9 @@ impl MemWalWorkerRegistry { caller_ordinals.as_range() ))); } + for (logical_id, update) in &confirmed_token_updates { + update.validate(worker.key.identity, logical_id)?; + } let charge = queued.charge(); let mut mode = worker.mode.lock().await; let current = match &*mode { @@ -2883,6 +3381,13 @@ impl MemWalWorkerRegistry { } } + { + let mut overlay = worker + .confirmed_token_overlay + .lock() + .expect("MemWAL token overlay poisoned"); + overlay.extend(confirmed_token_updates); + } *mode = WorkerMode::Admit(reserved); *worker .last_used @@ -3869,6 +4374,7 @@ mod tests { max_resident_writers_root: 4, max_resident_writers_per_table: 1, max_reserved_arrow_bytes: 4 * B1_MAX_GENERATION_ARROW_BYTES, + max_b2_preprocessing_bytes: B2_MAX_PREPROCESSING_BYTES_ROOT, max_inflight_calls: 2, max_pending_generations: 4, put_deadline: Duration::from_secs(1), @@ -3929,7 +4435,7 @@ mod tests { } #[test] - fn accounting_charges_exact_post_tombstone_arrow_memory() { + fn accounting_charges_exact_post_tombstone_logical_slice_memory() { let batch = batch(3); let charged = post_tombstone_accounting(&batch).unwrap(); let schema = schema_with_tombstone(batch.schema().as_ref()); @@ -3940,7 +4446,7 @@ mod tests { assert_eq!(charged.batches, 1); assert_eq!( charged.arrow_bytes, - u64::try_from(stored.get_array_memory_size()).unwrap() + b1_logical_batch_bytes(&stored).unwrap() ); } @@ -4003,6 +4509,45 @@ mod tests { registry.release_resident_reservation(identity); } + #[test] + fn b2_preprocessing_is_root_bounded_before_payload_allocation() { + let root = format!("memory://b2-preprocessing/{}", ShardId::new_v4()); + let mut preprocessing_limits = limits(); + preprocessing_limits.max_inflight_calls = 3; + let registry = MemWalWorkerRegistry::for_root(&root, preprocessing_limits).unwrap(); + let first = registry.reserve_b2_preprocessing().unwrap(); + let second = registry.reserve_b2_preprocessing().unwrap(); + let error = match registry.reserve_b2_preprocessing() { + Ok(_) => panic!("a third worst-case B2 envelope must exceed the root bound"), + Err(error) => error, + }; + assert!(matches!( + error, + OmniError::ResourceLimitExceeded { + ref resource, + limit, + actual, + } if resource == "stream_b2_preprocessing_bytes" + && limit == B2_MAX_PREPROCESSING_BYTES_ROOT + && actual == 3 * B2_PREPROCESSING_RESERVATION_BYTES + )); + { + let usage = registry.usage.lock().unwrap(); + assert_eq!( + usage.b2_preprocessing_bytes, + B2_MAX_PREPROCESSING_BYTES_ROOT + ); + // The failed third reservation releases the inflight slot it + // acquired before discovering root scratch pressure. + assert_eq!(usage.inflight_calls, 2); + } + drop(first); + drop(second); + let usage = registry.usage.lock().unwrap(); + assert_eq!(usage.b2_preprocessing_bytes, 0); + assert_eq!(usage.inflight_calls, 0); + } + #[test] fn queued_input_is_charged_before_detach_and_released_without_transfer() { let root = format!("memory://queued-input/{}", ShardId::new_v4()); @@ -4097,6 +4642,7 @@ mod tests { resident_writers_root: 1, resident_writers_by_table: HashMap::from([(identity, 1)]), reserved_arrow_bytes: 1_024, + b2_preprocessing_bytes: 0, reserved_by_key: HashMap::from([(key, initial)]), fold_required_by_key: HashMap::new(), inflight_calls: 1, diff --git a/crates/omnigraph/tests/failpoints.rs b/crates/omnigraph/tests/failpoints.rs index 531db3c1..a416f253 100644 --- a/crates/omnigraph/tests/failpoints.rs +++ b/crates/omnigraph/tests/failpoints.rs @@ -2,6 +2,7 @@ mod helpers; +use std::future::Future; use std::process::Command; use std::sync::Arc; @@ -31,6 +32,28 @@ const SCHEMA_V1: &str = "node Person { name: String @key }\n"; const SCHEMA_V2_ADDED_TYPE: &str = "node Person { name: String @key }\nnode Company { name: String @key }\n"; +/// Run one composed debug-build recovery scenario outside libtest's 2-MiB +/// thread. The production operations are independently exercised on ordinary +/// Tokio stacks; this helper bounds only the large future assembled by a test +/// that chains several complete recovery cycles in one body. +fn on_big_stack(body: impl FnOnce() -> F + Send + 'static) +where + F: Future, +{ + std::thread::Builder::new() + .stack_size(64 * 1024 * 1024) + .spawn(move || { + tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .unwrap() + .block_on(body()); + }) + .unwrap() + .join() + .unwrap(); +} + fn assert_open_stream_lifecycle_conflict(error: OmniError, operation: &str) { let OmniError::Manifest(manifest_error) = error else { panic!("expected typed manifest lifecycle conflict, got {error:?}"); @@ -668,7 +691,8 @@ fn rfc023_external_writer_process() { /// an enrolled multi-table writer has committed table 1 and parked before /// table 2, where publishing a competing graph commit would be both unnecessary /// and semantically wrong for the recovery assertion. The source schema is -/// restricted to the two-column `name @key` fixture used by that test. +/// restricted to the `name @key` fixture used by that test; the canonical +/// hidden attribution column is supplied as physical null metadata. async fn commit_raw_fenced_name_row(table_uri: &str, id: &str) { let base = Arc::new(Dataset::open(table_uri).await.unwrap()); let schema = Arc::new(Schema::from(base.schema())); @@ -678,14 +702,16 @@ async fn commit_raw_fenced_name_row(table_uri: &str, id: &str) { .iter() .map(|field| field.name().as_str()) .collect::>(), - vec!["id", "name"], + vec!["id", "name", "__omnigraph_stream_v1$"], "raw conflict injector is intentionally limited to the name-only fixture" ); + let hidden = arrow_array::new_null_array(schema.field(2).data_type(), 1); let batch = RecordBatch::try_new( schema.clone(), vec![ Arc::new(StringArray::from(vec![id])), Arc::new(StringArray::from(vec![id])), + hidden, ], ) .unwrap(); @@ -2543,15 +2569,17 @@ async fn rfc023_disjoint_retryable_strict_conflict_reprepares_without_key_confli .iter() .map(|field| field.name().as_str()) .collect::>(), - ["id", "name", "score"], + ["id", "name", "score", "__omnigraph_stream_v1$"], "raw disjoint-conflict injector is schema-specific" ); + let hidden = arrow_array::new_null_array(schema.field(3).data_type(), 1); let foreign = RecordBatch::try_new( schema, vec![ Arc::new(StringArray::from(vec!["foreign-disjoint"])), Arc::new(StringArray::from(vec!["foreign-disjoint"])), Arc::new(Int32Array::from(vec![1])), + hidden, ], ) .unwrap(); @@ -4019,9 +4047,13 @@ async fn recovery_rolls_forward_load_overwrite() { ); } -#[tokio::test] +#[test] #[serial] -async fn recovery_rolls_forward_ensure_indices_on_feature_branch() { +fn recovery_rolls_forward_ensure_indices_on_feature_branch() { + on_big_stack(recovery_rolls_forward_ensure_indices_on_feature_branch_inner); +} + +async fn recovery_rolls_forward_ensure_indices_on_feature_branch_inner() { use lance::index::DatasetIndexExt; use omnigraph::loader::{LoadMode, load_jsonl}; diff --git a/crates/omnigraph/tests/forbidden_apis.rs b/crates/omnigraph/tests/forbidden_apis.rs index aaffa627..902dcf84 100644 --- a/crates/omnigraph/tests/forbidden_apis.rs +++ b/crates/omnigraph/tests/forbidden_apis.rs @@ -140,6 +140,7 @@ const ALLOW_LIST_FILES: &[&str] = &[ "db/manifest/namespace.rs", // Opens manifest datasets through the shared namespace. "db/manifest/publisher.rs", // Lowest row-level manifest publish gateway. "db/manifest/recovery.rs", // Recovery executor; exactly inventoried below. + "db/manifest/token_store.rs", // Graph-global stream-token participant gateway. "db/manifest/tests.rs", // Out-of-line tests for the trusted gateways. "instrumentation.rs", // The instrumented dataset opener. ]; @@ -203,7 +204,8 @@ const MERGE_V9: WriteProtocol = WriteProtocol::Exact("BranchMerge v9"); const INDICES_V9: WriteProtocol = WriteProtocol::Exact("EnsureIndices v9"); const OPTIMIZE_V9: WriteProtocol = WriteProtocol::Bounded("Optimize v9"); const STREAM_ENROLLMENT_V10: WriteProtocol = WriteProtocol::Exact("StreamEnrollment v10"); -const STREAM_FOLD_V11: WriteProtocol = WriteProtocol::Exact("StreamFold v11 / private B1"); +const STREAM_FOLD_V12: WriteProtocol = + WriteProtocol::Exact("StreamFold v12 / private B2 common core"); #[derive(Debug, Clone, Copy)] struct WriteSurface { @@ -232,7 +234,7 @@ write_surfaces! { "db/omnigraph.rs" => INDICES_V9 => ["ensure_indices", "ensure_indices_on"], "db/omnigraph.rs" => WriteProtocol::TestOnly => ["failpoint_publish_table_head_without_index_rebuild_for_test"], "db/omnigraph/stream_enrollment.rs" => WriteProtocol::TestOnly => ["failpoint_enroll_stream_table_for_test"], - "db/omnigraph/stream_ingest.rs" => WriteProtocol::TestOnly => ["failpoint_stream_b1_for_test"], + "db/omnigraph/stream_ingest.rs" => WriteProtocol::TestOnly => ["failpoint_stream_b1_for_test", "failpoint_stream_b2_for_test"], "db/omnigraph.rs" => OPTIMIZE_V9 => ["optimize"], "db/omnigraph.rs" => WriteProtocol::ManifestAdoption => ["repair"], "db/omnigraph.rs" => WriteProtocol::PhysicalOnly => ["cleanup"], @@ -246,6 +248,7 @@ write_surfaces! { // cannot evade discovery. const READ_ONLY_SURFACES: &[(&str, &str)] = &[ ("db/omnigraph.rs", "open_read_only"), + ("db/omnigraph/stream_ingest.rs", "failpoint_stream_incarnation_for_test"), ("db/omnigraph.rs", "plan_schema"), ("db/omnigraph.rs", "plan_schema_with_options"), ("db/omnigraph.rs", "preview_schema_apply_with_options"), @@ -641,21 +644,29 @@ durable_calls! { ("db/omnigraph/table_ops.rs", "write_sidecar(", 1, INDICES_V9), ("db/omnigraph/optimize.rs", "write_sidecar(", 1, OPTIMIZE_V9), ("db/omnigraph/stream_enrollment.rs", "write_sidecar(", 1, STREAM_ENROLLMENT_V10), - ("db/omnigraph/stream_ingest.rs", "write_sidecar(", 1, STREAM_FOLD_V11), + ("db/omnigraph/stream_ingest.rs", "write_sidecar(", 1, STREAM_FOLD_V12), + ("db/manifest/token_store.rs", "Dataset::write(", 1, WriteProtocol::Bootstrap), + ("db/manifest/token_store.rs", "MergeInsertBuilder::try_new(", 1, STREAM_FOLD_V12), + ("db/manifest/token_store.rs", ".execute_uncommitted(", 1, STREAM_FOLD_V12), + ("db/omnigraph/stream_ingest.rs", "stage_stream_token_upsert(", 1, STREAM_FOLD_V12), + ("db/manifest/recovery.rs", "stage_stream_token_upsert(", 1, WriteProtocol::RecoveryExecutor), ("exec/staging.rs", ".commit_staged_exact(", 1, WriteProtocol::Exact("Mutation/Load v9")), ("exec/merge.rs", ".commit_staged_exact(", 1, MERGE_V9), ("db/omnigraph/schema_apply.rs", ".commit_staged_create_exact(", 1, SCHEMA_V9), ("db/omnigraph/schema_apply.rs", ".commit_staged_exact(", 1, SCHEMA_V9), ("db/omnigraph/table_ops.rs", ".commit_staged_exact(", 1, INDICES_V9), - ("db/omnigraph/stream_ingest.rs", ".commit_staged_exact(", 1, STREAM_FOLD_V11), + ("db/omnigraph/stream_ingest.rs", ".commit_staged_exact(", 2, STREAM_FOLD_V12), + ("db/manifest/recovery.rs", ".commit_staged_exact(", 1, WriteProtocol::RecoveryExecutor), ("db/omnigraph/table_ops.rs", ".commit_staged(", 1, WriteProtocol::Composed("shared merge/Optimize index tail")), ("db/omnigraph/table_ops.rs", ".fork_branch_from_state(", 1, WriteProtocol::Composed("adapter-owned first-touch data ref")), ("exec/staging.rs", "confirm_occ_sidecar_v9(", 1, WriteProtocol::Exact("Mutation/Load v9")), ("exec/merge.rs", "confirm_branch_merge_sidecar_v9(", 1, MERGE_V9), ("db/omnigraph/schema_apply.rs", "confirm_schema_apply_sidecar_v9(", 1, SCHEMA_V9), ("db/omnigraph/table_ops.rs", "confirm_ensure_indices_sidecar_v9(", 1, INDICES_V9), - ("db/omnigraph/stream_ingest.rs", "complete_stream_fold_sidecar_v11(", 1, STREAM_FOLD_V11), - ("db/omnigraph/stream_ingest.rs", "finalize_effect_free_stream_fold_sidecar_v11(", 1, STREAM_FOLD_V11), + ("db/omnigraph/stream_ingest.rs", "confirm_stream_fold_sidecar_v12(", 1, STREAM_FOLD_V12), + ("db/manifest/recovery.rs", "confirm_stream_fold_sidecar_v12(", 1, WriteProtocol::RecoveryExecutor), + ("db/omnigraph/stream_ingest.rs", "complete_stream_fold_sidecar_v12(", 2, STREAM_FOLD_V12), + ("db/omnigraph/stream_ingest.rs", "finalize_effect_free_stream_fold_sidecar_v12(", 1, STREAM_FOLD_V12), ("exec/mutation.rs", "delete_sidecar(", 1, MUTATION_V9), ("loader/mod.rs", "delete_sidecar(", 1, LOAD_V9), ("exec/merge.rs", "delete_sidecar(", 1, MERGE_V9), @@ -678,7 +689,7 @@ durable_calls! { ("db/omnigraph.rs", ".write_text_if_absent(", 1, WriteProtocol::Bootstrap), ("db/omnigraph.rs", ".write_text(", 1, WriteProtocol::Bootstrap), ("db/schema_state.rs", ".write_text(", 2, WriteProtocol::Composed("schema state publication")), - ("db/manifest/recovery.rs", ".write_text(", 7, WriteProtocol::RecoveryExecutor), + ("db/manifest/recovery.rs", ".write_text(", 8, WriteProtocol::RecoveryExecutor), ("db/omnigraph/schema_apply.rs", ".write_text(", 1, SCHEMA_V9), ("db/omnigraph.rs", ".delete(", 1, WriteProtocol::Bootstrap), ("db/schema_state.rs", ".delete(", 3, WriteProtocol::Composed("schema staging cleanup")), @@ -714,11 +725,11 @@ durable_calls! { ("db/manifest/recovery.rs", ".publish_with_precondition(", 1, WriteProtocol::RecoveryExecutor), ("db/manifest/recovery.rs", ".publish(", 1, WriteProtocol::RecoveryExecutor), ("db/manifest/recovery.rs", ".restore(", 1, WriteProtocol::RecoveryExecutor), - ("db/manifest/recovery.rs", ".append(RecoveryAuditRecord", 9, WriteProtocol::RecoveryExecutor), - ("db/manifest/recovery.rs", "publish_recovery_commit(", 10, WriteProtocol::RecoveryExecutor), + ("db/manifest/recovery.rs", ".append(RecoveryAuditRecord", 10, WriteProtocol::RecoveryExecutor), + ("db/manifest/recovery.rs", "publish_recovery_commit(", 11, WriteProtocol::RecoveryExecutor), ("db/manifest/recovery.rs", "restore_table_to_version(", 3, WriteProtocol::RecoveryExecutor), - ("db/manifest/recovery.rs", "record_audit(", 11, WriteProtocol::RecoveryExecutor), - ("db/manifest/recovery.rs", "delete_sidecar_by_operation_id(", 22, WriteProtocol::RecoveryExecutor), + ("db/manifest/recovery.rs", "record_audit(", 12, WriteProtocol::RecoveryExecutor), + ("db/manifest/recovery.rs", "delete_sidecar_by_operation_id(", 24, WriteProtocol::RecoveryExecutor), ("db/manifest/recovery.rs", "delete_sidecar(", 3, WriteProtocol::RecoveryExecutor), ("db/recovery_audit.rs", ".raw_dataset_append(", 1, WriteProtocol::RecoveryExecutor), ("db/recovery_audit.rs", "Dataset::write(", 1, WriteProtocol::RecoveryExecutor), @@ -734,18 +745,19 @@ durable_calls! { ("db/omnigraph/repair.rs", ".dataset()", 1, WriteProtocol::ManifestAdoption), ("db/omnigraph/optimize.rs", ".dataset()", 5, WriteProtocol::Composed("Optimize v9 planning + physical cleanup")), ("db/omnigraph/optimize.rs", ".into_dataset()", 2, OPTIMIZE_V9), - ("db/omnigraph/stream_enrollment.rs", ".dataset()", 6, WriteProtocol::ReadOnlyAccess), + ("db/omnigraph/stream_enrollment.rs", ".dataset()", 7, WriteProtocol::ReadOnlyAccess), ("db/omnigraph/stream_enrollment.rs", ".into_dataset()", 1, STREAM_ENROLLMENT_V10), - ("db/omnigraph/stream_ingest.rs", ".dataset()", 10, STREAM_FOLD_V11), - ("db/omnigraph/stream_ingest.rs", ".mem_wal_writer(", 2, STREAM_FOLD_V11), - ("db/omnigraph/stream_ingest.rs", ".put(", 1, STREAM_FOLD_V11), + ("db/omnigraph/stream_ingest.rs", ".dataset()", 16, STREAM_FOLD_V12), + ("db/omnigraph/stream_ingest.rs", ".mem_wal_writer(", 2, STREAM_FOLD_V12), + ("db/omnigraph/stream_ingest.rs", ".put(", 1, STREAM_FOLD_V12), ("table_store/mem_wal.rs", ".initialize_mem_wal()", 1, STREAM_ENROLLMENT_V10), ("table_store/mem_wal.rs", ".mem_wal_writer(", 1, STREAM_ENROLLMENT_V10), - ("table_store/mem_wal/worker.rs", ".put_no_wait(", 1, STREAM_FOLD_V11), - ("table_store/mem_wal/worker.rs", ".abort()", 2, STREAM_FOLD_V11), - ("table_store/mem_wal/worker.rs", ".force_seal_active()", 1, STREAM_FOLD_V11), - ("table_store/mem_wal/worker.rs", ".wait_for_flush_drain()", 1, STREAM_FOLD_V11), + ("table_store/mem_wal/worker.rs", ".put_no_wait(", 1, STREAM_FOLD_V12), + ("table_store/mem_wal/worker.rs", ".abort()", 2, STREAM_FOLD_V12), + ("table_store/mem_wal/worker.rs", ".force_seal_active()", 1, STREAM_FOLD_V12), + ("table_store/mem_wal/worker.rs", ".wait_for_flush_drain()", 1, STREAM_FOLD_V12), ("db/omnigraph/optimize.rs", "SnapshotHandle::new(", 1, OPTIMIZE_V9), + ("db/omnigraph/stream_ingest.rs", "SnapshotHandle::new(", 1, STREAM_FOLD_V12), ("exec/merge.rs", "SnapshotHandle::new(", 5, MERGE_V9), } @@ -757,6 +769,10 @@ const DURABLE_PRIMITIVES: &[&str] = &[ "confirm_ensure_indices_sidecar_v9(", "complete_stream_fold_sidecar_v11(", "finalize_effect_free_stream_fold_sidecar_v11(", + "confirm_stream_fold_sidecar_v12(", + "complete_stream_fold_sidecar_v12(", + "finalize_effect_free_stream_fold_sidecar_v12(", + "stage_stream_token_upsert(", "delete_sidecar(", ".commit_staged_create_exact(", ".commit_staged_exact(", diff --git a/crates/omnigraph/tests/lifecycle.rs b/crates/omnigraph/tests/lifecycle.rs index 383cbd93..4d7d3ebf 100644 --- a/crates/omnigraph/tests/lifecycle.rs +++ b/crates/omnigraph/tests/lifecycle.rs @@ -107,10 +107,106 @@ async fn init_creates_graph() { assert_eq!(db.catalog().node_types.len(), 2); assert_eq!(db.catalog().edge_types.len(), 2); + assert!( + db.catalog().node_types.values().all(|node| node + .arrow_schema + .field_with_name("__omnigraph_stream_v1$") + .is_err()), + "public catalog reflection must omit protocol-private stream metadata" + ); assert_eq!( db.catalog().node_types["Person"].key_property(), Some("name") ); + let person = snap.open("node:Person").await.unwrap(); + assert!(person.schema().field("__omnigraph_stream_v1$").is_none()); + assert!( + person + .load_indices() + .await + .unwrap() + .iter() + .all(|index| index.name != "__lance_mem_wal"), + "public index reflection must omit Lance's private MemWAL system index" + ); + let private_filter = person + .count_rows(Some(r#""__OMNIGRAPH_STREAM_V1$" IS NULL"#.to_string())) + .await + .unwrap_err(); + assert!( + private_filter + .to_string() + .contains("reserved for OmniGraph storage protocol metadata"), + "{private_filter:?}" + ); + assert_eq!( + person + .count_rows(Some( + "name = '__omnigraph_stream_v1$'".to_string(), + )) + .await + .unwrap(), + 0, + "the private spelling remains legal as ordinary user data" + ); + let mut literal_filter = person.scan(); + literal_filter + .filter("name = '__omnigraph_stream_v1$'") + .unwrap(); + literal_filter.try_into_stream().await.unwrap(); + let mut private_projection = person.scan(); + let private_projection = match private_projection.project(&["__OMNIGRAPH_STREAM_V1$"]) { + Ok(_) => panic!("public projection must reject protocol-private stream metadata"), + Err(error) => error, + }; + assert!( + private_projection + .to_string() + .contains("reserved for OmniGraph storage protocol metadata"), + "{private_projection:?}" + ); + let mut structured_private_filter = person.scan(); + structured_private_filter + .filter_expr(datafusion::prelude::col("__omnigraph_stream_v1$").is_null()); + let structured_private_filter = match structured_private_filter.try_into_stream().await { + Ok(_) => panic!("structured filters must not expose protocol-private stream metadata"), + Err(error) => error, + }; + assert!( + structured_private_filter + .to_string() + .contains("reserved for OmniGraph storage protocol metadata"), + "{structured_private_filter:?}" + ); +} + +#[tokio::test] +async fn public_snapshot_wildcard_omits_protocol_metadata() { + let dir = tempfile::tempdir().unwrap(); + let uri = dir.path().to_str().unwrap(); + let mut db = Omnigraph::init(uri, TEST_SCHEMA).await.unwrap(); + omnigraph::loader::load_jsonl( + &mut db, + r#"{"type":"Person","data":{"name":"Alice","age":30}}"#, + omnigraph::loader::LoadMode::Merge, + ) + .await + .unwrap(); + + let snapshot = snapshot_main(&db).await.unwrap(); + let person = snapshot.open("node:Person").await.unwrap(); + let mut scanner = person.scan(); + scanner.project(&["*"]).unwrap(); + let batches: Vec = futures::TryStreamExt::try_collect( + scanner.try_into_stream().await.unwrap(), + ) + .await + .unwrap(); + assert_eq!(batches.iter().map(|batch| batch.num_rows()).sum::(), 1); + assert!(batches.iter().all(|batch| batch + .schema() + .field_with_name("__omnigraph_stream_v1$") + .is_err())); } #[tokio::test] @@ -151,6 +247,26 @@ async fn open_reads_existing_graph() { ); } +#[tokio::test] +async fn open_refuses_a_missing_manifest_selected_stream_token_dataset() { + let dir = tempfile::tempdir().unwrap(); + let uri = dir.path().to_str().unwrap(); + let db = Omnigraph::init(uri, TEST_SCHEMA).await.unwrap(); + drop(db); + + fs::remove_dir_all(dir.path().join("_stream_tokens.lance")).unwrap(); + let error = match Omnigraph::open(uri).await { + Ok(_) => panic!("v9 open must validate its manifest-selected token authority"), + Err(error) => error, + }; + assert!( + error + .to_string() + .contains("stream-token authority consistency failed"), + "{error:?}" + ); +} + #[tokio::test] async fn open_refuses_missing_identity_contract_without_bootstrap() { let dir = tempfile::tempdir().unwrap(); diff --git a/crates/omnigraph/tests/memwal_stream.rs b/crates/omnigraph/tests/memwal_stream.rs index 35b5b07b..09b8dd1a 100644 --- a/crates/omnigraph/tests/memwal_stream.rs +++ b/crates/omnigraph/tests/memwal_stream.rs @@ -22,7 +22,9 @@ use futures::StreamExt; use futures::stream::BoxStream; use lance::Dataset; use lance::dataset::mem_wal::{DatasetMemWalExt, ShardWriterConfig}; +use lance::index::DatasetIndexExt; use lance::io::WrappingObjectStore; +use lance_index::mem_wal::MEM_WAL_INDEX_NAME; use object_store::path::Path as ObjectPath; use object_store::{ CopyOptions, GetOptions, GetResult, ListResult, MultipartUpload, ObjectMeta, ObjectStore, @@ -483,6 +485,39 @@ async fn init_enrolled_with_schema(schema: &str) -> (tempfile::TempDir, Arc RecordBatch { let snapshot = db.snapshot_of(ReadTarget::branch("main")).await.unwrap(); let table = snapshot.open(TABLE).await.unwrap(); @@ -505,6 +540,32 @@ async fn physical_batch(db: &Omnigraph, rows: &[(String, i32)]) -> RecordBatch { RecordBatch::try_new(schema, vec![ids, scores]).unwrap() } +/// Exercise one private B2 compare-and-chain occurrence without exposing its +/// protocol types through the integration-test boundary. +#[allow(clippy::too_many_arguments)] +async fn b2_put_score( + db: &Arc, + stream_incarnation_id: &str, + logical_id: &str, + score: i32, + caller_ordinal: u64, + write_id: &str, + predecessor_token: Option<&str>, + contributor_id: &str, +) -> Result { + let batch = physical_batch(db, &[(logical_id.to_string(), score)]).await; + db.failpoint_stream_b2_for_test( + TABLE, + batch, + caller_ordinal, + stream_incarnation_id, + write_id, + predecessor_token, + contributor_id, + ) + .await +} + async fn physical_payload_batch(db: &Omnigraph, id: &str, payload_bytes: usize) -> RecordBatch { let snapshot = db.snapshot_of(ReadTarget::branch("main")).await.unwrap(); let table = snapshot.open(TABLE).await.unwrap(); @@ -899,7 +960,7 @@ async fn post_invocation_failure_is_ack_unknown_and_replay_is_fold_only() { #[tokio::test] #[serial] -async fn crash_after_table_effect_keeps_old_visibility_then_open_rolls_forward() { +async fn crash_after_base_effect_repairs_token_effect_then_open_rolls_forward() { let _scenario = FailScenario::setup(); let (dir, db) = init_enrolled().await; let batch = physical_batch(&db, &[("recover".to_string(), 33)]).await; @@ -908,11 +969,13 @@ async fn crash_after_table_effect_keeps_old_visibility_then_open_rolls_forward() .unwrap(); let error = { - let _failpoint = - ScopedFailPoint::new(names::STREAM_FOLD_POST_TABLE_COMMIT_PRE_CONFIRM, "return"); + let _failpoint = ScopedFailPoint::new( + names::STREAM_FOLD_POST_BASE_COMMIT_PRE_TOKEN_COMMIT, + "return", + ); db.failpoint_stream_b1_for_test(TABLE, None, 0) .await - .expect_err("the exact table effect must retain recovery-v11 ownership") + .expect_err("the exact base-only effect must retain recovery-v12 ownership") }; assert!( matches!(error, OmniError::RecoveryRequired { .. }), @@ -933,6 +996,44 @@ async fn crash_after_table_effect_keeps_old_visibility_then_open_rolls_forward() ); } +#[tokio::test] +#[serial] +async fn crash_after_both_effects_reconstructs_confirmation_then_open_rolls_forward() { + let _scenario = FailScenario::setup(); + let (dir, db) = init_enrolled().await; + let batch = physical_batch(&db, &[("recover-both".to_string(), 34)]).await; + db.failpoint_stream_b1_for_test(TABLE, Some(batch), 0) + .await + .unwrap(); + + let error = { + let _failpoint = ScopedFailPoint::new( + names::STREAM_FOLD_POST_TOKEN_COMMIT_PRE_CONFIRM, + "return", + ); + db.failpoint_stream_b1_for_test(TABLE, None, 0) + .await + .expect_err("both exact effects without confirmation must retain recovery-v12 ownership") + }; + assert!( + matches!(error, OmniError::RecoveryRequired { .. }), + "{error:?}" + ); + assert!( + visible_rows(&db).await.is_empty(), + "neither exact Lance effect is graph-visible before the manifest CAS" + ); + drop(db); + + let reopened = Omnigraph::open(dir.path().to_str().unwrap()) + .await + .expect("open-time recovery must reconstruct confirmation and roll forward"); + assert_eq!( + visible_rows(&reopened).await, + vec![("recover-both".to_string(), 34)] + ); +} + #[tokio::test] #[serial] async fn fold_sidecar_arm_failure_leaves_no_table_effect_and_retries_the_exact_cut() { @@ -1903,46 +2004,377 @@ async fn strict_fold_validation_failure_keeps_manifest_old_and_stream_fold_only( #[tokio::test] #[serial] -async fn ack_unknown_retry_can_overwrite_a_newer_same_key_without_reconciling_the_attempt() { +async fn b2_happy_fold_exact_retry_and_typed_conflicts_are_effect_free() { let _scenario = FailScenario::setup(); let (_dir, db) = init_enrolled().await; + let incarnation = db + .failpoint_stream_incarnation_for_test(TABLE) + .await + .unwrap(); + let first_write = "11111111-1111-4111-8111-111111111111"; + let token = b2_put_score( + &db, + &incarnation, + "same-key", + 1, + 7, + first_write, + None, + "agent:a", + ) + .await + .expect("one B2 occurrence must cross the complete durability boundary"); + assert!(visible_rows(&db).await.is_empty()); + db.failpoint_stream_b1_for_test(TABLE, None, 0) + .await + .expect("the attributed occurrence must fold through recovery-v12"); + assert_eq!(visible_rows(&db).await, vec![("same-key".to_string(), 1)]); + let manifest_after_fold = db + .snapshot_of(ReadTarget::branch("main")) + .await + .unwrap() + .version(); - let first_x = physical_batch(&db, &[("same-key".to_string(), 1)]).await; - let unknown = { - let _after_durable = ScopedFailPoint::new(names::STREAM_B1_AFTER_WATCHER_SUCCESS, "return"); - db.failpoint_stream_b1_for_test(TABLE, Some(first_x), 7) + // Every outcome below is decided from durable token/base authority. If + // any path reaches Lance, the failpoint turns the mistake into a loud test + // failure instead of letting an accidental second WAL put pass unnoticed. + let _must_not_invoke_put = ScopedFailPoint::new(names::STREAM_B1_BEFORE_PUT_INVOKE, "return"); + + let retry = b2_put_score( + &db, + &incarnation, + "same-key", + 1, + 7, + first_write, + None, + "agent:a", + ) + .await + .expect("an exact durable retry must be idempotent without a second put"); + assert_eq!(retry, token); + + let binding_error = b2_put_score( + &db, + "22222222-2222-4222-8222-222222222222", + "binding-conflict", + 2, + 8, + "33333333-3333-4333-8333-333333333333", + None, + "agent:a", + ) + .await + .expect_err("a stale stream incarnation must fail before WAL invocation"); + match binding_error { + OmniError::StreamBindingChanged { + current_stream_incarnation_id, + .. + } => assert_eq!(current_stream_incarnation_id, incarnation), + other => panic!("expected StreamBindingChanged, got {other:?}"), + } + + let sequence_error = b2_put_score( + &db, + &incarnation, + "same-key", + 2, + 8, + "44444444-4444-4444-8444-444444444444", + None, + "agent:a", + ) + .await + .expect_err("a missing predecessor must fail before WAL invocation"); + match sequence_error { + OmniError::StreamSequenceConflict { + logical_id, + current_token, + .. + } => { + assert_eq!(logical_id, "same-key"); + assert_eq!(current_token.as_deref(), Some(token.as_str())); + } + other => panic!("expected StreamSequenceConflict, got {other:?}"), + } + + let idempotency_error = b2_put_score( + &db, + &incarnation, + "same-key", + 99, + 7, + first_write, + None, + "agent:a", + ) + .await + .expect_err("reusing one occurrence for another payload must fail before WAL invocation"); + match idempotency_error { + OmniError::StreamIdempotencyConflict { + logical_id, + current_token, + .. + } => { + assert_eq!(logical_id, "same-key"); + assert_eq!(current_token, token); + } + other => panic!("expected StreamIdempotencyConflict, got {other:?}"), + } + + assert_eq!(visible_rows(&db).await, vec![("same-key".to_string(), 1)]); + assert_eq!( + db.snapshot_of(ReadTarget::branch("main")) .await - .expect_err("even watcher-success is caller-ambiguous after acknowledgement loss") - }; - assert!( - matches!(unknown, OmniError::AckUnknown { .. }), - "{unknown:?}" + .unwrap() + .version(), + manifest_after_fold, + "exact retries and typed conflicts must be graph-effect-free" ); - db.failpoint_stream_b1_for_test(TABLE, None, 0) +} + +#[tokio::test] +#[serial] +async fn b2_same_generation_chain_folds_only_the_latest_value() { + let _scenario = FailScenario::setup(); + let (_dir, db) = init_enrolled().await; + let incarnation = db + .failpoint_stream_incarnation_for_test(TABLE) .await .unwrap(); - assert_eq!(visible_rows(&db).await, vec![("same-key".to_string(), 1)]); - let newer_y = physical_batch(&db, &[("same-key".to_string(), 2)]).await; - db.failpoint_stream_b1_for_test(TABLE, Some(newer_y), 8) + let first = b2_put_score( + &db, + &incarnation, + "chained", + 10, + 20, + "55555555-5555-4555-8555-555555555555", + None, + "agent:a", + ) + .await + .unwrap(); + let second = b2_put_score( + &db, + &incarnation, + "chained", + 20, + 21, + "66666666-6666-4666-8666-666666666666", + Some(&first), + "agent:b", + ) + .await + .expect("the next occurrence may chain from the confirmed in-generation token"); + assert_ne!(first, second); + assert!(visible_rows(&db).await.is_empty()); + + db.failpoint_stream_b1_for_test(TABLE, None, 0) + .await + .expect("one fold must publish the latest chained winner and token together"); + assert_eq!(visible_rows(&db).await, vec![("chained".to_string(), 20)]); +} + +#[tokio::test] +#[serial] +async fn b2_ack_unknown_retry_cannot_overwrite_a_later_winner() { + let _scenario = FailScenario::setup(); + let (_dir, db) = init_enrolled().await; + let incarnation = db + .failpoint_stream_incarnation_for_test(TABLE) .await .unwrap(); + let x_write = "99999999-9999-4999-8999-999999999991"; + let y_write = "99999999-9999-4999-8999-999999999992"; + let retired = + helpers::failpoint::Rendezvous::park_first(names::STREAM_B1_AFTER_RETIREMENT_RELEASE); + + let error = { + let _after_invoke = + ScopedFailPoint::new(names::STREAM_B1_AFTER_PUT_INVOKE_BEFORE_WATCHER, "return"); + b2_put_score( + &db, + &incarnation, + "ambiguous-chain", + 10, + 40, + x_write, + None, + "agent:x", + ) + .await + .expect_err("post-invocation failure must be AckUnknown") + }; + let candidate_x = match error { + OmniError::AckUnknown { + admission_attempt_id, + logical_write_ids, + unconfirmed_candidate_token, + .. + } => { + let attempt = admission_attempt_id.expect("B2 ambiguity carries its attempt id"); + assert_eq!(attempt.len(), 36, "attempt id must be canonical UUID text"); + assert_eq!( + attempt + .bytes() + .enumerate() + .filter_map(|(index, byte)| (byte == b'-').then_some(index)) + .collect::>(), + [8, 13, 18, 23], + "attempt id must be canonical UUID text" + ); + assert_eq!(logical_write_ids, vec![x_write.to_string()]); + unconfirmed_candidate_token.expect("B2 ambiguity carries its candidate token") + } + other => panic!("expected AckUnknown, got {other:?}"), + }; + assert!(visible_rows(&db).await.is_empty()); + + // Let the detached watcher/abort owner settle before cold replay opens the + // exact durable prefix. The candidate above remains correlation only until + // this fold publishes both base and token authority. + retired.wait_until_reached().await; + retired.release(); db.failpoint_stream_b1_for_test(TABLE, None, 0) .await - .unwrap(); - assert_eq!(visible_rows(&db).await, vec![("same-key".to_string(), 2)]); + .expect("the ambiguous X occurrence must replay and fold exactly once"); - let retry_x = physical_batch(&db, &[("same-key".to_string(), 1)]).await; - db.failpoint_stream_b1_for_test(TABLE, Some(retry_x), 7) + { + let _must_not_put = + ScopedFailPoint::new(names::STREAM_B1_BEFORE_PUT_INVOKE, "return"); + let exact_x = b2_put_score( + &db, + &incarnation, + "ambiguous-chain", + 10, + 40, + x_write, + None, + "agent:x", + ) .await - .expect("B1 has no attribution/idempotency proof that can reject this retry"); + .expect("the exact retry becomes provably durable after fold"); + assert_eq!(exact_x, candidate_x); + } + + let token_y = b2_put_score( + &db, + &incarnation, + "ambiguous-chain", + 20, + 41, + y_write, + Some(&candidate_x), + "agent:y", + ) + .await + .expect("Y must compare-and-chain from X's now-durable token"); db.failpoint_stream_b1_for_test(TABLE, None, 0) + .await + .expect("Y must fold and become the current winner"); + + let stale_x = { + let _must_not_put = + ScopedFailPoint::new(names::STREAM_B1_BEFORE_PUT_INVOKE, "return"); + b2_put_score( + &db, + &incarnation, + "ambiguous-chain", + 10, + 40, + x_write, + None, + "agent:x", + ) + .await + .expect_err("an old exact occurrence cannot displace a later winner") + }; + match stale_x { + OmniError::StreamSequenceConflict { + current_token, .. + } => assert_eq!(current_token.as_deref(), Some(token_y.as_str())), + other => panic!("stale X must be a StreamSequenceConflict, got {other:?}"), + } + assert_eq!( + visible_rows(&db).await, + vec![("ambiguous-chain".to_string(), 20)], + "the ambiguous retry must never overwrite Y" + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[serial] +async fn b2_revalidates_authority_after_waiting_for_shared_admission() { + let _scenario = FailScenario::setup(); + let (_dir, db) = init_enrolled().await; + let incarnation = db + .failpoint_stream_incarnation_for_test(TABLE) .await .unwrap(); + let rendezvous = + helpers::failpoint::Rendezvous::park_first(names::STREAM_B2_AFTER_PROVISIONAL_AUTHORITY); + + // X captures an empty token/base authority, then parks before it owns + // shared admission or the same-key queue. + let stale_db = Arc::clone(&db); + let stale_incarnation = incarnation.clone(); + let stale = tokio::spawn(async move { + b2_put_score( + &stale_db, + &stale_incarnation, + "raced", + 1, + 30, + "77777777-7777-4777-8777-777777777777", + None, + "agent:x", + ) + .await + }); + rendezvous.wait_until_reached().await; + + // Y is the only live owner. It durably appends and folds while X remains + // outside the admission domain, changing both base and token authority. + let winner = b2_put_score( + &db, + &incarnation, + "raced", + 2, + 31, + "88888888-8888-4888-8888-888888888888", + None, + "agent:y", + ) + .await + .expect("the second arrival must pass the park-first rendezvous"); + db.failpoint_stream_b1_for_test(TABLE, None, 0) + .await + .expect("Y must publish before X acquires shared admission"); + + let must_not_put = ScopedFailPoint::new(names::STREAM_B1_BEFORE_PUT_INVOKE, "return"); + rendezvous.release(); + let stale_error = tokio::time::timeout(Duration::from_secs(20), stale) + .await + .expect("stale X did not resume after rendezvous release") + .unwrap() + .expect_err("X must classify against Y's final authority, not its provisional snapshot"); + drop(must_not_put); + match stale_error { + OmniError::StreamSequenceConflict { + logical_id, + current_token, + .. + } => { + assert_eq!(logical_id, "raced"); + assert_eq!(current_token.as_deref(), Some(winner.as_str())); + } + other => panic!("stale X must return StreamSequenceConflict, got {other:?}"), + } assert_eq!( visible_rows(&db).await, - vec![("same-key".to_string(), 1)], - "retrying ambiguous X after durable Y demonstrates the documented overwrite hazard; it does not prove X was reconciled" + vec![("raced".to_string(), 2)], + "the stale provisional capture must neither overwrite nor re-append Y" ); } diff --git a/crates/omnigraph/tests/schema_apply.rs b/crates/omnigraph/tests/schema_apply.rs index 6d71af15..5a577eab 100644 --- a/crates/omnigraph/tests/schema_apply.rs +++ b/crates/omnigraph/tests/schema_apply.rs @@ -49,6 +49,15 @@ async fn plan_schema_reports_supported_additive_change() { .. } if type_name == "Person" && property_name == "nickname" ))); + + let preview = db + .preview_schema_apply_with_options(&desired, omnigraph::db::SchemaApplyOptions::default()) + .await + .unwrap(); + assert!(preview.catalog.node_types.values().all(|node| node + .arrow_schema + .field_with_name("__omnigraph_stream_v1$") + .is_err())); } #[tokio::test] @@ -149,6 +158,25 @@ async fn long_lived_handle_uses_the_schema_catalog_bound_to_its_write_token() { ) ); schema_owner.apply_schema(&desired).await.unwrap(); + assert!( + schema_owner.catalog().node_types.values().all(|node| node + .arrow_schema + .field_with_name("__omnigraph_stream_v1$") + .is_err()), + "schema publication must keep protocol metadata out of the owner's public catalog" + ); + let reopened_after_apply = Omnigraph::open(uri).await.unwrap(); + assert!( + reopened_after_apply + .catalog() + .node_types + .values() + .all(|node| node + .arrow_schema + .field_with_name("__omnigraph_stream_v1$") + .is_err()), + "reopen must reconstruct the same sealed public catalog after schema apply" + ); // The same apply exercises both physical schema shapes: Person is rebuilt // through a staged overwrite for the added property, while Project is a // newly created table incarnation. Neither may drop the immutable v6 PK. diff --git a/crates/omnigraph/tests/warm_read_cost.rs b/crates/omnigraph/tests/warm_read_cost.rs index be3a4e64..ba0745a5 100644 --- a/crates/omnigraph/tests/warm_read_cost.rs +++ b/crates/omnigraph/tests/warm_read_cost.rs @@ -293,9 +293,10 @@ async fn cold_other_branch_resolution_uses_one_coherent_manifest_open() { /// Native branch controls must take one post-gate operation-local coordinator /// capture, not refresh the handle-local coordinator before and after table -/// gates. Delete additionally opens the native main ref once for its exact -/// BranchIdentifier-fenced classifier, but performs no second manifest row -/// scan. +/// gates. Delete additionally takes one fresh manifest-only dependency +/// snapshot for each surviving branch and opens the native main ref once for +/// its exact BranchIdentifier-fenced classifier. The fixture below has one +/// survivor (`main`). #[tokio::test] async fn native_branch_controls_use_one_post_gate_manifest_capture() { cost_harness(async { @@ -315,9 +316,9 @@ async fn native_branch_controls_use_one_post_gate_manifest_capture() { deleted.unwrap(); assert_eq!( (delete_io.internal_open_count, delete_io.manifest_scan_count), - (2, 1), - "branch delete needs one coherent target capture plus one native-ref opener, \ - without a second row scan" + (3, 2), + "branch delete needs one coherent target capture, one fresh manifest-only \ + snapshot for surviving main, and one native-ref opener" ); db.branch_create("feature").await.unwrap(); diff --git a/docs/dev/canon.md b/docs/dev/canon.md index 0179ed24..02e8850e 100644 --- a/docs/dev/canon.md +++ b/docs/dev/canon.md @@ -3,7 +3,7 @@ **Audience:** maintainers, contributors, and coding agents — internal **Type:** narrative reference ("the book"), read top-to-bottom **Status:** living document -**Surveyed:** OmniGraph 0.8.1 development (`main`), with RFC-026 Phase A and the private Phase B1 core present, near-cap fold closure green, and the private B2a unbounded retain-all gate implemented; Lance 9.0.0-rc.1 (git rev `cec0b7df`); internal manifest schema v8 +**Surveyed:** OmniGraph 0.8.1 development (`main`), with RFC-026's private B2 compare-and-chain/token-fold core and unbounded retain-all profile implemented; Lance 9.0.0-rc.1 (git rev `cec0b7df`); internal manifest schema v9 --- @@ -320,7 +320,7 @@ Two mechanisms make concurrent publishes safe: N-writer convergence tests). The internal manifest schema is stamped -(`omnigraph:internal_schema_version`, currently v8) and **strict +(`omnigraph:internal_schema_version`, currently v9) and **strict single-version** — see [the strand model](#schema-and-migration--the-strand-model). --- @@ -486,7 +486,7 @@ writer describes its physical effects to the shared coordinator: | EnsureIndices | v9 (`protocol_v8` payload) | one pre-minted *mixed* CreateIndex transaction per table (every missing BTREE + FTS + full-table vector together) | | Optimize | v9 (bounded payload) | compaction + index folds have **no** public caller-controlled Lance transaction identity, so Optimize keeps looser, bounded provenance inside the identity-bearing envelope: one graph-wide sidecar pinning the complete productive set, one monotonic batch CAS for visibility. Exact provenance is trigger-gated on upstream API + distributed fencing | | StreamEnrollment *(internal Phase A foundation)* | v10 (`protocol_v10` payload) | exact main-only `N -> N + 1` singleton MemWAL initialization plus one pre-minted empty unsharded shard. No effect retires; index-only provisions the shard; the complete state publishes pointer + `OPEN`. Once an effect exists, recovery is roll-forward-only. There is no production enrollment or row caller | -| StreamFold *(private Phase B1 core)* | v11 (`protocol_v11` payload) | one proven drained no-roll generation becomes one exact staged keyed transaction that also records its exact Lance `MergedGeneration`. Confirmation binds the cut, transaction, fixed lineage, table outcome, and refreshed `OPEN` witness/epoch floor before one graph-manifest CAS. It folds already-normalized physical rows, including already-normalized vectors, with no external embedding or unspecified derivation. Only the feature-gated private seam can invoke it | +| StreamFold *(private B2 core)* | v12 (`protocol_v12` payload) | one proven drained no-roll generation becomes an exact staged keyed base transaction carrying its Lance `MergedGeneration` plus an exact staged `_stream_tokens.lance` transaction. The intent binds both prestates/transactions, planned token winners, fixed lineage, lifecycle state-v2, and attribution. Only exact+exact reaches one graph-manifest CAS publishing both pointers and the fold commitment. It folds already-normalized physical rows, including already-normalized vectors, with no external embedding or unspecified derivation. V11 is historical v8 state; only the feature-gated private seam can invoke v12 | First-touch tables (a branch's first write to a table) follow **sidecar-before-ref** ordering: the recovery intent that names the @@ -498,7 +498,7 @@ any pending claim as a hard stop. Every handle for one canonical graph root shares a process-local `WriteQueueManager`: stream-admission domains → schema gate → branch gate → -sorted table gates, one +stream-token gate where applicable → sorted table gates, one deadlock-free order used by writers, healers, and the recovery sweep alike. These gates prevent same-process races and reduce publisher retries. **They are not distributed locks.** Cross-process safety comes from the publisher's exact @@ -511,23 +511,27 @@ row—including `SEALED`—fenced base-table, schema, maintenance, repair-adoption, and recovery effects because no guarded drain/fold witness-update adapter existed. Native branch create/delete was the narrow exception: it did not advance a table HEAD, so it could proceed at `SEALED`; -it still refused `OPEN` or `DRAINING`. Schema v8 preserves those exclusions -and adds only private B1's exact one-generation fold as a guarded HEAD/witness -transition. Private B2a adds no new format or transition: it implements the -unbounded retain-all storage/correctness gate over that v8 core, with no -canonical durable MemWAL deletion. RFC-026's B2-common inventory specifies the -next contract: compare-and-chain -write/predecessor tokens, trusted hidden row attribution, a manifest-selected -current-token participant, durable `OPEN -> DRAINING -> SEALED -> OPEN`, -and bounded `REPLACE`/`WITHDRAW` correction. Managed reclamation is a deferred -Lance-owned optimization, not an activation gate. None of those B2-common -contracts is implemented. Schema -v9/config-v3/state-v2/recovery-v12 -and every public row surface stay inactive until their crash, storage, and -cross-version evidence passes. Phase D owns future automatic operation-scoped -drain and atomic witness advancement/rebind. Resume rechecks the bounded -narrow main-only topology; an incompatible branch operation stays -`SEALED`. +it still refused `OPEN` or `DRAINING`. Historical schema v8 added private B1's +exact one-generation fold as a guarded HEAD/witness transition. Private B2a +added no format or transition: it established the unbounded retain-all posture, +with no canonical durable MemWAL deletion. + +Current schema v9/config-v3/state-v2/recovery-v12 implements the private +compare-and-chain row/fold subset. Canonical payload and token digests plus +trusted hidden row metadata bind each admission. A pre-wait authority capture +is provisional; the adapter recaptures lifecycle/binding/HEAD/token authority +after shared admission and same-key queue ownership. The manifest-selected +`_stream_tokens.lance` participant and base table commit through exact +transactions owned by one v12 intent, and their sole manifest CAS also stores +the fold-attribution commitment. V11 is historical only. Managed reclamation +remains a deferred Lance-owned optimization, not an activation gate. + +The public/control portion remains inactive: explicit production enrollment, +durable `OPEN -> DRAINING -> SEALED -> OPEN` operations, bounded +`REPLACE`/`WITHDRAW` correction, authoritative status, authorization, and every +public row surface. Phase D owns future automatic operation-scoped drain and +atomic witness advancement/rebind. Resume must recheck the bounded narrow +main-only topology; an incompatible branch operation stays `SEALED`. --- @@ -556,11 +560,12 @@ this recovery WAL: Private StreamFold has one deliberate pre-arm prelude: under exclusive admission it seals, drains, retires, and independently proves an immutable -fresh-tier generation before writing the v11 sidecar. That cut is not a -base-table or graph-visible effect and remains fold-only after an armed -no-table-effect failure. The sidecar is still durable before the exact staged -base-table transaction; only confirmed exact `N + 1` may publish the fixed -table, lineage, and refreshed lifecycle witness. +fresh-tier generation before writing the v12 sidecar. That cut is not a +base-table or graph-visible effect and remains fold-only after a proven +effect-free failure. The sidecar is durable before either exact staged Lance +transaction. Only the exact base plus exact token-table outcome may publish the +fixed pointers, lineage, attribution, and refreshed lifecycle witness; exact +base-only recovery may complete only the pre-minted token transaction. The exactness is the security property. Recovery never adopts a foreign commit that happens to sit at the expected version; never reparents a prepared write @@ -585,15 +590,17 @@ When recovery runs: The established writers emit identity-bearing schema-v9 envelopes. RFC-026 Phase A adds the dedicated schema-v10 `StreamEnrollment` envelope; its exact public-state classifier is roll-forward-only once an initializer or shard -effect exists. Private Phase B1 adds schema-v11 `StreamFold`: only exact no -effect or the planned `N + 1` table transaction plus exact MemWAL merge progress -is classifiable, and only `EffectsConfirmed` may publish the fixed table, -lineage, and refreshed lifecycle outcome. The five established graph-visible +effect exists. Current private B2 adds schema-v12 `StreamFold`: both exact +participants and exact MemWAL merge progress are classifiable, and only their +complete confirmed outcome may publish the fixed pointers, lifecycle, lineage, +and attribution. Schema-v11 is retained as historical v8 syntax but is refused +under v9 because it lacks the token participant and complete state-v2 outcome. +The five established graph-visible writer classes map to six `SidecarKind`s because Mutation and Load remain distinct; `StreamEnrollment` is the seventh and `StreamFold` the eighth. Historical payload field names such as `protocol_v3`, `protocol_v4`, `protocol_v7`, `protocol_v8`, `protocol_v10`, and -`protocol_v11` describe retained per-writer payload shapes, not older active +`protocol_v12` describe retained per-writer payload shapes, not older active envelopes. A pre-v9 file without explicit table identity is refused; recovery never infers ownership from an alias or path. @@ -646,7 +653,7 @@ shape is **research-blocked**. Lance tag guards prove exact sparse pins and the branch-delete caveat; the local 10→1,000 decision run rejects activation because compacted list/cleanup scan bytes grow 17,012→38,000 cold and 12,336→15,064 warm (exact show grows too). No checkpoint format or API is active; internal -schema v8 adds no checkpoint state. +schema v9 adds no checkpoint state. ### Three-way merge @@ -712,7 +719,7 @@ enum bumps no table version. Destructive or narrowing changes are refused **Storage versioning is strict single-version** (the strand model, [versioning.md](versioning.md)): this binary reads exactly one internal -manifest schema (`MIN_SUPPORTED == CURRENT == 8`). An older graph is refused +manifest schema (`MIN_SUPPORTED == CURRENT == 9`). An older graph is refused with a self-service export/import rebuild recipe naming the right old release; a newer graph is refused with "upgrade omnigraph". There is deliberately no in-place migration dispatcher — that machinery is permanent liability (every @@ -734,16 +741,18 @@ PK from creation, and every production insert/upsert uses the filter-bearing keyed adapter. Moving from v5 to v6 is rebuild-only; the genuine cross-version binary rebuild/refusal run passed on 2026-07-15. The historical v7 format preserved both contracts and added RFC-026's identity-keyed lifecycle rows plus -recoverable empty enrollment. The currently served v8 format preserves that -Phase-A foundation and adds config-v2 private one-generation admission plus -recovery-v11 fold. V6→v7 and v7→v8 each require export/init/load rebuilds; both -genuine old/new-binary refusal/rebuild strands are green. - -The common B2 inventory assigns the next strict strand—schema v9, stream-config v3, state protocol -v2, and recovery-v12—but does not activate it. V9 is where trusted stream-row -metadata and the manifest-selected current-token dataset would enter the -format. It must refuse/rebuild against v8 in both directions; no serde default -or in-place adoption may manufacture token or contributor evidence. +recoverable empty enrollment. Historical v8 added config-v2 private +one-generation admission plus recovery-v11 fold. + +The currently served v9 format preserves those foundations and activates +stream-config v3, state protocol v2, trusted hidden stream-row metadata, the +manifest-selected `_stream_tokens.lance` authority, and recovery-v12. The +genuine final-v8↔v9 CI strand proves refusal in both directions and strict +export/init/load rebuild. Rebuild preserves logical rows, supplied physical +vectors, and exact-`id` PK metadata; it deliberately does not export trusted +hidden stream metadata or token authority. No serde default or in-place +adoption manufactures token or contributor evidence. Every transition remains +rebuild-only. --- @@ -938,8 +947,9 @@ on the proposer.) primitives. Our recovery sidecars are *intents over Lance commits*, not a parallel log of data. RFC-026 Phase A already consumes Lance's MemWAL initializer for recoverable empty enrollment; private Phase B1 now consumes - its bounded data/ack/replay/fold path rather than building one. Public B2 - remains inactive. The selected profile forbids OmniGraph from deleting raw + its bounded data/ack/replay mechanics rather than building one, and private + B2 adds durable compare-and-chain/token-fold authority over those primitives. + Public B2 remains inactive. The selected profile forbids OmniGraph from deleting raw `_mem_wal` paths and accepts monotonic storage plus loud provider exhaustion; a future managed-reclamation profile would require the missing operation to live in Lance — **(draft RFC-026)**. @@ -986,37 +996,37 @@ recovery's single-writer-process boundary; merge cost at divergence (full-width classification). **Implemented:** substrate-native key-conflict fencing **(RFC-023, internal -schema v6, preserved by v8)**, including the sealed production routing/PK/error/recovery +schema v6, preserved by v9)**, including the sealed production routing/PK/error/recovery contract, the inductive insertion-absence certificate, bounded branch replay, cross-version rebuild/refusal evidence, duplicate-repair runbook, and passed 10K/100K production latency/RSS acceptance series. **Implemented foundation:** RFC-026 Phase A in internal schema v7—recoverable main-only/unsharded empty MemWAL enrollment, identity-keyed lifecycle authority, -process-local admission/exclusion, and strict v6↔v7 refusal/rebuild. Schema v8 +process-local admission/exclusion, and strict v6↔v7 refusal/rebuild. Schema v9 preserves that historical foundation. -**Implemented private core:** RFC-026 Phase B1 in internal schema v8/config-v2 -with a schema-v11 `StreamFold` envelope—one root-scoped, no-roll generation; +**Implemented historical worker core:** RFC-026 Phase B1 in internal schema +v8/config-v2 used a schema-v11 `StreamFold` envelope—one root-scoped, no-roll generation; watcher success plus the same writer's post-durability epoch check before a clean acknowledgement; conservative replay; exact seal/retirement/fold; and atomic table-pointer plus lifecycle-witness publish. The cost, genuine v7↔v8 refusal/rebuild, and graph-level suite remain green. -Gate R0's legal high-entropy near-cap failure is repaired: fold scanning now -charges logical slices and densifies selected rows, so the batch folds and -publishes without lowering the 8,192-row/32-MiB admission cap. The measured +Gate R0's legal high-entropy near-cap failure is repaired: admission, replay, +and fold charge the same logical dense-slice Arrow bytes, and fold densifies selected rows, so the batch folds and +publishes without lowering the 8,192-row/32-MiB logical admission cap. The measured one-exclusive-fold RSS delta is 284,934,144 bytes (~272 MiB); the 384-MiB CI threshold is a remeasurement tripwire, not a runtime allocator limit. B1 is reachable only through a feature-gated private engine seam and folds only already-normalized physical rows/vectors without external embedding or unspecified derivation. -RFC-026 remains Draft: there is no production enrollment, row put/ack/fold, -drain/resume, or fresh-read surface. +V9 preserves the worker/closure mechanics but refuses historical recovery-v11. -**Implemented private B2a retain-all gate and the inactive public contract:** +**Implemented private B2a retain-all profile:** The first profile is unbounded retain-all on stock Lance. OmniGraph performs no MemWAL GC, never deletes a canonical durable MemWAL object, and claims no -retained-byte/file-count bound; provider exhaustion is an accepted loud -operational risk. Lance may remove only a losing manifest-CAS temporary staging +retained-byte, object-count, file-count, or history quota; provider exhaustion +is an accepted loud operational risk. Lance may remove only a losing +manifest-CAS temporary staging object. Complete and partial orphan output remains non-authoritative and is not descended into, read, mutated, adopted, or deleted through retry/reopen, though parent shard discovery may observe its prefix. The local/configured-RustFS @@ -1026,13 +1036,24 @@ adapter, advisory object, and RSS terms. Warm-ack operation shape stays flat while serialized authority and combined retained-history work grow; LIST bytes, wall times, and RSS are diagnostics rather than quotas or SLOs. RC.1's missing durable cross-open materialization-attempt receipt and complete physical-output envelope remain documented limits on any -future bounded-storage claim, not blockers for this profile. No schema or -product surface was activated by B2a. - -The common B2 inventory defines explicit -enrollment, compare-and-chain tokens, trusted visible-winner/current-withdrawal -attribution, persistent lifecycle/correction with monotonic revisions and -bounded terminal management receipts. The retain-all profile adds no storage +future bounded-storage claim, not blockers for this profile. B2a itself +activated no schema or product surface. + +**Implemented private B2 token/fold core; public controls remain inactive:** +Internal schema v9/config-v3/state-v2 adds canonical payload and token digests, +trusted hidden row metadata, exact compare-and-chain/idempotency classification, +same-generation token overlays, and the manifest-selected graph-global +`_stream_tokens.lance` authority. After acquiring shared admission and the +same-key queue, admission recaptures all mutable authority before invoking +Lance. Recovery-v12 owns exact base and token transactions and only their exact +joint result can publish both pointers, lifecycle, lineage, and graph-commit +fold attribution. Recovery-v11 is historical only. The genuine v8↔v9 gate +proves two-way refusal and strict rebuild fidelity while hidden trusted metadata +remains non-exported. + +Explicit production enrollment, persistent lifecycle/correction with monotonic +revisions and bounded terminal receipts, authoritative status, authorization, +and product parity remain inactive. The retain-all profile adds no storage admission watermark. Managed reclamation remains deferred Lance-owned work. Two checked-in RC.1 guards prove generic cleanup does not reclaim @@ -1050,10 +1071,10 @@ physical access shape)**; checkpoint-pinned retention **(RFC-025, research-blocked after Gate 0 rejected the current compacted registry-access shape)**; public MemWAL row admission and later lifecycle/read phases **(RFC-026, draft; -private one-generation Phase B1 implemented and closure-green; private B2a -unbounded retain-all gate implemented on stock RC.1; managed reclamation is optional later work; -public strict activation still requires the common token/attribution, -correction, lifecycle, cross-version, and product-parity evidence)**; +private v9 compare-and-chain/token-fold core and unbounded retain-all profile +implemented; managed reclamation is optional later work; public strict +activation still requires explicit enrollment, correction/status, lifecycle, +authorization, and product-parity evidence)**; lineage-based merge deltas **(RFC-027, research-blocked)**; background reconciler; planner statistics/cost model; policy pushdown; ingest-time embeddings; per-query @@ -1071,7 +1092,7 @@ resource budgets. | **R4: Manifest authority access grows with commit count.** Current-state resolution folds history; a selective index does not by itself bound the complete physical read. | Medium | `optimize` compacts internal tables (keeps periodically-optimized shipped paths flat where separately cost-gated). RFC-024 Gate A rejected durable heads because representative RustFS latest-manifest reads/bytes grow despite flat exact-BTREE row/range work. RFC-025 Gate 0 independently rejected checkpoint-registry activation: at local 10→1,000 on RC.1, uncompacted reconciled work and the eight-fragment tail stay flat, but compacted list/cleanup scan bytes grow 17,012→38,000 cold and 12,336→15,064 warm; exact-show bytes and operation counts also grow. Both RFCs are research-blocked; v8 retains the journal fold and internal-table *cleanup* remains deferred behind the resurrection watermark. | | **R5: Schema identity corruption or alias/identity drift.** Internal schema v5 introduced stable IDs/incarnation as durable authority; v6, v7, and v8 preserve them. | Medium | Open/init validate the SchemaIR domain and exact bidirectional IR↔manifest identity/path/alias contract; every active recovery envelope carries the identity pair; zero, duplicate, missing, or mismatched identity fails closed. | | **R6: Merge cost at divergence** — full-width classification and history-growing manifest folds. | Medium | Coherent coordinator scans plus retained probe handles reduced the pre-slice measured depth-5/depth-80 baseline from 59/651 manifest reads to 40/410 and cap the common fast-forward route at three internal opens and three scans, but the uncompacted-history slope remains. `merge_cost.rs` keeps both facts visible; O(delta) merge is blocked on a real deletion-delta source **(RFC-027)**; fragment adoption is **(draft RFC-0001)**. | -| **R7: No public streaming row path** — production writes are still capped by the `graph_head` CAS rate; high-frequency small writes remain wasteful outside the private evidence seam. | Medium | MemWAL is the strategic substrate. Phase A makes its opaque initializer + separate shard effect recoverable for one main-only/unsharded/single-live-writer-process empty enrollment, durably records lifecycle authority, and excludes competing local effects. Private B1 implements one root-scoped, no-rollover generation with watcher-plus-post-fence acknowledgement, conservative replay, and one recovery-v11 fold. Its legal high-entropy near-cap shape now closes through logical-slice accounting plus dense `take`; the measured one-exclusive-fold RSS delta is 284,934,144 bytes (~272 MiB), with a 384-MiB CI remeasurement tripwire. Private B2a implements the selected unbounded retain-all storage/correctness gate: no canonical durable MemWAL deletion, inert complete/partial orphan roots, typed provider failures, and advisory 1/8/32/128 local/RustFS evidence. It accepts loud provider exhaustion and growing combined retained-history cost without claiming a quota or SLO. Managed reclamation and a whole-root history budget are optional later work. Public activation remains closed on the common token/attribution/lifecycle/correction, private-format, cross-version, and product-parity contracts—not on a storage receipt. A public exact enrollment receipt plus reversible admission seal gates broader overlapping-process topology. | +| **R7: No public streaming row path** — production writes are still capped by the `graph_head` CAS rate; high-frequency small writes remain wasteful outside the private evidence seam. | Medium | MemWAL is the strategic substrate. Phase A makes its opaque initializer + separate shard effect recoverable for one main-only/unsharded/single-live-writer-process empty enrollment. The bounded worker provides watcher-plus-post-fence acknowledgement and conservative replay; its legal high-entropy near-cap shape closes under logical dense-slice accounting, with physical RSS guarded only as a remeasurement tripwire. Private B2a selects unbounded retain-all with no canonical MemWAL deletion or storage quota. Current v9 adds canonical compare-and-chain tokens, trusted attribution, post-admission authority recapture, graph-global token authority, and recovery-v12 exact base+token publication with durable fold attribution; genuine v8↔v9 refusal/rebuild is green. Public activation remains closed on explicit enrollment, lifecycle/correction/status, authorization, and product parity. Managed reclamation and a whole-root history budget are optional later work; a public exact enrollment receipt plus reversible admission seal gates broader overlapping-process topology. | | **R8: Some operations lack enforced memory/time budgets.** | Medium | Known gap, narrowed and accepted for RFC-023. Its direct-substrate instrument rejected the first whole-delta fenced adopt (~447 MB peak at 100K × 256 versus ~74 MB Append), and the first corrected production 10K series failed at 30.0× / 108,625,920 bytes overhead; both negative results remain evidence. Mutation/Load now refuses a keyed table above 8,192 rows / 32 MiB before arm, while BranchMerge uses a recovery-enrolled chain with the same per-chunk bounds and a 1,024-transaction ceiling. The inductive certificate route removes the general diff, temporary delta, target preflight, and target join without weakening that chain. Final five-pair production medians passed at 31/8 ms (3.875×) for 10K and 136/35 ms (~3.886×) for 100K; maximum signed paired RSS overheads were 24,297,472 and 32,604,160 bytes. Inclusive row/transaction ceilings, byte refusal (including materialized blobs), operation-wide validation retention, exact source/target incarnation revalidation, second-generation certificate composition, and both between-chunk recovery directions are pinned; other operations still need explicit bounds. | | **R9: Local-FS conditional-write emulation** (`write_text_if_match` check-then-act gap). | Low | All current callers sit behind the cluster lock protocol; S3 uses true conditional puts; close before admitting any lock-free caller. | | **R10: Doc/spec drift as the system grows** — this document included. | Low | Maintenance contract (same-PR doc updates, `check-agents-md.sh` link CI, "don't lie" stale markers); this canon defers to area docs by construction. | @@ -1099,13 +1120,15 @@ Live design questions, each owned by an RFC or a known gap — not a wishlist: shape must pass the original cold/warm, compacted/uncompacted local and object-store gate; the answer is not to weaken the gate or add a second authority dataset. -4. **Which capability owns the next rebuild boundary after v8?** RFC-028 +4. **Which capability owns the next rebuild boundary after v9?** RFC-028 activated stable identity in v5; RFC-023 assigned exact-`id` fencing to v6; RFC-026 Phase A assigned lifecycle/enrollment authority to v7, and private Phase B1 assigned the data-bearing config-v2/recovery-v11 core to v8. The - common B2 inventory assigns token/lifecycle/correction to schema - v9/config-v3/state-v2/ - recovery-v12, but that strand is not active. + current private B2 row/fold slice assigns config-v3/state-v2, graph-global + token authority, and recovery-v12 to v9; the genuine v8↔v9 gate is green. + Lifecycle management, correction/status, and public surfaces can build on + that format only when their own gates pass; another durable shape change + would require a later strand. RFC-024's heads, RFC-025's retention, and later RFC-026 phases remain independently reviewable. Any later format activation requires its own export/init/load rebuild unless capabilities deliberately co-release after @@ -1138,7 +1161,7 @@ The plan of record is the RFC-022…028 family (all under | [0023 — Key-conflict fencing](../rfcs/0023-key-conflict-fencing.md) | Substrate-native keyed-write fencing via Lance's unenforced-PK filter; fleet/format activation barrier | **Implemented** (2026-07-15) | | [0024 — Durable table heads](../rfcs/0024-durable-table-heads.md) | Materialized head-row research; the first exact-BTREE candidate bounded scan work but failed the full latest-manifest/object-byte cost gate | **Research blocked** | | [0025 — Checkpoint-pinned retention](../rfcs/0025-checkpoint-retention.md) | Named checkpoints as authoritative retention roots, materialized as Lance tags; current in-manifest registry lookup rejected by Gate 0 | **Research-blocked** | -| [0026 — MemWAL streaming ingest](../rfcs/0026-memwal-streaming-ingest.md) | Durability-first streaming writes: one-generation private watcher-plus-post-fence acknowledgement, then graph-atomic fold; legal near-cap closure is green; private B2a retains every canonical durable MemWAL object with no OmniGraph byte/file limit and pins inert orphan/provider/history behavior; common token/attribution/lifecycle/correction contracts remain inactive | **Draft; Phase A/B1 private core and B2a retain-all gate implemented; public inactive** | +| [0026 — MemWAL streaming ingest](../rfcs/0026-memwal-streaming-ingest.md) | Durability-first streaming writes: bounded watcher-plus-post-fence acknowledgement, canonical compare-and-chain tokens and trusted hidden attribution, then recovery-v12 exact base+token graph-atomic fold; legal near-cap closure and genuine v8↔v9 rebuild are green; unbounded retain-all forbids OmniGraph MemWAL deletion; lifecycle/correction/status and product contracts remain inactive | **Draft; private B2 token/fold core and retain-all profile implemented; public inactive** | | [0027 — Lineage merge deltas](../rfcs/0027-lineage-merge-deltas.md) | O(delta) merge classification from row-version lineage | Research-blocked | Deliberately split, not one mega-format: identity, key fencing, head rows, diff --git a/docs/dev/execution.md b/docs/dev/execution.md index 668674d4..40874b39 100644 --- a/docs/dev/execution.md +++ b/docs/dev/execution.md @@ -225,7 +225,7 @@ same pre-arm resource error above 32 MiB without reading payload bytes. Overwrite does accept `WriteParams` and preserves the external reference. `Append` is a user-facing mode name, not the selected Lance operation. On the -current v8 format it preserves the v6 strict-insert contract and routes through +current v9 format it preserves the v6 strict-insert contract and routes through filtered merge-insert with `WhenMatched::Fail`; bare Lance `Append` is unreachable from production graph writes. Use `Merge` when an existing `id` should be updated. This distinction is diff --git a/docs/dev/invariants.md b/docs/dev/invariants.md index 4d26fd8e..76aa5f9a 100644 --- a/docs/dev/invariants.md +++ b/docs/dev/invariants.md @@ -25,7 +25,7 @@ physically stored. > physical state — fragment layout, index coverage, compaction output, and > caches — may lag, retry, or be rebuilt. Pending physical state — including an > unpublished Lance HEAD, staged effects, or acknowledged MemWAL data (including -> the private B1 path) — is not graph-visible, but must remain durable and +> the private RFC-026 path) — is not graph-visible, but must remain durable and > recoverable until publication or a contract-defined terminal disposition; > compensation must preserve every acknowledgement guarantee. Neither category > may silently change logical correctness. @@ -109,7 +109,7 @@ converge the physical state. manifest-committed state. Any weaker mode must be explicit, read-only, auditable, and non-default. A stream-admission acknowledgement is a distinct durability contract, not acknowledgement of a graph-visible write. In the - private B1 profile it requires both successful MemWAL watcher completion and + private RFC-026 profile it requires both successful MemWAL watcher completion and the same `ShardWriter`'s successful post-durability `check_fenced()`; any fence or read ambiguity is `AckUnknown` plus worker retirement. It follows [RFC-026](../rfcs/0026-memwal-streaming-ingest.md) and never weakens @@ -128,9 +128,10 @@ converge the physical state. substitutes for graph-level identity. Internal schema v5 introduced the accepted SchemaIR v2 identity domain and keys manifest ownership, recovery, and physical paths by `(stable_table_id, incarnation_id)`. V6 added RFC-023 - key fencing; v7 added identity-keyed RFC-026 stream-lifecycle authority, and - the currently served v8 format preserves those contracts while adding the - private schema-v8/config-v2/recovery-v11 B1 stream core. See + key fencing; v7 added identity-keyed RFC-026 stream-lifecycle authority; v8 + added the private B1 stream core; and the currently served v9 format adds + config-v3/state-v2, manifest-selected graph-global token authority, and the + private recovery-v12 B2 compare-and-chain/fold core. See [RFC-028](../rfcs/0028-stable-schema-identity.md). 9. **Schema and data integrity failures are loud.** Type, required-field, @@ -190,9 +191,9 @@ converge the physical state. |---|---|---| | Multi-table commit | Manifest CAS plus recovery sidecars; not a single Lance primitive | [writes.md](writes.md), [architecture.md](architecture.md) | | Constructive mutations | In-memory `MutationStaging`, one end-of-query table commit per touched table, then one manifest publish | [writes.md](writes.md), [execution.md](execution.md) | -| Keyed writes | Every current v8 node/edge table declares exact non-null physical `id` as Lance's unenforced PK from creation, using the v6-introduced, v7/v8-preserved exact-`id` fence. General production strict insert and upsert use the sealed exact-`id`, forced-v2 MergeInsert adapter; strict insert exact-probes its pinned parent before minting `omnigraph.insert_absence=v1`, while an all-new upsert may mint it only when completed effect statistics prove one attempt inserted every source row with zero updates, deletes, or skipped duplicates. Certificate admission is optional: BranchMerge uses the shortcut only when every transaction in the complete contiguous source interval carries v1 and its persisted operation is the full pure-insert `Update` shape (exact parent, no removed or updated fragments, nonempty new fragments with `physical_rows`, no field or generation rewrites, `RewriteRows`, exact-`id` filter, full nested schema preorder, and matching physical-row total). It then rechecks both source and target native ref incarnations and passes owned batches through an opaque capability whose one production mint site is structurally guarded. The proven publisher stages immutable fragments with `InsertBuilder`, replaces the uncommitted Append operation with that filter-bearing `Update`, and performs zero target preflights, target merge joins, or committed Appends. Missing, cleaned, unknown, or malformed proof uses the general ordered diff. Mutation/Load remains one keyed transaction per table and rejects more than 8,192 rows or 32 MiB before arm; BranchMerge keeps its bounded v4 chain and exact-recovery limits. Raw Lance graph writers are unsupported, and the certificate is an internal non-cryptographic capability rather than an authenticity mechanism. The final production cost gate passed at 10K (3.875× median; 24,297,472-byte max paired RSS overhead) and 100K (~3.886×; 32,604,160 bytes) | [RFC-023](../rfcs/0023-key-conflict-fencing.md), [writes.md](writes.md), [execution.md](execution.md) | +| Keyed writes | Every current v9 node/edge table declares exact non-null physical `id` as Lance's unenforced PK from creation, using the v6-introduced and later-preserved exact-`id` fence. General production strict insert and upsert use the sealed exact-`id`, forced-v2 MergeInsert adapter; strict insert exact-probes its pinned parent before minting `omnigraph.insert_absence=v1`, while an all-new upsert may mint it only when completed effect statistics prove one attempt inserted every source row with zero updates, deletes, or skipped duplicates. Certificate admission is optional: BranchMerge uses the shortcut only when every transaction in the complete contiguous source interval carries v1 and its persisted operation is the full pure-insert `Update` shape (exact parent, no removed or updated fragments, nonempty new fragments with `physical_rows`, no field or generation rewrites, `RewriteRows`, exact-`id` filter, full nested schema preorder, and matching physical-row total). It then rechecks both source and target native ref incarnations and passes owned batches through an opaque capability whose one production mint site is structurally guarded. The proven publisher stages immutable fragments with `InsertBuilder`, replaces the uncommitted Append operation with that filter-bearing `Update`, and performs zero target preflights, target merge joins, or committed Appends. Missing, cleaned, unknown, or malformed proof uses the general ordered diff. Mutation/Load remains one keyed transaction per table and rejects more than 8,192 rows or 32 MiB before arm; BranchMerge keeps its bounded v4 chain and exact-recovery limits. Raw Lance graph writers are unsupported, and the certificate is an internal non-cryptographic capability rather than an authenticity mechanism. The final production cost gate passed at 10K (3.875× median; 24,297,472-byte max paired RSS overhead) and 100K (~3.886×; 32,604,160 bytes) | [RFC-023](../rfcs/0023-key-conflict-fencing.md), [writes.md](writes.md), [execution.md](execution.md) | | Deletes | Staged like inserts/updates (`stage_delete` via Lance 7.0 `DeleteBuilder::execute_uncommitted`, MR-A) — no inline HEAD advance; mixed insert/update/delete in one query rejected by D2 as a deliberate boundary (constructive XOR destructive per query; compose via separate mutations or a branch) | [query-language.md](../user/queries/index.md), [writes.md](writes.md) | -| Streaming ingest | RFC-026 adopts Lance MemWAL as the strategic substrate and remains Draft; public row streaming is inactive. Historical Phase A schema v7 implemented exact bounded enrollment recovery, identity-keyed lifecycle authority, process-local admission/exclusion, current-HEAD witnesses, compatible-open validation, partial-format refusal, and strict v6↔v7 rebuild for the main-only, one-unsharded-shard, one-live-writer-process profile. Current schema v8/config-v2/recovery-v11 adds the private B1 core behind one feature-gated, doc-hidden engine seam: one normalized physical batch enters one hard-capped 8,192-row/32-MiB generation; watcher success plus the same writer's post-durability `check_fenced()` is required for a clean acknowledgement; replayed or flushed-unmerged residue is fold-only; and one exact strict fold publishes table pointer, lifecycle witness, and lineage only at the `__manifest` CAS. Fold scanning charges each logical slice against the generation cap and copies every scanner emission into dense owned arrays before retaining it. The deterministic 8,192-row high-entropy near-cap cell therefore closes and publishes; its isolated fold RSS delta measured 284,934,144 bytes (about 272 MiB), below the 384-MiB remeasurement tripwire. The private B2a gate now implements unbounded retain-all: no retained-byte, object-count, file-count, or history quota; no deletion of a canonical durable `_mem_wal` object; typed provider failures; and complete or partial unreferenced generation residue kept non-authoritative and untouched below its root through retry and reopen. Parent shard discovery may observe the orphan prefix, and Lance may remove only a losing manifest-CAS `.binpb.tmp.` staging object; neither is adoption or retained-history reclamation. The 1/8/32/128 local/RustFS instrument separates warm acknowledgement, cold replay, fold, visibility, and retained-history terms; its LIST totals, wall times, and RSS are advisory, not quotas or SLOs. Provider exhaustion is loud and may block admission, fold, or recovery; row/memory/deadline/retry/ambiguity bounds remain. RC.1's absent cross-open materialization-attempt receipt and complete physical-output envelope are not activation blockers for this profile, but remain inputs to the future B2b managed-reclamation route. Compare-and-chain tokens, attribution, lifecycle receipts, correction, authorization, and product parity remain unimplemented. There is no production first-use caller, schema/SDK/HTTP/CLI/OpenAPI surface, operator drain/resume, correction, GC, fresh read, or schema v9 | [RFC-026](../rfcs/0026-memwal-streaming-ingest.md), [writes.md](writes.md) | +| Streaming ingest | RFC-026 adopts Lance MemWAL as the strategic substrate and remains Draft; public row streaming is inactive. Historical schema v7 implemented recoverable main-only empty enrollment, and historical v8/config-v2/recovery-v11 added the private B1 worker and one-generation fold mechanics. Current schema v9/config-v3/state-v2 activates the private B2 token/fold slice behind the feature-gated, doc-hidden seam: canonical payload and compare-and-chain token digests bind table identity, logical key, accepted schema, stream incarnation, predecessor, write ID, trusted contributor, and payload; hidden trusted metadata follows the row; and same-generation overlays preserve idempotency and chaining before fold. Admission treats any pre-wait capture as provisional, then recaptures schema, lifecycle, binding, HEAD, and token authority after shared admission and same-key queue ownership, before `put_no_wait`. The graph-global `_stream_tokens.lance` dataset is durable but its raw HEAD is never authority; `__manifest` selects its exact witness. Recovery-v12 owns exact pre-minted base and token transactions, and only their exact joint outcome may publish both pointers, lifecycle state-v2, lineage, and the fold-attribution commitment in one manifest CAS. Recovery-v11 is historical only and is refused under v9 because it lacks the token participant and complete state-v2 outcome. Admission, replay, and fold share the existing 8,192-row/32-MiB logical dense-slice Arrow gate; backing-buffer capacity and physical RSS are evidence, not admission authority. The selected profile remains unbounded retain-all: no retained-byte, object-count, file-count, or history quota; no deletion of a canonical durable `_mem_wal` object; typed provider failures; and inert complete/partial residue. The genuine v8↔v9 gate proves two-way refusal and export/init/load rebuild while preserving logical rows, vectors, and exact-`id` PK metadata; hidden trusted stream metadata is not exported. Explicit production enrollment, revisioned lifecycle management, correction, authoritative status, authorization, SDK/HTTP/CLI/OpenAPI, generation GC, and fresh reads remain inactive. | [RFC-026](../rfcs/0026-memwal-streaming-ingest.md), [writes.md](writes.md) | | Branch create/delete | `__manifest` `BranchContents` is the single logical authority. Lance create is physically two-phase, so OmniGraph prevalidates names, enforces path-prefix-disjoint live graph names, reclaims an absent-ref clone-only tree, and uses a bounded completion classifier; delete removes authority before tree cleanup, so an absent ref is success and derived tree reclaim may converge later. Neither control emits graph lineage. Under schema/source-target/all-table gates, each control uses one operation-local post-gate manifest/namespace capture rather than refreshing the handle-local coordinator around table-gate acquisition; successful ref movement explicitly invalidates derived read caches. Per-table forks are derived state, reclaimed best-effort with `cleanup` as backstop. A target-scoped unresolved sidecar may be made unreachable by deletion and is then audit-discarded by recovery; graph-global SchemaApply still blocks | [branches-commits.md](../user/branching/index.md), [maintenance.md](../user/operations/maintenance.md), [writes.md](writes.md) | | Cleanup retention | Explicit cleanup derives exact `keep` cutoffs from Lance's available version list, caps each main-table GC cutoff at the oldest exact main version inherited by any live lazy graph branch, and refuses uncovered main HEAD drift. Lance protects native per-table branch refs itself. The graph-wide live-reference preflight fails closed before the first table GC, after which individual table failures remain fault-isolated | [maintenance.md](../user/operations/maintenance.md), [writes.md](writes.md) | | Optimize visibility | One operation-local accepted catalog and fresh main snapshot are planned under schema → main → all-table gates. Every productive table shares one identity-bearing v9 recovery envelope; bounded-parallel physical effects become visible through at most one monotonic manifest/lineage CAS. Complete crash residuals roll forward together and partial residuals compensate before visibility. Optimize's payload remains a bounded maintenance adapter rather than an exact caller-minted Lance transaction proof, so it is supported within the single-writer-process recovery boundary. Exact provenance is deferred until Lance exposes a stable public caller-controlled maintenance transaction API and OmniGraph has distributed recovery fencing | [maintenance.md](../user/operations/maintenance.md), [writes.md](writes.md), [RFC-022](../rfcs/0022-unified-write-path.md) | @@ -228,8 +229,8 @@ incomplete would let a legacy writer prepare from unresolved physical state. Do not hide these behind invariant wording. Either move them forward or keep them explicit. -- **Private B1 closure and B2a retain-all gate are implemented; public row - streaming is not active:** RC.1's public +- **Private B2 token/fold core and B2a retain-all profile are implemented; + public row streaming is not active:** RC.1's public initializer commits the singleton MemWAL index internally and shard claim is a separate durable effect. RFC-026 no longer treats an upstream release as the only calendar path: a production-neutral harness proved exact @@ -253,7 +254,7 @@ them explicit. reset after MemTable rollover, `put_no_wait` may mutate before returning an error, replay leaves its BatchStore flush watermark unset, and a late drain waiter can miss - a completed failure. Current schema v8/config-v2/recovery-v11 implements the + a completed failure. Historical schema v8/config-v2/recovery-v11 implemented the bounded private response: one root-scoped worker and one nominally bounded, no-rollover generation per writer; admission held before epoch claim; watcher-backed durability followed by a same-writer post-durability @@ -269,15 +270,39 @@ them explicit. already-normalized physical rows and vectors; it performs no external embedding call or unspecified fold-derived-field materialization. It is not a product surface and performs no GC or correction. - The closure defect found by Gate R0 is repaired at the fold boundary: the - scanner charges logical slice bytes against the same 32-MiB generation limit, - takes every emitted row into dense owned Arrow arrays, and drops the sparse + The closure defect found by Gate R0 is repaired at the fold boundary: + admission, replay, and the scanner all charge + `ArrayData::get_slice_memory_size` logical bytes against the same 32-MiB + generation limit; fold takes every emitted row into dense owned Arrow arrays and drops the sparse parent representation before retaining the batch. The deterministic 8,192-row high-entropy near-cap generation now folds and publishes exactly once. Its isolated fold RSS delta was 284,934,144 bytes (about 272 MiB), below the 384-MiB remeasurement tripwire; crossing that tripwire requires remeasurement rather than a silent admission change. + Current schema v9/config-v3/state-v2 retains those worker mechanics and adds + the private compare-and-chain data plane. Canonical payload/token digests and + trusted hidden metadata bind each admitted key occurrence. B2 preprocessing + is root-bounded before allocation: each 128-MiB envelope covers the original + 32-MiB row, a possible 32-MiB materialized replacement, and 64 MiB of + canonical bytes. Its inflight slot transfers into the queued corridor and + its scratch charge remains until hashing completes; pressure is an + effect-free typed refusal. The private profile admits exactly two envelopes + (256 MiB root-wide), preserving the two-caller stale-authority race without + unbounded preprocessing. Any authority + captured before waiting is provisional: after shared admission and same-key + queue ownership, the adapter recaptures schema, binding, lifecycle, HEAD, and + manifest-selected token authority before invoking Lance. Same-generation + overlays make an exact durable retry idempotent and carry a valid chain before + fold. Recovery-v12 makes the base dataset and graph-global + `_stream_tokens.lance` exact participants under one durable plan. It publishes + only exact+exact, can complete the exact missing token effect after an exact + base effect, and records a commitment to the winning contributors and writes + in graph-commit metadata. Recovery-v11 is historical and cannot recover under + v9 because it lacks that token participant and complete state-v2 authority. + The genuine v8↔v9 gate proves two-way refusal and rebuild fidelity without + exporting trusted hidden metadata. + The private B2a profile implements unbounded retain-all. A flush may write a fresh randomized generation subtree before the shard-manifest CAS, and a cold replay may therefore leave additional materialization roots. OmniGraph @@ -303,12 +328,11 @@ them explicit. complete-output envelope are consequently not blockers for unbounded retain-all. They remain relevant to the future B2b Lance-owned inspect/plan/execute, checkpoint, inventory/accounting, and reclamation - design. Compare-and-chain tokens, trusted attribution, graph-global - current-token authority, lifecycle receipts, strict correction, - authorization, and product parity remain required before a public profile; + design. Explicit production enrollment, lifecycle management receipts, + strict correction/status, authorization, and product parity remain required + before a public profile; a `GraphHistoryBudget` or retained-storage watermark belongs only to a future - bounded/managed profile, not this one. Schema - v9/config-v3/state-v2/recovery-v12 and every product surface remain inactive. + bounded/managed profile, not this one. Every product surface remains inactive. The exact upstream receipt and cross-process seal remain required to broaden topology, not to run the existing private seam inside its current process boundary. See @@ -485,7 +509,7 @@ them explicit. 27→10 rows and 17→10 ranges. The full gate still fails: representative RustFS 20→80 uncompacted cold reads grow 34→94 and bytes 61,947→121,592, while compacted cold/warm byte terms also grow. RFC-024 is - therefore research-blocked and production remains on the current v8 journal + therefore research-blocked and production remains on the current v9 journal fold (the fold design predates and is preserved from v6). Do not reinterpret flat scan counters as a history-flat current-state path. - **Read-path re-derivation (largely closed by the query-latency work):** @@ -576,7 +600,7 @@ case is exceptional. - Cross-query `BEGIN`/`COMMIT` transactions in the OSS engine. Use branches and merges for multi-query workflows. - Acknowledging a graph-visible write before its Lance and manifest persistence, - or acknowledging a private B1 stream append before both its MemWAL durability + or acknowledging a private RFC-026 stream append before both its MemWAL durability watcher and same-writer post-durability epoch check have succeeded. - Silent fallback to eventual consistency, partial results, or dropped rows. - State that drifts from Lance or the manifest when it can be derived. diff --git a/docs/dev/lance-memwal-pr.md b/docs/dev/lance-memwal-pr.md index 0e97096d..4c82fef6 100644 --- a/docs/dev/lance-memwal-pr.md +++ b/docs/dev/lance-memwal-pr.md @@ -54,18 +54,19 @@ number of agents continuously submitting small changes. ### The private MemWAL core exists -RFC-026 Phase A and private Phase B1 are implemented and evidence-green at a -deliberately narrow boundary: +RFC-026 Phase A, private Phase B1, and the private common-B2 token/fold slice +are implemented and evidence-green at a deliberately narrow boundary: - one main-branch, unsharded MemWAL binding per enrolled table; - one live OmniGraph writer process for the graph; - one bounded generation of at most 8,192 rows and 32 MiB; - acknowledgement only after Lance's durability watcher succeeds; - replayed or flushed-but-unmerged state routes to fold only; -- one strict fold stages a normal RFC-022 table effect and publishes it through - `__manifest`; and -- current internal schema v8, stream-config v2, and recovery-v11 describe this - private implementation. +- one strict fold stages exact base-table and `_stream_tokens.lance` + participants and publishes both through `__manifest`; and +- current internal schema v9, stream-config v3, lifecycle state v2, and + recovery-v12 describe this private implementation. Historical v8/config-v2/ + recovery-v11 state crosses that boundary only through export/init/load. The core is intentionally reachable only through a feature-gated, doc-hidden engine seam. There is no `@stream` schema intent, public enrollment, SDK @@ -86,7 +87,7 @@ runtime allocator limit. ### Public streaming is still specified but inactive -RFC-026 Phase B2-0 specifies the remaining public contract: +RFC-026 Phase B2 specifies the remaining public contract: - compare-and-chain write tokens for safe same-key retries; - trusted durable contributor attribution; @@ -97,8 +98,12 @@ RFC-026 Phase B2-0 specifies the remaining public contract: - explicit acceptance of loud provider-capacity exhaustion rather than a hard retained-storage admission promise. -Those contracts are not implemented. Schema v9, stream-config v3, state v2, -recovery-v12, and all public streaming surfaces remain inactive. +The private storage/correctness subset is implemented: schema v9, stream-config +v3, lifecycle state v2, canonical compare-and-chain tokens, trusted hidden row +attribution, manifest-selected token authority, and recovery-v12's exact +base-plus-token publication. Explicit production enrollment, lifecycle +management, correction/status, authorization, SDK/HTTP/CLI/OpenAPI, and every +other product surface remain inactive. ## The missing Lance capabilities @@ -236,7 +241,8 @@ no “missing pointer means empty” fallback, and readers never use a best-effo latest hint as authority. The legacy details kind remains available for existing users and for -OmniGraph's private schema-v8 B1 state. Enabling the new kind is explicit. +OmniGraph's historical schema-v8 B1 state, whose worker mechanics the current +v9 format preserves. Enabling the new kind is explicit. ### 2. Post-success writer fencing @@ -339,13 +345,14 @@ The OmniGraph PR should remain deliberately thin: - prove stock RC.1 refuses the new details kind rather than silently ignoring it or falling back to a latest hint; - preserve structural checks forbidding raw `_mem_wal` deletion in OmniGraph; -- rerun the complete private B1 behavior, crash, race, cost, and cross-version - suites; and +- rerun the complete private B1/common-B2 behavior, crash, race, cost, and + cross-version suites; and - update RFC-026 and the write-path state ledger with the exact reviewed Lance revision and the boundary that has become green. -OmniGraph schema v8 remains the active format during this slice. The new Lance -retention kind is qualified but not yet activated by a production stream. +OmniGraph schema v9 remains the active format during this optional future +slice. The new Lance retention kind is qualified but is not thereby activated +by a production stream. ## Acceptance evidence diff --git a/docs/dev/lance.md b/docs/dev/lance.md index 432222bc..55b438e0 100644 --- a/docs/dev/lance.md +++ b/docs/dev/lance.md @@ -182,7 +182,7 @@ epoch-1 enrollment on main; no production caller can put or acknowledge a row, fold a generation, or operate drain/resume. On 2026-07-19, Phase B1 added the private data-bearing core and moved the -current format to internal schema v8, stream-config v2, and recovery-v11 +then-current format to internal schema v8, stream-config v2, and recovery-v11 `StreamFold`. One feature-gated, doc-hidden engine seam can admit one exact already-normalized physical batch, acknowledge only after that put's durability watcher succeeds, prevent rollover, and retire the writer before any @@ -195,6 +195,15 @@ graph-visible only at one `__manifest` CAS. This is private implementation and evidence machinery, not public activation: RFC-026 remains Draft, with no schema/SDK/HTTP/CLI/OpenAPI caller or operator drain/resume surface. +On 2026-07-22, the private common-B2 slice moved the current format to internal +schema v9, stream-config v3, lifecycle state v2, and recovery-v12. It adds the +grammar-impossible `__omnigraph_stream_v1$` trusted base-row field and one +manifest-selected `_stream_tokens.lance` participant, with compare-and-chain +attribution and exact base-plus-token recovery. V8 crosses this boundary only +through export/init/load; the pinned genuine-v8 cell also proves that a v8 user +property named `__omnigraph_stream_v1` remains ordinary data. This still does +not activate a production SDK/HTTP/CLI streaming surface. + The Phase-B1 source audit also narrows the RC.1 row contract. `put_no_wait` returns a `WriteResult` plus an optional `BatchDurableWatcher`; the watcher's successful `Result<()>` is the only per-put durable-completion signal. @@ -651,7 +660,7 @@ Behavior-affecting findings in this audit: This result blocks the in-manifest BTREE access shape, not checkpoint rows as logical authority or Lance tags as physical pins. RFC-025 is research-blocked and no retention format ships. Schema v6 was production - truth when the gate ran; current schema v8 likewise carries no retention + truth when the gate ran; current schema v9 likewise carries no retention state. A successor needs a history-flat current-authority lookup or revised evidence-backed operational contract without adding a second authority dataset. diff --git a/docs/dev/rfc-022-027-architecture-review.md b/docs/dev/rfc-022-027-architecture-review.md index 23096653..34c050a9 100644 --- a/docs/dev/rfc-022-027-architecture-review.md +++ b/docs/dev/rfc-022-027-architecture-review.md @@ -3,9 +3,11 @@ **Status:** RFC-022, RFC-023, and RFC-028 implemented; RFC-024, RFC-025, and RFC-027 research-blocked; RFC-026 Phase A implemented, private B1's legal near-cap closure repaired and green, private B2a unbounded retain-all gate -implemented, B2b/common contracts inactive, and no public stream surface exists +implemented, the private common-B2 compare-and-chain/token-fold core +implemented, B2b optional, and public streaming plus lifecycle +management/correction/status inactive **Date:** 2026-07-11 -**Last updated:** 2026-07-21 +**Last updated:** 2026-07-22 **Audience:** RFC authors, engine/storage maintainers, and release reviewers **Reviewed against:** OmniGraph 0.8.1; Lance 9.0.0-beta.15 at `f24e42c11a742581365e1cbe17c906ea2dac1bc6`; full Lance transaction, @@ -59,9 +61,13 @@ operation shape stays flat while serialized authority and combined history work grow; LIST bytes, timings, and RSS remain advisory. The missing materialization-attempt receipt and complete envelope therefore do not block B2a. No test-only attempt ledger or `GraphHistoryBudget` is on its immediate path. B2b remains optional future -Lance-owned managed reclamation. Common explicit enrollment, -compare-and-chain tokens, trusted attribution, revisioned lifecycle and -management receipts, correction, authorization, and product parity remain +Lance-owned managed reclamation. The private common-B2 core subsequently +activated internal schema v9, stream-config v3, lifecycle state-v2, trusted +hidden row attribution, canonical payload/token digests, the +manifest-selected graph-global `_stream_tokens.lance` authority, and +recovery-v12 exact base-plus-token fold publication. Genuine v8↔v9 +refusal/rebuild evidence is green. Explicit production enrollment, lifecycle +management, correction/status, authorization, and product parity remain specified, required, and inactive. Two RC.1 surface guards prove generic cleanup non-ownership and the stale-writer hazard created by deleting the successor's empty WAL fence sentinel; neither is an implementation of reclamation. @@ -74,7 +80,7 @@ audit and RFC-025. **RFC-023 substrate evidence revalidated against:** the same beta.21 revision; filter-shape and conflict-order probes are recorded in RFC-023 §2. Internal schema v6 introduced that evidence through exact-`id` fenced production -routing, and current v8 preserves it. Its final insertion-absence certificate/no-target-preflight route and +routing, and current v9 preserves it. Its final insertion-absence certificate/no-target-preflight route and predeclared 10K/100K production series now satisfy the remaining implementation and acceptance gates. **RFC-026 substrate contract revalidated against:** RC.1 at the current revision; @@ -95,7 +101,7 @@ so the candidate and format are research-blocked rather than accepted. RFC-025 Gate 0 was measured on 2026-07-17: Lance tag semantics pass, but the current in-manifest checkpoint-registry BTREE shape has history-sensitive compacted scan bytes and crosses another scan-operation boundary at 1,000 -commits. RFC-025 is therefore also research-blocked; current internal schema v8 +commits. RFC-025 is therefore also research-blocked; current internal schema v9 still contains no retention state. The bucket-gated S3/RustFS cost cell is checked in but was not run for that decision. @@ -497,17 +503,20 @@ history quota. Structural guards, typed local/configured-RustFS provider failures, complete/partial inert orphan evidence, and the 1/8/32/128 advisory history instrument pin that boundary. Lance losing manifest-CAS temp staging is not retained authority. Gate R0's historical no-go applies -only to claiming a finite storage bound on stock RC.1. Public activation remains -closed on the common contracts below, not on the missing materialization-attempt -receipt or physical-output envelope. RC.1 also lacks the ideal caller-owned +only to claiming a finite storage bound on stock RC.1. The private common-B2 +compare-and-chain/token-fold core is implemented in schema v9. Public +activation remains closed on the explicit enrollment, lifecycle-control, +authorization, and product contracts below, not on the missing +materialization-attempt receipt or physical-output envelope. RC.1 also lacks the ideal caller-owned enrollment receipt and cross-process shard-admission seal, but those remain broader-topology gates rather than dependencies of the implemented private single-live-writer-process profile. Gate E0's exact-classification and ABA cells passed using known-version probes and strict object-store classification. Phase A then activated a main-only, unsharded, single-live-writer-process empty enrollment foundation in internal schema v7. Private B1 activated the bounded -data-bearing schema-v8/config-v2 core. RFC-026 remains draft and public row -streaming remains inactive. +data-bearing schema-v8/config-v2 core. The private common-B2 slice activated +schema v9/config-v3/state-v2 and recovery-v12 for exact base-plus-token fold +publication. RFC-026 remains draft and public row streaming remains inactive. Enrollment creates persistent MemWAL metadata and `stream_state` changes the correctness preconditions for schema, branch, maintenance, and data operations. @@ -535,13 +544,14 @@ and carries genuine v7↔v8 rebuild/refusal evidence. B2 is a third strict strand, not an in-place reinterpretation of v8. Private B2a adds no schema change; it closes the stock-v8 storage/correctness gate for -the selected retain-all profile. The common -B2 inventory assigns internal schema v9, stream-config v3, stream-state -protocol v2, and recovery-v12 to trusted hidden row metadata, the -manifest-selected current-token participant, persistent revisioned -lifecycle/correction, bounded terminal management receipts, and their -multi-participant recovery. Those versions are specified but inactive; genuine -v8↔v9 refusal/rebuild remains an implementation gate. The selected unbounded +the selected retain-all profile. The implemented private common-B2 core assigns +internal schema v9, stream-config v3, stream-state protocol v2, and +recovery-v12 to trusted hidden row metadata, canonical payload/token digests, +the manifest-selected current-token participant, and exact base-plus-token fold +publication. Genuine v8↔v9 refusal/rebuild evidence is green. Explicit +production enrollment, persistent lifecycle management/correction/status and +its bounded terminal receipts, authorization, and product parity remain +inactive. The selected unbounded B2a profile adds no storage watermark or `GraphHistoryBudget`; those mechanisms would belong only to a separately justified future bounded/managed profile. @@ -584,17 +594,20 @@ reopen only after publication. `AckUnknown` preserves possible residue through replay but remains permanently caller-ambiguous. Fold now charges each scanner logical slice against the generation cap and takes each emission into dense owned arrays before retaining it; the legal 8,192-row high-entropy near-cap -cell folds and publishes successfully. B2 later owns public strict activation. -The common B2 inventory specifies explicit enrollment, durable trusted -contributor attribution, compare-and-chain tokens, a manifest-selected -current-token participant, bounded `REPLACE`/`WITHDRAW` correction, persistent +cell folds and publishes successfully. The current private B2 core adds durable +trusted contributor attribution, compare-and-chain tokens, a manifest-selected +current-token participant, and exact joint recovery/publication of the base and +token effects. Public strict activation remains a later B2 gate. The remaining +common B2 inventory specifies explicit production enrollment, bounded +`REPLACE`/`WITHDRAW` correction, persistent revisioned status/quiesce/resume/abort-drain/rebuild with bounded management receipts, authorization, and SDK/CLI/HTTP/OpenAPI product parity. B2a selects unbounded retain-all and loud provider exhaustion with no raw WAL deletion or storage/history quota. B2b managed reclamation is an optional future Lance-owned route, not an activation dependency for B2a. The common -implementation plus its crash, cross-version, authorization, and product -evidence remain gates. A `SEALED` B2 stream permits export/rebuild; in-place +private token/attribution/fold implementation and its crash and cross-version +evidence are present; enrollment/control, authorization, and product evidence +remain gates. A `SEALED` B2 stream permits export/rebuild; in-place maintenance still waits for Phase D. The private single-live-writer-process candidate makes that support restriction @@ -1103,7 +1116,7 @@ protected by symmetry in either case. > scalar-indexed/default-v1 path still emits `None`, keyed Append remains > reachable in today's engine, and beta.21 still permits unfiltered Update or > Append to land second after a filtered Update. Internal schema v6 introduced -> the routing closure, and current v8 preserves it: every production +> the routing closure, and current v9 preserves it: every production > insertion-bearing graph path uses > the exact-`id`, forced-v2 keyed adapter, generic Append is test-only, and the > adapter verifies the emitted field-ID filter. Upstream symmetry is therefore @@ -1229,7 +1242,7 @@ The review does not require all RFCs to land together. A safe order is: 4. RFC-024's independent physical lookup evaluation completed on 2026-07-15: the exact BTREE's scan work is flat, but uncompacted RustFS cold object reads/bytes and compacted byte terms grow, so the format is research-blocked - and current production remains on internal schema v8 without table heads; + and current production remains on internal schema v9 without table heads; 5. keep RFC-025 research-blocked after its 2026-07-17 Gate 0 no-go; reconsider only after a history-flat current-authority lookup shape or revised evidence-backed operational contract passes the full physical-I/O boundary, @@ -1247,10 +1260,13 @@ The review does not require all RFCs to land together. A safe order is: quota, surface provider exhaustion loudly, and retain the local/RustFS provider plus 1/8/32/128 advisory evidence. Do not add a materialization-attempt ledger, reserve-first complete physical-output envelope, storage watermark, - or `GraphHistoryBudget` to that immediate path. Implement and prove the - common explicit-enrollment, compare-and-chain-token, trusted-attribution, - revisioned-lifecycle/management-receipt, correction, authorization, and - SDK/CLI/HTTP/OpenAPI parity contracts before exposing B2. Keep B2b + or `GraphHistoryBudget` to that immediate path. Preserve the implemented + private schema-v9/config-v3/state-v2/recovery-v12 compare-and-chain/token-fold + core, including hidden trusted attribution, the manifest-selected + `_stream_tokens.lance` participant, exact joint publication, and genuine + v8↔v9 refusal/rebuild evidence. Implement and prove explicit production + enrollment, revisioned lifecycle management and receipts, correction/status, + authorization, and SDK/CLI/HTTP/OpenAPI parity before exposing B2. Keep B2b Lance-owned managed reclamation as an optional future route with its own evidence and format amendment. Retain the public exact enrollment receipt and cross-process admission seal as the gate for broader topology; diff --git a/docs/dev/testing.md b/docs/dev/testing.md index 1809764e..b42c786d 100644 --- a/docs/dev/testing.md +++ b/docs/dev/testing.md @@ -7,7 +7,7 @@ This file is the always-on map of the test surface. **Consult it before every ta | Crate | Path | Style | |---|---|---| | `omnigraph` (engine) | `crates/omnigraph/tests/` | Integration tests (one file per behavior area — see the table below), fixture-driven, share `tests/helpers/mod.rs` | -| `omnigraph-cli` | `crates/omnigraph-cli/tests/` | Per-area suites (post-modularization): `cli_cluster.rs` (cluster command surface + operator-actor cascade), `cli_cluster_e2e.rs` (spawned-binary lifecycle compositions — lost-state re-import recovery, out-of-band drift, graph-root destruction, multi-graph mixed-disposition convergence), `cli_data.rs` (load/read/change/branch/commit/export/snapshot/policy/embed/maintenance + operator format cascade), `cli_schema_config.rs` (init/config, schema plan/apply), `cli_queries.rs`, `parity_matrix.rs` (RFC-009 Phase 1: the embedded-vs-remote referee — every forked verb run against both arms with matched Cedar policy and the same actor, scrubbed-JSON + exit-code equality; divergences are pinned in its `KNOWN_DIVERGENCES` ledger, never silently repaired), `system_local.rs` (full-cycle cluster lifecycle with a spawned `--cluster` server, applied-policy enforcement over HTTP, keyed-credential auth, operator aliases), `system_remote.rs`, `crossversion_upgrade.rs` (gated genuine v3→v8 and v4/v5/v6/v7↔v8 rebuild/refusal harness — see below); share `tests/support/mod.rs` (hermetic `OMNIGRAPH_HOME` by default) | +| `omnigraph-cli` | `crates/omnigraph-cli/tests/` | Per-area suites (post-modularization): `cli_cluster.rs` (cluster command surface + operator-actor cascade), `cli_cluster_e2e.rs` (spawned-binary lifecycle compositions — lost-state re-import recovery, out-of-band drift, graph-root destruction, multi-graph mixed-disposition convergence), `cli_data.rs` (load/read/change/branch/commit/export/snapshot/policy/embed/maintenance + operator format cascade), `cli_schema_config.rs` (init/config, schema plan/apply), `cli_queries.rs`, `parity_matrix.rs` (RFC-009 Phase 1: the embedded-vs-remote referee — every forked verb run against both arms with matched Cedar policy and the same actor, scrubbed-JSON + exit-code equality; divergences are pinned in its `KNOWN_DIVERGENCES` ledger, never silently repaired), `system_local.rs` (full-cycle cluster lifecycle with a spawned `--cluster` server, applied-policy enforcement over HTTP, keyed-credential auth, operator aliases), `system_remote.rs`, `crossversion_upgrade.rs` (genuine v3→v9 and v4/v5/v6/v7/v8↔v9 rebuild/refusal harness — see below); share `tests/support/mod.rs` (hermetic `OMNIGRAPH_HOME` by default) | | `omnigraph-cluster` | mostly in-source `#[cfg(test)] mod tests`; `tests/failpoints.rs` (feature-gated); `tests/s3_cluster.rs` (bucket-gated full lifecycle on object storage) | Cluster config parser, local JSON state diff, state CAS/lock handling/recovery, read-only validate/plan/status plus explicit refresh/import graph observations, config-only apply (content-addressed payload publish, disposition gating, composite-digest convergence, idempotent re-apply), catalog payload verification (status read-only, refresh drift + self-heal), failpoint crash-mid-apply / CAS-race coverage, Stage 4A graph creation (create executor, recovery sidecars + sweep rows, create crash windows), Stage 4B schema apply (migration previews in plan, schema executor, schema-apply sweep classification, schema crash windows), Stage 4C gated deletes (digest-bound approvals, delete executor + tombstones, delete sweep rows, delete crash windows), and 5A policy binding metadata (applies_to in the applied revision, binding-change diffing + convergence, pre-5A backfill), and the 5B serving-snapshot read API (converged read, refusal rows) | | `omnigraph-server` | `crates/omnigraph-server/tests/` | Per-area suites (post-modularization): `auth_policy.rs`, `data_routes.rs`, `schema_routes.rs`, `stored_queries.rs`, `multi_graph.rs` (cluster-mode boot — converged serving, policy binding wiring, boot refusals — + the concurrent branch-ops matrix), `boot_settings.rs` (mode inference, PolicySource), `s3.rs` (bucket-gated: single-graph serving + config-free `--cluster s3://` boot), `openapi.rs` (OpenAPI drift / regeneration); share `tests/support/mod.rs` | | `omnigraph-compiler` | mostly in-source `#[cfg(test)] mod tests` | Parser, type-checker, IR lowering, lint. Schema parser and SchemaIR validation tests both reject the five exact Lance virtual system-column property names while preserving near-miss identifiers | @@ -25,18 +25,18 @@ The engine's `tests/` is the principal coverage surface; most graph-shaped behav | `writes.rs` | Direct-publish writes: cancellation, RFC-022 non-strict full-attempt reprepare from fresh branch authority, strict stale-write conflicts, multi-statement atomicity, MR-794 staged-write rewire (D₂ rejection, insert+update coalesce, multi-append coalesce, partial-failure recovery, load RI/cardinality recovery); RFC-023 pins the inclusive 8,192-row keyed input ceiling, the same exact/+1 boundary on streamed mutation-update matches, no-effect state for both refusals, and oversized stored-Blob rejection before payload read. Crate-internal pending-scan cells pin inclusive/+1 32 MiB accounting plus pending-key shadow-before-charge. The lance#7444 row-id-overlap regression (`filtered_read_after_merge_update_and_delete_keeps_row_ids_consistent` — merge-load → same-key merge-load → delete → keyed point lookup, green only under the vendored lance-table patch — plus its append-only control) | | `src/table_store/staged_tests.rs` | Crate-internal staged primitives. RFC-023 pins one exact target preflight for general StrictInsert, durable v1 mint/commit/reopen/history persistence, exact-`id` filter emission, typed `KeyConflict`, and missing/wrong PK refusal. `all_new_upsert_certifies_insert_absence_and_persists_it_in_history` proves an all-new completed Upsert receives the optional certificate, a mixed/update Upsert does not, unrelated transaction properties survive, and UUID rebinding does not erase it. Proven-insert cells show the opaque path performs zero strict preflights; stages with `InsertBuilder` but commits the full pure-insert `Update` shape (exact parent and `id` filter, `RewriteRows`, no updates/removals, full nested schema preorder, physical rows); persists/re-admits its own output for proof composition; leaves new fragments outside old index coverage; and fails same-key races loudly in proven/proven and proven/general orders. The in-source `exec/merge.rs` certificate unit table rejects missing/unknown properties, wrong parent/filter/full-preorder/mode/offsets, rewrite/removal shapes, missing `physical_rows`, and Append. Source-interval cells pin exact selection, lazy retained-parent splitting, coalescing, and pinned Lance's approximate raw-emission boundary while every normalized/writer chunk remains hard-capped. Generic `stage_append`/`stage_merge_insert` remain primitive tests only. The file also owns index staging and `commit_staged{,_exact}` | | `forbidden_apis.rs` | Defense-in-depth syntax-tree/source guard over the whole engine. The primary boundary is Rust visibility: raw storage/coordinator/handle-cache modules are crate-private; public `Snapshot::open` returns `SnapshotTable`; and `SnapshotScanner` executes reads without exposing Lance's raw scanner or physical plan. The guard pins those visibility/return-type boundaries, classifies public async inherent `Omnigraph` methods plus loader conveniences, classifies every crate-visible async method on `GraphCoordinator` / `ManifestCoordinator`, and exact-counts registered method/UFCS durable-call shapes including recovery. RFC-023 rejects production graph call sites of generic `stage_append{,_stream}` and `proven_insert_capability_has_one_production_mint_site` pins `ProvenInsertChunk::from_verified_history` to the complete-history classifier in `exec/merge.rs`, preventing the no-preflight capability from becoming a reusable bypass. At the RFC-026 Phase-A checkpoint the guard registered only the exact v10 enrollment gateway and feature-gated test seam, counted its sidecar/index/shard durability primitives, and kept every row-put/ack/fold surface absent; the Phase-B1 owner below now exact-counts only the approved crate-private put/fold durable-call sites while retaining the absence of public schema, SDK, HTTP, CLI, and OpenAPI side doors. B2a inventories the only production `_mem_wal` literal owners, forbids MemWAL reclamation/adoption symbols and destructive primitives in the adapter, keeps generic maintenance unaware of MemWAL, and keeps raw inventory/classifier helpers private. This remains defense in depth rather than macro expansion, alias, or data-flow analysis; the visibility boundary and behavior tests are still primary. It also counts selected raw `SnapshotHandle` / Dataset shapes, rejects renamed-owner/macro/include/path-lookalike forms, skips structurally test-only code, and pins retired escape hatches absent. `// forbidden-api-allow: ` exempts reviewed inline-Lance lines only | -| `lance_surface_guards.rs` | Pins the Lance API surfaces omnigraph depends on (named runtime + compile-only guards; see [lance.md](lance.md)) — the first smoke check on any Lance version bump. `cached_and_zero_cache_sessions_share_store_registry_not_metadata_cache` proves a cached data Session and zero-cache control Session reuse one live `ObjectStoreRegistry` client while their metadata caches remain isolated. `_compile_uncommitted_full_table_vector_index_shape` pins the public `IndexMetadata` shape suitable for `Operation::CreateIndex`; `compact_files_succeeds_on_blob_columns` pins blob-v2 compaction; Guard 9 pins clone-only branch reclaim semantics. RFC-023's `unenforced_pk_filter_shape_is_route_dependent` explicitly forces v2 versus indexed routes and pins the `Some(populated)` / `Some(empty)` / `None` key-filter shapes; `unenforced_pk_conflict_matrix_is_directional` pins the directional filtered/unfiltered and filtered/Append matrix. RFC-024's compile guard pins the public `BranchIdentifier` + current table version + current `Transaction.uuid` + `ManifestLocation.e_tag` current-HEAD witness; the local/shared-`Session` guard proves unchanged-reopen stability, ordinary-commit movement, and same-version ABA, while RustFS covers object-store ABA. RFC-025 adds exact main/named-branch tag-target, sparse cleanup pin/unpin, and branch-tree-deletion guards. RFC-026 pins doc-hidden `has_successor_version`, initializer/readback/shard-writer/durability/fencing, flush/drain, replay watermark, scanner, and merged-generation shapes; runtime Gate E0 classification belongs to `memwal_enrollment_gate.rs`, while v7 Phase-A and v8 B1 publication/recovery belong to the manifest/failpoint suites. B2b adds `cleanup_old_versions_does_not_reclaim_mem_wal_objects` and `mem_wal_deleted_fence_slot_allows_stale_writer_success_on_pinned_lance`: the first proves generic cleanup leaves the present MemWAL fixture unchanged and the second proves deleting the successor's empty fence sentinel is unsafe. The pinned source audit, not those two tests alone, establishes that stock RC.1 exposes no owned MemWAL reclamation API. The RC.1 compiler guard pins the five surveyed public Lance virtual system-column constants to early `.pg` rejection. These guards prove substrate shapes/tokens and negative ownership boundaries; they do not by themselves prove heads/checkpoint activation, the v8 publisher, or a safe reclamation implementation | +| `lance_surface_guards.rs` | Pins the Lance API surfaces omnigraph depends on (named runtime + compile-only guards; see [lance.md](lance.md)) — the first smoke check on any Lance version bump. `cached_and_zero_cache_sessions_share_store_registry_not_metadata_cache` proves a cached data Session and zero-cache control Session reuse one live `ObjectStoreRegistry` client while their metadata caches remain isolated. `_compile_uncommitted_full_table_vector_index_shape` pins the public `IndexMetadata` shape suitable for `Operation::CreateIndex`; `compact_files_succeeds_on_blob_columns` pins blob-v2 compaction; Guard 9 pins clone-only branch reclaim semantics. RFC-023's `unenforced_pk_filter_shape_is_route_dependent` explicitly forces v2 versus indexed routes and pins the `Some(populated)` / `Some(empty)` / `None` key-filter shapes; `unenforced_pk_conflict_matrix_is_directional` pins the directional filtered/unfiltered and filtered/Append matrix. RFC-024's compile guard pins the public `BranchIdentifier` + current table version + current `Transaction.uuid` + `ManifestLocation.e_tag` current-HEAD witness; the local/shared-`Session` guard proves unchanged-reopen stability, ordinary-commit movement, and same-version ABA, while RustFS covers object-store ABA. RFC-025 adds exact main/named-branch tag-target, sparse cleanup pin/unpin, and branch-tree-deletion guards. RFC-026 pins doc-hidden `has_successor_version`, initializer/readback/shard-writer/durability/fencing, flush/drain, replay watermark, scanner, and merged-generation shapes; runtime Gate E0 classification belongs to `memwal_enrollment_gate.rs`, while v7 Phase-A and v8 B1 publication/recovery belong to the manifest/failpoint suites. B2b adds `cleanup_old_versions_does_not_reclaim_mem_wal_objects` and `mem_wal_deleted_fence_slot_allows_stale_writer_success_on_pinned_lance`: the first proves generic cleanup leaves the present MemWAL fixture unchanged and the second proves deleting the successor's empty fence sentinel is unsafe. The pinned source audit, not those two tests alone, establishes that stock RC.1 exposes no owned MemWAL reclamation API. The RC.1 compiler guard pins the five surveyed public Lance virtual system-column constants to early `.pg` rejection. These guards prove substrate shapes/tokens and negative ownership boundaries; they do not by themselves prove heads/checkpoint activation, the current publisher, or a safe reclamation implementation | | `memwal_enrollment_gate.rs` | RFC-026's green production-neutral Gate E0 harness, isolated from the production manifest and graph writer. Fourteen substantive local cells plus one explicit unconfigured-S3 skip cover exact no-effect / `N + 1` index / pre-minted empty-shard classification, buried-effect refusal, marker survival, strict inventory/error handling, and the broad fail-closed matrix. The rejected first instrument used `checkout_latest` plus `IOTracker`, which missed local `read_dir`. The accepted exact-version classifier pins doc-hidden `has_successor_version`; its `AttemptTracker` records failed/`NotFound` attempts before forwarding and proves the identical complete six-attempt shape at baseline versions 8/80: four successful manifest HEADs, one `NotFound` manifest HEAD, one successful manifest GET, zero lists. A Unix execute-only `_versions` tripwire proves exact probing works when latest enumeration fails and an unreadable exact HEAD errors. The configured RustFS exact cell passes non-vacuously with the same zero-list shape and owns the positive lost-result/index/empty-shard/reopen sequence plus foreign shard, malformed/loose root, durable WAL, persisted cursor, and corrupt-manifest negatives. S3 ABA remains in `lance_surface_guards.rs`; CI rejects skipped E0/ABA cells. This file never mutates production manifest/schema state or deletes ambiguous artifacts; Phase A consumes its classifier through the private adapter | -| `memwal_stream.rs` | Feature-gated RFC-026 private B1 behavior plus B2a provider-failure evidence. B1 owns bounded put/ack/replay/fold, authority, cancellation, crash, and manifest-only visibility. B2a injects a recording/failing store at the real Lance table-store boundary and covers effect-free cold-claim failure, post-invocation WAL ambiguity, complete post-cut unreferenced generation output, and partial generation output after the data object. Exact local and configured-RustFS cells prove typed `Lance` / `AckUnknown` / `RecoveryRequired` outcomes, blocked fresh progress, one fresh authoritative retry root, retained non-authoritative residue, and zero access below the orphan root through retry and cold reopen. Parent shard discovery may observe the common prefix; it may not descend into or adopt the subtree. | +| `memwal_stream.rs` | Feature-gated RFC-026 private B1 mechanics, implemented private B2-common token/fold behavior, and B2a provider-failure evidence. B1 owns bounded put/ack/replay/fold, authority, cancellation, and manifest-only visibility. B2-common owns compare-and-chain/idempotency conflicts, same-generation overlays, stale-authority recapture after shared admission, exact base+token fold, recovery-v12 crash completion, and durable attribution. B2a injects a recording/failing store at the real Lance table-store boundary and covers effect-free cold-claim failure, post-invocation WAL ambiguity, complete post-cut unreferenced generation output, and partial generation output after the data object. Exact local and configured-RustFS cells prove typed `Lance` / `AckUnknown` / `RecoveryRequired` outcomes, blocked fresh progress, one fresh authoritative retry root, retained non-authoritative residue, and zero access below the orphan root through retry and cold reopen. Parent shard discovery may observe the common prefix; it may not descend into or adopt the subtree. | | `memwal_stream_cost.rs` | Feature-gated RFC-026 B1, Gate-R0, and B2a decision instrument. It separately measures warm already-claimed durability acknowledgement, cold replay, selected-generation fold scanning, visibility, retained merged metadata, the uncompacted graph-manifest term, legal no-roll estimates, and paired peak RSS. Gate R0 adds a revision-pinned source-audit tripwire, strict current-object classification/reference census, listed path/class/size retain-all comparisons at one/four/eight folds, referenced-cut retry reuse, and deterministic high-entropy near-cap local/configured-RustFS cells. The near-cap cell proves that an admitted 8,192-row/near-32-MiB generation acknowledges without graph visibility, then folds and publishes exactly once after logical-slice charging plus dense per-scanner-batch take. The isolated fold RSS delta measured 284,934,144 bytes (about 272 MiB), below a 384-MiB remeasurement tripwire; that tripwire is not a runtime allocator limit. B2a adds 1/8/32/128 local and configured-RustFS retained-history sweeps whose terms remain separate: warm ack, cold reopen/replay, fold, visibility, table/graph-manifest/adapter work, advisory current-object bytes, and whole-process peak RSS. Older retained roots must receive zero reads, writes, or deletes. The only allowed delete shape is Lance's losing manifest-CAS `.binpb.tmp.` staging; canonical durable MemWAL delete requests remain zero. LIST totals, wall times, and RSS are advisory—not a quota, SLO, isolated WAL slope, or provider billing. A green test proves private closure/retention behavior; it does not activate a public API. | | `durable_head_lookup_cost.rs` | RFC-024 Gate A decision instrument, isolated from the production manifest schema/publisher. At fixed catalog width 10 it runs the full absent/reconciled/one-uncovered/eight-uncovered/reconciled-after-tail matrix over compacted and uncompacted histories, with cold-open and warm-repeat measurements on local FS and bucket-gated S3/RustFS. Default depths are 20/80; the ignored decision-scale cell runs 10/100/1,000. Correct exact heads, flat indexed `rows_scanned`/range work, an index-absent growing negative control, and observable bounded tails all pass; after the eight-fragment tail, `optimize_indices` returns coverage to zero uncovered and representative `rows_scanned`/range work from 27→10 / 17→10. The test deliberately pins the no-go: uncompacted RustFS cold object reads/bytes and compacted byte terms grow, while RC.1 also crosses a bounded one-operation boundary by 1,000 commits, so RFC-024 remains research-blocked. `rows_scanned` is an RC.1 debug proxy, not a universal decoded-row counter. Object-store wrapper bytes and Lance execution-summary bytes are separate fixture-owned metrics and are not additive | -| `checkpoint_retention_cost.rs` | RFC-025 Gate 0 decision instrument, isolated from the production manifest schema. It models three live checkpoints at catalog width 10 and measures complete list, exact show, and cleanup-root authority reads across absent/reconciled/eight-uncovered index states, compacted/uncompacted layouts, and cold/warm access. It also owns the reference V1 name-normalization matrix. Default local depths 20/80 pass the checked-in **no-go-preservation** assertions; the RC.1 ignored 10/100/1,000 run shows reconciled uncompacted work and the bounded tail flat, but rejects the current format shape after compaction: list/cleanup scan bytes grow 17,012→38,000 cold and 12,336→15,064 warm; show grows 29,348→53,064 and 24,672→30,128; scan operations add one at 1,000. The S3/RustFS cell is bucket-gated and was not run for this decision. The result keeps RFC-025 research-blocked; current v8 adds no checkpoint state | +| `checkpoint_retention_cost.rs` | RFC-025 Gate 0 decision instrument, isolated from the production manifest schema. It models three live checkpoints at catalog width 10 and measures complete list, exact show, and cleanup-root authority reads across absent/reconciled/eight-uncovered index states, compacted/uncompacted layouts, and cold/warm access. It also owns the reference V1 name-normalization matrix. Default local depths 20/80 pass the checked-in **no-go-preservation** assertions; the RC.1 ignored 10/100/1,000 run shows reconciled uncompacted work and the bounded tail flat, but rejects the current format shape after compaction: list/cleanup scan bytes grow 17,012→38,000 cold and 12,336→15,064 warm; show grows 29,348→53,064 and 24,672→30,128; scan operations add one at 1,000. The S3/RustFS cell is bucket-gated and was not run for this decision. The result keeps RFC-025 research-blocked; current v9 adds no checkpoint state | | `warm_read_cost.rs` | Cost-budget tests for the warm read/control path (query-latency work), measured at the object-store boundary with Lance `IOTracker` (the LanceDB IO-counted pattern): a warm same-branch read does 0 manifest opens, 1 version probe, validates the schema once (Fix 1 / finding A / Fix 2 at commit-history depth); a cold other-branch resolution derives snapshot state and lineage from one coherent manifest open/scan; native branch create and create-from each use one post-gate open/scan, while delete uses one target capture plus one native-ref opener and only one row scan; stale same-branch reads perform exactly 2 probes and refresh manifest-only; recreated non-main branches with the same Lance version refresh by incarnation; recreated branch-owned table handles are distinguished by table e_tag or refresh-time cache clearing; recreated traversal topology is protected by per-edge-table e_tag in the graph-index cache key or refresh-time cache clearing; a warm *repeat* read does 0 table opens via the held-handle cache and a write re-opens only the changed table at its new version/e_tag (Fix 3/6A). Also the CSR topology-build cost guards: `fresh_branch_traversal_reuses_main_graph_index` (A1 — a lazy-fork branch reuses main's cached CSR index, 0 rebuilds via `graph_build_count`) and `single_edge_query_builds_only_referenced_edge` (A2 — a one-edge query builds only that edge via `graph_edges_built`); both force CSR via the scoped `with_traversal_mode` seam, so they need no `#[serial]`. See "Cost-budget tests" below. | | `write_cost.rs` | Cost-budget tests for the WRITE path (RFC-013), the latency twin of `warm_read_cost.rs` on the **shared `helpers::cost` harness** (`measure`/`IoCounts`/`assert_flat`/`local_graph`). Runs on **local FS**; gates the **internal-table** term (`__manifest` scans flat in commit-history depth, lineage rows included — `internal_table_scans_are_flat_in_history`, now **green every-PR** since RFC-013 step 2 brought the internal tables into `optimize`; the test compacts at each depth before measuring), graph-visible maintenance arbitration (`ensure_indices_manifest_reads_are_flat_in_history` and `optimize_manifest_reads_are_flat_in_history`), plus green every-PR guards (single-insert `data_writes` bounded, a per-write read-op ceiling that fails the moment a round-trip is added, and a `measure_with_staged` fitness assert that a keyed insert routes through the exact-`id` fenced adapter once with no bare `stage_append`/vector-index build). Also gates the batched committed `@unique` probe: `unique_probe_io_is_flat_in_delta_rows` sweeps DELTA size (4 vs 64 rows) at fixed shallow history and asserts `data_open_count`/`data_scan_reads` flat — red when the cross-version probe regresses to per-row scans/opens. The **data-table opener** term is S3-only — see `write_cost_s3.rs` and the backend-split note in "Cost-budget tests" below. RFC-023's representative row-count and peak-RSS decision measurements use the scenario harness, not this every-PR I/O budget | | `write_cost_s3.rs` | Bucket-gated (skips without `OMNIGRAPH_S3_TEST_BUCKET`) twin of `write_cost.rs` on the same `helpers::cost` harness: gates the **data-table opener** term (per-write latest-version resolution flat across commit depth on a real object store — per-version GETs are invisible on local FS). A cost gate, not a correctness test — run on demand, not in the every-merge `rustfs_integration` job (see the backend-split note in "Cost-budget tests" below) | | `helpers/cost.rs` | The shared cost-budget harness (not a test): `IoCounts`/`StagedCounts` (counts by table class), `measure`/`measure_with_staged` (the one place the `with_query_io_probes` + `MergeWriteProbes` task-local + `IOTracker` wiring lives; reads per-op deltas via lance's `incremental_stats()`, the upstream per-request idiom from `rust/lance/src/dataset/tests/dataset_io.rs`), `cost_harness`/`GraphIoMeter` (installs ONE `__manifest` `IOTracker` for a whole test body so the graph opens **under** it and `manifest_reads` is **ground truth** — every read regardless of handle age, the warm-coordinator freshness probe included — closing the blind spot where a per-op tracker installed at measure time cannot see a long-lived handle's reads; outside `cost_harness`, `measure` falls back to fresh per-op tracking, so `write_cost_s3.rs` is unaffected), `open_tracked_lance_dataset` (attaches a caller-owned `IOTracker` before `DatasetBuilder::load`, so a cold-open fixture includes latest-manifest resolution), `last_manifest_reads()` (the manifest read log for `assert_io_eq!`-style failure diagnostics), `assert_flat(curve, select, slack, what)`, and store-agnostic `local_graph`/`s3_graph` fixtures. The general `IoCounts` vocabulary remains operation counts; RFC-024's decision fixture owns its object/plan byte metrics. `warm_read_cost.rs`, `write_cost.rs`, `write_cost_s3.rs`, and the RFC-024 instrument consume the relevant seams | | `benchmark_scenario_contract.rs` | Source/protocol contract for the non-CI scenario harness. RFC-023 pins the production route's explicit `strict_insert_preflight_calls == 0` assertion and emitted `probe_strict_insert_preflight_calls` field, alongside route labels, clean-tree/binary identity, child-protocol refusal, and exact-content verification fields. A benchmark record therefore cannot silently claim the proven path after paying a target preflight | -| `lifecycle.rs` | Graph lifecycle and schema state, including the v6-origin creation invariant—preserved through current v8—that every fresh node/edge table declares exactly physical `id` as its Lance unenforced PK | +| `lifecycle.rs` | Graph lifecycle and schema state, including the v6-origin creation invariant—preserved through current v9—that every fresh node/edge table declares exactly physical `id` as its Lance unenforced PK | | `point_in_time.rs` | Snapshots, time travel (`snapshot_at_version`, `entity_at`) | | `changes.rs` | `diff_between` / `diff_commits` | | `consistency.rs` | Cross-table snapshot isolation and atomic publish; RFC-023 cells prove `LoadMode::Append` is strict (existing `id` rejected without update/version movement), pin the inclusive 8,192-row load ceiling with a one-over pre-effect refusal, reject an input above 32 MiB through the shared Mutation/Load staging seam with raw table HEAD/manifest/sidecar unchanged, reject an oversized external blob on a lazy branch from object metadata before payload access/ref creation/sidecar arm, and use a barrier-synchronized stress cell over 16 pre-opened handles to prove one same-key winner, 15 typed `KeyConflict` losers, exactly one stored row carrying the winner's value, and survival of disjoint IDs | @@ -50,7 +50,7 @@ The engine's `tests/` is the principal coverage surface; most graph-shaped behav | `ordering.rs` | ORDER BY contract: descending, multi-key precedence, deterministic key-column tie-break (total order, so `ORDER … LIMIT` is deterministic), NULL placement (`nulls_first = !descending`) | | `literal_filters.rs` | Execution goldens for non-string/non-integer scalar literal filters (F64/F32/Bool/Date/DateTime) across both the in-memory comparison arm and the Lance-pushdown arm | | `aggregation.rs` | `count`, `sum`, `avg`, `min`, `max` | -| `export.rs` | NDJSON streaming export filters; RFC-023's blob fixture also performs a later `LoadMode::Append` strict insert into a populated current-v8 table (preserving the v6 PK contract) and verifies both exact blob bytes and exact-`id` PK metadata afterward. `export_jsonl_round_trips_branch_snapshot` separately exports `main` and a named feature branch, rebuilds each into a main-only graph, and proves independent identity domains plus disjoint, self-contained histories | +| `export.rs` | NDJSON streaming export filters; RFC-023's blob fixture also performs a later `LoadMode::Append` strict insert into a populated current-v9 table (preserving the v6 PK contract) and verifies both exact blob bytes and exact-`id` PK metadata afterward. `export_jsonl_round_trips_branch_snapshot` separately exports `main` and a named feature branch, rebuilds each into a main-only graph, and proves independent identity domains plus disjoint, self-contained histories | | `s3_storage.rs` | S3-backed graph (skipped unless `OMNIGRAPH_S3_TEST_BUCKET` is set). Includes `s3_fresh_branch_traversal_reuses_main_graph_index_with_etags` — the CSR topology cache-key test on a **real** per-table e_tag (`None` on local FS, so `warm_read_cost.rs` can't reach this path); forces CSR via the scoped `with_traversal_mode` seam | | `lance_version_columns.rs` | Per-row `_row_last_updated_at_version` behavior | | `validators.rs` | Schema constraint enforcement (enum, range, unique, cardinality) across JSONL load, mutation insert/update. ALL THREE write surfaces — mutation, bulk load, AND merge — route through the unified `crate::validate` evaluator (Δ-scoped, index-backed, reusing these leaf checks). Cross-version-uniqueness closure: `cross_version_unique_rejected_on_mutation_insert` + `reinsert_existing_key_is_upsert_not_unique_violation` (mutation path); `cross_version_unique_rejected_on_append_load` + `merge_load_reupsert_existing_key_is_not_unique_violation` (load path). Per-table `Overwrite`: `overwrite_load_validates_ri_against_new_image` (an edges-only overwrite still resolves RI against retained committed nodes) + `append_load_rejects_orphan_edge`. The evaluator's own unit tests live in `src/validate.rs` (`#[cfg(test)]`); its merge-conflict equivalence is pinned by `merge_truth_table.rs` (OrphanEdge) + `branching.rs` (Unique/Cardinality merge tests). Intra-batch duplicate-`@key` rejection on every load mode is pinned by `consistency.rs::loader_rejects_intra_batch_duplicate_keys`; the mutation-coalesce counterpart (insert+update / chained updates of one id are NOT a self-collision) by `writes.rs`. Non-String `@unique` columns probe committed state with a TYPED literal (not a stringified key): `cross_version_unique_rejected_on_date_column` + `noncolliding_write_to_date_unique_column_succeeds` (a `Date @unique` collision is a proper `@unique` violation, and a distinct value does not raise a Date32-vs-Utf8 coercion error). Cardinality is keyed by edge id, last-wins (matching commit's `dedupe_merge_batches_by_id`): `merge_load_edge_src_move_rechecks_vacated_src_cardinality` (a Merge-load moving an edge recounts the vacated src for `@card` min) + `merge_load_duplicate_edge_id_counts_once_per_card` (a dup edge id under two srcs in one batch counts once, no spurious max violation). Direct deletes capture the ids they remove (from the delete op's own scan) into the change-set's `deleted_ids`, so a delete emptying a src is validated: `mutation_delete_edge_below_card_min_rejected` (a `delete Edge` dropping a src below `@card` min is rejected, not silently committed). | @@ -60,7 +60,7 @@ The engine's `tests/` is the principal coverage surface; most graph-shaped behav | `maintenance.rs` | `ensure_indices`, `optimize` (compaction), `repair` (explicit uncovered-drift publish), and `cleanup` (version GC): empty/idempotent/no-op edges, policy validation, head preservation. EnsureIndices refuses uncovered drift before arming its identity-bearing v9 envelope and keeps untrainable Vector work pending. Cleanup pins exact keep-count behavior, lazy-branch retention, graph-wide fail-closed ordering, and refusal of uncovered main HEAD drift before GC. Optimize's bounded payload inside the v9 envelope publishes multiple productive data tables through one graph commit, emits no lineage/sidecar at steady state, skips uncovered drift, refuses pending recovery, and compacts blob-v2 tables. Repair previews/heals verified maintenance drift and requires `--force` for semantic drift | | `failpoints.rs` | Failure-injection coverage (gated on `failpoints` feature). RFC-026 Phase A owns exact enrollment no-effect, index-only, and index-plus-empty-shard crash recovery; named-branch enrollment refusal; uncovered-index format refusal; typed maintenance/GC/index-build exclusion on an `OPEN` lifecycle; and disjoint-table maintenance/repair allowance. RFC-022 includes deterministic post-stage/pre-effect races for mutation/load uniqueness and strict disjoint-head changes, plus the cross-handle post-effect `RecoveryRequired` → read-write-open rollback cell. Branch merge adds the captured-source advance cell; post-confirm target-winner compensation; mixed physical + pointer-only delta recovery with fixed commit id/actor/parents; both sidecar-before-first-ref and ambiguous-ref-create recovery; and an 8,193-delete between-chunk crash proving an `Armed` exact-transaction prefix is rolled back before the successful retry. Identity-bearing v9 SchemaApply is pinned by `schema_apply_phase_b_failure_recovered_on_next_open` (exact confirmed roll-forward with fixed commit id + initiating actor), `schema_apply_partial_table_effect_rolls_back_exactly` (Armed proper-prefix compensation), `schema_apply_recovery_reclaims_owned_add_type_target_and_retry_succeeds` (strict owned first-touch cleanup), `schema_apply_first_touch_foreign_winner_is_preserved_not_adopted` (foreign unregistered winner preservation), `schema_apply_post_effect_disjoint_winner_is_preserved` (winner-preserving compensation), `schema_apply_post_effect_same_table_winner_fails_closed` (buried-effect refusal), `schema_apply_recovers_partial_schema_promotion_after_commit_crash` (read-only refusal for both valid and corrupt intents in the torn manifest/schema window, followed by fixed-outcome completion of a partial source/IR/state promotion), and `schema_apply_live_query_waits_for_coherent_schema_publication` (same-handle publication wait plus pre-apply-handle query/export/whole-graph-index capture from the operation-local accepted catalog). Metadata-only before/after-staging and rollback-retry cells keep the empty-effect v9 boundary pinned. EnsureIndices v9 recovery retains both boundaries in `recovery_rolls_forward_ensure_indices_on_feature_branch`: the first residual rolls forward on the next read-write open, and a second roll-forward-eligible `EffectsConfirmed` residual under an unchanged captured token is completed by a same-handle retry before new planning. `ensure_indices_complete_armed_effects_roll_back` keeps the authority-clean complete-effect Armed rollback rule isolated, while `ensure_indices_entry_barrier_refuses_partial_armed_before_staging` leaves one of two table effects pending and proves the original `RecoveryRequired` wins before the remaining index can reach the post-stage failpoint. Its remaining cells are `ensure_indices_stage_btree_failure_leaves_existing_tables_writable` (after a clean entry barrier, expensive mixed-index staging remains outside the final authority/gates), `ensure_indices_first_touch_crash_before_ref_recovers_cleanly` (sidecar-before-ref no-effect recovery), `ensure_indices_mixed_first_touch_rollback_does_not_delete_moved_ref` (owned-effect rollback and sibling first-touch cleanup), and the no-work/no-sidecar failpoint cell; the recovery module separately pins existing + first-touch payload round-trip and identity-less-input refusal. Optimize's graph-wide identity-bearing v9 envelope is pinned by `optimize_phase_b_failure_recovered_on_next_open` (two-table roll-forward), `optimize_multi_table_partial_effect_rolls_back_under_one_v2_sidecar` (one shared sidecar, no partial visibility, compensation), `optimize_post_manifest_failure_finalizes_multi_table_v2_sidecar` (lost publish acknowledgement), and `optimize_excludes_pending_only_vector_table_from_v2_sidecar` (pending status cannot poison sibling recovery), plus its late-sidecar/main-gate/retry cells. Native controls are pinned by `native_branch_controls_reclassify_lost_acknowledgements` (matching create and absent-ref delete, with no version/lineage movement); `armed_first_touch_recovery_accepts_missing_target_ref` additionally forges and reclaims the clone-only/no-`BranchContents` table state. Legacy path overlap has both sides pinned: `armed_first_touch_recovery_defers_legacy_path_overlap_until_leaf_delete` permits open only for a proven no-effect intent, while `partial_first_touch_recovery_fails_closed_on_legacy_path_overlap` leaves one exact multi-table effect and verifies open fails closed until offline leaf cleanup lets rollback converge. Other control/recovery race cells include `first_touch_post_create_open_error_keeps_recovery_ownership`, `branch_delete_orphans_sidecar_armed_after_initial_barrier`, `branch_merge_fences_target_delete_recreate_aba`, `branch_merge_fences_concurrent_sync_on_same_handle`, `branch_merge_rejects_fresh_target_manifest_change_before_effects`, `branch_merge_rechecks_late_sidecar_after_table_gates`, `optimize_rechecks_late_schema_apply_sidecar_after_main_gate` (late zero-pin graph-global intent), `optimize_rechecks_late_disjoint_main_sidecar_after_main_gate` (table-disjoint intent sharing `graph_head:main`), `optimize_holds_main_gate_through_disjoint_table_effects` (post-relist branch-gate lifetime), `cleanup_rechecks_sidecars_under_gc_gates`, `full_recovery_rereads_sidecar_body_after_discovery`, `recovery_discovery_skips_sidecar_deleted_after_list` (an unrelated write succeeds after a listed sidecar is published/deleted), and `read_only_recovery_discovery_skips_sidecar_deleted_after_list` (read-only open succeeds against that same concurrent completion). The suite also includes the established per-writer effect → manifest-CAS recovery tests, write-entry in-process heal contract, storage-fault matrix, S3 recovery twin, and convergence-idempotent roll-forward regression. | | `failpoint_names_guard.rs` | Source-walk guard (same defense-in-depth shape as `forbidden_apis.rs`): every failpoint call site across engine + cluster (`maybe_fail`, `ScopedFailPoint::new`/`with_callback`, `Rendezvous::park_first`) must reference a compile-checked `failpoints::names` const, never a bare string literal — a typo'd literal compiles but silently never fires | -| `recovery.rs` | Open-time recovery sweep — identity-bearing schema-v9 envelopes for Mutation/Load exact transaction identity, BranchMerge exact chains/ref-only effects + complete delta, SchemaApply exact overwrite/create ownership + schema promotion, EnsureIndices exact mixed-index transactions/authority/lineage/delta/first-touch identity, and Optimize's bounded maintenance payload, plus RFC-026's dedicated schema-v10 roll-forward-only StreamEnrollment envelope and schema-v11 StreamFold envelope binding the exact config-v2 profile, shard cut, merged generation, Lance transaction, and fixed lineage. Explicit refusal replaces alias inference for identity-less input; restartable compensation, fixed logical/rollback IDs, branch-token comparison, fresh under-gate reread/reparse, all-or-nothing roll-forward/rollback/refusal, recovery audit, and read-only guards remain pinned. RFC-023 additionally asserts that restoring a feature-branch sidecar leaves the selected feature ref with exact-`id` PK metadata | +| `recovery.rs` | Open-time recovery sweep — identity-bearing schema-v9 envelopes for Mutation/Load exact transaction identity, BranchMerge exact chains/ref-only effects + complete delta, SchemaApply exact overwrite/create ownership + schema promotion, EnsureIndices exact mixed-index transactions/authority/lineage/delta/first-touch identity, and Optimize's bounded maintenance payload, plus RFC-026's schema-v10 roll-forward-only StreamEnrollment envelope and current schema-v12 StreamFold envelope binding exact base and `_stream_tokens.lance` participants, generation cut, complete lifecycle state-v2, lineage, and attribution. Its unit matrix permits publication only for exact+exact and pins base-only completion plus corrupt/foreign refusal; historical schema-v11 is refused under v9. Explicit refusal replaces alias inference for identity-less input; restartable compensation, fixed logical/rollback IDs, branch-token comparison, fresh under-gate reread/reparse, all-or-nothing roll-forward/rollback/refusal, recovery audit, and read-only guards remain pinned. RFC-023 additionally asserts that restoring a feature-branch sidecar leaves the selected feature ref with exact-`id` PK metadata | | `composite_flow.rs` | Compositional/narrative end-to-end stories — multi-step flows that compose mechanics covered by other test files. Catches integration regressions where individual operations all pass their unit tests but their composition breaks (sequential merges, post-merge main writes, time-travel through merge DAG, reopen consistency over multi-merge histories, post-optimize and post-cleanup strict writes). | RFC-026 reclamation qualification: the two B2b runtime guards do not prove @@ -256,10 +256,13 @@ ack-cost claim. These debug timings still are not a product latency or group- commit claim, and B2 must re-qualify any higher resident-writer/resource limit before exposing public admission. -Format tests own genuine unenrolled-v7 ↔ v8/config-v2 refusal/rebuild. The v7 -binary exposes no production enrollment route, so this proves the real format -fence and no in-place adoption but does not claim recovery of retained physical -config-v1 state. Focused behavior also covers empty-batch refusal, exact-cap admission, one-row/one-byte +Format tests own genuine unenrolled-v7 ↔ v8/config-v2 and v8/config-v2 ↔ +v9/config-v3 refusal/rebuild. The v7 binary exposes no production enrollment +route, so that leg proves its real format fence and no in-place adoption but +does not claim recovery of retained physical config-v1 state. The v8 leg also +pins that the new trusted physical attribution column is absent from logical +exports. Focused behavior covers empty-batch refusal, exact-cap admission, +one-row/one-byte over-cap refusal before `put_no_wait` with no row/WAL batch, automatic-rollover refusal, higher-epoch reopen, restart reconstruction of the exact post-tombstone row/Arrow-byte reservation @@ -370,7 +373,33 @@ Run the decision-scale local sweep explicitly with: cargo test -p omnigraph-engine --features failpoints --test memwal_stream_cost b2a_retained_history_decision_scale_sweeps_to_128_generations -- --ignored --exact --nocapture ``` -B1/B2a do not add parser/server/CLI tests because they have no public surface. The +### RFC-026 private B2-common coverage ownership (implemented) + +The private B2-common slice extends the existing owners rather than creating a +parallel test stack. `memwal_stream.rs` owns compare-and-chain retries and +conflicts, same-generation token overlays, stale-authority recapture, exact +base+token fold, and the recovery-v12 crash boundaries. Unit tests in +`db/manifest/stream_token.rs` own canonical token/payload derivation and corrupt +base/token refusal; `db/manifest/recovery.rs` owns the v12 two-participant +effect matrix and confirmation. Manifest tests own the graph-global selected +token pointer and fold-attribution publication. `lifecycle.rs` proves hidden +trusted metadata and the MemWAL system index stay out of public reflection, +while `forbidden_apis.rs` proves no SDK/CLI/HTTP/OpenAPI streaming side door was +introduced. + +Run the focused private B2-common owners with: + +```bash +cargo test -p omnigraph-engine --features failpoints --test memwal_stream b2_ +cargo test -p omnigraph-engine --features failpoints --test memwal_stream crash_after_ +cargo test -p omnigraph-engine stream_token:: +cargo test -p omnigraph-engine stream_fold_v12 +cargo test -p omnigraph-engine --test lifecycle public_snapshot_wildcard_omits_protocol_metadata -- --exact +cargo test -p omnigraph-engine --test forbidden_apis +``` + +B1/B2a and the private B2-common slice do not add parser/server/CLI tests because +they have no public surface. The common product contract inventory specifies the missing contracts without activating them. Any public implementation must extend the existing compiler, server/OpenAPI, CLI parity, Cedar, shutdown, @@ -537,51 +566,67 @@ negatives, and the six-attempt zero-list shape. The RustFS CI job rejects a Outside that configured job the test skips explicitly when the bucket is absent. CI separately rejects a skipped S3 ABA surface guard. -## Cross-version upgrade (genuine old binaries → v8) +## Cross-version upgrade (genuine old binaries → v9) `crates/omnigraph-cli/tests/crossversion_upgrade.rs` contains genuine-binary coverage—not the stamp-rewind stand-in in `db/manifest/tests.rs::sub_current_graph_is_refused_then_rebuilt_via_export_import`. The long-baseline case mints internal schema v3 with OmniGraph 0.7.2; the v4 -case uses 0.8.1. Both prove current-v8 refusal, export/init/load into a different -v8 root, row/vector fidelity, and exact-`id` PK metadata on every rebuilt graph +case uses 0.8.1. Both prove current-v9 refusal, export/init/load into a different +v9 root, row/vector fidelity, and exact-`id` PK metadata on every rebuilt graph table; the v4 case also pins reverse refusal by the old binary. RFC-023 added its then-immediate-predecessor case gated on `OMNIGRAPH_V5_BIN`, built from the final internal-v5 commit. It mints a genuine SchemaIR-v2 v5 graph, -proves v8 refuses it with the 0.9.x rebuild guidance, exports with v5, rebuilds -under v8, checks row/vector/blob fidelity, exact blob bytes, and exact-`id` PK -metadata, then proves the v5 binary refuses the v8 root. The same cell injects -a duplicate logical ID into the v5 export: v8 rejects the load atomically, +proves v9 refuses it with the 0.9.x rebuild guidance, exports with v5, rebuilds +under v9, checks row/vector/blob fidelity, exact blob bytes, and exact-`id` PK +metadata, then proves the v5 binary refuses the v9 root. The same cell injects +a duplicate logical ID into the v5 export: v9 rejects the load atomically, leaves every initialized target table empty, and a canonical re-export proves the v5 source unchanged. The initialized empty target remains a valid graph; the operator must not serve it and should discard it after the failed rebuild. RFC-026 Phase A added its immediate-predecessor seam -`OMNIGRAPH_V6_BIN`. It mints a genuine internal-v6 graph, proves v8 refuses it -before serving, exports with v6, rebuilds into a different v8 root, verifies +`OMNIGRAPH_V6_BIN`. It mints a genuine internal-v6 graph, proves v9 refuses it +before serving, exports with v6, rebuilds into a different v9 root, verifies row/vector fidelity and exact-`id` PK metadata, and proves the v6 binary refuses -the v8 root across the v7 foundation. This is format-boundary evidence that a +the v9 root across the v7 foundation. This is format-boundary evidence that a stamp-rewind test cannot supply. RFC-026 Phase B1 adds the current immediate-predecessor seam `OMNIGRAPH_V7_BIN`. It mints a genuine internal-v7 graph with no physical -enrollment (the v7 binary exposes no production enrollment route), proves v8 +enrollment (the v7 binary exposes no production enrollment route), proves v9 refuses it before serving, exports with v7, rebuilds into a different -v8/config-v2 root, verifies row/vector fidelity and exact-`id` PK metadata, and -proves the v7 binary refuses the v8 root. This owns the real format fence, not a +v9/config-v3 root, verifies row/vector fidelity and exact-`id` PK metadata, and +proves the v7 binary refuses the v9 root. This owns the real format fence, not a retained-enrollment/config-v1 recovery claim. Run it with: ```bash OMNIGRAPH_V7_BIN=/path/to/final-v7/omnigraph \ cargo test -p omnigraph-cli --test crossversion_upgrade --locked \ - current_v8_refuses_and_rebuilds_genuine_v7_and_v7_refuses_v8 -- --exact --nocapture + current_v9_refuses_and_rebuilds_genuine_v7_and_v7_refuses_v9 -- --exact --nocapture +``` + +RFC-026 Phase B2 adds the immediate-predecessor `OMNIGRAPH_V8_BIN` seam. It +mints a genuine internal-v8/config-v2 graph, proves v9 refuses it with 0.12.x +rebuild guidance, exports with v8, rebuilds a distinct v9/config-v3 root, and +proves row/vector fidelity plus exact-`id` PK metadata. The v9 re-export must +not expose the physical `__omnigraph_stream_v1$` attribution column, and the v8 +fixture's ordinary user property `__omnigraph_stream_v1` must retain its value; +the v8 binary must refuse the v9 root. CI pins and builds the last merged schema-v8 +commit (`725793af83394235bf4b848b6c2c4454ac1f95e1`) before the workspace suite, +so this immediate boundary runs non-vacuously on full CI. Run it locally with: + +```bash +OMNIGRAPH_V8_BIN=/path/to/final-v8/omnigraph \ + cargo test -p omnigraph-cli --test crossversion_upgrade --locked \ + current_v9_refuses_and_rebuilds_genuine_v8_and_v8_refuses_v9 -- --exact --nocapture ``` -Cross-version suites are deliberately outside default CI because old-source -builds are expensive and this seam changes only at format/release boundaries. -They are gated on absolute old-binary paths and skip gracefully when unset; a -set but invalid path fails loudly rather than making the proof vacuous. +Older cross-version seams remain gated on absolute old-binary paths and skip +gracefully when unset because rebuilding every historical source revision in +default CI would be expensive. A set but invalid path, including +`OMNIGRAPH_V8_BIN`, fails loudly rather than making the proof vacuous. ## System e2e requirements and suppression diff --git a/docs/dev/versioning.md b/docs/dev/versioning.md index 4da9a874..f6f0dcbc 100644 --- a/docs/dev/versioning.md +++ b/docs/dev/versioning.md @@ -44,32 +44,31 @@ declares exactly non-null physical `id` as Lance's unenforced primary key from creation, and production strict insert/upsert routes use the exact-`id` filter-bearing adapter. -Internal schema **v8 is the currently served format** and maps to OmniGraph -**0.12.x**. It preserves the v5 identity, v6 key-fencing, and v7 stream- -lifecycle contracts, then activates RFC-026 Phase B1's private data-bearing -MemWAL core: exact stream-config v2, one admission-bounded no-roll generation, -watcher success plus the same writer's post-durability epoch check before clean -acknowledgement, conservative replay, explicit seal/drain, and a dedicated -schema-v11 `StreamFold` recovery envelope. The fold publishes its -base-table pointer, lifecycle HEAD witness, and fixed lineage in one manifest -CAS; no MemWAL state is graph-visible merely because it is durable. - -This remains an internal storage/recovery capability, not a public streaming -feature. V8 has no `@stream`, production SDK/HTTP/CLI surface, persistent -quiesce/resume controls, correction lane, or fresh-read surface. Its private -integration seam exists only under the feature-gated crash/evidence suite. -Gate R0 originally found that one legal high-entropy near-cap generation could -acknowledge and materialize but then exceed the fold's 32-MiB closure charge. -The private implementation now charges scanner output by logical slice and -copies each emission into dense owned arrays; that exact 8,192-row shape folds -and publishes successfully. This repair does not create a new format: no -schema v9 or product surface is activated by the selected unbounded retain-all -profile or the optional later B2b managed-reclamation profile. - -A v7/0.11.x graph is not reinterpreted or migrated in place: config-v1 never -becomes data-bearing config-v2 state. Export it with the v7 binary, initialize a -different v8 root, and load through the v8 writer. Because -`MIN_SUPPORTED == CURRENT == 8`, v8 refuses v7 and a v7 binary refuses v8. +Internal schema **v9 is the currently served format** and maps to OmniGraph +**0.13.x**. It preserves v8's private data-bearing MemWAL core, then activates +RFC-026's common B2 storage/recovery contract: stream-config v3, lifecycle +state v2, the grammar-impossible trusted base-row field +`__omnigraph_stream_v1$`, one manifest-selected `_stream_tokens.lance` +authority, compare-and-chain token attribution, and recovery-v12's exact base +plus token participants. One manifest CAS still owns graph visibility. The +selected B2a profile retains every canonical MemWAL object indefinitely; it +does not claim a retained-storage bound or online GC. + +This remains a private storage/recovery capability, not a public streaming +feature. V9 has no `@stream`, production SDK/HTTP/CLI surface, persistent +quiesce/resume controls, correction lane, or fresh-read surface. Its test seam +is feature-gated and doc-hidden. Row count, logical dense-slice bytes, +canonical payloads, token projections, recovery JSON, and exact-authority +lookup retention are bounded; the measured near-cap fold RSS remains evidence, +not a runtime allocator promise. + +A v8/0.12.x graph is not reinterpreted or migrated in place: config-v2 and +recovery-v11 never become config-v3/state-v2/recovery-v12 authority. Export it +with the v8 binary, initialize a different v9 root, and load through the v9 +writer. The physical field's trailing `$` is outside the `.pg` identifier +grammar, so a genuine v8 user property named `__omnigraph_stream_v1` remains +ordinary user data and round-trips unchanged. Because +`MIN_SUPPORTED == CURRENT == 9`, v9 refuses v8 and a v8 binary refuses v9. There is no in-place migration dispatcher. The single source file `db/manifest/migrations.rs` holds only the version constant, the stamp read/write, @@ -107,11 +106,11 @@ independently and a hard gate there would force lockstep redeploys for every fie addition. So that axis is additive — old and new coexist — and the OpenAPI-drift test is the guard that a change stayed additive rather than breaking the shape. RFC-023 follows that rule: `ErrorOutput.key_conflict` is optional, and its -`key` member remains optional on the wire for additive compatibility. The v8 +`key` member remains optional on the wire for additive compatibility. The v9 engine returns `KeyConflict` only after a fresh exact-ID probe identifies an attempted key; Lance's broader retryable conflict class is not serialized as a key conflict without that evidence. `ErrorOutput.resource_limit` is likewise -optional and additive; v8 servers use it with HTTP 413 for pre-arm keyed-write +optional and additive; v9 servers use it with HTTP 413 for pre-arm keyed-write ceilings. ## When you change each axis diff --git a/docs/dev/wal-thinking.md b/docs/dev/wal-thinking.md index 3293ac3c..d5a00399 100644 --- a/docs/dev/wal-thinking.md +++ b/docs/dev/wal-thinking.md @@ -1,28 +1,29 @@ # WAL Thinking -Working notes, updated 2026-07-21. Plain-language grounding for the WAL/streaming +Working notes, updated 2026-07-22. Plain-language grounding for the WAL/streaming discussion ([RFC-018](../rfcs/0018-ingest-wal.md) → [RFC-026](../rfcs/0026-memwal-streaming-ingest.md)). Three parts: the contract difference between an interactive graph commit and durable stream admission, the expected performance shape and evidence still required, and the build inventory. -Current boundary: RFC-026 Phase A, the Phase-B1 private core, and the private -B2a unbounded retain-all gate are built, but public streaming is not. Gate R0 found and the follow-up fixed the one known +Current boundary: RFC-026 Phase A, the Phase-B1 private core, private B2a +retain-all, and the first common-B2 compare-and-chain row/fold core are built, +but public streaming is not. Gate R0 found and the follow-up fixed the one known all-shape closure failure: a legal high-entropy near-cap generation is durably acknowledged, materialized, folded, and published without lowering the 8,192-row/32-MiB admission cap. The fold now charges logical Arrow slices and densifies selected rows before retaining them. The measured full-fold RSS delta was 284,934,144 bytes (about 272 MiB); CI carries a 384-MiB remeasurement tripwire for one exclusive fold, not a runtime allocator limit. Current -internal schema v8 preserves +internal schema v9 preserves Phase A's historical v7 foundation: exact empty main-only MemWAL enrollment recovery, identity-keyed lifecycle authority, exclusion of ordinary local table/schema/ maintenance/repair/recovery effects for a lifecycle-bound table, and partial-format refusal. Native branch controls alone may proceed at `SEALED`, because they do not move table HEAD. -V8 adds stream-config v2 and recovery-v11 `StreamFold`. One feature-gated, +V8 added stream-config v2 and recovery-v11 `StreamFold`. One feature-gated, doc-hidden engine seam can privately admit an already-normalized physical batch, acknowledge only after its Lance durability watcher and the same writer's post-durability epoch check both succeed, route replay or one @@ -31,7 +32,11 @@ flushed-unmerged generation fold-only, and publish one exact fold at the successor generation can put. This is implementation/evidence machinery, not a product surface. There is still no `@stream`, public enrollment or row admission, SDK/HTTP/CLI/OpenAPI route, operator drain/resume workflow, or fresh -read. RFC-026 remains Draft. +read. V9 adds config-v3/state-v2, the grammar-impossible trusted attribution +field, manifest-selected token authority, compare-and-chain admission, and +recovery-v12's exact base-plus-token publication. These remain crate-private; +management correction/lifecycle operations and every product surface are still +inactive. RFC-026 remains Draft. RFC-026 selects **unbounded retain-all on stock Lance** as the first storage profile, and its private B2a gate is implemented. OmniGraph deletes no canonical @@ -153,10 +158,10 @@ but the watcher outcome is lost, OmniGraph returns typed `AckUnknown`; replay preserves possible durable residue but cannot attribute it to that caller attempt. The attempt remains ambiguous and OmniGraph must never claim “not durable.” Cancellation cannot abandon the worker once invocation starts. -The current private B1 seam has only cardinality-level same-key retry behavior: -ambiguous `X(id)`, then durable `Y(id)`, then retry `X(id)` can make stale `X` -newest again. The common B2 contract inventory closes that public-contract hole with an explicit -compare-and-chain token. A caller-stable `write_id` plus the opaque +The current private B2 seam closes B1's cardinality-only same-key retry hole +with an explicit compare-and-chain token: ambiguous `X(id)`, later durable +`Y(id)`, then retry `X(id)` cannot silently restore stale `X`. A caller-stable +`write_id` plus the opaque `predecessor_token` returned by the server derives one stable successor token; the predecessor must equal the complete current token before Lance is called. An exact current retry is `already_durable`; a retry behind newer `Y` is @@ -456,7 +461,7 @@ workflow is implemented. | Responsibility | Required contract | |---|---| -| **Format capability and refusal** | **Phase A historical foundation:** v7 stamp, strict v6↔v7 refusal, export/init/load rebuild, exact lifecycle validation, and uncovered-partial-format refusal. **Private B1 current format:** schema v8, stream-config v2, and recovery-v11 `StreamFold`; genuine v7↔v8 old/new-binary refusal and rebuild evidence remains green, and the widest legal fold now closes. **Specified common B2 format:** schema v9, config-v3, state-v2, and recovery-v12 add hidden token/attribution/lifecycle state; they remain inactive until their genuine v8↔v9 refusal/rebuild evidence passes. Retain-all requires no storage-budget format. | +| **Format capability and refusal** | **Phase A historical foundation:** v7 stamp and enrollment authority. **Private B1 historical core:** schema v8, config-v2, recovery-v11, and the bounded one-generation worker/fold. **Private common B2 current format:** schema v9, config-v3, state-v2, grammar-impossible trusted attribution, manifest-selected token authority, and recovery-v12's base-plus-token publication. Genuine v8↔v9 refusal/rebuild includes the former-name user-property collision. Retain-all requires no storage-budget format. | | **`@stream` intent and enrollment** | **Foundation only:** Phase A binds stable table/incarnation identity, location/main ref, never-reused enrollment ID, one empty shard, fixed configuration, and the mutable current-HEAD witness under recovery. B1 is implemented but remains private. B2 makes `@stream` declaration leave the type `UNENROLLED`; an explicit request-idempotent enroll operation creates the logical stream incarnation and physical binding. Rebind remains later. | | **Public surface** | **Phase B2, after private evidence:** explicit enroll, `POST /graphs/{graph_id}/streams/{type_name}/ingest`, minimum status/block-inspection/fold/quiesce/resume/abort-drain/correct/rebuild-preflight controls, `omnigraph stream …` commands, Cedar, and OpenAPI parity. Every mutating management call after enrollment compares a lifecycle revision and durably returns its bounded terminal receipt. Existing `/ingest` remains the deprecated load alias. | | **Writer registry and routing** | **Phase B1 implemented privately:** one root-scoped, cross-handle registry owns the full binding and reuses the common table-identity plus resolved-physical-ref admission key; one serialized owner serves the initial `(table, main)` profile. One no-rollover generation has an 8,192-row / 32-MiB admission cap. Puts use exact charge → shared admission → same-key input queue → worker-mode inspection. Claim/replay starts under the shared admission lease. Empty reopen may admit; non-empty replay and one flushed-unmerged generation are fold-only, with exact accounting and the refusal marker installed before the opener releases its queue. Already-charged callers can overlap recovered replay transiently; the ledger records that overlap while refusing new charge. The exclusive fold validates replayed rows and uses the pinned public BatchStore watermark bridge before reseal. A cold fold reserves the full generation/resident/pending envelope before its owned opener and retains exclusive authority across the original seal deadline. Retirement stops puts before public `abort`; `close` is not durability evidence. Fold scanning now charges logical slices and densifies selected rows, so the near-cap closure cell succeeds. These are not public defaults. | @@ -617,8 +622,9 @@ E0 itself added no `@stream`, lifecycle rows, a sidecar schema, public APIs, WAL acknowledgements, or a format stamp. Phase A subsequently consumed that proof in historical schema v7: the v10 enrollment intent, lifecycle authority, writer exclusion, and refusal/rebuild. Private B1 now consumes that foundation -in current schema v8/config-v2 with private admission machinery and the -recovery-v11 `StreamFold` envelope. +in historical schema v8/config-v2 with private admission machinery and the +recovery-v11 `StreamFold` envelope. Current v9/config-v3/state-v2/recovery-v12 +adds private compare-and-chain token authority and trusted attribution. This still is not stream release. The exact upstream receipt/seal remains the preferred simplification and the broader-topology gate. @@ -653,7 +659,7 @@ The plan is: 2. keep Phase A's historical v7 enrollment recovery, all-lifecycle effect exclusion, narrow `SEALED` native-branch exception, lifecycle state, admission lease, - and refusal/rebuild gates as the foundation preserved by current v8; + and refusal/rebuild gates as the foundation preserved by current v9; 3. retain the repaired logical-slice/dense fold and its local/configured-RustFS near-cap closure plus full-fold RSS remeasurement instruments as regression gates; do not silently shrink the 8,192-row/32-MiB admission cap; @@ -661,15 +667,15 @@ The plan is: canonical durable MemWAL object, add no file/byte admission limit, keep complete/partial orphan roots inert, expose observed growth only as advisory, and surface provider-capacity failure loudly through recovery; -5. implement the B2-common compare-and-chain token, trusted attribution, - manifest-selected token participant, protocol-v2 lifecycle, and bounded - `REPLACE`/`WITHDRAW` correction as the implementation contract; +5. preserve the implemented B2-common compare-and-chain token, trusted + attribution, manifest-selected token participant, same-generation overlay, + exact authority bounds, and recovery-v12 crash matrix; 6. keep B2b Lance-owned reclamation and a whole-root finite-lifetime budget as optional future profiles, activated only by a new measured need and their own RFC/evidence; do not block retain-all on either; -7. implement schema v9/config-v3/state-v2/recovery-v12 privately for the token, - attribution, lifecycle, and correction contracts, and keep every product - surface absent until those cross-version and crash matrices are green; +7. implement the remaining protocol-v2 lifecycle controls and bounded + `REPLACE`/`WITHDRAW` correction privately, and keep every product surface + absent until their cross-version and crash matrices are green; 8. add schema/SDK/HTTP/CLI/Cedar/OpenAPI surfaces only after that private core passes; then activate Phase B2; 9. design and measure any later group-commit policy before claiming a @@ -679,12 +685,12 @@ The plan is: lands, and use the latter to simplify enrollment and broaden topology. We tested the narrower support contract instead of waiting on the upstream -calendar. Phase A's v7 foundation and private B1's current v8/config-v2/v11 -worker/fold core are implemented, and the widest admitted high-entropy shape -now closes. RFC-026 remains Draft because the public token, attribution, -lifecycle/correction, authorization, and product surfaces are not implemented. +calendar. Phase A's v7 foundation, B1's historical v8/config-v2/v11 worker/fold +core, and v9's first private common-B2 token/attribution/recovery-v12 slice are +implemented, and the widest admitted high-entropy shape now closes. RFC-026 +remains Draft because lifecycle/correction, authorization, and product surfaces +are not implemented. The selected first storage posture is the implemented private B2a unbounded retain-all gate: no OmniGraph MemWAL GC and no file/byte limit. Two negative RC.1 reclamation guards remain -checked in as rationale for that no-deletion rule. No schema-v9 contract, -production enrollment, acknowledgement, fold, operator, or public stream path -exists. +checked in as rationale for that no-deletion rule. No production enrollment, +operator control, or public stream path exists. diff --git a/docs/dev/writes.md b/docs/dev/writes.md index f81a75b4..91baa24c 100644 --- a/docs/dev/writes.md +++ b/docs/dev/writes.md @@ -19,7 +19,7 @@ authority. - No `omnigraph run *` CLI subcommands and no `/runs/*` HTTP endpoints. - No `__run__` staging branches; `__run__*` is no longer a reserved name. The branch-name guard was removed in MR-770. Historically, the v2→v3 - in-place migration swept stale `__run__*` entries; the current v8 strand is + in-place migration swept stale `__run__*` entries; the current v9 strand is strict single-version, so older graphs are refused and rebuilt by export/init/load rather than migrated on open. (Inert `_graph_runs.lance` bytes in an old export source remain irrelevant to the rebuilt graph.) @@ -543,13 +543,17 @@ Empty-table overwrite is represented as a valid zero-fragment Lance `Overwrite` transaction, not as truncate-then-append. -### RFC-026 private stream foundation, B1 core, and B2a retain-all gate +### RFC-026 private stream foundation, B2 core, and retain-all profile -Internal schema v8 activates the deliberately bounded, data-bearing MemWAL -format. Phase A's enrollment foundation and Phase B1's one-generation row/fold -core are always compiled but remain crate-private. Only the doc-hidden callable -integration-test seam is feature-gated; there is no schema, SDK, CLI, HTTP, -Cedar, OpenAPI, or production caller. +Internal schema v9 is the current data-bearing MemWAL format. It preserves +Phase A's enrollment foundation and Phase B1's bounded one-generation worker, +then activates the private B2 compare-and-chain/token-fold slice through +stream-config v3, lifecycle state-v2, recovery-v12, and the manifest-selected +graph-global `_stream_tokens.lance` participant. The code is always compiled +but remains crate-private; only the doc-hidden callable integration-test seam is +feature-gated. There is no accepted-schema declaration, production enrollment, +SDK, CLI, HTTP, Cedar, OpenAPI, lifecycle-management, correction, or status +surface. Phase A still owns physical enrollment: @@ -573,7 +577,7 @@ Phase B1 adds the private row path: - Independently opened handles share one root-scoped registry keyed by stable table identity, enrollment ID, and shard ID. The qualified B1 envelope permits one resident writer for the graph root, one generation, at most 8,192 rows, - 8,192 batches, and 32 MiB of Arrow payload. The worker config prevents Lance + 8,192 batches, and 32 MiB of logical dense-slice Arrow payload. The worker config prevents Lance auto-rollover. - A put accepts one non-empty, already-normalized physical `RecordBatch` and one contiguous caller-ordinal range. It rechecks schema, lifecycle, binding, @@ -582,15 +586,18 @@ Phase B1 adds the private row path: admission repeats the same checks immediately before invocation. - Cheap raw row/byte bounds reject obviously over-cap input before recovery I/O; a raw-fit batch then receives exact post-tombstone validation at that same - pre-recovery boundary. After any recovery/authority prelude, the exact charge - is recomputed and reserved against the root's 32-MiB aggregate budget. Every + pre-recovery boundary. After any recovery/authority prelude, the logical + dense-slice charge is recomputed with `ArrayData::get_slice_memory_size` and + reserved against the root's 32-MiB aggregate budget. Every put then follows one order: exact charge → shared admission → same-key input queue → worker-mode inspection, before detached ownership or cold claim. Queue-first acquisition is forbidden because it can deadlock with the fair exclusive fold admission path. The queued permit is released on effect-free refusal or transferred without double-counting into the resident generation. Thus concurrent callers cannot - accumulate an unmeasured queue outside the stated memory ceiling. + accumulate an unmeasured queue outside the stated 32-MiB logical dense-slice + Arrow charge ceiling. Physical allocation and RSS are measured separately; + they are not admission authority. - The worker invokes Lance `put_no_wait` in a detached, worker-owned task and acknowledges only after its durability watcher succeeds. Dropping the caller or reaching a deadline does not cancel an invoked Lance future or release the @@ -611,7 +618,7 @@ Phase B1 adds the private row path: admission. Seal/fold takes exclusive admission and holds it continuously across cut, independent drain proof, scan, staged table effect, and manifest publication. No second writer can claim the shard in that interval. Before a - cold fold invokes Lance's opener, it reserves the full 32-MiB generation plus + cold fold invokes Lance's opener, it reserves the full 32-MiB logical generation plus resident/pending slots and the fold-only marker. The owned opener shares the original seal deadline; timeout retains the opener, exclusive authority, inflight permit, reservation, and opening slot until unclaimed release or @@ -619,23 +626,31 @@ Phase B1 adds the private row path: The strict fold scans only the selected fresh generation with `LsmScanner::without_base_table`. It charges each scanner batch by the logical -memory of its selected slices against the same 8,192-row/32-MiB generation +dense-slice memory of its selected arrays against the same 8,192-row/32-MiB generation limit, then uses an Arrow take of every selected row to build dense owned arrays before retaining that batch. This avoids carrying a small logical slice through the fold with the scanner's much larger sparse parent buffers. The fold -validates those already-normalized rows against one captured graph snapshot and -stages an exact-`id` upsert plus exactly one `MergedGeneration` marker in the -same Lance `Update`. It then arms a -recovery-v11 `StreamFold` sidecar containing the physical cut, pre-minted -transaction identity, fixed lineage, and complete pointer/lifecycle outcome. -Under admission → schema → main → table gates, the final barrier re-lists all -main-branch recovery intents, revalidates authority, commits with zero -transparent conflict retries, durably confirms the exact effect, and publishes -one `__manifest` CAS. That CAS is the only graph-visibility point: it advances -the table pointer, `CurrentHeadWitness`, epoch floor/lifecycle, and graph lineage -together. Exact no-effect can retire and retry within the bounded attempt; exact -planned `N + 1` is rolled forward on recovery. A foreign, buried, differently -marked, or authority-mismatched effect fails closed. +validates those already-normalized rows against one captured graph snapshot, +plans the exact per-key token winners and attribution commitment, stages an +exact-`id` base upsert with one `MergedGeneration` marker, and stages the +corresponding exact keyed update to `_stream_tokens.lance`. + +Recovery-v12 binds the immutable cut, both pre-minted transaction identities, +planned token rows, prior and complete next lifecycle state-v2, fixed lineage, +and fold-attribution summary before either Lance HEAD moves. Under admission → +schema → main → stream-token → table gates, the final barrier re-lists all +main-branch recovery intents, recaptures manifest/token authority, revalidates +the winner set, and commits with zero transparent conflict retries. Only exact +base plus exact token effects may reach the one `__manifest` visibility CAS. +That CAS advances the base pointer, manifest-selected token pointer, +`CurrentHeadWitness`, epoch floor/lifecycle, graph lineage, and durable +graph-commit attribution together. Exact no-effect on both participants can +retire and retry; if the exact base effect landed but the token effect did not, +recovery may complete only the exact token transaction described by the durable +plan. A foreign, buried, differently marked, partially ambiguous, or +authority-mismatched outcome fails closed. Recovery-v11 is historical v8 state +and is refused under v9 because it has no token participant or complete +state-v2 outcome. B1 performs no fresh-tier reads and no generation GC. Acknowledged rows become query-visible only after fold. The support boundary remains main-only, @@ -646,7 +661,7 @@ Gate R0's closure defect is now repaired. The deterministic legal high-entropy near-cap cell acknowledges without moving either manifest or base-table version, materializes one generation, then folds 8,192 rows and publishes one table version through one `__manifest` visibility CAS. The dense scanner-batch -copy keeps the accepted shape inside its logical 32-MiB limit. A separate +copy keeps the accepted shape inside its logical dense-slice 32-MiB limit. A separate subprocess measurement recorded a 284,934,144-byte isolated fold RSS delta (about 272 MiB), below the 384-MiB remeasurement tripwire. That tripwire guards the measured implementation shape; changing the admission or fold strategy @@ -682,24 +697,51 @@ authority and combined retained-history work grow; the observations are not a quota, latency SLO, provider bill, or isolated WAL slope. B2a activates no new format or product state. -RFC-026's common B2 inventory specifies—but does not implement—the next product -contract. -`@stream` declares eligibility but explicit, request-idempotent enrollment -creates the stream incarnation and physical binding. A caller-stable `write_id` -and exact predecessor then form an -opaque compare-and-chain token; trusted contributor/digest/tagged-origin and -same-generation chain-certificate metadata travels with the row. A -manifest-selected graph-global token dataset is the sole post-fold per-key -sequencing authority and publishes atomically with the base pointer under the -shared stream-token gate/recovery envelope. Because that participant is -graph-global, every token-moving sidecar blocks every manifest/main writer, -including table-disjoint writers. A final-barrier late discovery releases all -admission, schema, branch, token, and table gates and restarts from the root -barrier; it never recovers while retaining a leaf gate. Durable -`OPEN -> DRAINING -> SEALED -> OPEN` state carries a monotonic lifecycle -revision. Every external mutating management call compares that revision and -records a bounded, complete terminal receipt; roll-forward-only resume/abort and -bounded `REPLACE`/`WITHDRAW` correction use recovery-v12. The B2b +The implemented private B2 slice owns the row-sequencing and fold-authority +part of RFC-026's common contract. A caller-stable `write_id` and exact +predecessor form an opaque compare-and-chain token. Its canonical v1 digest +binds stable table identity, logical ID, stream incarnation, predecessor, +write ID, trusted contributor, and a payload digest; the payload digest binds +the accepted schema and canonical normalized row bytes. Trusted contributor, +digest, tagged origin, and same-generation chain-certificate metadata travel +inside the hidden physical row field. A same-key overlay becomes usable only +after watcher success plus the post-durability fence check, so a second put in +the same generation can chain from durable-but-unfolded authority and an exact +retry performs no second WAL write. + +B2 reserves its complete worst-case preprocessing scratch before it can +materialize blob cells or build the canonical payload: 32 MiB for the original +legal Arrow row, 32 MiB for a possible normalized replacement, and 64 MiB for +canonical bytes. The private limits admit two 128-MiB envelopes (256 MiB total) +per root: the minimum overlap needed for a waiting provisional caller to +revalidate against a concurrent winner. Each ordinary inflight permit transfers +into queued/worker ownership, while its +scratch reservation releases only after the owned payload digest is derived. +Pressure is an effect-free typed `stream_b2_preprocessing_bytes` refusal, not +permission to allocate first and account later. + +Any authority captured before queueing is provisional. After the caller owns +shared admission and the same-key queue, the adapter re-lists relevant +recovery, recaptures schema/binding/lifecycle/HEAD authority, and reads the +manifest-selected token snapshot before classification and `put_no_wait`. +Stale authority therefore produces an effect-free typed conflict or retry; it +never authorizes a WAL effect. `_stream_tokens.lance` is the sole post-fold +per-key sequencing authority. Its raw HEAD is not trusted: one manifest row +selects its exact main-branch version and Lance transaction UUID. The durable +token witness intentionally omits provider-local ETags, which are inode-derived +on local storage and therefore change when an exact graph is copied. Because +that participant is graph-global, +every token-moving sidecar blocks every manifest/main writer, including +table-disjoint writers. A final-barrier late discovery releases the complete +gate suffix and restarts from the root barrier; it never recovers while +retaining a leaf gate. + +The remaining product contract is still inactive. `@stream` eligibility, +request-idempotent public enrollment, authoritative status, revisioned +`OPEN -> DRAINING -> SEALED -> OPEN` management, bounded terminal receipts, +`REPLACE`/`WITHDRAW` correction, authorization, and wire/SDK/CLI parity have no +production caller yet. State-v2 reserves and validates their durable slots; it +does not imply that their transitions are implemented. The B2b managed-reclamation profile requires Lance-owned durable reclamation with whole-cut proof, attempt/receipt recovery, bounded history checkpointing, strong PUT/DELETE inventory plus multipart accounting/abort or durable accounting, a post-success @@ -718,10 +760,10 @@ stays reserved. Versioned/soft-delete/Object-Lock stores are refused unless all retained physical bytes are countable and eligible versions permanently deletable. Measurements validate those bounds; they do not create them. Generic Lance cleanup does not reclaim `_mem_wal`, and OmniGraph must never delete its -raw paths. Internal -schema v9/config-v3/state-v2/recovery-v12 and every public policy/wire surface -remain inactive until the common crash, no-delete/provider-failure, -cross-version, authorization, and parity evidence passes. The selected +raw paths. Internal schema v9/config-v3/state-v2/recovery-v12 is active for the +private row/fold slice; every public policy/wire surface remains inactive until +its authorization, cancellation, lifecycle/correction/status, compatibility, +and parity evidence passes. The selected unbounded retain-all profile has no physical watermark and no graph-history quota. If a future B2b bounded/managed profile is pursued, its physical watermark is per binding and does not bound base/token or shared manifest @@ -853,22 +895,23 @@ identity, incarnation, path, and Lance version. > reader of `recovery.rs`, `failpoints.rs`, or this document only > encounters phase letters in the per-writer context. -RFC-026's named “Phase A foundation”, “Phase B1 private core”, and “B2a -retain-all gate” are RFC slice names, not steps in that four-phase convention. Recovery-v10 +RFC-026's named “Phase A foundation”, “Phase B1 private core”, “B2a +retain-all gate”, and “private B2 token/fold core” are RFC slice names, not +steps in that four-phase convention. Recovery-v10 `StreamEnrollment` uses a dedicated exact initializer classifier: no effect retires, index-only provisions the fixed empty shard, and -index-plus-empty-shard publishes pointer + lifecycle. Recovery-v11 `StreamFold` -accepts only exact no effect or its pre-minted `N + 1` Update carrying the fixed -generation marker; a confirmed exact effect publishes its fixed -pointer/lifecycle/lineage outcome. Neither envelope grants permission to adopt -an ambiguous artifact. +index-plus-empty-shard publishes pointer + lifecycle. Current recovery-v12 +`StreamFold` owns exact base and token transactions and publishes only their +exact joint pointer/lifecycle/lineage/attribution outcome. Recovery-v11 is the +historical v8 one-participant fold and is not recoverable under v9. No envelope +grants permission to adopt an ambiguous artifact. A failure between Phase A and Phase D leaves the sidecar on disk. The next `Omnigraph::open` (gated on `OpenMode::ReadWrite`) runs the recovery sweep in `crates/omnigraph/src/db/manifest/recovery.rs`: The established writers emit sidecar schema v9. RFC-026 additionally emits the -dedicated recovery-v10 `StreamEnrollment` and recovery-v11 `StreamFold` +dedicated recovery-v10 `StreamEnrollment` and recovery-v12 `StreamFold` envelopes. The JSON field names `protocol_v3`, `protocol_v4`, `protocol_v7`, and `protocol_v8` are retained payload-version names for mutation/load, BranchMerge, SchemaApply, and @@ -915,11 +958,16 @@ EnsureIndices respectively; they do not mean the outer envelope is pre-v9. forward through the dedicated completion path; any foreign, buried, data-bearing, mismatched, or ambiguous state fails closed with the sidecar intact. - Schema-v11 `StreamFold` requires the exact binding, immutable generation cut, - pre-minted transaction, `MergedGeneration` marker, fixed lineage, and complete - pointer/lifecycle outcome. Exact no effect retires; only its planned `N + 1` - effect rolls forward. Foreign, buried, differently marked, or - authority-mismatched state fails closed with the sidecar intact. + Schema-v12 `StreamFold` requires the exact binding, immutable generation cut, + pre-minted base and token transactions, `MergedGeneration` marker, planned + token winners, fixed lineage, attribution summary, and complete + pointer/lifecycle outcome. Exact no effect on both participants retires; an + exact base-only state may complete only the exact token effect in the durable + plan; only exact base plus exact token may publish. Foreign, buried, + differently marked, partially ambiguous, or authority-mismatched state fails + closed with the sidecar intact. Historical schema-v11 folds are refused under + internal schema v9 because they lack the token participant and complete + state-v2 lifecycle authority. First-touch rollback deletes only the exact owned version-one dataset and only while no manifest registration or competing recovery claim owns the path. A foreign winner at an unregistered first-touch path is left untouched and is @@ -952,8 +1000,8 @@ EnsureIndices respectively; they do not mean the outer envelope is pre-v9. `RecoveryRequired` when the fixed original manifest outcome is visible and the target schema identity is not yet fully live. It always refuses a pending schema-v10 `StreamEnrollment` intent because enrollment may carry uncovered - format effects, and it always refuses a pending schema-v11 `StreamFold` - because that intent may own an unpublished table effect. A read-write open + format effects, and it always refuses a pending schema-v12 `StreamFold` + because that intent may own unpublished base/token effects. A read-write open completes exact recoverable state. - On a live handle, query, export, graph-index, and blob-read capture takes the process-local schema gate just long enough to bind one manifest snapshot to one @@ -979,8 +1027,9 @@ EnsureIndices respectively; they do not mean the outer envelope is pre-v9. interrupted writer's fixed lineage intent, including its original actor. Schema-v10 `StreamEnrollment` likewise publishes its fixed lineage and records the exact `N -> N + 1` enrollment outcome when recovery completes an - effect. Schema-v11 `StreamFold` publishes its fixed fold lineage and records - the exact recovered fold outcome in the same recovery audit. + effect. Schema-v12 `StreamFold` publishes its fixed fold lineage and durable + attribution summary and records both exact participant outcomes in the same + recovery audit. Their rollback paths reuse pre-minted recovery commit ids and durable audit plans, with the recovery actor. Other rollback and legacy recovery commits use `actor_id = "omnigraph:recovery"`. Ordinary @@ -1211,7 +1260,7 @@ does not yet have a public CLI query. `db/manifest/migrations.rs` is the single place the on-disk `__manifest` shape is reconciled with what the binary expects. Storage is **strict-single-version** (the strand model): this binary reads exactly ONE internal-schema version -(`MIN_SUPPORTED == CURRENT == 8`), so there is no in-place migration. +(`MIN_SUPPORTED == CURRENT == 9`), so there is no in-place migration. - **Graph creation** stamps `omnigraph:internal_schema_version` at CURRENT, so a fresh graph always opens. @@ -1240,13 +1289,26 @@ root moves to v8 only through export/init/load into a different root; v8 refuses v7 and a v7 binary refuses v8. This format capability does not expose a public streaming API. +V9 is the current development strand. It preserves the bounded B1 worker +mechanics while activating stream-config v3, lifecycle state-v2, hidden trusted +row metadata, the manifest-selected graph-global `_stream_tokens.lance` +authority, and recovery-v12's exact base-plus-token fold. Recovery-v11 remains +historical v8 syntax and is refused under v9. The genuine v8↔v9 CI gate builds +the final v8 binary, proves refusal in both directions, and rebuilds through +export/init/load while preserving logical rows, physical vector values, and +exact-`id` PK metadata. Export deliberately omits v9's trusted hidden stream +metadata and does not transfer token authority. This remains a private-core +format, not a public streaming API. + The stamp history (v1 PK-less, v2 unenforced-PK, v3 `__run__*` sweep, v4 lineage in `__manifest` with the commit-graph tables retired, v5 stable table identity, v6 exact-`id` PK metadata plus fenced keyed routing, v7 identity-keyed stream lifecycle authority plus the recoverable empty-enrollment foundation, v8 -stream-config v2 plus the private recovery-v11 row/fold core) is recorded on the -`INTERNAL_MANIFEST_SCHEMA_VERSION` doc-comment; only v8 is served. An earlier-stamped -graph is rebuilt via export/import, not migrated in place. +stream-config v2 plus the private recovery-v11 row/fold core, and v9 +stream-config v3/state-v2 plus manifest-selected token authority and +recovery-v12) is recorded on the `INTERNAL_MANIFEST_SCHEMA_VERSION` doc-comment; +only v9 is served. An earlier-stamped graph is rebuilt via export/import, not +migrated in place. ## Mid-query partial failure: closed by MR-794 diff --git a/docs/dev/writing-path-state-of-affairs.md b/docs/dev/writing-path-state-of-affairs.md index adfdd518..dcf686b0 100644 --- a/docs/dev/writing-path-state-of-affairs.md +++ b/docs/dev/writing-path-state-of-affairs.md @@ -1,29 +1,29 @@ # Write Path: State of Affairs **Type:** living architecture and execution summary -**Status:** current as of 2026-07-21 -**Surveyed:** OmniGraph 0.8.1 development, internal manifest schema v8, +**Status:** current as of 2026-07-22 +**Surveyed:** OmniGraph 0.8.1 development, internal manifest schema v9, Lance 9.0.0-rc.1 at `cec0b7df` **Scope:** the direct-publish graph write path, its RFC-022–028 family, adjacent control and maintenance operations, known blockers, and the next decision points **Change-set boundary:** this page describes current main through RFC-026's -Phase A foundation, private Phase B1 core, the Gate R0 findings, the subsequent -all-shape closure repair, and the implemented private B2a unbounded retain-all -gate. +Phase A foundation, historical private Phase B1 core, the Gate R0/near-cap +closure work, the B2a unbounded retain-all gate, and the current private B2 +compare-and-chain/token-fold slice. RFC-024, RFC-025, and RFC-027 remain research-blocked at their recorded evidence gates. Gate E0 first proved the bounded public Lance state classifier; Phase A activated internal schema v7 with recoverable empty enrollment, durable identity-keyed lifecycle authority, process-local admission/exclusion, and -strict format refusal/rebuild. Internal schema v8 now preserves that foundation -and adds stream-config v2, a root-scoped one-generation worker, -watcher-backed durability followed by a same-writer post-durability epoch -check before clean acknowledgement, exact replay/seal/retirement, and -recovery-v11 `StreamFold`. Gate R0 found that a legal high-entropy near-cap +strict format refusal/rebuild. Historical schema v8 added stream-config v2, a +root-scoped one-generation worker, watcher-backed durability followed by a +same-writer post-durability epoch check, exact replay/seal/retirement, and the +one-participant recovery-v11 fold. Gate R0 found that a legal high-entropy near-cap generation could be acknowledged and materialized but could not fold because -shared-buffer capacity was charged instead of logical slice size. The fold now -charges logical slices and rebuilds dense arrays with `take`; that exact shape +shared-buffer capacity was charged instead of logical slice size. Admission, +replay, and fold now share `ArrayData::get_slice_memory_size` logical charging, +and fold rebuilds dense arrays with `take`; that exact shape acknowledges, materializes, folds, and publishes. The measured RSS delta for one exclusive full fold is 284,934,144 bytes (about 272 MiB). CI's 384-MiB threshold is a remeasurement tripwire, not a runtime allocator or hard memory @@ -33,8 +33,9 @@ Gate R0 historically rejected a *bounded, finite-lifetime* retain-all claim on stock RC.1 because materialization has no durable attempt receipt or complete physical-output envelope. We have deliberately dropped that claim. The first profile is unbounded retain-all. Its private B2a gate is implemented: OmniGraph -never deletes a canonical durable MemWAL object, sets no retained-byte or file- -count limit, and treats provider exhaustion as a loud operational failure. +never deletes a canonical durable MemWAL object, sets no retained-byte, +object-count, file-count, or history quota, and treats provider exhaustion as a +loud operational failure. Lance may remove only a losing shard-manifest-CAS temporary staging object. Complete and partial orphan output stays non-authoritative and is not descended into, read, mutated, adopted, or deleted through retry/reopen; parent discovery @@ -44,14 +45,27 @@ this profile. The 1/8/32/128 retained-history instrument keeps acknowledgement, replay, fold, visibility, table, graph-manifest, adapter, object, and RSS terms separate. It shows flat warm-ack operation shape alongside growing serialized authority/combined history work; its timings, RSS, and LIST totals are advisory, -not quotas or SLOs. Managed reclamation remains optional Lance-owned work for a later -profile. The common explicit enrollment, compare-and-chain token, trusted -attribution, and lifecycle-receipt/correction contracts remain specified and -inactive; a `GraphHistoryBudget` is not part of the immediate plan. +not quotas or SLOs. Managed reclamation remains optional Lance-owned work for a +later profile; a `GraphHistoryBudget` is not part of the immediate plan. + +Internal schema v9 now activates stream-config v3, lifecycle state-v2, trusted +hidden stream-row metadata, and a manifest-selected graph-global +`_stream_tokens.lance` authority. Private admission derives canonical payload +and compare-and-chain token digests, supports exact idempotent retry and +same-generation chains, and treats every pre-wait authority capture as +provisional: it recaptures schema, binding, lifecycle, HEAD, and token authority +after shared admission and same-key queue ownership. Recovery-v12 owns exact +base and token transactions; only their exact joint outcome may reach the sole +manifest CAS, which advances both pointers, lifecycle, lineage, and a durable +fold-attribution commitment together. Recovery-v11 is historical v8 state and +is refused under v9. The genuine v8↔v9 gate proves two-way refusal and strict +export/init/load rebuild without exporting hidden trusted metadata. + The RFC remains Draft and the implementation remains reachable only through a -feature-gated private engine seam: there is still no schema v9/config-v3/ -recovery-v12 activation, `@stream` syntax, production/public enrollment, -put/ack/fold API, operator drain/resume surface, or fresh-read mode. +feature-gated private engine seam. There is still no `@stream` syntax, +production/public enrollment or put/ack/fold API, operator +status/drain/resume/correction surface, Cedar/HTTP/CLI/OpenAPI contract, or +fresh-read mode. This page answers four practical questions: @@ -91,13 +105,15 @@ The central chassis is implemented: [RFC-022](../rfcs/0022-unified-write-path.md support boundary. The remaining RFCs are not a backlog to implement in numeric order. RFC-024, RFC-025, and RFC-027 remain research-blocked. RFC-026 remains strategic and Draft. Its production-neutral Gate E0 and Phase A foundation -passed their bounded gates. Internal -schema v7 introduced exact main-only empty MemWAL enrollment and local -lifecycle exclusion; schema v8/config-v2 adds one private no-rollover -generation behind a root-scoped worker, watcher-backed durability followed by -a same-writer post-durability epoch check before acknowledgement, conservative -replay, exact seal/retirement, and one recovery-v11 strict fold. -It admits and folds only exact already-normalized physical rows—including +passed their bounded gates. Internal schema v7 introduced exact main-only empty +MemWAL enrollment and local lifecycle exclusion; historical schema v8/config-v2 +added one private no-rollover generation behind a root-scoped worker, +watcher-backed durability followed by a same-writer post-durability epoch check, +conservative replay, exact seal/retirement, and one recovery-v11 strict fold. +Current schema v9/config-v3/state-v2 preserves that bounded worker and activates +the private compare-and-chain row/fold core: canonical digests, trusted hidden +attribution, graph-global manifest-selected token authority, and exact +base-plus-token recovery-v12. It admits and folds only exact already-normalized physical rows—including already-normalized vector values—and neither calls an external embedding provider nor invents unspecified derived fields. Native branch controls alone may proceed at `SEALED`, because they do not move table HEAD. This remains a @@ -108,13 +124,14 @@ shape after durable acknowledgement; logical-slice accounting plus dense rebuilding now closes that shape. Second, stock RC.1 cannot prove a lifetime bound for materialization attempts or their complete physical growth. The selected first profile makes no such claim. Its private B2a gate retains every -canonical durable MemWAL object without an OmniGraph file/byte limit, accepts -loud provider exhaustion, keeps complete/partial orphan roots inert, and -measures retained-history terms locally and on configured RustFS. +canonical durable MemWAL object without an OmniGraph retained-byte, +object-count, file-count, or history quota, accepts loud provider exhaustion, +keeps complete/partial orphan roots inert, and measures retained-history terms +locally and on configured RustFS. Neither a test-only attempt ledger nor managed reclamation is on the immediate -activation path. The next work is the common private token, attribution, -lifecycle, and correction machinery, followed by product contracts only after -their evidence is green. B1's +activation path. The next work is the remaining private lifecycle, +correction/status, and explicit-enrollment machinery, followed by product +contracts only after their evidence is green. B1's adapter recheck contains a stale epoch from becoming a clean OmniGraph acknowledgement; it is not a substrate retention/fencing primitive. @@ -134,7 +151,7 @@ resolve or refuse relevant recovery -> prepare and validate the complete operation -> acquire stream-admission domains -> StreamFold only: seal, drain, retire, and prove one immutable fresh-tier cut - -> acquire schema -> branch -> sorted-table gates + -> acquire schema -> branch -> stream-token -> sorted-table gates where applicable -> relist recovery and revalidate authority plus physical baselines -> durably arm an identity-bearing recovery intent -> apply writer-specific graph/table Lance effects @@ -154,14 +171,18 @@ validation result; Optimize likewise reopens and replans a maintenance retry. The five established graph-visible writer classes emit schema-v9 recovery envelopes before their first independently durable effects. Phase A adds the internal `StreamEnrollment` adapter with a dedicated schema-v10 envelope; -private Phase B1 adds the schema-v11 `StreamFold` adapter. Table ownership is +current private B2 adds the schema-v12 `StreamFold` adapter with exact +base-table and `_stream_tokens.lance` participants. Table ownership is keyed by `(stable_table_id, table_incarnation_id)`, never inferred from an alias, path, Lance version, or field ID. Mutation, Load, BranchMerge, SchemaApply, and EnsureIndices use exact protocols in which `Armed` is rollback-only and `EffectsConfirmed` may roll forward only under the captured authority. StreamFold is also exact, but its immutable fresh-tier cut is proved -before arm: `Armed` plus exact table `N` is effect-free for the base table and -leaves the cut fold-only, while only confirmed exact `N + 1` may publish. +before arm: only the exact planned base plus token outcome may publish both +pointers, lifecycle state-v2, lineage, and attribution. An exact base-only +outcome may complete only the pre-minted token transaction; foreign, buried, +token-only, or ambiguous partial outcomes fail closed. Recovery-v11 is +historical v8 syntax and is refused under v9. Optimize is the deliberate exception: its v9 envelope carries identity-bound pins and a bounded complete-set maintenance classifier, with no exact authority/fixed-lineage confirmation phase, because Lance exposes no @@ -187,7 +208,7 @@ gap between those Lance effects and graph visibility. | EnsureIndices | One staged mixed `CreateIndex` transaction per productive table | Indexes remain derived state. Exact transactions and the complete pointer delta publish once; untrainable vector work stays pending instead of breaking logical writes. | | Optimize | Lance compaction, incremental index optimization, and missing-index materialization across productive tables | One graph-wide v9 envelope and at most one monotonic manifest/lineage publish. Provenance is bounded rather than exact because Lance does not expose a caller-minted maintenance transaction surface. | | StreamEnrollment (internal Phase A foundation) | Lance's public internally committing singleton MemWAL initializer plus one pre-minted empty shard | One v10 intent classifies only no effect, exact `N + 1` index, or that index plus the exact empty shard, then publishes the pointer and `OPEN` lifecycle together. Once an effect exists it rolls forward only. The adapter is crate-private and has no row-admission caller. | -| StreamFold (private Phase B1 core) | Before arm, seal/drain/retirement proves one immutable fresh-tier cut; afterward that generation becomes one exact staged keyed transaction that also marks its exact Lance `MergedGeneration` | One v11 intent binds the cut, table prestate, fixed transaction, lineage, and lifecycle outcome. Exact confirmation permits one manifest CAS to publish the table pointer plus refreshed `OPEN` HEAD witness and epoch floor. An armed no-table-effect attempt leaves the generation fold-only for retry. It folds already-normalized physical rows without external embedding or unspecified derivation and is reachable only through the feature-gated private seam. | +| StreamFold (private B2 core) | Before arm, seal/drain/retirement proves one immutable fresh-tier cut; afterward that generation becomes one exact staged keyed base transaction carrying its Lance `MergedGeneration` plus one exact staged `_stream_tokens.lance` transaction | One v12 intent binds the cut, both prestates and pre-minted transactions, planned token winners, fixed lineage, lifecycle state-v2 outcome, and attribution summary. Only exact base + exact token permits one manifest CAS to publish both pointers, refreshed `OPEN` witness/epoch floor, lineage, and attribution. An effect-free attempt leaves the generation fold-only for retry; an exact base-only residual may complete only its planned token effect. Foreign or ambiguous state fails closed. | ### Explicit adjacent paths @@ -206,18 +227,20 @@ These operations do not create a side door around the protocol: | Publication surface | The historical Run state machine and `__run__*` staging branches are gone. The graph-write storage surface is crate-private, sealed, and staged-only; source guards make a new durable gateway an explicit review event. | | Lineage | `graph_commit` and `graph_head` live in `__manifest` and land in the same CAS as table pointers. The former secondary commit-graph datasets are retired. | | Mutation / load | Multi-statement writes accumulate in `MutationStaging`, provide read-your-writes, stage deletes as well as constructive writes, and publish once after complete validation. D2 deliberately keeps one query constructive or destructive. | -| Identity / schema | Accepted SchemaIR v2 owns graph-scoped, monotonic, no-reuse type/property/incarnation IDs. Supported renames preserve logical identity and table lifetime; a property rename rewrites that lifetime, while only a pure type rename also preserves physical/index history. Drop/re-add mints a new lifetime. The strict strand serves only internal schema v8 and upgrades by export/init/load rebuild. | +| Identity / schema | Accepted SchemaIR v2 owns graph-scoped, monotonic, no-reuse type/property/incarnation IDs. Supported renames preserve logical identity and table lifetime; a property rename rewrites that lifetime, while only a pure type rename also preserves physical/index history. Drop/re-add mints a new lifetime. The strict strand serves only internal schema v9 and upgrades by export/init/load rebuild. | | Key fencing | Every graph table has exact non-null physical `id` as Lance's unenforced PK. Strict insert/upsert use the sealed exact-`id` filtered adapter; bare keyed Append is forbidden; effect-free conflicts have typed reprepare or `KeyConflict` outcomes. | | BranchMerge | The ordinary ordered diff remains the correctness path. A completely verified `omnigraph.insert_absence = "v1"` history permits the proven-insert shortcut without committing Append or weakening the final filter. | | Validation | Value, enum, uniqueness, edge-RI, and cardinality use one catalog-derived, delta-scoped evaluator across mutation, load, and merge. Committed non-key uniqueness probes are batched by constraint group and bounded chunks. | -| Recovery | Same-process handles share ordered queues for the same canonical local root or identical normalized opaque remote/custom URI. Stream-admission domains are acquired outside schema → branch → table gates. Mutation/load, SchemaApply, BranchMerge, EnsureIndices, StreamFold, and `refresh` heal roll-forward-only; Optimize, Repair, and Cleanup refuse pending recovery, while branch controls use specialized barriers. Read-write open performs the quiesced full sweep, including exact v10 enrollment and v11 fold completion; read-only open never repairs and refuses unresolved stream recovery. A resolved intent is audited internally with the original actor when present; no-effect cleanup need not create graph lineage. | -| RFC-026 Phase A (v7 foundation) | Internal schema v7 introduced identity-keyed lifecycle rows and exact empty main/unsharded enrollment. At that Phase-A-only boundary, any lifecycle row—including `SEALED`—rejected base-table, schema, maintenance, repair-adoption, and recovery effects under a process-local admission lease because no drain/fold witness-update adapter existed. Native branch create/delete alone could proceed at `SEALED` because it did not move table HEAD; `OPEN`/`DRAINING` refused it. Enrollment and open-time validation rejected named-branch overlap and any uncovered lifecycle/MemWAL mismatch. Schema v8 preserves these foundation guarantees. | -| RFC-026 Phase B1 (private core) | Internal schema v8/config-v2 adds one root-scoped, cross-handle serialized worker and a hard-bounded 8,192-row/32-MiB no-roll generation. Watcher success proves durability; a clean `DurableBatchAck` additionally requires the same `ShardWriter::check_fenced()` to succeed immediately afterward. Fence loss, epoch-read failure, owner-task failure, or deadline ambiguity is post-invocation `AckUnknown` plus worker retirement. Reopen/replay is conservative, exact drain proof precedes quiesced abort, and recovery-v11 folds one already-normalized generation then atomically refreshes the table pointer and `OPEN` lifecycle witness. Gate R0's deterministic legal high-entropy near-cap cell exposed sparse scanner arrays retaining oversized backing buffers. Fold now charges logical slices and copies each scanner emission into dense owned arrays; the exact 8,192-row shape acknowledges, materializes, folds, and publishes. The isolated reference run measured a 284,934,144-byte fold RSS delta, below the 384-MiB CI remeasurement tripwire. B1 remains private. | +| Recovery | Same-process handles share ordered queues for the same canonical local root or identical normalized opaque remote/custom URI. Stream-admission domains are acquired outside schema → branch → stream-token → table gates where applicable. Mutation/load, SchemaApply, BranchMerge, EnsureIndices, StreamFold, and `refresh` heal roll-forward-only; Optimize, Repair, and Cleanup refuse pending recovery, while branch controls use specialized barriers. Read-write open performs the quiesced full sweep, including exact v10 enrollment and v12 base+token fold completion; read-only open never repairs and refuses unresolved stream recovery. Historical v11 folds are refused under v9. A resolved intent is audited internally with the original actor when present; no-effect cleanup need not create graph lineage. | +| RFC-026 Phase A (v7 foundation) | Internal schema v7 introduced identity-keyed lifecycle rows and exact empty main/unsharded enrollment. At that Phase-A-only boundary, any lifecycle row—including `SEALED`—rejected base-table, schema, maintenance, repair-adoption, and recovery effects under a process-local admission lease because no drain/fold witness-update adapter existed. Native branch create/delete alone could proceed at `SEALED` because it did not move table HEAD; `OPEN`/`DRAINING` refused it. Enrollment and open-time validation rejected named-branch overlap and any uncovered lifecycle/MemWAL mismatch. Schema v9 preserves these foundation guarantees. | +| RFC-026 Phase B1 (historical private core) | Internal schema v8/config-v2 added one root-scoped, cross-handle serialized worker and a hard-bounded 8,192-row/32-MiB logical dense-slice Arrow no-roll generation. Watcher success proves durability; a clean `DurableBatchAck` additionally requires the same `ShardWriter::check_fenced()` to succeed immediately afterward. Fence loss, epoch-read failure, owner-task failure, or deadline ambiguity is post-invocation `AckUnknown` plus worker retirement. Reopen/replay is conservative and exact drain proof precedes quiesced abort. Gate R0's deterministic legal high-entropy near-cap cell exposed sparse scanner arrays; logical-slice charging plus dense copies repaired it. Physical RSS is evidence only. V9 preserves those worker/closure mechanics, but recovery-v11 itself is historical only. | +| RFC-026 private B2 token/fold core | Internal schema v9/config-v3/state-v2 adds canonical payload/token digests, hidden trusted row metadata, exact idempotency and compare-and-chain classification, same-generation token overlays, and post-admission authority recapture. `_stream_tokens.lance` is graph-global durable sequencing state selected only by its manifest witness. Recovery-v12 owns exact base+token transactions and atomically publishes both pointers, lifecycle, lineage, and durable fold attribution. The genuine v8↔v9 refusal/rebuild gate is green. The slice remains private; lifecycle management, correction/status, public enrollment, authorization, and product surfaces are inactive. | | Lance access | One process-wide `ObjectStoreRegistry` reuses clients. Each `Omnigraph` handle owns its cached data-table `Session`; one process-wide zero-cache control `Session` opens mutable tips. Only the object-store registry is shared between the data and control sessions. This is “cache the past, never the present,” not one global cached session. | | Maintenance | EnsureIndices stages exact missing-index transactions. Optimize coordinates graph-wide compaction/index work under one bounded recovery envelope. Periodic optimize compacts `__manifest`, but unmaintained history-dependent paths are not globally flat. | Selected enforced limits are 8,192 rows / 32 MiB per keyed Mutation/Load table, -8,192 rows / 32 MiB for the complete private B1 generation, at most 32 +8,192 rows / 32 MiB of logical dense-slice Arrow bytes for the complete private +stream generation (physical RSS remains an evidence tripwire), at most 32 pre-effect full reprepares, 8,192 rows / 32 MiB per BranchMerge chunk, one aggregate 32-MiB retained merge-validation budget, at most 1,024 logical data transactions per merged table, and a 1,026-version exact-recovery scan bound. @@ -233,12 +256,12 @@ retry rules, recovery classification, and the full owned limits. | RFC | Current disposition | What that means now | |---|---|---| | [022 — Unified graph-write protocol](../rfcs/0022-unified-write-path.md) | **Implemented** | The shared correctness state machine, per-writer adapters, recovery barrier, one visibility point, control exceptions, and test lattice are the stable chassis. Completion is scoped to one writer process per graph for destructive recovery. | -| [023 — Key-conflict fencing](../rfcs/0023-key-conflict-fencing.md) | **Implemented** | Internal schema v6 introduced exact-`id` PK metadata, closed keyed routing, typed conflicts, bounded replay, rebuild/refusal, and accepted performance evidence; v8 preserves that contract. | +| [023 — Key-conflict fencing](../rfcs/0023-key-conflict-fencing.md) | **Implemented** | Internal schema v6 introduced exact-`id` PK metadata, closed keyed routing, typed conflicts, bounded replay, rebuild/refusal, and accepted performance evidence; v9 preserves that contract. | | [024 — Durable table heads](../rfcs/0024-durable-table-heads.md) | **Research-blocked** | The first in-manifest BTREE candidate has a specified logical contract and flat indexed row/range work, but fails the complete physical-I/O gate. No head rows or heads format are active. | | [025 — Checkpoint retention](../rfcs/0025-checkpoint-retention.md) | **Research-blocked** | Lance tag/pin semantics pass, but the proposed in-manifest registry access shape is not history-flat after compaction. No checkpoint rows, `ogcp_` production tags, API, or cleanup integration are active. | -| [026 — MemWAL streaming ingest](../rfcs/0026-memwal-streaming-ingest.md) | **Draft; Phase A/B1 private core and B2a retain-all gate implemented; public inactive** | Schema v8 preserves v7/recovery-v10 enrollment and adds config-v2 one-generation admission plus recovery-v11 strict fold. A clean acknowledgement requires watcher success and a same-writer post-durability epoch check; logical-slice charging plus dense copies close the legal near-cap fold shape. Gate R0's historical no-go applies to a finite storage/lifetime promise: stock RC.1 cannot prove an attempt cap or complete physical-growth envelope. Private B2a makes no such promise: it never deletes a canonical durable MemWAL object, keeps complete/partial orphan roots non-authoritative and untouched below their roots through retry/reopen, sets no retained-byte/object/file/history quota, pins provider failures locally and on configured RustFS, and records advisory 1/8/32/128 retained-history evidence. Lance's losing manifest-CAS temp staging is not retained authority. Common enrollment/token/attribution/lifecycle/correction, authorization, and product-parity contracts remain specified and inactive. `GraphHistoryBudget` and B2b managed reclamation belong to an optional future bounded profile. No schema v9 or product surface is active. | +| [026 — MemWAL streaming ingest](../rfcs/0026-memwal-streaming-ingest.md) | **Draft; private B2 token/fold core and retain-all profile implemented; public inactive** | Schema v9 preserves v7/recovery-v10 enrollment and the bounded B1 worker while activating config-v3/state-v2, canonical payload/token digests, trusted hidden attribution, manifest-selected `_stream_tokens.lance`, stale-authority revalidation after shared admission, and recovery-v12 exact base+token fold. One manifest CAS publishes both pointers, lifecycle, lineage, and fold attribution. Recovery-v11 is historical only. Gate R0's no-go still prohibits a finite storage promise, so the selected profile never deletes a canonical MemWAL object and sets no retained-byte/object/file/history quota. Genuine v8↔v9 refusal/rebuild is green. Explicit production enrollment, lifecycle management/correction/status, authorization, and all product surfaces remain inactive; B2b managed reclamation is optional future work. | | [027 — Lineage merge deltas](../rfcs/0027-lineage-merge-deltas.md) | **Research-blocked** | The desired O(delta) classifier and fallback contract are specified. Selective live-row and deletion-delta discovery are not yet bounded, so `OrderedTableCursor` remains the correctness path. | -| [028 — Stable schema identity](../rfcs/0028-stable-schema-identity.md) | **Implemented** | Rename-stable IDs, table incarnation, identity-derived paths, schema/recovery integration, and strict rebuild activation were introduced in v5 and remain active in v8. | +| [028 — Stable schema identity](../rfcs/0028-stable-schema-identity.md) | **Implemented** | Rename-stable IDs, table incarnation, identity-derived paths, schema/recovery integration, and strict rebuild activation were introduced in v5 and remain active in v9. | ## Blockers and constraints discovered @@ -247,7 +270,7 @@ retry rules, recovery classification, and the full owned limits. | Distributed recovery fence | Process-local queues cannot stop a live foreign process; Lance restore may orphan its commits and native refs lack conditional compare-delete. Supported destructive recovery remains one writer process per graph. | A separately designed and adversarially tested distributed fence before multi-process writers, background compensation, or cross-process exact maintenance recovery. | | History-flat authority | RFC-024/025 show flat BTREE rows/ranges/pages can coexist with history-growing manifest discovery or compacted bytes. No heads/checkpoint format is active; mutable tip caches and a second authority remain rejected. | A new Lance-native access shape—or revised measured operational contract—passes the original cold/warm, compacted/uncompacted, local/object-store gate. | | Internal history GC | Safe live-writer cleanup needs a durable resurrection/retention boundary; otherwise a stalled writer can recreate a collected version. | An evidence-backed cleanup watermark/fence before automated `__manifest` version GC. | -| MemWAL delivery and closure | RC.1 initializes the system index and claims shards as separate effects without a caller-minted combined receipt or cross-process seal. Phase A recovers that gap exactly for main-only, one-shard, one-live-writer-process empty enrollment. At the row boundary, the durability watermark is writer-wide while batch positions reset after MemTable rollover, `put_no_wait` may mutate before returning `Err`, replay leaves its fresh BatchStore WAL watermark unset, and `wait_for_flush_drain` can lose a completed failure before a late waiter snapshots it. Neither batch positions nor WAL statistics are durable receipts. Gate R0 also found a private closure bug caused by sparse scanner arrays retaining oversized backing buffers; logical-slice charging and dense copies now close the exact legal near-cap shape. | Implement and prove the common compare-and-chain token, trusted attribution, revisioned lifecycle receipts, bounded correction, authorization, and product-parity contracts privately before adding a public caller. A public receipt/seal or accepted distributed fence remains the exit for overlapping processes and failover, not for the current single-live-writer-process profile. | +| MemWAL delivery and closure | RC.1 initializes the system index and claims shards as separate effects without a caller-minted combined receipt or cross-process seal. Phase A recovers that gap exactly for main-only, one-shard, one-live-writer-process empty enrollment. At the row boundary, the durability watermark is writer-wide while batch positions reset after MemTable rollover, `put_no_wait` may mutate before returning `Err`, replay leaves its fresh BatchStore WAL watermark unset, and `wait_for_flush_drain` can lose a completed failure before a late waiter snapshots it. Neither batch positions nor WAL statistics are durable receipts. The private v9 slice now closes compare-and-chain sequencing, trusted attribution, stale-authority admission, and exact base+token fold recovery. | Implement and prove explicit production enrollment, revisioned lifecycle management, bounded correction/status, authorization, cancellation/shutdown, and product parity before adding a public caller. A public receipt/seal or accepted distributed fence remains the exit for overlapping processes and failover, not for the current single-live-writer-process profile. | | MemWAL retained growth | A clean retained generation has measurable current objects, and the Gate R0 sweep proves that referenced currently listed immutable paths retain their class and size at one/four/eight folds. RC.1 still provides no durable cross-open attempt cap, complete physical-output receipt, or provider-billed-byte inventory. That prevents OmniGraph from promising a finite retained-storage bound. | Private B2a implements the deliberately unbounded profile: never delete a canonical durable `_mem_wal` object, keep complete/partial orphan subtrees non-authoritative and untouched below their roots through retry/reopen, impose no retained-byte/object/file/history quota, and fail loudly if the provider refuses further writes. Lance may remove only its losing manifest-CAS temp staging. The 1/8/32/128 local/RustFS instrument records separate advisory terms and shows combined retained-history work grows; it does not create a bound. Add attempt ledgers or physical-growth reservation only if a later profile claims bounded retention. | | MemWAL reclamation (optional B2b) | Stock RC.1 exposes evidence-level raw listing and manifest reads, not a complete classified inventory or safe MemWAL delete/GC primitive. Generic `cleanup_old_versions` leaves `_mem_wal` unchanged. Worse, deleting the successor's empty WAL fence sentinel can let a stale writer complete a WAL PUT and report watcher success because RC.1 has no post-success epoch check. Private B1 contains that result for its own acknowledgement but cannot retract durable bytes or protect raw Lance callers. Raw path deletion in OmniGraph remains forbidden. | Keep B2b as optional Lance-owned durable inspect/plan/execute, attempt/receipt recovery, post-success fencing, bounded history checkpoint, strong inventory/accounting, and enforced-watermark work. It is not on the immediate retain-all activation path. | | O(delta) merge | A version-column predicate is still O(rows) without a selective source, and deleted rows have no live version columns. Full ID differencing remains correct. | Bounded live-row and deletion/change discovery, exact shadow agreement, and a table-size-flat one-row-delete gate. | @@ -265,14 +288,18 @@ must publish atomically with the table pointer. Other writer/control/ maintenance paths touching the table refuse pre-effect or drain first. Gate E0 proved the classifier and witness model with complete direct-probe and object-store evidence; Phase A established the reversible support restriction. -Private B1 now consumes it for one nominally bounded generation, watcher-backed +The private worker consumes it for one nominally bounded generation, watcher-backed durability plus the same-writer post-durability epoch check, replay, and strict folding without pretending the process-local lease is a distributed fence, inventing a durable WAL offset, or reusing a watcher across rollover. Fence loss or uncertainty after durability is `AckUnknown`, never a clean ack. Gate R0 exposed a sparse-buffer closure gap; the dense-copy repair closes the exact -legal near-cap shape without changing admission. That result authorizes neither -raw MemWAL reclamation, a public product, nor a broader topology. +legal near-cap shape without changing admission. V9 then adds durable +compare-and-chain authority: post-wait recapture prevents a stale admission, +same-generation overlays preserve exact retry/ordering, and recovery-v12 makes +the base and token effects one manifest-visible outcome with fold attribution. +Those results authorize neither raw MemWAL reclamation, a public product, nor a +broader topology. The full known-gap ledger, including adjacent local-CAS and unsupported multi-version-topology details, remains in @@ -313,16 +340,16 @@ or slightly lower incremental RSS. That is not a roadmap signal. must receive zero IO. Keep 384 MiB as a CI remeasurement tripwire for the isolated fold RSS delta, not as a runtime allocation promise or a retained- storage limit. Treat 1/8/32/128 LIST, timing, and RSS results as advisory. -2. **Implement the common B2 contracts privately for the selected retain-all - profile.** Add explicit first-use enrollment, compare-and-chain token - authority, trusted hidden contributor attribution, persistent revisioned - quiesce/resume/abort-drain receipts, and bounded `REPLACE`/`WITHDRAW` - correction. Keep every step under recovery and the manifest visibility CAS. - Do not add a storage quota, materialization-attempt ledger, +2. **Finish the remaining B2 control plane privately.** Build + request-idempotent first-use enrollment, authoritative status, persistent + revisioned quiesce/resume/abort-drain receipts, and bounded + `REPLACE`/`WITHDRAW` correction on the v9 token/fold authority already in + place. Keep every step under recovery and the manifest visibility CAS. Do + not add a storage quota, materialization-attempt ledger, `GraphHistoryBudget`, or raw `_mem_wal` deletion. -3. **Add product surfaces last.** Only after the private common machinery and - evidence are green should schema intent, SDK, HTTP, CLI, Cedar, OpenAPI, - shutdown ownership, and authoritative status converge on the same core. +3. **Add product surfaces last.** Only after those private lifecycle, + correction/status, and crash/race gates are green should schema intent, SDK, + HTTP, CLI, Cedar, OpenAPI, and shutdown ownership converge on the same core. 4. **Keep B2b managed reclamation independent and optional.** If a bounded profile is scheduled later, author Lance-owned durable inspect/plan/execute, attempt/receipt recovery, post-success epoch fencing, strong inventory or @@ -330,14 +357,14 @@ or slightly lower incremental RSS. That is not a roadmap signal. graph-global `GraphHistoryBudget` would require its own RFC and every-writer evidence. Never delete `_mem_wal` paths from OmniGraph. 5. **Keep RFC-024/025/027 stopped at their research no-gos.** Their blockers are - independent of the v8 stream core; do not add RFC-024 heads, RFC-025 graph + independent of the v9 stream core; do not add RFC-024 heads, RFC-025 graph checkpoint rows/format, or RFC-027 lineage-delta state as incidental B2 work. 6. **Give a distributed recovery fence its own design and evidence gate.** Define authority, expiry/renewal, fencing tokens, and crash semantics before implementation; require adversarial multi-process tests on local and object storage. -7. **Continue low-risk v8 hardening.** Add missing resource/time budgets, +7. **Continue low-risk v9 hardening.** Add missing resource/time budgets, preserve cost-at-history-depth gates, and reduce constant factors only where the existing authority model remains intact. 8. **Coordinate upstream without making it the calendar.** The additional @@ -348,12 +375,12 @@ or slightly lower incremental RSS. That is not a roadmap signal. ### Only when an evidence trigger fires -- **Public MemWAL row activation:** Phase A passed its bounded gate; private B1 - now closes the exact legal near-cap shape, and private B2a implements the - selected unbounded retain-all storage posture. The RFC remains Draft and public activation remains - off until explicit enrollment, durable contributor attribution, - compare-and-chain sequencing, strict correction/disposition, persistent - revisioned lifecycle with bounded management receipts, authorization, +- **Public MemWAL row activation:** Phase A passed its bounded gate; the private + v9 core now covers near-cap closure, unbounded retain-all, canonical + compare-and-chain sequencing, trusted attribution, and exact base+token + folding. The RFC remains Draft and public activation remains off until + request-idempotent explicit enrollment, strict correction/disposition, + persistent revisioned lifecycle with bounded management receipts, authorization, schema/SDK/API/CLI parity, cancellation ownership, and authoritative status pass their gates. Retained storage has no OmniGraph byte/file/history limit; provider exhaustion fails loudly. The exact upstream enrollment receipt/seal @@ -382,10 +409,9 @@ Do not: manifest-access cost; - implement RFC-024, RFC-025, or RFC-027 production paths behind a nominal feature flag before their blocking gate closes; -- treat the private schema-v8 B1 core or repaired near-cap closure as permission - to expose a put/ack endpoint before explicit enrollment, attribution, - compare-and-chain token, correction, persistent lifecycle, authorization, - and product-parity contracts are implemented and evidence-green; +- treat the private schema-v9 token/fold core as permission to expose a put/ack + endpoint before explicit enrollment, correction, persistent lifecycle/status, + authorization, and product-parity contracts are implemented and evidence-green; - delete or rewrite `_mem_wal` objects from OmniGraph, or interpret generic Lance version cleanup as MemWAL reclamation; - permit a base-table writer to advance any Phase A lifecycle's HEAD (including @@ -420,8 +446,9 @@ Do not: | MemWAL private Phase B1 admission/fold/crash and cost evidence | [`memwal_stream.rs`](../../crates/omnigraph/tests/memwal_stream.rs), [`memwal_stream_cost.rs`](../../crates/omnigraph/tests/memwal_stream_cost.rs), worker/recovery unit tests, [RFC-026 §12.3](../rfcs/0026-memwal-streaming-ingest.md) | | MemWAL Gate R0 retention decision, current-object census, attempt reuse, repaired near-cap closure, and fold RSS tripwire | [`memwal_stream_cost.rs`](../../crates/omnigraph/tests/memwal_stream_cost.rs), [RFC-026 §0.2 and §12.4](../rfcs/0026-memwal-streaming-ingest.md) | | MemWAL private B2a no-delete structure, provider-failure/orphan behavior, and 1/8/32/128 retained-history instrument | [`forbidden_apis.rs`](../../crates/omnigraph/tests/forbidden_apis.rs), [`memwal_stream.rs`](../../crates/omnigraph/tests/memwal_stream.rs), [`memwal_stream_cost.rs`](../../crates/omnigraph/tests/memwal_stream_cost.rs), shared [`helpers/memwal.rs`](../../crates/omnigraph/tests/helpers/memwal.rs), [RFC-026 §12.5](../rfcs/0026-memwal-streaming-ingest.md) | +| MemWAL private B2 canonical token/admission, stale-authority race, same-generation chain, recovery-v12 base+token fold, and attribution | [`memwal_stream.rs`](../../crates/omnigraph/tests/memwal_stream.rs), manifest/recovery/token unit tests, [`forbidden_apis.rs`](../../crates/omnigraph/tests/forbidden_apis.rs), [RFC-026 §12.6](../rfcs/0026-memwal-streaming-ingest.md) | | MemWAL B2b reclamation ownership/no-go guards | [`lance_surface_guards.rs`](../../crates/omnigraph/tests/lance_surface_guards.rs), [RFC-026 §4.5.2](../rfcs/0026-memwal-streaming-ingest.md) | -| Cross-version refusal/rebuild, including v6↔v7 and v7↔v8 | [`crossversion_upgrade.rs`](../../crates/omnigraph-cli/tests/crossversion_upgrade.rs) | +| Cross-version refusal/rebuild, including the genuine final-v8↔v9 gate | [`crossversion_upgrade.rs`](../../crates/omnigraph-cli/tests/crossversion_upgrade.rs), [CI workflow](../../.github/workflows/ci.yml) | Use [testing.md](testing.md) to find the existing owner before adding coverage. Decision instruments and production correctness tests serve different purposes: @@ -456,8 +483,14 @@ roots receive zero IO, canonical durable MemWAL delete requests remain zero, and only Lance's exact losing manifest-CAS temp staging may be removed. Warm ack operation shape stays flat while serialized authority and combined retained- history work grow; those measurements are advisory and create no ceiling. The -RFC remains Draft and every public surface remains inactive pending the common -correctness and product contracts. B2b's two additional guards prove why stock +private v9 token/fold suite additionally pins canonical digests, typed binding/ +sequence/idempotency conflicts, exact retry, same-generation chaining, +post-admission stale-authority refusal, exact two-participant recovery, and +durable fold attribution. The genuine final-v8↔v9 CI strand proves strict +two-way refusal and rebuild fidelity while excluding hidden trusted metadata +from export. The RFC remains Draft and every public surface remains inactive +pending lifecycle/correction/status, authorization, and product contracts. +B2b's two additional guards prove why stock RC.1 generic cleanup and raw successor-fence-sentinel deletion cannot implement a later managed profile; they are a no-go boundary for reclamation, not for retain-all. diff --git a/docs/rfcs/0024-durable-table-heads.md b/docs/rfcs/0024-durable-table-heads.md index 42a99e74..6c097c91 100644 --- a/docs/rfcs/0024-durable-table-heads.md +++ b/docs/rfcs/0024-durable-table-heads.md @@ -72,7 +72,7 @@ The first Gate A instrument rejected the proposed in-manifest BTREE shape on 2026-07-15. Exact indexed scan work is flat at fixed catalog width, but the complete required cost is not: latest-manifest discovery on uncompacted RustFS grows in object reads and bytes, and compacted reads still grow in bytes. -No heads-format production code ships from this RFC; current internal schema v8 +No heads-format production code ships from this RFC; current internal schema v9 preserves the journal-fold current-state path while research looks for a different substrate/access shape. @@ -657,7 +657,7 @@ coverage and does run against RustFS in CI. - Checkpoint retention is deferred. - The first in-manifest BTREE candidate is rejected: flat indexed scan work is insufficient while latest-manifest/object byte work grows with history. -- Production remains on current internal schema v8 without table heads; no +- Production remains on current internal schema v9 without table heads; no heads-format number or partial implementation is assigned. ### Gate status diff --git a/docs/rfcs/0025-checkpoint-retention.md b/docs/rfcs/0025-checkpoint-retention.md index 04154910..9cabe533 100644 --- a/docs/rfcs/0025-checkpoint-retention.md +++ b/docs/rfcs/0025-checkpoint-retention.md @@ -519,7 +519,7 @@ introduced here. ## 8. Format activation and rebuild compatibility Gate 0 was production-neutral and returned a no-go for the surveyed access -shape. The current shipped format is now internal schema v8, but none of the +shape. The current shipped format is now internal schema v9, but none of the activation or rebuild behavior below exists. The remainder of this section is the contingent format contract for a successor that first clears the same physical-I/O boundary; the Gate 0 result itself authorizes no implementation or @@ -569,7 +569,7 @@ strict-rebuild/refusal boundary plus functional engine create, list, show, and delete, the retention claim, exact tag verification, recovery, and pin reconciliation. Development-only pieces may land behind test-only seams before that slice, but production must stay on its then-current non-retention format -(currently v8) until the minimum usable contract is complete. +(currently v9) until the minimum usable contract is complete. ## 9. Observability and bounds @@ -628,7 +628,7 @@ Separately, all 23 Lance surface guards pass on RC.1. The RFC-025 cells prove ex main and named-branch tag targets, sparse cleanup pin/unpin behavior, and that a tag does not protect the named branch tree. These results validate the physical-pin architecture but cannot compensate for the failed registry-access -gate. Production remains on current schema v8 without retention until a +gate. Production remains on current schema v9 without retention until a successor access shape or revised operational contract passes the full boundary. @@ -683,7 +683,7 @@ boundary. | Phase | Content | Gate | |---|---|---| -| Gate 0 — pre-activation decision | production-neutral tag/cleanup semantics, physical-ref ABA, retention-claim, and checkpoint-registry physical-I/O instruments | **No-go (2026-07-17):** tags pass; compacted local registry scan bytes grow at 10→1,000 and scan operations add one boundary. S3 cost cell not run. Schema v6 remained unchanged at the gate; current v8 also ships no retention state | +| Gate 0 — pre-activation decision | production-neutral tag/cleanup semantics, physical-ref ABA, retention-claim, and checkpoint-registry physical-I/O instruments | **No-go (2026-07-17):** tags pass; compacted local registry scan bytes grow at 10→1,000 and scan operations add one boundary. S3 cost cell not run. Schema v6 remained unchanged at the gate; current v9 also ships no retention state | | A — minimum usable activation | row schemas and engine DTOs; deterministic V1 name/tag encoding; retention claim; create/list/show/delete; create sidecar, delete-operation marker, pin reconciler; strict-format/refusal/rebuild activation | **Blocked on a successor Gate 0 access shape or revised evidence-backed operational contract.** Then: schema and deterministic encoding vectors; old/new refusal; complete crash matrix; sparse-pin correctness; no inert format stamp | | B — destructive retention | offline cleanup integration and GC-boundary enforcement | blocked with A; then quiescence/refusal tests, complete root proof, and cost budgets | | C — operator surface | CLI, policy, audit, observability, docs, and rebuild runbook | blocked with A; then CLI outputs, policy parity, and genuine upgrade test | diff --git a/docs/rfcs/0026-memwal-streaming-ingest.md b/docs/rfcs/0026-memwal-streaming-ingest.md index 7e27af54..c930979f 100644 --- a/docs/rfcs/0026-memwal-streaming-ingest.md +++ b/docs/rfcs/0026-memwal-streaming-ingest.md @@ -10,8 +10,9 @@ owner: OmniGraph maintainers # RFC-026 — MemWAL streaming ingest -**Status:** Draft / Phase A, private Phase B1, and the private B2a unbounded -retain-all evidence gate implemented; common B2/public streaming inactive +**Status:** Draft / private B2 compare-and-chain/token-fold core and B2a +unbounded retain-all profile implemented; public streaming and +lifecycle management/correction/status inactive **Date:** 2026-07-10 **Gate E0 evaluated:** 2026-07-18 **Phase A foundation completed:** 2026-07-18 @@ -24,7 +25,7 @@ success, private B1 requires the same `ShardWriter` to pass `check_fenced()` before a clean acknowledgement; fence loss or an unreadable or unsettled check is `AckUnknown` plus worker retirement) **Phase B1 near-cap closure repaired:** 2026-07-21 (fresh scanner batches are -densified before retention; the legal high-entropy 32-MiB generation now folds +densified before retention; the legal high-entropy 32-MiB logical generation now folds and publishes exactly once; see §0.2 and §12.3) **Phase B2 contract inventory:** 2026-07-19 (§4.1–§4.6 specified the common admission/token, attribution, revision-fenced lifecycle, correction, retention, @@ -40,6 +41,13 @@ first profile; managed reclamation remains deferred (§4.5) guard, complete/partial provider-residue recovery matrix, and local/configured- RustFS retained-history evidence passed (§12.5); no schema or product surface was activated +**Private Phase B2 token/fold core completed:** 2026-07-22 — internal schema +v9, stream-config v3, lifecycle state-v2, manifest-selected graph-global +`_stream_tokens.lance`, canonical payload/token digests, post-admission +authority recapture, recovery-v12 exact base+token fold, durable graph-commit +attribution, and the genuine v8↔v9 refusal/rebuild gate passed (§11/§12.6); +explicit production enrollment, lifecycle management, correction/status, and +all product surfaces remain inactive **Author track:** Maintainer design series **Depends on:** [RFC-022](0022-unified-write-path.md)'s unified write and generic recovery-sidecar protocol, plus @@ -126,24 +134,33 @@ refusal/rebuild. It still exposes no production enrollment entry point and cannot append or acknowledge a row. RFC-026 therefore remains draft and public streaming remains inactive until the later gates close. -The implementation remains deliberately split. **Phase B1 is implemented -privately** for one admission-bounded, no-roll generation, from admission -through crash replay and one strict RFC-022 fold. The 2026-07-21 dense-scan -repair closes the legal near-cap row shape that Gate R0 exposed. The core is -reachable only through one feature-gated, doc-hidden engine test seam and is -not a product surface. The private **B2a unbounded retain-all** evidence gate -is also implemented: stock Lance owns the WAL, OmniGraph never deletes or +The implementation remains deliberately split. Historical **Phase B1** supplied +one admission-bounded, no-roll generation from admission through crash replay +and strict fold mechanics. The 2026-07-21 dense-scan repair closes the legal +near-cap row shape that Gate R0 exposed. The private **B2a unbounded +retain-all** evidence gate is also implemented: stock Lance owns the WAL, +OmniGraph never deletes or reclaims a canonical durable MemWAL object, and physical storage is allowed to grow monotonically without a file, object, or byte quota. Lance may delete its own losing `.binpb.tmp.` shard-manifest CAS staging files before they become canonical objects; this is atomic-write cleanup, not MemWAL GC. **B2b** remains the deferred managed-reclamation profile using a Lance-owned primitive. -The token, trusted-attribution, -recovery, lifecycle, correction, and product-parity contracts in §4.1–§4.4 and -§4.6 still apply. `GraphHistoryBudget`, physical-storage admission, and -aggregate receipt-capacity reservations do not. No schema intent, production -first use, SDK/HTTP/CLI/OpenAPI/Cedar surface, durable contributor attribution, -correction path, or public same-key `AckUnknown` retry contract is implemented. + +The current private B2 row/fold slice activates internal schema v9, +stream-config v3, lifecycle state-v2, canonical payload/token digests, trusted +hidden row attribution, manifest-selected graph-global token authority, and +recovery-v12. Admission recaptures mutable authority after shared admission and +same-key queue ownership; fold publishes exact base plus token effects, lifecycle, +lineage, and attribution only at one manifest CAS. Recovery-v11 is historical +v8 state. The core remains reachable only through one feature-gated, doc-hidden +engine test seam and is not a product surface. + +The remaining lifecycle, correction, explicit public-enrollment, status, +authorization, and product-parity contracts in §4.1–§4.4 and §4.6 still apply. +`GraphHistoryBudget`, physical-storage admission, and aggregate receipt-capacity +reservations do not. No schema intent, production first use, +SDK/HTTP/CLI/OpenAPI/Cedar surface, correction path, or public same-key +`AckUnknown` retry contract is implemented. ### 0.2 Gate R0: historical bounded-retention result and current disposition @@ -180,12 +197,12 @@ Lance RC.1.** Three findings were recorded: authority and repeated reopen can accumulate additional attempts. 2. RC.1 exposes no admission-grade, reserve-first conservative physical-output estimator or complete post-attempt storage receipt. The - 32-MiB Arrow admission cap is not a source-derived bound for + 32-MiB logical dense-slice Arrow admission cap is not a source-derived bound for data/Blob/transaction/manifest/deletion/PK/Bloom objects, local staging residue, or multipart/provider residue. Measurements can validate a formula; they cannot create one. 3. The deterministic high-entropy near-cap cell acknowledged a legal - 33,228,232-byte post-tombstone Arrow generation and retains 33,174,630 + 33,228,232-byte logical post-tombstone Arrow generation and retains 33,174,630 currently listed immutable bytes after acknowledgement. Materialization raised the listed retained total to about 65.1 million bytes. The old fold retained sparse `LsmScanner` slices whose variable-width arrays still owned @@ -232,10 +249,12 @@ quota. The 8,192-row and 32-MiB logical generation limits, one-resident-writer topology, exclusive-fold ownership, typed failures, and recovery barriers all remain. -This decision and repair authorize no internal schema -v9/config-v3/state-v2/recovery-v12 state, public stream contract, production -caller, or deletion. They only remove bounded physical retention from the -critical path and make the already-private B1 generation close for the widest +This decision and repair did not themselves authorize internal schema +v9/config-v3/state-v2/recovery-v12 state, a public stream contract, a production +caller, or deletion. The subsequent private B2 slice activates that format and +recovery state without changing retain-all or adding a product surface. Gate R0 +only removed bounded physical retention from the critical path and made the +already-private B1 generation close for the widest admitted shape. Public activation still requires the correctness and product gates in §12.6. @@ -510,16 +529,18 @@ intent conflicts; a different ID after enrollment returns effect-free exact incarnation and returns `StreamNotEnrolled` before body admission on `UNENROLLED`; it never auto-enrolls or invents a first-row exception. -## 4. Future public activation - -This section contains the future-public contract, not part of the Phase-B1 -private engine slice. **B2a unbounded retain-all** is the selected first -profile: it deletes no MemWAL object and performs no physical-storage admission -or accounting. **B2b** is the deferred managed-reclamation profile through the -Lance-owned protocol in §4.5.2. Token safety, trusted attribution, recovery, -revision-fenced lifecycle, correction, and a persistent quiesce/rebuild escape -remain common correctness contracts. B2a retains individually bounded protocol -records indefinitely and paginates their inspection; it has no aggregate +## 4. B2 contract and future public activation + +This section owns both the implemented private B2 row/fold contract and the +remaining future-public/control contract. Internal schema v9 now implements the +§4.1 token/attribution substrate and recovery-v12 fold portion of §4.4. It does +not expose a production caller. Explicit enrollment, lifecycle management, +correction, authoritative status, authorization, and product parity remain +future gates. **B2a unbounded retain-all** is the selected first profile: it +deletes no MemWAL object and performs no physical-storage admission or +accounting. **B2b** is the deferred managed-reclamation profile through the +Lance-owned protocol in §4.5.2. B2a retains individually bounded protocol +records indefinitely and paginates their future inspection; it has no aggregate receipt-count/byte cap, `GraphHistoryBudget`, retained-storage quota, or closure-capacity reservation. In-place SchemaApply and maintenance on an enrolled table stay refused until Phase D supplies automatic drain, witness @@ -599,15 +620,18 @@ this `sha256-lowerhex-v1` representation; changing it requires a new wire/config version rather than permissive dual decoding. The trusted metadata is stored in one reserved nullable physical struct, -provisionally `__omnigraph_stream_v1`, in every stream-capable base-table +`__omnigraph_stream_v1$`, in every stream-capable base-table schema. Pre-stream/direct rows have a null top-level struct. For a stream row, the incarnation, contributor, write ID, stream token, chain depth, payload digest, and origin tag are non-null; predecessor and fold-base tokens are nullable when the chain begins at null. The tagged origin has variant-specific nullability: exactly the `Admission` or `Correction` children selected by its -non-null tag are populated. The compiler reserves -the field; logical query results, ordinary export, user schema reflection, and -user-declared indexes never expose it. Lance therefore carries the exact +non-null tag are populated. The trailing `$` is deliberately outside the `.pg` +identifier grammar; accepted-SchemaIR validation still rejects a hand-authored +lookalike. The v8-valid user property `__omnigraph_stream_v1` (without `$`) +remains ordinary data and survives rebuild. Logical query results, ordinary +export, user schema reflection, and user-declared indexes never expose the +physical field. Lance therefore carries the exact metadata atomically with the row through WAL, replay, flushed generation, fold, and base-table publication. This is intentionally a base-schema field: RC.1's `put_no_wait` accepts the base schema and adds only Lance's own @@ -623,10 +647,29 @@ descriptor; a payload that still requires post-ack dereferencing is not self-contained and is refused. Stream-config v3 pins the digest version and canonical encoding. +Before B2 can materialize a blob or allocate that canonical encoding, it must +reserve one root-scoped worst-case preprocessing envelope: the original legal +32-MiB Arrow row, a possible 32-MiB materialized replacement, and at most 64 MiB +of canonical bytes (128 MiB total), plus the ordinary inflight-call slot. The +inflight slot transfers into queued/worker ownership; the preprocessing +reservation remains held until the owned payload digest exists, then releases. +The private profile permits two such envelopes root-wide (256 MiB total), the +minimum overlap that lets one provisional caller wait while another establishes +the authority it must later revalidate. A third concurrent preprocessor fails +before allocating or invoking Lance with typed resource +`stream_b2_preprocessing_bytes`; this bound is process memory admission, not a +retained-storage quota. + A graph-internal Lance dataset, provisionally `_stream_tokens.lance`, is the sole **post-fold** per-key sequencing authority. It is initialized by the new format strand and addressed only through the exact version selected by -`__manifest`, never by its raw Lance HEAD. Its current row is keyed by +`__manifest`, never by its raw Lance HEAD. The token pointer's durable witness +is main-branch version plus Lance transaction UUID. Its shared +`CurrentHeadWitness.manifest_e_tag` slot is canonically `None`: local object +stores derive ETags partly from the inode, so copying an otherwise exact graph +changes that provider-local value. Strict transactions and recovery still +fence every token effect; a provider-local ETag is not graph identity. Its +current row is keyed by `(stable_table_id, incarnation_id, logical_id)` and carries the immutable `origin_enrollment_id` under which that token became current, stream incarnation, current stream token, write ID, predecessor token, @@ -674,6 +717,22 @@ overlay; an exact duplicate becomes `already_durable`, while a successor must have obtained and named the confirmed token in a later request. The adapter never treats an opaque pending candidate as caller authority. +Exact token/base probes stream their result rather than collecting all batches. +They stop after requested-key-count plus one row, reject duplicates/foreign +keys, bound each Arrow emission, and charge retained current-authority maps +against the same fixed 32-MiB token-projection ceiling. Separately valid large +historical authorities therefore fail loudly in aggregate instead of making a +later small generation allocate memory proportional to prior per-key maxima. + +The initial schema/lifecycle/HEAD capture used to select the admission domain +is explicitly provisional. A caller may wait behind an exclusive fold after +that capture. Once it owns shared admission and the same-key queue, the +implemented adapter re-lists relevant recovery and recaptures the accepted +schema, stable binding, lifecycle revision, current HEAD witness, stream +incarnation, and manifest-selected token witness before token classification or +`put_no_wait`. Any movement returns an effect-free typed conflict/retry. A stale +provisional capture can locate a gate; it can never authorize a WAL effect. + The overlay also mints a compact fold certificate for each same-key chain. The first row for a key in one generation stores the manifest-selected current token as `fold_base_token` and depth 1; every later admitted row inherits that @@ -1159,7 +1218,7 @@ pageable without a tie whose order can flip. It does not store or expose whole row payloads. Under exclusive admission, the read-only block- inspection operation lists relevant recovery without resolving it, binds the complete `DRAINING` row/block/base witness, scans the at-most-8,192-row/32-MiB -cut, reruns that pinned validator, and requires count/digest equality. It returns +logical dense-slice cut, reruns that pinned validator, and requires count/digest equality. It returns pages in that canonical order, capped by both entry count and serialized bytes, with an opaque cursor bound to `(block_token, correction_view_digest, lifecycle_revision, last_entry_ordinal)`, then rereads the complete authority @@ -1222,7 +1281,7 @@ Unmentioned keys retain the original generation's LWW winner. The correction does not append another MemWAL generation and does not create a custom side log. Under exclusive admission it scans the one immutable cut, applies the bounded overlay in memory, reruns complete fold validation, and enforces the -8,192-row/32-MiB post-overlay bound. A validation failure creates no sidecar, +8,192-row/32-MiB logical dense-slice post-overlay bound. A validation failure creates no sidecar, base effect, token effect, lifecycle movement, or correction success. Once valid, recovery-v12 `StreamCorrection` binds the prior complete @@ -1234,7 +1293,7 @@ applies one bounded keyed-upsert batch with at most one image per resulting then publishes at most 8,192 resulting `PRESENT | WITHDRAWN` current-token rows in deterministic chunks and the one `CorrectionReceipt` row in a final separate transaction on that same dataset. Each transaction independently obeys the -sealed 8,192-row/32-MiB bound; the receipt transaction also obeys the +sealed 8,192-row/32-MiB logical dense-slice bound; the receipt transaction also obeys the source-derived single-receipt byte bound. Stream-config v3 fixes a source- derived `max_correction_token_transactions`, and admission proves that the worst-case token projection for the complete legal generation plus the receipt @@ -1281,7 +1340,7 @@ provider capacity. This changes only the physical-retention contract. Row and memory admission remain bounded at one no-roll generation of at most 8,192 rows and 32 MiB of -logical Arrow data; worker count, queues, deadlines, execution retries, and +logical dense-slice Arrow data; worker count, queues, deadlines, execution retries, and fold/correction transaction shapes remain explicitly bounded. Every acknowledgement still requires watcher durability plus the same-writer fence check. Every authority-changing effect still uses the existing manifest and @@ -1815,8 +1874,8 @@ passes a non-contiguous range to B1. Previously acknowledged runs in the same request remain durable; the response is a stream, not an all-request transaction. Ordering, cancellation, and retry rules are explicit: -Before adding a normalized row to a run, the adapter computes its exact stored -Arrow charge after tombstone/hidden-metadata injection. A single row above +Before adding a normalized row to a run, the adapter computes its logical +dense-slice Arrow charge after tombstone/hidden-metadata injection. A single row above 32 MiB is `stream_input_too_large`; the adapter closes the preceding run and may continue with later lines. Otherwise it forms only runs that fit both the per-call bound @@ -1949,9 +2008,10 @@ restart. Phase B1 admits exactly one non-empty normalized `RecordBatch` per Lance `put_no_wait`. Each call and the **entire active generation**, including any duplicate batch submitted while that live generation is still active, are -capped at 8,192 rows and 32 MiB of materialized Arrow memory. The charged -representation is the exact stored MemWAL batch after normalization/defaulting and internal -`_tombstone=false` injection. These are hard OmniGraph reservations made before +capped at 8,192 rows and 32 MiB of logical dense-slice Arrow bytes. The charged +representation is the selected slice after normalization/defaulting and internal +`_tombstone=false` injection; backing-buffer capacity and physical allocation +are deliberately excluded. These are hard OmniGraph reservations made before the row put and use the sealed keyed writer's numeric limits; they are not Lance's soft thresholds. Cheap raw row/byte bounds reject obviously over-cap input before recovery I/O; a raw-fit batch then receives exact post-tombstone @@ -2062,7 +2122,7 @@ stay empty; a widest-legal normalized generation guard proves Lance's trigger estimate—BatchStore bytes plus the PK Bloom filter—cannot reach either byte threshold. RC.1 omits the mandatory PK index from that estimate, so its actual memory is covered only by the separate RSS evidence gate, never by the hard -32-MiB Arrow reservation. The worker explicitly seals and drains its one +32-MiB logical dense-slice Arrow reservation. The worker explicitly seals and drains its one generation, retires that writer without admitting into the replacement MemTable, folds the generation, then reopens the binding at a higher epoch before the next put. An observed automatic rollover is @@ -2210,7 +2270,7 @@ Reopen has an exact fail-closed routing table before any put: fold-only and resumes that generation; 4. an unmerged flushed generation plus non-empty active data, more than one unmerged generation, any frozen MemTable, an unexpected generation number, - or replay beyond the 8,192-row/32-MiB cap fails closed. + or replay beyond the 8,192-row/32-MiB logical dense-slice cap fails closed. This is stricter than merely rebuilding a reservation. In pinned RC.1, `replay_memtable_from_wal` inserts the replayed batches into a fresh @@ -2234,15 +2294,16 @@ not a second WAL implementation. Generation reservation is restart-derived, not a drifting runtime counter. After claim/replay the worker snapshots `active.batch_store` and sums every -physical stored batch's rows plus `RecordBatch::get_array_memory_size()` in the -same post-tombstone representation charged pre-put. An empty admissible reopen +stored batch's rows plus each array's `ArrayData::get_slice_memory_size()` in +the same post-tombstone logical representation charged pre-put. An empty +admissible reopen starts at zero; a non-empty result is validated against the hard cap and routed to fold-only, never used to continue admission. The serialized worker updates the derived total after each completed invocation while that one live generation remains open. Surface guards pin `in_memory_memtable_refs`, public BatchStore iteration/StoredBatch data, the watermark bridge, and accounting for -replayed duplicates; Lance's different `MemTableStats::estimated_size` is not -substituted for the 32-MiB contract. +replayed duplicates; Lance's different `MemTableStats::estimated_size` and +backing-buffer capacity are not substituted for the 32-MiB logical contract. ## 6. Fold protocol @@ -2259,7 +2320,7 @@ all flush scheduling can report success, and its handler/channel or object- store work can stall. The registry task—not the requesting future—owns the seal/drain/abort sequence and the exclusive admission lease until it settles. Before a cold fold invokes the writer opener, it atomically reserves one -resident slot, one pending generation, the full 32-MiB generation budget, and +resident slot, one pending generation, the full 32-MiB logical generation budget, and the same-key fold-only marker. The opener runs in an owned task and is awaited under the same seal deadline used by the later cut. A caller timeout never cancels that task: the continuation retains the opener, exclusive authority, @@ -2327,11 +2388,13 @@ re-applied. B1 folds the already-normalized physical rows it acknowledged; physical vector columns pass through like other columns. It does not call an external embedding provider or materialize unspecified fold-derived fields. The deduplicated output is rechecked against the one sealed keyed-transaction -limit of 8,192 rows and 32 MiB. An over-limit result is strict-blocked before -any table effect and leaves the acknowledged generation durable. The current -B1 transform has no output-expansion source; a future derived-field transform -must add separate evidence for that bound rather than inheriting this claim. -B1 never splits one generation across transactions and never marks it merged +limit of 8,192 rows and 32 MiB of logical dense-slice Arrow bytes. Backing-buffer +capacity and process RSS are not the admission metric; RSS is retained only as +remeasurement evidence. An over-limit result is strict-blocked before any table +effect and leaves the acknowledged generation durable. The current transform +has no output-expansion source; a future derived-field transform must add +separate evidence for that bound rather than inheriting this claim. The fold +never splits one generation across transactions and never marks it merged after a partial prefix. With or without RFC-024, the `ReadSet` carries schema identity, the complete stream binding/configuration/generation cut, the base table's exact @@ -2344,28 +2407,33 @@ commit. RFC-024 may later narrow false contention with table heads but is not a correctness dependency. The commit phase then: 1. stages accepted rows with Lance merge-insert and includes the exact - `MergedGeneration` cut in that same transaction; -2. acquires every affected queue in canonical sorted order and revalidates the - complete `ReadSet`; any mismatch discards and replans the whole fold; -3. writes one dedicated schema-v11 `StreamFold` payload inside RFC-022's shared - recovery envelope before `commit_staged`; the payload carries stable table - identity, the exact physical binding and prior witness, shard/generation - cut, exact transaction identity plus merged-generation set, and fixed - lineage; -4. commits the staged Lance transaction with zero transparent conflict retries; -5. durably confirms the achieved table version, transaction, and merge-progress - effect; -6. captures the achieved base-table witness and publishes the table version, - next lifecycle `CurrentHeadWitness`, and fixed lineage in one - `__manifest` CAS, including table heads when RFC-024 is active; -7. deletes the sidecar after successful publication. - -That seven-step list is the implemented B1 one-table fold. B2 preserves its -ordering but recovery-v12 adds the exact `_stream_tokens.lance` transaction and -fixed attribution summary. The base and token effects are both classified -before the same manifest CAS; neither may publish independently. A correction -uses the same multi-participant envelope described in §4.4. It does not turn a -generation into multiple base-table keyed transactions. + `MergedGeneration` cut in that same base transaction; +2. derives and validates the exact winner token rows and durable fold- + attribution commitment, then stages one exact keyed transaction against the + manifest-selected `_stream_tokens.lance` version; +3. acquires every affected queue in canonical order and revalidates the + complete `ReadSet` and winner set; any mismatch discards the effect-free plan + and replans the whole fold; +4. writes one dedicated schema-v12 `StreamFold` payload before either commit; + it carries stable table identity, exact binding and prior witnesses, + shard/generation cut, both pre-minted transaction identities, planned token + rows, complete lifecycle state-v2 outcome, fixed lineage, and attribution; +5. commits the exact base transaction and then the exact token transaction with + zero transparent conflict retries; +6. durably confirms both achieved versions, transactions, merge progress, + witnesses, and the complete fixed manifest outcome; +7. publishes the base pointer, token pointer, next lifecycle + `CurrentHeadWitness`, fixed lineage, and fold attribution in one + `__manifest` CAS; and +8. deletes the sidecar after successful publication. + +That eight-step list is the implemented private B2 fold. Neither participant +may publish independently. Exact no-effect on both participants may retire and +replan; if the exact base effect landed but the token effect did not, recovery +may execute only the pre-minted token transaction from the durable plan. A +foreign, buried, token-only, or ambiguous partial outcome fails closed. A future +correction uses the same multi-participant envelope described in §4.4; it does +not turn a generation into multiple base-table keyed transactions. Because `_stream_tokens.lance` is one graph-global physical participant, B2 adds one root-shared stream-token gate. The universal order for a manifest @@ -2374,8 +2442,9 @@ stream token -> sorted graph tables`; a writer with no relevant stream admission starts at schema, and no caller acquires a newly discovered stream lease after that gate. Every fold or correction captures the token dataset's exact manifest-selected version, -transaction UUID, and e_tag in its `ReadSet`; it never opens raw token HEAD as -current authority. The final gate rechecks that witness plus every winner's +transaction UUID in its `ReadSet`; it never persists or compares a +provider-local e_tag and never opens raw token HEAD as current authority. The +final gate rechecks that witness plus every winner's stream incarnation/fold-base certificate before sidecar arm. A B2 recovery sidecar that can move the token participant is graph-global relevant to **every** operation that may publish `__manifest` or main authority: stream @@ -2394,9 +2463,9 @@ still amortize many row acknowledgements, but B2 does not claim independent per-table token commits. A token conflict proven effect-free across **both** participants may finalize and fully reprepare from fresh manifest authority; any base or token effect, or ambiguous classification, retains recovery -ownership and returns `RecoveryRequired`. Recovery rolls the exact pair forward -or compensates it under the captured authority; it never adopts a token-table -HEAD merely because its rows look compatible. +ownership and returns `RecoveryRequired`. Recovery converges only the exact +pre-minted pair under captured authority; it never adopts a token-table HEAD +merely because its rows look compatible. The adapter is a distinct writer kind, not a Mutation payload with extra fields. The sidecar is mandatory even though merge-insert is staged. After @@ -2428,12 +2497,13 @@ waiters, and never aborts an acknowledged row. Default graph reads remain on the old manifest pointer after the table commit and see the rows only after the single manifest CAS. -Phase B1 records fixed fold mechanism lineage only because it has no public -caller. B2's accepted design embeds trusted contributor/write metadata in the -stored row and publishes the §4.2 winner summary plus current token evidence. -It never infers provenance from MemTable positions or WAL cursor statistics. -The design is not active until the v9/config-v3/recovery-v12 implementation and -evidence gates pass; restart-stable dead-letter row identity remains Phase C. +The private B2 core embeds trusted contributor/write metadata in the stored row +and publishes the §4.2 winner summary plus current-token evidence. The graph +commit stores a durable commitment to the visible contributor/write winner set; +ordinary commits carry no fold summary. Provenance is never inferred from +MemTable positions or WAL cursor statistics. This row/fold design is active in +v9; public exposure, lifecycle/correction/status, and restart-stable dead-letter +row identity remain later gates. ## 7. Fold-time rejection is atomic @@ -2515,7 +2585,7 @@ covers the higher-epoch claim while the gate remains exclusively closed and is resolved before any ack path proceeds. The drain itself is not one giant sidecar spanning multiple commits. -`DRAINING` is not allowed to become an operator trap. B2 implements the +`DRAINING` is not allowed to become an operator trap. Public B2 must implement the protocol-v2 descriptor and recovery-v12 transition specified in §4.3, including a crash-safe abort transition for a quiesce that cannot finish: only when no guarded schema/maintenance operation has begun, the exact current DRAINING row @@ -2673,7 +2743,7 @@ Richer index-catchup, age, and reject detail may follow in Phase C. Lance's labeled hints; neither is presented as a durable per-row receipt or a way to resolve one `AckUnknown` attempt. -Before B2 activation, internal metrics must cover ack latency, +Before public B2 activation, internal metrics must cover ack latency, durability-wait batching, fenced writers, replayed entries, fold rows/bytes/generations, fold retries, lag, reject counts, and sidecar recovery. They also cover compare-and-chain conflicts/idempotent hits, token-table @@ -2682,6 +2752,10 @@ profile claim outcomes. B2b additionally covers its reservation ledger, admission watermark, reclaim/checkpoint attempts, and receipts. Defaults for every active row/memory/count/time bound must be documented, and configuration changes are observable behavior. B2a has no storage-limit default to report. +The implemented private B2 default includes two 128-MiB root preprocessing +reservations (256 MiB total) acquired before blob materialization/canonical encoding; they are +separate from the 32-MiB queued-generation Arrow charge and does not weaken the +retain-all storage profile. The accepted acknowledgement-cost instrument scopes a warm, already-claimed writer in steady state and includes both WAL append and synchronous in-memory @@ -2702,12 +2776,15 @@ but cannot claim history-flat cost while scanning manifest history. ## 11. Format activation and rebuild Streaming is a graph-format capability, not a feature activated by the first -enrollment. Internal schema v8 now preserves the Phase-A foundation and adds -the private B1 data-bearing capability on every new root: identity-keyed -lifecycle rows are a recognized authority kind, v10 enrollment and v11 fold -intents are recoverable, and uncovered lifecycle/MemWAL mismatches are refused. -A physical enrollment adds one table's MemWAL index, empty shard, and exact -lifecycle row; it does not change the graph stamp. +enrollment. Internal schema v9 is now the only served format. It preserves the +Phase-A enrollment foundation and bounded B1 worker mechanics, recognizes +identity-keyed lifecycle state-v2, initializes manifest-selected +`_stream_tokens.lance` authority, and recovers v10 enrollment plus v12 +base+token fold intents. A historical v11 fold is refused because it lacks the +token participant and complete state-v2 outcome. Uncovered lifecycle, token, +or MemWAL mismatches are refused. A physical enrollment adds one table's +MemWAL index, empty shard, and exact lifecycle row; it does not change the graph +stamp. V7 activation followed Gate E0 and the bounded enrollment/recovery, writer-exclusion, lifecycle, crash, and refusal/rebuild evidence. It is not @@ -2727,7 +2804,7 @@ refuses it and requires export/init/load into a different v8 root. The genuine v7↔v8 old-binary/new-format and new-binary/old-format refusal/rebuild evidence now passes (§12.3). V8 remains a private-core format with no public B2 caller. -B2 is a third strict strand: internal schema v9, stream-state protocol v2, +B2 is the third strict strand: internal schema v9, stream-state protocol v2, stream-config v3, and recovery-v12. V9 adds the reserved nullable stream-row metadata to stream-capable physical schemas and initializes the manifest-selected `_stream_tokens.lance` authority. It does **not** add a @@ -2738,10 +2815,14 @@ effects; unreferenced physical residue remains retained and inert. V8 rows, lifecycle payloads, and private acknowledgements are never reinterpreted or assigned contributor/token evidence with serde defaults. The -implementation must prove genuine v8↔v9 refusal plus export/init/load rebuild -before any production caller exists. These version assignments are part of the -future B2 contract; this design amendment does not activate them in code or -change the currently served v8 format. +genuine v8↔v9 gate now builds the final schema-v8 binary, proves old-binary/new- +format and new-binary/old-format refusal, and performs strict +export/init/load rebuild. The rebuilt v9 graph preserves logical rows, +caller-supplied physical vector values, and exact-`id` PK metadata. Ordinary +export deliberately omits `__omnigraph_stream_v1$` and does not transfer token +authority. The pinned fixture includes a genuine v8 user property named +`__omnigraph_stream_v1` and proves its value round-trips unchanged. These +assignments are active private-core format state, not a public B2 contract. Old binaries refuse a stream-capable graph before reading or writing any table. A stream-capable binary refuses an older graph before running recovery. On a @@ -2761,9 +2842,8 @@ and not permission to attach old shards. Recovery classifies the old and new bindings independently and retains the old artifacts until their sidecars and read guards permit reclamation. -There is no in-place activation or rollback to an old format. A v6 graph moves -to v7, and a v7 graph moves to v8, only through the strict -export/init/load strand. Neither v6 nor v7 can contain publicly acknowledged +There is no in-place activation or rollback to an old format. V6→v7, v7→v8, +and v8→v9 move only through the strict export/init/load strand. Neither v6 nor v7 can contain publicly acknowledged MemWAL rows, so the stream-specific quiesce steps below are vacuous for those transitions; they become load-bearing for any later rebuild from a format that exposes durable admission: @@ -2998,8 +3078,8 @@ RFC remains draft. and fixed lineage; default reads remain old until that CAS. - Resource reservations cover queued input, the resident generation, and the sealed cut through fold publication. The evidence-qualified private limit is - one resident writer and a nominal 32-MiB aggregate Arrow admission budget per - graph root; cold replay discovered after callers were charged is the narrow + one resident writer and a nominal 32-MiB aggregate logical dense-slice Arrow + admission budget per graph root; cold replay discovered after callers were charged is the narrow transient-overlap exception above. A cold fold reserves the full budget plus resident/pending slots before its owned opener and carries the original seal deadline through that open. Any higher resident concurrency or larger steady @@ -3013,7 +3093,7 @@ RFC remains draft. **Checked evidence** - In-source worker tests pin exact config-v2 reconstruction/no-roll settings, - physical Arrow accounting, ordinal validation, root-registry sharing, + logical dense-slice Arrow accounting, ordinal validation, root-registry sharing, pre-effect resource refusals, fold-reservation lifetime, and the rule that a deadline continuation owns its authority until one settled handoff. A source guard rejects direct timeout cancellation around Lance futures. @@ -3037,7 +3117,7 @@ RFC remains draft. - Forbidden-API coverage keeps the implementation private and rejects schema, SDK, server, CLI, OpenAPI, or generic raw-Lance row side doors. - The genuine cached-binary v7 ↔ v8/config-v2 matrix passes in both directions: - current v8 refuses the final v7 image before table use, v7 refuses a genuine + the then-current v8 binary refuses the final v7 image before table use, v7 refuses a genuine v8 root, and v7 export → v8 init/load → v8 re-export preserves the logical rows and exact-`id` PK contract. The v7 fixture is unenrolled because that binary exposes no production enrollment route; this proves the real format @@ -3073,14 +3153,14 @@ RFC remains draft. 38.426/49.253 ms. Rerun this cell before a current object-store ack-cost claim. These are evidence-run observations, not a latency SLO or group- commit multiplier. -- The widest one-batch generation reserves 33,228,232 Arrow bytes with a +- The widest one-batch dense generation reserves 33,228,232 logical Arrow bytes with a 33,203,240-byte RC trigger estimate. The worst fragmented 8,192-one-row-batch shape reserves 33,103,872 Arrow bytes with a 29,343,744-byte trigger estimate; both remain below the explicit 1-GiB no-roll threshold and 8,192 remains below the 8,193 row/batch threshold. Isolated one-batch whole-process peak RSS moves from 74,334,208 to 206,471,168 bytes (+132,136,960), explicitly including Arrow, PK-index, runtime, and allocator overhead. That result qualifies one - resident writer with one 32-MiB aggregate Arrow reservation. Queued batches + resident writer with one 32-MiB aggregate logical Arrow reservation. Queued batches share that reservation rather than accumulating outside it. The repaired high-entropy widest fold publishes all 8,192 rows and measures an isolated whole-process peak-RSS delta of 284,934,144 bytes (about 272 MiB / 285 MB) @@ -3115,8 +3195,9 @@ RFC remains draft. protocol. The Lance-owned post-success fence and retention patch therefore remain B2b managed-reclamation and broader-topology gates. They did not block the completed private no-delete, single-live-writer-process B2a gate, which - retains the wrapper's post-watcher fence check. Public activation still waits - on B2-common. + retains the wrapper's post-watcher fence check. The later private v9 + token/fold slice passed independently; public activation still waits on the + remaining lifecycle/correction/status, authorization, and product gates. **Phase B1 disposition — accepted 2026-07-19, closure gap found 2026-07-20 and repaired 2026-07-21** @@ -3136,11 +3217,14 @@ latency SLO, group commit, Lance-internal failure injection, or the unavailable forged pre-claim/pre-put stream-sidecar cell. Gate R0 correctly found that the old scanner retained sparse backing buffers. -The dense copy releases those buffers and restores the intended logical -32-MiB closure check; the near-cap cell now closes. This amendment changes no -product surface. The B2-common contracts in §4.1–§4.4 and §4.6 remain -unimplemented; §4.5.1's private B2a gate is implemented, while §4.5.2's B2b -managed-reclamation profile remains optional and inactive. +The dense copy releases those buffers and restores the intended 32-MiB logical +dense-slice Arrow closure check; physical RSS remains a separate evidence +tripwire. The near-cap cell now closes. This amendment changed no product +surface. The subsequent v9 slice implements §4.1's private token/attribution +and §4.4's base+token fold core. Explicit enrollment, lifecycle management, +correction/status, and product parity remain inactive; §4.5.1's B2a profile is +implemented, while §4.5.2's B2b managed-reclamation profile remains optional +and inactive. ### 12.4 Gate R0 — historical bounded-retention no-go; closure repaired @@ -3229,126 +3313,133 @@ reads; this is the real combined accumulated base-table, graph-manifest, and retained-WAL shape, not an isolated WAL slope. Wall times and whole-process RSS are printed as diagnostics only and enforce no product threshold. -This result adds no schema v9, production caller, SDK/HTTP/CLI/OpenAPI/Cedar -surface, or lifecycle/correction implementation. Those remain §12.6 work. - -### 12.6 Common public and B2b managed-reclamation activation gates - -The private §12.5 B2a gate has passed. The common public contracts in -§4.1–§4.4 and §4.6 are therefore the next implementation work. This section -owns those common gates and B2b's optional managed-reclamation gates. B2a does -not need any B2b-only bullet. Public activation still waits for every common -implementation/evidence item to be green. +This B2a result itself added no schema or product surface. The subsequent +private B2-common row/fold slice activated schema v9; production callers and +lifecycle/correction/status remain §12.6 work. + +### 12.6 Private B2-common implementation and remaining public/B2b gates + +The private §12.5 B2a gate and the private B2 row/fold core have passed. The +implemented core covers schema v9/config-v3/state-v2 format activation, +canonical payload/token derivation, trusted row attribution, graph-global token +authority, stale-authority admission revalidation, same-generation chaining, +recovery-v12 exact base+token folding, durable graph-commit attribution, and +genuine v8↔v9 refusal/rebuild. Public activation still waits for explicit +production enrollment, lifecycle/correction/status, authorization, +cancellation/shutdown, API compatibility, and product parity. This section also +owns B2b's optional managed-reclamation gates; B2a does not need any B2b-only +bullet. The design does not waive the persistent escape requirement: a user must never be left with a table that ordinary writers refuse but cannot be corrected, quiesced, or rebuilt. -- `@stream(mode="upsert", on_reject="strict")`, production first use, SDK, - HTTP, CLI, Cedar, and OpenAPI all route through the same private B1 core. The - existing `/ingest` behavior remains compatible, and embedded/remote command - parity is tested. Accepted-schema and runtime guards refuse `@stream` on a - type requiring `@embed` or any external/provider-derived field; caller- - supplied physical vectors round-trip without provider invocation. -- Explicit enrollment tests cover `UNENROLLED` status, request-ID same/different +- **Inactive product surface:** `@stream(mode="upsert", on_reject="strict")`, + production first use, SDK, HTTP, CLI, Cedar, and OpenAPI must all route + through the same private core. Existing `/ingest` behavior must remain + compatible, and embedded/remote command parity must be tested. + Accepted-schema and runtime guards must refuse `@stream` on a type requiring + `@embed` or any external/provider-derived field; caller-supplied physical + vectors must round-trip without provider invocation. +- **Inactive explicit enrollment:** tests must cover `UNENROLLED` status, request-ID same/different intent, every selected-profile bootstrap/shard/lifecycle crash boundary, lost result after sidecar deletion through the persisted request-ID/intent receipt, `already_enrolled`, `stream_manage` authorization with Cedar/ embedded/remote parity, first returned stream incarnation, and - request-level `StreamNotEnrolled` before ingest body admission. Ingest never - creates physical enrollment implicitly. B2b additionally covers every + request-level `StreamNotEnrolled` before ingest body admission. Ingest must + never create physical enrollment implicitly. B2b must additionally cover every genesis body/pointer/new-details crash boundary. -- The public acknowledgement is status-only and caller-ordered. It maps the - private durable batch result back to each caller ordinal and reports the +- **Inactive public acknowledgement adapter:** the response must be status-only + and caller-ordered. It must map the private durable batch result back to each + caller ordinal and report the stream incarnation, write ID, and current binding, not a WAL position, generation, or Lance `batch_positions` value. Only durable/current outcomes - report a confirmed token and persisted tagged origin; `ack_unknown` labels + may report a confirmed token and persisted tagged origin; `ack_unknown` must label its candidate unconfirmed, while invalid/conflict/not-invoked outcomes mint neither. Every exact response variant in §4.6, including `invalid`, `stream_input_too_large`, lifecycle/authority/recovery blockers, and - `stream_retry_required`, has a schema. The public adapter owns contiguous + `stream_retry_required`, must have a schema. The public adapter must own contiguous physical-run boundaries around invalid lines and token dispositions plus its - bounded reorder buffer. Tests cross a partially full generation, row/logical- + bounded reorder buffer. Tests must cross a partially full generation, row/logical- memory and queue/deadline limits, an intrinsically oversized row on an empty generation, authority/lifecycle movement or `RecoveryRequired` between runs, and a pre-invocation queue deadline after earlier lines are already durable; the - blocking and remaining otherwise-admissible lines receive their exact effect- + blocking and remaining otherwise-admissible lines must receive their exact effect- free capacity/backpressure status while later parse/schema/normalization failures remain `invalid` and intrinsically oversized rows retain their own - status. The same precedence is pinned after `AckUnknown`. No handler converts + status. The same precedence must be pinned after `AckUnknown`. No handler may convert partial NDJSON success into an HTTP error. -- Internal schema v9/config-v3/state-v2/recovery-v12 provisions the hidden row - metadata and manifest-selected token dataset. Every base/token effect is one - recovery-covered publication. The graph-global token participant follows - canonical `sorted relevant stream admission -> schema -> main branch -> - stream token -> sorted graph tables` gate order; folds capture - its exact manifest-selected - witness in the read set and never use raw token-table HEAD. If either the base - or token participant has an effect or ambiguous outcome, recovery owns both; - transparent reprepare is allowed only after proving both effect-free. - Current-token mismatch, stream-incarnation mismatch, reused write ID with a - different digest/actor, historic UUID reuse against a newer predecessor, and - the exact `X(unknown) -> Y -> retry X` sequence all follow §4.1 before - `put_no_wait`; idempotent-current retry performs zero WAL writes. Fold proves - each LWW winner's `fold_base_token`/`chain_depth` certificate. An unconfirmed - candidate is never accepted as a predecessor, and each physical run contains - at most one fresh occurrence per key. OpenAPI/SDK/CLI round-trips pin the exact - `sha256:` plus 64 lowercase-hex wire form for stream/block tokens and protocol - digests; uppercase, base64, whitespace, bad prefix, and wrong length refuse - pre-effect. Tests also pin - the universal global-token-sidecar barrier across every ordinary manifest - writer and the release-all-gates/restart rule after late discovery. -- Minimum operator controls expose authoritative status, explicit fold, plus +- **Implemented privately:** schema v9/config-v3/state-v2 provisions the hidden + row metadata and manifest-selected token dataset. Canonical payload/token + digests bind accepted schema, table/key identity, stream incarnation, + predecessor, write ID, trusted contributor, and normalized payload. After + acquiring shared admission and the same-key queue, admission recaptures live + schema/binding/lifecycle/HEAD/token authority before `put_no_wait`; a stale + provisional capture is effect-free. Current-token mismatch, + stream-incarnation mismatch, reused write ID with a different digest/actor, + exact retry, and same-generation chains are typed and tested. Recovery-v12 + owns exact pre-minted base and `_stream_tokens.lance` transactions, planned + token winners, state-v2 outcome, fixed lineage, and attribution. Only exact + base+token may publish both pointers and the graph-commit fold commitment; + exact base-only recovery may complete only the planned token effect. V11 is + historical and refused under v9. The graph-global token gate and + release-all-gates/restart rule cover every manifest writer. +- **Still inactive at the product boundary:** OpenAPI/SDK/CLI round-trips must + pin the exact `sha256:` plus 64 lowercase-hex wire form for public tokens and + protocol digests; uppercase, base64, whitespace, bad prefix, and wrong length + must refuse pre-effect. Public caller ordering, mixed request shaping, + cancellation/shutdown, and transport parity retain their gates above. +- **Inactive operator controls:** minimum controls must expose authoritative status, explicit fold, plus persistent quiesce, resume/abort-drain, correction, and rebuild preflight. - Status takes an exclusive cut, settles owners without mutating recovery, and - includes exact lifecycle/binding/revision, epoch, advisory pending + Status must take an exclusive cut, settle owners without mutating recovery, and + include exact lifecycle/binding/revision, epoch, advisory pending generations/bytes, paginated correction and management/claim receipt summaries, last fold summary, and strict-blocked state. Every mutating call after - enrollment carries an operation ID and expected revision; same-ID/same-digest - returns the complete bounded terminal receipt, same-ID/different-digest - conflicts, and a stale revision never retargets. Quiesce requires a caller - drain ID, has a crash-safe cut, never records terminal success at the initial - `DRAINING` CAS, never auto-reopens, and - proves that every acknowledged row is folded before export/cutover. Resume - revalidates the same binding, no-named-branch topology, and epoch authority - before advancing its epoch. Abort-drain can return `DRAINING -> OPEN` only + enrollment must carry an operation ID and expected revision; same-ID/same-digest + must return the complete bounded terminal receipt, same-ID/different-digest + must conflict, and a stale revision must never retarget. Quiesce must require a caller + drain ID, have a crash-safe cut, never record terminal success at the initial + `DRAINING` CAS, never auto-reopen, and + prove that every acknowledged row is folded before export/cutover. Resume + must revalidate the same binding, no-named-branch topology, and epoch authority + before advancing its epoch. Abort-drain may return `DRAINING -> OPEN` only when no guarded operation began, every owner settled, and no unmerged residue - or strict block remains. Correction clears a matching block but preserves - `DRAINING` and the original drain goal; it is never itself a reopen. Tests - delay a quiesce/resume retry across a later cycle and prove receipt/stale- - revision handling. They hold two streams strict-blocked simultaneously, then + or strict block remains. Correction must clear a matching block but preserve + `DRAINING` and the original drain goal; it must never itself be a reopen. Tests + must delay a quiesce/resume retry across a later cycle and prove receipt/stale- + revision handling. They must hold two streams strict-blocked simultaneously, then close one while preserving the other's independent block/correction/ - abort-drain/requiesce/`SEALED` authority. The WAL watermark is never reported + abort-drain/requiesce/`SEALED` authority. The WAL watermark must never be reported as a whole-graph history bound. - **Future bounded-root profile only:** a separately accepted `GraphHistoryBudget` amendment must force every manifest writer through one reserve-first gate, account for complete physical effects and recovery, and prove crash-safe settlement and independent stream closure reserves. None of those tests gates unbounded retain-all. -- Every public batch durably records its authenticated contributor before rows - can become unattributable, and the fold audit consumes that evidence. A - current `WITHDRAWN` token remains durable even when the graph row is absent; - neither attribution nor sequencing is inferred from MemTable positions, +- Every public batch must durably record its authenticated contributor before + rows can become unattributable, and the fold audit must consume that + evidence. A current `WITHDRAWN` token must remain durable even when the graph + row is absent; neither attribution nor sequencing may be inferred from MemTable positions, mutable WAL statistics, age, raw path shape, or benchmark maxima. - **B2b only:** the reviewed Lance inspect/plan/execute patch, durable attempt/receipt, post-success WAL fence check, bounded history checkpoint, and enforced retained-byte/object admission watermark bound growth. Exact inventory requires strong HEAD/GET/LIST visibility after PUT/DELETE plus exact multipart accounting/abort, or Lance-owned durable accounting. -- Admission cancellation and server shutdown cannot abandon an in-flight - durability waiter. A background-owned admission task reaches a durable - outcome or typed `AckUnknown`. Status/replay preserves possible residue but - never claims to reconcile that caller's attempt. -- The accepted retry contract prevents an ambiguous `X(id)`, newer durable +- Admission cancellation and server shutdown must not abandon an in-flight + durability waiter. A background-owned admission task must reach a durable + outcome or typed `AckUnknown`. Status/replay must preserve possible residue + but never claim to reconcile that caller's attempt. +- The public retry contract must prevent an ambiguous `X(id)`, newer durable `Y(id)`, retry `X(id)` sequence from silently restoring the stale `X` value. - Tests pin that exact interleaving, stream-incarnation reset, historic UUID + Tests must pin that exact interleaving, stream-incarnation reset, historic UUID reuse with a newer predecessor, actor change, same-generation chains across separate durable puts, the one-fresh-occurrence-per-key physical-run rule, reachable same-key `new/already/conflict`, and mixed-key `new/conflict/new` within one request, across separate requests, after restart, and through correction. One-row LWW cardinality - alone is not accepted as semantic idempotency. -- Correction tests cover `REPLACE` and `WITHDRAW` over an exact blocked cut, + alone must not be accepted as semantic idempotency. +- Correction tests must cover `REPLACE` and `WITHDRAW` over an exact blocked cut, lost fold response/restart followed by paginated block inspection, exact offending-key/token/action view digest, cursor/block movement, permission refusal, missing-cut corruption, multiple violations for one key across @@ -3357,34 +3448,34 @@ ordinary writers refuse but cannot be corrected, quiesced, or rebuilt. engine-derived canonical plan digest plus actor binding, same/different correction-ID replay, stale block token, unknown key, exact row/byte and response-page bounds, cold-replay reconstruction of terminal receipts, and - no-generation quiesce through `SEALED` rebuild. They also cover + no-generation quiesce through `SEALED` rebuild. They must also cover 8,192 distinct corrected current-token rows plus the separately bounded one- row receipt transaction, depth-8,192 replacement to exact maximum 8,193, concurrent fold/resume, and every sidecar/effect/publication crash boundary. No second generation, - skip-invalid path, or graph delete is admitted. -- Quiesce tests cover crashes before/after `OPEN -> DRAINING`, epoch claim, + skip-invalid path, or graph delete may be admitted. +- Quiesce tests must cover crashes before/after `OPEN -> DRAINING`, epoch claim, seal/drain, each fold, equal descriptor/top-level witness advancement, historical and selected-profile-authenticated current fence sentinels, strict-block summary, engine-minted failure-drain identity/actor before and after its conditional - CAS, empty proof, and `SEALED` publication. Status covers concurrent - put/flush/selected-profile retention exclusion. Resume and abort-drain cover + CAS, empty proof, and `SEALED` publication. Status tests must cover concurrent + put/flush/selected-profile retention exclusion. Resume and abort-drain tests must cover arm/claim/confirm/CAS/finalize, caller operation-ID/expected-revision retry, same-ID/different-intent, delayed retry after a later drain/resume cycle, named-branch refusal, strict- block refusal, reopen, and lost replies. Every selected-profile claim crashes - at its declared authority/effect boundaries and proves exact classification + at its declared authority/effect boundaries and prove exact classification before admitting a put. Under B2b, cold open, quiesce, resume/abort, and checkpoint additionally cover reclaim-successor sentinels and crash at patched claim-attempt/sentinel/manifest - boundaries and prove that their ordinary sentinel-first claims preserve the + boundaries and must prove that their ordinary sentinel-first claims preserve the prior replay cursor and classify its full tail; only the whole-cut reclaim - case may advance that cursor to its new sentinel. Quiesce settles and - publishes every retention sidecar before the final proof; B2b - reclaim/checkpoint refuse while `SEALED`, and resume consumes rather than - mutates the old proof. `SEALED` never auto-opens. -- **B2b only:** reclamation tests cover stock-cleanup non-ownership, the stale-writer deleted + case may advance that cursor to its new sentinel. Quiesce must settle and + publish every retention sidecar before the final proof; B2b + reclaim/checkpoint must refuse while `SEALED`, and resume must consume rather than + mutate the old proof. `SEALED` must never auto-open. +- **B2b only:** reclamation tests must cover stock-cleanup non-ownership, the stale-writer deleted successor-sentinel slot, exact local/RustFS inventory, HEAD/GET/LIST and multipart capability refusal or durable accounting, and explicit refusal of versioned/soft-delete/Object-Lock storage unless every retained version, @@ -3420,15 +3511,17 @@ ordinary writers refuse but cannot be corrected, quiesced, or rebuilt. Mutation/Load, SchemaApply, and maintenance remain refused for every enrolled lifecycle, including `SEALED`, until Phase D defines a token-aware direct- write/witness/rebind transition. -- Genuine cross-version tests prove old-binary/new-format and - new-binary/old-format refusal. Under B2b, they prove stock RC.1 rejects the +- **Implemented v8↔v9 gate:** CI builds the immutable final-v8 binary, proves + old-binary/new-format and new-binary/old-format refusal, and performs an + export/init/load rebuild that preserves logical rows, caller-supplied vectors, + and exact-`id` PK metadata while omitting hidden trusted stream metadata. + Future public rebuild tests must additionally prove acknowledged rows survive + only after quiesce/fold and fail closed when one remains unfolded. Under B2b, + compatibility tests must prove stock RC.1 rejects the patched new MemWAL details type URL/system-index kind rather than ignoring a field or falling back to a latest-version hint. A future bounded-root amendment must separately own `GraphHistoryBudget` bootstrap, mismatch, crash, and cap-too-small refusal tests; those are not v9/B2a activation gates. - Rebuild tests prove acknowledged rows survive - only after quiesce/fold, and fail closed when an acknowledged row remains - unfolded. ### 12.7 Later expansion gates @@ -3455,11 +3548,11 @@ ordinary writers refuse but cannot be corrected, quiesced, or rebuilt. |---|---|---| | E0 | production-neutral public-surface enrollment/witness classifier; no schema, API, sidecar, or format activation | **Passed 2026-07-18:** 14 substantive local cells, complete six-attempt zero-list 8/80 cost shape, Unix no-list/error tripwire, and one non-vacuous configured RustFS positive-plus-negative cell (§12.1) | | A | bounded main/unsharded/single-live-writer enrollment adapter, all-lifecycle effect exclusion with only the `SEALED` native-branch exception, lifecycle/admission lease, then graph-format capability/refusal and strict rebuild | **Implemented 2026-07-18 (§12.2):** internal schema v7, recovery-v10 enrollment, durable lifecycle CAS, process-local exclusion, crash/partial-format refusal, and genuine v6↔v7 strand evidence; no public enrollment or row path | -| B1 | **Implemented privately 2026-07-19; acknowledgement containment added 2026-07-20; widest-shape closure repaired 2026-07-21:** internal schema v8/config-v2, root-scoped one-generation admission worker, durability-watcher success followed by a same-writer post-durability epoch check, conservative active-state reopen/replay, the pinned RC.1 replay-watermark bridge, and one explicit strict RFC-022 fold; no production caller | The graph-level behavior/crash/race suite and genuine v7↔v8 refusal/rebuild remain green. Fold now charges logical slices and copies each scanner emission into dense owned arrays. The legal 8,192-row high-entropy near-cap generation folds and publishes exactly once without changing the 32-MiB admission cap; its isolated fold RSS delta is guarded by the 384-MiB remeasurement tripwire (§12.4) | +| B1 | **Implemented privately 2026-07-19; acknowledgement containment added 2026-07-20; widest-shape closure repaired 2026-07-21:** internal schema v8/config-v2, root-scoped one-generation admission worker, durability-watcher success followed by a same-writer post-durability epoch check, conservative active-state reopen/replay, the pinned RC.1 replay-watermark bridge, and one explicit strict RFC-022 fold; no production caller | The graph-level behavior/crash/race suite and genuine v7↔v8 refusal/rebuild remain green. Fold charges logical dense-slice Arrow bytes and copies each scanner emission into dense owned arrays. The legal 8,192-row high-entropy near-cap generation folds and publishes exactly once without changing the logical 32-MiB admission cap; physical RSS is guarded only by the 384-MiB remeasurement tripwire (§12.4). Recovery-v11 is historical under current v9 | | R0 | production-neutral retained-growth/source audit; current-object census; referenced-cut retry; legal high-entropy near-cap materialize/fold cell; no schema, public caller, or deletion | **Historical bounded-retention no-go 2026-07-20; disposition amended 2026-07-21 (§0.2/§12.4):** RC.1 still exposes neither a complete reserve-first physical envelope/receipt nor a durable cross-open randomized-attempt cap. Those facts prohibit a finite storage promise but do not block selected unbounded retain-all. The formerly red widest cell is now green locally and on the configured-RustFS CI path; current-object observations remain advisory retention evidence, not provider billing/accounting | -| B2a | selected unbounded retain-all/no-GC profile on stock Lance | **Private gate implemented 2026-07-21 (§12.5):** no OmniGraph byte/object/file/history quota; zero canonical `_mem_wal` deletion; complete/partial provider residue remains retained, unreferenced, and untouched below its root through retry/reopen; provider failures are loud; local/configured-RustFS history sweeps are advisory. No schema v9 or product surface is active | +| B2a | selected unbounded retain-all/no-GC profile on stock Lance | **Private gate implemented 2026-07-21 (§12.5):** no OmniGraph byte/object/file/history quota; zero canonical `_mem_wal` deletion; complete/partial provider residue remains retained, unreferenced, and untouched below its root through retry/reopen; provider failures are loud; local/configured-RustFS history sweeps are advisory. This gate itself activated no schema or product surface; the later private B2-common slice activates v9 | | B2b | candidate managed-reclamation retention profile | Inactive. Requires the Lance-owned durable inspect/plan/execute + receipt, post-success fencing, bounded checkpoint/inventory/accounting, local/RustFS enforced-bound validation, and the profile-specific crash matrix (§4.5.2/§12.6). Passing it alone activates no product surface | -| B2-common | explicit enrollment, compare-and-chain token/attribution, revision-fenced lifecycle/correction, schema v9/config-v3/recovery-v12, SDK, HTTP, CLI, Cedar, and OpenAPI | Inactive. Public activation requires the common crash/no-delete/provider-failure/cross-version, authorization, cancellation/shutdown, API-compatibility, and product-parity gates in §12.6. `GraphHistoryBudget` belongs only to a future bounded/managed profile, not selected retain-all | +| B2-common | schema v9/config-v3/state-v2, compare-and-chain token/attribution, graph-global token authority, recovery-v12 base+token fold; then explicit enrollment, revision-fenced lifecycle/correction/status, SDK, HTTP, CLI, Cedar, and OpenAPI | **Private row/fold subset implemented 2026-07-22 (§11/§12.6):** canonical digests, hidden attribution, stale-authority revalidation after shared admission, same-generation chains, exact two-participant recovery/publication, durable fold attribution, retain-all, and genuine v8↔v9 refusal/rebuild are green. Explicit production enrollment, lifecycle/correction/status, authorization, cancellation/shutdown, API compatibility, and product parity remain inactive. `GraphHistoryBudget` belongs only to a future bounded/managed profile | | C | restart-stable reject-row identity, atomic dead letter, richer status, and evidence-backed configurable bounds | reject crash matrix; reject-retention proof; backpressure and RSS/latency evidence | | D | automatic operation drain, schema/branch/upgrade integration, and rematerialization rebind | two-coordinator race, old/new physical-binding crash matrix, and format-transition suite | | E | fresh cuts and maintained-index reads; cross-process `Fresh` ships only if the substrate generation-retention guard exists (§9), otherwise same-process only | cut consistency; merged-generation exclusion | diff --git a/docs/rfcs/0028-stable-schema-identity.md b/docs/rfcs/0028-stable-schema-identity.md index b9979016..ab5ed7c2 100644 --- a/docs/rfcs/0028-stable-schema-identity.md +++ b/docs/rfcs/0028-stable-schema-identity.md @@ -466,10 +466,13 @@ are one internal storage-format capability. RFC-028 landed as internal manifest schema v5; that release kept `MIN_SUPPORTED == CURRENT == 5` and added the identity version to the release/stamp history. V6 preserved this contract and added RFC-023 key fencing; v7 preserved both and added RFC-026 identity-keyed -stream-lifecycle authority. The currently served v8 format preserves the -v5/v6/v7 contracts and adds RFC-026 stream-config v2 plus recovery-v11 for the -private B1 row/fold core. None of the later formats reinterprets or backfills -v5 in place. A v5 graph was never served with a +stream-lifecycle authority. Historical v8 preserved those contracts and added +RFC-026 stream-config v2 plus recovery-v11 for the private B1 row/fold core. +The currently served v9 format preserves those worker mechanics and activates +stream-config v3, lifecycle state v2, trusted hidden attribution, the +manifest-selected token participant, and recovery-v12's exact base-plus-token +fold. None of the later formats reinterprets or backfills v5 in place. A v5 +graph was never served with a partial combination such as identity-bearing IR over name-keyed manifest rows. There is no v1→v2 IR backfill and no new-binary in-place conversion of an old diff --git a/docs/user/concepts/storage.md b/docs/user/concepts/storage.md index 555496cc..ef2f045e 100644 --- a/docs/user/concepts/storage.md +++ b/docs/user/concepts/storage.md @@ -7,7 +7,8 @@ Every node type and every edge type is its own Lance dataset: - **Columnar Arrow storage**: each property is a column; nullable per Arrow schema. - **Fragments**: data is partitioned into fragments; new writes create new fragments. - **Manifest versioning**: every commit produces a new dataset version; old versions remain readable. -- **Stable row IDs**: stable row IDs are enabled on every Lance dataset OmniGraph creates — node and edge data tables, `__manifest`, `_graph_commit_recoveries.lance`, and any future system tables. This is an architectural invariant: the flag is one-way at dataset create, so a future change that introduces a Lance dataset must preserve it. Consequences: `_row_created_at_version` and `_row_last_updated_at_version` are available on every dataset (load-bearing for change-feed validators); indices survive `omnigraph optimize`. Pre-0.4.x graphs created before this code path settled may have datasets without the flag and cannot be retrofitted in place — the supported path is dump-and-reload. The rewrite path used by `schema_apply` preserves the flag. +- **Stable row IDs**: stable row IDs are enabled on every Lance dataset OmniGraph creates — node and edge data tables, `__manifest`, `_stream_tokens.lance`, `_graph_commit_recoveries.lance`, and any future system tables. This is an architectural invariant: the flag is one-way at dataset create, so a future change that introduces a Lance dataset must preserve it. Consequences: `_row_created_at_version` and `_row_last_updated_at_version` are available on every dataset (load-bearing for change-feed validators); indices survive `omnigraph optimize`. Pre-0.4.x graphs created before this code path settled may have datasets without the flag and cannot be retrofitted in place — the supported path is dump-and-reload. The rewrite path used by `schema_apply` preserves the flag. +- **Private v9 row attribution**: every node/edge table physically carries one nullable `__omnigraph_stream_v1$` struct. The trailing `$` is outside the `.pg` identifier grammar. SDK catalog/schema reflection, scans, filters, index reflection, and export omit or reject this storage-protocol field; it is not a user property. - **Append / delete / `merge_insert`**: native Lance write modes. - **Per-dataset branches** (Lance native): copy-on-write at the dataset level. - **Object-store agnostic**: file://, s3://, gs://, az://, http (read-only via Lance) — OmniGraph wires file:// and s3://. @@ -21,17 +22,18 @@ OmniGraph is **not** a single Lance dataset; it is a *graph* of datasets coordin - `nodes/{stable_table_id:016x}-{table_incarnation_id:016x}` — one Lance dataset per node-table lifetime - `edges/{stable_table_id:016x}-{table_incarnation_id:016x}` — one Lance dataset per edge-table lifetime - `__manifest/` — the catalog of all sub-tables and their published versions, **and** the graph commit lineage (RFC-013 Phase 7: `graph_commit` / `graph_head` rows). Graph-level branches are Lance branches on these datasets. + - `_stream_tokens.lance` — graph-global RFC-026 current-token state. Its raw Lance HEAD is not authority; `__manifest` selects one exact witness. - `_graph_commit_recoveries.lance` — the crash-recovery audit log (one row per recovery action; see below). The former `_graph_commits.lance` / `_graph_commit_actors.lance` lineage tables are **retired**: lineage lives in `__manifest`, so a graph this binary creates has neither. - **Manifest row schema** (`object_id, object_type, location, metadata, base_objects, table_key, stable_table_id, table_incarnation_id, table_version, table_branch, row_count`): - - `object_type` ∈ `table | table_version | table_tombstone | graph_commit | graph_head | stream_state` - - `table_key` ∈ `node: | edge:` (empty for `graph_commit` / `graph_head` lineage rows). On `stream_state` it is diagnostic only; identity and binding come from the stable pair and metadata payload. - - `(stable_table_id, table_incarnation_id)` is the immutable table coordinate; both fields are nonzero on table and `stream_state` rows and null on lineage rows. `table_key` is the current human-readable alias and may change on rename. + - `object_type` ∈ `table | table_version | table_tombstone | graph_commit | graph_head | stream_state | stream_token_authority` + - `table_key` ∈ `node: | edge:` (empty for lineage and `stream_token_authority` rows). On `stream_state` it is diagnostic only; identity and binding come from the stable pair and metadata payload. + - `(stable_table_id, table_incarnation_id)` is the immutable table coordinate; both fields are nonzero on table and `stream_state` rows and null on lineage and graph-global `stream_token_authority` rows. `table_key` is the current human-readable alias and may change on rename. - `table_branch` is `null` for the main lineage and the branch name otherwise - **Graph lineage rows** (RFC-013 Phase 7): one immutable `graph_commit` row per commit (`object_id` = the commit ULID; `metadata` JSON carries parent / merged-parent / actor / timestamp) plus one mutable `graph_head:` pointer per branch (`graph_head:main` for main). The in-memory commit DAG is a projection of these rows. - **Snapshot reconstruction**: latest visible `table_version` per stable table ID + incarnation minus tombstones scoped to that same pair, joined to the pair's current registration for its alias and path. Two live pairs cannot expose the same alias. A drop/re-add therefore starts an independent version sequence and cannot be hidden by the old lifetime's tombstone. -- **Atomic publish**: multi-dataset commits publish so that a single write to `__manifest` flips all the new sub-table versions visible at once. +- **Atomic publish**: multi-dataset commits publish so that a single write to `__manifest` flips all the new sub-table versions visible at once. A private stream fold includes both its base-table pointer and the exact `_stream_tokens.lance` witness in that same visibility decision. - **Row-level CAS on the merge-insert join key**: `object_id` carries an unenforced-primary-key annotation so Lance's bloom-filter conflict resolver rejects two concurrent commits that land the same `object_id` row. Without this annotation, Lance's transparent rebase would admit silent duplicates from racing publishers. -- **Optimistic concurrency control on publish**: legacy writers assert the manifest's current latest non-tombstoned version for each touched table; a mismatch surfaces as `ExpectedVersionMismatch`. RFC-022-enrolled mutation/load attempts use a stronger, branch-wide contract: preparation captures the Lance-native branch identity, the exact `graph_head` (including absence), the accepted schema identity/catalog, and one base table snapshot. Under root-shared schema → branch → sorted-table gates, the engine revalidates that complete authority before any physical effect, then the publisher rechecks the exact native branch identity/head plus the touched-table versions. An insert-only mutation or Append/Merge load whose authority changed before effects discards and fully reprepares the bounded attempt; Update/Delete/Overwrite returns `ReadSetChanged`. Once any Lance effect is durable, any later failure leaves the recovery sidecar authoritative and returns `RecoveryRequired` instead of silently rebasing or replaying the prepared plan. +- **Optimistic concurrency control on publish**: legacy writers assert the manifest's current latest non-tombstoned version for each touched table; a mismatch surfaces as `ExpectedVersionMismatch`. RFC-022-enrolled mutation/load attempts use a stronger, branch-wide contract: preparation captures the Lance-native branch identity, the exact `graph_head` (including absence), the accepted schema identity/catalog, and one base table snapshot. Under root-shared schema → branch → stream-token → sorted-table gates, the engine revalidates that complete authority before any physical effect, then the publisher rechecks the exact native branch identity/head plus the touched-table versions. An insert-only mutation or Append/Merge load whose authority changed before effects discards and fully reprepares the bounded attempt; Update/Delete/Overwrite returns `ReadSetChanged`. Once any Lance effect is durable, any later failure leaves the recovery sidecar authoritative and returns `RecoveryRequired` instead of silently rebasing or replaying the prepared plan. ### Internal schema versioning @@ -44,7 +46,7 @@ The on-disk shape of `__manifest` is reconciled with the binary via a single ver - The stamp is read with no object-store writes, so the check is safe under a read-only open. Operators can see a graph's stamp with `omnigraph snapshot` and the binary's served version with `omnigraph version` (the `internal-schema` line). The stamp values below are historical; this binary serves only the current one -(`v8`). An earlier-stamped graph is rebuilt via export/import, not migrated in +(`v9`). An earlier-stamped graph is rebuilt via export/import, not migrated in place. | Stamp | Shape | @@ -56,7 +58,8 @@ place. | v5 | RFC-028 SchemaIR v2 plus graph-domain stable schema IDs; manifest table rows, OCC, recovery ownership, and physical paths keyed by stable table ID + incarnation. | | v6 | Preserves v5 identity and activates RFC-023: every graph node/edge dataset has exact non-null physical `id` as Lance's unenforced PK, and every production strict insert/upsert uses the exact-`id` filter-bearing adapter. | | v7 | Preserves v5/v6 identity and keyed-write contracts, and adds RFC-026 identity-keyed `stream_state` authority: physical enrollment binding, mutable current-HEAD witness, lifecycle, and per-shard epoch floor. Phase A can recover one exact empty private enrollment; no production row-streaming API is active. | -| v8 | Preserves the v5/v6/v7 contracts and activates RFC-026 stream-config v2 plus recovery-v11 for the private B1 row/fold core: one no-roll generation, watcher success plus a same-writer post-durability epoch check before clean acknowledgement, and one exact fold whose manifest publish advances the table pointer, lifecycle witness, and graph lineage together. The legal near-cap closure cell is green after logical-slice accounting and dense scanner-batch copies. Unbounded retain-all/no-GC is selected for the first profile, but schema-v9 and every public row-streaming surface remain inactive. **The only version this binary serves.** | +| v8 | Preserves the v5/v6/v7 contracts and activates RFC-026 stream-config v2 plus recovery-v11 for the private B1 row/fold core: one no-roll generation, watcher success plus a same-writer post-durability epoch check before clean acknowledgement, and one exact fold whose manifest publish advances the table pointer, lifecycle witness, and graph lineage together. The legal near-cap closure cell is green after logical-slice accounting and dense scanner-batch copies. | +| v9 | Preserves the bounded B1 worker mechanics and activates RFC-026 stream-config v3, lifecycle state v2, the grammar-impossible trusted-attribution field, manifest-selected `_stream_tokens.lance` authority, compare-and-chain tokens, and recovery-v12's exact base-plus-token fold. The selected B2a profile retains all canonical MemWAL objects and imposes no retained-byte/object/file/history quota. Public enrollment, lifecycle management, correction/status, SDK/HTTP/CLI/OpenAPI, generation GC, and fresh reads remain inactive. **The only version this binary serves.** | ## On-disk layout @@ -72,6 +75,7 @@ flowchart TB manifest["__manifest/
L2 catalog of sub-tables"]:::l2 nodes["nodes/{stable-id}-{incarnation}/
one dataset per node-table lifetime"]:::l2 edges["edges/{stable-id}-{incarnation}/
one dataset per edge-table lifetime"]:::l2 + tokens["_stream_tokens.lance/
manifest-selected current-token authority"]:::l2 cgraph["_graph_commit_recoveries.lance/
crash-recovery audit log"]:::l2 recovery["__recovery/{ulid}.json
recovery sidecars (transient)"]:::l2 refs["_refs/branches/{name}.json
graph-level branches"]:::l2 @@ -79,6 +83,7 @@ flowchart TB graph --> manifest graph --> nodes graph --> edges + graph --> tokens graph --> cgraph graph --> recovery graph --> refs @@ -93,6 +98,7 @@ flowchart TB nodes -.-> dataset edges -.-> dataset + tokens -.-> dataset manifest -.-> dataset ``` @@ -101,13 +107,14 @@ flowchart TB - **Graph root** is one directory (or S3 prefix). Everything below is part of one OmniGraph graph. - **`__manifest/`** is a Lance dataset whose rows describe which sub-table version is published at which graph-branch. Reading a snapshot starts here. - **`nodes/`** and **`edges/`** are sibling directories holding one Lance dataset per live table lifetime. Names encode the stable table ID and incarnation, so a public type rename keeps its path while a drop/re-add receives a fresh one. +- **`_stream_tokens.lance`** stores at most one current RFC-026 authority row per logical graph key. Only the exact version selected by the matching `stream_token_authority` manifest row is authoritative; operators must not advance, rewrite, clean, or delete it directly. - The graph commit DAG lives in **`__manifest`** as `graph_commit` / `graph_head` rows written in the publish CAS (RFC-013 Phase 7). The former `_graph_commits.lance` / `_graph_commit_actors.lance` lineage tables are retired — a graph this binary creates has neither. - **`_graph_commit_recoveries.lance`** — one internal row per completed crash-recovery action, including its exact per-table outcomes and the original actor. It joins by `graph_commit_id` to the graph commit lineage in `__manifest`. An exact v9 writer roll-forward keeps the interrupted writer's original actor; rollback and legacy recovery commits use `omnigraph:recovery`. The CLI does not currently expose this internal table. - **`__recovery/{ulid}.json`** — transient sidecar files written by a writer before it advances the underlying dataset, deleted once the matching manifest publish succeeds. A sidecar persisting after process exit means the writer crashed mid-commit; the next read-write open processes it. Steady-state directory is empty. - **`_refs/branches/{name}.json`** is graph-level branch metadata — pointers from a branch name to the manifest version it heads. - **Inside each Lance dataset** (orange): the standard Lance directory layout. `_versions/{n}.manifest` records every commit; `data/` holds the actual Arrow fragments; `_indices/{uuid}/` holds index segments with their own `fragment_bitmap` for partial coverage; `_refs/` holds Lance-native per-dataset branches and tags. -The split — L2 owns the cross-dataset catalog; L1 owns the per-dataset internals — means that schema work (which adds or removes datasets) updates `__manifest`, while data work (which adds fragments) updates `_versions/` inside the affected dataset and then bumps `__manifest`. +The split — L2 owns cross-dataset authority; L1 owns each dataset's internals — means that schema work (which adds or removes datasets) updates `__manifest`, while data work (which adds fragments) updates `_versions/` inside the affected dataset and then bumps `__manifest`. A private stream fold stages both base and token effects first, then publishes their exact witnesses together. ## URI scheme support diff --git a/docs/user/operations/upgrade.md b/docs/user/operations/upgrade.md index 9e26f7c7..a4020856 100644 --- a/docs/user/operations/upgrade.md +++ b/docs/user/operations/upgrade.md @@ -17,9 +17,9 @@ message that **names the release line that wrote it** and the exact commands — so you can fetch the right old binary without guessing: ``` -__manifest is stamped at internal schema v7, but this omnigraph reads only v8. -This graph was created by omnigraph 0.11.x. Rebuild it: with an omnigraph -0.11.x binary run `omnigraph export > graph.jsonl`, then with this +__manifest is stamped at internal schema v8, but this omnigraph reads only v9. +This graph was created by omnigraph 0.12.x. Rebuild it: with an omnigraph +0.12.x binary run `omnigraph export > graph.jsonl`, then with this binary run `omnigraph init --schema ` and `omnigraph load --mode overwrite --data graph.jsonl `. (Data, vectors, and blobs are preserved; commit history and branches are not.) See docs/user/operations/upgrade.md. @@ -39,7 +39,8 @@ from that line (the latest is safest): | internal schema v5 | omnigraph 0.9.x | the latest 0.9.x | | internal schema v6 | omnigraph 0.10.x | the latest 0.10.x | | internal schema v7 | omnigraph 0.11.x | the latest 0.11.x | -| internal schema v8 | omnigraph 0.12.x | — current format; no rebuild needed | +| internal schema v8 | omnigraph 0.12.x | the latest 0.12.x | +| internal schema v9 | omnigraph 0.13.x | — current format; no rebuild needed | You can also check versions before you hit a refusal: @@ -110,6 +111,28 @@ complete. Do not use force-init to turn the old root into the new format. - **Server deployments**: take the graph out of the serving set, rebuild it offline with the CLI, then point the cluster at the rebuilt graph (`cluster apply`). +## Migrating from internal schema v8 to v9 + +Internal schema v9 activates RFC-026's private common-B2 storage contract: +stream-config v3, lifecycle state v2, a manifest-selected +`_stream_tokens.lance` authority, trusted per-row attribution, and +recovery-v12's atomic base-plus-token publication. These are physical +correctness foundations; v9 does not by itself expose a public streaming API. + +A v8 graph must use the standard rebuild recipe above. Quiesce every v8 writer, +export with the latest 0.12.x binary, initialize a **different** root with the +0.13.x binary, load the export, and verify the v9 stamp plus row/vector/blob +fidelity before fleet cutover. Keep the v8 root unchanged through the rollback +window. A v9 binary refuses v8, and a v8 binary refuses v9. + +V9's physical attribution field is named `__omnigraph_stream_v1$`. The trailing +`$` is deliberately outside the `.pg` property-name grammar, so it cannot +collide with a user property. In particular, v8 legitimately allowed a user +property named `__omnigraph_stream_v1` (without `$`); export/init/load preserves +that property and its values unchanged. Do not rename or delete it as protocol +metadata. Conversely, v9 export omits only the exact physical `$`-suffixed +field and never transfers token authority into the rebuilt logical snapshot. + ## Migrating to v0.8.0 v0.8.0 is the first release with a storage-format change since v0.4.0. Any graph diff --git a/docs/user/reference/constants.md b/docs/user/reference/constants.md index ed2c1e44..350a5821 100644 --- a/docs/user/reference/constants.md +++ b/docs/user/reference/constants.md @@ -3,6 +3,8 @@ | Name | Value | Area | |---|---|---| | `MANIFEST_DIR` | `__manifest` | manifest layout | +| Stream-token authority dataset | `_stream_tokens.lance` | RFC-026 graph-global current-token participant; only the exact witness selected by `__manifest` is authoritative | +| Trusted stream metadata column | `__omnigraph_stream_v1$` | private nullable v9 base-row attribution; trailing `$` is outside the `.pg` identifier grammar and public reflection/export omit it | | Commit graph dirs (retired) | `_graph_commits.lance` / `_graph_commit_actors.lance` | retired in Phase B; lineage lives in `__manifest` (`graph_commit` / `graph_head` rows) since RFC-013 Phase 7. A graph this binary creates has neither. | | Recovery audit dir | `_graph_commit_recoveries.lance` | internal exact record of completed crash-recovery actions; no public CLI query yet | | BranchMerge logical data-transaction ceiling | `MAX_BRANCH_MERGE_DATA_TRANSACTIONS = 1024` | maximum strict-insert/upsert/delete transactions one table may arm in a `protocol_v4` chain; a larger plan fails before sidecar arm | @@ -10,12 +12,15 @@ | Run branch prefix (legacy, removed) | `__run__` | pre-v0.4.0 Run state machine; no longer a reserved name. A graph still carrying `__run__*` branches is sub-v4 and refused on open (rebuild via export/import). | | Schema apply lock | `__schema_apply_lock__` | schema apply | | Manifest publisher retry budget | `PUBLISHER_RETRY_BUDGET = 5` | manifest publish | -| Internal manifest schema version | `INTERNAL_MANIFEST_SCHEMA_VERSION = 8` | strict RFC-026 B1 strand; preserves v5 identity, v6 keyed fencing, and v7 stream-lifecycle authority while activating stream-config v2 plus recovery-v11 for the private data-bearing core. V7 and older graphs require export/init/load rebuild | +| Internal manifest schema version | `INTERNAL_MANIFEST_SCHEMA_VERSION = 9` | strict RFC-026 common-B2 strand; preserves v5 identity, v6 keyed fencing, v7 stream-lifecycle authority, and v8's bounded B1 worker while activating stream-config v3, lifecycle state v2, manifest-selected token authority, trusted attribution, and recovery-v12. V8 and older graphs require export/init/load rebuild | | Keyed-write row ceiling | `KEYED_WRITE_MAX_ROWS = 8192` | one Mutation/Load keyed table or one BranchMerge chunk; inclusive | | Keyed-write Arrow-memory ceiling | `KEYED_WRITE_MAX_BYTES = 33,554,432` (32 MiB) | accumulated Mutation/Load keyed input (including pending state plus a streamed mutation-update match set) or one BranchMerge row/upsert/delete-filter chunk; a single larger row is refused before sidecar arm. Stored update Blobs and keyed external-URI ranges/object sizes are charged before payload reads. The complete retained BranchMerge delete plan and the operation-wide projected scalar validation delta are separately capped at the same value; ordered merge and validation scans explicitly apply it as Lance's per-batch decoded-byte ceiling. Overwrite retains external-reference semantics | | Private RFC-026 B1 generation ceiling | `8,192 rows`, `8,192 batches`, `33,554,432` Arrow bytes | one no-roll MemWAL generation; the private seam refuses the next put before exceeding any bound | | Private RFC-026 B1 resident-writer ceiling | `1` per graph root and table | evidence-qualified process-local worker admission; not a public throughput contract. B2 must requalify any higher multi-resident limit with an RSS cell | | Private RFC-026 B1 aggregate Arrow reservation | `33,554,432` bytes (32 MiB) per graph root | cheap raw caller row/byte bounds reject obviously over-cap input before recovery I/O; raw-fit input then receives exact post-tombstone validation at that same pre-recovery boundary. After any recovery/authority prelude, the exact charge is recomputed and reserved against the aggregate before any same-key queue wait, shared admission, detached ownership, or cold claim; the permit transfers into the resident generation and remains charged through fold publication. Distinct from whole-process RSS | +| Private RFC-026 B2 canonical-payload ceiling | `B2_MAX_CANONICAL_PAYLOAD_BYTES = 67,108,864` (64 MiB) per normalized row | bounded deterministic token-digest input; not a public request-body limit | +| Private RFC-026 B2 token-projection ceiling | `B2_MAX_TOKEN_PROJECTION_ARROW_BYTES = 33,554,432` (32 MiB) | exact winning token projection and cumulative exact-authority lookup retention | +| Private RFC-026 B2 recovery-JSON ceiling | `B2_MAX_TOKEN_RECOVERY_JSON_BYTES = 33,554,432` (32 MiB) | bounded token rows embedded in one recovery-v12 sidecar | | Maintenance concurrency | `OMNIGRAPH_MAINTENANCE_CONCURRENCY=8` | optimize/cleanup | | Graph index cache size | `8` (LRU) | runtime cache | | Expand indexed-path frontier ceiling | `OMNIGRAPH_EXPAND_INDEXED_MAX_FRONTIER=1024` | traversal |