RFC-026: add private B2 token authority and exact fold recovery

[aaltshuler]

Summary

• advances the private RFC-026 implementation to internal schema v9, stream config v3, and lifecycle state v2
• adds canonical compare-and-chain payload/token digests, grammar-impossible trusted row metadata, same-generation overlays, and a manifest-selected graph-global _stream_tokens.lance authority
• folds base rows and token winners through exact pre-minted transactions owned by recovery-v12, with one __manifest CAS publishing both pointers, lifecycle state, lineage, and durable attribution
• recaptures lifecycle/binding/HEAD/token authority after shared admission and same-key queue ownership, and bounds B2 preprocessing to two 128 MiB envelopes
• adds the genuine v8↔v9 refusal/export-init-load rebuild gate while preserving logical rows, vectors, exact-id PK metadata, and ordinary __omnigraph_stream_v1 user data
• keeps the implementation crate-private and feature-gated: no production enrollment, lifecycle management, SDK, CLI, HTTP, Cedar, or OpenAPI surface is activated; B2a remains unbounded retain-all with no MemWAL GC
• realigns the existing branch-delete cost baseline with its current operation-local capture shape

Why

Private B1 proved durable admission and bounded folding, but it did not yet provide durable per-key sequencing, idempotent retry classification, trusted contributor attribution, or an exact two-participant fold outcome. This slice closes that correctness gap without exposing a product API or inventing an OmniGraph WAL/transaction manager beside Lance.

Compatibility and risk

This is an intentional internal storage-format boundary: v8 graphs are refused by v9 and must cross through export/init/load. Recovery-v11 remains historical and is refused under v9 because it lacks the token participant and complete lifecycle-v2 authority. The topology remains main-only, unsharded, and one live writer process; fresh reads and managed generation reclamation remain inactive.

Validation

• cargo test --workspace --locked
• cargo test --locked -p omnigraph-engine --features failpoints --test failpoints — 141 passed, 1 ignored
• cargo test --locked -p omnigraph-engine --features failpoints --test memwal_stream — 34 passed
• cargo test -p omnigraph-server --features aws --locked
• scripts/check-agents-md.sh
• git diff --check

Greptile Summary

Advances the private RFC-026 stream implementation to schema v9 and recovery-v12.

• Adds graph-global token authority, canonical token/payload digests, trusted physical row metadata, and same-generation compare-and-chain overlays.
• Adds exact two-participant base/token fold recovery and atomic manifest publication of pointers, lifecycle, lineage, and attribution.
• Introduces the v8-to-v9 export/rebuild compatibility fence and expands lifecycle, failpoint, and cost coverage.
• Updates internal schema validation, public projections, export handling, CI, and architecture documentation.

Confidence Score: 3/5

This PR should not merge until repeated same-key writes within one generation can fold to their final chained winner.

Same-generation admission accepts and durably acknowledges a successor for an already-written key, but the fold subsequently rejects the resulting duplicate logical ID, leaving the acknowledged generation unable to publish.

crates/omnigraph/src/db/omnigraph/stream_ingest.rs and crates/omnigraph/src/table_store/mem_wal/worker.rs

Important Files Changed

Filename	Overview
crates/omnigraph/src/db/omnigraph/stream_ingest.rs	Adds B2 admission and two-participant folding, but fold attribution rejects the repeated same-key rows that same-generation chaining admits.
crates/omnigraph/src/table_store/mem_wal/worker.rs	Adds preprocessing admission and confirmed token overlays used to sequence same-generation writes.
crates/omnigraph/src/db/manifest/recovery.rs	Adds recovery-v12 exact base/token effect classification, missing-token completion, and atomic roll-forward.
crates/omnigraph/src/db/manifest/token_store.rs	Implements the manifest-selected token authority dataset, bounded exact lookup, and staged token upserts.
crates/omnigraph/src/db/manifest/stream_token.rs	Defines canonical compare-and-chain tokens, trusted row metadata, authority rows, and validation.
crates/omnigraph/src/db/manifest.rs	Adds protocol-column filtering and public snapshot projection boundaries.
crates/omnigraph-compiler/src/schema/mod.rs	Reserves the grammar-impossible trusted stream metadata column from user schemas.
crates/omnigraph-cli/tests/crossversion_upgrade.rs	Extends genuine cross-version coverage through the v8-to-v9 refusal and export/rebuild boundary.
.github/workflows/ci.yml	Builds the final v8 CLI in CI so cross-version tests exercise a genuine predecessor binary.

Sequence Diagram

sequenceDiagram
  participant A as Put A
  participant O as Same-generation overlay
  participant B as Put B
  participant W as MemWAL generation
  participant F as Fold
  A->>W: Append key K, token A
  W-->>A: Durable acknowledgement
  A->>O: Record K → token A
  B->>O: Validate predecessor token A
  O-->>B: Admit successor token B
  B->>W: Append key K, token B
  W-->>B: Durable acknowledgement
  F->>W: Scan fresh generation
  W-->>F: K/token A, K/token B
  F-->>F: Reject duplicate logical ID K

Loading

_{Reviews (1): Last reviewed commit: "feat: add private RFC-026 B2 token autho..." | Re-trigger Greptile}

Greptile also left 1 inline comment on this PR.

Context used (5)

• Context used - AGENTS.md (source)
• Knowledge Base — OmniGraph core storage: manifest, table store, and Lance access
• Knowledge Base — OmniGraph write path
• Knowledge Base — Compiler Schema and Catalog
• Knowledge Base — Dev conventions and invariants

[greptile-apps]

Same-key chains cannot fold

When two sequential puts target the same logical ID in one MemWAL generation, admission accepts and durably acknowledges the second put against the first put's confirmed overlay, but the fold rejects the resulting repeated logical ID instead of selecting the final chained winner. Every retry then fails with stream fold scanner returned duplicate winner id, preventing the acknowledged generation from publishing and blocking subsequent stream progress.

Context Used: AGENTS.md (source)

Knowledge Base Used:

• OmniGraph write path
• OmniGraph core storage: manifest, table store, and Lance access