Select unbounded MemWAL retention and close fold boundary

[aaltshuler]

Summary

• Select RFC-026 B2a as an unbounded retain-all profile on stock Lance: OmniGraph performs no raw _mem_wal deletion and promises no retained byte, object, file, or history quota.
• Repair the private B1 near-cap fold boundary by charging logical Arrow slices and copying each scanner emission into dense owned arrays before retention.
• Turn the 8,192-row high-entropy case into a positive closure regression: acknowledgement stays graph-invisible, then one fold advances the table and __manifest exactly once.
• Add revision-pinned Gate R0 inventory, retained-growth, retry, local/RustFS closure, and fold-RSS evidence, with CI polarity and non-vacuity checks.
• Align RFC-026, the architecture/canon/testing docs, the WAL decision record, and the deferred Lance managed-reclamation proposal.

Why

Gate R0 exposed two separate facts. First, a legal near-32-MiB generation could be acknowledged and materialized but then fail folding because sparse LsmScanner slices retained much larger backing buffers. Second, stock Lance RC.1 cannot prove a finite lifetime or complete provider-billed growth envelope for every materialization attempt.

The first issue is a correctness/closure bug and is fixed here. The second prevents a bounded-storage promise, but it does not block a profile that explicitly retains everything. This lets OmniGraph move forward without waiting for upstream reclamation APIs or introducing an unsafe raw-path collector.

Impact

The row and logical-memory limits remain 8,192 rows / 32 MiB, with one resident writer and one exclusive fold. Provider exhaustion is loud and may stop progress; it never permits false acknowledgement, silent loss, or partial graph publication.

This does not activate public streaming. Explicit enrollment, compare-and-chain tokens, trusted attribution, revisioned lifecycle receipts, correction, authorization, SDK/CLI/HTTP/OpenAPI parity, and their evidence remain future work. Managed reclamation and any GraphHistoryBudget belong to an optional later bounded profile.

Validation

• cargo test --workspace --locked
• cargo test --locked -p omnigraph-engine --features failpoints --test memwal_stream — 25 passed
• cargo test --locked -p omnigraph-engine --features failpoints --test memwal_stream_cost — 19 passed, 1 subprocess helper ignored
• exact high-entropy closure and fold-RSS regression cells
• git diff --check
• targeted rustfmt --check for the modified Rust implementation and test files

Greptile Summary

This PR selects an unbounded retain-all MemWAL profile and closes the near-cap fold boundary. The main changes are:

• Logical Arrow slice accounting and dense owned fold batches.
• Positive local and RustFS closure coverage for the widest legal generation.
• Retention, retry, inventory, and fold-memory evidence in CI.
• Updated RFC, architecture, storage, testing, and operational documentation.

Confidence Score: 5/5

The updated code looks safe to merge.

• The latest diagnostic fix preserves local Gate R0 test output when a command fails.
• No distinct blocking issue remains within the follow-up review scope.

Important Files Changed

Filename	Overview
crates/omnigraph/src/db/omnigraph/stream_ingest.rs	Charges logical scanner slices and retains dense owned arrays so legal near-cap generations can fold.
crates/omnigraph/tests/memwal_stream_cost.rs	Adds closure, retention-growth, retry, inventory, and fold-memory coverage.
.github/workflows/ci.yml	Adds local and RustFS Gate R0 execution, evidence, and non-vacuity checks.

_{Reviews (4): Last reviewed commit: "Preserve Gate R0 failure diagnostics" | Re-trigger Greptile}

Context used:

• Context used - AGENTS.md (source)
• Context used - CLAUDE.md (source)