Exp 160: incremental view maintenance — the v2 stream engine

[danReynolds]

This is an architectural change, not a tuning experiment. It introduces a second execution strategy for reactive streams — materialized results maintained incrementally from row deltas — with the existing re-query path retained as the universal, fail-closed fallback. It is the largest change to the reactive engine since its creation, and it carries the experiment-protocol evidence trail (exp 160) on top of that.

The architecture

Today's engine answers every invalidation by re-executing the query (hash suppression saves decode/emission, never execution). This PR adds the machinery for the engine to instead understand queries and maintain their results:

preupdate hook ──► bounded row deltas ──► writer reply ──► stream engine
                                                              │
                                       classifier (at registration, fail-closed)
                                          ├─ full maintenance   → patch cached rows, emit
                                          ├─ windowed (LIMIT K) → top-K cache, patch/evict
                                          ├─ skip-only          → prove misses, else re-query
                                          ├─ aggregates         → maintain exact COUNT/SUM/MIN/MAX/AVG
                                          └─ everything else    → today's re-query path, unchanged

New durable infrastructure (each usable beyond this PR)

1. Row-delta channel (native/resqlite.c, lib/src/row_deltas.dart) — the preupdate hook captures per-row old/new values (bounded: 256 rows / 32 cols / 256 KB per cycle; poisoned by savepoint rollback or overflow), ships them in writer replies, decoded lazily. This is a CDC-grade primitive: diff-emitting streams, sync engines, and undo journals can all consume it later without touching native code again.
2. Query classifier (lib/src/stream_ivm.dart) — a strict registration-time grammar (bare columns, aggregates with aliases, AND-ed comparisons, composite ORDER BY with pk tiebreak, LIMIT) producing sealed admission shapes. Fail-closed by construction: anything unparsed or unprovable costs a re-query, never correctness.
3. Maintained state machines — full / windowed / skip-only / aggregate states with clone-on-write patching, complete-set window tracking, and exact integer aggregate maintenance.
4. A new ordering primitive: writer-ordered state builds. All IVM baselines (caches, aggregate snapshots) are built through reads on the writer port, whose FIFO totally orders a snapshot against the write replies that carry deltas — reader results and writer replies have no cross-port happens-before (a defect class this PR discovered, fixed, and recorded as a JOURNAL lesson). Paired invariant: maintained state survives only an unbroken chain of processed cycles; any bypassed delta-bearing cycle drops it for an exact rebuild.
5. Randomized equivalence harness (test/stream_ivm_equivalence_test.dart) — seeded write storms (rowid changes, NULLs, savepoint rollbacks, overflow batches) across every admission mode, every emission checked against a fresh select. It caught two ~1-in-5 ordering races deterministic replays never fired; it is the permanent safety net for all future IVM work.

Architectural properties

• Zero public API change — db.stream() is untouched; admission is an internal execution strategy.
• Fail-closed at every layer — capture overflow, classifier miss, unprovable cell, cache inconsistency, schema drift all degrade to today's path. Zero bails observed across all measured workloads.
• One-line revertible — remove the maintain check in the dirty loop and the system is byte-for-byte today's behavior.
• Emission semantics sharpen: admitted streams deliver each write's patch instead of coalescing behind re-query latency (the semantics of an infinitely fast re-query). The exp 045 microtask-batch pattern is the pre-identified mitigation if a workload prefers coalescing.

Evidence (full record: experiments/160-stream-delta-ivm.md)

evidence	result
Tracelite primary gate	many-streams −18.2%, formal improved verdict — the first cleared gate in this direction's history; reproduced order-flipped (+19.5% main-slower)
App-shaped admission audit	7/9 stream shapes admitted (tier 1 alone: 3/8); 7,643 invalidation decisions resolved with zero reader re-queries, 0 bails; burst wall 132.9 ms vs main ≈200 ms while delivering per-write patches
Exp-147 audit	overlap −19% with 500 vs 27 emissions delivered; keyed-PK −14% (matched pair)
Honest costs	high-cardinality density +3.5–4.8% (both collection orders): O(admitted-streams) main-isolate predicate eval replaces reader-distributed suppression — v3 fix specified (per-table equality-predicate index, O(1) dispatch); keyed-PK neutral for the full stack
Found & fixed en route	the diagnostics×reader NOMUTEX race (landed separately as #156) and the two cross-port ordering defects above

What this is not (yet) — the mapped ladder

Tier 4 one-hop joins, the equality-predicate dispatch index, rebuild-storm coalescing, and diff-carrying emissions are specified in signals.json with evidence trails, deliberately not built here.

Test plan

• 315/315 tests incl. 38 classifier/state-machine units and the equivalence harness; 20/20 clean harness loops post-fix
• dart analyze lib test benchmark clean; finalizer green
• Tracelite gate, two passes (standard + order-flipped), both committed in the writeup
• Release-suite record + profile aggregates committed (markdown only per CI guard)

🤖 Generated with Claude Code

[danReynolds]

Added the admission-rate benchmarking + an encapsulation pass (see latest commit): tier-1 admits 30/62 distinct entries on an app-shaped chat+feed stream mix and eliminates 3,000 re-queries in the burst with 0 bails — but the rejected set holds the highest-churn DESC+LIMIT screens (~3,100 remaining reader replies), which is the quantified case for tier-2 (composite-key ordering + LIMIT K+buffer). Per-cycle delta decode also moved out of the engine into RowDeltaBatch; engine-side IVM state is now just StreamEntry.ivm + the table_info cache. 298/298 tests, analyze clean, finalizer green.

[danReynolds]

Pushed three updates (branch rebased onto main with #153 merged — conflicts in the experiments metadata resolved by union):

1. Fix for a pre-existing data race this PR surfaced (own commit, cherry-pickable if you'd rather land it separately): Database.diagnostics() calls sqlite3_db_status on reader connections guarded by in_use — dead code since exp 030's dedicated-reader assignment — so the main isolate was reading live NOMUTEX connections; SCHEMA_USED measures via the connection's pnBytesFreed dry-run, corrupting a mid-query reader's allocation accounting (flaky reader SEGV, ~1-in-30 stream_test runs once this PR's detached admission reads made readers busy at diagnostics-poll time). Read workers now bracket each request with resqlite_reader_set_busy (two ~ns leaf FFI calls); the sacrifice path clears it before Isolate.exit. Bisection + 100/100 clean stress runs in the experiment doc.
2. Test hardening: the e2e IVM tests used fixed 60ms settles and flaked under full-suite load; now poll-until-count / quiet-window. 8/8 clean full-suite runs (~1-in-3 flaked before).
3. JOURNAL: added the transferable lesson (a dead guard flag is a data race waiting for a traffic pattern).

Full suite 8×, analyze, and finalizer all green post-rebase.

[danReynolds]

v2 tier stack pushed (per the goal to take this as far as it goes — all isolated on this branch). Summary of what's now here beyond tier 1:

Tiers — one fail-closed grammar, three admission modes:

• Full maintenance extended with composite ORDER BY intCol [DESC], pk [DESC] (explicit tiebreak ⇒ exact tie order) and LIMIT K windows (top-K cache, complete-set tracking, departures-from-full-windows fall back); TEXT equality under introspected no-COLLATE tables.
• Skip-only (tier 1.5): proven-miss elision for unmaintainable-result shapes (DESC w/o tiebreak, OFFSET, DISTINCT…) — no cached state at all.
• Aggregates (tier 3): COUNT/SUM/MIN/MAX/AVG … AS alias, exactly seeded and maintained; extremum departures re-seed.

Headline results:

evidence	result
Tracelite gate	many-streams −18.2%, formal improved verdict — first cleared primary gate in this direction's history; reproduced order-flipped (+19.5% main-slower)
App-shaped admission audit	7/9 shapes admitted (3/8 at tier 1); 7,643 invalidation decisions with zero reader re-queries, 0 bails; burst wall 132.9 ms vs main's ~200 ms while delivering per-write patches
Honest costs	high-cardinality +3.5–4.8% both orders (O(admitted-streams) main-isolate eval at 100-streams-one-table density — v3 fix identified: per-table equality-predicate index for O(1) dispatch); keyed-PK neutral for v2 (tier-1 alone won it)

Two ordering defects found and fixed by the new randomized equivalence harness (3 seeds × 12 rounds × 9 streams, every emission vs fresh select; both fired ~1-in-5 under load, never in deterministic replays): reader-built IVM baselines race late writer replies → all state now builds through writer-ordered reads; and maintained state survives only an unbroken chain of processed cycles → any bypassed delta-bearing cycle drops the state (a hash-suppressed re-query validates emissions without re-syncing state). Both are recorded as a JOURNAL lesson (cross-port replies have no happens-before).

315/315 tests, 20/20 equivalence-loop runs, analyze + finalizer green.