store/fjall: carry stream_id in the $all index so read_all/subscribe_all surface it

[joeldsouzax]

STATUS UPDATE (2026-06-21) — no longer a #145 blocker, no longer pre-freeze-urgent.
The export/import design (#145) moved to a per-stream export model: it reads read_stream(id) (where stream_id is the call argument) and enumerates streams via a new additive StreamLister::list_streams trait — it does not use read_all/the $all index at all. So export never needed stream-id-on-the-global-path. This card's export motivation is gone.
What remains is the original second justification only: a live subscribe_all consumer (a multi-stream projection / router) wanting stream attribution. No such consumer exists yet (Agency's runner is future), so by YAGNI this is speculative — keep on the backlog, off the pre-freeze critical path, until a real cross-stream subscriber needs it.
If it does proceed, lead it as a generic store-contract change (RawEventStore / PersistedEnvelope / AllStream item), with fjall + in-memory as implementors — not the fjall-led framing below.

Original body (export-framed) preserved for reference:

Summary

read_all (and the subscribe_all cursor built on it) cannot tell a consumer which stream an event belongs to. The $all index stores no stream id — not in its key, not in its value — and PersistedEnvelope has no stream_id field. We must carry the stream id in the $all index and surface it on the all-streams read path.

This is pre-freeze: it changes a storage structure (the fjall events_global partition) and the public all-streams read surface, both of which export/import (#145) and any cross-stream consumer depend on.

The gap (verified against current code, 2026-06-21)

• Wire frame (nexus-store/src/wire.rs:13) = [u64 global_seq][u32 schema_version][u16 et_len][u32 meta_len][event_type][metadata?][payload]. No stream_id, no version.
• fjall events key (nexus-fjall/src/encoding.rs:67) = [u16 BE id_len][id_bytes][u64 BE version]. The stream id lives only here, in the per-stream partition key.
• fjall $all index (events_global, nexus-fjall/src/store.rs:41, :196-197) = key [u64 BE global_seq][u64 BE version], value = the wire frame (the same slice bytes inserted into events). Stream id appears in neither the key nor the value.
• PersistedEnvelope (nexus-store/src/envelope.rs:309) fields = version, global_seq, schema_version, value, event_type_range, payload_range, metadata_range. Accessors expose version / global_seq / schema_version / event_type / payload / metadata — no id / stream_id (grep for stream_id/StreamId/\bid\b in envelope.rs returns nothing).

Why it was never needed before:

• read_stream(id, from) — the id is the query argument; every returned envelope belongs to it by construction. Redundant to stamp.
• append(id, …) — the id is a call argument, not a field.
• read_all(from) — no id argument (it's cross-stream), and the index it scans dropped the id to keep the key small. So on this path the id is genuinely unavailable, at every layer.

Why it matters

• Blocks Add event export/import paths for cross-device replication #145 export-all-streams. Export's "a device dumps its whole feed" path reads read_all and must label each exported block with its stream_id (the import side files events under an origin-namespaced stream — see Add event export/import paths for cross-device replication #145 §4/§5). Today it cannot, because read_all can't say which stream each event came from. A per-block stream_id is mandatory in the chunk format.
• Hurts every cross-stream consumer. Any subscribe_all reader (projections over multiple streams, routers, audit tailers) gets events with no way to attribute them to a stream without a side lookup that doesn't exist.

Proposed change

Two independent decisions:

A. Where the id goes in the $all index

• (A1) In the value — wrap the stored bytes as [u16 id_len][id_bytes][frame] instead of the bare frame. Leaves the key ([global_seq][version]) and its keyset-resume-on-global_seq scan untouched. Costs 2 + id_len bytes per $all row; the two partitions (events, events_global) stop sharing identical value bytes (today they share one slice — store.rs:194-197).
• (A2) In the key — [global_seq][id_len][id][version]. Key still sorts by global_seq first, so global ordering and keyset resume are preserved, but the key is now variable-length.

Leaning A1 (value) — keeps the key fixed-width and the global-order scan logic unchanged; the id is data, not an ordering term.

B. How the id reaches the consumer

• (B1) Add a stream_id to PersistedEnvelope, populated on both paths (stamped from the query arg on read_stream, read from the index on read_all). Uniform; one item type everywhere.
• (B2) Leave PersistedEnvelope as-is; make the AllStream item a richer pair carrying the id alongside the envelope. Avoids putting a "came-from-the-query" field on read_stream envelopes.

Leaning B1 — uniform envelope, simplest for consumers; the id is a real property of a persisted event regardless of how it was read.

The id type: the store layer is generic over Id; surface it as the stored id bytes (&[u8] / owned Bytes) at the raw layer, reconstituting the typed Id at the repository layer, consistent with how append/read_stream already take id bytes.

Atomicity / safety

The id is written into events_global in the same transaction as the event (store.rs:194-197 already), so adding it is trivially atomic — no new multi-step access (CLAUDE rule 1).

Touch points

• nexus-fjall: encoding.rs (encode_global_key / a new value wrapper), store.rs (append insert + read_all decode), all_stream.rs (yield the id).
• nexus-store: store.rs (RawEventStore::read_all / AllStream item contract — document id semantics), envelope.rs (if B1), testing.rs (InMemoryStore must match the new contract).
• Adapters defend their own boundary: a $all row missing/!UTF-8 id is a CorruptValue, distinct from input errors (CLAUDE rule 3).

Tests (the 4 cross-cutting categories first — CLAUDE rule 7)

1. Sequence/Protocol — append across interleaved streams, read_all, assert each event's stream_id matches the stream it was appended to (exact assert_eq, not "contains").
2. Lifecycle — write across streams → close → reopen → read_all still attributes every event to the right stream (write-close-reopen-verify); a $all row with a corrupted id surfaces CorruptValue, not a wrong stream.
3. Defensive boundary — feed the decoder a $all value with id_len exceeding the buffer; assert a structured decode error, no panic, no out-of-bounds.
4. Linearizability/isolation — concurrent writers on different streams + a read_all/subscribe_all reader; assert every observed event carries the correct stream id under a Barrier-synchronized overlap.

Gating

• Milestone 1 — Feature-Complete (pre-freeze): changes the $all storage format and the public all-streams read contract. Pre-1.0 / experimental, so no on-disk migration is owed — existing dev stores rebuild.
• Blocks Add event export/import paths for cross-device replication #145 (export/import) — export-all-streams cannot label blocks with stream_id until this lands.
• Builds on Global $all subscription cursor (all-streams, GlobalSeq-ordered) #214 / feat: global $all subscription cursor (GlobalSeq-ordered, all-streams) (#214) #216 (the $all index / subscribe_all cursor that introduced events_global).