docs: record the v1 distributed-execution design

[contrasam]

Documents where the correctness lives, which is the part that is expensive to
revisit once anything is built on it.

The decision: the log is the arbiter, not a coordination service.
Single-writer-per-execution is enforced by a conditional append
(append(id, event, expectedSeq)) rather than a distributed lock. A lock
service can tell a node it holds the lock but not that it still holds it at
the instant it writes — a GC pause or partition between those moments is
enough for two nodes to both believe they own the stream. Pushing the check
to where the write lands removes that class of failure entirely, and
Catalyst's dense per-execution seq is already a natural fence.

The consequence matters more than the mechanism: placement becomes an
optimisation rather than a correctness dependency. A stale writer is rejected
by storage even if the coordination layer is wrong, so the choice of actor
system is a late, reversible decision.

Surveyed both sibling projects for this. Cajun's ClusterActorSystem is the
closest fit for placement and can be swapped in behind KeyedLock — the seam
was left for exactly this — but nothing rests on its maturity. Bayou does not
distribute at all (single process, no node identity), yet it shares Gumbo
with Catalyst, so it is useful within a node and could later consume the same
primitives to become clustered itself. That is the argument for putting these
primitives in Gumbo rather than in an actor system: it is the layer both
already share.

The three gaps are all storage-side — conditional append, lease CAS with TTL,
and a claimable-work index. The last is the least obvious: there is currently
no way to ask the log what needs running, since every read path starts from an
id you already hold.

Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01G54yVgLZGZ3mzBv4kPc9F3

[greptile-apps]

Greptile Summary

Records the proposed v1 distributed-execution architecture.

• Establishes storage-backed conditional append as the single-writer correctness fence.
• Documents lease-based work claiming, Gumbo’s multi-writer prerequisites, and phased implementation.
• Qualifies recovery guarantees for in-flight model completions and records the remaining duplicate-call window.

Confidence Score: 5/5

The documentation-only PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
docs/distribution.md	Documents the distributed-execution invariants, storage primitives, operational model, limitations, and implementation phases.
ROADMAP.md	Expands the distributed-execution roadmap and records the in-doubt model-completion work.

Sequence Diagram

sequenceDiagram
    participant A as Node A
    participant B as Node B
    participant L as Shared Log
    A->>L: Claim execution lease
    L-->>A: Lease granted
    A->>L: "append(event, expectedSeq=N)"
    L-->>A: "Append accepted, seq=N+1"
    Note over A: Node stalls and lease expires
    B->>L: Claim expired lease
    L-->>B: Lease granted
    B->>L: "append(event, expectedSeq=N+1)"
    L-->>B: "Append accepted, seq=N+2"
    A->>L: "append(event, expectedSeq=N+1)"
    L-->>A: Reject stale writer

Loading

_{Reviews (5): Last reviewed commit: "docs: don't let conditional append degra..." | Re-trigger Greptile}