Skip to content

Upgrade to Gumbo 0.3.0 and fix shared-log tail reads - #20

Merged
contrasam merged 1 commit into
mainfrom
claude/catalyst-0.3.0-verify-ephul0
Jul 26, 2026
Merged

Upgrade to Gumbo 0.3.0 and fix shared-log tail reads#20
contrasam merged 1 commit into
mainfrom
claude/catalyst-0.3.0-verify-ephul0

Conversation

@contrasam

Copy link
Copy Markdown
Contributor

Summary

Upgrade Catalyst to Gumbo 0.3.0 and fix a critical bug in GumboEventLog.readFrom() where tail reads were keyed on the log's global seqnum instead of the execution's own streamVersion. This caused silent data corruption when multiple executions shared a log: snapshot warm reads would return the entire stream and the reducer would re-apply events already folded into the snapshot, double-counting every step, token, and cost.

Key Changes

  • Upgrade Gumbo dependency from 0.2.0 to 0.3.0 across all build files and documentation
  • Fix readFrom() to use version-keyed reads: Replace readAfter(seqnum) with readAfterVersion(streamVersion) so tail reads are keyed on the execution's own stream position, not the log's global position
  • Fix latestSeq() cold path: Use getLatestVersion() instead of scanning the entire stream
  • Update terminology: Replace localId with streamVersion throughout (0.3.0 renamed the accessor; old names still work but are deprecated)
  • Add single-writer enforcement: Catch LogAlreadyOpenException from the file adapter and surface it as a clear UncheckedIOException with the message at the top level (not buried as a cause), since a directory held by another process is a configuration mistake, not a debugging scenario
  • Add comprehensive regression tests:
    • readFromReturnsOnlyThisExecutionsTailWhenAnotherExecutionSharesTheLog: in-memory log with two interleaved executions
    • readFromOnAFileBackedSharedLogReadsOnlyThisExecutionsTail: file-backed log with two executions across a reopen
    • aSecondWriterOnTheSameDirectoryIsRefusedWithAClearMessage: verify single-writer lock and that it releases on close
    • warmInspectMatchesColdWhenAnotherExecutionSharesTheLog: end-to-end acceptance test reproducing the snapshot warm-fold corruption and verifying the fix

Implementation Details

The bug was invisible in the existing test suite because every test used a single execution per log, where streamVersion and global seqnum are the same number. With a second execution present, readAfter(seqnum) would return events from the entire log tail (not just this execution's tail), and the warm snapshot fold would re-apply the snapshot's own prefix.

The fix is straightforward: readAfterVersion() is keyed on the execution tag's own stream position. The cold path for latestSeq() now calls getLatestVersion() directly instead of scanning, which is both faster and correct.

The file adapter's exclusive lock (new in 0.3.0) is now surfaced clearly: LogAlreadyOpenException is caught and re-thrown as UncheckedIOException with the original message intact, so operators see "directory already open" rather than a cause chain. The lock is released on close, so crash→resume still works.

https://claude.ai/code/session_017v6M83hQFt4KuLP6Dvk7Sg

Gumbo 0.3.0 lands the log-layer fixes from docs/gumbo-requirements.md. The
upgrade on its own changed nothing: 0.3.0 *adds* readAfterVersion rather than
altering readAfter, so the old call kept compiling and kept being wrong, and
the whole suite stayed green across the bump. The caller had to move.

D4 is the one that was corrupting state. Catalyst's seq is a per-tag position,
but readFrom passed it to readAfter, which is keyed on the log's global seqnum.
The two numbers are equal only while a log holds a single execution — true in
every test, false in every real deployment. With a second execution present the
snapshot warm read returned the entire stream, and the reducer re-folded the
snapshot's own prefix on top of itself: timeline steps, tokens, cost and
attempt counters all double-counted, with no error at the seam. Measured here
at 253 events returned for a tail that should have been under 100.

readFrom now uses readAfterVersion and latestSeq's cold path uses
getLatestVersion() instead of scanning the whole stream. Accessors move to
streamVersion; localId was a fossil of Boki's per-engine id and 0.2.0's
accessors still work, deprecated.

The file adapter now takes an exclusive directory lock, so a second writer is
refused rather than silently assigning duplicate seqs and clobbering index.dat.
GumboEventLog surfaces that message at the top level instead of burying it
under "Failed to open Gumbo log" — the value of the check is that it is
legible. The lock releases on close and on process death, so crash → resume
still reopens (M0's kill -9 demo verifies this).

Tests are the point of the change, since the defect shipped precisely because
every log-layer fixture used one execution in a fresh log — the single
configuration where a stream version and a global seqnum are indistinguishable:

- GumboEventLogTest: tail reads with a second execution sharing the log, on
  both the in-memory and file-backed adapters (the latter across a reopen), and
  the single-writer refusal
- SnapshotAcceptanceTest.warmInspectMatchesColdWhenAnotherExecutionSharesTheLog:
  the end-to-end warm-fold corruption. Verified to fail when the seqnum-keyed
  read is restored, so it has teeth

Full reactor green and all 15 exit demos pass against the 0.3.0 release tag.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017v6M83hQFt4KuLP6Dvk7Sg
@greptile-apps

greptile-apps Bot commented Jul 26, 2026

Copy link
Copy Markdown

Greptile Summary

Upgrades Catalyst to Gumbo 0.3.0 and corrects shared-log stream handling.

  • Uses per-execution stream versions for appends, decoding, tail reads, and latest-sequence lookup.
  • Surfaces Gumbo’s exclusive file-lock failure as a clear unchecked I/O error.
  • Adds in-memory, file-backed, snapshot, and lock-release regression coverage.
  • Updates build configuration and documentation for Gumbo 0.3.0.

Confidence Score: 5/5

The PR appears safe to merge, with the shared-log correction covered across tail reads, persistence reopen, snapshot folding, and writer-lock lifecycle.

The changed implementation consistently uses per-execution stream versions, preserves the EventLog boundary behavior for full and exclusive tail reads, and adds targeted tests for the previously corrupting multi-execution scenario.

Important Files Changed

Filename Overview
catalyst-gumbo/src/main/java/com/cajunsystems/catalyst/gumbo/GumboEventLog.java Replaces global-sequence tail reads with per-stream version reads, uses the stream tip for cold latest-sequence lookups, and translates exclusive-lock failures clearly.
catalyst-gumbo/src/test/java/com/cajunsystems/catalyst/gumbo/GumboEventLogTest.java Adds shared-log regression coverage for both persistence adapters and verifies exclusive writer locking and release.
catalyst-api/src/test/java/com/cajunsystems/catalyst/api/SnapshotAcceptanceTest.java Adds an end-to-end test proving warm snapshot folds match cold folds when executions share a log.
pom.xml Updates the managed Gumbo dependency from 0.2.0 to 0.3.0.

Sequence Diagram

sequenceDiagram
    participant Runtime as CatalystRuntime
    participant Snapshot as Snapshot Store
    participant Log as GumboEventLog
    participant Stream as Gumbo Execution Stream
    Runtime->>Snapshot: Load snapshot through streamVersion N
    Runtime->>Log: readFrom(executionId, N)
    Log->>Stream: readAfterVersion(N)
    Stream-->>Log: "Events with streamVersion > N"
    Log-->>Runtime: Decoded execution tail
    Runtime->>Runtime: Fold snapshot plus tail
Loading

Reviews (1): Last reviewed commit: "Upgrade to Gumbo 0.3.0 and adopt the ver..." | Re-trigger Greptile

@contrasam
contrasam merged commit 75b8cc6 into main Jul 26, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants