Skip to content

Add scratch-file offload for elided tool output (ports OpenDev's truncation.rs pattern)#6

Merged
maxkilla merged 1 commit into
mainfrom
comb/scratch-file-offload
Jul 16, 2026
Merged

Add scratch-file offload for elided tool output (ports OpenDev's truncation.rs pattern)#6
maxkilla merged 1 commit into
mainfrom
comb/scratch-file-offload

Conversation

@maxkilla

Copy link
Copy Markdown
Owner

What

Ports a pattern from OpenDev's truncation.rs (opendev-to/opendev, crates/opendev-tools-impl/src/truncation.rs) into comb: before eliding the middle of oversized tool output, save the full original to disk so it's recoverable.

Why

comb's compress() currently elides the middle of oversized output permanently — there was no way to get it back if it turned out to matter. OpenDev solves the same problem in its own tool-output truncation path by always writing the full output to a scratch file first and returning a path alongside the truncated preview.

What changed

  • saveOverflow(text): writes the full untouched text to ~/.claude/comb/tool-output/tool_<timestamp>_<8-hex>, mode 0o600. Never throws — on any FS error it returns null and compress() just omits the recovery hint; the file write is best-effort, the compression itself is not.
  • 1MB cap on the saved file (MAX_OVERFLOW_BYTES), matching OpenDev's constant. Past the cap, the saved file is itself capped head 75% / tail 25%, not a flat cutoff — so both how the output started and how it ended survive even in the worst case.
  • compress()'s marker now includes , full output: <path> when a save succeeded.
  • COMB_COMPRESS_SAVE_FULL=0 opt-out. Defaults on (opt-out rather than opt-in) since silently losing tool output seems worse than an occasional stray file on disk.
  • cleanupOldFiles(): 7-day retention sweep. Exported but not auto-invoked by the hook itself — PostToolUse runs in the hot path on every matched tool call, so a directory scan there is wasted work every time. Meant to be wired into a SessionStart hook or cron if disk growth in OVERFLOW_DIR becomes a problem.

Testing

  • All 19 pre-existing tests in test/compress.test.js pass unmodified — nothing about the existing elision/salvage/gate/rule-store behavior changed.
  • 9 new tests in test/overflow.test.js:
    • saveOverflow writes the exact full text and returns a path
    • the 1MB cap actually caps, with head/tail both surviving
    • compress()'s marker contains a working path, and the saved file matches the original byte-for-byte
    • COMB_COMPRESS_SAVE_FULL=0 writes nothing and omits the path from the marker
    • cleanupOldFiles() keeps recent files, removes files past retention (verified via fs.utimesSync, not a real 7-day wait), leaves non-tool_* files alone, and returns 0 gracefully when the directory doesn't exist yet
  • Also ran the hook end-to-end via actual stdin/stdout (as Claude Code would invoke it) with a synthetic oversized Bash tool_response, confirmed the returned JSON's updatedToolOutput.output contains a working path and the file on disk matches the pre-elision original exactly.

What I deliberately did NOT port from OpenDev

  • No line-count threshold alongside the existing character-count threshold — comb's HEAD_CHARS/TAIL_CHARS/THRESHOLD are already purely character-based, and OpenDev's dual line+byte trigger wasn't something comb's actual design was missing; would've been scope creep.
  • Kept COMB_COMPRESS_SAVE_FULL opt-out (vs. no toggle at all in OpenDev) since comb already has a convention of explicit env-var toggles for anything that touches the filesystem/network (see COMB_RULE_STORE_URL).

Ports OpenDev's truncation.rs recovery pattern (crates/opendev-tools-impl):
before compress() elides the middle of oversized output, save the full
original to ~/.claude/comb/tool-output/ and embed the path in the marker,
so elision is recoverable instead of permanent data loss.

- saveOverflow(): writes full text, never throws, caps at 1MB with
  head75/tail25 split (not a flat cutoff) if exceeded, matching OpenDev's
  save_overflow behavior
- cleanupOldFiles(): 7-day retention sweep, exported but not auto-invoked
  on the hot path (same reasoning as OpenDev's cleanup_old_files)
- COMB_COMPRESS_SAVE_FULL=0 opt-out (defaults on, since silently losing
  data is worse than a stray file on disk)
- 9 new tests in test/overflow.test.js covering the write path, the
  1MB cap, the opt-out, and cleanup; all 19 pre-existing tests still pass
  unmodified

Found while reviewing OpenDev's compaction/truncation source for comb's
design - see crates/opendev-tools-impl/src/truncation.rs upstream.
@maxkilla maxkilla merged commit b31813a into main Jul 16, 2026
@maxkilla maxkilla deleted the comb/scratch-file-offload branch July 16, 2026 11:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant