fix: cap the consolidation response, and stop reading a store too large to read (#346)

[fdaviddpt]

Closes #346

The reporter's root cause is wrong, and the way it is wrong is the bug

The report says recent.md and archive.md "only ever get appended to". They have never been appended to: run-consolidation.sh:159-160 cps both wholesale from the model's response, and those two lines are unchanged since the initial commit. There is exactly one writer in the tree.

Consolidation caps its input and never caps its output.

• pipeline/consolidate.py:315-322 refuses to send a prompt over consolidate_max_bytes (default 600000).
• pipeline/haiku.py:686 runs the CLI with capture_output=True, timeout=timeout — bounded by a wall clock, not by bytes. The size of a response was never a quantity anything downstream measured.
• pipeline/shell.py writes result.recent to a temp file verbatim and run-consolidation.sh:159 copies it over recent.md. No size check on that path.

Measured on a stub: one round wrote 52,428,825 bytes — 87× the cap the same function had refused to send seconds earlier.

One oversized write is permanent, and that is what produces the reported symptom. recent.md is part of the input the cap is measured on, so the round after an oversized write assembles an oversized prompt, raises ConsolidationTooLarge, and skips — and so does every round after it. _rotate_archive is no escape when the bulk is recent.md. The file then never grows again and never shrinks either, which from outside is indistinguishable from a file that is only ever appended to. The observation was right; the mechanism was inside out.

What this changes

1. Refuse an oversized response (consolidate()). Deliberately a plain ConsolidationSkipped, not ConsolidationTooLarge: that subclass means "the input was too big, shrink it and retry", and the caller acts on it by rotating archive.md. Nothing about the input was wrong here, so a retry would spend another model call to be handed another oversized response and would rotate away a healthy archive for nothing.

2. stat the store before reading it (cmd_consolidate()). The cap was enforced on the assembled prompt, so a 6.4 GB recent.md had to be read into memory and a prompt built around it before the pipeline was allowed to notice it was too large to send. That is the allocation that took the reporter's machine down, from a script that runs disowned beside a live session. It cannot produce a false skip: the assembled prompt is the template plus per-file labels plus these bytes, so it is strictly larger than their sum.

3. Name, do not inject, an oversized memory file (session-start-hook.sh). Nothing above helps the 6.4 GB file someone already has on disk, and that file is what froze every claude launch in that project. Over thresholds.memory_inject_max_bytes (new, default 200000) the file is listed with its size instead of cat'd — the same "kept but not injected" trade Recall should read rotated archive-<date>.md siblings #124 makes for rotated archives, reached by size rather than by filename. The bytes stay on disk and stay greppable.

Not fixed here

• A store that is already broken is not recovered. After this, such a store starts sessions and skips consolidation forever; the only cure is still deleting the file. Filed separately.
• now.md / today-*.md are the genuinely append-only files — staging_append is a literal >> with no cap, and five documented branches append a duplicate span without rolling it back. Different bug, different files, filed separately.
• The 600-token compression instruction is advisory. Nothing enforces it, so a faithful re-emitter grows recent.md ~2 KB/round; measured hitting the cap at round 298. Bounded by the cap by construction, so it cannot reach a gigabyte on its own — it is the slow grower, not the multiplier. Noted in the CHANGELOG.

Verification

TDD: the test file was written first and failed 5/6 (the pass is the control — a normal response must still be written). Representative RED:

assert 'CONSOLIDATION_STATUS=skip' in 'STAGING_COUNT=1\nCONSOLIDATION_STATUS=ok\nRECENT_OUT=/var/...'
AssertionError: recent.md was read into memory to discover it is too large to read
AssertionError: an oversized recent.md was cat'd into the session (2400447 bytes of hook output)

GREEN, full suite:

1576 passed, 43 skipped, 1 warning in 1139.91s
Required test coverage of 80% reached. Total coverage: 92.95%

An existing test caught a real defect in the first attempt: the up-front rotation was not undone when the model call raised. Fixed with a trailing except Exception: _restore_rotation(); raise.