diff --git a/CHANGELOG.md b/CHANGELOG.md index 11ecb90..06f69a7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,26 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [Unreleased] + +### Fixed + +- **`recent.md` and `archive.md` could be written without bound, and the write that broke the store disabled the only thing that could repair it** ([#346](https://github.com/Digital-Process-Tools/claude-remember/issues/346)) — a reporter's store reached **6.4 GB** (`recent.md`) and **1.8 GB** (`archive.md`). The `SessionStart` hook `cat`s both into every session, so every `claude` launch in that project froze; on macOS iTerm2 reached ~56 GB and the machine needed a restart. + + **The reported cause was that these two files have no rotation and are only ever appended to. Nothing appends to them.** The only writer in the tree is `cp "$RECENT_OUT" "$RECENT_FILE"` in `scripts/run-consolidation.sh`, which *replaces* the file wholesale with the consolidation's output, and it has done exactly that since the initial commit. The observation behind the report was still precisely right — the files only ever grew and never shrank — but the route there is the opposite of an append. + + **Consolidation capped its input and not its output.** `consolidate()` refuses to *send* a prompt over `thresholds.consolidate_max_bytes` (staging + `recent.md` + `archive.md`, default 600000). Between the model call returning and the `cp`, no byte count was ever taken: `capture_output=True` bounds the CLI subprocess by a wall clock and not by bytes, so the size of a response was not a quantity anything downstream measured. `cmd_consolidate` wrote `result.recent` to a temp file verbatim and the shell copied it over `recent.md`. + + **One oversized write is permanent, and that is what makes it this bug rather than one bad round.** `recent.md` is part of the input the cap is measured on. So the round after an oversized write assembles an oversized prompt, raises `ConsolidationTooLarge`, and skips — and so does every round after that, forever. Rotating `archive.md` ([#122](https://github.com/Digital-Process-Tools/claude-remember/issues/122)/[#123](https://github.com/Digital-Process-Tools/claude-remember/issues/123)) is no escape when the bulk is `recent.md`. The file can then never grow again and never shrink either, which from outside is indistinguishable from a file that is only ever appended to. + + **The same number now caps both directions.** A response larger than `consolidate_max_bytes` is refused as non-conforming, which is the established non-destructive path ([#89](https://github.com/Digital-Process-Tools/claude-remember/issues/89)/[#202](https://github.com/Digital-Process-Tools/claude-remember/issues/202)): staging and memory are both left intact and the next run retries. Deliberately *not* `ConsolidationTooLarge` — that subclass means "the input was too big, shrink it and retry", and acting on it would spend another model call to be handed another oversized response while rotating away a healthy archive for nothing. + + **The store is now sized before it is read.** The cap was enforced on the assembled prompt, so the whole store had to be read into memory and a prompt built around it before the pipeline was allowed to notice it was too large to send — several times the store's size in allocation to reach a decision `stat` answers for free, from a script that runs disowned beside a live session. That is the part that took the reporter's machine down. It cannot be a false skip: the prompt is the template plus per-file labels plus those bytes, so a sum already over the cap is proof the prompt would be. `archive.md` being the bulk still rotates, now before the read rather than after it, and an up-front rotation is undone on every path where the round does not go through. + + **And a store that is *already* broken no longer freezes the session**, because none of the above helps the 6.4 GB file someone has on disk today. A memory file over `thresholds.memory_inject_max_bytes` (new, default 200000) is named with its size instead of being `cat`'d — [#124](https://github.com/Digital-Process-Tools/claude-remember/issues/124)'s "kept but not injected" vocabulary, reached by size instead of by filename. The bytes stay on disk and stay greppable; what stops is pouring them into a context window that cannot hold them. A session that starts and says the store is broken is worth more than one that hangs. + + Two growth paths were found and only one is this bug. The other: consolidation is *told* to keep `recent.md` under 600 tokens and nothing enforces it, so a model that faithfully re-emits the file plus one day per round grows it monotonically — measured at ~2 KB/round, reaching the 600000 cap after ~298 rounds and then freezing. That one is bounded by the cap by construction and cannot reach a gigabyte; it is a compression-compliance question, filed separately. + ## [0.19.0] — A compaction is not a new session ### Changed diff --git a/README.md b/README.md index 6969f7d..e431825 100644 --- a/README.md +++ b/README.md @@ -422,7 +422,8 @@ Put cross-project preferences (timezone, cooldowns) in `~/.remember/config.json` | `prompt_stamp` | `full` | What the `UserPromptSubmit` hook injects into the model's context. `full` — `[14:30 CEST — jack — 45%]`, unchanged from every previous release. `stable` — `[jack]` only: the clock and the context percentage both change between turns, and the percentage climbs on *every* prompt for anyone running the status line, so dropping only the clock would leave the line volatile. The `>= 95` context warning is **kept**, because it is threshold-gated and changes bytes only when it changes behaviour. `off` — nothing at all, warning included. An unrecognised value reads as `full` ([#301](https://github.com/Digital-Process-Tools/claude-remember/issues/301)). | | `model` | `haiku` | Model used for the summarization / consolidation `claude -p` call. `REMEMBER_MODEL` overrides it. Documented as an env var only until #176, though `config.json` is the source of truth. | | `reject_pattern` | _(empty)_ | Overrides the reject-gate regex that keeps model refusals out of the memory layer. Empty → the narrow built-in default; `none` → gate off; anything else → a case-insensitive regex. An invalid regex falls back to the default. `REMEMBER_REJECT_PATTERN` overrides it. | -| `thresholds.consolidate_max_bytes` | `600000` | Max UTF-8 size of the staging content sent to the consolidation model. Read by `run-consolidation.sh`; documented in `config.example.json` but missing from this table until #176. | +| `thresholds.consolidate_max_bytes` | `600000` | Max UTF-8 size of the staging content sent to the consolidation model. Read by `run-consolidation.sh`; documented in `config.example.json` but missing from this table until #176. Since [#346](https://github.com/Digital-Process-Tools/claude-remember/issues/346) the same number also caps what consolidation may **write**: a response larger than this is refused rather than copied over `recent.md`/`archive.md`. | +| `thresholds.memory_inject_max_bytes` | `200000` | A memory file larger than this is **named with its size instead of injected** at session start ([#346](https://github.com/Digital-Process-Tools/claude-remember/issues/346)). It stays on disk and stays greppable; what stops is pouring it into a context window that cannot hold it — a multi-GB `recent.md` froze every `claude` launch in the reporter's project. A healthy memory file is kilobytes, so this only ever fires on a store that is already broken. Set `0` to disable. | | `debug` | _(unset)_ | Verbose logging for cooldowns and locks. Unset, each script keeps its own default — `save-session.sh` is verbose, the git-backup hook is quiet — which is what they did before this option was wired up (#176). `REMEMBER_DEBUG` overrides it. | | `haiku.oauth_token` | _(empty)_ | OAuth token the plugin hands to the nested `claude -p` **only when the host did not put `CLAUDE_CODE_OAUTH_TOKEN` in the hook subprocess env** — some desktop / Agent-SDK hosts withhold it from spawned children, so `claude -p` is unauthenticated and nothing ever saves ([#129](https://github.com/Digital-Process-Tools/claude-remember/issues/129)/[#131](https://github.com/Digital-Process-Tools/claude-remember/issues/131)). Create one with `claude setup-token`. The plugin holds this credential and passes it to the summarization CLI, so set it deliberately. A host-provided token always wins; `REMEMBER_OAUTH_TOKEN` overrides this. A malformed value is refused and reported in the daily log, never passed to the CLI. | diff --git a/config.example.json b/config.example.json index fe24663..ae856c4 100644 --- a/config.example.json +++ b/config.example.json @@ -11,7 +11,8 @@ "max_summary_failures": 3, "delta_lines_trigger": 50, "extract_max_bytes": 300000, - "consolidate_max_bytes": 600000 + "consolidate_max_bytes": 600000, + "memory_inject_max_bytes": 200000 }, "features": { "ndc_compression": true, @@ -71,7 +72,21 @@ "rather than overflowing Haiku's window with 'Prompt is too long' and", "stalling all saves. Unlike extract_max_bytes the input is NOT truncated,", "because consolidation rewrites recent/archive and truncating would drop", - "archived memory. Set 0 to disable the guard." + "archived memory. Set 0 to disable the guard.", + "Since #346 the same number caps the OUTPUT too: a model response larger", + "than this is refused instead of being written to recent.md/archive.md.", + "Capping only the input meant one oversized response became the permanent", + "record AND pushed the store past this cap, so every later run skipped and", + "the file could never shrink again." + ], + "memory_inject_max_bytes": [ + "A memory file larger than this (UTF-8 bytes) is NAMED with its size at", + "session start instead of being cat'd into context. Default 200000.", + "The bytes stay on disk and stay greppable — what stops is pouring them", + "into a context window that cannot hold them. A healthy memory file is", + "kilobytes, so this only fires on a store that is already broken; in #346", + "a 6.4GB recent.md froze every `claude` launch in that project. Set 0 to", + "disable." ], "prompt_stamp": [ "What the UserPromptSubmit hook injects into the model's context.", diff --git a/pipeline/consolidate.py b/pipeline/consolidate.py index a3df4be..bf2586c 100644 --- a/pipeline/consolidate.py +++ b/pipeline/consolidate.py @@ -333,6 +333,43 @@ def consolidate( "(SKIP or missing ===RECENT===/entry headers)" ) + # The cap above is the size this pipeline refuses to SEND. Refusing to + # WRITE more than that is the same number in the other direction, and the + # other direction is where #346 came from: ``capture_output=True`` bounds + # the CLI subprocess by a wall clock and not by bytes, so the size of a + # response was never a quantity anything downstream measured. + # ``cmd_consolidate`` writes ``result.recent`` to a temp file verbatim and + # run-consolidation.sh copies it over recent.md — an overwrite, never an + # append, and unbounded either way. + # + # One oversized write is permanent, which is what makes this the bug + # rather than one bad round. recent.md is part of the input the cap is + # measured on, so the round after an oversized write assembles an + # oversized prompt and skips, and so does every round after that. + # Rotating the archive is no escape when the bulk is recent.md. The file + # can then never grow again and never shrink either — which from outside + # is indistinguishable from a file that is only ever appended to, and is + # exactly how the reporter read it. + # + # Deliberately NOT ConsolidationTooLarge: that subclass means "the input + # was too big, shrink it and retry", and the caller acts on it by rotating + # archive.md. Nothing about the input was wrong here, so a retry would + # spend another model call to be handed another oversized response and + # would rotate away a healthy archive for nothing. + # + # Skipping is the established non-destructive direction (see + # ``_echoes_the_prompt``): staging and memory are both left intact and the + # next run retries, so a false positive costs one round and a false + # negative costs the permanent record. + if max_prompt_bytes > 0: + response_bytes = len(result.text.encode("utf-8")) + if response_bytes > max_prompt_bytes: + raise ConsolidationSkipped( + f"consolidation response too large ({response_bytes} bytes > " + f"{max_prompt_bytes} cap) -- refusing to write it to memory; " + f"staging + memory left untouched" + ) + recent_new, archive_new = parse_consolidation_response(result.text) return ConsolidationResult( diff --git a/pipeline/shell.py b/pipeline/shell.py index 9bc42b9..1939ae6 100644 --- a/pipeline/shell.py +++ b/pipeline/shell.py @@ -461,6 +461,56 @@ def cmd_consolidate(staging_dir: str, recent_file: str, archive_file: str, print("STAGING_COUNT=0") return + def _emit_skip() -> None: + # Skip status so the shell leaves recent.md/archive.md untouched and does + # NOT rename the source staging files to .done.md — they remain available + # for the next run. STAGING_COUNT is non-zero (we found files) but the + # shell gates on CONSOLIDATION_STATUS. + print(f"STAGING_COUNT={len(staging_contents)}") + print("CONSOLIDATION_STATUS=skip") + + # Size the store before reading it (#346). The cap is enforced on the + # assembled prompt, so the whole store had to be read into memory and a + # prompt built around it before the pipeline was allowed to notice it was + # too large to send: several times the store's size in allocation to reach + # a decision ``stat`` answers for free. Against the reporter's 6.4 GB + # recent.md that is what took the machine down, from a script that runs + # disowned beside a live session. + # + # It can never be a false skip. The assembled prompt is the template plus + # per-file labels plus these bytes, so it is strictly larger than their + # sum, and a sum already over the cap is proof the prompt would be. + rotated: str | None = None + + def _restore_rotation() -> None: + """Undo an up-front rotation when the round did not go through. + + Existence-checked because the handlers inside ``ConsolidationTooLarge`` + below do their own restore and then re-raise; an exception raised + inside an except clause does not re-enter its siblings, but the guard + keeps that a property of this function rather than of Python's + control flow. + """ + if rotated is not None and os.path.exists(rotated): + os.replace(rotated, archive_file) + + recent_size = os.path.getsize(recent_file) if os.path.exists(recent_file) else 0 + archive_size = os.path.getsize(archive_file) if os.path.exists(archive_file) else 0 + if max_prompt_bytes > 0: + embedded = sum(staging_raw_bytes.values()) + recent_size + archive_size + if embedded > max_prompt_bytes: + # Same recovery ConsolidationTooLarge gets below, taken before the + # read rather than after it: if archive.md is the bulk, rotate it + # to a dated sibling and carry on with a fresh one. Only when + # dropping it still would not fit — recent.md is the bulk, #346's + # shape — is nothing read at all. + if embedded - archive_size <= max_prompt_bytes: + rotated = _rotate_archive(archive_file) + if rotated is None: + _emit_skip() + return + archive_size = 0 + recent = "" if os.path.exists(recent_file): with open(recent_file, encoding="utf-8", errors="replace") as f: @@ -471,14 +521,6 @@ def cmd_consolidate(staging_dir: str, recent_file: str, archive_file: str, with open(archive_file, encoding="utf-8", errors="replace") as f: archive = f.read() - def _emit_skip() -> None: - # Skip status so the shell leaves recent.md/archive.md untouched and does - # NOT rename the source staging files to .done.md — they remain available - # for the next run. STAGING_COUNT is non-zero (we found files) but the - # shell gates on CONSOLIDATION_STATUS. - print(f"STAGING_COUNT={len(staging_contents)}") - print("CONSOLIDATION_STATUS=skip") - try: result = consolidate(staging_contents, recent, archive, max_prompt_bytes=max_prompt_bytes) @@ -488,7 +530,12 @@ def _emit_skip() -> None: # empty archive, so consolidation keeps progressing instead of skipping # every run forever. If there is nothing to rotate, or the retry still # overflows (staging + recent alone exceed the cap), restore and skip. - rotated = _rotate_archive(archive_file) + # Only reachable when the stat guard above let the round through and + # the template plus per-file labels tipped it over, or when the guard + # is disabled. An up-front rotation has already happened in the first + # case, so do not rotate a second time and orphan the first sibling. + if rotated is None: + rotated = _rotate_archive(archive_file) if rotated is None: _emit_skip() return @@ -505,13 +552,25 @@ def _emit_skip() -> None: except SummarizerSpawnDeclined as declined: # The spawn guard refused (#204). Staging is left exactly as it is and # the next run consolidates it — not a skip, which retires staging, and - # not a failure either. + # not a failure either. An up-front rotation is undone for the same + # reason: this round never happened, so nothing it moved may persist. + _restore_rotation() print(f"consolidate declined: {declined}", file=sys.stderr) sys.exit(EXIT_SPAWN_DECLINED) except ConsolidationSkipped: - # Model declined (SKIP) or returned non-conforming output. + # Model declined (SKIP), returned non-conforming output, or returned + # more bytes than the pipeline is willing to write (#346). + _restore_rotation() _emit_skip() return + except Exception: + # A transient failure (the model call erroring, say) must not leave the + # up-front rotation applied: nothing was consolidated, so archive.md + # has to be where the next run expects it. The post-hoc rotation path + # has restored itself on this branch since #123; the pre-read one owes + # the same guarantee. + _restore_rotation() + raise # Write results to temp files fd_r, recent_out = tempfile.mkstemp(prefix="remember-recent-", suffix=".md") diff --git a/scripts/session-start-hook.sh b/scripts/session-start-hook.sh index 7d91c2b..d2a10e0 100755 --- a/scripts/session-start-hook.sh +++ b/scripts/session-start-hook.sh @@ -967,17 +967,45 @@ if [ -n "$HAS_MEMORY" ]; then # Named rather than dropped, which is the #124 vocabulary for "kept but # not injected": a file nobody names is a file nobody greps, and a recap # that shrinks in silence is indistinguishable from a store that emptied. + # The last defence, and the only one that helps a store that is ALREADY + # broken (#346). A memory file is written by consolidation, and a bounded + # writer does nothing for the 6.4 GB recent.md someone already has on + # disk: this loop cat'd it into every session, which is what froze every + # `claude` launch in that project and took the reporter's iTerm2 to ~56 GB. + # + # Named rather than injected, which is the #124 vocabulary a few lines + # below for rotated archives — the same trade, reached by size instead of + # by filename. The bytes stay on disk and stay greppable; what stops is + # pouring them into a context window that cannot hold them. A file this + # size is a broken store either way, and a session that starts and says so + # is worth more than one that hangs. + MEMORY_INJECT_MAX_BYTES=$(config ".thresholds.memory_inject_max_bytes" 200000) + case "$MEMORY_INJECT_MAX_BYTES" in (''|*[!0-9]*) MEMORY_INJECT_MAX_BYTES=200000 ;; esac + OVERSIZED_MEMORY="" for MFILE in "${MEMORY_FILES[@]}"; do if [ -f "$MFILE" ] && [ -s "$MFILE" ]; then if [ "$SESSION_START_SOURCE" = "compact" ] && [ "$MFILE" != "$IDENTITY_FILE" ]; then continue fi + MFILE_BYTES=$(wc -c < "$MFILE" | tr -d ' ') + case "$MFILE_BYTES" in (''|*[!0-9]*) MFILE_BYTES=0 ;; esac + if [ "$MEMORY_INJECT_MAX_BYTES" -gt 0 ] && [ "$MFILE_BYTES" -gt "$MEMORY_INJECT_MAX_BYTES" ]; then + OVERSIZED_MEMORY="${OVERSIZED_MEMORY}${MFILE} (${MFILE_BYTES} bytes) +" + continue + fi BASENAME=$(basename "$MFILE") echo "--- $BASENAME ---" cat "$MFILE" echo "" fi done + if [ -n "$OVERSIZED_MEMORY" ]; then + echo "--- too large to inject (kept on disk; grep on request) ---" + printf '%s' "$OVERSIZED_MEMORY" + printf 'A healthy memory file is kilobytes. One this size means consolidation wrote a response nobody bounded (see thresholds.memory_inject_max_bytes) and has been skipping ever since; run /remember:doctor.\n' + echo "" + fi if [ "$SESSION_START_SOURCE" = "compact" ]; then # Built before the header is printed, so the header is never printed # over an empty list — a store can hold identity.md and nothing else. diff --git a/tests/test_unbounded_memory_write_346.py b/tests/test_unbounded_memory_write_346.py new file mode 100644 index 0000000..65cd792 --- /dev/null +++ b/tests/test_unbounded_memory_write_346.py @@ -0,0 +1,296 @@ +"""recent.md / archive.md can be written without bound (#346). + +The reporter's store reached 6.4 GB (`recent.md`) and 1.8 GB (`archive.md`), +and every `claude` launch in that project froze because the SessionStart hook +`cat`s both into context. Their diagnosis was "unlike now.md / today-*.md, +these two have no rotation — they only ever get appended to". + +That diagnosis does not survive the code. Nothing appends to either file. The +only writer in the tree is ``cp "$RECENT_OUT" "$RECENT_FILE"`` at +scripts/run-consolidation.sh:159, which REPLACES the file wholesale with the +consolidation's output. The observation behind the diagnosis was still exactly +right — the files only ever grew and never shrank — but the route there is the +opposite of an append: + +**The consolidation caps its input and does not cap its output.** + +``consolidate()`` refuses to SEND a prompt over ``max_prompt_bytes`` +(pipeline/consolidate.py:315) — staging + recent.md + archive.md, default +600000. Past that it raises and nothing is written. But between ``call_haiku`` +returning and the ``cp``, no byte count is taken: ``cmd_consolidate`` writes +``result.recent`` to a temp file verbatim (pipeline/shell.py:517-523) and the +shell copies it over recent.md. ``capture_output=True`` on the CLI subprocess +(pipeline/haiku.py:686) is bounded by a wall clock, not by bytes, so the size +of that response is not a quantity this pipeline has ever measured. + +The two halves of the defect are one mechanism, and the second half is what +makes it permanent: + +1. **A single response of any size is written.** Not "a few KB per session" — + one round, arbitrarily many bytes, straight into the permanent record. +2. **That same write disables the only thing that could repair it.** The cap + is measured on the INPUT, and recent.md is part of the input. So the round + after an oversized write assembles an oversized prompt, raises + ``ConsolidationTooLarge``, and skips. Forever. ``_rotate_archive`` is no + escape: it can only shrink archive.md, and the bulk here is recent.md. + The file cannot grow again and cannot shrink either — frozen at its worst + size, which is precisely "only ever gets appended to" as a user sees it. + +There is a second, much slower grower, and it is deliberately NOT this bug: +consolidation is told to keep recent under 600 tokens but nothing enforces it, +so a model that faithfully re-emits the file plus one day per round grows it +monotonically — measured at ~2 KB/round, reaching the 600000 cap after ~298 +rounds and then freezing. That one is bounded by the cap by construction and +cannot reach a gigabyte. Distinguishing them matters: capping the output is +what closes the multi-GB path; the slow path is a compression-compliance +question, filed separately. + +The last defence has to hold on a store that is ALREADY broken, because the +fix above does nothing for the 6.4 GB file the reporter already had. Two +readers walk into it unguarded: the SessionStart hook `cat`s the file into +every session (which is what froze `claude` and took iTerm2 to ~56 GB), and +``cmd_consolidate`` reads the whole file into memory and assembles a prompt +around it BEFORE the cap gets to say it is too large — spending several times +the file's size in RAM to discover it should not have been read. + +Direction of every guard here: refusing is non-destructive. A skipped +consolidation leaves staging and memory intact and the next run retries; a +memory file named instead of injected stays on disk and greppable, and the +store already has the vocabulary for that (#124, rotated archives). Writing +an unbounded response, or reading one into a hook, is what costs the machine. +""" + +from __future__ import annotations + +import io +import json +import os +import subprocess +import sys +import time +from contextlib import redirect_stdout +from pathlib import Path +from unittest.mock import patch + +import pytest + +REPO_ROOT = Path(__file__).resolve().parent.parent +SESSION_START = REPO_ROOT / "scripts" / "session-start-hook.sh" + +sys.path.insert(0, str(REPO_ROOT)) + +from pipeline import consolidate as consolidate_mod +from pipeline import shell as shell_mod +from pipeline.consolidate import ConsolidationSkipped, consolidate +from pipeline.slug import session_dir_slug as _slug +from pipeline.types import HaikuResult, TokenUsage + +CAP = 600000 +STAGING = {"today-2026-01-01.md": "## 10:00 | main\n\nA day of work.\n"} +RECENT_BEFORE = "# Recent\n\n## 2025-12-31\n\nSmall and healthy.\n" +ARCHIVE_BEFORE = "# Archive\n\n## Week of 2025-12-22\n\nSmall and healthy.\n" + + +def _response(recent_body: str, archive_body: str = "fine") -> HaikuResult: + """A well-formed consolidation envelope. Only its SIZE is under test. + + Deliberately valid on every other axis: it carries the ``===RECENT===`` + envelope and a real ``## YYYY-MM-DD`` entry header, and none of the + template's instruction lines, so it passes ``_is_valid_consolidation`` and + the #202 echo guard. A test that leaned on those would pass against a + pipeline that still had no size guard at all. + """ + return HaikuResult( + text=(f"===RECENT===\n# Recent\n\n## 2026-01-01\n\n{recent_body}\n\n" + f"===ARCHIVE===\n# Archive\n\n## Week of 2025-12-29\n\n{archive_body}\n"), + is_skip=False, + is_rejected=False, + tokens=TokenUsage(input=0, output=0, cache=0, cost_usd=0.0), + ) + + +# ── 1. The write itself ──────────────────────────────────────────────────── + + +def test_oversized_response_is_refused_instead_of_written(): + """A response larger than the cap must not become the permanent record. + + The pipeline already knows this number: it is the same cap that made it + refuse to SEND a prompt this size one function call earlier. Accepting + back what it would not send is the whole defect. + """ + huge = _response("y" * (CAP * 3)) + + with patch.object(consolidate_mod, "call_haiku", lambda p, timeout=180: huge): + with pytest.raises(ConsolidationSkipped): + consolidate(STAGING, RECENT_BEFORE, ARCHIVE_BEFORE, max_prompt_bytes=CAP) + + +def test_oversized_archive_section_is_refused_too(): + """archive.md reached 1.8 GB in the report — both halves are written.""" + huge = _response("small", "z" * (CAP * 3)) + + with patch.object(consolidate_mod, "call_haiku", lambda p, timeout=180: huge): + with pytest.raises(ConsolidationSkipped): + consolidate(STAGING, RECENT_BEFORE, ARCHIVE_BEFORE, max_prompt_bytes=CAP) + + +def test_a_normal_response_is_still_written(): + """The guard must not be a cap on consolidation working at all. + + Paired with the two above on purpose: a size guard that rejected + everything would satisfy them and silently stop the plugin from ever + consolidating, which is a worse bug than the one being fixed. + """ + ok = _response("a compressed day, a few hundred bytes long") + + with patch.object(consolidate_mod, "call_haiku", lambda p, timeout=180: ok): + result = consolidate(STAGING, RECENT_BEFORE, ARCHIVE_BEFORE, max_prompt_bytes=CAP) + + assert result.recent.startswith("# Recent") + assert "a compressed day" in result.recent + + +def test_refusal_leaves_memory_and_staging_untouched(tmp_path): + """The refusal has to take the established non-destructive path. + + ``CONSOLIDATION_STATUS=skip`` with no ``RECENT_OUT`` is the contract + run-consolidation.sh reads: it means do not overwrite memory and do not + retire the staging files to ``.done.md``. A refusal that instead wrote an + empty file, or that let the shell retire staging, would lose the day it + was protecting. + """ + recent_f = tmp_path / "recent.md" + archive_f = tmp_path / "archive.md" + recent_f.write_text(RECENT_BEFORE, encoding="utf-8") + archive_f.write_text(ARCHIVE_BEFORE, encoding="utf-8") + (tmp_path / "today-2026-01-01.md").write_text("## 10:00 | main\n\nWork.\n", encoding="utf-8") + + huge = _response("y" * (CAP * 3)) + buf = io.StringIO() + with patch.object(consolidate_mod, "call_haiku", lambda p, timeout=180: huge): + with redirect_stdout(buf): + shell_mod.cmd_consolidate(str(tmp_path), str(recent_f), str(archive_f), CAP, "") + + out = buf.getvalue() + assert "CONSOLIDATION_STATUS=skip" in out, out + assert "RECENT_OUT=" not in out, out + assert recent_f.read_text(encoding="utf-8") == RECENT_BEFORE + assert archive_f.read_text(encoding="utf-8") == ARCHIVE_BEFORE + + +# ── 2. The ratchet: an already-oversized store must not be read whole ────── + + +def test_oversized_store_is_refused_by_size_not_by_reading_it(tmp_path): + """Discovering the store is too large must not cost the store's size in RAM. + + Today the order is: read recent.md whole, read archive.md whole, build a + prompt string around both, encode it to count bytes, and only then raise + ``ConsolidationTooLarge``. On the reporter's 6.4 GB file that is several + times 6.4 GB of allocation to reach a decision that ``os.path.getsize`` + answers for free — and it runs disowned in the background, next to a live + session, on a machine that then needed a restart. + + Asserted by watching the reads rather than the timing: the file is a real + file of the right size, and the test fails if anything opens it. + """ + recent_f = tmp_path / "recent.md" + archive_f = tmp_path / "archive.md" + # Sparse: the size is the point, the bytes are not. + with open(recent_f, "wb") as f: + f.truncate(CAP * 4) + archive_f.write_text(ARCHIVE_BEFORE, encoding="utf-8") + (tmp_path / "today-2026-01-01.md").write_text("## 10:00 | main\n\nWork.\n", encoding="utf-8") + + real_open = io.open + opened: list[str] = [] + + def spy(file, *a, **kw): + opened.append(str(file)) + return real_open(file, *a, **kw) + + called = [] + buf = io.StringIO() + with patch.object(consolidate_mod, "call_haiku", + lambda p, timeout=180: called.append(1) or _response("x")): + with patch("builtins.open", spy): + with redirect_stdout(buf): + shell_mod.cmd_consolidate(str(tmp_path), str(recent_f), str(archive_f), CAP, "") + + assert "CONSOLIDATION_STATUS=skip" in buf.getvalue(), buf.getvalue() + assert not called, "an oversized store must not reach the model call" + assert str(recent_f) not in opened, ( + "recent.md was read into memory to discover it is too large to read — " + f"opened: {opened}" + ) + + +# ── 3. The last defence: a broken store must not freeze the session ─────── + + +pytestmark_bash = pytest.mark.skipif( + sys.platform == "win32", + reason="bash hook subprocess + POSIX semantics — not portable to Windows runners", +) + +SESSION = "dddddddd-0000-4000-8000-000000000346" + + +@pytestmark_bash +def test_session_start_names_an_oversized_memory_file_instead_of_injecting_it(tmp_path): + """The symptom the reporter actually hit: every `claude` launch froze. + + A memory file past the injection threshold is named with its size and left + on disk, exactly as rotated archives already are (#124) — kept, greppable, + and not poured into a context window. Injecting it is what hung the launch + and took iTerm2 to ~56 GB, and no fix to the writer helps a store that is + already in that state. + """ + home = tmp_path / "home" + project = tmp_path / "project" + remember = project / ".remember" + (remember / "tmp").mkdir(parents=True) + (home / ".claude" / "projects" / _slug(str(project))).mkdir(parents=True) + + (remember / "now.md").write_text("NOW-BODY-346\n", encoding="utf-8") + # Big, and with a sentinel at the very front so a truncating implementation + # (rather than a refusing one) would still be caught by the size line. + with open(remember / "recent.md", "w", encoding="utf-8") as f: + f.write("RECENT-BODY-346\n") + f.write("q" * (CAP * 4)) + oversized_bytes = (remember / "recent.md").stat().st_size + + payload = json.dumps({ + "session_id": SESSION, + "transcript_path": f"/does/not/matter/{SESSION}.jsonl", + "hook_event_name": "SessionStart", + "source": "startup", + "cwd": "/does/not/matter", + }) + + proc = subprocess.run( + ["bash", str(SESSION_START)], + input=payload, + capture_output=True, + text=True, + timeout=120, + env={ + **os.environ, + "HOME": str(home), + "CLAUDE_PROJECT_DIR": str(project), + "CLAUDE_PLUGIN_ROOT": str(REPO_ROOT), + "REMEMBER_DIR": str(remember), + "_LIB_MEMORY_DIR_LOADED": "1", + }, + ) + + assert proc.returncode == 0, proc.stderr + out = proc.stdout + + assert "NOW-BODY-346" in out, "healthy memory files must still be injected" + assert "RECENT-BODY-346" not in out, ( + f"an oversized recent.md was cat'd into the session ({len(out)} bytes of hook output)" + ) + assert "recent.md" in out, "an oversized file must still be NAMED, not silently dropped" + assert str(oversized_bytes) in out, "the size must be reported so the state is diagnosable"