fix: cap the consolidation response, and stop reading a store too large to read (#346)

CHANGELOG.md

@@ -5,6 +5,26 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [Unreleased]
+
+### Fixed
+
+- **`recent.md` and `archive.md` could be written without bound, and the write that broke the store disabled the only thing that could repair it** ([#346](https://github.com/Digital-Process-Tools/claude-remember/issues/346)) — a reporter's store reached **6.4 GB** (`recent.md`) and **1.8 GB** (`archive.md`). The `SessionStart` hook `cat`s both into every session, so every `claude` launch in that project froze; on macOS iTerm2 reached ~56 GB and the machine needed a restart.
+
+  **The reported cause was that these two files have no rotation and are only ever appended to. Nothing appends to them.** The only writer in the tree is `cp "$RECENT_OUT" "$RECENT_FILE"` in `scripts/run-consolidation.sh`, which *replaces* the file wholesale with the consolidation's output, and it has done exactly that since the initial commit. The observation behind the report was still precisely right — the files only ever grew and never shrank — but the route there is the opposite of an append.
+
+  **Consolidation capped its input and not its output.** `consolidate()` refuses to *send* a prompt over `thresholds.consolidate_max_bytes` (staging + `recent.md` + `archive.md`, default 600000). Between the model call returning and the `cp`, no byte count was ever taken: `capture_output=True` bounds the CLI subprocess by a wall clock and not by bytes, so the size of a response was not a quantity anything downstream measured. `cmd_consolidate` wrote `result.recent` to a temp file verbatim and the shell copied it over `recent.md`.
+
+  **One oversized write is permanent, and that is what makes it this bug rather than one bad round.** `recent.md` is part of the input the cap is measured on. So the round after an oversized write assembles an oversized prompt, raises `ConsolidationTooLarge`, and skips — and so does every round after that, forever. Rotating `archive.md` ([#122](https://github.com/Digital-Process-Tools/claude-remember/issues/122)/[#123](https://github.com/Digital-Process-Tools/claude-remember/issues/123)) is no escape when the bulk is `recent.md`. The file can then never grow again and never shrink either, which from outside is indistinguishable from a file that is only ever appended to.
+
+  **The same number now caps both directions.** A response larger than `consolidate_max_bytes` is refused as non-conforming, which is the established non-destructive path ([#89](https://github.com/Digital-Process-Tools/claude-remember/issues/89)/[#202](https://github.com/Digital-Process-Tools/claude-remember/issues/202)): staging and memory are both left intact and the next run retries. Deliberately *not* `ConsolidationTooLarge` — that subclass means "the input was too big, shrink it and retry", and acting on it would spend another model call to be handed another oversized response while rotating away a healthy archive for nothing.
+
+  **The store is now sized before it is read.** The cap was enforced on the assembled prompt, so the whole store had to be read into memory and a prompt built around it before the pipeline was allowed to notice it was too large to send — several times the store's size in allocation to reach a decision `stat` answers for free, from a script that runs disowned beside a live session. That is the part that took the reporter's machine down. It cannot be a false skip: the prompt is the template plus per-file labels plus those bytes, so a sum already over the cap is proof the prompt would be. `archive.md` being the bulk still rotates, now before the read rather than after it, and an up-front rotation is undone on every path where the round does not go through.
+
+  **And a store that is *already* broken no longer freezes the session**, because none of the above helps the 6.4 GB file someone has on disk today. A memory file over `thresholds.memory_inject_max_bytes` (new, default 200000) is named with its size instead of being `cat`'d — [#124](https://github.com/Digital-Process-Tools/claude-remember/issues/124)'s "kept but not injected" vocabulary, reached by size instead of by filename. The bytes stay on disk and stay greppable; what stops is pouring them into a context window that cannot hold them. A session that starts and says the store is broken is worth more than one that hangs.
+
+  Two growth paths were found and only one is this bug. The other: consolidation is *told* to keep `recent.md` under 600 tokens and nothing enforces it, so a model that faithfully re-emits the file plus one day per round grows it monotonically — measured at ~2 KB/round, reaching the 600000 cap after ~298 rounds and then freezing. That one is bounded by the cap by construction and cannot reach a gigabyte; it is a compression-compliance question, filed separately.
+
 ## [0.19.0] — A compaction is not a new session
 
 ### Changed

README.md

@@ -422,7 +422,8 @@ Put cross-project preferences (timezone, cooldowns) in `~/.remember/config.json`
 | `prompt_stamp`                   | `full`           | What the `UserPromptSubmit` hook injects into the model's context. `full` — `[14:30 CEST — jack — 45%]`, unchanged from every previous release. `stable` — `[jack]` only: the clock and the context percentage both change between turns, and the percentage climbs on *every* prompt for anyone running the status line, so dropping only the clock would leave the line volatile. The `>= 95` context warning is **kept**, because it is threshold-gated and changes bytes only when it changes behaviour. `off` — nothing at all, warning included. An unrecognised value reads as `full` ([#301](https://github.com/Digital-Process-Tools/claude-remember/issues/301)). |
 | `model`                          | `haiku`          | Model used for the summarization / consolidation `claude -p` call. `REMEMBER_MODEL` overrides it. Documented as an env var only until #176, though `config.json` is the source of truth. |
 | `reject_pattern`                 | _(empty)_        | Overrides the reject-gate regex that keeps model refusals out of the memory layer. Empty → the narrow built-in default; `none` → gate off; anything else → a case-insensitive regex. An invalid regex falls back to the default. `REMEMBER_REJECT_PATTERN` overrides it. |
-| `thresholds.consolidate_max_bytes` | `600000`       | Max UTF-8 size of the staging content sent to the consolidation model. Read by `run-consolidation.sh`; documented in `config.example.json` but missing from this table until #176. |
+| `thresholds.consolidate_max_bytes` | `600000`       | Max UTF-8 size of the staging content sent to the consolidation model. Read by `run-consolidation.sh`; documented in `config.example.json` but missing from this table until #176. Since [#346](https://github.com/Digital-Process-Tools/claude-remember/issues/346) the same number also caps what consolidation may **write**: a response larger than this is refused rather than copied over `recent.md`/`archive.md`. |
+| `thresholds.memory_inject_max_bytes` | `200000`     | A memory file larger than this is **named with its size instead of injected** at session start ([#346](https://github.com/Digital-Process-Tools/claude-remember/issues/346)). It stays on disk and stays greppable; what stops is pouring it into a context window that cannot hold it — a multi-GB `recent.md` froze every `claude` launch in the reporter's project. A healthy memory file is kilobytes, so this only ever fires on a store that is already broken. Set `0` to disable. |
 | `debug`                          | _(unset)_        | Verbose logging for cooldowns and locks. Unset, each script keeps its own default — `save-session.sh` is verbose, the git-backup hook is quiet — which is what they did before this option was wired up (#176). `REMEMBER_DEBUG` overrides it.                                                                                                                                                                                                |
 | `haiku.oauth_token`              | _(empty)_        | OAuth token the plugin hands to the nested `claude -p` **only when the host did not put `CLAUDE_CODE_OAUTH_TOKEN` in the hook subprocess env** — some desktop / Agent-SDK hosts withhold it from spawned children, so `claude -p` is unauthenticated and nothing ever saves ([#129](https://github.com/Digital-Process-Tools/claude-remember/issues/129)/[#131](https://github.com/Digital-Process-Tools/claude-remember/issues/131)). Create one with `claude setup-token`. The plugin holds this credential and passes it to the summarization CLI, so set it deliberately. A host-provided token always wins; `REMEMBER_OAUTH_TOKEN` overrides this. A malformed value is refused and reported in the daily log, never passed to the CLI. |
 

config.example.json

@@ -11,7 +11,8 @@
     "max_summary_failures": 3,
     "delta_lines_trigger": 50,
     "extract_max_bytes": 300000,
-    "consolidate_max_bytes": 600000
+    "consolidate_max_bytes": 600000,
+    "memory_inject_max_bytes": 200000
   },
   "features": {
     "ndc_compression": true,
@@ -71,7 +72,21 @@
       "rather than overflowing Haiku's window with 'Prompt is too long' and",
       "stalling all saves. Unlike extract_max_bytes the input is NOT truncated,",
       "because consolidation rewrites recent/archive and truncating would drop",
-      "archived memory. Set 0 to disable the guard."
+      "archived memory. Set 0 to disable the guard.",
+      "Since #346 the same number caps the OUTPUT too: a model response larger",
+      "than this is refused instead of being written to recent.md/archive.md.",
+      "Capping only the input meant one oversized response became the permanent",
+      "record AND pushed the store past this cap, so every later run skipped and",
+      "the file could never shrink again."
+    ],
+    "memory_inject_max_bytes": [
+      "A memory file larger than this (UTF-8 bytes) is NAMED with its size at",
+      "session start instead of being cat'd into context. Default 200000.",
+      "The bytes stay on disk and stay greppable — what stops is pouring them",
+      "into a context window that cannot hold them. A healthy memory file is",
+      "kilobytes, so this only fires on a store that is already broken; in #346",
+      "a 6.4GB recent.md froze every `claude` launch in that project. Set 0 to",
+      "disable."
     ],
     "prompt_stamp": [
       "What the UserPromptSubmit hook injects into the model's context.",

pipeline/consolidate.py

@@ -333,6 +333,43 @@ def consolidate(
             "(SKIP or missing ===RECENT===/entry headers)"
         )
 
+    # The cap above is the size this pipeline refuses to SEND. Refusing to
+    # WRITE more than that is the same number in the other direction, and the
+    # other direction is where #346 came from: ``capture_output=True`` bounds
+    # the CLI subprocess by a wall clock and not by bytes, so the size of a
+    # response was never a quantity anything downstream measured.
+    # ``cmd_consolidate`` writes ``result.recent`` to a temp file verbatim and
+    # run-consolidation.sh copies it over recent.md — an overwrite, never an
+    # append, and unbounded either way.
+    #
+    # One oversized write is permanent, which is what makes this the bug
+    # rather than one bad round. recent.md is part of the input the cap is
+    # measured on, so the round after an oversized write assembles an
+    # oversized prompt and skips, and so does every round after that.
+    # Rotating the archive is no escape when the bulk is recent.md. The file
+    # can then never grow again and never shrink either — which from outside
+    # is indistinguishable from a file that is only ever appended to, and is
+    # exactly how the reporter read it.
+    #
+    # Deliberately NOT ConsolidationTooLarge: that subclass means "the input
+    # was too big, shrink it and retry", and the caller acts on it by rotating
+    # archive.md. Nothing about the input was wrong here, so a retry would
+    # spend another model call to be handed another oversized response and
+    # would rotate away a healthy archive for nothing.
+    #
+    # Skipping is the established non-destructive direction (see
+    # ``_echoes_the_prompt``): staging and memory are both left intact and the
+    # next run retries, so a false positive costs one round and a false
+    # negative costs the permanent record.
+    if max_prompt_bytes > 0:
+        response_bytes = len(result.text.encode("utf-8"))
+        if response_bytes > max_prompt_bytes:
+            raise ConsolidationSkipped(
+                f"consolidation response too large ({response_bytes} bytes > "
+                f"{max_prompt_bytes} cap) -- refusing to write it to memory; "
+                f"staging + memory left untouched"
+            )
+
     recent_new, archive_new = parse_consolidation_response(result.text)
 
     return ConsolidationResult(

pipeline/shell.py

@@ -461,6 +461,56 @@ def cmd_consolidate(staging_dir: str, recent_file: str, archive_file: str,
         print("STAGING_COUNT=0")
         return
 
+    def _emit_skip() -> None:
+        # Skip status so the shell leaves recent.md/archive.md untouched and does
+        # NOT rename the source staging files to .done.md — they remain available
+        # for the next run. STAGING_COUNT is non-zero (we found files) but the
+        # shell gates on CONSOLIDATION_STATUS.
+        print(f"STAGING_COUNT={len(staging_contents)}")
+        print("CONSOLIDATION_STATUS=skip")
+
+    # Size the store before reading it (#346). The cap is enforced on the
+    # assembled prompt, so the whole store had to be read into memory and a
+    # prompt built around it before the pipeline was allowed to notice it was
+    # too large to send: several times the store's size in allocation to reach
+    # a decision ``stat`` answers for free. Against the reporter's 6.4 GB
+    # recent.md that is what took the machine down, from a script that runs
+    # disowned beside a live session.
+    #
+    # It can never be a false skip. The assembled prompt is the template plus
+    # per-file labels plus these bytes, so it is strictly larger than their
+    # sum, and a sum already over the cap is proof the prompt would be.
+    rotated: str | None = None
+
+    def _restore_rotation() -> None:
+        """Undo an up-front rotation when the round did not go through.
+
+        Existence-checked because the handlers inside ``ConsolidationTooLarge``
+        below do their own restore and then re-raise; an exception raised
+        inside an except clause does not re-enter its siblings, but the guard
+        keeps that a property of this function rather than of Python's
+        control flow.
+        """
+        if rotated is not None and os.path.exists(rotated):
+            os.replace(rotated, archive_file)
+
+    recent_size = os.path.getsize(recent_file) if os.path.exists(recent_file) else 0
+    archive_size = os.path.getsize(archive_file) if os.path.exists(archive_file) else 0
+    if max_prompt_bytes > 0:
+        embedded = sum(staging_raw_bytes.values()) + recent_size + archive_size
+        if embedded > max_prompt_bytes:
+            # Same recovery ConsolidationTooLarge gets below, taken before the
+            # read rather than after it: if archive.md is the bulk, rotate it
+            # to a dated sibling and carry on with a fresh one. Only when
+            # dropping it still would not fit — recent.md is the bulk, #346's
+            # shape — is nothing read at all.
+            if embedded - archive_size <= max_prompt_bytes:
+                rotated = _rotate_archive(archive_file)
+            if rotated is None:
+                _emit_skip()
+                return
+            archive_size = 0
+
     recent = ""
     if os.path.exists(recent_file):
         with open(recent_file, encoding="utf-8", errors="replace") as f:
@@ -471,14 +521,6 @@ def cmd_consolidate(staging_dir: str, recent_file: str, archive_file: str,
         with open(archive_file, encoding="utf-8", errors="replace") as f:
             archive = f.read()
 
-    def _emit_skip() -> None:
-        # Skip status so the shell leaves recent.md/archive.md untouched and does
-        # NOT rename the source staging files to .done.md — they remain available
-        # for the next run. STAGING_COUNT is non-zero (we found files) but the
-        # shell gates on CONSOLIDATION_STATUS.
-        print(f"STAGING_COUNT={len(staging_contents)}")
-        print("CONSOLIDATION_STATUS=skip")
-
     try:
         result = consolidate(staging_contents, recent, archive,
                              max_prompt_bytes=max_prompt_bytes)
@@ -488,7 +530,12 @@ def _emit_skip() -> None:
         # empty archive, so consolidation keeps progressing instead of skipping
         # every run forever. If there is nothing to rotate, or the retry still
         # overflows (staging + recent alone exceed the cap), restore and skip.
-        rotated = _rotate_archive(archive_file)
+        # Only reachable when the stat guard above let the round through and
+        # the template plus per-file labels tipped it over, or when the guard
+        # is disabled. An up-front rotation has already happened in the first
+        # case, so do not rotate a second time and orphan the first sibling.
+        if rotated is None:
+            rotated = _rotate_archive(archive_file)
         if rotated is None:
             _emit_skip()
             return
@@ -505,13 +552,25 @@ def _emit_skip() -> None:
     except SummarizerSpawnDeclined as declined:
         # The spawn guard refused (#204). Staging is left exactly as it is and
         # the next run consolidates it — not a skip, which retires staging, and
-        # not a failure either.
+        # not a failure either. An up-front rotation is undone for the same
+        # reason: this round never happened, so nothing it moved may persist.
+        _restore_rotation()
         print(f"consolidate declined: {declined}", file=sys.stderr)
         sys.exit(EXIT_SPAWN_DECLINED)
     except ConsolidationSkipped:
-        # Model declined (SKIP) or returned non-conforming output.
+        # Model declined (SKIP), returned non-conforming output, or returned
+        # more bytes than the pipeline is willing to write (#346).
+        _restore_rotation()
         _emit_skip()
         return
+    except Exception:
+        # A transient failure (the model call erroring, say) must not leave the
+        # up-front rotation applied: nothing was consolidated, so archive.md
+        # has to be where the next run expects it. The post-hoc rotation path
+        # has restored itself on this branch since #123; the pre-read one owes
+        # the same guarantee.
+        _restore_rotation()
+        raise
 
     # Write results to temp files
     fd_r, recent_out = tempfile.mkstemp(prefix="remember-recent-", suffix=".md")