v0.11.1 — dynamic tool guide and memory store caps

docs/DESIGN.md

@@ -553,7 +553,7 @@ The runtime should expose `/compact status` so the user can see current context
 
 #### ContextAssembler
 
-The system prompt is assembled from a fixed, ordered set of blocks — base prompt, behavior instructions, memory, the conditional skills group (a `skills index` block plus per-skill bodies and triggered references, section 23), and (when a plan is live) a compact plan-state block. A pure `ContextAssembler` (no file or model I/O) captures that assembly as a structured `ContextSnapshot` of `ContextBlock`s, each carrying a block name, source, token estimate, an `injected` flag, and an optional skip reason. The snapshot is the single source of truth: the same structure produces both the live model prompt (joining injected blocks with `\n\n`) and the `/context` breakdown, so the figures shown to the user cannot drift from what the model receives. Block order is load-bearing; the base prompt is always present, while behavior, memory, skills, and plan state are injected only when their trigger or non-empty condition holds.
+The system prompt is assembled from a fixed, ordered set of blocks — base prompt, dynamic tool guide, behavior instructions, memory, the conditional skills group (a `skills index` block plus per-skill bodies and triggered references, section 23), and (when a plan is live) a compact plan-state block. A pure `ContextAssembler` (no file or model I/O) captures that assembly as a structured `ContextSnapshot` of `ContextBlock`s, each carrying a block name, source, token estimate, an `injected` flag, and an optional skip reason. The snapshot is the single source of truth: the same structure produces both the live model prompt (joining injected blocks with `\n\n`) and the `/context` breakdown, so the figures shown to the user cannot drift from what the model receives. Block order is load-bearing; the base prompt is always present, while the tool guide, behavior, memory, skills, and plan state are injected only when their trigger or non-empty condition holds.
 
 Skill activation uses an explicit `TriggerContext(plan_status, web_enabled, enabled)` rendered by the runtime. `plan_status` is the live plan status when a plan sidecar exists, `web_enabled` is true only when both `web_search` and `web_fetch` are registered in the `ToolRegistry`, and `enabled` is the `[skills] enabled` tuple. This keeps boot-only settings drift from accidentally activating web guidance.
 
@@ -1361,8 +1361,9 @@ Implemented in v0.3.0 (settled 2026-06-11) following this section's design, with
 
 - Stores: global `memory.json` in the user config dir, project `.shellpilot/memory.json` with `project_id`. Versioned schema, explicit validation (unknown versions rejected), atomic writes, secrets redacted before disk.
 - Facts carry an `id` (e.g. `fact_001`) — an addition to the 16.5 example schema — so `/memory forget` can address them.
-- Tools: `memory_read` (read-only, auto) and `memory_propose_update` (the model's only write path; MEDIUM risk, approval required in every profile, with a diff-style preview). The runtime injects a budget-capped Memory block into the system prompt each turn.
-- `/memory compact` optimization (16.4) covers preferences; facts are structured and excluded for now. A deterministic guard rejects any optimization that drops or invents ids, or drops a `source: user` entry — regardless of what the model returns.
+- Tools: `memory_read` (read-only, auto) and `memory_propose_update` (the model's only write path; MEDIUM risk, approval required in every profile, with a diff-style preview). The runtime injects a budget-capped Memory block into the system prompt each turn, grouping preferences into global and project sections so project preferences can override global ones inside the workspace.
+- Memory store files are capped at 1800 serialized characters per file (global and project separately). Writes that would exceed the cap are rejected with a clear instruction to forget or compact existing entries, forcing stored memory to stay selective rather than relying on prompt truncation. Legacy over-cap files can still shrink: the cap ratchets against normalized current payload size and rejects growth until the store complies.
+- `/memory compact` optimization (16.4) covers preferences; facts are structured and excluded for now. A deterministic guard rejects any optimization that drops or invents ids, or drops a `source: user` entry — regardless of what the model returns. Replacements are preflighted for every store before any file is written, so a rejected compact cannot partially mutate global or project memory.
 - AGENTS.md remains read-only and is loaded alongside memory; `trusted-local` auto-save (16.3) remains moot until that profile exists.
 
 V1 behavior instructions were static AGENTS.md only; that path still works unchanged.
@@ -1910,7 +1911,8 @@ Prompt principles:
 
 - **Proposal-time rules live in the base prompt** (`shellpilot/prompts/system.py`, `_BASE`): call `propose_plan` once for real multi-step work (3+ distinct steps), include all known setup/work, do not plan trivial one-step command/edit/inspection tasks, and never write plans as prose. The base prompt keeps a single bridge sentence ("After a plan is approved, keep working in this same turn…") and carries no `update_plan` mechanics. The base prompt is always present.
 - **Plan-mode discipline is the builtin `planning` skill** (`shellpilot/skills/builtin/planning/`): `SKILL.md` is only a tiny harness-managed-plan preamble, and mode-specific guidance lives in `references/proposed.md`, `references/active.md`, and `references/blocked.md`. The loader assigns those three references to `PLAN_PROPOSED`, `PLAN_ACTIVE`, and `PLAN_BLOCKED` by filename convention; templates are discovered metadata only.
-- **Rationale.** The 8K-context target model pays for every system-prompt token on every turn, and system blocks are never compacted (section 20.2). `update_plan` mechanics are dead weight before an active/blocked plan, while expanded proposal guidance is dead weight outside proposal mode. Selecting planning references by exact plan status keeps plan-free turns lean while guaranteeing the relevant discipline is present exactly when it can be acted on. The split is enforced by tests: the base prompt must contain compact proposal rules and no `update_plan`; planning references must carry the mode mechanics. `PROMPT_VERSION` is bumped to 4 for the Skills v2 builtin-resource layout.
+- **Dynamic tool guide (v0.11.1).** A compact `Tool guide` block is rendered from the same profile-filtered registry definitions offered to the model. Optional tools (`memory_*`, `web_*`, `skill_read`) appear only when registered for the session. `update_plan` is omitted on plan-free/proposed turns and appears only when a plan is active or blocked, preserving the proposal-vs-execution split.
+- **Rationale.** The 8K-context target model pays for every system-prompt token on every turn, and system blocks are never compacted (section 20.2). `update_plan` mechanics are dead weight before an active/blocked plan, while expanded proposal guidance is dead weight outside proposal mode. Selecting planning references by exact plan status keeps plan-free turns lean while guaranteeing the relevant discipline is present exactly when it can be acted on. The split is enforced by tests: the base prompt must contain compact proposal rules and no `update_plan`; planning references must carry the mode mechanics. The dynamic guide bumps `PROMPT_VERSION` to 6.
 
 ### 19.1 Unified System Prompt Themes
 

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "shellpilot"
-version = "0.11.0"
+version = "0.11.1"
 description = "A local-first AI shell harness for your terminal, powered by Ollama."
 readme = "README.md"
 requires-python = ">=3.11"

shellpilot/__init__.py

@@ -1,3 +1,3 @@
 """ShellPilot: a local-first AI shell harness."""
 
-__version__ = "0.11.0"
+__version__ = "0.11.1"

shellpilot/cli/slash.py

@@ -34,6 +34,7 @@
 )
 from shellpilot.config.overrides import load_overrides, overrides_path, save_overrides
 from shellpilot.llm.client import LLMClient
+from shellpilot.memory.store import MemoryFormatError
 from shellpilot.runtime.conversation import ConversationRuntime
 from shellpilot.skills.model import SkillTrigger
 
@@ -851,7 +852,13 @@ def _memory(self, args: list[str]) -> None:
                 self._console.print("Usage: /memory add <preference text>")
                 return
             if self._confirm(f'Add global preference "{text}"?'):
-                preference = stores.global_store.add_preference(text, scope="global", source="user")
+                try:
+                    preference = stores.global_store.add_preference(
+                        text, scope="global", source="user"
+                    )
+                except MemoryFormatError as exc:
+                    self._console.print(f"[red]Memory update rejected:[/red] {exc}")
+                    return
                 self._audit_memory(f"add {preference.id}")
                 self._console.print(f"Saved {preference.id}.")
             return
@@ -865,7 +872,11 @@ def _memory(self, args: list[str]) -> None:
                 self._console.print(f"[red]No memory entry {entry_id}.[/red] See /memory show.")
                 return
             if self._confirm(f"Forget {entry_id}?"):
-                store.remove(entry_id)
+                try:
+                    store.remove(entry_id)
+                except MemoryFormatError as exc:
+                    self._console.print(f"[red]Memory update rejected:[/red] {exc}")
+                    return
                 self._audit_memory(f"forget {entry_id}")
                 self._console.print(f"Removed {entry_id}.")
             return
@@ -942,7 +953,8 @@ def _memory_compact(self, stores: object) -> None:
         kept = len(final_ids)
         if not self._confirm(f"Apply optimization: {len(all_preferences)} preferences -> {kept}?"):
             return
-        for name, store in by_store.items():
+        replacements: dict[str, list[Preference]] = {}
+        for name in by_store:
             new_preferences = []
             for entry in final_entries:
                 if not isinstance(entry, dict):
@@ -961,7 +973,15 @@ def _memory_compact(self, stores: object) -> None:
                         updated_at=original.updated_at,
                     )
                 )
-            store.replace_all(new_preferences, list(store.facts))
+            replacements[name] = new_preferences
+        try:
+            for name, store in by_store.items():
+                store.validate_replacement(replacements[name], list(store.facts))
+        except MemoryFormatError as exc:
+            self._console.print(f"[red]Optimization rejected:[/red] {exc}")
+            return
+        for name, store in by_store.items():
+            store.replace_all(replacements[name], list(store.facts))
         self._audit_memory(f"compact {len(all_preferences)} -> {kept}")
         self._console.print(f"Memory compacted: {len(all_preferences)} -> {kept} preferences.")
 

shellpilot/memory/store.py

@@ -22,6 +22,7 @@
 from shellpilot.runtime.budget import truncate_to_tokens
 
 SCHEMA_VERSION = 1
+MAX_MEMORY_FILE_CHARS = 1800
 VALID_SCOPES = ("global", "project")
 VALID_CONFIDENCE = ("observed", "stated", "inferred")
 
@@ -141,6 +142,54 @@ def _next_id(prefix: str, existing: list[str]) -> str:
     def _clean(self, text: str) -> str:
         return redact_secrets(text) if self._redact else text
 
+    def _payload(
+        self,
+        preferences: list[Preference] | None = None,
+        facts: list[ProjectFact] | None = None,
+    ) -> dict[str, Any]:
+        stored_preferences = self._preferences if preferences is None else preferences
+        stored_facts = self._facts if facts is None else facts
+        payload: dict[str, Any] = {
+            "version": SCHEMA_VERSION,
+            "preferences": [asdict(p) for p in stored_preferences],
+            "facts": [asdict(f) for f in stored_facts],
+        }
+        if self.project_id is not None:
+            payload["project_id"] = self.project_id
+        return payload
+
+    def _payload_text(self, payload: dict[str, Any]) -> str:
+        return json.dumps(payload, indent=2, sort_keys=True) + "\n"
+
+    def _current_payload_size(self) -> int:
+        return len(self._payload_text(self._payload()))
+
+    def _ensure_within_file_cap(self, payload: dict[str, Any]) -> None:
+        size = len(self._payload_text(payload))
+        current_size = self._current_payload_size()
+        if size > MAX_MEMORY_FILE_CHARS and size <= current_size:
+            return
+        if size > MAX_MEMORY_FILE_CHARS:
+            raise MemoryFormatError(
+                f"{self.path}: memory file would be {size} characters; "
+                f"limit is {MAX_MEMORY_FILE_CHARS} characters. "
+                "Forget or compact existing memories before adding more."
+            )
+
+    def validate_replacement(self, preferences: list[Preference], facts: list[ProjectFact]) -> None:
+        """Raise if replacing this store would violate the memory file cap."""
+        self._ensure_within_file_cap(self._payload(preferences=preferences, facts=facts))
+
+    def _save_payload(
+        self,
+        *,
+        preferences: list[Preference] | None = None,
+        facts: list[ProjectFact] | None = None,
+    ) -> None:
+        payload = self._payload(preferences=preferences, facts=facts)
+        self._ensure_within_file_cap(payload)
+        atomic_write_json(self.path, payload)
+
     def add_preference(self, text: str, *, scope: str, source: str) -> Preference:
         if scope not in VALID_SCOPES:
             raise MemoryFormatError(f"invalid scope {scope!r}; use one of {VALID_SCOPES}")
@@ -151,8 +200,9 @@ def add_preference(self, text: str, *, scope: str, source: str) -> Preference:
             source=source,
             updated_at=_now_iso(),
         )
-        self._preferences.append(preference)
-        self.save()
+        preferences = [*self._preferences, preference]
+        self._save_payload(preferences=preferences)
+        self._preferences = preferences
         return preference
 
     def add_fact(
@@ -176,34 +226,33 @@ def add_fact(
             confidence=confidence,
             source=source,
         )
-        self._facts.append(fact)
-        self.save()
+        facts = [*self._facts, fact]
+        self._save_payload(facts=facts)
+        self._facts = facts
         return fact
 
     def remove(self, entry_id: str) -> bool:
         before = len(self._preferences) + len(self._facts)
-        self._preferences = [p for p in self._preferences if p.id != entry_id]
-        self._facts = [f for f in self._facts if f.id != entry_id]
-        if len(self._preferences) + len(self._facts) == before:
+        preferences = [p for p in self._preferences if p.id != entry_id]
+        facts = [f for f in self._facts if f.id != entry_id]
+        if len(preferences) + len(facts) == before:
             return False
-        self.save()
+        self._save_payload(preferences=preferences, facts=facts)
+        self._preferences = preferences
+        self._facts = facts
         return True
 
     def replace_all(self, preferences: list[Preference], facts: list[ProjectFact]) -> None:
         """Wholesale replacement used by /memory compact after approval."""
-        self._preferences = list(preferences)
-        self._facts = list(facts)
-        self.save()
+        next_preferences = list(preferences)
+        next_facts = list(facts)
+        self.validate_replacement(next_preferences, next_facts)
+        atomic_write_json(self.path, self._payload(preferences=next_preferences, facts=next_facts))
+        self._preferences = next_preferences
+        self._facts = next_facts
 
     def save(self) -> None:
-        payload: dict[str, Any] = {
-            "version": SCHEMA_VERSION,
-            "preferences": [asdict(p) for p in self._preferences],
-            "facts": [asdict(f) for f in self._facts],
-        }
-        if self.project_id is not None:
-            payload["project_id"] = self.project_id
-        atomic_write_json(self.path, payload)
+        self._save_payload()
 
 
 @dataclass(frozen=True)
@@ -216,18 +265,29 @@ class MemoryStores:
     def render(self, max_tokens: int, *, meta: bool = False) -> str:
         """Render the memory block. ``meta`` annotates each preference with its
         (scope, source) for the ``/memory show`` view; the default is the
-        injected prompt format and must stay byte-identical."""
-        preferences = list(self.global_store.preferences) + list(self.project_store.preferences)
+        injected prompt format, grouped by global/project scope."""
+        global_preferences = list(self.global_store.preferences)
+        project_preferences = list(self.project_store.preferences)
         facts = list(self.global_store.facts) + list(self.project_store.facts)
-        if not preferences and not facts:
+        if not global_preferences and not project_preferences and not facts:
             return ""
         lines = ["## Memory"]
-        if preferences:
-            lines.append("Preferences:")
+        if global_preferences:
+            lines.append("Global preferences:")
+            if meta:
+                lines.extend(
+                    f"- [{p.id}] ({p.scope}, {p.source}) {p.text}" for p in global_preferences
+                )
+            else:
+                lines.extend(f"- [{p.id}] {p.text}" for p in global_preferences)
+        if project_preferences:
+            lines.append("Project preferences:")
             if meta:
-                lines.extend(f"- [{p.id}] ({p.scope}, {p.source}) {p.text}" for p in preferences)
+                lines.extend(
+                    f"- [{p.id}] ({p.scope}, {p.source}) {p.text}" for p in project_preferences
+                )
             else:
-                lines.extend(f"- [{p.id}] {p.text}" for p in preferences)
+                lines.extend(f"- [{p.id}] {p.text}" for p in project_preferences)
         if facts:
             lines.append("Project facts:")
             lines.extend(f"- [{f.id}] ({f.kind}) {f.label}: {f.value}" for f in facts)