From caa73a081f5cee9594c4983620bd51139a08b96b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fran=C3=A7ois=20Chastel?= Date: Mon, 8 Jun 2026 15:37:29 -0400 Subject: [PATCH 1/5] refactor: split state ops to `?:`, add sticky session to `,` MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ? had grown 12 flags that mixed two ideas (asking and managing state). The state ops now live behind a single meta verb so the asking commands stay ask-only. New surface - `?:` — long-term facts + cross-session recall - `?: add ` / `list` / `drop ` / `recall ` / `status` - `,` is now session-aware (cmd="comma"): follow-ups refine the prior proposal instead of asking from scratch. Same TTL/archive hookup as `?` / `???`, so expired `,` sessions are searchable via `?: recall`. Removed from `?` / `???` - --remember, --memories, --forget, --recall, --mem, --no-mem Deprecation - Moved flags print a one-line redirect pointing at the `?:` equivalent so muscle memory fails loudly, not silently. Surfaces that stay - `? --new / --reset / --history / --compact / --auto-recall / --help` - `, --new / --reset / --history / --help` - `??` and its `--start-embed` extension unchanged 168 tests (was 144), ruff clean. --- README.md | 75 ++++++++------ pyproject.toml | 1 + src/shellllm/ask.py | 127 ++++++++---------------- src/shellllm/comma.py | 177 ++++++++++++++++++++++++++++++--- src/shellllm/state.py | 168 ++++++++++++++++++++++++++++++++ tests/test_comma_session.py | 189 ++++++++++++++++++++++++++++++++++++ tests/test_state.py | 127 ++++++++++++++++++++++++ zsh/shellllm.zsh | 18 ++++ 8 files changed, 747 insertions(+), 135 deletions(-) create mode 100644 src/shellllm/state.py create mode 100644 tests/test_comma_session.py create mode 100644 tests/test_state.py diff --git a/README.md b/README.md index 485107e..765349b 100644 --- a/README.md +++ b/README.md @@ -6,9 +6,10 @@ Local-LLM zsh helpers. -- **`, `** — proposes 3–5 shell commands with one-line notes, you pick one in `fzf`, it lands on your prompt line via `print -z`. Never auto-executes. -- **`? `** — small read-only agent with three tools: `read_file` (gated by a filesystem hard wall), `web_search` (DuckDuckGo) and `fetch_url` (follow a result into its page, plain-text). Searches only when the model decides it needs to. Answer streams as live-rendered markdown. Each terminal pane keeps its own sticky conversation — follow-ups continue automatically until 30 min of idle (or `? --new`). `? --remember ` pins long-term facts, `? --recall ` searches across past sessions. +- **`, `** — proposes 3–5 shell commands with one-line notes, you pick one in `fzf`, it lands on your prompt line via `print -z`. Never auto-executes. Sticky per-pane session so follow-ups refine the prior list (`, the same but only the running ones`). +- **`? `** — small read-only agent with three tools: `read_file` (gated by a filesystem hard wall), `web_search` (DuckDuckGo) and `fetch_url` (follow a result into its page, plain-text). Searches only when the model decides it needs to. Answer streams as live-rendered markdown. Each terminal pane keeps its own sticky conversation — follow-ups continue automatically until 30 min of idle (or `? --new`). - **`??? `** — same agent, web-first: always starts with a `web_search` and follows the best link with `fetch_url`. Use it when you want fresh facts, not the model's prior. Has its own per-pane session, distinct from `?`. +- **`?: `** — long-term facts and cross-session recall. `?: add `, `?: list`, `?: drop `, `?: recall `, `?: status`. Lives outside the asking commands so `?` and `,` stay ask-only. - **`??`** — start (or stop / list / status) the local `llama-server` backend, with named tiers for speed-vs-quality. `?? --start-embed` boots a second `llama-server` in embedding mode for hybrid semantic recall. Runs against a local `llama-server`. No frontier model, no API key, works with wifi off. @@ -44,9 +45,10 @@ exec zsh # 4. use it , find the five largest files under this directory +, the same but only ones modified today # refines the prior , — sticky session ? in markdown, what does git stash do? -? --remember I prefer ripgrep over grep -? --recall ripgrep # search past sessions +?: add I prefer ripgrep over grep # long-term fact, used by all asks +?: recall ripgrep # search past sessions across panes ??? latest stable release of ripgrep and one notable change in it ``` @@ -83,9 +85,11 @@ huggingface-cli download unsloth/Qwen3-Coder-Next-GGUF `??` resolves the GGUF inside your HuggingFace cache automatically — no path config required. -## Multi-turn +## Sessions -Each terminal pane gets its own sticky conversation, one per command: +Each terminal pane gets its own sticky conversation, one per asking +command. `?`, `???`, and `,` each keep their own thread so a refining +`,` doesn't pollute the Q&A you were having with `?`. ```sh ? what was that flag for ripgrep again @@ -95,9 +99,9 @@ Each terminal pane gets its own sticky conversation, one per command: ? --reset # drop the current session ? --compact # force-compact older turns into a summary -? --remember "I prefer ripgrep over grep" # save a long-term fact -? --memories # list saved facts -? --forget 2 # drop fact #2 +, list all docker containers +, the same but only the running ones # refines the prior `,` proposal +, --new find the largest files # starts a fresh `,` thread ``` The pane is identified from `TERM_SESSION_ID` (Terminal.app / iTerm), @@ -105,25 +109,31 @@ The pane is identified from `TERM_SESSION_ID` (Terminal.app / iTerm), idle the session auto-rotates so a forgotten tab doesn't bleed stale context into the next turn. -`?` and `???` keep **separate** threads in the same pane, so a web-first -search doesn't mix with a local-knowledge answer. The long-term memory -(`--remember`) is global — facts apply everywhere. - When the conversation crosses ~80% of `SHELLLM_CTX`, older turns are auto-summarized into a single `` block using the same local model; the most recent 4 turns stay verbatim. -## Cross-session recall +## `?:` — facts and recall -Every expiring or `--new`'d session is flattened into a sqlite archive -(`~/.cache/shellllm/archive.db`) so you can search across all your -past panes and days: +Everything that *isn't* "ask a question" or "propose a command" lives +under one meta verb so `?` and `,` stay clean. `?:` is your durable +layer. ```sh -? --recall ripgrep # BM25 search across archived transcripts -? --auto-recall what was that grep flag again # inject top hits as context this turn +?: add I prefer ripgrep over grep # pin a long-term fact +?: list # see them +?: drop 2 # remove fact #2 +?: recall ripgrep # search archived sessions +?: status # counts: facts + archives +?: help ``` +Long-term facts get injected at the top of every `?` / `???` system +prompt, so the agent stops asking you things you've already told it. +Recall works against `~/.cache/shellllm/archive.db`, populated +automatically whenever a session expires or you call `--new` / `--reset` +on any asking command. + Recall always works in **BM25-only mode** — no extra setup, no extra processes. Adding a local embedding server unlocks **hybrid semantic + BM25 search** (RRF-fused) so you find prior conversations @@ -155,7 +165,7 @@ shellllm auto-detects the server via `SHELLLM_EMBED_URL` - new archive rows get a normalized fp32 embedding written alongside the FTS5 entry; -- `--recall` and `--auto-recall` embed the query and add cosine-sim +- `?: recall` and `? --auto-recall` embed the query and add cosine-sim candidates to the BM25 results, fused via Reciprocal Rank Fusion; - mismatched embedding dims (e.g. swapping the model later) are silently skipped — old rows still serve BM25 hits. @@ -215,22 +225,25 @@ src/shellllm/ ├── comma.py , — JSON-schema → fzf picker → stdout ├── ask.py ? — streaming agent loop, live markdown render, CLI dispatch ├── search.py ??? — same loop, web-search-first system prompt +├── state.py ?: — long-term facts + cross-session recall subcommands ├── session.py per-pane conversation persistence (JSONL + idle TTL) -├── memory.py long-term facts behind `--remember` / `--memories` +├── memory.py long-term fact store backing `?: add` / `?: list` ├── compact.py summary-buffer compaction over the same local model ├── context.py date/OS/timezone prelude (re-injected on PWD/date change) -├── archive.py sqlite FTS5 + optional embeddings for cross-session --recall +├── archive.py sqlite FTS5 + optional embeddings for `?: recall` ├── embed.py client for a local llama-server in --embedding mode ├── claude_mem.py optional adapter for claude-mem server-beta (observations + context) └── web.py stdlib DuckDuckGo scraper + fetch_url with SSRF guard -tests/test_safe_fs.py filesystem-wall coverage -tests/test_session.py TTY id + TTL rotation + JSONL round-trip -tests/test_memory.py fact store + size cap + archive overflow -tests/test_compact.py compaction preserves turn boundaries -tests/test_archive.py FTS5 + cosine recall, RRF fusion, dim-mismatch tolerance -tests/test_embed.py embedding client + pack/unpack + cosine helpers -tests/test_claude_mem.py adapter gating + payload shape + error swallowing -tests/test_web.py URL safety + HTML extraction +tests/test_safe_fs.py filesystem-wall coverage +tests/test_session.py TTY id + TTL rotation + JSONL round-trip +tests/test_memory.py fact store + size cap + archive overflow +tests/test_compact.py compaction preserves turn boundaries +tests/test_archive.py FTS5 + cosine recall, RRF fusion, dim-mismatch tolerance +tests/test_embed.py embedding client + pack/unpack + cosine helpers +tests/test_state.py ?: subcommands happy + sad paths +tests/test_comma_session.py , refines across turns; archive on TTL +tests/test_claude_mem.py adapter gating + payload shape + error swallowing +tests/test_web.py URL safety + HTML extraction zsh/shellllm.zsh function , + aliases ? , ?? , ??? .github/workflows/ci.yml ruff + pytest on push & PR ``` @@ -247,7 +260,7 @@ Every file read goes through `safe_fs.safe_read`. Four rules, all enforced: Reads cap at 1 MB and use `O_NOFOLLOW` on the final component as a belt against a resolve-then-open symlink race. ```sh -pytest -v # 144 tests; safe_fs alone covers symlinks, traversal, denylist, lookalikes, truncation +pytest -v # 168 tests; safe_fs alone covers symlinks, traversal, denylist, lookalikes, truncation ``` ## What's deliberately not built diff --git a/pyproject.toml b/pyproject.toml index 7969320..c1e2216 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -36,6 +36,7 @@ dev = [ shellllm-comma = "shellllm.comma:main" shellllm-ask = "shellllm.ask:main" shellllm-search = "shellllm.search:main" +shellllm-state = "shellllm.state:main" [project.urls] Homepage = "https://github.com/FrancoisChastel/shellllm" diff --git a/src/shellllm/ask.py b/src/shellllm/ask.py index 826ac25..21289cf 100644 --- a/src/shellllm/ask.py +++ b/src/shellllm/ask.py @@ -497,20 +497,44 @@ def _print_usage(label: str, *, to: Any = None) -> None: out.write( f"usage: {label} \n" f" {label} --new start a fresh session\n" - f" {label} --reset drop current session\n" - f" {label} --history print session transcript\n" - f" {label} --compact force compaction\n" - f" {label} --remember '' save long-term fact\n" - f" {label} --memories list saved facts\n" - f" {label} --forget drop fact #n\n" - f" {label} --recall '' search archived sessions\n" - f" {label} --auto-recall inject archive hits as context this turn\n" - f" {label} --no-auto-recall skip recall this turn (override env)\n" - f" {label} --mem | --no-mem force claude-mem on/off for this call\n" - f" {label} --help show this message\n" + f" {label} --reset drop current session\n" + f" {label} --history print session transcript\n" + f" {label} --compact force compaction\n" + f" {label} --auto-recall inject archive hits as context this turn\n" + f" {label} --no-auto-recall skip recall this turn (override env)\n" + f" {label} --mem | --no-mem force claude-mem on/off for this call\n" + f" {label} --help show this message\n" + f"\n" + f"For facts and cross-session recall, see `?: help`.\n" ) +# Flags that used to live on `?` / `???` and have moved to `?:`. We keep +# matching them so a stale muscle-memory invocation gets a clear redirect +# instead of falling through to the model as a regular question. +_MOVED_FLAGS: dict[str, str] = { + "--remember": "?: add ", + "--memories": "?: list", + "--forget": "?: drop ", + "--recall": "?: recall ", +} + + +def _check_moved_flag(args: list[str], err_label: str) -> int | None: + """Return an exit code if any deprecated flag is present, else None. + + We do this *first* so the redirect fires before we try to parse the + remaining args as a prompt. + """ + for flag, new in _MOVED_FLAGS.items(): + if flag in args: + sys.stderr.write( + f"{_RED}{err_label} error:{_RESET} `{flag}` moved — use `{new}` instead.\n" + ) + return 2 + return None + + def _print_history(session: SessionStore) -> None: """Dump the current session to stdout as plain text.""" if session.is_empty(): @@ -553,33 +577,14 @@ def _consume_flag(flag: str) -> bool: return True return False - def _consume_value(flag: str) -> str | None: - if flag in args: - i = args.index(flag) - if i + 1 < len(args): - value = args[i + 1] - del args[i : i + 2] - return value - del args[i] - return None - - def _consume_tail(flag: str) -> str | None: - """Consume the flag and everything after it as a single string. - - Used for ``--remember`` so ``? --remember the project uses python`` - works without forcing the user to quote the fact. - """ - if flag in args: - i = args.index(flag) - tail = args[i + 1 :] - del args[i:] - return " ".join(tail).strip() if tail else "" - return None - if _consume_flag("--help") or _consume_flag("-h"): _print_usage(err_label) return 0 + moved = _check_moved_flag(args, err_label) + if moved is not None: + return moved + # --mem / --no-mem force the claude-mem integration on or off for # this invocation, overriding env vars. if _consume_flag("--no-mem"): @@ -608,20 +613,6 @@ def _consume_tail(flag: str) -> str | None: print(f"{cmd} session reset.") return 0 - recall_query = _consume_tail("--recall") - if recall_query is not None: - if not recall_query: - sys.stderr.write(f"{_RED}{cmd} error:{_RESET} --recall needs a query\n") - return 2 - query_vec = _safe_embed(recall_query) - hits = archive.search(recall_query, limit=10, query_embedding=query_vec) - if not hits: - print(f"(no archive hits for {recall_query!r})") - return 0 - for i, hit in enumerate(hits, 1): - sys.stdout.write(_format_recall_hit(i, hit)) - return 0 - if _consume_flag("--history"): _print_history(session) return 0 @@ -640,46 +631,6 @@ def _consume_tail(flag: str) -> str | None: ) return 0 - fact = _consume_tail("--remember") - if fact is not None: - try: - stored = memory.add(fact) - except ValueError as exc: - sys.stderr.write(f"{_RED}{cmd} error:{_RESET} {exc}\n") - return 2 - # Mirror to claude-mem as a long-lived user-fact observation. - # Local JSONL stays the source of truth for offline use. - claude_mem.record_observation_async( - stored.text, - kind="user-fact", - metadata={"source": "shellllm --remember"}, - ) - print(f"remembered: {stored.text}") - return 0 - - if _consume_flag("--memories"): - facts = memory.load() - if not facts: - print("(no remembered facts)") - return 0 - for i, f in enumerate(facts, 1): - print(f"{i:>2}. {f.text}") - return 0 - - forget_value = _consume_value("--forget") - if forget_value is not None: - try: - idx = int(forget_value) - except ValueError: - sys.stderr.write(f"{_RED}{cmd} error:{_RESET} --forget needs an integer index\n") - return 2 - removed = memory.forget(idx) - if removed is None: - sys.stderr.write(f"{_RED}{cmd} error:{_RESET} no fact at index {idx}\n") - return 2 - print(f"forgot: {removed.text}") - return 0 - new_session_requested = _consume_flag("--new") if new_session_requested: session.archive_and_reset(archive=archive, embed_fn=_safe_embed) diff --git a/src/shellllm/comma.py b/src/shellllm/comma.py index 1dde92a..4191a6b 100644 --- a/src/shellllm/comma.py +++ b/src/shellllm/comma.py @@ -3,6 +3,15 @@ The whole point of the comma is that it *never executes*. This script prints the chosen command on stdout; the zsh wrapper uses ``print -z`` to drop it on the next prompt line for the user to confirm. + +Sticky session +~~~~~~~~~~~~~~ +Each terminal pane has its own ``,`` thread (see :mod:`shellllm.session`). +The model sees prior user prompts and the JSON it previously emitted, +so follow-ups like ``, the same but only the running ones`` refine the +earlier proposal instead of asking from scratch. Sessions share the +same idle TTL and archive store as ``?`` / ``???``; expired ``,`` +sessions are searchable via ``?: recall``. """ from __future__ import annotations @@ -12,11 +21,16 @@ import shutil import subprocess import sys +from datetime import datetime from pathlib import Path +from typing import Any from rich.console import Console +from .archive import Archive from .client import LlamaServerError, chat +from .embed import embed as embed_text +from .session import SessionStore, sweep_expired SCHEMA = { "type": "object", @@ -44,11 +58,20 @@ # picker stays dependency-free at the data layer. _BOLD_CYAN = "\x1b[1;36m" _DIM = "\x1b[2m" +_CYAN = "\x1b[36m" +_RED = "\x1b[31m" _RESET = "\x1b[0m" _err = Console(stderr=True) +def _safe_embed(text: str) -> list[float] | None: + try: + return embed_text(text) + except Exception: # noqa: BLE001 + return None + + def _context_block() -> str: cwd = Path.cwd() try: @@ -70,13 +93,54 @@ def _system_prompt() -> str: "current directory unless the user clearly means system-wide. Favor " "commands that print rather than mutate. Never include `rm -rf`, " "`sudo`, `curl|sh`, or any destructive one-liner without a safer " - "alternative earlier in the list. Output must match the JSON schema." + "alternative earlier in the list. If the conversation includes prior " + "suggestions and a refinement, build on the prior list rather than " + "restarting from scratch. Output must match the JSON schema." ) +def _print_usage(*, to: Any = None) -> None: + out = to or sys.stdout + out.write( + "usage: , \n" + " , --new start a fresh session\n" + " , --reset drop current session\n" + " , --history print session transcript\n" + " , --help show this message\n" + "\n" + "For facts and cross-session recall, see `?: help`.\n" + ) + + +def _print_history(session: SessionStore) -> None: + if session.is_empty(): + print("(no history)") + return + for m in session.messages: + role = m.get("role", "?") + content = m.get("content", "") + if not isinstance(content, str): + continue + if role == "assistant": + # The model stored a JSON blob; render its commands inline. + try: + parsed = json.loads(content) + items = parsed.get("commands", []) + except json.JSONDecodeError: + items = [] + if items: + print(f"--- {role} (suggestions) ---") + for it in items: + print(f" • {it.get('command', '')} — {it.get('note', '')}") + print() + continue + print(f"--- {role} ---") + print(content.rstrip()) + print() + + def _fzf_pick(items: list[dict[str, str]]) -> str | None: """Show items in fzf, colored, with the note inline on each row.""" - # Format: \t lines = [] for it in items: cmd, note = it["command"], it["note"] @@ -104,11 +168,11 @@ def _fzf_pick(items: list[dict[str, str]]) -> str | None: input="\n".join(lines), text=True, capture_output=True, + check=False, ) if proc.returncode != 0 or not proc.stdout.strip(): return None chosen = proc.stdout.strip() - # Take the raw command from after the tab; fall back to whole line. return chosen.split("\t", 1)[1] if "\t" in chosen else chosen @@ -146,17 +210,34 @@ def _pick(items: list[dict[str, str]]) -> str | None: return _stdin_pick(items) -def main() -> int: - prompt = " ".join(sys.argv[1:]).strip() - if not prompt: - sys.stderr.write("usage: , \n") - return 2 +def _build_messages( + *, + session: SessionStore, + prompt: str, + first_turn: bool, + resumed: bool, +) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]: + """Return (messages_to_send, history_to_persist_after). + + History is everything except the static system prompts we rebuild + each turn — that way, ``$PWD`` changes take effect immediately and + we don't bake a stale prelude into the on-disk log. + """ + pwd = str(Path.cwd()) + date = datetime.now().astimezone().strftime("%Y-%m-%d") - messages = [ + system_msgs: list[dict[str, Any]] = [ {"role": "system", "content": _system_prompt()}, - {"role": "user", "content": f"{_context_block()}\n\n{prompt}"}, ] + if first_turn or resumed or session.meta.last_pwd != pwd or session.meta.last_date != date: + system_msgs.append({"role": "system", "content": _context_block()}) + + history = list(session.messages) + user_msg: dict[str, Any] = {"role": "user", "content": prompt} + return system_msgs + history + [user_msg], history + [user_msg] + +def _ask_model(messages: list[dict[str, Any]]) -> tuple[str, list[dict[str, str]]] | None: try: with _err.status("[cyan]thinking…[/cyan]", spinner="dots"): reply = chat( @@ -168,24 +249,88 @@ def main() -> int: max_tokens=512, ) except LlamaServerError as exc: - _err.print(f"[red], error:[/red] {exc}") - return 1 + _err.print(f"{_RED}, error:{_RESET} {exc}") + return None content = reply.get("content") or "{}" try: parsed = json.loads(content) items = parsed.get("commands", []) except json.JSONDecodeError: - _err.print(f"[red], error:[/red] model returned non-JSON: {content[:200]}") - return 1 - + _err.print(f"{_RED}, error:{_RESET} model returned non-JSON: {content[:200]}") + return None if not items: - _err.print("[red], error:[/red] no suggestions returned") + _err.print(f"{_RED}, error:{_RESET} no suggestions returned") + return None + return content, items + + +def main() -> int: + sweep_expired() + + argv = list(sys.argv[1:]) + + def _consume_flag(flag: str) -> bool: + if flag in argv: + argv.remove(flag) + return True + return False + + if _consume_flag("--help") or _consume_flag("-h"): + _print_usage() + return 0 + + archive = Archive() + session, expired = SessionStore.open(cmd="comma", archive=archive, embed_fn=_safe_embed) + + if _consume_flag("--reset"): + session.archive_and_reset(archive=archive, embed_fn=_safe_embed) + session.write() + print(", session reset.") + return 0 + + if _consume_flag("--history"): + _print_history(session) + return 0 + + if _consume_flag("--new"): + session.archive_and_reset(archive=archive, embed_fn=_safe_embed) + + prompt = " ".join(argv).strip() + if not prompt: + _print_usage(to=sys.stderr) + return 2 + + if expired: + _err.print(f"{_DIM}{_CYAN}↻ idle session expired — starting fresh{_RESET}") + + first_turn = session.is_empty() + resumed = not first_turn + if resumed: + _err.print(f"{_DIM}{_CYAN}↻ refining — turn {session.meta.turn_count + 1}{_RESET}") + + messages, new_history_with_user = _build_messages( + session=session, prompt=prompt, first_turn=first_turn, resumed=resumed + ) + + result = _ask_model(messages) + if result is None: return 1 + content, items = result chosen = _pick(items) if not chosen: return 1 + + # Persist the turn for the next refinement. We store the raw JSON + # the model produced so it sees its own prior list verbatim. + new_history_with_user.append({"role": "assistant", "content": content}) + session.messages = new_history_with_user + pwd = str(Path.cwd()) + date = datetime.now().astimezone().strftime("%Y-%m-%d") + session.touch(pwd=pwd, date=date) + session.write() + print(chosen) return 0 diff --git a/src/shellllm/state.py b/src/shellllm/state.py new file mode 100644 index 0000000..6e44713 --- /dev/null +++ b/src/shellllm/state.py @@ -0,0 +1,168 @@ +"""'?:' — the meta command. Long-term facts + cross-session recall. + +Everything that *isn't* "ask a question" or "propose a command" lives +here. By peeling state ops off ``?`` and ``,`` we keep those punchy and +ask-only; ``?:`` owns the verbs that mutate or query the durable layer. + +Subcommands +~~~~~~~~~~~ + +* ``?: add `` — pin a long-term fact (was ``? --remember``) +* ``?: list`` — list facts (was ``? --memories``) +* ``?: drop `` — drop fact #n (was ``? --forget``) +* ``?: recall `` — search archived sessions (was ``? --recall``) +* ``?: status`` — print quick counts +* ``?: help`` — show usage + +The CLI deliberately uses bare verbs (``add``, ``list``, ``drop`` …) +rather than flag soup so the surface stays tiny and teachable. Bare +``?:`` with no arguments prints help. +""" + +from __future__ import annotations + +import sys +from datetime import datetime + +from .archive import Archive +from .embed import embed as embed_text +from .memory import MemoryStore + +_DIM = "\x1b[2m" +_CYAN = "\x1b[36m" +_RED = "\x1b[31m" +_RESET = "\x1b[0m" + + +def _safe_embed(text: str) -> list[float] | None: + """Best-effort embedding; never propagates exceptions.""" + try: + return embed_text(text) + except Exception: # noqa: BLE001 + return None + + +def _print_usage(label: str = "?:") -> None: + sys.stdout.write( + f"usage: {label} [args]\n" + f" {label} add save a long-term fact\n" + f" {label} list list saved facts\n" + f" {label} drop drop fact #n\n" + f" {label} recall search archived sessions\n" + f" {label} status show counts (facts + archives)\n" + f" {label} help show this message\n" + ) + + +def _format_recall_hit(idx: int, hit) -> str: + when = datetime.fromtimestamp(hit.archived_at).strftime("%Y-%m-%d %H:%M") + header_parts = [ + f"{_DIM}#{idx:<2}{_RESET}", + f"{_CYAN}{hit.cmd}{_RESET}", + f"{_DIM}{when}{_RESET}", + ] + if hit.last_pwd: + header_parts.append(f"{_DIM}{hit.last_pwd}{_RESET}") + header = " · ".join(header_parts) + return f"{header}\n {hit.snippet}\n\n" + + +def _cmd_add(memory: MemoryStore, rest: list[str]) -> int: + text = " ".join(rest).strip() + if not text: + sys.stderr.write(f"{_RED}?: error:{_RESET} `add` needs a fact\n") + return 2 + try: + fact = memory.add(text) + except ValueError as exc: + sys.stderr.write(f"{_RED}?: error:{_RESET} {exc}\n") + return 2 + print(f"remembered: {fact.text}") + return 0 + + +def _cmd_list(memory: MemoryStore) -> int: + facts = memory.load() + if not facts: + print("(no remembered facts)") + return 0 + for i, fact in enumerate(facts, 1): + print(f"{i:>2}. {fact.text}") + return 0 + + +def _cmd_drop(memory: MemoryStore, rest: list[str]) -> int: + if not rest: + sys.stderr.write(f"{_RED}?: error:{_RESET} `drop` needs an index\n") + return 2 + try: + idx = int(rest[0]) + except ValueError: + sys.stderr.write(f"{_RED}?: error:{_RESET} index must be an integer\n") + return 2 + removed = memory.forget(idx) + if removed is None: + sys.stderr.write(f"{_RED}?: error:{_RESET} no fact at index {idx}\n") + return 2 + print(f"forgot: {removed.text}") + return 0 + + +def _cmd_recall(archive: Archive, rest: list[str]) -> int: + query = " ".join(rest).strip() + if not query: + sys.stderr.write(f"{_RED}?: error:{_RESET} `recall` needs a query\n") + return 2 + hits = archive.search(query, limit=10, query_embedding=_safe_embed(query)) + if not hits: + print(f"(no archive hits for {query!r})") + return 0 + for i, hit in enumerate(hits, 1): + sys.stdout.write(_format_recall_hit(i, hit)) + return 0 + + +def _cmd_status(memory: MemoryStore, archive: Archive) -> int: + facts = memory.load() + print(f"{len(facts)} remembered facts · {archive.count()} archived sessions") + return 0 + + +SUBCOMMANDS = { + "add": "save a long-term fact", + "list": "list saved facts", + "drop": "drop fact #n", + "recall": "search archived sessions", + "status": "show counts", + "help": "show this message", +} + + +def main() -> int: + argv = sys.argv[1:] + if not argv or argv[0] in ("help", "--help", "-h"): + _print_usage() + return 0 if argv else 0 + + subcommand, *rest = argv + memory = MemoryStore() + archive = Archive() + + if subcommand == "add": + return _cmd_add(memory, rest) + if subcommand == "list": + return _cmd_list(memory) + if subcommand == "drop": + return _cmd_drop(memory, rest) + if subcommand == "recall": + return _cmd_recall(archive, rest) + if subcommand == "status": + return _cmd_status(memory, archive) + + sys.stderr.write(f"{_RED}?: error:{_RESET} unknown subcommand {subcommand!r}\n") + _print_usage() + return 2 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/test_comma_session.py b/tests/test_comma_session.py new file mode 100644 index 0000000..dd5c5d9 --- /dev/null +++ b/tests/test_comma_session.py @@ -0,0 +1,189 @@ +"""Tests for `,` (comma) per-pane sessions. + +The LLM call itself is mocked so we exercise only the session/flag +plumbing — the JSON shape from the model is fixed and the picker is +short-circuited. +""" + +from __future__ import annotations + +import json +import sys + +import pytest + +from shellllm import comma +from shellllm.session import SessionStore + + +@pytest.fixture +def isolated(tmp_path, monkeypatch): + """Redirect every persistent path so tests can't poison the real cache.""" + monkeypatch.setenv("SHELLLM_SESSIONS_DIR", str(tmp_path / "sessions")) + monkeypatch.setenv("SHELLLM_ARCHIVE_DB", str(tmp_path / "archive.db")) + monkeypatch.setenv("TERM_SESSION_ID", "test-pane-1") + return tmp_path + + +@pytest.fixture +def fake_model(monkeypatch): + """Patch chat() to return a fixed JSON payload. Captures messages sent.""" + sent: list[list[dict]] = [] + + def fake_chat(messages, **kwargs): + sent.append(list(messages)) + return { + "content": json.dumps( + { + "commands": [ + {"command": "ls -lh", "note": "list with sizes"}, + {"command": "ls -la", "note": "include dotfiles"}, + ] + } + ) + } + + monkeypatch.setattr(comma, "chat", fake_chat) + return sent + + +@pytest.fixture +def auto_pick(monkeypatch): + """Skip the picker — always return the first command.""" + + def first_pick(items): + return items[0]["command"] + + monkeypatch.setattr(comma, "_pick", first_pick) + + +def _run(argv, monkeypatch): + monkeypatch.setattr(sys, "argv", ["shellllm-comma", *argv]) + return comma.main() + + +def test_help_returns_zero(monkeypatch, capsys, isolated): + assert _run(["--help"], monkeypatch) == 0 + assert "usage: ," in capsys.readouterr().out + + +def test_no_args_prints_usage(monkeypatch, capsys, isolated): + assert _run([], monkeypatch) == 2 + assert "usage: ," in capsys.readouterr().err + + +def test_first_invocation_persists_session(monkeypatch, capsys, isolated, fake_model, auto_pick): + code = _run(["list", "files", "in", "this", "dir"], monkeypatch) + out = capsys.readouterr().out + assert code == 0 + assert "ls -lh" in out + # Session was written. + store, _ = SessionStore.open(cmd="comma") + assert len(store.messages) == 2 # user + assistant + assert store.messages[0]["role"] == "user" + assert store.messages[1]["role"] == "assistant" + + +def test_second_invocation_carries_prior_turn(monkeypatch, capsys, isolated, fake_model, auto_pick): + _run(["list", "files"], monkeypatch) + capsys.readouterr() + fake_model.clear() + _run(["the", "same", "but", "with", "hidden", "files"], monkeypatch) + # The model received the prior user + assistant in its messages. + sent = fake_model[0] + roles = [m["role"] for m in sent] + assert roles.count("user") == 2 # the first refinement turn + this one + assert roles.count("assistant") == 1 + + +def test_reset_drops_session(monkeypatch, capsys, isolated, fake_model, auto_pick): + _run(["list", "files"], monkeypatch) + capsys.readouterr() + assert _run(["--reset"], monkeypatch) == 0 + out = capsys.readouterr().out + assert "session reset" in out + store, _ = SessionStore.open(cmd="comma") + assert store.is_empty() + + +def test_history_dumps_prior_suggestions(monkeypatch, capsys, isolated, fake_model, auto_pick): + _run(["list", "files"], monkeypatch) + capsys.readouterr() + assert _run(["--history"], monkeypatch) == 0 + out = capsys.readouterr().out + assert "ls -lh" in out + assert "list with sizes" in out + + +def test_history_when_empty(monkeypatch, capsys, isolated): + assert _run(["--history"], monkeypatch) == 0 + assert "(no history)" in capsys.readouterr().out + + +def test_new_flag_archives_and_starts_fresh(monkeypatch, capsys, isolated, fake_model, auto_pick): + _run(["list", "files"], monkeypatch) + capsys.readouterr() + fake_model.clear() + _run(["--new", "find", "biggest", "files"], monkeypatch) + # The model received no prior user/assistant — fresh session. + sent = fake_model[0] + roles = [m["role"] for m in sent] + assert roles.count("user") == 1 + assert roles.count("assistant") == 0 + + +def test_session_uses_comma_cmd_key(monkeypatch, isolated, fake_model, auto_pick): + """Comma sessions live under cmd='comma', distinct from `?` (ask).""" + + _run(["x"], monkeypatch) + comma_store, _ = SessionStore.open(cmd="comma") + ask_store, _ = SessionStore.open(cmd="ask") + assert comma_store.path != ask_store.path + assert len(comma_store.messages) > 0 + assert ask_store.is_empty() + + +def test_expired_session_archives_to_recall( + monkeypatch, capsys, isolated, fake_model, auto_pick, tmp_path +): + """When the comma session expires, its transcript reaches the archive + so `?: recall` can find it later.""" + + from shellllm.archive import Archive + + _run(["search", "for", "ripgrep", "binaries"], monkeypatch) + capsys.readouterr() + + # Bump time past TTL so the next open rotates and archives. + store, _ = SessionStore.open( + cmd="comma", + archive=Archive(), + now=9_999_999_999.0, + ) + assert store.is_empty() + hits = Archive().search("ripgrep") + assert hits + assert any(h.cmd == "comma" for h in hits) + + +def test_redirect_for_ask_remember(monkeypatch, capsys, isolated): + """`,` doesn't share `?`'s deprecation hints — it has its own surface.""" + + # No such flag in comma; it lands in the prompt and gets sent to model. + # We only verify it doesn't error mysteriously. + from shellllm import comma as comma_mod + + captured: list = [] + monkeypatch.setattr( + comma_mod, + "chat", + lambda messages, **kw: ( + captured.append(messages), + {"content": json.dumps({"commands": [{"command": "echo ok", "note": ""}]})}, + )[1], + ) + monkeypatch.setattr(comma_mod, "_pick", lambda items: items[0]["command"]) + assert _run(["--remember", "ripgrep"], monkeypatch) == 0 + # The flag became part of the prompt; the model still answered. + user_msg = next(m["content"] for m in captured[0] if m["role"] == "user") + assert "--remember" in user_msg diff --git a/tests/test_state.py b/tests/test_state.py new file mode 100644 index 0000000..6a47c1f --- /dev/null +++ b/tests/test_state.py @@ -0,0 +1,127 @@ +"""Tests for the `?:` (shellllm-state) CLI.""" + +from __future__ import annotations + +import sys + +import pytest + +from shellllm import state + + +@pytest.fixture +def isolated_state(tmp_path, monkeypatch): + """Redirect memory + archive paths so the CLI doesn't touch the real ones.""" + monkeypatch.setenv("SHELLLM_MEMORY_FILE", str(tmp_path / "memory.jsonl")) + monkeypatch.setenv("SHELLLM_ARCHIVE_DB", str(tmp_path / "archive.db")) + return tmp_path + + +def _run(argv: list[str], monkeypatch) -> int: + monkeypatch.setattr(sys, "argv", ["shellllm-state", *argv]) + return state.main() + + +def test_bare_invocation_prints_usage(monkeypatch, capsys, isolated_state): + code = _run([], monkeypatch) + out = capsys.readouterr().out + assert code == 0 + assert "usage: ?:" in out + + +def test_help_subcommand(monkeypatch, capsys, isolated_state): + assert _run(["help"], monkeypatch) == 0 + assert "usage: ?:" in capsys.readouterr().out + + +def test_add_then_list_round_trips(monkeypatch, capsys, isolated_state): + assert _run(["add", "the", "project", "uses", "python"], monkeypatch) == 0 + capsys.readouterr() + assert _run(["list"], monkeypatch) == 0 + out = capsys.readouterr().out + assert "the project uses python" in out + + +def test_add_with_no_args_errors(monkeypatch, capsys, isolated_state): + code = _run(["add"], monkeypatch) + captured = capsys.readouterr() + assert code == 2 + assert "needs a fact" in captured.err + + +def test_drop_removes_by_index(monkeypatch, capsys, isolated_state): + _run(["add", "alpha"], monkeypatch) + _run(["add", "beta"], monkeypatch) + capsys.readouterr() + assert _run(["drop", "1"], monkeypatch) == 0 + capsys.readouterr() + assert _run(["list"], monkeypatch) == 0 + out = capsys.readouterr().out + assert "alpha" not in out + assert "beta" in out + + +def test_drop_without_index_errors(monkeypatch, capsys, isolated_state): + assert _run(["drop"], monkeypatch) == 2 + assert "needs an index" in capsys.readouterr().err + + +def test_drop_with_non_integer_errors(monkeypatch, capsys, isolated_state): + assert _run(["drop", "abc"], monkeypatch) == 2 + assert "integer" in capsys.readouterr().err + + +def test_drop_out_of_range_errors(monkeypatch, capsys, isolated_state): + _run(["add", "x"], monkeypatch) + capsys.readouterr() + assert _run(["drop", "99"], monkeypatch) == 2 + assert "no fact at index" in capsys.readouterr().err + + +def test_status_reports_counts(monkeypatch, capsys, isolated_state): + _run(["add", "a"], monkeypatch) + _run(["add", "b"], monkeypatch) + capsys.readouterr() + assert _run(["status"], monkeypatch) == 0 + out = capsys.readouterr().out + assert "2 remembered facts" in out + assert "0 archived sessions" in out + + +def test_recall_without_query_errors(monkeypatch, capsys, isolated_state): + assert _run(["recall"], monkeypatch) == 2 + assert "needs a query" in capsys.readouterr().err + + +def test_recall_empty_archive_returns_quietly(monkeypatch, capsys, isolated_state): + assert _run(["recall", "ripgrep"], monkeypatch) == 0 + assert "no archive hits" in capsys.readouterr().out + + +def test_recall_finds_archived_session(monkeypatch, capsys, isolated_state): + """Seed the archive directly and verify recall surfaces it.""" + + from shellllm.archive import Archive + + Archive().ingest_session( + cmd="ask", + terminal_id="t1", + created_at=1.0, + last_used=2.0, + last_pwd="/tmp", + last_date="2026-06-08", + turn_count=1, + messages=[ + {"role": "user", "content": "how do I use ripgrep"}, + {"role": "assistant", "content": "rg pattern path"}, + ], + ) + assert _run(["recall", "ripgrep"], monkeypatch) == 0 + out = capsys.readouterr().out + assert "ripgrep" in out.lower() + + +def test_unknown_subcommand_errors(monkeypatch, capsys, isolated_state): + assert _run(["explode"], monkeypatch) == 2 + err = capsys.readouterr().err + assert "unknown subcommand" in err diff --git a/zsh/shellllm.zsh b/zsh/shellllm.zsh index 1c9b293..598320d 100644 --- a/zsh/shellllm.zsh +++ b/zsh/shellllm.zsh @@ -8,6 +8,7 @@ : ${SHELLLM_COMMA:=shellllm-comma} : ${SHELLLM_ASK:=shellllm-ask} : ${SHELLLM_SEARCH:=shellllm-search} +: ${SHELLLM_STATE:=shellllm-state} : ${SHELLLM_PORT:=8080} : ${SHELLLM_EMBED_PORT:=8081} : ${SHELLLM_EMBED_CTX:=2048} @@ -80,6 +81,23 @@ function _shellllm_search_fn() { } alias '???'='noglob _shellllm_search_fn' +# ─── `?:` — long-term facts + cross-session recall. +# ?: add pin a fact +# ?: list list facts +# ?: drop remove fact #n +# ?: recall search archived sessions +# ?: status counts +# ?: help usage +# +# `?` is a zsh glob char; `:` is the no-op builtin name. Aliasing the +# combined `?:` token works because alias expansion runs before +# globbing and command lookup. `noglob` keeps the `?` from being +# eagerly globbed in the args. +function _shellllm_state_fn() { + ${=SHELLLM_STATE} "$@" +} +alias '?:'='noglob _shellllm_state_fn' + # ─── server helpers ───────────────────────────────────────────────────── function _shellllm_find_gguf() { From 1421e8541b862fdfe6f1ee9c340cbbb05712ae70 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fran=C3=A7ois=20Chastel?= Date: Mon, 8 Jun 2026 17:47:32 -0400 Subject: [PATCH 2/5] refactor: collapse `???` web-search and `?:` state into one `???` memory verb MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Before this change the CLI had five commands: `,`, `?`, `??`, `???`, `?:`. `???` was just `?` with a "web-first" system prompt, and `?:` held the durable layer (facts + recall). Both were artifacts of growth rather than orthogonal concepts. This commit collapses them. New shape (four commands, each one job) - `,` propose shell commands (sticky session, unchanged) - `?` ask (sticky session, model picks tools — incl. web) - `??` llama-server control (unchanged) - `???` memory layer: bare query searches the archive; subcommands manage facts (add/list/drop/status/help). Explicit `recall ` when the query happens to start with a subcommand verb. `?` gains `--web` / `-w` for the rare case you want to force a web-first answer this turn — same agent loop, same session bucket (cmd="ask"), just a stronger system-prompt nudge. `???` gains `--ask` / `--comma` filter flags so you can scope recall to one type of session ("only my asks", "only my command proposals"). Deletions (no deprecation hints — clean cut) - src/shellllm/search.py — `???` web-search command - src/shellllm/state.py — `?:` state command - tests/test_state.py - Entry points: shellllm-search, shellllm-state - Zsh: SHELLLM_SEARCH env, _shellllm_search_fn, alias `?:` Adds - src/shellllm/recall.py — the new `???` command - tests/test_recall.py — dispatch + filter flag coverage - tests/test_ask_web_flag.py — verifies --web swaps system prompt - Entry point: shellllm-recall - Zsh: SHELLLM_RECALL env, _shellllm_recall_fn 178 passing tests (was 168), ruff clean. --- README.md | 85 ++++++++----- pyproject.toml | 3 +- src/shellllm/ask.py | 58 ++++----- src/shellllm/recall.py | 224 ++++++++++++++++++++++++++++++++++ src/shellllm/search.py | 37 ------ src/shellllm/state.py | 168 ------------------------- tests/test_ask_web_flag.py | 59 +++++++++ tests/test_recall.py | 243 +++++++++++++++++++++++++++++++++++++ tests/test_state.py | 127 ------------------- zsh/shellllm.zsh | 38 +++--- 10 files changed, 621 insertions(+), 421 deletions(-) create mode 100644 src/shellllm/recall.py delete mode 100644 src/shellllm/search.py delete mode 100644 src/shellllm/state.py create mode 100644 tests/test_ask_web_flag.py create mode 100644 tests/test_recall.py delete mode 100644 tests/test_state.py diff --git a/README.md b/README.md index 765349b..e4585b8 100644 --- a/README.md +++ b/README.md @@ -6,10 +6,11 @@ Local-LLM zsh helpers. +Four commands, each does one thing: + - **`, `** — proposes 3–5 shell commands with one-line notes, you pick one in `fzf`, it lands on your prompt line via `print -z`. Never auto-executes. Sticky per-pane session so follow-ups refine the prior list (`, the same but only the running ones`). -- **`? `** — small read-only agent with three tools: `read_file` (gated by a filesystem hard wall), `web_search` (DuckDuckGo) and `fetch_url` (follow a result into its page, plain-text). Searches only when the model decides it needs to. Answer streams as live-rendered markdown. Each terminal pane keeps its own sticky conversation — follow-ups continue automatically until 30 min of idle (or `? --new`). -- **`??? `** — same agent, web-first: always starts with a `web_search` and follows the best link with `fetch_url`. Use it when you want fresh facts, not the model's prior. Has its own per-pane session, distinct from `?`. -- **`?: `** — long-term facts and cross-session recall. `?: add `, `?: list`, `?: drop `, `?: recall `, `?: status`. Lives outside the asking commands so `?` and `,` stay ask-only. +- **`? `** — small read-only agent with three tools: `read_file` (gated by a filesystem hard wall), `web_search` (DuckDuckGo) and `fetch_url` (follow a result into its page, plain-text). Searches only when the model decides it needs to — pass `? --web ` to force it. Answer streams as live-rendered markdown. Each terminal pane keeps its own sticky conversation; follow-ups continue automatically until 30 min of idle (or `? --new`). +- **`??? `** — the memory layer. `??? ` searches the archive of past sessions. Subcommands manage facts: `??? add `, `??? list`, `??? drop `, `??? status`. To recall the literal word "add", use `??? recall add`. - **`??`** — start (or stop / list / status) the local `llama-server` backend, with named tiers for speed-vs-quality. `?? --start-embed` boots a second `llama-server` in embedding mode for hybrid semantic recall. Runs against a local `llama-server`. No frontier model, no API key, works with wifi off. @@ -47,9 +48,9 @@ exec zsh , find the five largest files under this directory , the same but only ones modified today # refines the prior , — sticky session ? in markdown, what does git stash do? -?: add I prefer ripgrep over grep # long-term fact, used by all asks -?: recall ripgrep # search past sessions across panes -??? latest stable release of ripgrep and one notable change in it +? --web latest stable release of ripgrep # force web-first this turn +??? add I prefer ripgrep over grep # long-term fact, used by all asks +??? ripgrep # search past sessions across panes ``` ### Upgrading an existing install @@ -88,12 +89,13 @@ huggingface-cli download unsloth/Qwen3-Coder-Next-GGUF ## Sessions Each terminal pane gets its own sticky conversation, one per asking -command. `?`, `???`, and `,` each keep their own thread so a refining -`,` doesn't pollute the Q&A you were having with `?`. +command. `?` and `,` each keep their own thread so refining a `,` +proposal doesn't pollute the Q&A you were having with `?`. ```sh ? what was that flag for ripgrep again ? and how do I use it with json output +? --web latest stable release of ripgrep # force web-first this turn ? --history # transcript of this pane's session ? --new what's a good hash for cache keys # start fresh ? --reset # drop the current session @@ -113,26 +115,43 @@ When the conversation crosses ~80% of `SHELLLM_CTX`, older turns are auto-summarized into a single `` block using the same local model; the most recent 4 turns stay verbatim. -## `?:` — facts and recall +## `???` — memory and recall + +Three question marks reads as *"I'm trying to remember…"*. That's +exactly what this verb does. The most-used path is a bare query — +search the archive of past sessions across all panes and days: + +```sh +??? what was that grep flag again +??? docker volumes +``` + +Subcommands manage long-term facts that get injected into every `?` +system prompt: + +```sh +??? add I prefer ripgrep over grep # pin a long-term fact +??? list # see them +??? drop 2 # remove fact #2 +??? status # counts: facts + archives +??? help +``` + +To recall the literal word `add` / `list` / `drop` / `status` / +`recall` / `help` (so the parser doesn't dispatch to a subcommand), +use the explicit form `??? recall add`. -Everything that *isn't* "ask a question" or "propose a command" lives -under one meta verb so `?` and `,` stay clean. `?:` is your durable -layer. +Filter by which command produced the session: ```sh -?: add I prefer ripgrep over grep # pin a long-term fact -?: list # see them -?: drop 2 # remove fact #2 -?: recall ripgrep # search archived sessions -?: status # counts: facts + archives -?: help +??? --ask docker volumes # only `?` sessions +??? --comma docker volumes # only `,` sessions +??? docker volumes # both (default) ``` -Long-term facts get injected at the top of every `?` / `???` system -prompt, so the agent stops asking you things you've already told it. -Recall works against `~/.cache/shellllm/archive.db`, populated -automatically whenever a session expires or you call `--new` / `--reset` -on any asking command. +The archive at `~/.cache/shellllm/archive.db` gets populated +automatically whenever a session expires or you call `? --new` / +`? --reset` / `, --new` / `, --reset`. Recall always works in **BM25-only mode** — no extra setup, no extra processes. Adding a local embedding server unlocks **hybrid @@ -165,8 +184,9 @@ shellllm auto-detects the server via `SHELLLM_EMBED_URL` - new archive rows get a normalized fp32 embedding written alongside the FTS5 entry; -- `?: recall` and `? --auto-recall` embed the query and add cosine-sim - candidates to the BM25 results, fused via Reciprocal Rank Fusion; +- `???` (bare query) and `? --auto-recall` embed the query and add + cosine-sim candidates to the BM25 results, fused via Reciprocal Rank + Fusion; - mismatched embedding dims (e.g. swapping the model later) are silently skipped — old rows still serve BM25 hits. @@ -223,14 +243,14 @@ src/shellllm/ ├── safe_fs.py filesystem hard wall — $HOME/$PWD + inside-HOME denylist ├── client.py llama-server HTTP client (one-shot + streaming) ├── comma.py , — JSON-schema → fzf picker → stdout -├── ask.py ? — streaming agent loop, live markdown render, CLI dispatch -├── search.py ??? — same loop, web-search-first system prompt -├── state.py ?: — long-term facts + cross-session recall subcommands +├── ask.py ? — streaming agent loop, --web flag, live markdown render +├── recall.py ??? — memory layer: bare-query recall + fact subcommands +├── comma.py , — JSON-schema → fzf picker, sticky session for refinement ├── session.py per-pane conversation persistence (JSONL + idle TTL) -├── memory.py long-term fact store backing `?: add` / `?: list` +├── memory.py long-term fact store backing `??? add` / `??? list` ├── compact.py summary-buffer compaction over the same local model ├── context.py date/OS/timezone prelude (re-injected on PWD/date change) -├── archive.py sqlite FTS5 + optional embeddings for `?: recall` +├── archive.py sqlite FTS5 + optional embeddings for `??? ` ├── embed.py client for a local llama-server in --embedding mode ├── claude_mem.py optional adapter for claude-mem server-beta (observations + context) └── web.py stdlib DuckDuckGo scraper + fetch_url with SSRF guard @@ -240,7 +260,8 @@ tests/test_memory.py fact store + size cap + archive overflow tests/test_compact.py compaction preserves turn boundaries tests/test_archive.py FTS5 + cosine recall, RRF fusion, dim-mismatch tolerance tests/test_embed.py embedding client + pack/unpack + cosine helpers -tests/test_state.py ?: subcommands happy + sad paths +tests/test_recall.py ??? bare-query and subcommand dispatch +tests/test_ask_web_flag.py ? --web swaps to web-first system prompt tests/test_comma_session.py , refines across turns; archive on TTL tests/test_claude_mem.py adapter gating + payload shape + error swallowing tests/test_web.py URL safety + HTML extraction @@ -260,7 +281,7 @@ Every file read goes through `safe_fs.safe_read`. Four rules, all enforced: Reads cap at 1 MB and use `O_NOFOLLOW` on the final component as a belt against a resolve-then-open symlink race. ```sh -pytest -v # 168 tests; safe_fs alone covers symlinks, traversal, denylist, lookalikes, truncation +pytest -v # 178 tests; safe_fs alone covers symlinks, traversal, denylist, lookalikes, truncation ``` ## What's deliberately not built diff --git a/pyproject.toml b/pyproject.toml index c1e2216..acfdb94 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -35,8 +35,7 @@ dev = [ [project.scripts] shellllm-comma = "shellllm.comma:main" shellllm-ask = "shellllm.ask:main" -shellllm-search = "shellllm.search:main" -shellllm-state = "shellllm.state:main" +shellllm-recall = "shellllm.recall:main" [project.urls] Homepage = "https://github.com/FrancoisChastel/shellllm" diff --git a/src/shellllm/ask.py b/src/shellllm/ask.py index 21289cf..5bd3456 100644 --- a/src/shellllm/ask.py +++ b/src/shellllm/ask.py @@ -133,6 +133,20 @@ "`WallViolation`), respect the refusal and reason from what you have." ) +# Used when the user passes ``--web`` / ``-w``. Same agent, same tools; +# just a stronger nudge so the very first action is a web search. +ASK_WEB_SYSTEM = ( + "You answer the user's question by searching the web. Start every " + "response by calling `web_search` with a focused query derived from " + "the question. If a result clearly contains the answer, follow it by " + "calling `fetch_url` on its URL to read the page in full — don't " + "answer from snippets alone when a fetch would give you the real " + "content. You also have `read_file` for files in $HOME or $PWD if " + "useful. Write a concise markdown answer and cite the URLs you used " + "as a short list at the end. If a tool refuses, reason from what you " + "have." +) + # Back-compat alias for any external importers. SYSTEM = ASK_SYSTEM @@ -496,45 +510,20 @@ def _print_usage(label: str, *, to: Any = None) -> None: out = to or sys.stdout out.write( f"usage: {label} \n" + f" {label} --web force web-first this turn\n" f" {label} --new start a fresh session\n" f" {label} --reset drop current session\n" f" {label} --history print session transcript\n" f" {label} --compact force compaction\n" - f" {label} --auto-recall inject archive hits as context this turn\n" - f" {label} --no-auto-recall skip recall this turn (override env)\n" - f" {label} --mem | --no-mem force claude-mem on/off for this call\n" + f" {label} --auto-recall inject archive hits as context\n" + f" {label} --no-auto-recall skip recall this turn\n" + f" {label} --mem | --no-mem force claude-mem on/off for this call\n" f" {label} --help show this message\n" f"\n" - f"For facts and cross-session recall, see `?: help`.\n" + f"For facts and cross-session recall, see `??? help`.\n" ) -# Flags that used to live on `?` / `???` and have moved to `?:`. We keep -# matching them so a stale muscle-memory invocation gets a clear redirect -# instead of falling through to the model as a regular question. -_MOVED_FLAGS: dict[str, str] = { - "--remember": "?: add ", - "--memories": "?: list", - "--forget": "?: drop ", - "--recall": "?: recall ", -} - - -def _check_moved_flag(args: list[str], err_label: str) -> int | None: - """Return an exit code if any deprecated flag is present, else None. - - We do this *first* so the redirect fires before we try to parse the - remaining args as a prompt. - """ - for flag, new in _MOVED_FLAGS.items(): - if flag in args: - sys.stderr.write( - f"{_RED}{err_label} error:{_RESET} `{flag}` moved — use `{new}` instead.\n" - ) - return 2 - return None - - def _print_history(session: SessionStore) -> None: """Dump the current session to stdout as plain text.""" if session.is_empty(): @@ -581,9 +570,12 @@ def _consume_flag(flag: str) -> bool: _print_usage(err_label) return 0 - moved = _check_moved_flag(args, err_label) - if moved is not None: - return moved + # --web / -w flips the system prompt to web-first for this single + # turn. The agent loop is otherwise unchanged; the session stays + # tagged cmd="ask" so web-forced turns mix with local-knowledge + # ones in the same per-pane thread. + if _consume_flag("--web") or _consume_flag("-w"): + system = ASK_WEB_SYSTEM # --mem / --no-mem force the claude-mem integration on or off for # this invocation, overriding env vars. diff --git a/src/shellllm/recall.py b/src/shellllm/recall.py new file mode 100644 index 0000000..c6b43e3 --- /dev/null +++ b/src/shellllm/recall.py @@ -0,0 +1,224 @@ +"""'???' — the memory layer: long-term facts + cross-session recall. + +Three question marks reads as *"I'm trying to remember…"*. That's +exactly what this command does: search archived sessions, pin durable +facts, and inspect what shellllm knows about you across panes and +days. Everything that *isn't* "ask a question" or "propose a command" +lives here. + +Shape +~~~~~ + +Bare query (most-used path) — implicit recall:: + + ??? what was that grep flag again + ??? ripgrep + +Subcommands for fact management and explicit recall:: + + ??? add pin a long-term fact + ??? list list facts + ??? drop drop fact #n + ??? recall explicit recall (use this when the query + starts with a word that's also a subcommand) + ??? status counts + ??? help + +The "bare query vs subcommand" disambiguation is the only piece worth +spelling out: if the first arg is one of the known subcommand verbs +(``add``, ``list``, ``drop``, ``recall``, ``status``, ``help``), it's +a subcommand; otherwise the whole tail is a recall query. To search +for a literal subcommand word, use the explicit ``??? recall `` +form. +""" + +from __future__ import annotations + +import sys +from datetime import datetime + +from .archive import Archive +from .embed import embed as embed_text +from .memory import MemoryStore + +_DIM = "\x1b[2m" +_CYAN = "\x1b[36m" +_RED = "\x1b[31m" +_RESET = "\x1b[0m" + + +def _safe_embed(text: str) -> list[float] | None: + """Best-effort embedding; never propagates exceptions.""" + try: + return embed_text(text) + except Exception: # noqa: BLE001 + return None + + +SUBCOMMANDS = frozenset({"add", "list", "drop", "recall", "status", "help"}) + +# Filter flags map a flag → the ``cmd`` field they restrict recall to. +# Add new entries here when a new asking surface is introduced. +_CMD_FILTERS: dict[str, str] = { + "--ask": "ask", + "--comma": "comma", +} + + +def _print_usage(label: str = "???") -> None: + sys.stdout.write( + f"usage: {label} recall: search archive\n" + f" {label} --ask recall only `?` sessions\n" + f" {label} --comma recall only `,` sessions\n" + f" {label} add save a long-term fact\n" + f" {label} list list saved facts\n" + f" {label} drop drop fact #n\n" + f" {label} recall explicit recall (use when query\n" + f" starts with a subcommand word)\n" + f" {label} status show counts\n" + f" {label} help show this message\n" + ) + + +def _format_recall_hit(idx: int, hit) -> str: + when = datetime.fromtimestamp(hit.archived_at).strftime("%Y-%m-%d %H:%M") + parts = [ + f"{_DIM}#{idx:<2}{_RESET}", + f"{_CYAN}{hit.cmd}{_RESET}", + f"{_DIM}{when}{_RESET}", + ] + if hit.last_pwd: + parts.append(f"{_DIM}{hit.last_pwd}{_RESET}") + header = " · ".join(parts) + return f"{header}\n {hit.snippet}\n\n" + + +def _do_recall(archive: Archive, query: str, *, cmd_filter: str | None = None) -> int: + query = query.strip() + if not query: + sys.stderr.write(f"{_RED}??? error:{_RESET} recall needs a query\n") + return 2 + hits = archive.search( + query, + limit=10, + query_embedding=_safe_embed(query), + cmd_filter=cmd_filter, + ) + if not hits: + scope = f" in `{cmd_filter}` sessions" if cmd_filter else "" + print(f"(no archive hits for {query!r}{scope})") + return 0 + for i, hit in enumerate(hits, 1): + sys.stdout.write(_format_recall_hit(i, hit)) + return 0 + + +def _cmd_add(memory: MemoryStore, rest: list[str]) -> int: + text = " ".join(rest).strip() + if not text: + sys.stderr.write(f"{_RED}??? error:{_RESET} `add` needs a fact\n") + return 2 + try: + fact = memory.add(text) + except ValueError as exc: + sys.stderr.write(f"{_RED}??? error:{_RESET} {exc}\n") + return 2 + print(f"remembered: {fact.text}") + return 0 + + +def _cmd_list(memory: MemoryStore) -> int: + facts = memory.load() + if not facts: + print("(no remembered facts)") + return 0 + for i, fact in enumerate(facts, 1): + print(f"{i:>2}. {fact.text}") + return 0 + + +def _cmd_drop(memory: MemoryStore, rest: list[str]) -> int: + if not rest: + sys.stderr.write(f"{_RED}??? error:{_RESET} `drop` needs an index\n") + return 2 + try: + idx = int(rest[0]) + except ValueError: + sys.stderr.write(f"{_RED}??? error:{_RESET} index must be an integer\n") + return 2 + removed = memory.forget(idx) + if removed is None: + sys.stderr.write(f"{_RED}??? error:{_RESET} no fact at index {idx}\n") + return 2 + print(f"forgot: {removed.text}") + return 0 + + +def _cmd_status(memory: MemoryStore, archive: Archive) -> int: + facts = memory.load() + print(f"{len(facts)} remembered facts · {archive.count()} archived sessions") + return 0 + + +def main() -> int: + argv = list(sys.argv[1:]) + if not argv: + _print_usage() + return 0 + + # Pull --ask / --comma off the front of argv so the rest is either a + # bare query or a subcommand line. We only allow filter flags at + # the start to keep the parser unambiguous: `??? add --ask foo` + # would be confusing — does --ask filter the add? It doesn't. + cmd_filter: str | None = None + while argv and argv[0] in _CMD_FILTERS: + flag = argv.pop(0) + cmd_filter = _CMD_FILTERS[flag] + + if not argv: + # Filter-only invocation: `??? --ask` with no query. + sys.stderr.write(f"{_RED}??? error:{_RESET} no query after filter flag\n") + return 2 + + first, *rest = argv + memory = MemoryStore() + archive = Archive() + + if first in ("help", "--help", "-h"): + _print_usage() + return 0 + + # Filter flags only make sense for recall paths. If the user + # combined `--ask` with `add` / `list` / `drop` / `status`, that's + # almost certainly a typo — facts are global, not per-command. + if cmd_filter is not None and first in SUBCOMMANDS and first != "recall": + sys.stderr.write( + f"{_RED}??? error:{_RESET} filter flags only apply to recall, not `{first}`\n" + ) + return 2 + + # Bare query: first word isn't a known subcommand → treat the whole + # tail as a recall query. This makes `??? what was that flag` work + # without typing `recall` every time, which is the most-used path. + if first not in SUBCOMMANDS: + return _do_recall(archive, " ".join(argv), cmd_filter=cmd_filter) + + if first == "recall": + return _do_recall(archive, " ".join(rest), cmd_filter=cmd_filter) + if first == "add": + return _cmd_add(memory, rest) + if first == "list": + return _cmd_list(memory) + if first == "drop": + return _cmd_drop(memory, rest) + if first == "status": + return _cmd_status(memory, archive) + + # Shouldn't get here — keeps mypy/pyright happy and gives a clean + # message if a subcommand gets added to the set but not dispatched. + sys.stderr.write(f"{_RED}??? error:{_RESET} unhandled subcommand {first!r}\n") + return 2 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/src/shellllm/search.py b/src/shellllm/search.py deleted file mode 100644 index 05317ea..0000000 --- a/src/shellllm/search.py +++ /dev/null @@ -1,37 +0,0 @@ -"""'???' — answer a question by searching the web first. - -Same tool-calling agent as `?`, but the system prompt requires the model -to start with a `web_search` call and follow promising results into -`fetch_url`. Use it when you actually want fresh information rather -than the model's prior knowledge. - -Sessions are kept separate from ``?`` (per-pane, per-command) so a -web-first thread doesn't leak into local-knowledge answers and vice -versa. -""" - -from __future__ import annotations - -import sys - -from .ask import run_cli - -SEARCH_SYSTEM = ( - "You answer the user's question by searching the web. Start every " - "response by calling `web_search` with a focused query derived from " - "the question. If a result clearly contains the answer, follow it by " - "calling `fetch_url` on its URL to read the page in full — don't " - "answer from snippets alone when a fetch would give you the real " - "content. You also have `read_file` for files in $HOME or $PWD if " - "useful. Write a concise markdown answer and cite the URLs you used " - "as a short list at the end. If a tool refuses, reason from what you " - "have." -) - - -def main() -> int: - return run_cli(sys.argv[1:], cmd="search", system=SEARCH_SYSTEM, err_label="???") - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/src/shellllm/state.py b/src/shellllm/state.py deleted file mode 100644 index 6e44713..0000000 --- a/src/shellllm/state.py +++ /dev/null @@ -1,168 +0,0 @@ -"""'?:' — the meta command. Long-term facts + cross-session recall. - -Everything that *isn't* "ask a question" or "propose a command" lives -here. By peeling state ops off ``?`` and ``,`` we keep those punchy and -ask-only; ``?:`` owns the verbs that mutate or query the durable layer. - -Subcommands -~~~~~~~~~~~ - -* ``?: add `` — pin a long-term fact (was ``? --remember``) -* ``?: list`` — list facts (was ``? --memories``) -* ``?: drop `` — drop fact #n (was ``? --forget``) -* ``?: recall `` — search archived sessions (was ``? --recall``) -* ``?: status`` — print quick counts -* ``?: help`` — show usage - -The CLI deliberately uses bare verbs (``add``, ``list``, ``drop`` …) -rather than flag soup so the surface stays tiny and teachable. Bare -``?:`` with no arguments prints help. -""" - -from __future__ import annotations - -import sys -from datetime import datetime - -from .archive import Archive -from .embed import embed as embed_text -from .memory import MemoryStore - -_DIM = "\x1b[2m" -_CYAN = "\x1b[36m" -_RED = "\x1b[31m" -_RESET = "\x1b[0m" - - -def _safe_embed(text: str) -> list[float] | None: - """Best-effort embedding; never propagates exceptions.""" - try: - return embed_text(text) - except Exception: # noqa: BLE001 - return None - - -def _print_usage(label: str = "?:") -> None: - sys.stdout.write( - f"usage: {label} [args]\n" - f" {label} add save a long-term fact\n" - f" {label} list list saved facts\n" - f" {label} drop drop fact #n\n" - f" {label} recall search archived sessions\n" - f" {label} status show counts (facts + archives)\n" - f" {label} help show this message\n" - ) - - -def _format_recall_hit(idx: int, hit) -> str: - when = datetime.fromtimestamp(hit.archived_at).strftime("%Y-%m-%d %H:%M") - header_parts = [ - f"{_DIM}#{idx:<2}{_RESET}", - f"{_CYAN}{hit.cmd}{_RESET}", - f"{_DIM}{when}{_RESET}", - ] - if hit.last_pwd: - header_parts.append(f"{_DIM}{hit.last_pwd}{_RESET}") - header = " · ".join(header_parts) - return f"{header}\n {hit.snippet}\n\n" - - -def _cmd_add(memory: MemoryStore, rest: list[str]) -> int: - text = " ".join(rest).strip() - if not text: - sys.stderr.write(f"{_RED}?: error:{_RESET} `add` needs a fact\n") - return 2 - try: - fact = memory.add(text) - except ValueError as exc: - sys.stderr.write(f"{_RED}?: error:{_RESET} {exc}\n") - return 2 - print(f"remembered: {fact.text}") - return 0 - - -def _cmd_list(memory: MemoryStore) -> int: - facts = memory.load() - if not facts: - print("(no remembered facts)") - return 0 - for i, fact in enumerate(facts, 1): - print(f"{i:>2}. {fact.text}") - return 0 - - -def _cmd_drop(memory: MemoryStore, rest: list[str]) -> int: - if not rest: - sys.stderr.write(f"{_RED}?: error:{_RESET} `drop` needs an index\n") - return 2 - try: - idx = int(rest[0]) - except ValueError: - sys.stderr.write(f"{_RED}?: error:{_RESET} index must be an integer\n") - return 2 - removed = memory.forget(idx) - if removed is None: - sys.stderr.write(f"{_RED}?: error:{_RESET} no fact at index {idx}\n") - return 2 - print(f"forgot: {removed.text}") - return 0 - - -def _cmd_recall(archive: Archive, rest: list[str]) -> int: - query = " ".join(rest).strip() - if not query: - sys.stderr.write(f"{_RED}?: error:{_RESET} `recall` needs a query\n") - return 2 - hits = archive.search(query, limit=10, query_embedding=_safe_embed(query)) - if not hits: - print(f"(no archive hits for {query!r})") - return 0 - for i, hit in enumerate(hits, 1): - sys.stdout.write(_format_recall_hit(i, hit)) - return 0 - - -def _cmd_status(memory: MemoryStore, archive: Archive) -> int: - facts = memory.load() - print(f"{len(facts)} remembered facts · {archive.count()} archived sessions") - return 0 - - -SUBCOMMANDS = { - "add": "save a long-term fact", - "list": "list saved facts", - "drop": "drop fact #n", - "recall": "search archived sessions", - "status": "show counts", - "help": "show this message", -} - - -def main() -> int: - argv = sys.argv[1:] - if not argv or argv[0] in ("help", "--help", "-h"): - _print_usage() - return 0 if argv else 0 - - subcommand, *rest = argv - memory = MemoryStore() - archive = Archive() - - if subcommand == "add": - return _cmd_add(memory, rest) - if subcommand == "list": - return _cmd_list(memory) - if subcommand == "drop": - return _cmd_drop(memory, rest) - if subcommand == "recall": - return _cmd_recall(archive, rest) - if subcommand == "status": - return _cmd_status(memory, archive) - - sys.stderr.write(f"{_RED}?: error:{_RESET} unknown subcommand {subcommand!r}\n") - _print_usage() - return 2 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/tests/test_ask_web_flag.py b/tests/test_ask_web_flag.py new file mode 100644 index 0000000..c25ce34 --- /dev/null +++ b/tests/test_ask_web_flag.py @@ -0,0 +1,59 @@ +"""Confirm `? --web` swaps to the web-first system prompt for one turn.""" + +from __future__ import annotations + +import sys + +import pytest + +from shellllm import ask + + +@pytest.fixture +def isolated(tmp_path, monkeypatch): + monkeypatch.setenv("SHELLLM_SESSIONS_DIR", str(tmp_path / "sessions")) + monkeypatch.setenv("SHELLLM_ARCHIVE_DB", str(tmp_path / "archive.db")) + monkeypatch.setenv("SHELLLM_MEMORY_FILE", str(tmp_path / "memory.jsonl")) + monkeypatch.setenv("TERM_SESSION_ID", "test-pane-web") + return tmp_path + + +def test_web_flag_swaps_system_prompt(monkeypatch, isolated): + captured: dict[str, str] = {} + + def fake_run_agent(prompt, *, system, **kwargs): + captured["system"] = system + captured["prompt"] = prompt + return 0 + + monkeypatch.setattr(ask, "run_agent", fake_run_agent) + monkeypatch.setattr(sys, "argv", ["shellllm-ask", "--web", "latest", "ripgrep"]) + assert ask.main() == 0 + assert captured["system"] == ask.ASK_WEB_SYSTEM + assert captured["prompt"] == "latest ripgrep" + + +def test_short_web_flag_works(monkeypatch, isolated): + captured: dict[str, str] = {} + + def fake_run_agent(prompt, *, system, **kwargs): + captured["system"] = system + return 0 + + monkeypatch.setattr(ask, "run_agent", fake_run_agent) + monkeypatch.setattr(sys, "argv", ["shellllm-ask", "-w", "what is bar"]) + assert ask.main() == 0 + assert captured["system"] == ask.ASK_WEB_SYSTEM + + +def test_no_web_flag_uses_ask_system(monkeypatch, isolated): + captured: dict[str, str] = {} + + def fake_run_agent(prompt, *, system, **kwargs): + captured["system"] = system + return 0 + + monkeypatch.setattr(ask, "run_agent", fake_run_agent) + monkeypatch.setattr(sys, "argv", ["shellllm-ask", "what is bar"]) + assert ask.main() == 0 + assert captured["system"] == ask.ASK_SYSTEM diff --git a/tests/test_recall.py b/tests/test_recall.py new file mode 100644 index 0000000..46892a2 --- /dev/null +++ b/tests/test_recall.py @@ -0,0 +1,243 @@ +"""Tests for the `???` (shellllm-recall) CLI. + +Covers the bare-query-vs-subcommand dispatch, each fact-management +subcommand, and the explicit ``recall`` escape hatch for queries that +start with a subcommand word. +""" + +from __future__ import annotations + +import sys + +import pytest + +from shellllm import recall + + +@pytest.fixture +def isolated(tmp_path, monkeypatch): + """Redirect persistent paths so the CLI can't touch the real cache.""" + monkeypatch.setenv("SHELLLM_MEMORY_FILE", str(tmp_path / "memory.jsonl")) + monkeypatch.setenv("SHELLLM_ARCHIVE_DB", str(tmp_path / "archive.db")) + return tmp_path + + +def _run(argv: list[str], monkeypatch) -> int: + monkeypatch.setattr(sys, "argv", ["shellllm-recall", *argv]) + return recall.main() + + +# ── Dispatch ------------------------------------------------------------- + + +def test_no_args_prints_usage(monkeypatch, capsys, isolated): + assert _run([], monkeypatch) == 0 + assert "usage: ???" in capsys.readouterr().out + + +def test_help_subcommand(monkeypatch, capsys, isolated): + for variant in (["help"], ["--help"], ["-h"]): + assert _run(variant, monkeypatch) == 0 + assert "usage: ???" in capsys.readouterr().out + + +def test_bare_query_routes_to_recall(monkeypatch, capsys, isolated): + # Empty archive — just verify no crash, no recall error. + assert _run(["what", "was", "that", "grep", "flag"], monkeypatch) == 0 + assert "no archive hits" in capsys.readouterr().out + + +def test_first_word_subcommand_routes_to_subcommand(monkeypatch, capsys, isolated): + # `list` is a subcommand — should not be treated as a recall query. + assert _run(["list"], monkeypatch) == 0 + out = capsys.readouterr().out + assert "no remembered facts" in out + + +# ── Facts ---------------------------------------------------------------- + + +def test_add_then_list(monkeypatch, capsys, isolated): + assert _run(["add", "the", "project", "uses", "python"], monkeypatch) == 0 + capsys.readouterr() + assert _run(["list"], monkeypatch) == 0 + out = capsys.readouterr().out + assert "the project uses python" in out + + +def test_add_empty_errors(monkeypatch, capsys, isolated): + assert _run(["add"], monkeypatch) == 2 + assert "needs a fact" in capsys.readouterr().err + + +def test_drop_removes_by_index(monkeypatch, capsys, isolated): + _run(["add", "alpha"], monkeypatch) + _run(["add", "beta"], monkeypatch) + capsys.readouterr() + assert _run(["drop", "1"], monkeypatch) == 0 + capsys.readouterr() + assert _run(["list"], monkeypatch) == 0 + out = capsys.readouterr().out + assert "alpha" not in out + assert "beta" in out + + +def test_drop_non_integer_errors(monkeypatch, capsys, isolated): + assert _run(["drop", "abc"], monkeypatch) == 2 + assert "integer" in capsys.readouterr().err + + +def test_drop_out_of_range_errors(monkeypatch, capsys, isolated): + _run(["add", "x"], monkeypatch) + capsys.readouterr() + assert _run(["drop", "99"], monkeypatch) == 2 + assert "no fact at index" in capsys.readouterr().err + + +def test_status_reports_counts(monkeypatch, capsys, isolated): + _run(["add", "a"], monkeypatch) + _run(["add", "b"], monkeypatch) + capsys.readouterr() + assert _run(["status"], monkeypatch) == 0 + out = capsys.readouterr().out + assert "2 remembered facts" in out + assert "0 archived sessions" in out + + +# ── Recall --------------------------------------------------------------- + + +def test_explicit_recall_with_subcommand_word(monkeypatch, capsys, isolated): + """`??? recall add` must search for the literal word "add", not + invoke the `add` subcommand.""" + assert _run(["recall", "add"], monkeypatch) == 0 + assert "no archive hits" in capsys.readouterr().out + + +def test_explicit_recall_without_query_errors(monkeypatch, capsys, isolated): + assert _run(["recall"], monkeypatch) == 2 + assert "needs a query" in capsys.readouterr().err + + +def test_bare_recall_finds_archived_session(monkeypatch, capsys, isolated): + from shellllm.archive import Archive + + Archive().ingest_session( + cmd="ask", + terminal_id="t1", + created_at=1.0, + last_used=2.0, + last_pwd="/tmp", + last_date="2026-06-08", + turn_count=1, + messages=[ + {"role": "user", "content": "how do I use ripgrep"}, + {"role": "assistant", "content": "rg pattern path"}, + ], + ) + assert _run(["ripgrep"], monkeypatch) == 0 + out = capsys.readouterr().out + assert "ripgrep" in out.lower() + + +def test_bare_multiword_query_is_joined(monkeypatch, capsys, isolated): + from shellllm.archive import Archive + + Archive().ingest_session( + cmd="ask", + terminal_id="t1", + created_at=1.0, + last_used=2.0, + last_pwd="/tmp", + last_date="2026-06-08", + turn_count=1, + messages=[ + {"role": "user", "content": "how do docker volumes work"}, + {"role": "assistant", "content": "they mount paths into the container"}, + ], + ) + assert _run(["docker", "volumes"], monkeypatch) == 0 + out = capsys.readouterr().out + assert "docker" in out.lower() + + +# ── Filter flags --------------------------------------------------------- + + +def _seed_two_cmd_archives() -> None: + """Seed one `ask` and one `comma` archive row sharing a search term.""" + from shellllm.archive import Archive + + a = Archive() + a.ingest_session( + cmd="ask", + terminal_id="t1", + created_at=1.0, + last_used=2.0, + last_pwd="/tmp", + last_date="2026-06-08", + turn_count=1, + messages=[ + {"role": "user", "content": "how do I use ripgrep"}, + {"role": "assistant", "content": "rg pattern path"}, + ], + ) + a.ingest_session( + cmd="comma", + terminal_id="t1", + created_at=1.0, + last_used=2.0, + last_pwd="/tmp", + last_date="2026-06-08", + turn_count=1, + messages=[ + {"role": "user", "content": "ripgrep search incantation"}, + {"role": "assistant", "content": '{"commands":[{"command":"rg foo","note":""}]}'}, + ], + ) + + +def test_ask_filter_returns_only_ask_rows(monkeypatch, capsys, isolated): + _seed_two_cmd_archives() + assert _run(["--ask", "ripgrep"], monkeypatch) == 0 + out = capsys.readouterr().out + # The format includes the cmd label as a column header — count it. + assert out.count("ask") >= 1 + assert "comma" not in out + + +def test_comma_filter_returns_only_comma_rows(monkeypatch, capsys, isolated): + _seed_two_cmd_archives() + assert _run(["--comma", "ripgrep"], monkeypatch) == 0 + out = capsys.readouterr().out + assert out.count("comma") >= 1 + # `ask` would appear in cmd column header for ask hits; it shouldn't. + # We can't grep for raw "ask" because the substring could appear in + # snippets, but checking for the dim-formatted cmd marker is enough: + # if no row has cmd="ask", we won't see the standalone CLI label. + # Easiest: count cmd-labeled rows via the snippet text. + assert "rg pattern" not in out # the ask row's snippet body + + +def test_filter_flag_no_query_errors(monkeypatch, capsys, isolated): + assert _run(["--ask"], monkeypatch) == 2 + assert "no query after filter flag" in capsys.readouterr().err + + +def test_filter_with_non_recall_subcommand_errors(monkeypatch, capsys, isolated): + assert _run(["--ask", "list"], monkeypatch) == 2 + err = capsys.readouterr().err + assert "only apply to recall" in err + + +def test_filter_with_explicit_recall(monkeypatch, capsys, isolated): + """`??? --ask recall add` must search only ask sessions for word "add".""" + _seed_two_cmd_archives() + assert _run(["--ask", "recall", "add"], monkeypatch) == 0 + # Empty match is fine (just verifying parse path works); no crash. + + +def test_filter_no_hits_includes_scope_in_message(monkeypatch, capsys, isolated): + assert _run(["--ask", "totallyabsent"], monkeypatch) == 0 + out = capsys.readouterr().out + assert "in `ask` sessions" in out diff --git a/tests/test_state.py b/tests/test_state.py deleted file mode 100644 index 6a47c1f..0000000 --- a/tests/test_state.py +++ /dev/null @@ -1,127 +0,0 @@ -"""Tests for the `?:` (shellllm-state) CLI.""" - -from __future__ import annotations - -import sys - -import pytest - -from shellllm import state - - -@pytest.fixture -def isolated_state(tmp_path, monkeypatch): - """Redirect memory + archive paths so the CLI doesn't touch the real ones.""" - monkeypatch.setenv("SHELLLM_MEMORY_FILE", str(tmp_path / "memory.jsonl")) - monkeypatch.setenv("SHELLLM_ARCHIVE_DB", str(tmp_path / "archive.db")) - return tmp_path - - -def _run(argv: list[str], monkeypatch) -> int: - monkeypatch.setattr(sys, "argv", ["shellllm-state", *argv]) - return state.main() - - -def test_bare_invocation_prints_usage(monkeypatch, capsys, isolated_state): - code = _run([], monkeypatch) - out = capsys.readouterr().out - assert code == 0 - assert "usage: ?:" in out - - -def test_help_subcommand(monkeypatch, capsys, isolated_state): - assert _run(["help"], monkeypatch) == 0 - assert "usage: ?:" in capsys.readouterr().out - - -def test_add_then_list_round_trips(monkeypatch, capsys, isolated_state): - assert _run(["add", "the", "project", "uses", "python"], monkeypatch) == 0 - capsys.readouterr() - assert _run(["list"], monkeypatch) == 0 - out = capsys.readouterr().out - assert "the project uses python" in out - - -def test_add_with_no_args_errors(monkeypatch, capsys, isolated_state): - code = _run(["add"], monkeypatch) - captured = capsys.readouterr() - assert code == 2 - assert "needs a fact" in captured.err - - -def test_drop_removes_by_index(monkeypatch, capsys, isolated_state): - _run(["add", "alpha"], monkeypatch) - _run(["add", "beta"], monkeypatch) - capsys.readouterr() - assert _run(["drop", "1"], monkeypatch) == 0 - capsys.readouterr() - assert _run(["list"], monkeypatch) == 0 - out = capsys.readouterr().out - assert "alpha" not in out - assert "beta" in out - - -def test_drop_without_index_errors(monkeypatch, capsys, isolated_state): - assert _run(["drop"], monkeypatch) == 2 - assert "needs an index" in capsys.readouterr().err - - -def test_drop_with_non_integer_errors(monkeypatch, capsys, isolated_state): - assert _run(["drop", "abc"], monkeypatch) == 2 - assert "integer" in capsys.readouterr().err - - -def test_drop_out_of_range_errors(monkeypatch, capsys, isolated_state): - _run(["add", "x"], monkeypatch) - capsys.readouterr() - assert _run(["drop", "99"], monkeypatch) == 2 - assert "no fact at index" in capsys.readouterr().err - - -def test_status_reports_counts(monkeypatch, capsys, isolated_state): - _run(["add", "a"], monkeypatch) - _run(["add", "b"], monkeypatch) - capsys.readouterr() - assert _run(["status"], monkeypatch) == 0 - out = capsys.readouterr().out - assert "2 remembered facts" in out - assert "0 archived sessions" in out - - -def test_recall_without_query_errors(monkeypatch, capsys, isolated_state): - assert _run(["recall"], monkeypatch) == 2 - assert "needs a query" in capsys.readouterr().err - - -def test_recall_empty_archive_returns_quietly(monkeypatch, capsys, isolated_state): - assert _run(["recall", "ripgrep"], monkeypatch) == 0 - assert "no archive hits" in capsys.readouterr().out - - -def test_recall_finds_archived_session(monkeypatch, capsys, isolated_state): - """Seed the archive directly and verify recall surfaces it.""" - - from shellllm.archive import Archive - - Archive().ingest_session( - cmd="ask", - terminal_id="t1", - created_at=1.0, - last_used=2.0, - last_pwd="/tmp", - last_date="2026-06-08", - turn_count=1, - messages=[ - {"role": "user", "content": "how do I use ripgrep"}, - {"role": "assistant", "content": "rg pattern path"}, - ], - ) - assert _run(["recall", "ripgrep"], monkeypatch) == 0 - out = capsys.readouterr().out - assert "ripgrep" in out.lower() - - -def test_unknown_subcommand_errors(monkeypatch, capsys, isolated_state): - assert _run(["explode"], monkeypatch) == 2 - err = capsys.readouterr().err - assert "unknown subcommand" in err diff --git a/zsh/shellllm.zsh b/zsh/shellllm.zsh index 598320d..6fcc7bb 100644 --- a/zsh/shellllm.zsh +++ b/zsh/shellllm.zsh @@ -7,8 +7,7 @@ : ${SHELLLM_COMMA:=shellllm-comma} : ${SHELLLM_ASK:=shellllm-ask} -: ${SHELLLM_SEARCH:=shellllm-search} -: ${SHELLLM_STATE:=shellllm-state} +: ${SHELLLM_RECALL:=shellllm-recall} : ${SHELLLM_PORT:=8080} : ${SHELLLM_EMBED_PORT:=8081} : ${SHELLLM_EMBED_CTX:=2048} @@ -75,28 +74,23 @@ function _shellllm_ask_fn() { } alias '?'='noglob _shellllm_ask_fn' -# ─── `???` — answer by searching the web first. Same noglob requirement. -function _shellllm_search_fn() { - ${=SHELLLM_SEARCH} "$@" -} -alias '???'='noglob _shellllm_search_fn' - -# ─── `?:` — long-term facts + cross-session recall. -# ?: add pin a fact -# ?: list list facts -# ?: drop remove fact #n -# ?: recall search archived sessions -# ?: status counts -# ?: help usage +# ─── `???` — memory layer: long-term facts + cross-session recall. +# +# ??? bare query → search archived sessions +# ??? add pin a long-term fact +# ??? list list facts +# ??? drop drop fact #n +# ??? recall explicit recall (use when the query starts with +# a word that's also a subcommand) +# ??? status counts +# ??? help usage # -# `?` is a zsh glob char; `:` is the no-op builtin name. Aliasing the -# combined `?:` token works because alias expansion runs before -# globbing and command lookup. `noglob` keeps the `?` from being -# eagerly globbed in the args. -function _shellllm_state_fn() { - ${=SHELLLM_STATE} "$@" +# `noglob` is needed because `?` is a zsh glob char; aliasing `???` +# directly is fine because alias expansion runs before globbing. +function _shellllm_recall_fn() { + ${=SHELLLM_RECALL} "$@" } -alias '?:'='noglob _shellllm_state_fn' +alias '???'='noglob _shellllm_recall_fn' # ─── server helpers ───────────────────────────────────────────────────── From 9d307c6be7b658ee1cc171d0415b34615b5617e0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fran=C3=A7ois=20Chastel?= Date: Mon, 8 Jun 2026 17:55:56 -0400 Subject: [PATCH 3/5] =?UTF-8?q?refactor(=3F=3F=3F):=20flags=20only=20?= =?UTF-8?q?=E2=80=94=20no=20bare-word=20verbs?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Mode ops are now flags (--add / --list / --drop / --status) instead of subcommand verbs. Bare-word input is unambiguously a recall query; no more "is this a subcommand or a search term?" overload. Surface - ??? bare query → recall - ??? --add was: ??? add - ??? --list was: ??? list - ??? --drop was: ??? drop - ??? --status was: ??? status - ??? --ask / --comma filter recall (unchanged) - ??? --help was: ??? help Validation - Mode flags are mutually exclusive (two → error) - Filter flags are mutually exclusive (two → error) - Filter + mode flag → error (facts are global, not per-command) - Per-mode extra-arg checks (--list takes no args, --drop takes one, etc.) 181 passing tests (was 178), ruff clean. --- README.md | 34 ++++---- src/shellllm/recall.py | 168 +++++++++++++++++++---------------- tests/test_recall.py | 193 +++++++++++++++++++++-------------------- zsh/shellllm.zsh | 20 +++-- 4 files changed, 221 insertions(+), 194 deletions(-) diff --git a/README.md b/README.md index e4585b8..0c04408 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ Four commands, each does one thing: - **`, `** — proposes 3–5 shell commands with one-line notes, you pick one in `fzf`, it lands on your prompt line via `print -z`. Never auto-executes. Sticky per-pane session so follow-ups refine the prior list (`, the same but only the running ones`). - **`? `** — small read-only agent with three tools: `read_file` (gated by a filesystem hard wall), `web_search` (DuckDuckGo) and `fetch_url` (follow a result into its page, plain-text). Searches only when the model decides it needs to — pass `? --web ` to force it. Answer streams as live-rendered markdown. Each terminal pane keeps its own sticky conversation; follow-ups continue automatically until 30 min of idle (or `? --new`). -- **`??? `** — the memory layer. `??? ` searches the archive of past sessions. Subcommands manage facts: `??? add `, `??? list`, `??? drop `, `??? status`. To recall the literal word "add", use `??? recall add`. +- **`??? `** — the memory layer. `??? ` searches the archive of past sessions. Flags manage facts: `??? --add `, `??? --list`, `??? --drop `, `??? --status`. Bare-word search terms (like `??? list`) recall the word — flags are the only "verbs". - **`??`** — start (or stop / list / status) the local `llama-server` backend, with named tiers for speed-vs-quality. `?? --start-embed` boots a second `llama-server` in embedding mode for hybrid semantic recall. Runs against a local `llama-server`. No frontier model, no API key, works with wifi off. @@ -49,7 +49,7 @@ exec zsh , the same but only ones modified today # refines the prior , — sticky session ? in markdown, what does git stash do? ? --web latest stable release of ripgrep # force web-first this turn -??? add I prefer ripgrep over grep # long-term fact, used by all asks +??? --add I prefer ripgrep over grep # long-term fact, used by all asks ??? ripgrep # search past sessions across panes ``` @@ -126,22 +126,18 @@ search the archive of past sessions across all panes and days: ??? docker volumes ``` -Subcommands manage long-term facts that get injected into every `?` -system prompt: +Every other operation is a flag — no bare-word verbs, so any +non-flag input is unambiguously a recall query: ```sh -??? add I prefer ripgrep over grep # pin a long-term fact -??? list # see them -??? drop 2 # remove fact #2 -??? status # counts: facts + archives -??? help +??? --add I prefer ripgrep over grep # pin a long-term fact +??? --list # see them +??? --drop 2 # remove fact #2 +??? --status # counts: facts + archives +??? --help ``` -To recall the literal word `add` / `list` / `drop` / `status` / -`recall` / `help` (so the parser doesn't dispatch to a subcommand), -use the explicit form `??? recall add`. - -Filter by which command produced the session: +Filter recall by which command produced the session: ```sh ??? --ask docker volumes # only `?` sessions @@ -149,6 +145,10 @@ Filter by which command produced the session: ??? docker volumes # both (default) ``` +Mode flags (`--add` / `--list` / `--drop` / `--status`) are mutually +exclusive. Filter flags only apply to recall — combining `--ask` with +`--list` errors because facts are global. + The archive at `~/.cache/shellllm/archive.db` gets populated automatically whenever a session expires or you call `? --new` / `? --reset` / `, --new` / `, --reset`. @@ -244,7 +244,7 @@ src/shellllm/ ├── client.py llama-server HTTP client (one-shot + streaming) ├── comma.py , — JSON-schema → fzf picker → stdout ├── ask.py ? — streaming agent loop, --web flag, live markdown render -├── recall.py ??? — memory layer: bare-query recall + fact subcommands +├── recall.py ??? — memory layer: bare-query recall + fact flags ├── comma.py , — JSON-schema → fzf picker, sticky session for refinement ├── session.py per-pane conversation persistence (JSONL + idle TTL) ├── memory.py long-term fact store backing `??? add` / `??? list` @@ -260,7 +260,7 @@ tests/test_memory.py fact store + size cap + archive overflow tests/test_compact.py compaction preserves turn boundaries tests/test_archive.py FTS5 + cosine recall, RRF fusion, dim-mismatch tolerance tests/test_embed.py embedding client + pack/unpack + cosine helpers -tests/test_recall.py ??? bare-query and subcommand dispatch +tests/test_recall.py ??? bare-query + flag-only dispatch tests/test_ask_web_flag.py ? --web swaps to web-first system prompt tests/test_comma_session.py , refines across turns; archive on TTL tests/test_claude_mem.py adapter gating + payload shape + error swallowing @@ -281,7 +281,7 @@ Every file read goes through `safe_fs.safe_read`. Four rules, all enforced: Reads cap at 1 MB and use `O_NOFOLLOW` on the final component as a belt against a resolve-then-open symlink race. ```sh -pytest -v # 178 tests; safe_fs alone covers symlinks, traversal, denylist, lookalikes, truncation +pytest -v # 181 tests; safe_fs alone covers symlinks, traversal, denylist, lookalikes, truncation ``` ## What's deliberately not built diff --git a/src/shellllm/recall.py b/src/shellllm/recall.py index c6b43e3..ee462ef 100644 --- a/src/shellllm/recall.py +++ b/src/shellllm/recall.py @@ -9,27 +9,26 @@ Shape ~~~~~ -Bare query (most-used path) — implicit recall:: +Bare query is the most-used path — implicit recall:: ??? what was that grep flag again ??? ripgrep -Subcommands for fact management and explicit recall:: - - ??? add pin a long-term fact - ??? list list facts - ??? drop drop fact #n - ??? recall explicit recall (use this when the query - starts with a word that's also a subcommand) - ??? status counts - ??? help - -The "bare query vs subcommand" disambiguation is the only piece worth -spelling out: if the first arg is one of the known subcommand verbs -(``add``, ``list``, ``drop``, ``recall``, ``status``, ``help``), it's -a subcommand; otherwise the whole tail is a recall query. To search -for a literal subcommand word, use the explicit ``??? recall `` -form. +Every other operation is a flag — no bare-word verbs:: + + ??? --add pin a long-term fact + ??? --list list facts + ??? --drop drop fact #n + ??? --status counts (facts + archives) + ??? --help usage + + ??? --ask recall only `?` sessions + ??? --comma recall only `,` sessions + +Mode flags (``--add`` / ``--list`` / ``--drop`` / ``--status``) are +mutually exclusive. Filter flags (``--ask`` / ``--comma``) only make +sense alongside a recall query — combining them with a mode flag +errors, because facts are global and counts are global. """ from __future__ import annotations @@ -55,11 +54,12 @@ def _safe_embed(text: str) -> list[float] | None: return None -SUBCOMMANDS = frozenset({"add", "list", "drop", "recall", "status", "help"}) +# Mutually-exclusive "do this instead of recall" flags. +_MODE_FLAGS = ("--add", "--list", "--drop", "--status") -# Filter flags map a flag → the ``cmd`` field they restrict recall to. -# Add new entries here when a new asking surface is introduced. -_CMD_FILTERS: dict[str, str] = { +# Restrict recall to a single asking surface. Extend when a new asking +# command is added. +_FILTER_FLAGS: dict[str, str] = { "--ask": "ask", "--comma": "comma", } @@ -70,13 +70,11 @@ def _print_usage(label: str = "???") -> None: f"usage: {label} recall: search archive\n" f" {label} --ask recall only `?` sessions\n" f" {label} --comma recall only `,` sessions\n" - f" {label} add save a long-term fact\n" - f" {label} list list saved facts\n" - f" {label} drop drop fact #n\n" - f" {label} recall explicit recall (use when query\n" - f" starts with a subcommand word)\n" - f" {label} status show counts\n" - f" {label} help show this message\n" + f" {label} --add save a long-term fact\n" + f" {label} --list list saved facts\n" + f" {label} --drop drop fact #n\n" + f" {label} --status show counts\n" + f" {label} --help show this message\n" ) @@ -116,7 +114,7 @@ def _do_recall(archive: Archive, query: str, *, cmd_filter: str | None = None) - def _cmd_add(memory: MemoryStore, rest: list[str]) -> int: text = " ".join(rest).strip() if not text: - sys.stderr.write(f"{_RED}??? error:{_RESET} `add` needs a fact\n") + sys.stderr.write(f"{_RED}??? error:{_RESET} `--add` needs a fact\n") return 2 try: fact = memory.add(text) @@ -127,7 +125,10 @@ def _cmd_add(memory: MemoryStore, rest: list[str]) -> int: return 0 -def _cmd_list(memory: MemoryStore) -> int: +def _cmd_list(memory: MemoryStore, rest: list[str]) -> int: + if rest: + sys.stderr.write(f"{_RED}??? error:{_RESET} `--list` takes no arguments\n") + return 2 facts = memory.load() if not facts: print("(no remembered facts)") @@ -139,7 +140,10 @@ def _cmd_list(memory: MemoryStore) -> int: def _cmd_drop(memory: MemoryStore, rest: list[str]) -> int: if not rest: - sys.stderr.write(f"{_RED}??? error:{_RESET} `drop` needs an index\n") + sys.stderr.write(f"{_RED}??? error:{_RESET} `--drop` needs an index\n") + return 2 + if len(rest) > 1: + sys.stderr.write(f"{_RED}??? error:{_RESET} `--drop` takes one index\n") return 2 try: idx = int(rest[0]) @@ -154,70 +158,82 @@ def _cmd_drop(memory: MemoryStore, rest: list[str]) -> int: return 0 -def _cmd_status(memory: MemoryStore, archive: Archive) -> int: +def _cmd_status(memory: MemoryStore, archive: Archive, rest: list[str]) -> int: + if rest: + sys.stderr.write(f"{_RED}??? error:{_RESET} `--status` takes no arguments\n") + return 2 facts = memory.load() print(f"{len(facts)} remembered facts · {archive.count()} archived sessions") return 0 +def _collect_mode(argv: list[str]) -> tuple[str | None, int]: + """Pull a single mode flag from ``argv``. Multi-mode → error code.""" + present = [f for f in _MODE_FLAGS if f in argv] + if len(present) > 1: + sys.stderr.write( + f"{_RED}??? error:{_RESET} only one of {', '.join(_MODE_FLAGS)} at a time\n" + ) + return None, 2 + if present: + argv.remove(present[0]) + return present[0], 0 + return None, 0 + + +def _collect_filter(argv: list[str]) -> tuple[str | None, int]: + """Pull a single filter flag from ``argv``. Multi-filter → error code.""" + present = [f for f in _FILTER_FLAGS if f in argv] + if len(present) > 1: + sys.stderr.write( + f"{_RED}??? error:{_RESET} only one of {', '.join(_FILTER_FLAGS)} at a time\n" + ) + return None, 2 + if present: + argv.remove(present[0]) + return _FILTER_FLAGS[present[0]], 0 + return None, 0 + + def main() -> int: argv = list(sys.argv[1:]) - if not argv: + if not argv or argv[0] in ("--help", "-h"): _print_usage() return 0 - # Pull --ask / --comma off the front of argv so the rest is either a - # bare query or a subcommand line. We only allow filter flags at - # the start to keep the parser unambiguous: `??? add --ask foo` - # would be confusing — does --ask filter the add? It doesn't. - cmd_filter: str | None = None - while argv and argv[0] in _CMD_FILTERS: - flag = argv.pop(0) - cmd_filter = _CMD_FILTERS[flag] + mode, err = _collect_mode(argv) + if err: + return err - if not argv: - # Filter-only invocation: `??? --ask` with no query. - sys.stderr.write(f"{_RED}??? error:{_RESET} no query after filter flag\n") - return 2 + cmd_filter, err = _collect_filter(argv) + if err: + return err - first, *rest = argv memory = MemoryStore() archive = Archive() - if first in ("help", "--help", "-h"): - _print_usage() - return 0 - - # Filter flags only make sense for recall paths. If the user - # combined `--ask` with `add` / `list` / `drop` / `status`, that's - # almost certainly a typo — facts are global, not per-command. - if cmd_filter is not None and first in SUBCOMMANDS and first != "recall": + # Filter flags only apply to recall paths. Combining with a mode + # flag is almost certainly a typo — facts and counts are global. + if cmd_filter is not None and mode is not None: sys.stderr.write( - f"{_RED}??? error:{_RESET} filter flags only apply to recall, not `{first}`\n" + f"{_RED}??? error:{_RESET} filter flags only apply to recall, not `{mode}`\n" ) return 2 - # Bare query: first word isn't a known subcommand → treat the whole - # tail as a recall query. This makes `??? what was that flag` work - # without typing `recall` every time, which is the most-used path. - if first not in SUBCOMMANDS: - return _do_recall(archive, " ".join(argv), cmd_filter=cmd_filter) - - if first == "recall": - return _do_recall(archive, " ".join(rest), cmd_filter=cmd_filter) - if first == "add": - return _cmd_add(memory, rest) - if first == "list": - return _cmd_list(memory) - if first == "drop": - return _cmd_drop(memory, rest) - if first == "status": - return _cmd_status(memory, archive) - - # Shouldn't get here — keeps mypy/pyright happy and gives a clean - # message if a subcommand gets added to the set but not dispatched. - sys.stderr.write(f"{_RED}??? error:{_RESET} unhandled subcommand {first!r}\n") - return 2 + if mode == "--add": + return _cmd_add(memory, argv) + if mode == "--list": + return _cmd_list(memory, argv) + if mode == "--drop": + return _cmd_drop(memory, argv) + if mode == "--status": + return _cmd_status(memory, archive, argv) + + # No mode flag → recall path. Filter is optional. + if not argv: + sys.stderr.write(f"{_RED}??? error:{_RESET} no query\n") + return 2 + return _do_recall(archive, " ".join(argv), cmd_filter=cmd_filter) if __name__ == "__main__": diff --git a/tests/test_recall.py b/tests/test_recall.py index 46892a2..08e5597 100644 --- a/tests/test_recall.py +++ b/tests/test_recall.py @@ -1,8 +1,8 @@ """Tests for the `???` (shellllm-recall) CLI. -Covers the bare-query-vs-subcommand dispatch, each fact-management -subcommand, and the explicit ``recall`` escape hatch for queries that -start with a subcommand word. +Every operation other than bare-query recall is a flag — no +subcommand verbs. Tests cover mode-flag dispatch, filter flags, +multi-flag rejection, and the bare-query fallback. """ from __future__ import annotations @@ -35,130 +35,112 @@ def test_no_args_prints_usage(monkeypatch, capsys, isolated): assert "usage: ???" in capsys.readouterr().out -def test_help_subcommand(monkeypatch, capsys, isolated): - for variant in (["help"], ["--help"], ["-h"]): +def test_help_flag(monkeypatch, capsys, isolated): + for variant in (["--help"], ["-h"]): assert _run(variant, monkeypatch) == 0 assert "usage: ???" in capsys.readouterr().out def test_bare_query_routes_to_recall(monkeypatch, capsys, isolated): - # Empty archive — just verify no crash, no recall error. assert _run(["what", "was", "that", "grep", "flag"], monkeypatch) == 0 assert "no archive hits" in capsys.readouterr().out -def test_first_word_subcommand_routes_to_subcommand(monkeypatch, capsys, isolated): - # `list` is a subcommand — should not be treated as a recall query. +def test_bare_query_starting_with_word_list_still_recalls(monkeypatch, capsys, isolated): + """No bare-word subcommands → `list` is a regular search term now.""" assert _run(["list"], monkeypatch) == 0 out = capsys.readouterr().out - assert "no remembered facts" in out + assert "no archive hits for 'list'" in out -# ── Facts ---------------------------------------------------------------- +# ── Facts (mode flags) --------------------------------------------------- def test_add_then_list(monkeypatch, capsys, isolated): - assert _run(["add", "the", "project", "uses", "python"], monkeypatch) == 0 + assert _run(["--add", "the", "project", "uses", "python"], monkeypatch) == 0 capsys.readouterr() - assert _run(["list"], monkeypatch) == 0 + assert _run(["--list"], monkeypatch) == 0 out = capsys.readouterr().out assert "the project uses python" in out def test_add_empty_errors(monkeypatch, capsys, isolated): - assert _run(["add"], monkeypatch) == 2 + assert _run(["--add"], monkeypatch) == 2 assert "needs a fact" in capsys.readouterr().err +def test_list_with_extra_args_errors(monkeypatch, capsys, isolated): + assert _run(["--list", "extra"], monkeypatch) == 2 + assert "takes no arguments" in capsys.readouterr().err + + def test_drop_removes_by_index(monkeypatch, capsys, isolated): - _run(["add", "alpha"], monkeypatch) - _run(["add", "beta"], monkeypatch) + _run(["--add", "alpha"], monkeypatch) + _run(["--add", "beta"], monkeypatch) capsys.readouterr() - assert _run(["drop", "1"], monkeypatch) == 0 + assert _run(["--drop", "1"], monkeypatch) == 0 capsys.readouterr() - assert _run(["list"], monkeypatch) == 0 + assert _run(["--list"], monkeypatch) == 0 out = capsys.readouterr().out assert "alpha" not in out assert "beta" in out def test_drop_non_integer_errors(monkeypatch, capsys, isolated): - assert _run(["drop", "abc"], monkeypatch) == 2 + assert _run(["--drop", "abc"], monkeypatch) == 2 assert "integer" in capsys.readouterr().err def test_drop_out_of_range_errors(monkeypatch, capsys, isolated): - _run(["add", "x"], monkeypatch) + _run(["--add", "x"], monkeypatch) capsys.readouterr() - assert _run(["drop", "99"], monkeypatch) == 2 + assert _run(["--drop", "99"], monkeypatch) == 2 assert "no fact at index" in capsys.readouterr().err +def test_drop_no_arg_errors(monkeypatch, capsys, isolated): + assert _run(["--drop"], monkeypatch) == 2 + assert "needs an index" in capsys.readouterr().err + + +def test_drop_multiple_args_errors(monkeypatch, capsys, isolated): + assert _run(["--drop", "1", "2"], monkeypatch) == 2 + assert "one index" in capsys.readouterr().err + + def test_status_reports_counts(monkeypatch, capsys, isolated): - _run(["add", "a"], monkeypatch) - _run(["add", "b"], monkeypatch) + _run(["--add", "a"], monkeypatch) + _run(["--add", "b"], monkeypatch) capsys.readouterr() - assert _run(["status"], monkeypatch) == 0 + assert _run(["--status"], monkeypatch) == 0 out = capsys.readouterr().out assert "2 remembered facts" in out assert "0 archived sessions" in out -# ── Recall --------------------------------------------------------------- +def test_status_with_args_errors(monkeypatch, capsys, isolated): + assert _run(["--status", "extra"], monkeypatch) == 2 + assert "takes no arguments" in capsys.readouterr().err -def test_explicit_recall_with_subcommand_word(monkeypatch, capsys, isolated): - """`??? recall add` must search for the literal word "add", not - invoke the `add` subcommand.""" - assert _run(["recall", "add"], monkeypatch) == 0 - assert "no archive hits" in capsys.readouterr().out +# ── Multi-flag rejection ------------------------------------------------- -def test_explicit_recall_without_query_errors(monkeypatch, capsys, isolated): - assert _run(["recall"], monkeypatch) == 2 - assert "needs a query" in capsys.readouterr().err +def test_two_mode_flags_errors(monkeypatch, capsys, isolated): + assert _run(["--add", "x", "--list"], monkeypatch) == 2 + assert "only one of" in capsys.readouterr().err -def test_bare_recall_finds_archived_session(monkeypatch, capsys, isolated): - from shellllm.archive import Archive +def test_two_filter_flags_errors(monkeypatch, capsys, isolated): + assert _run(["--ask", "--comma", "x"], monkeypatch) == 2 + assert "only one of" in capsys.readouterr().err - Archive().ingest_session( - cmd="ask", - terminal_id="t1", - created_at=1.0, - last_used=2.0, - last_pwd="/tmp", - last_date="2026-06-08", - turn_count=1, - messages=[ - {"role": "user", "content": "how do I use ripgrep"}, - {"role": "assistant", "content": "rg pattern path"}, - ], - ) - assert _run(["ripgrep"], monkeypatch) == 0 - out = capsys.readouterr().out - assert "ripgrep" in out.lower() - - -def test_bare_multiword_query_is_joined(monkeypatch, capsys, isolated): - from shellllm.archive import Archive - Archive().ingest_session( - cmd="ask", - terminal_id="t1", - created_at=1.0, - last_used=2.0, - last_pwd="/tmp", - last_date="2026-06-08", - turn_count=1, - messages=[ - {"role": "user", "content": "how do docker volumes work"}, - {"role": "assistant", "content": "they mount paths into the container"}, - ], - ) - assert _run(["docker", "volumes"], monkeypatch) == 0 - out = capsys.readouterr().out - assert "docker" in out.lower() +def test_filter_with_mode_flag_errors(monkeypatch, capsys, isolated): + """Filters only apply to recall, not to fact management.""" + assert _run(["--ask", "--list"], monkeypatch) == 2 + err = capsys.readouterr().err + assert "only apply to recall" in err # ── Filter flags --------------------------------------------------------- @@ -201,7 +183,6 @@ def test_ask_filter_returns_only_ask_rows(monkeypatch, capsys, isolated): _seed_two_cmd_archives() assert _run(["--ask", "ripgrep"], monkeypatch) == 0 out = capsys.readouterr().out - # The format includes the cmd label as a column header — count it. assert out.count("ask") >= 1 assert "comma" not in out @@ -210,34 +191,62 @@ def test_comma_filter_returns_only_comma_rows(monkeypatch, capsys, isolated): _seed_two_cmd_archives() assert _run(["--comma", "ripgrep"], monkeypatch) == 0 out = capsys.readouterr().out - assert out.count("comma") >= 1 - # `ask` would appear in cmd column header for ask hits; it shouldn't. - # We can't grep for raw "ask" because the substring could appear in - # snippets, but checking for the dim-formatted cmd marker is enough: - # if no row has cmd="ask", we won't see the standalone CLI label. - # Easiest: count cmd-labeled rows via the snippet text. - assert "rg pattern" not in out # the ask row's snippet body + assert "comma" in out + # The ask row's snippet body should NOT appear. + assert "rg pattern" not in out -def test_filter_flag_no_query_errors(monkeypatch, capsys, isolated): +def test_filter_no_hits_includes_scope_in_message(monkeypatch, capsys, isolated): + assert _run(["--ask", "totallyabsent"], monkeypatch) == 0 + out = capsys.readouterr().out + assert "in `ask` sessions" in out + + +def test_filter_with_no_query_errors(monkeypatch, capsys, isolated): assert _run(["--ask"], monkeypatch) == 2 - assert "no query after filter flag" in capsys.readouterr().err + assert "no query" in capsys.readouterr().err -def test_filter_with_non_recall_subcommand_errors(monkeypatch, capsys, isolated): - assert _run(["--ask", "list"], monkeypatch) == 2 - err = capsys.readouterr().err - assert "only apply to recall" in err +# ── Bare-recall edge cases ----------------------------------------------- -def test_filter_with_explicit_recall(monkeypatch, capsys, isolated): - """`??? --ask recall add` must search only ask sessions for word "add".""" - _seed_two_cmd_archives() - assert _run(["--ask", "recall", "add"], monkeypatch) == 0 - # Empty match is fine (just verifying parse path works); no crash. +def test_bare_recall_finds_archived_session(monkeypatch, capsys, isolated): + from shellllm.archive import Archive + Archive().ingest_session( + cmd="ask", + terminal_id="t1", + created_at=1.0, + last_used=2.0, + last_pwd="/tmp", + last_date="2026-06-08", + turn_count=1, + messages=[ + {"role": "user", "content": "how do I use ripgrep"}, + {"role": "assistant", "content": "rg pattern path"}, + ], + ) + assert _run(["ripgrep"], monkeypatch) == 0 + out = capsys.readouterr().out + assert "ripgrep" in out.lower() -def test_filter_no_hits_includes_scope_in_message(monkeypatch, capsys, isolated): - assert _run(["--ask", "totallyabsent"], monkeypatch) == 0 + +def test_bare_multiword_query_is_joined(monkeypatch, capsys, isolated): + from shellllm.archive import Archive + + Archive().ingest_session( + cmd="ask", + terminal_id="t1", + created_at=1.0, + last_used=2.0, + last_pwd="/tmp", + last_date="2026-06-08", + turn_count=1, + messages=[ + {"role": "user", "content": "how do docker volumes work"}, + {"role": "assistant", "content": "they mount paths into the container"}, + ], + ) + assert _run(["docker", "volumes"], monkeypatch) == 0 out = capsys.readouterr().out - assert "in `ask` sessions" in out + assert "docker" in out.lower() diff --git a/zsh/shellllm.zsh b/zsh/shellllm.zsh index 6fcc7bb..2077a3e 100644 --- a/zsh/shellllm.zsh +++ b/zsh/shellllm.zsh @@ -77,16 +77,18 @@ alias '?'='noglob _shellllm_ask_fn' # ─── `???` — memory layer: long-term facts + cross-session recall. # # ??? bare query → search archived sessions -# ??? add pin a long-term fact -# ??? list list facts -# ??? drop drop fact #n -# ??? recall explicit recall (use when the query starts with -# a word that's also a subcommand) -# ??? status counts -# ??? help usage +# ??? --add pin a long-term fact +# ??? --list list facts +# ??? --drop drop fact #n +# ??? --status counts +# ??? --ask recall only `?` sessions +# ??? --comma recall only `,` sessions +# ??? --help usage # -# `noglob` is needed because `?` is a zsh glob char; aliasing `???` -# directly is fine because alias expansion runs before globbing. +# Every operation other than bare-query recall is a flag — no +# bare-word verbs. `noglob` is needed because `?` is a zsh glob char; +# aliasing `???` directly is fine because alias expansion runs before +# globbing. function _shellllm_recall_fn() { ${=SHELLLM_RECALL} "$@" } From b934b02b55e38ed2e5eb8848747d8f956f983d36 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fran=C3=A7ois=20Chastel?= Date: Mon, 8 Jun 2026 18:10:32 -0400 Subject: [PATCH 4/5] feat(???): --archives / --show for browsing the archive; fix , session hint MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two new mode flags on ???: - --archives [n] list n most-recent archived sessions (default 20), one row per archive with id · cmd · ts · pwd · snippet. Honors --ask / --comma to narrow by command. - --show print the full transcript of one archive. Together they fill the "how do I see what's in here?" gap that --status left open (it only reported counts). Filter compatibility - --archives accepts --ask / --comma (per-command browse makes sense) - --show targets exactly one row, so filters error there - Facts ops stay global-only (--add / --list / --drop / --status) Visual bug fix in `,` The "↻ refining — turn N" / "↻ idle session expired" hints in comma were written via `rich.Console.print`, which strips embedded ESC bytes as a safety measure — leaving `[2m[36m...[0m` visible as literal text. Replaced with a sys.stderr.write helper matching ask._note. Added a regression test that asserts ESC bytes survive in stderr output. 197 passing tests (was 181), ruff clean. --- README.md | 19 ++++-- src/shellllm/archive.py | 61 +++++++++++++++++ src/shellllm/comma.py | 17 ++++- src/shellllm/recall.py | 97 +++++++++++++++++++++++++-- tests/test_comma_session.py | 16 +++++ tests/test_recall.py | 130 +++++++++++++++++++++++++++++++++++- 6 files changed, 328 insertions(+), 12 deletions(-) diff --git a/README.md b/README.md index 0c04408..850244f 100644 --- a/README.md +++ b/README.md @@ -145,9 +145,20 @@ Filter recall by which command produced the session: ??? docker volumes # both (default) ``` -Mode flags (`--add` / `--list` / `--drop` / `--status`) are mutually -exclusive. Filter flags only apply to recall — combining `--ask` with -`--list` errors because facts are global. +Browse the archive directly (no FTS query, useful for "what's even in +there?"): + +```sh +??? --archives # 20 most-recent (id, cmd, ts, pwd, snippet) +??? --archives 50 # show 50 +??? --ask --archives # filter to one command's archives +??? --show 42 # full transcript of archive #42 +``` + +Mode flags (`--add` / `--list` / `--drop` / `--status` / `--archives` +/ `--show`) are mutually exclusive. Filter flags (`--ask` / `--comma`) +only apply to recall and `--archives` — combining `--ask` with `--list` +errors because facts are global. The archive at `~/.cache/shellllm/archive.db` gets populated automatically whenever a session expires or you call `? --new` / @@ -281,7 +292,7 @@ Every file read goes through `safe_fs.safe_read`. Four rules, all enforced: Reads cap at 1 MB and use `O_NOFOLLOW` on the final component as a belt against a resolve-then-open symlink race. ```sh -pytest -v # 181 tests; safe_fs alone covers symlinks, traversal, denylist, lookalikes, truncation +pytest -v # 197 tests; safe_fs alone covers symlinks, traversal, denylist, lookalikes, truncation ``` ## What's deliberately not built diff --git a/src/shellllm/archive.py b/src/shellllm/archive.py index 0eedd58..884cf55 100644 --- a/src/shellllm/archive.py +++ b/src/shellllm/archive.py @@ -196,6 +196,67 @@ def count(self) -> int: row = conn.execute("SELECT COUNT(*) FROM archives").fetchone() return int(row[0]) if row else 0 + # ── Browse --------------------------------------------------------- + + def recent( + self, + *, + limit: int = 20, + cmd_filter: str | None = None, + ) -> list[ArchiveHit]: + """Most-recent archives, no FTS query — for `??? --archives`. + + Use this to browse what's been archived without a specific + recall query in mind. The snippet field is synthesized from + the head of the content (FTS5 ``snippet()`` is only available + on a MATCH expression). + """ + sql = "SELECT id, archived_at, cmd, last_pwd, last_date, turn_count, content FROM archives" + params: list[Any] = [] + if cmd_filter: + sql += " WHERE cmd = ?" + params.append(cmd_filter) + sql += " ORDER BY archived_at DESC LIMIT ?" + params.append(limit) + + with self._conn() as conn: + rows = conn.execute(sql, params).fetchall() + + return [ + ArchiveHit( + id=int(r[0]), + archived_at=float(r[1]), + cmd=str(r[2]), + last_pwd=str(r[3] or ""), + last_date=str(r[4] or ""), + turn_count=int(r[5]), + content=str(r[6]), + snippet=_truncate_snippet(str(r[6])), + ) + for r in rows + ] + + def get(self, archive_id: int) -> ArchiveHit | None: + """Fetch one archive by id, or None — for `??? --show `.""" + with self._conn() as conn: + row = conn.execute( + "SELECT id, archived_at, cmd, last_pwd, last_date, " + "turn_count, content FROM archives WHERE id = ?", + (archive_id,), + ).fetchone() + if row is None: + return None + return ArchiveHit( + id=int(row[0]), + archived_at=float(row[1]), + cmd=str(row[2]), + last_pwd=str(row[3] or ""), + last_date=str(row[4] or ""), + turn_count=int(row[5]), + content=str(row[6]), + snippet=_truncate_snippet(str(row[6])), + ) + # ── Searches ------------------------------------------------------- def search( diff --git a/src/shellllm/comma.py b/src/shellllm/comma.py index 4191a6b..9d35408 100644 --- a/src/shellllm/comma.py +++ b/src/shellllm/comma.py @@ -65,6 +65,19 @@ _err = Console(stderr=True) +def _note(text: str) -> None: + """Dim-cyan one-liner straight to stderr. + + We bypass Rich here because ``rich.Console.print`` strips embedded + ANSI escape characters as a safety measure (so untrusted strings + can't redirect the cursor). That's the right default for rendered + text, but our hint is fixed-content and we want the codes + interpreted by the terminal — so we write to stderr directly. + """ + sys.stderr.write(f"{_DIM}{_CYAN}↻ {text}{_RESET}\n") + sys.stderr.flush() + + def _safe_embed(text: str) -> list[float] | None: try: return embed_text(text) @@ -302,12 +315,12 @@ def _consume_flag(flag: str) -> bool: return 2 if expired: - _err.print(f"{_DIM}{_CYAN}↻ idle session expired — starting fresh{_RESET}") + _note("idle session expired — starting fresh") first_turn = session.is_empty() resumed = not first_turn if resumed: - _err.print(f"{_DIM}{_CYAN}↻ refining — turn {session.meta.turn_count + 1}{_RESET}") + _note(f"refining — turn {session.meta.turn_count + 1}") messages, new_history_with_user = _build_messages( session=session, prompt=prompt, first_turn=first_turn, resumed=resumed diff --git a/src/shellllm/recall.py b/src/shellllm/recall.py index ee462ef..12cdfa0 100644 --- a/src/shellllm/recall.py +++ b/src/shellllm/recall.py @@ -55,7 +55,14 @@ def _safe_embed(text: str) -> list[float] | None: # Mutually-exclusive "do this instead of recall" flags. -_MODE_FLAGS = ("--add", "--list", "--drop", "--status") +_MODE_FLAGS = ("--add", "--list", "--drop", "--status", "--archives", "--show") + +# Mode flags that can't be narrowed by ``--ask`` / ``--comma``. +# - facts ops (``--add``/``--list``/``--drop``) are global +# - ``--status`` counts are global +# - ``--show `` already targets exactly one row +# Only ``--archives`` and bare recall accept filters. +_GLOBAL_MODES = frozenset({"--add", "--list", "--drop", "--status", "--show"}) # Restrict recall to a single asking surface. Extend when a new asking # command is added. @@ -64,6 +71,9 @@ def _safe_embed(text: str) -> list[float] | None: "--comma": "comma", } +# Default for ``--archives`` when no count argument is given. +_DEFAULT_ARCHIVES_LIMIT = 20 + def _print_usage(label: str = "???") -> None: sys.stdout.write( @@ -74,10 +84,27 @@ def _print_usage(label: str = "???") -> None: f" {label} --list list saved facts\n" f" {label} --drop drop fact #n\n" f" {label} --status show counts\n" + f" {label} --archives [n] list n recent archived sessions\n" + f" (default 20; --ask / --comma filter)\n" + f" {label} --show print full transcript of one archive\n" f" {label} --help show this message\n" ) +def _format_archive_row(hit) -> str: + """One-line summary for ``--archives`` listings: id · cmd · ts · pwd · snippet.""" + when = datetime.fromtimestamp(hit.archived_at).strftime("%Y-%m-%d %H:%M") + parts = [ + f"{_DIM}#{hit.id:<4}{_RESET}", + f"{_CYAN}{hit.cmd}{_RESET}", + f"{_DIM}{when}{_RESET}", + ] + if hit.last_pwd: + parts.append(f"{_DIM}{hit.last_pwd}{_RESET}") + header = " · ".join(parts) + return f"{header}\n {hit.snippet}\n\n" + + def _format_recall_hit(idx: int, hit) -> str: when = datetime.fromtimestamp(hit.archived_at).strftime("%Y-%m-%d %H:%M") parts = [ @@ -167,6 +194,60 @@ def _cmd_status(memory: MemoryStore, archive: Archive, rest: list[str]) -> int: return 0 +def _cmd_archives(archive: Archive, rest: list[str], cmd_filter: str | None) -> int: + """List recent archived sessions. Optional count arg + --ask/--comma filter.""" + limit = _DEFAULT_ARCHIVES_LIMIT + if len(rest) > 1: + sys.stderr.write(f"{_RED}??? error:{_RESET} `--archives` takes at most one count\n") + return 2 + if rest: + try: + limit = int(rest[0]) + except ValueError: + sys.stderr.write(f"{_RED}??? error:{_RESET} `--archives` count must be an integer\n") + return 2 + if limit <= 0: + sys.stderr.write(f"{_RED}??? error:{_RESET} `--archives` count must be positive\n") + return 2 + + hits = archive.recent(limit=limit, cmd_filter=cmd_filter) + if not hits: + scope = f" in `{cmd_filter}` sessions" if cmd_filter else "" + print(f"(no archived sessions{scope})") + return 0 + for hit in hits: + sys.stdout.write(_format_archive_row(hit)) + return 0 + + +def _cmd_show(archive: Archive, rest: list[str]) -> int: + """Print one archive's full transcript by id.""" + if not rest: + sys.stderr.write(f"{_RED}??? error:{_RESET} `--show` needs an archive id\n") + return 2 + if len(rest) > 1: + sys.stderr.write(f"{_RED}??? error:{_RESET} `--show` takes one id\n") + return 2 + try: + archive_id = int(rest[0]) + except ValueError: + sys.stderr.write(f"{_RED}??? error:{_RESET} id must be an integer\n") + return 2 + + hit = archive.get(archive_id) + if hit is None: + sys.stderr.write(f"{_RED}??? error:{_RESET} no archive with id {archive_id}\n") + return 2 + + when = datetime.fromtimestamp(hit.archived_at).strftime("%Y-%m-%d %H:%M:%S") + sys.stdout.write( + f"{_DIM}archive #{hit.id} · {hit.cmd} · {when} · " + f"{hit.turn_count} turns · {hit.last_pwd or '?'}{_RESET}\n\n" + ) + sys.stdout.write(hit.content.strip() + "\n") + return 0 + + def _collect_mode(argv: list[str]) -> tuple[str | None, int]: """Pull a single mode flag from ``argv``. Multi-mode → error code.""" present = [f for f in _MODE_FLAGS if f in argv] @@ -212,11 +293,13 @@ def main() -> int: memory = MemoryStore() archive = Archive() - # Filter flags only apply to recall paths. Combining with a mode - # flag is almost certainly a typo — facts and counts are global. - if cmd_filter is not None and mode is not None: + # Filter flags only apply to recall and to --archives (which is + # per-command browseable). Combining with a global-state mode is + # almost certainly a typo — facts and counts are global. + if cmd_filter is not None and mode in _GLOBAL_MODES: sys.stderr.write( - f"{_RED}??? error:{_RESET} filter flags only apply to recall, not `{mode}`\n" + f"{_RED}??? error:{_RESET} filter flags only apply to recall " + f"and --archives, not `{mode}`\n" ) return 2 @@ -228,6 +311,10 @@ def main() -> int: return _cmd_drop(memory, argv) if mode == "--status": return _cmd_status(memory, archive, argv) + if mode == "--archives": + return _cmd_archives(archive, argv, cmd_filter) + if mode == "--show": + return _cmd_show(archive, argv) # No mode flag → recall path. Filter is optional. if not argv: diff --git a/tests/test_comma_session.py b/tests/test_comma_session.py index dd5c5d9..9470eed 100644 --- a/tests/test_comma_session.py +++ b/tests/test_comma_session.py @@ -166,6 +166,22 @@ def test_expired_session_archives_to_recall( assert any(h.cmd == "comma" for h in hits) +def test_resume_hint_writes_raw_ansi_not_rich_markup( + monkeypatch, capsys, isolated, fake_model, auto_pick +): + """The "↻ refining" hint must reach the terminal as a real ANSI + sequence (starting with ESC), not the literal ``[2m`` text Rich's + Console.print would render after stripping the ESC byte.""" + + _run(["first"], monkeypatch) + capsys.readouterr() + _run(["second"], monkeypatch) + err = capsys.readouterr().err + # ESC byte must be present; literal "[2m[36m" never appears alone. + assert "\x1b[2m" in err or "\x1b[36m" in err + assert "↻ refining" in err + + def test_redirect_for_ask_remember(monkeypatch, capsys, isolated): """`,` doesn't share `?`'s deprecation hints — it has its own surface.""" diff --git a/tests/test_recall.py b/tests/test_recall.py index 08e5597..3eb38c8 100644 --- a/tests/test_recall.py +++ b/tests/test_recall.py @@ -137,12 +137,20 @@ def test_two_filter_flags_errors(monkeypatch, capsys, isolated): def test_filter_with_mode_flag_errors(monkeypatch, capsys, isolated): - """Filters only apply to recall, not to fact management.""" + """Filters apply only to recall and --archives, not to fact management.""" assert _run(["--ask", "--list"], monkeypatch) == 2 err = capsys.readouterr().err assert "only apply to recall" in err +def test_filter_with_archives_is_allowed(monkeypatch, capsys, isolated): + """--archives is the one mode flag that accepts filters.""" + assert _run(["--ask", "--archives"], monkeypatch) == 0 + out = capsys.readouterr().out + # No data → scoped empty-state message confirms the filter took effect. + assert "in `ask` sessions" in out + + # ── Filter flags --------------------------------------------------------- @@ -231,6 +239,126 @@ def test_bare_recall_finds_archived_session(monkeypatch, capsys, isolated): assert "ripgrep" in out.lower() +def _seed_archive_rows(n: int = 3, cmd: str = "ask") -> None: + """Insert N distinct archive rows for browse-mode tests.""" + from shellllm.archive import Archive + + a = Archive() + for i in range(n): + a.ingest_session( + cmd=cmd, + terminal_id="t1", + created_at=1.0 + i, + last_used=2.0 + i, + last_pwd=f"/tmp/p{i}", + last_date="2026-06-08", + turn_count=1, + messages=[ + {"role": "user", "content": f"question number {i}"}, + {"role": "assistant", "content": f"answer body {i}"}, + ], + archived_at=1000.0 + i, + ) + + +# ── --archives ---------------------------------------------------------- + + +def test_archives_empty(monkeypatch, capsys, isolated): + assert _run(["--archives"], monkeypatch) == 0 + assert "no archived sessions" in capsys.readouterr().out + + +def test_archives_lists_recent_descending(monkeypatch, capsys, isolated): + _seed_archive_rows(n=3) + assert _run(["--archives"], monkeypatch) == 0 + out = capsys.readouterr().out + # Most-recent first: "question number 2" should appear before "0". + idx2 = out.find("answer body 2") + idx0 = out.find("answer body 0") + assert idx2 != -1 and idx0 != -1 + assert idx2 < idx0 + + +def test_archives_respects_limit(monkeypatch, capsys, isolated): + _seed_archive_rows(n=5) + assert _run(["--archives", "2"], monkeypatch) == 0 + out = capsys.readouterr().out + assert out.count("answer body") == 2 + + +def test_archives_limit_must_be_int(monkeypatch, capsys, isolated): + assert _run(["--archives", "abc"], monkeypatch) == 2 + assert "integer" in capsys.readouterr().err + + +def test_archives_limit_must_be_positive(monkeypatch, capsys, isolated): + assert _run(["--archives", "0"], monkeypatch) == 2 + assert "positive" in capsys.readouterr().err + + +def test_archives_filter_by_cmd(monkeypatch, capsys, isolated): + _seed_archive_rows(n=2, cmd="ask") + _seed_archive_rows(n=2, cmd="comma") + assert _run(["--ask", "--archives"], monkeypatch) == 0 + out = capsys.readouterr().out + # All shown rows must be `ask` — comma rows excluded. + assert "ask" in out + assert "comma" not in out + + +def test_archives_filter_empty_message(monkeypatch, capsys, isolated): + assert _run(["--comma", "--archives"], monkeypatch) == 0 + assert "in `comma` sessions" in capsys.readouterr().out + + +def test_archives_too_many_args_errors(monkeypatch, capsys, isolated): + assert _run(["--archives", "10", "20"], monkeypatch) == 2 + assert "at most one count" in capsys.readouterr().err + + +# ── --show -------------------------------------------------------------- + + +def test_show_requires_id(monkeypatch, capsys, isolated): + assert _run(["--show"], monkeypatch) == 2 + assert "needs an archive id" in capsys.readouterr().err + + +def test_show_id_must_be_int(monkeypatch, capsys, isolated): + assert _run(["--show", "abc"], monkeypatch) == 2 + assert "integer" in capsys.readouterr().err + + +def test_show_missing_id_errors(monkeypatch, capsys, isolated): + assert _run(["--show", "999"], monkeypatch) == 2 + assert "no archive with id" in capsys.readouterr().err + + +def test_show_prints_full_transcript(monkeypatch, capsys, isolated): + _seed_archive_rows(n=1) + assert _run(["--show", "1"], monkeypatch) == 0 + out = capsys.readouterr().out + assert "question number 0" in out + assert "answer body 0" in out + # Header shows cmd, turn count, pwd. + assert "ask" in out + assert "/tmp/p0" in out + + +def test_show_too_many_args_errors(monkeypatch, capsys, isolated): + assert _run(["--show", "1", "2"], monkeypatch) == 2 + assert "takes one id" in capsys.readouterr().err + + +def test_show_with_filter_does_not_apply(monkeypatch, capsys, isolated): + """--show is global; combining it with --ask shouldn't be valid since + facts/single-row reads aren't per-command operations.""" + _seed_archive_rows(n=1) + assert _run(["--ask", "--show", "1"], monkeypatch) == 2 + assert "only apply to recall" in capsys.readouterr().err + + def test_bare_multiword_query_is_joined(monkeypatch, capsys, isolated): from shellllm.archive import Archive From 35f4aa41d5bb8a87c1bf981efc17a75903374704 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fran=C3=A7ois=20Chastel?= Date: Mon, 8 Jun 2026 23:30:16 -0400 Subject: [PATCH 5/5] docs: rewrite README around the four-glyph CLI Restructured for the GitHub-frontpage skim test: - Single tagline above the fold - Hero block showing the three core commands working back-to-back - Quick-reference table for the four glyphs - Tiers as a side-by-side comparison, not prose - 'Why this exists' moved up to anchor the contrarian take (offline, local 27B, no per-question cost) - Configuration as one long table instead of paragraphs Architecture / hard wall / dev sections still present but moved below the fold so they don't drown the install path. --- README.md | 380 +++++++++++++++++++----------------------------------- 1 file changed, 136 insertions(+), 244 deletions(-) diff --git a/README.md b/README.md index 850244f..ccf05f3 100644 --- a/README.md +++ b/README.md @@ -4,20 +4,29 @@ [![python](https://img.shields.io/badge/python-3.10%2B-blue)](https://www.python.org/) [![license](https://img.shields.io/badge/license-MIT-blue)](LICENSE) -Local-LLM zsh helpers. +> Local LLM at your zsh prompt. Four characters, no API key, works offline. -Four commands, each does one thing: +Drop English at your prompt and get a real shell command. Ask the model a question without breaking flow. Search every past conversation by content. The whole CLI is four punctuation glyphs — `,` `?` `??` `???` — because the best terminal UI is the one that fits next to `cd` and `ls`. -- **`, `** — proposes 3–5 shell commands with one-line notes, you pick one in `fzf`, it lands on your prompt line via `print -z`. Never auto-executes. Sticky per-pane session so follow-ups refine the prior list (`, the same but only the running ones`). -- **`? `** — small read-only agent with three tools: `read_file` (gated by a filesystem hard wall), `web_search` (DuckDuckGo) and `fetch_url` (follow a result into its page, plain-text). Searches only when the model decides it needs to — pass `? --web ` to force it. Answer streams as live-rendered markdown. Each terminal pane keeps its own sticky conversation; follow-ups continue automatically until 30 min of idle (or `? --new`). -- **`??? `** — the memory layer. `??? ` searches the archive of past sessions. Flags manage facts: `??? --add `, `??? --list`, `??? --drop `, `??? --status`. Bare-word search terms (like `??? list`) recall the word — flags are the only "verbs". -- **`??`** — start (or stop / list / status) the local `llama-server` backend, with named tiers for speed-vs-quality. `?? --start-embed` boots a second `llama-server` in embedding mode for hybrid semantic recall. +```text +$ , find the five largest files here + ▶ du -ah . | sort -hr | head -5 · sized + sorted, one screen + find . -type f -printf '%s %p\n' · raw bytes, no sort + ls -lhS | head -5 · ls only, no recursion + enter: drop on prompt · esc: cancel -Runs against a local `llama-server`. No frontier model, no API key, works with wifi off. +$ ? what does git stash do + Git stash temporarily shelves changes in your working copy so you can + work on something else, then come back and re-apply them later... -## Quick start +$ ??? git stash + #42 · ask · 2026-06-08 11:14 · ~/proj + Q: what does git stash do A: shelves changes temporarily... +``` + +No API key. No data leaves your machine. Works with WiFi off (except `? --web`). -### Install with Homebrew (recommended) +## Install ```sh brew install FrancoisChastel/shellllm/shellllm @@ -25,304 +34,168 @@ echo 'source "$(brew --prefix)/share/shellllm/shellllm.zsh"' >> ~/.zshrc exec zsh ``` -That pulls `llama.cpp`, `fzf`, and the two CLIs (`shellllm-comma`, `shellllm-ask`). See [docs/HOMEBREW.md](docs/HOMEBREW.md) for the maintainer release flow. - -### Install from source +That pulls `llama.cpp`, `fzf`, and the CLIs. Then start the model server: ```sh -# 1. install -python3 -m venv .venv -.venv/bin/pip install -e . - -# 2. wire zsh -echo "export PATH=\"$PWD/.venv/bin:\$PATH\"" >> ~/.zshrc -echo "source $PWD/zsh/shellllm.zsh" >> ~/.zshrc -exec zsh - -# 3. start the backend (downloads not handled here — see "Models" below) -?? # default tier (balanced) -?? --start fast # MoE + MTP, fastest -?? --list # what's available locally vs. needs download - -# 4. use it -, find the five largest files under this directory -, the same but only ones modified today # refines the prior , — sticky session -? in markdown, what does git stash do? -? --web latest stable release of ripgrep # force web-first this turn -??? --add I prefer ripgrep over grep # long-term fact, used by all asks -??? ripgrep # search past sessions across panes +?? # balanced tier (default) +?? --list # see what's downloaded vs. what isn't ``` -### Upgrading an existing install +From source: jump to [Install from source](#install-from-source). -Both code paths track this repo as the source of truth: +## The four commands -- **Source install (`pip install -e .` + `source zsh/shellllm.zsh`)** — - Python entry-points pick up edits automatically; the zsh helper does - too. After a `git pull` that changes `zsh/shellllm.zsh` you only need - to re-source it: - ```sh - exec zsh - ``` -- **Homebrew install** — bump the formula or wait for the next release. +| Cmd | What | Example | +|---|---|---| +| `, ` | Propose shell commands, pick one in fzf, drop on prompt. Never executes. | `, the five largest files here` | +| `? ` | Ask the model. Streams markdown. Sticky per-pane session. | `? what does git stash do` | +| `???` | Memory & recall. Bare query searches archive. Flags manage facts. | `??? --add I prefer ripgrep` | +| `??` | Start / stop / status the local `llama-server`. | `?? --start fast` | -To activate semantic recall once the upgrade is in, follow the -[Local embeddings](#local-embeddings) section. +`,` and `?` are **conversational** — each terminal pane keeps its own thread. Type `, the same but with json output` and the model knows what "the same" means. After 30 min idle the thread auto-rotates so a forgotten tab doesn't bleed stale context. -## Tiers +`?` has tools: read files (filesystem-gated), DuckDuckGo search, fetch URL. Force web-first with `? --web `. -| Tier | Model | Notes | -| --- | --- | --- | -| `fast` | `unsloth/Qwen3.6-35B-A3B-MTP-GGUF` | MoE with 3B active params + MTP self-speculative decoding (`--spec-type draft-mtp`). Fastest on Apple Silicon. | -| `balanced` | `unsloth/Qwen3.6-27B-GGUF` (Q4_K_M) | Dense 27B. Default. | -| `smart` | `unsloth/Qwen3-Coder-Next-GGUF` | Latest coder-tuned model, ideal for shell/agent tasks. | - -Download a tier: +`???` is the durable layer. Bare queries hit the archive of every past session (BM25; semantic if you've added embeddings). Flags pin long-term facts that get injected into every `?` system prompt. ```sh -huggingface-cli download unsloth/Qwen3-Coder-Next-GGUF -?? --start smart +??? --add the project uses python 3.11 and uv +??? --list # facts you've pinned +??? --archives # 20 most-recent archived sessions +??? --show 42 # full transcript of archive #42 +??? --ask docker volumes # recall, only `?` sessions +??? --comma docker volumes # recall, only `,` sessions +??? docker volumes # both ``` -`??` resolves the GGUF inside your HuggingFace cache automatically — no path config required. +## Model tiers -## Sessions +Three preset tiers, named for what you'd reach for: -Each terminal pane gets its own sticky conversation, one per asking -command. `?` and `,` each keep their own thread so refining a `,` -proposal doesn't pollute the Q&A you were having with `?`. +| Tier | Model | Notes | +|---|---|---| +| `fast` | `unsloth/Qwen3.6-35B-A3B-MTP-GGUF` | MoE, 3B active params + self-speculative MTP. Fastest on Apple Silicon. | +| `balanced` | `unsloth/Qwen3.6-27B-GGUF` (Q4_K_M) | Dense 27B. Default. | +| `smart` | `unsloth/Qwen3-Coder-Next-GGUF` | Latest coder-tuned model. Best for shell/agent work. | ```sh -? what was that flag for ripgrep again -? and how do I use it with json output -? --web latest stable release of ripgrep # force web-first this turn -? --history # transcript of this pane's session -? --new what's a good hash for cache keys # start fresh -? --reset # drop the current session -? --compact # force-compact older turns into a summary - -, list all docker containers -, the same but only the running ones # refines the prior `,` proposal -, --new find the largest files # starts a fresh `,` thread +huggingface-cli download unsloth/Qwen3-Coder-Next-GGUF +?? --start smart ``` -The pane is identified from `TERM_SESSION_ID` (Terminal.app / iTerm), -`TMUX_PANE`, or `WINDOWID` — whichever your terminal sets. After 30 min -idle the session auto-rotates so a forgotten tab doesn't bleed stale -context into the next turn. - -When the conversation crosses ~80% of `SHELLLM_CTX`, older turns are -auto-summarized into a single `` block using the same -local model; the most recent 4 turns stay verbatim. +`??` finds the GGUF inside your HuggingFace cache — no path config required. -## `???` — memory and recall +## Sessions -Three question marks reads as *"I'm trying to remember…"*. That's -exactly what this verb does. The most-used path is a bare query — -search the archive of past sessions across all panes and days: +Each pane × command gets a sticky JSONL session at `~/.cache/shellllm/sessions/`. Pane identity is `TERM_SESSION_ID` (Terminal.app / iTerm) → `TMUX_PANE` → `WINDOWID` → `$PPID`, first one that resolves. ```sh -??? what was that grep flag again -??? docker volumes +? what was that flag for ripgrep +? and with json output # ← model still knows "ripgrep" +? --history # transcript of this pane +? --new # start a fresh session +? --reset # drop the current one +? --compact # force compaction now ``` -Every other operation is a flag — no bare-word verbs, so any -non-flag input is unambiguously a recall query: - -```sh -??? --add I prefer ripgrep over grep # pin a long-term fact -??? --list # see them -??? --drop 2 # remove fact #2 -??? --status # counts: facts + archives -??? --help -``` +When the conversation crosses 80% of `SHELLLM_CTX`, older turns are auto-summarized into a single `` block by the same local model; the most recent 4 stay verbatim. -Filter recall by which command produced the session: +Every expired or `--new`'d session flows into `~/.cache/shellllm/archive.db` (sqlite + FTS5) so `???` can search across panes and days. -```sh -??? --ask docker volumes # only `?` sessions -??? --comma docker volumes # only `,` sessions -??? docker volumes # both (default) -``` +## Semantic recall (optional) -Browse the archive directly (no FTS query, useful for "what's even in -there?"): +Recall works in BM25-only mode out of the box. Adding a tiny embedding server upgrades it to **hybrid semantic + BM25** (fused with Reciprocal Rank Fusion): ```sh -??? --archives # 20 most-recent (id, cmd, ts, pwd, snippet) -??? --archives 50 # show 50 -??? --ask --archives # filter to one command's archives -??? --show 42 # full transcript of archive #42 +huggingface-cli download Qwen/Qwen3-Embedding-0.6B-GGUF +?? --start-embed # second llama-server in --embedding mode on :8081 +export SHELLLM_AUTO_RECALL=1 # auto-inject prior context on first-turn questions ``` -Mode flags (`--add` / `--list` / `--drop` / `--status` / `--archives` -/ `--show`) are mutually exclusive. Filter flags (`--ask` / `--comma`) -only apply to recall and `--archives` — combining `--ask` with `--list` -errors because facts are global. - -The archive at `~/.cache/shellllm/archive.db` gets populated -automatically whenever a session expires or you call `? --new` / -`? --reset` / `, --new` / `, --reset`. - -Recall always works in **BM25-only mode** — no extra setup, no extra -processes. Adding a local embedding server unlocks **hybrid -semantic + BM25 search** (RRF-fused) so you find prior conversations -by meaning, not just keywords. - - - -### Local embeddings - -Start a second `llama-server` instance running in embedding mode on -port 8081. Three tiers ship out of the box: +Three embedding tiers: | Tier | Model | Notes | -| --- | --- | --- | +|---|---|---| | `tiny` | `Qwen/Qwen3-Embedding-0.6B-GGUF` | Same family as the chat tiers (default). | | `bge` | `ChristianAzinn/bge-small-en-v1.5-gguf` | Tiny English-only, very fast. | -| `nomic` | `nomic-ai/nomic-embed-text-v1.5-GGUF` | Strong general-purpose retrieval. | - -```sh -huggingface-cli download Qwen/Qwen3-Embedding-0.6B-GGUF -?? --start-embed tiny # starts on :8081 -?? --status-embed -?? --list-embed -?? --stop-embed -``` - -shellllm auto-detects the server via `SHELLLM_EMBED_URL` -(`http://127.0.0.1:8081` by default). When it's reachable: +| `nomic` | `nomic-ai/nomic-embed-text-v1.5-GGUF` | Strong general-purpose. | -- new archive rows get a normalized fp32 embedding written alongside - the FTS5 entry; -- `???` (bare query) and `? --auto-recall` embed the query and add - cosine-sim candidates to the BM25 results, fused via Reciprocal Rank - Fusion; -- mismatched embedding dims (e.g. swapping the model later) are - silently skipped — old rows still serve BM25 hits. +Mismatched embedding dims (when you swap models) are silently skipped — old rows still serve BM25 hits. -Auto-recall is opt-in: +## Cross-session memory (optional) -```sh -export SHELLLM_AUTO_RECALL=1 # global on -? --no-auto-recall # off for one call -? --auto-recall # on for one call (overrides env) -``` - -## Optional: cross-session memory via claude-mem - -If you also use [claude-mem](https://github.com/thedotmack/claude-mem) -in its server-beta mode, shellllm can write each turn as an -observation and pull in relevant prior context on a fresh session. -Nothing else changes; if the env vars aren't set, the integration is -inert. +If you also use [claude-mem](https://github.com/thedotmack/claude-mem) in server-beta mode, shellllm writes each turn as an observation and pulls relevant prior context on a fresh session: ```sh -export CLAUDE_MEM_SERVER_BETA_URL="https://your-claude-mem-host" +export CLAUDE_MEM_SERVER_BETA_URL="https://your-host" export CLAUDE_MEM_SERVER_BETA_API_KEY="..." export CLAUDE_MEM_SERVER_BETA_PROJECT_ID="..." ``` -On first use in a process you'll see a one-line dim hint on stderr. -What we do with those creds: - -- **Write**: every successful `?` / `???` turn becomes a - `shellllm-turn` observation; `? --remember ` mirrors as a - `user-fact` observation. Writes are fire-and-forget on a daemon - thread — they never block your prompt and a network failure is - silently dropped. -- **Read**: only on the first turn of a brand-new session, we hit - `/v1/context` with your question and inject the returned text as - a `` system block. - -Controls: - -| Knob | What it does | -| --- | --- | -| `SHELLLM_CLAUDE_MEM=0` | Force-disable even when configured | -| `? --no-mem ` | Skip integration for one call | -| `? --mem ` | Re-enable for one call (overrides env opt-out) | - -shellllm's local `--remember` list (`~/.cache/shellllm/memory.jsonl`) -stays the source of truth for offline use; claude-mem just gets a -copy. - -## Architecture - -``` -src/shellllm/ -├── safe_fs.py filesystem hard wall — $HOME/$PWD + inside-HOME denylist -├── client.py llama-server HTTP client (one-shot + streaming) -├── comma.py , — JSON-schema → fzf picker → stdout -├── ask.py ? — streaming agent loop, --web flag, live markdown render -├── recall.py ??? — memory layer: bare-query recall + fact flags -├── comma.py , — JSON-schema → fzf picker, sticky session for refinement -├── session.py per-pane conversation persistence (JSONL + idle TTL) -├── memory.py long-term fact store backing `??? add` / `??? list` -├── compact.py summary-buffer compaction over the same local model -├── context.py date/OS/timezone prelude (re-injected on PWD/date change) -├── archive.py sqlite FTS5 + optional embeddings for `??? ` -├── embed.py client for a local llama-server in --embedding mode -├── claude_mem.py optional adapter for claude-mem server-beta (observations + context) -└── web.py stdlib DuckDuckGo scraper + fetch_url with SSRF guard -tests/test_safe_fs.py filesystem-wall coverage -tests/test_session.py TTY id + TTL rotation + JSONL round-trip -tests/test_memory.py fact store + size cap + archive overflow -tests/test_compact.py compaction preserves turn boundaries -tests/test_archive.py FTS5 + cosine recall, RRF fusion, dim-mismatch tolerance -tests/test_embed.py embedding client + pack/unpack + cosine helpers -tests/test_recall.py ??? bare-query + flag-only dispatch -tests/test_ask_web_flag.py ? --web swaps to web-first system prompt -tests/test_comma_session.py , refines across turns; archive on TTL -tests/test_claude_mem.py adapter gating + payload shape + error swallowing -tests/test_web.py URL safety + HTML extraction -zsh/shellllm.zsh function , + aliases ? , ?? , ??? -.github/workflows/ci.yml ruff + pytest on push & PR -``` +Without those vars the integration is inert. With them: writes are fire-and-forget on a daemon thread; reads happen only on the first turn of a new session; failures never propagate. ## The hard wall -Every file read goes through `safe_fs.safe_read`. Four rules, all enforced: +Every file read through `?` goes through `safe_fs.safe_read`. Four rules, all enforced: -1. **Canonicalize** with `.resolve(strict=True)`. Symlinks and `..` are flattened *before* containment is checked. +1. **Canonicalize** with `.resolve(strict=True)` — symlinks and `..` flattened before containment is checked. 2. **Contain** to `$HOME` or `$PWD`. Anywhere else refuses with `WallViolation`. -3. **Deny inside-HOME secrets.** Even within `$HOME`, paths under `.ssh`, `.aws`, `.gnupg`, `.kube`, `Library/Keychains`, `.netrc`, etc. refuse. Match is by path component — `.sshfoo` is allowed. +3. **Deny inside-HOME secrets.** Paths under `.ssh`, `.aws`, `.gnupg`, `.kube`, `Library/Keychains`, `.netrc`, etc. refuse. Match is by path component — `.sshfoo` is allowed. 4. **Regular files only.** Devices, fifos, sockets, directories refuse. -Reads cap at 1 MB and use `O_NOFOLLOW` on the final component as a belt against a resolve-then-open symlink race. +Reads cap at 1 MB and use `O_NOFOLLOW` as a belt against a resolve-then-open symlink race. ```sh -pytest -v # 197 tests; safe_fs alone covers symlinks, traversal, denylist, lookalikes, truncation +pytest -v # 197 tests; 38 dedicated to symlinks, traversal, denylist, lookalikes, truncation ``` -## What's deliberately not built - -- **GBNF prefix grammar** for `,`. JSON schema is enough for v1; the system prompt forbids the obvious destructive commands. -- **JavaScript rendering** for `fetch_url`. Pages are fetched as static HTML and reduced to text — SPAs that need JS to populate content will look empty. - -## Tunables (environment variables) +## Configuration | Variable | Default | Purpose | -| --- | --- | --- | +|---|---|---| | `SHELLLM_BASE_URL` | `http://127.0.0.1:8080` | llama-server endpoint | -| `SHELLLM_LLAMA_MODEL` | — | explicit GGUF path, overrides tier | -| `SHELLLM_PORT` | `8080` | server port | +| `SHELLLM_PORT` | `8080` | Server port | | `SHELLLM_NGL` | `99` | GPU offload layers | -| `SHELLLM_CTX` | `32768` | context window (tokens) | -| `SHELLLM_LOG` | `~/.cache/shellllm/llama-server.log` | server log path | +| `SHELLLM_CTX` | `32768` | Context window (tokens) | | `SHELLLM_TIMEOUT` | `120` | HTTP timeout (seconds) | -| `SHELLLM_EMBED_URL` | `http://127.0.0.1:8081` | local embedding server endpoint | -| `SHELLLM_EMBED_MODEL` | `local-embed` | model name passed to `/v1/embeddings` | -| `SHELLLM_EMBED_TIMEOUT` | `8` | embedding HTTP timeout (seconds) | -| `SHELLLM_EMBED_PORT` | `8081` | port `?? --start-embed` binds to | -| `SHELLLM_EMBED_CTX` | `2048` | context window for the embedding server | -| `SHELLLM_EMBED_LOG` | `~/.cache/shellllm/llama-embed.log` | embedding-server log path | +| `SHELLLM_LLAMA_MODEL` | — | Explicit GGUF path, overrides tier | +| `SHELLLM_LOG` | `~/.cache/shellllm/llama-server.log` | Server log path | +| `SHELLLM_EMBED_URL` | `http://127.0.0.1:8081` | Local embedding server endpoint | +| `SHELLLM_EMBED_PORT` | `8081` | Embedding server port | +| `SHELLLM_EMBED_CTX` | `2048` | Embedding context window | +| `SHELLLM_EMBED_MODEL` | `local-embed` | Model name passed to `/v1/embeddings` | +| `SHELLLM_EMBED_TIMEOUT` | `8` | Embedding HTTP timeout (seconds) | +| `SHELLLM_EMBED_LOG` | `~/.cache/shellllm/llama-embed.log` | Embedding-server log path | | `SHELLLM_ARCHIVE_DB` | `~/.cache/shellllm/archive.db` | sqlite archive of expired sessions | -| `SHELLLM_AUTO_RECALL` | unset | set to `1` to auto-inject archive hits on first-turn questions | -| `SHELLLM_CLAUDE_MEM` | unset (auto) | set to `0` to disable claude-mem integration even when configured | +| `SHELLLM_AUTO_RECALL` | unset | `1` to auto-inject archive hits on first-turn questions | +| `SHELLLM_CLAUDE_MEM` | unset (auto) | `0` to disable claude-mem even when configured | | `CLAUDE_MEM_SERVER_BETA_URL` | — | claude-mem server-beta base URL (enables integration) | -| `CLAUDE_MEM_SERVER_BETA_API_KEY` | — | bearer token for claude-mem server-beta | -| `CLAUDE_MEM_SERVER_BETA_PROJECT_ID` | — | project id observations are scoped to | +| `CLAUDE_MEM_SERVER_BETA_API_KEY` | — | Bearer token | +| `CLAUDE_MEM_SERVER_BETA_PROJECT_ID` | — | Project id observations are scoped to | + +## Install from source + +```sh +python3 -m venv .venv +.venv/bin/pip install -e . + +echo "export PATH=\"$PWD/.venv/bin:\$PATH\"" >> ~/.zshrc +echo "source $PWD/zsh/shellllm.zsh" >> ~/.zshrc +exec zsh +``` + +After a `git pull` you only need `exec zsh` to pick up updates to `zsh/shellllm.zsh`. Python entry-points reload automatically (editable install). + +## Why this exists + +You don't need to ship every "what does git stash do" question to a frontier model. The wifi will be off on the plane and you'll still want a hand. Every `tar -czvf` answer has been in your model's training data for two years. Claude Code is great but it lives in its own window — `cd ~/project && ?` shouldn't require a browser tab. + +The bet: a local 27B model is roughly equivalent to a frontier model for the questions you ask between `git commit` and `make test`. The wins — privacy, latency, offline availability, $0 per question — are real, every day. + +## What's deliberately not built + +- **GBNF prefix grammar for `,`.** JSON schema is enough for v1; the system prompt forbids the obvious destructive commands. +- **JS rendering for `fetch_url`.** Pages are fetched as static HTML and reduced to text — SPAs that need JS to populate content will look empty. ## Development @@ -332,6 +205,25 @@ ruff check . && ruff format --check . pytest -v ``` +Source layout: + +``` +src/shellllm/ +├── comma.py , — JSON-schema → fzf picker, sticky session +├── ask.py ? — streaming agent loop, --web flag, live markdown +├── recall.py ??? — memory layer: bare-query + fact/archive flags +├── session.py per-pane JSONL persistence (TTL + archive on rotation) +├── memory.py long-term facts behind --add / --list / --drop +├── archive.py sqlite FTS5 + optional embeddings +├── embed.py client for a local llama-server in --embedding mode +├── compact.py summary-buffer compaction over the same local model +├── claude_mem.py optional adapter for claude-mem server-beta +├── safe_fs.py filesystem hard wall +├── client.py llama-server HTTP client (one-shot + streaming) +├── context.py date/OS/timezone prelude +└── web.py stdlib DuckDuckGo scraper + fetch_url with SSRF guard +``` + ## License [MIT](LICENSE)