docs(letta): document Letta-backed agent mode + the streaming usage caveat

[juniperbevensee]

Why

git grep -i letta over origin/main for *.md returned nothing. The Letta bridge is genuinely merged — gateway/letta_brain.py plus a dispatch branch at the top of _run_agent_inner in gateway/run.py — with two test modules behind it, but no documentation anywhere. Anyone told that a Hermes gateway can front a second, heterogeneous runtime had no way to find out how to turn it on, what reaches the Letta agent, or where the integration stops short.

Docs only. No behavior change.

What's here

File	Change
website/docs/user-guide/features/letta-brain-mode.md	New page — the full contract
README.md	New Alternate Runtimes section (native / Codex / Letta / proxy) with a compact Letta subsection and the caveat inline
website/docs/reference/environment-variables.md	LETTA_BRAIN_URL, LETTA_BRAIN_AGENT_ID, LETTA_BRAIN_API_KEY, LETTA_BRAIN_STREAMING
website/sidebars.ts	Sidebar entry next to codex-app-server-runtime

⚠️ The load-bearing caveat

Stated prominently in all three places, because a launch narrative is likely to lean on "a Letta-brained door inherits Hermes' whole policy stack." Most of it does. Budget accounting does not.

• stream_message (gateway/letta_brain.py:285-288) parses only assistant-text deltas off the SSE feed. Letta's terminal usage_statistics event is never captured, so a streamed turn reports prompt_tokens = 0 / completion_tokens = 0 — documented in the LettaTurn docstring at lines 44-45.
• Streaming is on unless explicitly disabled: gateway/run.py:18086-18087 reads LETTA_BRAIN_STREAMING and only takes the blocking path when the value is off / 0 / false / no. So zero-token turns are the default, not an edge case.
• Downstream, _run_agent_via_letta records last_prompt_tokens: 0 for those turns, so context-window tracking, compression decisions, and the runtime footer all read 0.
• Real counts come back only from the blocking client. The audit-parity tests that assert real token recording set LETTA_BRAIN_STREAMING=off (tests/gateway/test_letta_audit_parity.py:116) — that is exactly the scope of the guarantee.
• Separately, and on either path: the credits tracker lives in the native loop (run_agent.py), which brain turns never enter, so there is no cost/spend accounting for them at all. Letta's tool calls likewise don't reach Hermes' tool audit log. Both are explicit non-goals in the test module's docstring, and the page says so rather than implying coverage.

The page's summary line: the door's authorization and audit story holds; its cost-control story does not.

Code facts verified before writing

Everything documented was read off origin/main, not inferred:

• Seam — dispatch is a branch at the top of _run_agent_inner (gateway/run.py:18628-18640), checked before proxy mode, reusing the proxy-mode result-dict shape and _setup_platform_stream_consumer. Authorization runs upstream at _is_user_authorized (~line 9668) → _handle_message_with_agent (11582) → _run_agent (12489) → _run_agent_inner (18600), so the bridge is not an authorization bypass.
• Env contract — _get_letta_brain (17862): env first (LETTA_BRAIN_URL + LETTA_BRAIN_AGENT_ID, optional LETTA_BRAIN_API_KEY), then gateway.letta_brain.{base_url, agent_id, api_key}. Mode activates when both URL and agent ID are present; no separate flag.
• Door context — build_sender_tag / apply_sender_tag prepend [from Alice in #family]. DMs get sender only; group/channel/thread also gets chat_name, falling back to {chat_type}:{chat_id} (gateway/run.py:18069-18072).
• Persistence — returns agent_persisted: False so the gateway writes transcript rows itself; the user message is persisted even on a failed round-trip.
• Concurrency — per-agent asyncio.Lock keyed base_url|agent_id, with generation re-checks after the lock, mid-stream, and post-turn.

Honesty notes

The page is explicit that this is a productionized prototype and lists what it does not do: one agent per gateway (no per-platform/group/profile routing); Hermes personality, memory, context files, skills, and toolset are not forwarded; no per-sender isolation on the Letta side; the SSE path has unit coverage for line parsing only (no end-to-end streaming test); superseding a turn suppresses delivery but does not cancel the Letta server's in-flight work; the 120s timeout is not configurable. No capabilities were invented.

Before merging

• A live session is working in this repo (uncommitted changes in the main checkout at the time this was written). This branch was made from a fresh clone of origin/main and touches only README.md, website/sidebars.ts, website/docs/reference/environment-variables.md, and one new docs page — but please rebase onto current main before merging in case that session has landed anything.
• CI's docs-site-checks (ascii-guard lint --exclude-code-blocks docs + a Docusaurus build) has not been run locally — ascii-guard==2.3.0 would not install in this environment and npm ci was out of scope. The one ASCII diagram is inside a fenced code block (excluded by that linter), and the page was scanned for MDX-unsafe {} / <tag> outside code spans. Let CI confirm.
• Not merging. Review the caveat wording especially — if any of it overstates or understates the gap, that's the part to correct.

🤖 Generated with Claude Code

[juniperbevensee]

Independent audit

Re-verified every load-bearing claim against origin/main from a fresh clone (not the working checkout — a live session holds that). Findings below; one real error found and fixed in 071c693.

Verified correct (the caveat holds)

Claim	Evidence on origin/main	Verdict
Streamed turns report 0 usage	gateway/letta_brain.py:285-288 (stream_message docstring: "the terminal usage_statistics event is not captured, so streamed turns report 0 prompt/completion tokens") and :44-45 (LettaTurn docstring). Code confirms: parse_stream_line returns only assistant_message deltas; nothing reads usage_statistics.	✅ exact
Streaming defaults ON	gateway/run.py:18086-18087 — _streaming_pref = os.getenv("LETTA_BRAIN_STREAMING", "").strip().lower() then if _streaming_pref not in {"off","0","false","no"}:. Unset → "" → not in the set → streaming attempted.	✅ exact
Audit-parity tests force streaming off before asserting tokens	tests/gateway/test_letta_audit_parity.py:116 — monkeypatch.setenv("LETTA_BRAIN_STREAMING", "off") in the shared _bootstrap, consumed by test_letta_turn_records_prompt_tokens. Module docstring names credits_tracker and tool_calls.log as explicit non-goals.	✅ exact
No credits/spend accounting for brain turns	credits_tracker appears in run_agent.py, agent/*, tui_gateway/server.py — and zero occurrences in gateway/run.py. Brain turns never enter that loop.	✅
Dispatch precedes proxy mode	gateway/run.py:18631 if self._get_letta_brain(): sits above if self._get_proxy_url(): at 18641, both at the top of _run_agent_inner (18600).	✅
Authz chain is upstream	Call sites, not definitions: _is_user_authorized at 9668 (defined gateway/authz_mixin.py:298), _handle_message_with_agent 11582, _run_agent call at 12489, _run_agent_inner 18600. Not an authorization bypass.	✅
Env contract	_get_letta_brain (17862): env first (LETTA_BRAIN_URL + LETTA_BRAIN_AGENT_ID, optional LETTA_BRAIN_API_KEY), then gateway.letta_brain.*; base_url.rstrip("/") confirms "trailing slashes are stripped".	✅
Sender tag	build_sender_tag/apply_sender_tag (letta_brain.py:88-120) produce [from Alice in #family]; run.py:18069-18072 sets _group only when chat_type != "dm", falling back to f"{chat_type}:{chat_id}".	✅
api_calls: 1, empty tools, agent_persisted: False, last_prompt_tokens	Result dict at run.py:18200-18213.	✅
Streaming fallback table	retryable=True only for missing aiohttp (293), 404/405 (313), and ClientConnectorError/ConnectionRefusedError (329); run.py:18129-18150 implements the three documented outcomes.	✅
"SSE has unit coverage for line parsing only"	test_letta_brain.py has test_parse_stream_line_* and no end-to-end streaming test.	✅
No pre-existing Letta docs	`git ls-tree -r origin/main	grep -i letta→ only the two.py` files and two test modules. The "no documentation anywhere" premise is real.

❌ One error found and fixed (071c693)

The page claimed:

If the URL and agent ID are missing when a turn arrives, the gateway replies with ⚠️ Letta brain not configured (...) rather than silently falling back to the native loop.

This is backwards. Dispatch at run.py:18631 is gated on if self._get_letta_brain(): — the same check the guard inside _run_agent_via_letta (18032-18040) repeats. With either variable absent, _get_letta_brain() returns None, the branch is skipped, and the turn proceeds to proxy mode or the native AIAgent loop. The ⚠️ string is effectively unreachable through the gateway path, and the fallback is exactly the silent one the sentence promised it wasn't.

This is worth more than a wording fix: a typo in LETTA_BRAIN_AGENT_ID produces a normal-looking native Hermes turn, so none of this page's caveats apply to it — while the native toolset, possibly not intended for that door, does. Corrected in all three places (page :::warning, README sentence, env-var table row) plus a Known-limitations bullet.

Merge conditions

• CI — green on the base commit; re-running on 071c693 (docs-site-checks builds Docusaurus + lint:diagrams).
• File overlap — checked all 7 other open PRs (fix(web): warn when a configured search/extract backend is discarded #128, fix(discord): gateway channel-scoped access (opt-in) + fail closed on unparseable allowlists #127, test(image_gen): pin the availability check to the dispatch chain #126, feat(telegram): auto-approve groups via HSM when admin adds the bot #120, chore(actions)(deps): bump the actions-minor-patch group across 1 directory with 2 updates #103, feat(slack): auto-approve channels the bot is added to (resolves #68) #78). None touches README.md, website/**, or sidebars.ts.
• Live session — its docs branch (dev/juniperbevensee/sm-plugin-docs) edits the README banner at lines 4-11; that content already landed on main via docs(swarm_map_policy): name the plugin's counterpart Swarm Map, not HSM #124. This PR's hunks start at line 184. Test-merged into current origin/main: Auto-merging README.md / Automatic merge went well. No conflict.
• Rebase note addressed — verified against main @ 2c19f56 (docs(swarm_map_policy): name the plugin's counterpart Swarm Map, not HSM #124), not the stale base.

Merging on that basis. The streaming/budget caveat is prominent in three places and, as far as I can verify, accurate.