TODO-pass: continuous-conversation memory (v67), cross-source recall tool, cbm/MCP startup hardening#22
Conversation
admin_socket.c had regrown to 2,554 lines, back over the 2,500-line hard limit. Extract the 4 music-DB handlers (stats/search/list/rescan) into a new admin_socket_music.c — the lowest-coupling seam, since they depend only on send_text_response. Core is now 2,423 lines. Handlers renamed admin_handle_music_* to avoid a link-time collision with webui_music_handlers.c's extern handle_music_search (the admin copies were static before the move, so the clash only surfaced once they went extern). Pure code movement: dispatch switch and wire protocol unchanged. Build clean (0 warnings), 88/88 ctest -L ci.
From a startup-log review: - config_parser: add note_extraction_guard to the [memory] known-keys allowlist. It was parsed and applied but missing from the allowlist, so it logged a spurious "unknown key (typo?)" warning every boot. - mcp_client: downgrade the inner handshake initialize/tools-list failures from ERROR to WARNING. The bridge already treats a failed optional server as skip-and-continue and reports the outcome at WARN; an optional MCP server that isn't up yet at boot shouldn't throw red. Build clean, format clean. (A dawn.toml mis-placement that silently dropped the [memory.embeddings] recompute keys was fixed locally — gitignored, not in this commit.)
Under the gateway, any slug that didn't prefix-match the direct-provider tables fell back to a 128K default — wrong for OpenRouter-only vendors (deepseek/mistral/qwen/meta) and divergent names. llm_context now fetches openrouter.ai/api/v1/models on first use, caches slug -> context_length (256-entry table, 1h TTL, single-flight), and looks up the active model's true window. Falls back to the old vendor-strip + table probe when offline / no key / cache miss — no regression. 2MB cap for the catalog (exceeds the 64K /props cap). Build clean, format clean; live-verified context now reports correctly under the gateway.
…ort.js memory.js was 2,094 lines, over the 1,500 JS soft limit. Move the Import/Export cluster (16 functions + import state) into a new memory_import.js, mirroring the memory_aliases.js sibling pattern: window.DawnMemoryImport with init(ctx) for shared escapeHtml + onMemoriesChanged, and thin handle*Response forwarders kept on DawnMemory so dawn.js dispatch is unchanged. memory.js -> 1,634; index.html loads memory_import.js before memory.js. Pure code movement, no behavior change. node --check + Prettier clean. Export live-verified in the Memory panel; Import path moved verbatim but not re-exercised live.
At boot, cbm-mcp (mcp-proxy fronting a stdio child) answers `initialize` before its child is ready for `tools/list`, returning -32602. DAWN treated the failed handshake as fatal, destroyed the server slot, and never reconnected — so a later code-project import reported "no code server connected" until a daemon restart, with no logs (the availability check was a passive cached-state read). - mcp_client: retry tools/list across the proxy cold-start window (RPC error / timeout only); log the JSON-RPC error body so failures aren't a bare "(7)". - mcp_bridge: keep the server slot when the startup connect fails; add mcp_bridge_ensure_connected() for lazy reconnect on first use. cbm's availability gate now reconnects instead of reporting it permanently absent. Tool registration stays at startup only (registry locks post-init). - mcp_bridge_register_tool: null the moved-in param set immediately so a failed registry insert frees it exactly once (fixes a double-free crash reachable on any registration failure). - dawn-server.service: weak After=/Wants=cbm-mcp.service ordering. Tested live: cbm down at startup keeps the slot (0 connected, not dropped); reconnect on `mcp reset`/first use succeeds without crashing; warm path registers all 12 tools. 44 MCP/code-project/auth unit tests pass.
A `dawn-admin` import passes requester user_id 0 ("operator import"), and
with global defaulting to false the row was written user_id=NULL/is_global=0
— owned by no one and shared with no one. Such a row is invisible to every
code_project_db_list_visible() caller: the LLM `code_project` tool and all
non-admin users. Only admins saw it, via the WebUI's list_all fallback (which
is why it showed in the UI but the assistant reported it "not indexed").
Force global=true when requester_user_id <= 0 in code_project_import, so an
ownerless import is shared and visible. WebUI imports (real auth_user_id) are
unaffected. Existing ownerless rows need a one-time `UPDATE ... SET is_global=1`.
The cbm name-translation map (clean project name <-> cbm path slug) was captured from mcp_bridge_init(), which runs inside tools_register_all() — before auth_db_init(). code_project_db wasn't open yet, so the capture failed instantly and silently, leaving the map empty. It only ever populated as a side effect of an index, so after a plain restart with no new index every cbm graph tool received an untranslated clean name, cbm returned "project not found", and the LLM had to fall back to raw "var-lib-dawn-source-*" slugs. - dawn.c: capture the map after auth_db_init + the MCP admin-grant bootstrap, where the DB is open and cbm is connected. This is the real startup capture. - mcp_bridge_tool.c: drop the premature no-op capture in mcp_bridge_init; re-capture on cbm lazy reconnect (mcp_bridge_ensure_connected) so a server that was down at boot rebuilds its map when it comes up. Tested live: startup now logs "captured 6 project mapping(s)" with no index; clean names resolve on the first cbm call. 20 MCP/code-project tests pass.
cbm's search_graph declares 14 parameters, but the MCP bridge rejected any tool with more than TOOL_PARAM_MAX (12) properties — so DAWN's primary code-search tool was silently dropped at registration and never reached the LLM. TOOL_PARAM_MAX sizes no array anywhere (tool_metadata_t.params is a pointer + count; the bridge mallocs its set); it's purely a validation cap. Raise it to 20 — admits real MCP tools like search_graph while still bounding an untrusted upstream schema. Generic: any MCP server's 13–20-param tools now register too. Verified live: cbm now registers 13 tools incl. cbm_search_graph.
The live `role:visual` handler only rendered a <dawn-visual> into the active streaming entry; when that entry didn't exist — a diagram-only turn, where the render_visual result arrives during the tool phase before the answer streams — the `if (sEntry)` block was skipped with no fallback, so the diagram was silently dropped from the live view and merely stashed for save. It then appeared only on reload (via extractVisuals in addNormalEntry). finalizeStream drains pending visuals into the save payload, not the live transcript, so there was no other live path. Add the else branch: render the visual as its own assistant entry via renderVisuals (the same path reload uses). No duplicate — finalize never renders visuals live.
…k (v67) Replace fork-on-compaction (archive parent + create a continuation row, the source of the both-locked bug) with an in-conversation watermark. On compaction, conv_db_set_compaction_watermark records the summary + last- compacted msg id on the same row (single atomic UPDATE, monotonic guard) — no archive, no new conversation. On reload, context is bounded to messages after the watermark (conv_db_get_messages_after) + the summary, while the UI still displays the full transcript. The summary is re-injected as an ASSISTANT message carrying a reconstructed [COMPACTED conv=N msgs=X-Y node=Z depth=D] marker (shared formatter), so the LLM keeps a context_expand handle to the originals across reloads. All bounding gated on watermark>0, so un-compacted conversations are byte-identical to before. The messaging forever-conversation loader (memory_history_load_from_db) goes through the same watermark path. Old split code left dormant (warns if hit). Schema v67 adds context_watermark_msg_id + one-time unlocks legacy archived conversations. Also fixes a latent bug: reloaded summaries were injected as system messages and silently dropped by the per-turn two-system-message rebuild — now assistant role, matching the live marker. Live-verified end-to-end (watermark set, bounded reload, marker visible, context_expand fired and retrieved verbatim originals). Build clean, 88/88 CI, +2 unit tests. Five-agent review applied.
Document the v67 fork→watermark change: single continuous conversations, in-row watermark + bounded reload, assistant-role [COMPACTED] marker re-injection, dormant split path. Flag the Phase 3/4 continued_from continuation/chain machinery as legacy (fires only for pre-v67 convs).
…imation resize Chart.js charts rendered at the default 300x150 in the sandboxed srcdoc iframe. Root causes: (1) maintainAspectRatio:true caches aspectRatio 2.0 from the default canvas, so height is always width/2, never the container; (2) resize() during the ~1s entrance animation is a no-op (Chart.animator.running guard), so rAF-timed nudges did nothing. Fix: wrap the canvas in a definite-size .dawn-chart-box (canonical Chart.js responsive container), force the canvas to fill it, and flip maintainAspectRatio:false + resize() after the animation settles. Canvas content only; SVG/non-chart HTML unaffected. chart.md guideline updated to match.
Focus-block candidate text was truncated at a fixed byte count (FOCUS_TEXT_MAX_BYTES=4608) with a raw memcpy in two places (focus_candidate_init and webui_broadcasts capped_text_view). When byte 4608 landed inside a multi-byte UTF-8 character, the truncated text ended in a partial sequence — invalid UTF-8 — which made the context_injection WebSocket text frame undecodable. The browser rejected the frame and dropped the socket (1006), looping reconnect→reload→drop so the conversation never rendered. DB content was valid; corruption was introduced at focus-compose/broadcast time. Add focus_utf8_safe_cap() (back the cut up to a character boundary) and use it at both sites. Surfaced on a conversation full of article punctuation (em-dashes, curly quotes) whose excerpts crossed the cap mid-character. Tested: new test_utf8_safe_cap (em-dash straddle, ASCII, whole, NULL); test_focus_source 21/21.
One high-level 'recall' tool that gathers across memory, notes, documents, and calendar in a single call, instead of forcing the LLM to fan out across the per-source search tools (which it reliably failed to do, answering shallowly from per-turn injection). Wraps the existing focus engine via focus_compose_ex() (a per-call trim-limit override; focus_compose() is now a NULL-limits wrapper), groups results by source with read-pointers, adds a RULES nudge, and demotes the per-source tools so the model calls recall first for what-do-we-know/status questions. New: src/tools/recall_tool.c, recall_format.c (+headers), [memory.recall] config, tests/test_recall_format.c. Removes the 'recall' alias from the memory tool. Eval (benchmarks/recall_eval, untracked): cross-source gather 8% -> ~54%; recall fires unprompted; completeness neutral (eval noise-limited at 13 prompts). Tests: test_recall_format 5/5, test_focus_source 20/20, test_config_validate 18/18. Build + format clean. Big-3 review applied (summary dead-pointer, redundant strlen, include_private, config/availability comments).
…etrieval config_write_toml emitted backfill_on_startup / model_id / recompute_* under [memory.graph_retrieval], but the parser reads them from [memory.embeddings]. Every settings save misplaced them, so the next load logged 'unknown key' warnings and silently fell back to defaults. Move the writes into the embeddings section to match the parser so saved config round-trips.
There was a problem hiding this comment.
Pull request overview
This PR is a broad “TODO/bug-fix backlog” pass that upgrades DAWN’s memory compaction model (v67) to keep conversations continuous, introduces a unified cross-source recall tool, and hardens MCP/cbm startup/connect behavior. It also includes WebUI rendering fixes (Chart.js sizing + diagram-only live visuals), OpenRouter context sizing via /api/v1/models, config round-trip fixes, and a couple of file-split refactors to stay within size limits.
Changes:
- Memory v67: replace fork-on-compaction with an in-conversation compaction watermark + bounded reload, including summary reinjection and new DB migration/tests.
- Add unified
recalltool (formatter + routing nudges + config), plus UTF-8-safe truncation for focus/context-injection payloads. - MCP/cbm: startup race hardening (lazy reconnect, handshake retries, logging), OpenRouter context catalog cache, WebUI visual rendering fixes, and code organization refactors.
Reviewed changes
Copilot reviewed 59 out of 59 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| www/js/ui/visual-render.js | Adds Chart.js sizing wrapper/CSS + runtime resize logic for iframe-rendered visuals. |
| www/js/ui/memory.js | Splits import/export UI logic out to memory_import.js and adds thin response forwarders. |
| www/js/ui/memory_import.js | New module owning Memory Import/Export modals, state, and WS response handling. |
| www/js/ui/history.js | Updates compaction handling for v67 (no longer continues into a new conversation). |
| www/js/dawn.js | Renders diagram-only visual turns live when no active streaming entry exists. |
| www/index.html | Loads memory_import.js before memory.js. |
| tool_instructions/render_visual/chart.md | Updates Chart.js instructions (no canvas max-height; require maintainAspectRatio:false). |
| tests/test_recall_format.c | Adds unit tests for recall result formatting and pointer/grouping behavior. |
| tests/test_focus_source.c | Adds UTF-8 safe cap unit test and links new helper. |
| tests/test_config_validate.c | Sets valid defaults for new memory.recall.* config fields in tests. |
| tests/test_auth_db.c | Adds v67 compaction watermark and bounded-message-load unit tests. |
| tests/CMakeLists.txt | Adds v67 migration source, links focus helper, and adds new recall formatter test. |
| src/webui/webui_server.c | Restores LLM context bounded by watermark; reinjects compaction summary as assistant message. |
| src/webui/webui_history.c | Leaves legacy continue_conversation handler dormant (warns if invoked); bounds restore path. |
| src/webui/webui_broadcasts.c | UTF-8-safe truncation for broadcast payload text. |
| src/webui/webui_auth_helpers.c | Adds stable system-prompt routing nudge to call recall first for broad context questions. |
| src/tools/tools_init.c | Registers the recall tool unconditionally (availability gated at runtime). |
| src/tools/recall_tool.c | Implements unified recall tool using focus engine + formatter. |
| src/tools/recall_format.c | Formats recall results into grouped source sections with read pointers. |
| src/tools/memory_tool.c | Removes recall alias from memory tool; updates guidance to prefer recall for broad context. |
| src/tools/mcp_transport_http_sse.c | Improves endpoint-ready logging to include resolved endpoint URL. |
| src/tools/mcp_client.c | Adds tools/list handshake retry and improves RPC error logging; downgrades handshake failures to WARNING. |
| src/tools/mcp_bridge_tool.c | Keeps server slots on failed startup connect; adds lazy reconnect; fixes params ownership/double-free risk. |
| src/tools/mcp_bridge_schema.c | Ties MCP schema property cap explicitly to TOOL_PARAM_MAX and clarifies intent. |
| src/tools/document_search.c | Updates tool description to position it as a targeted follow-up vs recall. |
| src/tools/document_grep_tool.c | Updates tool description to position it as a targeted follow-up vs recall. |
| src/tools/code_project_service.c | Forces global=true for ownerless (CLI/operator) imports so projects are visible. |
| src/tools/code_graph_provider_cbm.c | Availability check now attempts lazy reconnect (boot race self-heal). |
| src/memory/memory_history_loader.c | Mirrors v67 bounded restore behavior for messaging history loads. |
| src/llm/llm_context.c | Adds OpenRouter model catalog cache + TTL + single-flight lookup for context sizing; persists v67 watermark. |
| src/llm/llm_command_parser.c | Adds system rules nudge to call recall first for broad “what do we know” questions. |
| src/dawn.c | Captures cbm name-translation map after DB init (and after MCP init). |
| src/core/focus/focus_source.c | Introduces focus_compose_ex() with per-call trim overrides; keeps focus_compose() as wrapper. |
| src/core/focus/focus_candidate_helpers.c | Adds focus_utf8_safe_cap() and uses it for safe truncation of candidate text. |
| src/config/config_validate.c | Validates new [memory.recall] fields and bounds per_source_max. |
| src/config/config_parser.c | Parses [memory.recall] table and allowlists note_extraction_guard. |
| src/config/config_env.c | Fixes TOML writer to emit embedding keys under [memory.embeddings] (parser-consistent). |
| src/config/config_defaults.c | Adds defaults for unified recall tool limits. |
| src/auth/auth_db_statements.c | Adds v67 statements: conv watermark update and bounded message-select. |
| src/auth/auth_db_schema.c | Adds context_watermark_msg_id column to conversations schema. |
| src/auth/auth_db_migrations.c | Applies v67 migration and bumps schema readiness gating. |
| src/auth/auth_db_migrations_v67.c | New v67 migration: add watermark column + unlock legacy archived split conversations. |
| src/auth/auth_db_conv.c | Persists watermark + formats compaction context marker; adds bounded conv_db_get_messages_after(). |
| src/auth/admin_socket.c | Removes music handlers from main file; dispatches to extracted admin_socket_music.c. |
| src/auth/admin_socket_music.c | New extracted music admin handler implementations. |
| services/dawn-server/dawn-server.service | Adds weak ordering/wants for cbm-mcp service to reduce boot race window. |
| include/tools/tool_registry.h | Raises TOOL_PARAM_MAX to 20 to admit real MCP tools (e.g., cbm search_graph). |
| include/tools/recall_tool.h | New public header for registering the recall tool. |
| include/tools/recall_format.h | New public header for recall formatter. |
| include/tools/mcp_bridge.h | Adds mcp_bridge_ensure_connected() API. |
| include/core/focus/focus_source.h | Adds focus_limits_t and focus_compose_ex() declaration. |
| include/core/focus/focus_candidate_helpers.h | Declares focus_utf8_safe_cap(). |
| include/config/dawn_config.h | Adds recall_config_t under memory config. |
| include/auth/auth_db.h | Adds watermark field and new DB APIs to public auth DB header. |
| include/auth/auth_db_internal.h | Bumps schema version to 67; adds new stmt pointers and migration prototype. |
| include/auth/admin_socket_internal.h | Declares extracted music admin handler entrypoints. |
| docs/LCM_DESIGN.md | Documents Phase 5 compaction watermark model and legacy continuation behavior. |
| CMakeLists.txt | Adds v67 migration and extracted admin music file to build. |
| cmake/DawnTools.cmake | Always compiles recall tool sources and reports it enabled. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
PR Summary by QodoMemory v67 watermark, cross-source recall tool, MCP/cbm startup hardening Description
Diagram
High-Level Assessment
Files changed (59)
|
Code Review by Qodo
Context used✅ Compliance rules (platform):
27 rules 1. focus_compose_ex Doxygen params missing
|
- visual-render: gate the Chart.js sizing wrapper on `new Chart(` rather than any <canvas>, so a non-chart canvas visual isn't wrapped/clipped. - memory_import: give escapeHtml a real entity-escaping fallback (the old one returned input unchanged — an innerHTML XSS sink if ctx is missing); fix the stale "matches memory_aliases.js" comment. - memory_import: release an existing focus trap before installing a new one so a double-open can't leak competing traps. - mcp_bridge_tool: correct the self-heal comment (ensure_connected does NOT register tools, per the header) and note the return is intentionally unchecked (the NULL-client lookup below reports a failed reconnect). - focus_source.h / mcp_bridge.h: complete Doxygen (@return, @param). Comment/doc-only on the C side. test_focus_source 21/21, test_recall_format 5/5; format + node --check clean.
Summary
A pass over the TODO/bug-fix backlog: one memory-architecture change, a new
cross-source recall tool, a cluster of MCP/code-project (cbm) startup-robustness
fixes, OpenRouter context sizing, two WebUI rendering fixes, config/startup-log
cleanup, and two file-split refactors to stay under the size limits. 15 commits,
all build/format clean.
Highlights
Memory — single continuous conversations (v67)
4a2f5feReplace fork-on-compaction (archive parent + create a continuationrow — the source of the both-locked bug) with an in-conversation compaction
watermark. On compaction, the summary + last-compacted msg id are recorded on
the same row (single atomic UPDATE, monotonic guard); on reload, context is
bounded to messages after the watermark + the summary, while the UI still shows
the full transcript. The summary is re-injected as an assistant message
carrying a reconstructed
[COMPACTED …]marker so the LLM keeps acontext_expandhandle to the originals across reloads. All bounding gated onwatermark > 0, so un-compacted conversations are byte-identical to before.Schema v67; messaging forever-conversation loader goes through the same path;
old split code left dormant. Live-verified end-to-end, 88/88 CI, +2 unit tests,
five-agent review applied.
2ecf254LCM design doc: add Phase 5 (compaction watermark), mark thecontinued_fromcontinuation/chain machinery legacy (pre-v67 only).Recall / retrieval
087cef3New high-levelrecalltool that gathers across memory, notes,documents, and calendar in a single call (via a new
focus_compose_ex()trimoverride), instead of relying on the LLM to fan out across per-source search
tools — which it reliably failed to do. Groups results by source with
read-pointers + a RULES nudge; demotes the per-source tools. Eval (untracked):
cross-source gather 8% → ~54%; recall fires unprompted.
f046fd8UTF-8-safe focus-block truncation. A fixed-byte cut could landmid-multibyte-character, producing invalid UTF-8 in the
context_injectionWebSocket frame → browser drops the socket (1006) → reconnect/reload loop, so
the conversation never rendered. Added
focus_utf8_safe_cap()at both cut sites.MCP / code-project (cbm) startup robustness
c18335cSelf-heal the cbm connection instead of dropping it on a boot race(mcp-proxy answers
initializebefore its stdio child is ready fortools/list). Keep the server slot on failed startup connect; add lazyreconnect on first use; retry
tools/listacross the cold-start window; fix adouble-free reachable on any registration failure; weak service ordering.
cbfdb19Capture the cbm name-translation map afterauth_db_init(it ranbefore the DB was open, so the map was empty after a plain restart and clean
names didn't resolve). Re-capture on lazy reconnect.
16684a9Raise the MCP tool param cap 12 → 20 so cbm's 14-paramsearch_graph(DAWN's primary code-search tool) actually registers. The cap isvalidation-only; sizes no array.
68f3456Forceglobal=truefor operator/CLI (dawn-admin) imports(
requester_user_id <= 0), so an ownerless import is visible to thecode_projecttool and non-admin users instead of being owned by no one.
LLM
b54cebaResolve OpenRouter context sizes from/api/v1/models(cachedslug → context_length, 1h TTL, single-flight) instead of falling back to a wrong
128K default for OpenRouter-only vendors. Falls back to the old table probe when
offline/no key.
WebUI
ac230a4Size Chart.js visuals via a dedicated.dawn-chart-boxcontainer +maintainAspectRatio:false+ post-animationresize()— charts were stuck atthe default 300×150 in the sandboxed iframe.
b5d5611Render diagram-only turns live: therole:visualhandler had nofallback when there was no active streaming entry, so the diagram only appeared
on reload.
Config / startup
15dd30eWrite embedding keys to[memory.embeddings](the section theparser reads) instead of
[memory.graph_retrieval], so saved config round-tripsinstead of logging "unknown key" and falling back to defaults.
7db9ea8Quiet two startup-log false alarms: addnote_extraction_guardtothe
[memory]allowlist; downgrade optional-MCP-server handshake failures fromERROR to WARNING.
Refactors (size discipline)
55df2acExtract the Import/Export cluster frommemory.js(2,094 →1,634 lines) into
memory_import.js, mirroring thememory_aliases.jspattern.4241abbSplit the 4 music-DB handlers out ofadmin_socket.c(2,554 →2,423, back under the 2,500 hard limit) into
admin_socket_music.c.Testing
ctest -L ci.test_recall_format,test_focus_source(UTF-8 cap), +2 for v67.context sizing all live-verified per the individual commit messages.
🤖 Generated with Claude Code