feat(dream-daemon): gate per-mind memory consolidation behind dreamDaemon feature flag#369
Open
johnhain-msft wants to merge 27 commits into
Open
Conversation
… dbPath
Test-only accessor on the MindMemoryService interface, matching the existing `__resetMindMutexForTesting` pattern in consolidation-scheduler. Returns `{ daemon, dbPath } | null` so a Playwright driver attached via `electronApp.evaluate()` can drive forceRun()/getStatus() and read the per-mind dream.db from disk without us building a renderer-facing IPC bridge.
…e handle) Two narrow, env-gated hooks so the chamber-ui-tester Playwright driver can exercise the consolidation cycle deterministically without a renderer bridge: 1. DreamDaemon: `maybeTestSleep()` between the memory.md write and the log.md prune, gated on `CHAMBER_E2E=1 && CHAMBER_DREAM_TEST_SLEEP_MS > 0`. Lets M7 reproduce the mid-cycle append race (turn arrives after snapshot but before prune). Production builds never see CHAMBER_E2E=1, so this is a no-op by construction. 2. main.ts: when CHAMBER_E2E=1, stash mindMemoryService on globalThis.__chamberMindMemoryService. `electronApp.evaluate(...)` can then call `__debugGet(mindId).daemon.forceRun()` and read `dbPath` directly — no IPC channel, no preload bridge, no shared-package type contract.
…tation/sentinel scenarios
… (v0.59.7) Two production-blocking defects surfaced during manual testing of feature/dream-daemon-memory-consolidation: every fresh mind logged "[MindScaffold] Capability bootstrap failed (non-fatal): Error: Upgrade skill not found in genesis repo", and "[WorkingMemoryComposer] log.md is unstructured" repeated on every system-prompt rebuild. Root causes: 1. Epic ianphil#67 moved the upgrade skill from ianphil/genesis to ianphil/genesis-frontier in commit 17580f5, but MindScaffold.GENESIS_SOURCE was never updated. 2. MindScaffold.createStructure() wrote a zero-byte log.md placeholder which violated the chamber-structured-log/v1 sentinel contract, so WorkingMemoryComposer.readLog warned and skipped on every prompt rebuild until the first DailyLogWriter rotation fired. Fixes: - MindScaffold now points at ianphil/genesis-frontier@main; the upgrade-skill not-found error names the repo and branch. - createStructure() pre-seeds log.md with the sentinel + double newline (matching DailyLogWriter.seedFreshLog byte-for-byte) BEFORE the WORKING_MEMORY_FILES placeholder loop runs. - Genesis prompt no longer instructs the LLM to write to log.md (the file is reserved for DailyLogWriter frames). SOUL.md "Continuity" rewritten and the Mind Index drops the log.md bullet. - WorkingMemoryComposer.readLog downgrades the unstructured-log message from warn to info for the migration window. Validation: - New tests/integration/mindScaffold.integration.test.ts (7 tests) drives the full MindScaffold.create() against mkdtempSync with fake registry + SDK clients. Locks the on-disk contract for new minds AND the cross-cutting migration story for existing pre-fix minds (composer info -> DailyLogWriter rotates -> composer silent). - 49/49 targeted unit tests across the 4 affected files. - Full suite: 2035/2035 pass; lint clean. - Live Electron smoke (chamber-ui-tester) confirmed both error patterns are gone end-to-end. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…Node-major upgrades
Brings in A2A mailbox relay (v0.60.0, v0.61.0) and the v0.59.7 revert of "Refresh models" (issue ianphil#90 / docs/model-cache-investigation.md). Conflict resolutions: - CHANGELOG.md: dream-daemon entry renumbered to v0.62.0; master's v0.61.0, v0.60.0, v0.59.7 entries preserved below. - package.json: version 0.62.0; @github/copilot 1.0.45 (master). - config/vitest.config.ts: kept both new include entries (tests/integration/** + .github/extensions/**/*.mjs) and pool: 'forks'. - apps/desktop/src/main.ts: adopted master's A2A composition (activeA2AResolver, a2aRelayModeService, MessageRouter, A2aToolProvider, updated wireLifecycleEvents + setupA2AIPC signatures); kept mindMemoryComposition + CHAMBER_E2E hook; requestQuit now closes mindMemoryComposition, then mindManager.shutdown, then settles a2aRelayModeService.disconnect + stopMvpServer. - packages/services/src/mind/MindManager.ts: accepted master's deletion of recycleClientForMind; kept enableDreamDaemon/disableDreamDaemon/ toggleDreamDaemon/daemonToggling and patchChamberMindConfig/ rollbackToUnstructured imports. - package-lock.json: regenerated via full npm install (npm install --package-lock-only strips optional emnapi entries CI needs). Preflight: npm run lint clean, 2158/2159 tests pass (1 intentional skip).
Brings dream-daemon-memory-consolidation up to date with master: - Adopts master's chamber-sqlite-runtime + loadBetterSqlite3 injection. The bespoke better-sqlite3 loader in buildMindMemoryService.ts is gone; main.ts feeds the already-resolved Database constructor in. Both the task ledger and the dream daemon now resolve sqlite via the unified runtime. - Moves dream-daemon composition (buildMindMemoryService + mind:loaded/mind:unloaded listeners) from top-level into initializeRuntime(), right after mindManager.setProviders(...), so it slots into master's new boot sequence. Module-level mindMemoryComposition/mindMemoryService lets keep requestQuit() teardown and the CHAMBER_E2E global wired the same as before. - Reconciles ChatService constructor with master (mindManager, turnQueue, dateTimeContextProvider?, byoLlmModelsProvider?) and keeps add/removeObserver as field-initialized infra. Test sites use svc.addObserver(...). - Keeps both enableDreamDaemon/disableDreamDaemon (HEAD) and BYO LLM provider config integration (master) describe blocks in MindManager.test.ts. - forge.config.ts: drops better-sqlite3/bindings/file-uri-to-path from the ASAR unpack glob (they ship via chamber-sqlite-runtime extraResource now). - CHANGELOG: moves the dream-daemon v0.62.0 entry into Unreleased > Added (Keep-a-Changelog 1.1.0). Next: Phases 2-7 wire the dream-daemon surfaces behind a new app-level dreamDaemon feature flag, then Phase 8 runs full regression.
Adds dreamDaemon to AppFeatureFlags, DEFAULT_APP_FEATURE_FLAGS, getAppFeatureFlags, parseFeatureFlags, parseCompleteFeatureFlags. DEV defaults to true; remote policy (stable + insiders) defaults to false. Extends shared parser tests with dreamDaemon assertions and a missing-field rejection. Updates FeatureFlagService.test.ts complete-policy fixtures and renderer/test-helper fixtures so AppFeatureFlags-shaped literals satisfy the new required field. No runtime gating yet — Phase 3 wraps the dream-daemon composition in main.ts behind this flag.
…aemon flag
Layered defense-in-depth so the dream-daemon stack is fully inert when
the new app-level dreamDaemon feature flag is off:
- apps/desktop/src/main.ts: module-level dreamDaemonFeatureEnabled() reads
appFeatureFlags at call time; buildMindMemoryService composition + the
__chamberMindMemoryService E2E global are wrapped in if(flag); accessor
is passed positionally into IdentityLoader (arg 3), MindManager (arg 7),
MindProfileService (arg 4), setupMindIPC and setupGenesisIPC configs.
- main/ipc/mind.ts: SET_DREAM_DAEMON rejects when enabled && !flag, disable
is always allowed so a stable build can clean up insiders opt-in state.
- main/ipc/genesis.ts: new GenesisIPCOptions; CREATE handler coerces
enableDreamDaemon=false in sanitizedConfig server-side when flag is off.
- MindManager.doToggleDreamDaemon: throws Error('Dream Daemon is not
available in this build') before mind lookup when enabled && !flag.
- IdentityLoader.resolveComposerConfig: forces enabled=false in the
composer config when the accessor returns false, regardless of the
per-mind .chamber.json. Caps (lastKTurns, perTurnMaxBytes, memoryMaxBytes)
remain faithful so a future flip-on does not lose the user's settings.
- MindProfileService.getProfile: AND'd the accessor with the .chamber.json
value so the (now-hidden) UI never sees a stale ON state.
- renderer RoleScreen/GenesisFlow/AgentProfileModal: read featureFlags from
useAppState; hide the Switch / toggle row when flag is off and coerce
enableDreamDaemon=false at every emit boundary.
Tests: 6 new service-layer gate tests across MindManager, IdentityLoader,
and MindProfileService (146 passing). 4 new renderer flag-off tests across
RoleScreen, GenesisFlow, AgentProfileModal (25 passing). Existing tests
updated to pass testInitialState={ featureFlags: { dreamDaemon: true } }
where they assert on the dream-daemon surface.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add dreamDaemon row to the Current flags table (dev-only rollout: stable remote=false, insiders remote=false, dev=true). - New 'Dream Daemon' subsection enumerates every off-state behavior (buildMindMemoryService not called, IPC reject, genesis.create coercion, MindManager throw, IdentityLoader enabled=false override, MindProfileService payload override, RoleScreen/GenesisFlow/AgentProfileModal UI hide). - Update DEV_FEATURE_FLAGS snippet and the docs/flags/v1/flags.json example to include dreamDaemon. - Fix MD022/MD032 in AGENTS.md (blank lines around the Dream Daemon heading added on the branch) so 'npm run lint' is clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…(v0.64.0-insiders.4) Brings in: cron run history persistence, packaged copilot SDK 0.3.0 with RuntimeConnection.forStdio API, macOS code signing prepackaged scripts, prewarm-no-await black-screen fix.
….0 API SDK 0.3.0 CopilotSession exposes disconnect() not destroy(). The merge auto-took the dream-daemon-branch mock shape (destroy:) while production code (MindScaffold.generateSoul finally block) calls session.disconnect() — same as upstream master. Aligning fakes with the real SDK surface.
…olidation Resolves conflicts in MindManager.ts and main.ts constructor arg list. Both branches added a new positional arg after byoDefaultModelProvider: this branch added dreamDaemonFeatureEnabled (slot 7), upstream/master added managedSkillService (now slot 8). Both have defaults / are optional, so existing callers stay valid. The one upstream test (line 471) that explicitly passed managedSkillService at slot 7 is updated to pass undefined,managedSkillService. Brings in: author-your-own ttasks cron automation (fdc0891), skill loading fix (ed7181e), managed skills marketplace (61d0fce), v0.64.0 version bump (91ae68e). Regression: lint clean, npm test 2540/0fail, smoke:sdk + smoke:packaged-runtime green, smoke:desktop 24/16skip/6fail (4 pre-existing baseline + 2 environmental LLM-timeout tests on upstream's own new cron-tools spec and conversation-history flake).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the Dream Daemon — a per-mind background job that consolidates the structured
log.mdinto a refinedmemory.mdbetween turns — behind a newdreamDaemonapp feature flag. Defaults: off in stable, on in insiders. When the flag is off, no runtime is constructed, no IPC is reachable, no UI is shown, andIdentityLoaderreports the working-memory section as disabled regardless of any per-mind.chamber.jsonopt-in (so a stable build never includes log content in the system prompt for a mind that was opted in under insiders).What it does
MindMemoryServiceconstructed ininitializeRuntime(); wired tomind:loaded/mind:unloadedandapp.before-quitIPC.MIND.SET_DREAM_DAEMONresolves to updatedMindContextgenesis.create({ enableDreamDaemon: true })writes.chamber.jsonwithworkingMemory.consolidation.enabled: truefalseregardless of payloadWorkingMemoryComposerincludes consolidatedlog.mdexcerptsIdentityLoader.resolveComposerConfig; no log content in promptRoleScreenshows the Dream Daemon switch;AgentProfileModalshows the toggle rowMindProfileServicereportsdreamDaemonEnabled: false__chamberMindMemoryServiceexposed underCHAMBER_E2E=1Defense in depth: even if a renderer is compromised, the IPC throws, the genesis path coerces, and
MindManager.enableDreamDaemonitself rejects when the accessor returns false.Changes
dreamDaemonadded toAppFeatureFlags, defaults, resolver, complete-policy parser, with tests.docs/flags/v1/flags.jsonaddsdreamDaemon: false(stable) /true(insiders).dreamDaemonin remote success, cache hit, and all-false fallback paths.apps/desktop/src/main.ts) —buildMindMemoryServiceonly called when flag is on; uses the master-alignedloadBetterSqlite3()injection (no separate sqlite resolution).mind.tsandgenesis.tsacceptfeatureEnabledparameter; server-side coercion ongenesis.create.MindManager.enableDreamDaemon/disableDreamDaemonandIdentityLoader.resolveComposerConfigtake adreamDaemonFeatureEnabledaccessor;MindProfileServicemasks per-mind state when off.RoleScreen,GenesisFlow,AgentProfileModalconsumeuseAppFeatureFlags()and hide controls / coerce payloads when off.ai-docs/feature-flags.mdadds thedreamDaemonrow + behavior subsection.v0.64.0-insiders.4(cron run history persistence, SDK 0.3.0RuntimeConnection.forStdioAPI migration, macOS prepackaged code-signing scripts, prewarm-no-await black-screen fix). Test fakes that previously usedsession.destroyupdated tosession.disconnectto match the SDK 0.3 surface that upstream's master now uses.Verification
npm run lint— clean (tsc + eslint + deps:check + lint:yaml + lint:md)npm test— 2475 passed / 1 skipped / 0 failed across 208 test filesnpm run smoke:sdk— pinned runtime resolvesnpm run smoke:packaged-runtime— sqlite runtime + ledger path pack correctlynpm run smoke:desktop— 25 passed / 16 skipped / 4 failed; all 4 failures are pre-existing onmaster:agency-microsoft/genesis-mindsconversation-history-smoke.spec.tssidebar render lag — pre-existing on master baselinemodel-switch-context-smoke(failed pre-merge) now passes, likely fixed by upstream's cron-history workNotes for reviewer
dreamDaemonflag defaults to off in stable — merging this PR does not change behavior for stable users untildocs/flags/v1/flags.jsonis graduated.enableConfigDiscovery: false+configDirchange from97abd51is preserved; the dream-daemon prompt-path gate is orthogonal (per-mind.chamber.json, not the SDK config discovery flag).chamber-copilot-runtime/was bumped by upstream to SDK 0.3.0; nothing in this PR re-pins it.Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com