|
2426 | 2426 | "href": "17-memory-hierarchy.html#tier-4-dream-consolidation-nightly-batch-archival", |
2427 | 2427 | "title": "Memory Hierarchy", |
2428 | 2428 | "section": "Tier 4: Dream Consolidation – Nightly Batch Archival", |
2429 | | - "text": "Tier 4: Dream Consolidation – Nightly Batch Archival\nThe fourth tier is autoDream – a background consolidation pass that reviews memories accumulated across multiple sessions and reorganizes them. The name is apt: like sleep consolidation in neuroscience, it operates when the agent is not actively processing tasks, synthesizing fragmented observations into coherent long-term knowledge.\n\nTriple Gate\nAutoDream fires only when all three gates pass, evaluated in cheapest-first order to minimize per-turn cost:\n\nTime gate: hours since last consolidation >= minHours (default: 24 hours). This requires only one stat call on the lock file.\nSession gate: number of transcript files with mtime > lastConsolidatedAt >= minSessions (default: 5 sessions). This requires a directory scan, so it is gated behind a 10-minute scan throttle to avoid repeated scanning when the time gate passes but the session gate does not.\nLock gate: no other process is mid-consolidation. The lock file (~/.claude/projects/<slug>/memory/.consolidate-lock) stores the holder’s PID. A stale-PID reclamation mechanism handles crashes: if the PID is dead or the lock is older than 1 hour, the next process reclaims it.\n\nconst DEFAULTS: AutoDreamConfig = {\n minHours: 24,\n minSessions: 5,\n}\n\n\nConsolidation Prompt\nThe consolidation prompt (consolidationPrompt.ts) instructs the forked agent through four phases:\n\nOrient – ls the memory directory, read MEMORY.md, skim existing topic files\nGather recent signal – review daily logs, check for drifted memories, grep transcripts narrowly\nConsolidate – merge new signal into existing topic files (not create duplicates), convert relative dates to absolute, delete contradicted facts\nPrune and index – keep MEMORY.md under 200 lines and 25KB, demote verbose entries, resolve contradictions\n\nThe agent operates under the same createAutoMemCanUseTool permission sandbox as T3 extraction. Bash is restricted to read-only commands – the prompt explicitly states this to prevent the agent from probing.\n\n\nRollback on Failure\nIf the forked agent fails, the lock file’s mtime is rolled back to its pre-acquisition value via rollbackConsolidationLock. This ensures the time gate passes again on the next turn, rather than the failure appearing as a successful consolidation that delays the next attempt by 24 hours.", |
| 2429 | + "text": "Tier 4: Dream Consolidation – Nightly Batch Archival\nThe fourth tier is autoDream – a background consolidation pass that reviews memories accumulated across multiple sessions and reorganizes them. The name is apt: like sleep consolidation in neuroscience, it operates when the agent is not actively processing tasks, synthesizing fragmented observations into coherent long-term knowledge.\n\n\n\n\n\n\nNote\n\n\n\nRollout status: T4 Dream Consolidation is fully implemented in v2.1.88 but in progressive rollout. It is gated by a server-side GrowthBook flag (tengu_onyx_plover) that defaults to off. Users can opt in via the autoDreamEnabled setting in settings.json.\n\n\n\nTriple Gate\nAutoDream fires only when all three gates pass, evaluated in cheapest-first order to minimize per-turn cost:\n\nTime gate: hours since last consolidation >= minHours (default: 24 hours). This requires only one stat call on the lock file.\nSession gate: number of transcript files with mtime > lastConsolidatedAt >= minSessions (default: 5 sessions). This requires a directory scan, so it is gated behind a 10-minute scan throttle to avoid repeated scanning when the time gate passes but the session gate does not.\nLock gate: no other process is mid-consolidation. The lock file (~/.claude/projects/<slug>/memory/.consolidate-lock) stores the holder’s PID. A stale-PID reclamation mechanism handles crashes: if the PID is dead or the lock is older than 1 hour, the next process reclaims it.\n\nconst DEFAULTS: AutoDreamConfig = {\n minHours: 24,\n minSessions: 5,\n}\n\n\nConsolidation Prompt\nThe consolidation prompt (consolidationPrompt.ts) instructs the forked agent through four phases:\n\nOrient – ls the memory directory, read MEMORY.md, skim existing topic files\nGather recent signal – review daily logs, check for drifted memories, grep transcripts narrowly\nConsolidate – merge new signal into existing topic files (not create duplicates), convert relative dates to absolute, delete contradicted facts\nPrune and index – keep MEMORY.md under 200 lines and 25KB, demote verbose entries, resolve contradictions\n\nThe agent operates under the same createAutoMemCanUseTool permission sandbox as T3 extraction. Bash is restricted to read-only commands – the prompt explicitly states this to prevent the agent from probing.\n\n\nRollback on Failure\nIf the forked agent fails, the lock file’s mtime is rolled back to its pre-acquisition value via rollbackConsolidationLock. This ensures the time gate passes again on the next turn, rather than the failure appearing as a successful consolidation that delays the next attempt by 24 hours.", |
2430 | 2430 | "crumbs": [ |
2431 | 2431 | "Series Home", |
2432 | 2432 | "III. Context Engineering", |
|
2438 | 2438 | "href": "17-memory-hierarchy.html#tier-5-team-memory-sync-the-distributed-cache", |
2439 | 2439 | "title": "Memory Hierarchy", |
2440 | 2440 | "section": "Tier 5: Team Memory Sync – The Distributed Cache", |
2441 | | - "text": "Tier 5: Team Memory Sync – The Distributed Cache\nThe outermost tier is team memory sync (teamMemorySync/index.ts), a server-backed system that shares memories across all authenticated organization members working on the same repository. This is the distributed shared cache of the hierarchy – the slowest, most broadly scoped, and most complex tier.\n\nSync Semantics\nThe API contract is built around a single endpoint:\nGET /api/claude_code/team_memory?repo={owner/repo} → pull\nGET ...?repo={owner/repo}&view=hashes → checksums only\nPUT /api/claude_code/team_memory?repo={owner/repo} → push (upsert)\nThe semantics are deliberately asymmetric:\n\nPull is server-wins: remote entries overwrite local files. If a teammate pushed a correction, the next pull replaces the local version unconditionally.\nPush is local-wins-on-conflict: if a 412 (Precondition Failed) occurs during push, the system probes server checksums, recomputes the delta excluding keys where the teammate’s push matches ours, and retries. This preserves the local user’s active edit.\nDeletions do not propagate: deleting a local file does not remove it from the server; the next pull restores it locally.\n\n\n\nDelta Upload and Batching\nPush does not upload all local files. It computes a delta by comparing sha256:<hex> content hashes of local files against serverChecksums (populated from the server’s entryChecksums response). Only keys whose hash differs are included in the PUT. This is analogous to a cache write-back policy: only dirty lines are flushed.\nLarge deltas are split into batches under a 200KB body-size cap (MAX_PUT_BODY_BYTES) to stay under the API gateway’s limit. Each batch is an independent PUT with upsert semantics – if batch N fails, batches 1..N-1 are already committed. The serverChecksums map is updated after each successful batch, so a retry naturally resumes from the uncommitted tail.\n\n\nSecret Scanning\nThe upload format is JSON, not the local markdown files directly. Memory entries are serialized into a JSON payload before transmission, which decouples the on-disk representation from the wire protocol and allows the server to store and index entries in a structured format.\nBefore any file is uploaded, it passes through a secret scanner (secretScanner.ts) using patterns derived from gitleaks. Files containing detected secrets are silently excluded from the upload – they never leave the machine. Only the rule ID (not the secret value or file path) is logged for analytics.\nconst secretMatches = scanForSecrets(content)\nif (secretMatches.length > 0) {\n skippedSecrets.push({\n path: relPath,\n ruleId: firstMatch.ruleId,\n label: firstMatch.label,\n })\n return // file excluded from upload\n}\n\n\nConflict Resolution\nOn a 412 conflict, the push logic executes a lightweight resolution cycle:\n\nProbe GET ?view=hashes to refresh per-key checksums (no bodies – saves bandwidth)\nRecompute the delta against the refreshed checksums (keys where a teammate’s concurrent push matches ours are naturally excluded)\nRetry the PUT with the tighter delta\n\nThis cycle repeats up to MAX_CONFLICT_RETRIES = 2 times. The probe-and-recompute approach avoids full content downloads during conflict resolution – a significant optimization when the team memory contains hundreds of kilobytes of content.", |
| 2441 | + "text": "Tier 5: Team Memory Sync – The Distributed Cache\nThe outermost tier is team memory sync (teamMemorySync/index.ts), a server-backed system that shares memories across all authenticated organization members working on the same repository. This is the distributed shared cache of the hierarchy – the slowest, most broadly scoped, and most complex tier.\n\n\n\n\n\n\nNote\n\n\n\nRollout status: T5 Team Memory Sync is fully implemented in v2.1.88 but in progressive rollout. It is double-gated: a compile-time feature('TEAMMEM') flag (enabled in this build) and a server-side GrowthBook flag (tengu_herring_clock) that defaults to off. It also requires OAuth authentication and a GitHub remote, limiting it to Pro/Team/Enterprise users. The codebase contains production bug fixes (e.g., a device that emitted 167K push events over 2.5 days), confirming it has been active for early access users.\n\n\n\nSync Semantics\nThe API contract is built around a single endpoint:\nGET /api/claude_code/team_memory?repo={owner/repo} → pull\nGET ...?repo={owner/repo}&view=hashes → checksums only\nPUT /api/claude_code/team_memory?repo={owner/repo} → push (upsert)\nThe semantics are deliberately asymmetric:\n\nPull is server-wins: remote entries overwrite local files. If a teammate pushed a correction, the next pull replaces the local version unconditionally.\nPush is local-wins-on-conflict: if a 412 (Precondition Failed) occurs during push, the system probes server checksums, recomputes the delta excluding keys where the teammate’s push matches ours, and retries. This preserves the local user’s active edit.\nDeletions do not propagate: deleting a local file does not remove it from the server; the next pull restores it locally.\n\n\n\nDelta Upload and Batching\nPush does not upload all local files. It computes a delta by comparing sha256:<hex> content hashes of local files against serverChecksums (populated from the server’s entryChecksums response). Only keys whose hash differs are included in the PUT. This is analogous to a cache write-back policy: only dirty lines are flushed.\nLarge deltas are split into batches under a 200KB body-size cap (MAX_PUT_BODY_BYTES) to stay under the API gateway’s limit. Each batch is an independent PUT with upsert semantics – if batch N fails, batches 1..N-1 are already committed. The serverChecksums map is updated after each successful batch, so a retry naturally resumes from the uncommitted tail.\n\n\nSecret Scanning\nThe upload format is JSON, not the local markdown files directly. Memory entries are serialized into a JSON payload before transmission, which decouples the on-disk representation from the wire protocol and allows the server to store and index entries in a structured format.\nBefore any file is uploaded, it passes through a secret scanner (secretScanner.ts) using patterns derived from gitleaks. Files containing detected secrets are silently excluded from the upload – they never leave the machine. Only the rule ID (not the secret value or file path) is logged for analytics.\nconst secretMatches = scanForSecrets(content)\nif (secretMatches.length > 0) {\n skippedSecrets.push({\n path: relPath,\n ruleId: firstMatch.ruleId,\n label: firstMatch.label,\n })\n return // file excluded from upload\n}\n\n\nConflict Resolution\nOn a 412 conflict, the push logic executes a lightweight resolution cycle:\n\nProbe GET ?view=hashes to refresh per-key checksums (no bodies – saves bandwidth)\nRecompute the delta against the refreshed checksums (keys where a teammate’s concurrent push matches ours are naturally excluded)\nRetry the PUT with the tighter delta\n\nThis cycle repeats up to MAX_CONFLICT_RETRIES = 2 times. The probe-and-recompute approach avoids full content downloads during conflict resolution – a significant optimization when the team memory contains hundreds of kilobytes of content.", |
2442 | 2442 | "crumbs": [ |
2443 | 2443 | "Series Home", |
2444 | 2444 | "III. Context Engineering", |
|
0 commit comments