Stage 5

README.md

@@ -17,6 +17,8 @@ With the latest update, Engram also supports memory evolution: when a session's
 
 Engram also supports collective memory: multiple agents can share a global knowledge graph, sessions can be marked public or private, and conflicting facts across sessions are surfaced via a dedicated endpoint.
 
+Engram also has cognitive version control: because every committed change is an entry in the Raft log, Engram can reconstruct a session's exact state at any past point — by log index, by timestamp, or by a named checkpoint you pin. Reconstruction replays the log against the nearest retained snapshot into a throwaway, read-only state machine, so time-travel reads never touch live state and need no consensus. You can also diff any two points to see what entities, relationships, facts, and summaries changed between them.
+
 ## architecture
 
 ```mermaid
@@ -33,12 +35,14 @@ graph TD
     router --> visibilityhandler["visibility handler"]
     router --> globalhandler["global knowledge handler"]
     router --> consolidationhandler["consolidation handler"]
+    router --> historyhandler["history handler\n/at /history /diff /checkpoints"]
 
     memoryhandler -->|write| raft["raft consensus\nopenraft 0.9\ncluster mode only"]
     corememhandler -->|write| raft
     sessionhandler -->|delete or register agent| raft
     visibilityhandler -->|write| raft
     consolidationhandler -->|ApplySummary| raft
+    historyhandler -->|CreateCheckpoint| raft
     raft -.->|grpc append entries| peers["peer nodes (port 9001)"]
     raft -.->|grpc install snapshot| laggers["lagging followers"]
     raft -->|state machine apply| shortterm["short-term memory trait"]
@@ -51,6 +55,13 @@ graph TD
     raft -->|knowledge job| knowledgequeue["knowledge worker pool<br/>bounded channel"]
     raft --> redb[("redb\npersistent raft log\n+ snapshot store")]
     redb -.->|"startup recovery"| raft
+    raft -->|CreateCheckpoint| checkpointstore["checkpoint store\nname -> log index"]
+
+    historyhandler -->|reconstruct / diff / history| reconstructor["reconstructor\nreplay log vs nearest snapshot\nread-only, node-local"]
+    reconstructor -->|nearest retained snapshot| redb
+    reconstructor -->|log tail above snapshot| redb
+    reconstructor -->|resolve checkpoint| checkpointstore
+    retention["retention\nkeep last HISTORY_SNAPSHOTS\n+ log tail above them"] -.->|floors log purge| redb
 
     shortterm --> redis[("redis<br/>volatile, fast")]
     shortterm --> inmem["in-memory store<br/>test fallback"]
@@ -205,6 +216,11 @@ docker compose up -d
 | GET    | /knowledge/global/conflicts                                   | list conflicting facts across sessions       |
 | GET    | /sessions/{session_id}/summaries                              | list consolidated summaries for a session    |
 | POST   | /sessions/{session_id}/consolidate                            | manually trigger consolidation (leader only) |
+| GET    | /sessions/{session_id}/at?index=N\|checkpoint=X\|at=RFC3339   | reconstruct session state at a past point    |
+| GET    | /sessions/{session_id}/history                                | list the session's committed timeline        |
+| GET    | /sessions/{session_id}/diff?from=A&to=B                       | structural diff between two points           |
+| POST   | /sessions/{session_id}/checkpoints                            | pin a named checkpoint to a log index        |
+| GET    | /sessions/{session_id}/checkpoints                            | list a session's named checkpoints           |
 | GET    | /cluster                                                      | cluster status (cluster mode only)           |
 | POST   | /cluster/init                                                 | initialize cluster                           |
 | POST   | /cluster/add-learner                                          | add a learner node                           |
@@ -249,6 +265,7 @@ the application reads configuration from environment variables:
 | CLUSTER_HTTP_PEERS    | comma-separated HTTP peers as `id:host:port` (for leader redirect URLs)  | `2:node-2:3000,3:node-3:3000`     |
 | RAFT_DB_PATH          | path to the redb file for the persistent Raft log and snapshot store     | `./data/raft/engram.redb`         |
 | SNAPSHOT_LOG_THRESHOLD| number of committed log entries after which the leader compacts the log  | `1000`                            |
+| HISTORY_SNAPSHOTS     | number of recent snapshots retained for time-travel reads; log purge is floored above the oldest retained snapshot, and points below the window return 410 Gone | `5` |
 
 values like `similarity_threshold` and `max_tokens` are controlled per request through query parameters on the context endpoint.
 
@@ -285,6 +302,12 @@ values like `similarity_threshold` and `max_tokens` are controlled per request t
 - `ApplySummary` Raft command: one atomic replicated transition stores the summary, trims consumed raw messages, and updates metrics; idempotent by `summary_id`
 - `GET /sessions/{id}/summaries` and `POST /sessions/{id}/consolidate` endpoints; followers 307-redirect to the leader
 - five new Prometheus metrics for consolidation throughput and queue depth
+- cognitive version control: reconstruct a session's state at any past point by log index, RFC 3339 timestamp, or named checkpoint (`GET /sessions/{id}/at`)
+- session timeline (`GET /sessions/{id}/history`) and structural diff between two points (`GET /sessions/{id}/diff`)
+- named checkpoints via idempotent, non-repointing `CreateCheckpoint` Raft command (`POST/GET /sessions/{id}/checkpoints`)
+- read-only reconstruction: replay the log against the nearest retained snapshot into a throwaway state machine, node-local and consensus-free
+- bounded history retention (`HISTORY_SNAPSHOTS`): log purge floored above the oldest retained snapshot; out-of-window points return 410 Gone
+- four new Prometheus metrics for history: retained snapshots, checkpoints, reconstructions, and reconstruction duration
 - LongMemEval and BEAM benchmark harnesses
 
 ## quickstart (3-node cluster)
@@ -305,7 +328,7 @@ docker compose -f docker-compose.cluster.yml up -d --build
 ./scripts/cluster-verify.sh
 ```
 
-the verify script checks 22 criteria: leader election, write replication to all nodes, 307 redirect from followers, failover, Prometheus metric presence, knowledge graph replication, delete-session cleanup, node restart and recovery from the Raft log, snapshot compaction, restart-then-verify that state is fully restored from the latest snapshot, session visibility propagation, global graph population from public sessions, agent registration, global entity and relationship count metrics, global entity queries, global conflict detection, global graph snapshot round-trip, consolidation threshold trigger, replicated determinism (all nodes byte-identical after consolidation), idempotency of re-applied summaries, persistence of summaries through cluster restart, and manual consolidate endpoint. it exits 0 only if all 22 criteria pass.
+the verify script checks 27 criteria: leader election, write replication to all nodes, 307 redirect from followers, failover, Prometheus metric presence, knowledge graph replication, delete-session cleanup, node restart and recovery from the Raft log, snapshot compaction, restart-then-verify that state is fully restored from the latest snapshot, session visibility propagation, global graph population from public sessions, agent registration, global entity and relationship count metrics, global entity queries, global conflict detection, global graph snapshot round-trip, consolidation threshold trigger, replicated determinism (all nodes byte-identical after consolidation), idempotency of re-applied summaries, persistence of summaries through cluster restart, manual consolidate endpoint, and (Stage 5) time-travel reconstruction identity across nodes, checkpoint recall and idempotency, history timeline projection, structural diff deltas, and read-only/retention-window semantics. it exits 0 only if all 27 criteria pass.
 
 see `docker-compose.cluster.yml` and the scripts in `scripts/` for details.
 

docker-compose.cluster.yml

@@ -26,6 +26,7 @@ services:
       LANCE_DB_PATH: "/data/lancedb"
       RAFT_DB_PATH: "/data/raft/engram.redb"
       SNAPSHOT_LOG_THRESHOLD: "${SNAPSHOT_LOG_THRESHOLD:-1000}"
+      HISTORY_SNAPSHOTS: "${HISTORY_SNAPSHOTS:-3}"
       ENGRAM_BIND_ADDR: "0.0.0.0:3000"
       OPENAI_API_KEY: "${OPENAI_API_KEY:-}"
       KNOWLEDGE_EXTRACTOR: "${KNOWLEDGE_EXTRACTOR:-mock}"
@@ -54,6 +55,7 @@ services:
       LANCE_DB_PATH: "/data/lancedb"
       RAFT_DB_PATH: "/data/raft/engram.redb"
       SNAPSHOT_LOG_THRESHOLD: "${SNAPSHOT_LOG_THRESHOLD:-1000}"
+      HISTORY_SNAPSHOTS: "${HISTORY_SNAPSHOTS:-3}"
       ENGRAM_BIND_ADDR: "0.0.0.0:3000"
       OPENAI_API_KEY: "${OPENAI_API_KEY:-}"
       KNOWLEDGE_EXTRACTOR: "${KNOWLEDGE_EXTRACTOR:-mock}"
@@ -82,6 +84,7 @@ services:
       LANCE_DB_PATH: "/data/lancedb"
       RAFT_DB_PATH: "/data/raft/engram.redb"
       SNAPSHOT_LOG_THRESHOLD: "${SNAPSHOT_LOG_THRESHOLD:-1000}"
+      HISTORY_SNAPSHOTS: "${HISTORY_SNAPSHOTS:-3}"
       ENGRAM_BIND_ADDR: "0.0.0.0:3000"
       OPENAI_API_KEY: "${OPENAI_API_KEY:-}"
       KNOWLEDGE_EXTRACTOR: "${KNOWLEDGE_EXTRACTOR:-mock}"

docs/API.md

@@ -23,6 +23,11 @@ This document describes every REST endpoint exposed by Engram. All endpoints are
 | GET    | /knowledge/global/conflicts                                   | list conflicting facts across sessions                |
 | GET    | /sessions/{session_id}/summaries                              | list consolidated summaries for a session             |
 | POST   | /sessions/{session_id}/consolidate                            | manually trigger consolidation (leader only)          |
+| GET    | /sessions/{session_id}/at                                     | reconstruct session state at a past point (cluster)   |
+| GET    | /sessions/{session_id}/history                                | list the session's committed timeline (cluster)       |
+| GET    | /sessions/{session_id}/diff                                   | structural diff between two points (cluster)          |
+| POST   | /sessions/{session_id}/checkpoints                            | pin a named checkpoint to a log index (cluster)       |
+| GET    | /sessions/{session_id}/checkpoints                            | list a session's named checkpoints                    |
 | GET    | /health                                                       | health check                                          |
 | GET    | /metrics                                                      | Prometheus metrics                                    |
 | GET    | /api-docs/openapi.json                                        | OpenAPI specification                                 |
@@ -808,3 +813,197 @@ In cluster mode, followers return 307 with a `Location` header pointing to the l
 ```sh
 curl -X POST http://localhost:3000/sessions/{session_id}/consolidate
 ```
+
+---
+
+## temporal (cognitive version control) endpoints
+
+Stage 5 adds time-travel reads and named checkpoints. Reconstruction replays this node's own Raft log against its nearest retained snapshot into a throwaway in-memory state machine — the log records committed *results*, so replay is deterministic and read-only. Reads need no consensus and run node-locally; only creating a checkpoint is a cluster mutation and routes through Raft.
+
+Retention keeps the last `HISTORY_SNAPSHOTS` snapshots plus the log tail above the oldest retained snapshot. A point older than that window is gone: reconstructing or diffing it returns **410 Gone**.
+
+These endpoints require the reconstructor, which only exists in cluster mode. Standalone nodes return **501 Not Implemented** for `/at`, `/history`, `/diff`, and (for creation) `/checkpoints`.
+
+---
+
+## GET /sessions/{session_id}/at
+
+Reconstructs the session's state at a past point. Select the point with **exactly one** of the query parameters below.
+
+**path parameters:**
+- `session_id` (string): session identifier
+
+**query parameters (exactly one required):**
+- `index` (integer): a Raft log index
+- `checkpoint` (string): a named checkpoint, resolved to its pinned index
+- `at` (string, RFC 3339): resolves to the newest message at or before this timestamp
+
+**success response:**
+- status: 200
+- body:
+```json
+{
+  "at_index": 42,
+  "messages": [ { "id": "m1", "role": "user", "content": "hello", "timestamp": "2026-07-08T12:00:00Z" } ],
+  "facts": ["user prefers dark mode"],
+  "entities": [ { "name": "Alice", "entity_type": "Person", "attributes": {} } ],
+  "relationships": [ { "from": "Alice", "to": "OpenAI", "relationship_type": "works_at" } ],
+  "summaries": [ { "id": "...", "text": "...", "created_at_index": 30, "consumed_message_ids": [], "consumed_count": 3, "model": "gpt-4o-mini", "prompt_version": "summarize_v1" } ]
+}
+```
+
+**error responses:**
+- 400: none, more than one, or invalid selector (e.g. malformed `at` timestamp)
+- 404: named checkpoint not found, or no message at/before the given timestamp
+- 410: the requested point is below the retained window
+- 501: standalone mode (no reconstructor)
+
+**examples:**
+```sh
+curl "http://localhost:3000/sessions/{session_id}/at?index=42"
+curl "http://localhost:3000/sessions/{session_id}/at?checkpoint=v1.0"
+curl "http://localhost:3000/sessions/{session_id}/at?at=2026-07-08T12:00:00Z"
+```
+
+---
+
+## GET /sessions/{session_id}/history
+
+Returns the session's committed changes in log-index order over the retained window.
+
+**path parameters:**
+- `session_id` (string): session identifier
+
+**success response:**
+- status: 200
+- body (each entry is `{index, kind, session_id}`):
+```json
+{
+  "history": [
+    { "index": 40, "kind": "AddMessage", "session_id": "abc123" },
+    { "index": 41, "kind": "ApplySummary", "session_id": "abc123" },
+    { "index": 42, "kind": "CreateCheckpoint", "session_id": "abc123" }
+  ]
+}
+```
+
+**error responses:**
+- 501: standalone mode (no reconstructor)
+
+**example:**
+```sh
+curl http://localhost:3000/sessions/{session_id}/history
+```
+
+---
+
+## GET /sessions/{session_id}/diff
+
+Structural diff between two points. "Added" means present at `to` but not `from`; identities collide by a stable key per kind, so re-ordering or attribute-only churn does not register.
+
+**path parameters:**
+- `session_id` (string): session identifier
+
+**query parameters (both required):**
+- `from` (integer): earlier log index
+- `to` (integer): later log index; `from` must be `<= to`
+
+**success response:**
+- status: 200
+- body (`message_count_delta` is signed, so a consolidation trim shows negative):
+```json
+{
+  "entities_added": [],
+  "entities_removed": [],
+  "relationships_added": [],
+  "relationships_removed": [],
+  "facts_added": ["user prefers dark mode"],
+  "facts_removed": [],
+  "summaries_added": [],
+  "summaries_removed": [],
+  "message_count_delta": 2
+}
+```
+
+**error responses:**
+- 400: `from` > `to`
+- 410: either endpoint is below the retained window
+- 501: standalone mode (no reconstructor)
+
+**example:**
+```sh
+curl "http://localhost:3000/sessions/{session_id}/diff?from=30&to=42"
+```
+
+---
+
+## POST /sessions/{session_id}/checkpoints
+
+Pins a name to a log index on every node. Omitting `at_index` pins the current applied index ("now"), stamped before submit so replay stays deterministic. This is a cluster mutation and routes through Raft; a follower 307s to the leader.
+
+Create is **idempotent and non-repointing**: re-creating an existing name keeps its original index, and the response echoes the *stored* index, not the requested one.
+
+**path parameters:**
+- `session_id` (string): session identifier
+
+**request body:**
+```json
+{
+  "name": "v1.0",
+  "at_index": 42
+}
+```
+
+`name` is required and must not be empty. `at_index` is optional (defaults to the current applied index).
+
+**success response:**
+- status: 201
+- body:
+```json
+{
+  "session_id": "abc123",
+  "name": "v1.0",
+  "at_index": 42
+}
+```
+
+**error responses:**
+- 400: empty name
+- 307: redirect to leader (cluster mode, follower received the request)
+- 501: standalone mode
+- 500: replication failed
+
+**example:**
+```sh
+curl -X POST http://localhost:3000/sessions/{session_id}/checkpoints \
+  -H 'content-type: application/json' \
+  -d '{"name":"v1.0","at_index":42}'
+```
+
+---
+
+## GET /sessions/{session_id}/checkpoints
+
+Lists the session's checkpoints. Reads the shared checkpoint store directly, so it works on any node without reconstruction (available in standalone mode too).
+
+**path parameters:**
+- `session_id` (string): session identifier
+
+**success response:**
+- status: 200
+- body:
+```json
+{
+  "checkpoints": [
+    { "session_id": "abc123", "name": "v1.0", "at_index": 42 }
+  ]
+}
+```
+
+**error responses:**
+- 500: failed to list checkpoints
+
+**example:**
+```sh
+curl http://localhost:3000/sessions/{session_id}/checkpoints
+```