Skip to content

feat(store,mcp,tui): surface observation lifecycle state from existing review_after column #437

@goutham80808

Description

@goutham80808

Pre-flight Checks

  • I have searched existing issues and this is not a duplicate
  • I understand this issue needs status:approved before a PR can be opened

Problem Description

Engram computes review_after timestamps on new observations (decision=6mo, policy=12mo, preference=3mo) and stores them in the review_after column (store.go:851). This infrastructure was built but the last mile was never completed — the column is written once and never read again.

Why this matters — user pain points:

  1. Stale decisions pollute agent context forever. A decision observation saved 2 years ago about "use Redux for state management" still gets injected into FormatContext today — even if the team migrated to Zustand 6 months ago. Agents trust stale decisions as if they were made yesterday, leading to contradictory guidance and confused output.

  2. No way to audit what needs attention. Teams using engram across weeks/months have no mechanism to surface which decisions, policies, or preferences might be outdated. A project with 200+ observations has no way to answer the question "what should we revisit?" — you'd have to manually read every observation.

  3. Agents can't self-correct. When an agent encounters a decision observation in context, it has no signal about whether that decision is current or stale. It treats a 24-month-old policy the same as one saved 5 minutes ago. This means agents blindly follow outdated conventions even when the codebase has moved on.

  4. The decay infrastructure is 80% complete but useless. The review_after column is populated on every insert (store.go:2261), the decay offsets are configured (store.go:218-229), and migration tests verify it works (store_test.go:7416-7571). But none of this is surfaced to agents or users — it's a column that's written and never read.

  5. TUI shows no state information. The observation list (view.go:682) and detail view (view.go:278) render type, title, project, timestamps — but no indication of whether an observation is fresh or stale. Users scanning the TUI see a flat list with no lifecycle context.

The core gap: engram already knows when observations should be reviewed (the data is in the database), but this knowledge is invisible to every consumer — MCP agents, TUI users, and context injection.

Proposed Solution

Add a virtual observation state derived from the existing review_after column at query time:

State Condition
active review_after IS NULL or review_after > now()
needs_review review_after <= now() and review_after IS NOT NULL

No new database columns. The state is computed from data that already exists.

Phase 1 (this issue):

1. Store layer (internal/store/store.go)

  • Add ReviewAfter *string field to Observation struct
  • Expose it in all SELECT/Scan sites (9 query locations, 6 Scan call sites)
  • Add Observation.State() string method for virtual state computation
  • Add ObservationsNeedingReview(project string, limit int) ([]Observation, error) — queries observations where review_after <= datetime('now') and deleted_at IS NULL
  • Add MarkReviewed(id int64) error — extends review_after by the observation type's decay offset, resetting the review cycle

2. MCP server (internal/mcp/mcp.go)

  • New mem_review tool:
    • action: "list" — returns observations needing review, grouped by project
    • action: "mark_reviewed" — resets review_after on a specific observation, confirming it's still valid
  • Add computed state field to mem_save and mem_search response metadata — so agents know immediately whether a result is current or stale

3. TUI (internal/tui/view.go, styles.go)

  • State badge in renderObservationListItem — yellow warning needs_review when stale
  • State + review date rows in viewObservationDetail:
    State:  [active]           (green)
    Review: 2026-11-27         (review_after date)
    

Example usage (MCP):

Tool: mem_review
Arguments: { "action": "list", "project": "engram", "limit": 10 }

Returns observations where review_after has passed:
  #42 [decision] "Switched from sessions to JWT" - needs_review since 2026-05-15
  #67 [policy]   "Branch naming conventions"     - needs_review since 2026-04-01

Tool: mem_review
Arguments: { "action": "mark_reviewed", "observation_id": 42 }

Confirms decision is still valid. Resets review_after to now() + 6 months.

User impact after implementation:

  • Agents can query mem_review to surface stale decisions before they cause problems
  • Agents can mark decisions as reviewed, keeping the review cycle alive
  • TUI users see at a glance which observations need attention
  • FormatContext can later prioritize active observations over stale ones
  • Teams get a natural "memory hygiene" workflow without manual effort

Not in scope for Phase 1:

  • Background goroutine for auto-transitioning states (query-time derivation is sufficient)
  • expires_at activation (remains NULL per existing Phase 1 constraint)
  • New state column in the database (virtual state only)
  • FormatContext prioritization by state (can be a follow-up)
  • TUI "Review Queue" dedicated screen (can be a follow-up issue)

Affected Area

  • Store (database, queries) — primary
  • MCP Server (tools, transport) — secondary
  • TUI (terminal UI) — secondary

Alternatives Considered

  1. Materialized state column — Add a state TEXT DEFAULT 'active' column and update it via background worker. Rejected for Phase 1: adds migration complexity and a goroutine lifecycle to manage. Virtual state via State() method is simpler and equivalent for current scale. Can be revisited if query-time computation becomes a performance concern.

  2. Background decay worker — A goroutine that periodically scans and transitions states. Rejected: query-time derivation is sufficient. A background worker adds operational complexity (start/stop lifecycle, error handling, scheduling) for no user-facing benefit at this stage.

  3. Manual state management only — Let agents call mem_save with a state parameter. Rejected: defeats the purpose of automatic decay. Most agents would never set it, and the problem of stale context would remain unsolved.

  4. Do nothing — Leave review_after as a write-only column. Rejected: the infrastructure exists and is tested but provides zero value without this last-mile surfacing. Every day without it, agents operate on increasingly stale decisions.

Additional Context

File change map:

File Change
internal/store/store.go ReviewAfter field on Observation struct, State() method, update 9 SELECT + 6 Scan sites, new ObservationsNeedingReview + MarkReviewed methods
internal/mcp/mcp.go New mem_review tool registration + handler, state in save/search responses
internal/tui/view.go State badge in renderObservationListItem, state + review date rows in viewObservationDetail
internal/tui/styles.go stateActiveStyle, stateNeedsReviewStyle styles
internal/store/store_test.go State computation tests, needs review query tests, mark reviewed tests
internal/mcp/mcp_test.go mem_review tool handler tests
internal/tui/view_test.go State badge rendering tests

Sync compatibility: review_after already exists in the schema (store.go:851) and is already populated on insert. No new columns needed. The ListObservationSyncPayloads test (store.go:6469) verifies that review_after is excluded from the sync wire format, which remains true.

Related issues: #241 (weighted BM25 — approved), #352 (search improvements — approved), #257 (trigram+fuzzy — merged)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions