feat: RAG long-term memory with vector embeddings

[nickmeinhold]

Long-term Memory via RAG

Inspired by Dreamfinder's memory system.

Currently Gremlin has a 10-message sliding window and a 30-minute conversation TTL. Important context just... evaporates. This adds persistent long-term memory with semantic retrieval.

How it works

1. Embedding pipeline — As conversations happen (or during the dream cycle), important messages/decisions/context get embedded via Voyage AI and stored in the DB with their vector representations.
2. Memory retriever — On each new message, retrieve the top-N semantically similar memories using cosine similarity and inject them into the system prompt context window.
3. Memory consolidator — Periodically merge/deduplicate similar memories to avoid bloat.

What gets remembered

• Decisions made in chat
• Task assignments and outcomes
• Team preferences and conventions
• Recurring topics and project context
• Anything the dream cycle explicitly files as worth remembering

DB additions needed

• memories table: id, chat_id, content, embedding (blob/json), created_at, source (conversation/dream), similarity score metadata
• Index on chat_id for fast retrieval

System prompt injection

Retrieved memories injected as a ## Long-term Memory section near the top of the system prompt, formatted as a concise bullet list. Top ~5 most relevant memories per message.

Dependencies

• Voyage AI API key (VOYAGE_API_KEY env var) — or swap for any embeddings provider (OpenAI, local, etc.)
• Vector similarity search (cosine, done in-process over SQLite blob storage — no need for a vector DB at this scale)

Why this is good

Right now every conversation starts cold. With RAG memory, Gremlin actually knows the team over time — remembers that @Thinkerer prefers tasks in a specific format, that the auth system has a known quirk, that sprint planning always happens Tuesday. It gets smarter the longer it runs.

---

Reference implementation: lib/src/memory/ in Dreamfinder (Dart). Uses Voyage AI embeddings + cosine similarity retrieval.

[nickmeinhold]

Thanks for the thoughtful comment @garygegpt-droid! You're spot on about emotional weighting and tiered memory — and it turns out we built something very close to what you're describing in our sister project Dreamfinder.

RAG memory landed in Dreamfinder first:

• feat(memory): RAG-based long-term memory system — Voyage AI embeddings + cosine similarity retrieval over SQLite
• feat(memory): active recall & embedding backfill
• feat: agentic RAG with deep_search multi-source retrieval — Level 3 agentic RAG with parallel fan-out across memory embeddings, Outline wiki, and Kan task board. The agent reasons about when to search and how to evaluate results.

Dream consolidation — you nailed it, this is literally what Dreamfinder does:

• feat(dream): autonomous dream cycle with parallel branching — Modeled after real sleep stages (N1→N2→N3→REM). Daily interactions get processed into consolidated memories overnight, with "sparks" branching into parallel dream threads. Quiet days skip deeper phases automatically.

Your point about core identity facts persisting while contextual memories get overwritten maps closely to how the MemoryConsolidator works — summarising and deduplicating older memories while preserving high-signal content.

Dreamfinder is a Dart/Matrix bot rather than TypeScript/Telegram, but the memory architecture is exactly what this issue was describing. Worth a look if you're building something similar!

[m13v]

memory consolidation is the part most people underestimate here. without dedup at ingest time you end up with hundreds of near-identical memories after a few weeks of active use. we do a cosine similarity check on every new entry, and if it's above 0.92 vs an existing one we supersede the old entry rather than append. keeps the corpus clean without losing information. for the embedding pipeline, local ONNX runtime with nomic-embed-text avoids API costs entirely and retrieval quality is close enough to Voyage for memory-sized datasets.

[m13v]

our semantic dedup and embedding pipeline with the cosine similarity threshold for memory consolidation: https://github.com/m13v/ai-browser-profile/blob/main/ai_browser_profile/embeddings.py

[nickmeinhold]

Great points @m13v, thanks for sharing the implementation! The 0.92 cosine similarity threshold for ingest-time dedup is something we should steal. Dreamfinder currently handles dedup as a periodic batch job during the dream consolidation cycle — works fine, but it means the corpus gets noisy between consolidation runs. Checking at write time is cleaner.

Curious about a couple of things from your approach:

1. Supersede vs merge — when you detect a near-duplicate, do you just replace the old entry with the new one, or do you do any content merging? In DF's consolidator we summarise overlapping memories into a single entry, which sometimes captures nuance from both versions. But it's also more complex and occasionally lossy.

2. Threshold tuning — how'd you land on 0.92? We've found that the right threshold varies by embedding model and content type (short factual statements cluster tighter than conversational context). Did you experiment with different values or was it empirical?

The nomic-embed-text via ONNX runtime is a smart call for local-first use cases. For DF's scale (a few hundred memories per chat group) the Voyage API cost is negligible, but eliminating the network dependency is appealing — especially for the dream cycle which processes memories in bulk.

Your embeddings.py is a clean reference — the browser profile angle is a really interesting inversion of the same problem. We're extracting knowledge from conversations; you're extracting it from browsing behaviour. Different input, same consolidation challenge.