feat(platform): opt-in local Ollama providers for embeddings + recall LLM - #317
Merged
Conversation
… LLM With no API key set, the seam falls back to the deterministic fakes. That keeps CI offline (invariant §2.5) but makes local recall untestable: the fake embedder is hash-based, so any query whose wording differs from the stored text scores ~0.0004 cosine and `ask` always answers "I don't have a memory about X". Ask-conversation threads (#265) are never persisted either, because the §05.6 miss gate returns before the persist call — so the feature reads as broken when it is merely unexercised. You cannot dogfood recall out of the box. Add Ollama as a third real provider so a dev can run the whole memory loop on-box, with no keys and no data leaving the machine: - `OllamaEmbeddingProvider` (new) over Ollama's OpenAI-compatible `/embeddings`. Ollama cannot truncate to an arbitrary width the way OpenAI's v3 models can — the model's native output width IS the vector width — so we never send `dimensions` and instead VERIFY the returned width against EMBEDDING_DIM, failing loudly with a message naming a correct model rather than writing a vector pgvector will later reject. `mxbai-embed-large` is the 1024-d model matching the schema column. - The recall LLM reuses `OpenAiLlmProvider` against Ollama's OpenAI-compatible `/chat/completions`, so all eight call sites and the strict-JSON citation parsing are shared rather than reimplemented. The hardcoded URL becomes an injectable `baseUrl`; the default is unchanged. - The model router gains an `LLM_PROVIDER=ollama` short-circuit that forces the local vendor for EVERY class, mirroring the existing `'fake'` switch. Privacy fence (§2.2, M9): `isThirdParty` is false ONLY when the resolved base URL is loopback, and it is an explicit constructor argument defaulting to true. Moving OLLAMA_BASE_URL to a remote host therefore re-arms the fence and private memories stop flowing to it — a URL edit in `.env` can never silently disarm the fence. Both providers are opt-in and NEVER inferred: Ollama is a daemon, not a key, so there is no key-presence to infer from, and a dev box running Ollama for something unrelated must not be hijacked. Defaults are untouched — with no EMBEDDING_PROVIDER/LLM_PROVIDER set, the fakes still win. Verified: type-check, lint and build green; the 82 existing model-router and LLM unit tests pass unchanged. End-to-end against a live local stack with `mxbai-embed-large` + `gemma3:4b`, a semantic paraphrase ("When does the quarterly meeting happen?") correctly recalled a memory worded differently ("The zorblatt quarterly review is scheduled for March 14th…") and answered "March 14th" with 1 citation — a query the fake embedder scores at 0.0004 and always misses. No new tests: this adds provider implementations behind the existing seam, and the fakes remain forced in tests via vitest.config.ts. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds Ollama as a third real provider so the whole memory loop runs on-box with no API keys and nothing leaving the machine.
Why: with no key set the seam falls back to the fakes. That keeps CI offline (§2.5) but makes local recall untestable — the fake embedder is hash-based, so any query worded differently from the stored text scores ~0.0004 cosine and
askalways answers "I don't have a memory about X". Ask-conversation threads (#265) are never persisted either, because the §05.6 miss gate returns before the persist call, so the feature reads as broken when it is merely unexercised.What:
OllamaEmbeddingProviderover Ollama's OpenAI-compatible/embeddings. Ollama cannot truncate to an arbitrary width, so we never senddimensionsand instead VERIFY the returned width againstEMBEDDING_DIM, failing loudly rather than writing a vector pgvector rejects later.mxbai-embed-largeis the 1024-d model matching the schema.OpenAiLlmProvideragainst Ollama's/chat/completions— all eight call sites and the strict-JSON citation parsing are shared, not reimplemented. The hardcoded URL becomes an injectablebaseUrl; default unchanged.LLM_PROVIDER=ollamashort-circuit forcing the local vendor for every class, mirroring the existing'fake'switch.Privacy fence (§2.2, M9):
isThirdPartyis false ONLY for a loopback base URL, and it is an explicit constructor argument defaulting totrue. MovingOLLAMA_BASE_URLto a remote host re-arms the fence — a URL edit in.envcan never silently disarm it.Both providers are opt-in and NEVER inferred: Ollama is a daemon, not a key, so there is nothing to infer from, and a box running Ollama for something unrelated must not be hijacked. With no
EMBEDDING_PROVIDER/LLM_PROVIDERset, the fakes still win.type-check,lint,buildgreenmxbai-embed-large+gemma3:4b: the semantic paraphrase "When does the quarterly meeting happen?" recalled a differently-worded memory ("The zorblatt quarterly review is scheduled for March 14th…") and answered "March 14th" with 1 citation — a query the fake embedder scores 0.0004 and always misses.No new tests: this adds provider implementations behind the existing seam, and the fakes remain forced in tests via
vitest.config.ts.🤖 Generated with Claude Code