Skip to content

feat(platform): opt-in local Ollama providers for embeddings + recall LLM - #317

Merged
jayesh-keychain merged 1 commit into
mainfrom
feat/ollama-local-providers
Jul 31, 2026
Merged

feat(platform): opt-in local Ollama providers for embeddings + recall LLM#317
jayesh-keychain merged 1 commit into
mainfrom
feat/ollama-local-providers

Conversation

@jayesh-keychain

Copy link
Copy Markdown
Collaborator

Adds Ollama as a third real provider so the whole memory loop runs on-box with no API keys and nothing leaving the machine.

Why: with no key set the seam falls back to the fakes. That keeps CI offline (§2.5) but makes local recall untestable — the fake embedder is hash-based, so any query worded differently from the stored text scores ~0.0004 cosine and ask always answers "I don't have a memory about X". Ask-conversation threads (#265) are never persisted either, because the §05.6 miss gate returns before the persist call, so the feature reads as broken when it is merely unexercised.

What:

  • OllamaEmbeddingProvider over Ollama's OpenAI-compatible /embeddings. Ollama cannot truncate to an arbitrary width, so we never send dimensions and instead VERIFY the returned width against EMBEDDING_DIM, failing loudly rather than writing a vector pgvector rejects later. mxbai-embed-large is the 1024-d model matching the schema.
  • Recall LLM reuses OpenAiLlmProvider against Ollama's /chat/completions — all eight call sites and the strict-JSON citation parsing are shared, not reimplemented. The hardcoded URL becomes an injectable baseUrl; default unchanged.
  • Model router gains an LLM_PROVIDER=ollama short-circuit forcing the local vendor for every class, mirroring the existing 'fake' switch.

Privacy fence (§2.2, M9): isThirdParty is false ONLY for a loopback base URL, and it is an explicit constructor argument defaulting to true. Moving OLLAMA_BASE_URL to a remote host re-arms the fence — a URL edit in .env can never silently disarm it.

Both providers are opt-in and NEVER inferred: Ollama is a daemon, not a key, so there is nothing to infer from, and a box running Ollama for something unrelated must not be hijacked. With no EMBEDDING_PROVIDER/LLM_PROVIDER set, the fakes still win.

  • type-check, lint, build green
  • 82 existing model-router + LLM unit tests pass unchanged
  • End-to-end on a live stack with mxbai-embed-large + gemma3:4b: the semantic paraphrase "When does the quarterly meeting happen?" recalled a differently-worded memory ("The zorblatt quarterly review is scheduled for March 14th…") and answered "March 14th" with 1 citation — a query the fake embedder scores 0.0004 and always misses.
  • Verified against the seeded demo corpus: "What is Priya allergic to?" → "peanuts" (conf 0.75, 1 citation).

No new tests: this adds provider implementations behind the existing seam, and the fakes remain forced in tests via vitest.config.ts.

🤖 Generated with Claude Code

… LLM

With no API key set, the seam falls back to the deterministic fakes. That
keeps CI offline (invariant §2.5) but makes local recall untestable: the fake
embedder is hash-based, so any query whose wording differs from the stored
text scores ~0.0004 cosine and `ask` always answers "I don't have a memory
about X". Ask-conversation threads (#265) are never persisted either, because
the §05.6 miss gate returns before the persist call — so the feature reads as
broken when it is merely unexercised. You cannot dogfood recall out of the box.

Add Ollama as a third real provider so a dev can run the whole memory loop
on-box, with no keys and no data leaving the machine:

  - `OllamaEmbeddingProvider` (new) over Ollama's OpenAI-compatible
    `/embeddings`. Ollama cannot truncate to an arbitrary width the way
    OpenAI's v3 models can — the model's native output width IS the vector
    width — so we never send `dimensions` and instead VERIFY the returned
    width against EMBEDDING_DIM, failing loudly with a message naming a
    correct model rather than writing a vector pgvector will later reject.
    `mxbai-embed-large` is the 1024-d model matching the schema column.
  - The recall LLM reuses `OpenAiLlmProvider` against Ollama's
    OpenAI-compatible `/chat/completions`, so all eight call sites and the
    strict-JSON citation parsing are shared rather than reimplemented. The
    hardcoded URL becomes an injectable `baseUrl`; the default is unchanged.
  - The model router gains an `LLM_PROVIDER=ollama` short-circuit that forces
    the local vendor for EVERY class, mirroring the existing `'fake'` switch.

Privacy fence (§2.2, M9): `isThirdParty` is false ONLY when the resolved base
URL is loopback, and it is an explicit constructor argument defaulting to
true. Moving OLLAMA_BASE_URL to a remote host therefore re-arms the fence and
private memories stop flowing to it — a URL edit in `.env` can never silently
disarm the fence.

Both providers are opt-in and NEVER inferred: Ollama is a daemon, not a key,
so there is no key-presence to infer from, and a dev box running Ollama for
something unrelated must not be hijacked. Defaults are untouched — with no
EMBEDDING_PROVIDER/LLM_PROVIDER set, the fakes still win.

Verified: type-check, lint and build green; the 82 existing model-router and
LLM unit tests pass unchanged. End-to-end against a live local stack with
`mxbai-embed-large` + `gemma3:4b`, a semantic paraphrase ("When does the
quarterly meeting happen?") correctly recalled a memory worded differently
("The zorblatt quarterly review is scheduled for March 14th…") and answered
"March 14th" with 1 citation — a query the fake embedder scores at 0.0004 and
always misses. No new tests: this adds provider implementations behind the
existing seam, and the fakes remain forced in tests via vitest.config.ts.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@jayesh-keychain
jayesh-keychain merged commit cda34b8 into main Jul 31, 2026
6 checks passed
@jayesh-keychain
jayesh-keychain deleted the feat/ollama-local-providers branch July 31, 2026 21:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant