feat: add nomic-embed-text deep semantic embeddings

Embed nomic-embed-text-v1.5 (Q8_0 GGUF, 134MB) directly in the binary
via go:embed. Three-way hybrid scoring (BM25 0.3 + LSA 0.2 + Nomic 0.5)
with graceful 2-way fallback on unsupported platforms.

- New nomic/ package with platform build tags and direct CGO bindings
  to llama.cpp (Metal on macOS, CPU on Linux)
- session_embeddings PK changed to (session_id, model) for coexistence
- Incremental index update at checkpoint time for new sessions
- All llama.cpp stderr noise suppressed for clean CLI output
- Supported platforms: darwin/arm64, linux/amd64

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Author: Frank Guo

.gitattributes

@@ -0,0 +1 @@
+cmd/rekal/cli/nomic/models/*.gguf.gz filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -22,3 +22,6 @@
 # Local config (optional; uncomment if you don't want to track)
 # .config/
 .rekal/
+
+# Build dependencies (llama.cpp)
+.deps/

CLAUDE.md

@@ -28,6 +28,8 @@ This repo contains the CLI for Rekal — gives your agent precise memory.
   - `version.go`: Version constant (set via ldflags)
   - `versioncheck/`: Auto-update notification
   - `db/`: DuckDB backend (open, close, schema)
+  - `lsa/`: Latent Semantic Analysis embeddings
+  - `nomic/`: Nomic-embed-text deep semantic embeddings (platform build tags)
   - `integration_test/`: Integration tests (`//go:build integration`)
 
 ### Specifications (`docs/spec/`)
@@ -140,7 +142,7 @@ All commands except `init` and `clean` must call both:
 
 Two databases in `.rekal/`:
 - `data.db` — source of truth (sessions, checkpoints, files_touched, checkpoint_sessions)
-- `index.db` — derived index (turns_ft, tool_calls_index, files_index, session_facets, file_cooccurrence)
+- `index.db` — derived index (turns_ft, tool_calls_index, files_index, session_facets, file_cooccurrence, session_embeddings with LSA + nomic vectors)
 
 Use the `db` package to open connections:
 ```go

README.md

@@ -21,47 +21,48 @@ Your agent starts every session knowing *why* the code looks the way it does.
 
 ## What Makes Rekal Different
 
-- **Security first** — Everything stays local. Nothing leaves the boundaries of git. No external services, no cloud APIs, no telemetry. A single binary with zero runtime dependencies beyond git itself.
+- **Security first** — Everything stays local. Nothing leaves the boundaries of git. No external services, no cloud APIs, no telemetry. A single binary with zero runtime dependencies beyond git itself. Even the embedding model (nomic-embed-text-v1.5) ships inside the binary — no downloads, no servers, no setup.
 - **Immutable by design** — Session snapshots are append-only. Content-hash deduplication means two developers always write to disjoint rows — merge conflicts are structurally impossible. Rekal never updates or deletes a session row.
 - **Team-shared memory** — `rekal push` and `rekal sync` share session context across your entire team through git. Every developer's agent benefits from every other developer's prior sessions.
 - **Git-native** — No external infrastructure. Rekal data lives on standard orphan branches, syncs through your existing remote, and uses git's object store for point-in-time recovery. Every checkpoint is anchored to a commit SHA.
-- **DuckDB-powered** — Full-text search (BM25), LSA vector embeddings, and file co-occurrence graphs built on DuckDB. The index is local-only and rebuilt on demand from the shared data.
+- **DuckDB-powered** — Full-text search (BM25), LSA vector embeddings, and nomic-embed-text deep semantic embeddings — all running locally inside a single binary. Three-way hybrid scoring (BM25 + LSA + Nomic) with graceful fallback on unsupported platforms. File co-occurrence graphs built on DuckDB. The index is local-only and rebuilt on demand from the shared data.
 - **Agent-first** — Progressive context loading. `rekal <query>` returns scored snippets and metadata — just enough for the agent to decide what matters. `rekal query --session <id>` drills into a specific session for full turns. The agent controls how much context it loads.
 - **Signal, not bulk** — A 2-10 MB session file becomes a ~300 byte payload. The wire format is a custom binary codec with zstd compression, string interning via varint references, and append-only framing.
 
 ## How It Works
 
 ```mermaid
-flowchart TB
-    subgraph capture ["Capture (post-commit hook)"]
-        A["AI Session<br/>prompts, responses,<br/>tool calls, files"] -->|"rekal checkpoint"| B["data.db<br/><i>DuckDB — append-only</i>"]
+flowchart LR
+    subgraph capture ["Capture"]
+        A["AI Session"] -->|"rekal checkpoint<br/>(post-commit)"| B[("data.db<br/>append-only")]
     end
 
-    subgraph index ["Index (local-only, rebuilt on demand)"]
-        B -->|"rekal index"| C["index.db<br/><i>DuckDB — derived</i>"]
-        C --- D["BM25 full-text search"]
-        C --- E["LSA vector embeddings"]
-        C --- F["File co-occurrence graph"]
-        C --- G["Session facets &amp; filters"]
+    subgraph transport ["Transport"]
+        B -->|"rekal push"| C["Wire Format<br/>zstd + varint interning"]
+        C -->|"git push<br/>rekal/&lt;email&gt;"| D[("Remote<br/>orphan branch")]
     end
 
-    subgraph transport ["Transport (git orphan branches)"]
-        B -->|"rekal push"| H["Wire Format<br/><i>rekal.body + dict.bin</i><br/>zstd · varint interning · append-only"]
-        H -->|"git push origin<br/>rekal/&lt;email&gt;"| I["Remote<br/><i>per-user orphan branch</i>"]
-        I -->|"rekal sync"| C
+    subgraph index ["Index"]
+        B -->|"rekal index"| E[("index.db<br/>local-only")]
+        D -->|"rekal sync"| E
+        E --- F["BM25 FTS"]
+        E --- G["LSA Embeddings"]
+        E --- N["Nomic Deep Embeddings"]
+        E --- H["Co-occurrence"]
+        E --- I["Facets"]
     end
 
-    subgraph query ["Query (agent-driven)"]
-        J["rekal 'keyword'"] -->|"hybrid BM25 + LSA"| C
-        C -->|"scored JSON"| K["Agent"]
-        K -->|"rekal query --session &lt;id&gt;"| B
+    subgraph query ["Query"]
+        J["rekal 'keyword'"] -->|"hybrid search"| E
+        E -->|"scored JSON"| K["Agent"]
+        K -->|"rekal query<br/>--session &lt;id&gt;"| B
         B -->|"full conversation"| K
     end
 
-    style capture fill:#1a1a2e,stroke:#e94560,color:#eee
-    style index fill:#1a1a2e,stroke:#0f3460,color:#eee
-    style transport fill:#1a1a2e,stroke:#16213e,color:#eee
-    style query fill:#1a1a2e,stroke:#533483,color:#eee
+    style capture fill:#fff5f5,stroke:#e94560,color:#333
+    style transport fill:#f0fdf4,stroke:#22c55e,color:#333
+    style index fill:#f0f4ff,stroke:#3b82f6,color:#333
+    style query fill:#faf5ff,stroke:#a855f7,color:#333
 ```
 
 When you commit, Rekal automatically snapshots your active AI session into a local DuckDB database. `rekal push` shares it with your team on a per-user orphan branch — your git history stays clean.
@@ -77,9 +78,11 @@ When you commit, Rekal automatically snapshots your active AI session into a loc
 # Install
 curl -fsSL https://raw.githubusercontent.com/rekal-dev/cli/main/scripts/install.sh | bash
 
+# Install to a specific directory
+curl -fsSL https://raw.githubusercontent.com/rekal-dev/cli/main/scripts/install.sh | bash -s -- --target /opt/bin
 ```
 
-Install location: `~/.local/bin` (override with `REKAL_INSTALL_DIR`).
+Install location: `~/.local/bin` (override with `--target <dir>` or `REKAL_INSTALL_DIR`).
 
 ```bash
 # Initialize in a git repo
@@ -104,7 +107,7 @@ When a newer release is available, the CLI prints an update notice after each co
 | `rekal sync [--self]` | Sync team context from remote rekal branches |
 | `rekal index` | Rebuild the index DB from the data DB |
 | `rekal log [--limit N]` | Show recent checkpoints |
-| `rekal [filters...] [query]` | Recall — hybrid search (BM25 + LSA) over sessions |
+| `rekal [filters...] [query]` | Recall — hybrid search (BM25 + LSA + Nomic) over sessions |
 | `rekal query --session <id> [--full]` | Drill into a session (turns, tool calls, files) |
 | `rekal query "<sql>" [--index]` | Run raw SQL against the data or index DB |
 
@@ -161,7 +164,7 @@ rekal --file src/billing/ "why discount logic"
 Rekal uses two local DuckDB databases and a compact binary wire format:
 
 - **Data DB** (`.rekal/data.db`) — Append-only shared truth. Normalized tables: sessions, turns, tool calls, checkpoints, files touched. The local query interface via `rekal query`.
-- **Index DB** (`.rekal/index.db`) — Local-only search intelligence. Full-text indexes (BM25), LSA vector embeddings, file co-occurrence graphs. Never synced. Rebuild anytime with `rekal index`.
+- **Index DB** (`.rekal/index.db`) — Local-only search intelligence. Full-text indexes (BM25), LSA vector embeddings, nomic-embed-text deep semantic embeddings, file co-occurrence graphs. Never synced. Rebuild anytime with `rekal index`.
 - **Wire format** (`rekal.body` + `dict.bin`) — Stored on per-user orphan branches (`rekal/<email>`). Append-only binary frames with zstd compression. This is what gets pushed/synced via git — the DuckDB databases are rebuilt from it.
 
 The wire format can be inspected from any point in time using git:

cmd/rekal/cli/checkpoint.go

@@ -8,6 +8,7 @@ import (
 	"math/rand"
 	"os"
 	"os/exec"
+	"path/filepath"
 	"strings"
 	"time"
 
@@ -243,6 +244,12 @@ func doCheckpoint(gitRoot string, w io.Writer) error {
 		}
 	}
 
+	// Incrementally update the index for newly captured sessions.
+	if err := updateIndexIncremental(gitRoot, sessionIDs, checkpointID, w); err != nil {
+		// Non-fatal — index can be rebuilt later with 'rekal index'.
+		fmt.Fprintf(w, "rekal: warning: incremental index update failed: %v\n", err)
+	}
+
 	fmt.Fprintf(w, "rekal: %d session(s) captured\n", inserted)
 	return nil
 }
@@ -291,3 +298,39 @@ func sha256Hex(data []byte) string {
 	h := sha256.Sum256(data)
 	return hex.EncodeToString(h[:])
 }
+
+// updateIndexIncremental adds newly captured sessions to the index DB
+// without a full rebuild. Handles: turns_ft, tool_calls_index, session_facets,
+// files_index, and nomic embeddings. LSA is skipped (requires full corpus).
+// FTS pragma_create_fts_index is not re-run — new rows in turns_ft are
+// automatically indexed by DuckDB's FTS.
+func updateIndexIncremental(gitRoot string, sessionIDs []string, checkpointID string, w io.Writer) error {
+	indexPath := filepath.Join(gitRoot, ".rekal", "index.db")
+	if _, err := os.Stat(indexPath); err != nil {
+		// No index DB yet — skip incremental update. Next 'rekal index' or 'rekal sync' will build it.
+		return nil
+	}
+
+	indexDB, err := db.OpenIndex(gitRoot)
+	if err != nil {
+		return err
+	}
+	defer indexDB.Close()
+
+	// Populate index tables for new sessions.
+	if err := db.PopulateIndexIncremental(indexDB, gitRoot, sessionIDs, checkpointID); err != nil {
+		return fmt.Errorf("populate index: %w", err)
+	}
+
+	// Nomic embeddings for new sessions (non-fatal).
+	sessionContent, err := db.QuerySessionContentByIDs(indexDB, sessionIDs)
+	if err != nil || len(sessionContent) == 0 {
+		return err
+	}
+
+	if err := buildNomicEmbeddings(indexDB, sessionContent, w); err != nil {
+		fmt.Fprintf(w, "rekal: warning: nomic embeddings skipped: %v\n", err)
+	}
+
+	return nil
+}

cmd/rekal/cli/db/indexer.go

@@ -268,6 +268,119 @@ func float64SliceToDuckDB(v []float64) string {
 	return b.String()
 }
 
+// QueryEmbeddings returns session_id → embedding vector for a given model.
+func QueryEmbeddings(d *sql.DB, model string) (map[string][]float64, error) {
+	rows, err := d.Query("SELECT session_id, embedding FROM session_embeddings WHERE model = $1", model)
+	if err != nil {
+		return nil, fmt.Errorf("query embeddings: %w", err)
+	}
+	defer rows.Close() //nolint:errcheck
+
+	result := make(map[string][]float64)
+	for rows.Next() {
+		var sid string
+		var emb []float64
+		if err := rows.Scan(&sid, &emb); err != nil {
+			return nil, fmt.Errorf("scan embedding: %w", err)
+		}
+		result[sid] = emb
+	}
+	return result, rows.Err()
+}
+
+// PopulateIndexIncremental adds new sessions to the index without a full rebuild.
+// sessionIDs are the newly captured sessions. checkpointID is the new checkpoint.
+func PopulateIndexIncremental(d *sql.DB, gitRoot string, sessionIDs []string, checkpointID string) error {
+	dataPath := filepath.Join(gitRoot, ".rekal", "data.db")
+
+	if _, err := d.Exec(fmt.Sprintf("ATTACH '%s' AS data_db (READ_ONLY)", dataPath)); err != nil {
+		return fmt.Errorf("attach data_db: %w", err)
+	}
+	defer d.Exec("DETACH data_db") //nolint:errcheck
+
+	for _, sid := range sessionIDs {
+		// turns_ft
+		if _, err := d.Exec(`
+			INSERT INTO turns_ft (id, session_id, turn_index, role, content, ts)
+			SELECT id, session_id, turn_index, role, content, CAST(ts AS VARCHAR)
+			FROM data_db.turns WHERE session_id = $1
+		`, sid); err != nil {
+			return fmt.Errorf("incremental turns_ft: %w", err)
+		}
+
+		// tool_calls_index
+		if _, err := d.Exec(`
+			INSERT INTO tool_calls_index (id, session_id, call_order, tool, path, cmd_prefix)
+			SELECT id, session_id, call_order, tool, path, cmd_prefix
+			FROM data_db.tool_calls WHERE session_id = $1
+		`, sid); err != nil {
+			return fmt.Errorf("incremental tool_calls_index: %w", err)
+		}
+
+		// session_facets
+		if _, err := d.Exec(`
+			INSERT INTO session_facets (
+				session_id, user_email, git_branch, actor_type, agent_id,
+				captured_at, turn_count, tool_call_count, file_count,
+				checkpoint_id, git_sha
+			)
+			SELECT
+				s.id, s.user_email,
+				COALESCE(c.git_branch, s.branch),
+				s.actor_type, s.agent_id, s.captured_at,
+				(SELECT count(*) FROM data_db.turns t WHERE t.session_id = s.id),
+				(SELECT count(*) FROM data_db.tool_calls tc WHERE tc.session_id = s.id),
+				COALESCE(fc.cnt, 0),
+				c.id, c.git_sha
+			FROM data_db.sessions s
+			LEFT JOIN data_db.checkpoint_sessions cs ON cs.session_id = s.id
+			LEFT JOIN data_db.checkpoints c ON c.id = cs.checkpoint_id
+			LEFT JOIN (
+				SELECT cs2.session_id, count(DISTINCT ft.file_path) AS cnt
+				FROM data_db.checkpoint_sessions cs2
+				JOIN data_db.files_touched ft ON ft.checkpoint_id = cs2.checkpoint_id
+				WHERE cs2.session_id = $1
+				GROUP BY cs2.session_id
+			) fc ON fc.session_id = s.id
+			WHERE s.id = $1
+		`, sid); err != nil {
+			return fmt.Errorf("incremental session_facets: %w", err)
+		}
+	}
+
+	// files_index for the new checkpoint
+	if _, err := d.Exec(`
+		INSERT INTO files_index (checkpoint_id, session_id, file_path, change_type)
+		SELECT ft.checkpoint_id, cs.session_id, ft.file_path, ft.change_type
+		FROM data_db.files_touched ft
+		JOIN data_db.checkpoint_sessions cs ON cs.checkpoint_id = ft.checkpoint_id
+		WHERE ft.checkpoint_id = $1
+	`, checkpointID); err != nil {
+		return fmt.Errorf("incremental files_index: %w", err)
+	}
+
+	return nil
+}
+
+// QuerySessionContentByIDs returns session_id → concatenated turn content for specific sessions.
+func QuerySessionContentByIDs(d *sql.DB, sessionIDs []string) (map[string]string, error) {
+	result := make(map[string]string, len(sessionIDs))
+	for _, sid := range sessionIDs {
+		var content string
+		err := d.QueryRow(`
+			SELECT string_agg(content, ' ' ORDER BY turn_index)
+			FROM turns_ft WHERE session_id = $1
+		`, sid).Scan(&content)
+		if err != nil && err != sql.ErrNoRows {
+			return nil, fmt.Errorf("query session content for %s: %w", sid, err)
+		}
+		if content != "" {
+			result[sid] = content
+		}
+	}
+	return result, nil
+}
+
 // QuerySessionContent returns session_id → concatenated turn content for LSA.
 func QuerySessionContent(d *sql.DB) (map[string]string, error) {
 	rows, err := d.Query(`

cmd/rekal/cli/db/schema.go

@@ -138,10 +138,11 @@ CREATE TABLE IF NOT EXISTS file_cooccurrence (
 );
 
 CREATE TABLE IF NOT EXISTS session_embeddings (
-	session_id      VARCHAR PRIMARY KEY,
+	session_id      VARCHAR NOT NULL,
 	embedding       FLOAT[],
 	model           VARCHAR NOT NULL,
-	generated_at    TIMESTAMP NOT NULL
+	generated_at    TIMESTAMP NOT NULL,
+	PRIMARY KEY (session_id, model)
 );
 
 CREATE TABLE IF NOT EXISTS index_state (

cmd/rekal/cli/index_cmd.go

@@ -1,11 +1,14 @@
 package cli
 
 import (
+	"database/sql"
 	"fmt"
+	"io"
 	"strconv"
 
 	"github.com/rekal-dev/cli/cmd/rekal/cli/db"
 	"github.com/rekal-dev/cli/cmd/rekal/cli/lsa"
+	"github.com/rekal-dev/cli/cmd/rekal/cli/nomic"
 	"github.com/spf13/cobra"
 )
 
@@ -17,7 +20,8 @@ func newIndexCmd() *cobra.Command {
 
 The index is local-only and never synced. It contains:
   - Full-text search index (BM25) over conversation turns
-  - LSA vector embeddings for semantic similarity
+  - LSA vector embeddings for lightweight semantic similarity
+  - Nomic deep semantic embeddings (on supported platforms)
   - Session facets (author, branch, actor, counts) for fast filtering
   - File co-occurrence graph
   - Tool call indexes
@@ -108,7 +112,12 @@ func runIndex(cmd *cobra.Command, gitRoot string) error {
 				return fmt.Errorf("store embeddings: %w", err)
 			}
 			embeddingDim = model.Dim
-			fmt.Fprintf(w, "stored %d embeddings (%d dimensions)\n", len(vectors), embeddingDim)
+			fmt.Fprintf(w, "stored %d LSA embeddings (%d dimensions)\n", len(vectors), embeddingDim)
+		}
+
+		// Nomic pass (non-fatal).
+		if err := buildNomicEmbeddings(indexDB, sessionContent, w); err != nil {
+			fmt.Fprintf(w, "warning: nomic embeddings skipped: %v\n", err)
 		}
 	}
 
@@ -129,3 +138,29 @@ func runIndex(cmd *cobra.Command, gitRoot string) error {
 	fmt.Fprintf(w, "index rebuilt: %d sessions, %d turns\n", sessionCount, turnCount)
 	return nil
 }
+
+// buildNomicEmbeddings generates nomic-embed-text embeddings for all sessions
+// and stores them in the index DB. Non-fatal: returns error on any failure.
+func buildNomicEmbeddings(indexDB *sql.DB, sessionContent map[string]string, w io.Writer) error {
+	if !nomic.Supported() {
+		return nil
+	}
+
+	fmt.Fprintln(w, "building nomic deep semantic embeddings...")
+	embedder, err := nomic.NewEmbedder()
+	if err != nil {
+		return err
+	}
+	defer embedder.Close()
+
+	vectors, err := embedder.EmbedSessions(sessionContent)
+	if err != nil {
+		return err
+	}
+
+	if err := db.StoreEmbeddings(indexDB, vectors, nomic.ModelName); err != nil {
+		return err
+	}
+	fmt.Fprintf(w, "stored %d nomic embeddings (%d dimensions)\n", len(vectors), nomic.EmbedDim)
+	return nil
+}