diff --git a/AGENTS.md b/AGENTS.md index 92ee3b4..80afb4e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,6 +1,15 @@ # Agent Instructions -## Internal Documentation +## Codebase Context + +**At start of each task:** Call `get_memory` to load team conventions. + +**CRITICAL:** When user says "remember this" or "record this": +- STOP immediately and call `remember` tool FIRST +- DO NOT proceed with other actions until memory is recorded +- This is a blocking requirement, not optional + +## Internal Documentation (Submodule) This repository uses a private git submodule for internal notes. @@ -20,6 +29,6 @@ git pull --recurse-submodules git submodule update --remote --merge ``` -### Privacy & Security +### Privacy -The `internal-docs` repository is **Private**. It returns a 404 to unauthenticated users/APIs. Access requires a GitHub PAT or SSH keys with repository permissions. +The `internal-docs` repository is private. It returns a 404 to unauthenticated users. Access requires a GitHub PAT or SSH keys with repository permissions. diff --git a/MOTIVATION.md b/MOTIVATION.md index c406fe9..c95e429 100644 --- a/MOTIVATION.md +++ b/MOTIVATION.md @@ -1,6 +1,6 @@ # Motivation: Why This Exists -> **TL;DR**: AI coding assistants are smart but dangerous. Without guidance, they "vibe code" their way into technical debt. This MCP gives them **Context** (to know your patterns) and **Wisdom** (to keep your codebase healthy). +> **TL;DR**: AI coding assistants increase throughput but often degrade stability. Without codebase context, they generate code that works but violates team conventions and architectural rules. This MCP provides structured pattern data and recorded rationale so agents produce code that fits. --- @@ -28,7 +28,7 @@ AI drastically increases **Throughput** (more code/hour) but often kills **Stabi ## What This Does -We provide **Active Context**—not just raw data, but the *judgment* of a Senior Engineer. +This MCP provides **active context** - not raw data, but structured intelligence derived from actual codebase state. ### 1. Pattern Discovery (The "Map") - **Frequency Detection**: "97% use `inject()`, 3% use `constructor`." (Consensus) @@ -40,9 +40,8 @@ We provide **Active Context**—not just raw data, but the *judgment* of a Senio - **Health Context**: "⚠️ Careful, `UserService.ts` is a high-churn hotspot with circular dependencies. Add tests." ### Works with AGENTS.md -> **AGENTS.md is the Law. MCP is the Map.** -- **AGENTS.md** says: "We prefer functional functional programming." -- **MCP** shows: "Here are the 5 most recent functional patterns we actually used." +- **AGENTS.md** defines intent: "Use functional patterns." +- **MCP** provides evidence: "Here are the 5 most recent functional patterns actually used." --- @@ -50,9 +49,9 @@ We provide **Active Context**—not just raw data, but the *judgment* of a Senio | Limitation | Mitigation | |------------|--------| -| **Pattern frequency ≠ pattern quality** | We added **Pattern Momentum** (Rise/Fall trends) to fix this. | +| **Pattern frequency ≠ pattern quality** | **Pattern Momentum** (Rise/Fall trends) distinguishes adoption direction from raw count. | | **Stale index risk** | Manual re-indexing required for now. | -| **Framework coverage** | Angular-specialized. React/Vue analyzers extensible. | +| **Framework coverage** | Deep analysis for Angular. Generic analyzer covers 30+ languages. React/Vue specialized analyzers extensible. | | **File-level trend detection** | Trend is based on file modification date, not line-by-line content. A recently modified file may still contain legacy patterns on specific lines. Future: AST-based line-level detection. | --- @@ -61,7 +60,7 @@ We provide **Active Context**—not just raw data, but the *judgment* of a Senio 1. **Context alone is dangerous**: Giving AI "all the context" just confuses it or teaches it bad habits (Search Contamination). 2. **Decisions > Data**: AI needs *guidance* ("Use X"), not just *options* ("Here is X and Y"). -3. **Governance through Discovery**: We don't need to block PRs to be useful. If we show the AI that a pattern is "Declining" and "Dangerous," it self-corrects. +3. **Governance through Discovery**: Blocking PRs is not required. If the AI sees that a pattern is "Declining" and "Dangerous," it self-corrects. --- @@ -76,7 +75,3 @@ We provide **Active Context**—not just raw data, but the *judgment* of a Senio - **Search Contamination**: Without MCP, models copied legacy patterns 40% of the time. - **Momentum Success**: With "Trending" signals, models adopted modern patterns even when they were the minority (3%). ---- - -*Last updated: December 2025* - diff --git a/README.md b/README.md index 06c189a..6675fca 100644 --- a/README.md +++ b/README.md @@ -1,45 +1,183 @@ # codebase-context -**AI coding agents don't know your codebase. This MCP fixes that.** +[![npm version](https://img.shields.io/npm/v/codebase-context)](https://www.npmjs.com/package/codebase-context) [![license](https://img.shields.io/npm/l/codebase-context)](./LICENSE) [![node](https://img.shields.io/node/v/codebase-context)](./package.json) -Your team has internal libraries, naming conventions, and patterns that external AI models have never seen. This MCP server gives AI assistants real-time visibility into your codebase: which libraries your team actually uses, how often, and where to find canonical examples. +A second brain for AI coding agents. MCP server that remembers team decisions, tracks pattern evolution, and guides every edit with evidence. ## Quick Start -Add this to your MCP client config (Claude Desktop, VS Code, Cursor, etc.). +### Claude Desktop + +Add to `claude_desktop_config.json`: + +```json +{ + "mcpServers": { + "codebase-context": { + "command": "npx", + "args": ["-y", "codebase-context", "/path/to/your/project"] + } + } +} +``` + +### VS Code (Copilot) + +Add `.vscode/mcp.json` to your project root: ```json -"mcpServers": { - "codebase-context": { - "command": "npx", - "args": ["codebase-context", "/path/to/your/project"] +{ + "servers": { + "codebase-context": { + "command": "npx", + "args": ["-y", "codebase-context", "${workspaceFolder}"] + } } } ``` -If your environment prompts on first run, use `npx --yes ...` (or `npx -y ...`) to auto-confirm. +### Cursor + +Add to `.cursor/mcp.json` in your project: + +```json +{ + "mcpServers": { + "codebase-context": { + "command": "npx", + "args": ["-y", "codebase-context", "/path/to/your/project"] + } + } +} +``` + +### Windsurf + +Open Settings > MCP and add: + +```json +{ + "mcpServers": { + "codebase-context": { + "command": "npx", + "args": ["-y", "codebase-context", "/path/to/your/project"] + } + } +} +``` + +### Claude Code + +No config file needed. Add to `.claude/settings.json` or run: + +```bash +claude mcp add codebase-context -- npx -y codebase-context /path/to/your/project +``` + +## What Makes It a Second Brain + +Other tools help AI find code. This one helps AI make the right decisions — by remembering what your team does, tracking how patterns evolve, and warning before mistakes repeat. -## What You Get +### Remembers -- **Internal library discovery** → `@mycompany/ui-toolkit`: 847 uses vs `primeng`: 3 uses -- **Pattern frequencies** → `inject()`: 97%, `constructor()`: 3% -- **Pattern momentum** → `Signals`: Rising (last used 2 days ago) vs `RxJS`: Declining (180+ days) -- **Golden file examples** → Real implementations showing all patterns together -- **Testing conventions** → `Jest`: 74%, `Playwright`: 6% -- **Framework patterns** → Angular signals, standalone components, etc. -- **Circular dependency detection** → Find toxic import cycles between files -- **Memory system** → Record "why" behind choices so AI doesn't repeat mistakes +Decisions, rationale, and past failures persist across sessions. Not just what the team does — why. + +- Internal library usage: `@mycompany/ui-toolkit` (847 uses) vs `primeng` (3 uses) — and _why_ the wrapper exists +- "Tried direct PrimeNG toast, broke event system" — recorded as a failure memory, surfaced before the next agent repeats it +- Conventions from git history auto-extracted: `refactor:`, `migrate:`, `fix:`, `revert:` commits become memories with zero manual effort + +### Reasons + +Quantified pattern analysis with trend direction. Not "use inject()" — "97% of the team uses inject(), and it's rising." + +- `inject()`: 97% adoption vs `constructor()`: 3% — with trend direction (rising/declining) +- `Signals`: rising (last used 2 days ago) vs `RxJS BehaviorSubject`: declining (180+ days) +- Golden files: real implementations scoring highest on modern pattern density — canonical examples to follow +- Pattern conflicts detected: when two approaches in the same category both exceed 20% adoption + +### Protects + +Before an edit happens, the agent gets a preflight briefing: what to use, what to avoid, what broke last time. + +- Preflight card on `search_codebase` with `intent: "edit"` — risk level, preferred/avoid patterns, failure warnings, golden files, impact candidates +- Failure memories bump risk level and surface as explicit warnings +- Confidence decay: memories age (90-day or 180-day half-life). Stale guidance gets flagged, not blindly trusted +- Epistemic stress detection: when evidence is contradictory, stale, or too thin, the preflight card says "insufficient evidence" instead of guessing + +### Discovers + +Hybrid search (BM25 keyword 30% + vector embeddings 70%) with structured filters across 30+ languages: + +- **Framework**: Angular, React, Vue +- **Language**: TypeScript, JavaScript, Python, Go, Rust, and 25+ more +- **Component type**: component, service, directive, guard, interceptor, pipe +- **Architectural layer**: presentation, business, data, state, core, shared +- Circular dependency detection, style guide auto-detection, architectural layer classification + +## Measured Results + +Tested against a real enterprise Angular codebase (~30k files): + +| What was measured | Result | +| ---------------------------------- | -------------------------------------------------------- | +| Internal library detection | 336 uses of `@company/ui-toolkit` vs 3 direct PrimeNG | +| DI pattern consensus | 98% `inject()` adoption detected, constructor DI flagged | +| Test framework detection | 74% Jest, 26% Jasmine/Karma, per-module awareness | +| Wrapper discovery | `ToastEventService`, `DialogComponent` surfaced over raw | +| Golden file identification | Top 5 files scoring 4-6 modern patterns each | + +Without this context, AI agents default to generic patterns: raw PrimeNG imports, constructor injection, Jasmine syntax. With the second brain active, generated code matches the existing codebase on first attempt. ## How It Works -When generating code, the agent checks your patterns first: +The difference in practice: -| Without MCP | With MCP | +| Without second brain | With second brain | | ---------------------------------------- | ------------------------------------ | | Uses `constructor(private svc: Service)` | Uses `inject()` (97% team adoption) | | Suggests `primeng/button` directly | Uses `@mycompany/ui-toolkit` wrapper | | Generic Jest setup | Your team's actual test utilities | +### Preflight Card + +When using `search_codebase` with `intent: "edit"`, `"refactor"`, or `"migrate"`, the response includes a preflight card alongside search results: + +```json +{ + "preflight": { + "intent": "refactor", + "riskLevel": "medium", + "confidence": "fresh", + "evidenceLock": { + "mode": "triangulated", + "status": "pass", + "readyToEdit": true, + "score": 100, + "sources": [ + { "source": "code", "strength": "strong", "count": 5 }, + { "source": "patterns", "strength": "strong", "count": 3 }, + { "source": "memories", "strength": "strong", "count": 2 } + ] + }, + "preferredPatterns": [ + { "pattern": "inject() function", "category": "dependencyInjection", "adoption": "98%", "trend": "Rising" } + ], + "avoidPatterns": [ + { "pattern": "Constructor injection", "category": "dependencyInjection", "adoption": "2%", "trend": "Declining" } + ], + "goldenFiles": [ + { "file": "src/features/auth/auth.service.ts", "score": 6 } + ], + "failureWarnings": [ + { "memory": "Direct PrimeNG toast broke event system", "reason": "Must use ToastEventService" } + ] + }, + "results": [...] +} +``` + +One call. The second brain composes patterns, memories, failures, and risk into a single response. + ### Tip: Auto-invoke in your rules Add this to your `.cursorrules`, `CLAUDE.md`, or `AGENTS.md`: @@ -59,18 +197,22 @@ Now the agent checks patterns automatically instead of waiting for you to ask. ## Tools -| Tool | Purpose | -| ------------------------------ | --------------------------------------------- | -| `search_codebase` | Semantic + keyword hybrid search | -| `get_component_usage` | Find where a library/component is used | -| `get_team_patterns` | Pattern frequencies + canonical examples | -| `get_codebase_metadata` | Project structure overview | -| `get_indexing_status` | Indexing progress + last stats | -| `get_style_guide` | Query style guide rules | -| `detect_circular_dependencies` | Find import cycles between files | -| `remember` | Record memory (conventions/decisions/gotchas) | -| `get_memory` | Query recorded memory by category/keyword | -| `refresh_index` | Re-index the codebase | +| Tool | Purpose | +| ------------------------------ | -------------------------------------------------------------------- | +| `search_codebase` | Hybrid search with filters. Pass `intent: "edit"` for preflight card | +| `get_component_usage` | Find where a library/component is used | +| `get_team_patterns` | Pattern frequencies, golden files, conflict detection | +| `get_codebase_metadata` | Project structure overview | +| `get_indexing_status` | Indexing progress + last stats | +| `get_style_guide` | Query style guide rules | +| `detect_circular_dependencies` | Find import cycles between files | +| `remember` | Record memory (conventions/decisions/gotchas/failures) | +| `get_memory` | Query memory with confidence decay scoring | +| `refresh_index` | Re-index the codebase + extract git memories | + +## Language Support + +The Angular analyzer provides deep framework-specific analysis (signals, standalone components, control flow syntax, lifecycle hooks, DI patterns). A generic analyzer covers 30+ languages and file types as a fallback: JavaScript, TypeScript, Python, Java, Kotlin, C/C++, C#, Go, Rust, PHP, Ruby, Swift, Scala, Shell, and common config/markup formats. ## File Structure @@ -97,22 +239,27 @@ The MCP creates the following structure in your project: Patterns tell you _what_ the team does ("97% use inject"), but not _why_ ("standalone compatibility"). Use `remember` to capture rationale that prevents repeated mistakes: ```typescript -// AI won't change this again after recording the decision remember({ type: 'decision', category: 'dependencies', memory: 'Use node-linker: hoisted, not isolated', - reason: - "Some packages don't declare transitive deps. Isolated forces manual package.json additions." + reason: "Some packages don't declare transitive deps." }); ``` -Memories surface automatically in `search_codebase` results and `get_team_patterns` responses. +**Memory types:** `convention` (style rules), `decision` (architecture choices), `gotcha` (things that break), `failure` (tried X, failed because Y). + +**Confidence decay:** Memories age. Conventions never decay. Decisions have a 180-day half-life. Gotchas and failures have a 90-day half-life. Memories below 30% confidence are flagged as stale in `get_memory` responses. + +**Git auto-extraction:** During indexing, conventional commits (`refactor:`, `migrate:`, `fix:`, `revert:`) from the last 90 days are auto-recorded as memories. Zero manual effort. + +**Pattern conflicts:** `get_team_patterns` detects when two patterns in the same category are both above 20% adoption with different trends, and surfaces them as conflicts with both sides. + +Memories surface automatically in `search_codebase` results, `get_team_patterns` responses, and preflight cards. -**Early baseline — known quirks:** +**Known quirks:** - Agents may bundle multiple things into one entry -- Duplicates can happen if you record the same thing twice - Edit `.codebase-context/memory.json` directly to clean up - Be explicit: "Remember this: use X not Y" @@ -125,19 +272,19 @@ Memories surface automatically in `search_codebase` results and `get_team_patter | `CODEBASE_ROOT` | - | Project root to index (CLI arg takes precedence) | | `CODEBASE_CONTEXT_DEBUG` | - | Set to `1` to enable verbose logging (startup messages, analyzer registration) | -## Performance Note +## Performance -This tool runs **locally** on your machine using your hardware. +This tool runs locally on your machine. -- **Initial Indexing**: The first run works hard. It may take several minutes (e.g., ~2-5 mins for 30k files) to compute embeddings for your entire codebase. -- **Caching**: Subsequent queries are instant (milliseconds). -- **Updates**: Currently, `refresh_index` re-scans the codebase. True incremental indexing (processing only changed files) is on the roadmap. +- **Initial indexing**: First run may take several minutes (e.g., 2-5 min for 30k files) to compute embeddings. +- **Subsequent queries**: Instant (milliseconds) from cache. +- **Updates**: `refresh_index` re-scans the codebase. True incremental indexing (processing only changed files) is on the roadmap. ## Links -- 📄 [Motivation](./MOTIVATION.md) — Why this exists, research, learnings -- 📋 [Changelog](./CHANGELOG.md) — Version history -- 🤝 [Contributing](./CONTRIBUTING.md) — How to add analyzers +- [Motivation](./MOTIVATION.md) — Research and design rationale +- [Changelog](./CHANGELOG.md) — Version history +- [Contributing](./CONTRIBUTING.md) — How to add analyzers ## License diff --git a/package.json b/package.json index 5ce3ec0..94c8e9d 100644 --- a/package.json +++ b/package.json @@ -1,7 +1,7 @@ { "name": "codebase-context", "version": "1.4.1", - "description": "MCP server that helps AI agents understand your codebase - patterns, libraries, architecture, monorepo support", + "description": "MCP server for codebase intelligence — patterns, conventions, architecture, and rationale for AI coding agents", "type": "module", "main": "./dist/lib.js", "types": "./dist/lib.d.ts", @@ -45,17 +45,25 @@ }, "keywords": [ "mcp", + "mcp-server", "model-context-protocol", - "semantic-search", "codebase", - "indexing", - "embeddings", + "code-intelligence", + "code-patterns", + "code-conventions", + "semantic-search", "vector-search", - "angular", + "embeddings", "ai", "llm", - "code-understanding", - "developer-tools" + "claude", + "cursor", + "copilot", + "angular", + "react", + "developer-tools", + "static-analysis", + "code-quality" ], "repository": { "type": "git", diff --git a/src/constants/codebase-context.ts b/src/constants/codebase-context.ts index 92bbc41..e3cf2af 100644 --- a/src/constants/codebase-context.ts +++ b/src/constants/codebase-context.ts @@ -7,3 +7,4 @@ export const MEMORY_FILENAME = 'memory.json' as const; export const INTELLIGENCE_FILENAME = 'intelligence.json' as const; export const KEYWORD_INDEX_FILENAME = 'index.json' as const; export const VECTOR_DB_DIRNAME = 'index' as const; +export const MANIFEST_FILENAME = 'manifest.json' as const; diff --git a/src/constants/git-patterns.ts b/src/constants/git-patterns.ts new file mode 100644 index 0000000..4a5eee2 --- /dev/null +++ b/src/constants/git-patterns.ts @@ -0,0 +1,18 @@ +import type { MemoryType, MemoryCategory } from '../types/index.js'; + +export interface GitCommitPattern { + prefix: RegExp; + type: MemoryType; + category: MemoryCategory; +} + +/** + * Conventional commit patterns that produce auto-extracted memories. + * Shared between production (extractGitMemories) and tests. + */ +export const GIT_COMMIT_PATTERNS: GitCommitPattern[] = [ + { prefix: /^[a-f0-9]+ refactor[(!:]/i, type: 'decision', category: 'architecture' }, + { prefix: /^[a-f0-9]+ migrate[(!:]/i, type: 'decision', category: 'dependencies' }, + { prefix: /^[a-f0-9]+ fix[(!:]/i, type: 'gotcha', category: 'conventions' }, + { prefix: /^[a-f0-9]+ revert[(!:]/i, type: 'gotcha', category: 'architecture' } +]; diff --git a/src/core/indexer.ts b/src/core/indexer.ts index 895d4f6..437cee0 100644 --- a/src/core/indexer.ts +++ b/src/core/indexer.ts @@ -32,13 +32,23 @@ import { CODEBASE_CONTEXT_DIRNAME, INTELLIGENCE_FILENAME, KEYWORD_INDEX_FILENAME, + MANIFEST_FILENAME, VECTOR_DB_DIRNAME } from '../constants/codebase-context.js'; +import { + computeFileHashes, + readManifest, + writeManifest, + diffManifest, + type FileManifest, + type ManifestDiff +} from './manifest.js'; export interface IndexerOptions { rootPath: string; config?: Partial; onProgress?: (progress: IndexingProgress) => void; + incrementalOnly?: boolean; } export class CodebaseIndexer { @@ -46,11 +56,13 @@ export class CodebaseIndexer { private config: CodebaseConfig; private progress: IndexingProgress; private onProgressCallback?: (progress: IndexingProgress) => void; + private incrementalOnly: boolean; constructor(options: IndexerOptions) { this.rootPath = path.resolve(options.rootPath); this.config = this.mergeConfig(options.config); this.onProgressCallback = options.onProgress; + this.incrementalOnly = options.incrementalOnly ?? false; this.progress = { phase: 'initializing', @@ -166,9 +178,56 @@ export class CodebaseIndexer { console.error(`Found ${files.length} files to index`); + // Phase 1b: Incremental diff (if incremental mode) + const contextDir = path.join(this.rootPath, CODEBASE_CONTEXT_DIRNAME); + const manifestPath = path.join(contextDir, MANIFEST_FILENAME); + let diff: ManifestDiff | null = null; + let currentHashes: Record | null = null; + + if (this.incrementalOnly) { + this.updateProgress('scanning', 10); + console.error('Computing file hashes for incremental diff...'); + currentHashes = await computeFileHashes(files, this.rootPath); + + const oldManifest = await readManifest(manifestPath); + diff = diffManifest(oldManifest, currentHashes); + + console.error( + `Incremental diff: ${diff.added.length} added, ${diff.changed.length} changed, ` + + `${diff.deleted.length} deleted, ${diff.unchanged.length} unchanged` + ); + + stats.incremental = { + added: diff.added.length, + changed: diff.changed.length, + deleted: diff.deleted.length, + unchanged: diff.unchanged.length + }; + + // Short-circuit: nothing changed + if (diff.added.length === 0 && diff.changed.length === 0 && diff.deleted.length === 0) { + console.error('No files changed — skipping re-index.'); + this.updateProgress('complete', 100); + stats.duration = Date.now() - startTime; + stats.completedAt = new Date(); + return stats; + } + } + + // Build the set of files that need analysis + embedding (incremental: only added/changed) + const filesToProcess = diff + ? files.filter((f) => { + const rel = path.relative(this.rootPath, f).replace(/\\/g, '/'); + return diff!.added.includes(rel) || diff!.changed.includes(rel); + }) + : files; + // Phase 2: Analyzing & Parsing + // Intelligence tracking (patterns, libraries, import graph) runs on ALL files + // but embedding only runs on filesToProcess this.updateProgress('analyzing', 0); const allChunks: CodeChunk[] = []; + const changedChunks: CodeChunk[] = []; // Only chunks from added/changed files const libraryTracker = new LibraryUsageTracker(); const patternDetector = new PatternDetector(); const importGraph = new ImportGraph(); @@ -177,6 +236,9 @@ export class CodebaseIndexer { // Fetch git commit dates for pattern momentum analysis const fileDates = await getFileCommitDates(this.rootPath); + // When incremental, track which files need embedding + const filesToProcessSet = diff ? new Set(filesToProcess.map((f) => f)) : null; + for (let i = 0; i < files.length; i++) { const file = files[i]; this.progress.currentFile = file; @@ -190,7 +252,12 @@ export class CodebaseIndexer { const result = await analyzerRegistry.analyzeFile(file, content); if (result) { + const isFileChanged = !filesToProcessSet || filesToProcessSet.has(file); + allChunks.push(...result.chunks); + if (isFileChanged) { + changedChunks.push(...result.chunks); + } stats.indexedFiles++; stats.totalLines += content.split('\n').length; @@ -331,22 +398,29 @@ export class CodebaseIndexer { ? Math.round(allChunks.reduce((sum, c) => sum + c.content.length, 0) / allChunks.length) : 0; + // Determine which chunks to embed: in incremental mode, only changed/added file chunks + const chunksForEmbedding = diff ? changedChunks : allChunks; + // Memory safety: limit chunks to prevent embedding memory issues const MAX_CHUNKS = 5000; - let chunksToEmbed = allChunks; - if (allChunks.length > MAX_CHUNKS) { + let chunksToEmbed = chunksForEmbedding; + if (chunksForEmbedding.length > MAX_CHUNKS) { console.warn( - `WARNING: ${allChunks.length} chunks exceed limit. Indexing first ${MAX_CHUNKS} chunks.` + `WARNING: ${chunksForEmbedding.length} chunks exceed limit. Indexing first ${MAX_CHUNKS} chunks.` ); - chunksToEmbed = allChunks.slice(0, MAX_CHUNKS); + chunksToEmbed = chunksForEmbedding.slice(0, MAX_CHUNKS); } - // Phase 3: Embedding + // Phase 3: Embedding (only changed/added chunks in incremental mode) const chunksWithEmbeddings: CodeChunkWithEmbedding[] = []; - if (!this.config.skipEmbedding) { + if (!this.config.skipEmbedding && chunksToEmbed.length > 0) { this.updateProgress('embedding', 50); - console.error(`Creating embeddings for ${chunksToEmbed.length} chunks...`); + console.error( + `Creating embeddings for ${chunksToEmbed.length} chunks` + + (diff ? ` (${allChunks.length} total, ${chunksToEmbed.length} changed)` : '') + + '...' + ); // Initialize embedding provider const embeddingProvider = await getEmbeddingProvider(this.config.embedding); @@ -389,32 +463,58 @@ export class CodebaseIndexer { ); } } - } else { + } else if (this.config.skipEmbedding) { console.error('Skipping embedding generation (skipEmbedding=true)'); + } else if (chunksToEmbed.length === 0 && diff) { + console.error('No chunks to embed (all unchanged)'); } // Phase 4: Storing this.updateProgress('storing', 75); - const contextDir = path.join(this.rootPath, CODEBASE_CONTEXT_DIRNAME); await fs.mkdir(contextDir, { recursive: true }); if (!this.config.skipEmbedding) { - console.error(`Storing ${chunksToEmbed.length} chunks...`); - - // Store in LanceDB for vector search const storagePath = path.join(contextDir, VECTOR_DB_DIRNAME); const storageProvider = await getStorageProvider({ path: storagePath }); - await storageProvider.clear(); // Clear existing index - await storageProvider.store(chunksWithEmbeddings); + + if (diff) { + // Incremental: delete old chunks for changed + deleted files, then add new + const filesToDelete = [...diff.changed, ...diff.deleted].map((rel) => + path.join(this.rootPath, rel).replace(/\\/g, '/') + ); + // Also try with OS-native separators for matching + const filePathsForDelete = [...diff.changed, ...diff.deleted].map((rel) => + path.resolve(this.rootPath, rel) + ); + const allDeletePaths = [...new Set([...filesToDelete, ...filePathsForDelete])]; + + if (allDeletePaths.length > 0) { + await storageProvider.deleteByFilePaths(allDeletePaths); + } + if (chunksWithEmbeddings.length > 0) { + await storageProvider.store(chunksWithEmbeddings); + } + console.error( + `Incremental store: deleted chunks for ${diff.changed.length + diff.deleted.length} files, ` + + `added ${chunksWithEmbeddings.length} new chunks` + ); + } else { + // Full: clear and re-store everything + console.error(`Storing ${chunksToEmbed.length} chunks...`); + await storageProvider.clear(); + await storageProvider.store(chunksWithEmbeddings); + } } - // Also save JSON for keyword search (Fuse.js) - use chunksToEmbed for consistency + // Keyword index always uses ALL chunks (full regen) const indexPath = path.join(contextDir, KEYWORD_INDEX_FILENAME); - // Write without pretty-printing to save memory - await fs.writeFile(indexPath, JSON.stringify(chunksToEmbed)); + // Memory safety: cap keyword index too + const keywordChunks = + allChunks.length > MAX_CHUNKS ? allChunks.slice(0, MAX_CHUNKS) : allChunks; + await fs.writeFile(indexPath, JSON.stringify(keywordChunks)); - // Save library usage and pattern stats + // Save library usage and pattern stats (always full regen) const intelligencePath = path.join(contextDir, INTELLIGENCE_FILENAME); const libraryStats = libraryTracker.getStats(); @@ -451,14 +551,30 @@ export class CodebaseIndexer { }; await fs.writeFile(intelligencePath, JSON.stringify(intelligence, null, 2)); + // Write manifest (both full and incremental) + const manifest: FileManifest = { + version: 1, + generatedAt: new Date().toISOString(), + files: currentHashes ?? (await computeFileHashes(files, this.rootPath)) + }; + await writeManifest(manifestPath, manifest); + // Phase 5: Complete this.updateProgress('complete', 100); stats.duration = Date.now() - startTime; stats.completedAt = new Date(); - console.error(`Indexing complete in ${stats.duration}ms`); - console.error(`Indexed ${stats.indexedFiles} files, ${stats.totalChunks} chunks`); + if (diff) { + console.error( + `Incremental indexing complete in ${stats.duration}ms ` + + `(${diff.added.length} added, ${diff.changed.length} changed, ` + + `${diff.deleted.length} deleted, ${diff.unchanged.length} unchanged)` + ); + } else { + console.error(`Indexing complete in ${stats.duration}ms`); + console.error(`Indexed ${stats.indexedFiles} files, ${stats.totalChunks} chunks`); + } return stats; } catch (error) { diff --git a/src/core/manifest.ts b/src/core/manifest.ts new file mode 100644 index 0000000..04f7c61 --- /dev/null +++ b/src/core/manifest.ts @@ -0,0 +1,110 @@ +/** + * File hash manifest for incremental indexing. + * Tracks SHA-256 hashes of indexed files to detect changes between runs. + */ + +import { createHash } from 'crypto'; +import { promises as fs } from 'fs'; +import path from 'path'; + +export interface FileManifest { + version: 1; + generatedAt: string; + files: Record; // relativePath → SHA-256 hash (first 16 hex chars) +} + +export interface ManifestDiff { + added: string[]; // new files (not in old manifest) + changed: string[]; // hash differs + deleted: string[]; // in old manifest but not on disk + unchanged: string[]; // hash matches +} + +/** + * Hash file content using SHA-256, returning first 16 hex chars. + * 16 hex chars = 64 bits of entropy — collision-safe for per-project file counts. + */ +export function hashFileContent(content: string): string { + return createHash('sha256').update(content).digest('hex').slice(0, 16); +} + +/** + * Read a manifest from disk. Returns null if missing or corrupt. + */ +export async function readManifest(manifestPath: string): Promise { + try { + const raw = await fs.readFile(manifestPath, 'utf-8'); + const parsed = JSON.parse(raw); + if (parsed.version !== 1 || typeof parsed.files !== 'object') { + return null; + } + return parsed as FileManifest; + } catch { + return null; + } +} + +/** + * Write a manifest to disk. + */ +export async function writeManifest(manifestPath: string, manifest: FileManifest): Promise { + await fs.mkdir(path.dirname(manifestPath), { recursive: true }); + await fs.writeFile(manifestPath, JSON.stringify(manifest)); +} + +/** + * Compute SHA-256 hashes for a list of absolute file paths. + * Returns a map of relativePath → hash. + */ +export async function computeFileHashes( + files: string[], + rootPath: string, + readFile: (p: string) => Promise = (p) => fs.readFile(p, 'utf-8') +): Promise> { + const hashes: Record = {}; + for (const file of files) { + try { + const content = await readFile(file); + const relativePath = path.relative(rootPath, file).replace(/\\/g, '/'); + hashes[relativePath] = hashFileContent(content); + } catch { + // Skip files that can't be read + } + } + return hashes; +} + +/** + * Diff an old manifest against current file hashes. + * If oldManifest is null (first run), all files are "added". + */ +export function diffManifest( + oldManifest: FileManifest | null, + currentHashes: Record +): ManifestDiff { + const oldFiles = oldManifest?.files ?? {}; + const added: string[] = []; + const changed: string[] = []; + const unchanged: string[] = []; + const deleted: string[] = []; + + // Check current files against old manifest + for (const [filePath, hash] of Object.entries(currentHashes)) { + if (!(filePath in oldFiles)) { + added.push(filePath); + } else if (oldFiles[filePath] !== hash) { + changed.push(filePath); + } else { + unchanged.push(filePath); + } + } + + // Check for deleted files (in old manifest but not in current) + for (const filePath of Object.keys(oldFiles)) { + if (!(filePath in currentHashes)) { + deleted.push(filePath); + } + } + + return { added, changed, deleted, unchanged }; +} diff --git a/src/index.ts b/src/index.ts index ca53c1d..972881c 100644 --- a/src/index.ts +++ b/src/index.ts @@ -46,8 +46,11 @@ import { appendMemoryFile, readMemoriesFile, filterMemories, - applyUnfilteredLimit + applyUnfilteredLimit, + withConfidence } from './memory/store.js'; +import { parseGitLogLineToMemory } from './memory/git-memory.js'; +import { buildEvidenceLock } from './preflight/evidence-lock.js'; analyzerRegistry.register(new AngularAnalyzer()); analyzerRegistry.register(new GenericAnalyzer()); @@ -161,6 +164,11 @@ export interface IndexState { indexer?: CodebaseIndexer; } +// Read version from package.json so it never drifts +const PKG_VERSION: string = JSON.parse( + await fs.readFile(new URL('../package.json', import.meta.url), 'utf-8') +).version; + const indexState: IndexState = { status: 'idle' }; @@ -168,7 +176,7 @@ const indexState: IndexState = { const server: Server = new Server( { name: 'codebase-context', - version: '1.4.0' + version: PKG_VERSION }, { capabilities: { @@ -184,6 +192,8 @@ const TOOLS: Tool[] = [ description: 'Search the indexed codebase using natural language queries. Returns code summaries with file locations. ' + 'Supports framework-specific queries and architectural layer filtering. ' + + 'When intent is "edit", "refactor", or "migrate", returns a preflight card with risk level, ' + + 'patterns to use/avoid, impact candidates, related memories, and an evidence lock score — all in one call. ' + 'Use the returned filePath with other tools to read complete file contents.', inputSchema: { type: 'object', @@ -192,6 +202,14 @@ const TOOLS: Tool[] = [ type: 'string', description: 'Natural language search query' }, + intent: { + type: 'string', + enum: ['explore', 'edit', 'refactor', 'migrate'], + description: + 'Search intent. Use "explore" (default) for read-only browsing. ' + + 'Use "edit", "refactor", or "migrate" to get a preflight card with risk assessment, ' + + 'patterns to prefer/avoid, affected files, relevant team memories, and ready-to-edit evidence checks.' + }, limit: { type: 'number', description: 'Maximum number of results to return (default: 5)', @@ -359,8 +377,10 @@ const TOOLS: Tool[] = [ properties: { type: { type: 'string', - enum: ['convention', 'decision', 'gotcha'], - description: 'Type of memory being recorded' + enum: ['convention', 'decision', 'gotcha', 'failure'], + description: + 'Type of memory being recorded. Use "failure" for things that were tried and failed — ' + + 'prevents repeating the same mistakes.' }, category: { type: 'string', @@ -396,7 +416,7 @@ const TOOLS: Tool[] = [ type: { type: 'string', description: 'Filter by memory type', - enum: ['convention', 'decision', 'gotcha'] + enum: ['convention', 'decision', 'gotcha', 'failure'] }, query: { type: 'string', @@ -563,14 +583,55 @@ server.setRequestHandler(ReadResourceRequestSchema, async (request) => { throw new Error(`Unknown resource: ${uri}`); }); -async function performIndexing(): Promise { +/** + * Extract memories from conventional git commits (refactor:, migrate:, fix:, revert:). + * Scans last 90 days. Deduplicates via content hash. Zero friction alternative to manual memory. + */ +async function extractGitMemories(): Promise { + // Quick check: skip if not a git repo + if (!(await fileExists(path.join(ROOT_PATH, '.git')))) return 0; + + const { execSync } = await import('child_process'); + + let log: string; + try { + // Format: ISO-datehash subject (e.g. "2026-01-15T10:00:00+00:00\tabc1234 fix: race condition") + log = execSync('git log --format="%aI\t%h %s" --since="90 days ago" --no-merges', { + cwd: ROOT_PATH, + encoding: 'utf-8', + timeout: 5000 + }).trim(); + } catch { + // Git not available or command failed — silently skip + return 0; + } + + if (!log) return 0; + + const lines = log.split('\n').filter(Boolean); + let added = 0; + + for (const line of lines) { + const parsedMemory = parseGitLogLineToMemory(line); + if (!parsedMemory) continue; + + const result = await appendMemoryFile(PATHS.memory, parsedMemory); + if (result.status === 'added') added++; + } + + return added; +} + +async function performIndexing(incrementalOnly?: boolean): Promise { indexState.status = 'indexing'; - console.error(`Indexing: ${ROOT_PATH}`); + const mode = incrementalOnly ? 'incremental' : 'full'; + console.error(`Indexing (${mode}): ${ROOT_PATH}`); try { let lastLoggedProgress = { phase: '', percentage: -1 }; const indexer = new CodebaseIndexer({ rootPath: ROOT_PATH, + incrementalOnly, onProgress: (progress) => { // Only log when phase or percentage actually changes (prevents duplicate logs) const shouldLog = @@ -596,6 +657,18 @@ async function performIndexing(): Promise { stats.duration / 1000 ).toFixed(2)}s` ); + + // Auto-extract memories from git history (non-blocking, best-effort) + try { + const gitMemories = await extractGitMemories(); + if (gitMemories > 0) { + console.error( + `[git-memory] Extracted ${gitMemories} new memor${gitMemories === 1 ? 'y' : 'ies'} from git history` + ); + } + } catch { + // Git memory extraction is optional — never fail indexing over it + } } catch (error) { indexState.status = 'error'; indexState.error = error instanceof Error ? error.message : String(error); @@ -619,7 +692,7 @@ server.setRequestHandler(CallToolRequestSchema, async (request) => { try { switch (name) { case 'search_codebase': { - const { query, limit, filters } = args as any; + const { query, limit, filters, intent } = args as any; if (indexState.status === 'indexing') { return { @@ -716,18 +789,205 @@ server.setRequestHandler(CallToolRequestSchema, async (request) => { } } - // Load memories for keyword matching + // Load memories for keyword matching, enriched with confidence const allMemories = await readMemoriesFile(PATHS.memory); + const allMemoriesWithConf = withConfidence(allMemories); - const findRelatedMemories = (queryTerms: string[]): Memory[] => { - return allMemories.filter((m) => { + const queryTerms = query.toLowerCase().split(/\s+/); + const relatedMemories = allMemoriesWithConf + .filter((m) => { const searchText = `${m.memory} ${m.reason}`.toLowerCase(); - return queryTerms.some((term) => searchText.includes(term)); - }); - }; + return queryTerms.some((term: string) => searchText.includes(term)); + }) + .sort((a, b) => b.effectiveConfidence - a.effectiveConfidence); + + // Compose preflight card for edit/refactor/migrate intents + let preflight: any = undefined; + const preflightIntents = ['edit', 'refactor', 'migrate']; + if (intent && preflightIntents.includes(intent)) { + try { + const intelligenceContent = await fs.readFile(PATHS.intelligence, 'utf-8'); + const intelligence = JSON.parse(intelligenceContent); + + // --- Avoid / Prefer patterns --- + const avoidPatterns: any[] = []; + const preferredPatterns: any[] = []; + const patterns = intelligence.patterns || {}; + for (const [category, data] of Object.entries(patterns)) { + // Primary pattern = preferred if Rising or Stable + if (data.primary) { + const p = data.primary; + if (p.trend === 'Rising' || p.trend === 'Stable') { + preferredPatterns.push({ + pattern: p.name, + category, + adoption: p.frequency, + trend: p.trend, + guidance: p.guidance, + ...(p.canonicalExample && { example: p.canonicalExample.file }) + }); + } + } + // Also-detected patterns that are Declining = avoid + if (data.alsoDetected) { + for (const alt of data.alsoDetected) { + if (alt.trend === 'Declining') { + avoidPatterns.push({ + pattern: alt.name, + category, + adoption: alt.frequency, + trend: 'Declining', + guidance: alt.guidance + }); + } + } + } + } - const queryTerms = query.toLowerCase().split(/\s+/); - const relatedMemories = findRelatedMemories(queryTerms); + // --- Impact candidates (files importing the result files) --- + const impactCandidates: string[] = []; + const resultPaths = results.map((r) => r.filePath); + if (intelligence.internalFileGraph?.imports) { + const allImports = intelligence.internalFileGraph.imports as Record; + for (const [file, deps] of Object.entries(allImports)) { + if ( + deps.some((dep: string) => + resultPaths.some((rp) => dep.endsWith(rp) || rp.endsWith(dep)) + ) + ) { + if (!resultPaths.some((rp) => file.endsWith(rp) || rp.endsWith(file))) { + impactCandidates.push(file); + } + } + } + } + + // --- Risk level (based on circular deps + impact breadth) --- + let riskLevel: 'low' | 'medium' | 'high' = 'low'; + let cycleCount = 0; + if (intelligence.internalFileGraph) { + try { + const graph = InternalFileGraph.fromJSON(intelligence.internalFileGraph, ROOT_PATH); + // Use directory prefixes as scope (not full file paths) + // findCycles(scope) filters files by startsWith, so a full path would only match itself + const scopes = new Set( + resultPaths.map((rp) => { + const lastSlash = rp.lastIndexOf('/'); + return lastSlash > 0 ? rp.substring(0, lastSlash + 1) : rp; + }) + ); + for (const scope of scopes) { + const cycles = graph.findCycles(scope); + cycleCount += cycles.length; + } + } catch { + // Graph reconstruction failed — skip cycle check + } + } + if (cycleCount > 0 || impactCandidates.length > 10) { + riskLevel = 'high'; + } else if (impactCandidates.length > 3) { + riskLevel = 'medium'; + } + + // --- Golden files (exemplar code) --- + const goldenFiles = (intelligence.goldenFiles || []).slice(0, 3).map((g: any) => ({ + file: g.file, + score: g.score + })); + + // --- Confidence (index freshness) --- + let confidence: 'fresh' | 'aging' | 'stale' = 'stale'; + if (intelligence.generatedAt) { + const indexAge = Date.now() - new Date(intelligence.generatedAt).getTime(); + const hoursOld = indexAge / (1000 * 60 * 60); + if (hoursOld < 24) { + confidence = 'fresh'; + } else if (hoursOld < 168) { + confidence = 'aging'; + } + } + + // --- Failure memories (1.5x relevance boost) --- + const failureWarnings = relatedMemories + .filter((m) => m.type === 'failure' && !m.stale) + .map((m) => ({ + memory: m.memory, + reason: m.reason, + confidence: m.effectiveConfidence + })) + .slice(0, 3); + + const preferredPatternsForOutput = preferredPatterns.slice(0, 5); + const avoidPatternsForOutput = avoidPatterns.slice(0, 5); + + // --- Pattern conflicts (split decisions within categories) --- + const patternConflicts: Array<{ + category: string; + primary: { name: string; adoption: string }; + alternative: { name: string; adoption: string }; + }> = []; + for (const [cat, data] of Object.entries(patterns)) { + if (!data.primary || !data.alsoDetected?.length) continue; + const primaryFreq = parseFloat(data.primary.frequency) || 100; + if (primaryFreq >= 80) continue; + for (const alt of data.alsoDetected) { + const altFreq = parseFloat(alt.frequency) || 0; + if (altFreq >= 20) { + patternConflicts.push({ + category: cat, + primary: { name: data.primary.name, adoption: data.primary.frequency }, + alternative: { name: alt.name, adoption: alt.frequency } + }); + } + } + } + + const evidenceLock = buildEvidenceLock({ + results, + preferredPatterns: preferredPatternsForOutput, + relatedMemories, + failureWarnings, + patternConflicts + }); + + // Bump risk if there are active failure memories for this area + if (failureWarnings.length > 0 && riskLevel === 'low') { + riskLevel = 'medium'; + } + + // If evidence triangulation is weak, avoid claiming low risk + if (evidenceLock.status === 'block' && riskLevel === 'low') { + riskLevel = 'medium'; + } + + // If epistemic stress says abstain, bump risk + if (evidenceLock.epistemicStress?.abstain && riskLevel === 'low') { + riskLevel = 'medium'; + } + + preflight = { + intent, + riskLevel, + confidence, + evidenceLock, + ...(preferredPatternsForOutput.length > 0 && { + preferredPatterns: preferredPatternsForOutput + }), + ...(avoidPatternsForOutput.length > 0 && { + avoidPatterns: avoidPatternsForOutput + }), + ...(goldenFiles.length > 0 && { goldenFiles }), + ...(impactCandidates.length > 0 && { + impactCandidates: impactCandidates.slice(0, 10) + }), + ...(cycleCount > 0 && { circularDependencies: cycleCount }), + ...(failureWarnings.length > 0 && { failureWarnings }) + }; + } catch { + // Intelligence file not available — skip preflight, don't fail the search + } + } return { content: [ @@ -736,6 +996,7 @@ server.setRequestHandler(CallToolRequestSchema, async (request) => { text: JSON.stringify( { status: 'success', + ...(preflight && { preflight }), results: results.map((r) => ({ summary: r.summary, snippet: r.snippet, @@ -749,7 +1010,9 @@ server.setRequestHandler(CallToolRequestSchema, async (request) => { patternWarning: r.patternWarning })), totalResults: results.length, - ...(relatedMemories.length > 0 && { relatedMemories }) + ...(relatedMemories.length > 0 && { + relatedMemories: relatedMemories.slice(0, 5) + }) }, null, 2 @@ -776,7 +1039,8 @@ server.setRequestHandler(CallToolRequestSchema, async (request) => { totalFiles: indexState.stats.totalFiles, indexedFiles: indexState.stats.indexedFiles, totalChunks: indexState.stats.totalChunks, - duration: `${(indexState.stats.duration / 1000).toFixed(2)}s` + duration: `${(indexState.stats.duration / 1000).toFixed(2)}s`, + incremental: indexState.stats.incremental } : undefined, progress: progress @@ -804,10 +1068,7 @@ server.setRequestHandler(CallToolRequestSchema, async (request) => { const mode = incrementalOnly ? 'incremental' : 'full'; console.error(`Refresh requested (${mode}): ${reason || 'Manual trigger'}`); - // TODO: When incremental indexing is implemented (Phase 2), - // use `incrementalOnly` to only re-index changed files. - // For now, always do full re-index but acknowledge the intention. - performIndexing(); + performIndexing(incrementalOnly); return { content: [ @@ -818,12 +1079,9 @@ server.setRequestHandler(CallToolRequestSchema, async (request) => { status: 'started', mode, message: incrementalOnly - ? 'Incremental re-indexing requested. Check status with get_indexing_status.' + ? 'Incremental re-indexing started. Only changed files will be re-embedded.' : 'Full re-indexing started. Check status with get_indexing_status.', - reason, - note: incrementalOnly - ? 'Incremental mode requested. Full re-index for now; true incremental indexing coming in Phase 2.' - : undefined + reason }, null, 2 @@ -1045,6 +1303,40 @@ server.setRequestHandler(CallToolRequestSchema, async (request) => { // No memory file yet, that's fine - don't fail the whole request } + // Detect pattern conflicts: primary < 80% and any alternative > 20% + const conflicts: any[] = []; + const patternsData = intelligence.patterns || {}; + for (const [cat, data] of Object.entries(patternsData)) { + if (category && category !== 'all' && cat !== category) continue; + if (!data.primary || !data.alsoDetected?.length) continue; + + const primaryFreq = parseFloat(data.primary.frequency) || 100; + if (primaryFreq >= 80) continue; + + for (const alt of data.alsoDetected) { + const altFreq = parseFloat(alt.frequency) || 0; + if (altFreq < 20) continue; + + conflicts.push({ + category: cat, + primary: { + name: data.primary.name, + adoption: data.primary.frequency, + trend: data.primary.trend + }, + alternative: { + name: alt.name, + adoption: alt.frequency, + trend: alt.trend + }, + note: `Split decision: ${data.primary.frequency} ${data.primary.name} (${data.primary.trend || 'unknown'}) vs ${alt.frequency} ${alt.name} (${alt.trend || 'unknown'})` + }); + } + } + if (conflicts.length > 0) { + result.conflicts = conflicts; + } + return { content: [{ type: 'text', text: JSON.stringify(result, null, 2) }] }; @@ -1369,6 +1661,10 @@ server.setRequestHandler(CallToolRequestSchema, async (request) => { const filtered = filterMemories(allMemories, { category, type, query }); const limited = applyUnfilteredLimit(filtered, { category, type, query }, 20); + // Enrich with confidence decay + const enriched = withConfidence(limited.memories); + const staleCount = enriched.filter((m) => m.stale).length; + return { content: [ { @@ -1376,13 +1672,17 @@ server.setRequestHandler(CallToolRequestSchema, async (request) => { text: JSON.stringify( { status: 'success', - count: limited.memories.length, + count: enriched.length, totalCount: limited.totalCount, truncated: limited.truncated, + ...(staleCount > 0 && { + staleCount, + staleNote: `${staleCount} memor${staleCount === 1 ? 'y' : 'ies'} below 30% confidence. Consider reviewing or removing.` + }), message: limited.truncated ? 'Showing 20 most recent. Use filters (category/type/query) for targeted results.' : undefined, - memories: limited.memories + memories: enriched }, null, 2 diff --git a/src/memory/git-memory.ts b/src/memory/git-memory.ts new file mode 100644 index 0000000..27b5ad5 --- /dev/null +++ b/src/memory/git-memory.ts @@ -0,0 +1,55 @@ +import { createHash } from 'crypto'; +import { GIT_COMMIT_PATTERNS } from '../constants/git-patterns.js'; +import type { Memory } from '../types/index.js'; + +const MIN_COMMIT_MESSAGE_LENGTH = 10; + +/** + * Parse one git log line in format: + * \t + * Returns a normalized Memory when the subject matches supported commit patterns. + */ +export function parseGitLogLineToMemory(line: string): Memory | null { + const tabIdx = line.indexOf('\t'); + if (tabIdx === -1) return null; + + const commitDateRaw = line.substring(0, tabIdx).trim(); + const hashAndSubject = line.substring(tabIdx + 1).trim(); + if (!commitDateRaw || !hashAndSubject) return null; + + const commitMs = Date.parse(commitDateRaw); + if (!Number.isFinite(commitMs)) return null; + + for (const pattern of GIT_COMMIT_PATTERNS) { + if (!pattern.prefix.test(hashAndSubject)) continue; + + const message = hashAndSubject.replace(/^[a-f0-9]+ /i, '').trim(); + if (message.length < MIN_COMMIT_MESSAGE_LENGTH) return null; + + const hashContent = `git:${pattern.type}:${pattern.category}:${message}`; + const id = createHash('sha256').update(hashContent).digest('hex').substring(0, 12); + + return { + id, + type: pattern.type, + category: pattern.category, + memory: message, + reason: 'Auto-extracted from git commit history', + date: new Date(commitMs).toISOString(), + source: 'git' + }; + } + + return null; +} + +export function parseGitLogToMemories(log: string): Memory[] { + if (!log.trim()) return []; + + const memories: Memory[] = []; + for (const line of log.split('\n')) { + const parsed = parseGitLogLineToMemory(line); + if (parsed) memories.push(parsed); + } + return memories; +} diff --git a/src/memory/store.ts b/src/memory/store.ts index 358fbaf..16d222b 100644 --- a/src/memory/store.ts +++ b/src/memory/store.ts @@ -10,6 +10,7 @@ type RawMemory = Partial<{ decision: unknown; reason: unknown; date: unknown; + source: unknown; }>; export type MemoryFilters = { @@ -40,7 +41,8 @@ export function normalizeMemory(raw: unknown): Memory | null { if (!id || !category || !memory || !reason || !date) return null; - return { id, type, category, memory, reason, date }; + const source = m.source === 'git' ? ('git' as const) : undefined; + return { id, type, category, memory, reason, date, ...(source && { source }) }; } export function normalizeMemories(raw: unknown): Memory[] { @@ -112,6 +114,55 @@ export function sortMemoriesByRecency(memories: Memory[]): Memory[] { return withIndex.map((x) => x.m); } +/** + * Half-life in days per memory type. + * Convention memories never decay (Infinity). + * Decisions may be revisited. Gotchas and failures get fixed. + */ +const HALF_LIFE_DAYS: Record = { + convention: Infinity, + decision: 180, + gotcha: 90, + failure: 90 +}; + +export interface MemoryWithConfidence extends Memory { + effectiveConfidence: number; + stale: boolean; +} + +/** + * Compute confidence decay: confidence = 2^(-age_days / half_life) + * Conventions never decay. Memories below 0.3 are flagged stale. + */ +export function computeConfidence( + memory: Memory, + now?: Date +): { effectiveConfidence: number; stale: boolean } { + const halfLife = HALF_LIFE_DAYS[memory.type] ?? 180; + if (!Number.isFinite(halfLife)) { + return { effectiveConfidence: 1.0, stale: false }; + } + const memDate = Date.parse(memory.date); + if (!Number.isFinite(memDate)) { + return { effectiveConfidence: 0, stale: true }; + } + const ageDays = ((now ?? new Date()).getTime() - memDate) / (1000 * 60 * 60 * 24); + const confidence = Math.pow(2, -ageDays / halfLife); + const rounded = Math.round(confidence * 100) / 100; + return { effectiveConfidence: rounded, stale: rounded < 0.3 }; +} + +/** + * Enrich an array of memories with confidence decay metadata. + */ +export function withConfidence(memories: Memory[], now?: Date): MemoryWithConfidence[] { + return memories.map((m) => ({ + ...m, + ...computeConfidence(m, now) + })); +} + export function applyUnfilteredLimit( memories: Memory[], filters: MemoryFilters, diff --git a/src/preflight/evidence-lock.ts b/src/preflight/evidence-lock.ts new file mode 100644 index 0000000..630d3d6 --- /dev/null +++ b/src/preflight/evidence-lock.ts @@ -0,0 +1,190 @@ +import type { MemoryWithConfidence } from '../memory/store.js'; +import type { SearchResult } from '../types/index.js'; + +type EvidenceStrength = 'strong' | 'weak' | 'missing'; + +interface EvidenceSource { + source: 'code' | 'patterns' | 'memories'; + strength: EvidenceStrength; + count: number; + examples: string[]; +} + +export interface EpistemicStress { + level: 'low' | 'moderate' | 'high'; + triggers: string[]; + abstain: boolean; +} + +export interface EvidenceLock { + mode: 'triangulated'; + status: 'pass' | 'warn' | 'block'; + readyToEdit: boolean; + score: number; + sources: EvidenceSource[]; + gaps?: string[]; + nextAction?: string; + epistemicStress?: EpistemicStress; +} + +interface PatternConflict { + category: string; + primary: { name: string; adoption: string }; + alternative: { name: string; adoption: string }; +} + +interface BuildEvidenceLockInput { + results: SearchResult[]; + preferredPatterns: Array<{ pattern: string; example?: string }>; + relatedMemories: MemoryWithConfidence[]; + failureWarnings: Array<{ memory: string }>; + patternConflicts?: PatternConflict[]; +} + +function strengthFactor(strength: EvidenceStrength): number { + if (strength === 'strong') return 1; + if (strength === 'weak') return 0.5; + return 0; +} + +function truncate(text: string, max = 80): string { + if (text.length <= max) return text; + return `${text.slice(0, max - 1)}...`; +} + +export function buildEvidenceLock(input: BuildEvidenceLockInput): EvidenceLock { + const codeExamples = input.results + .slice(0, 3) + .map((r) => `${r.filePath}:${r.startLine}-${r.endLine}`); + const codeStrength: EvidenceStrength = + input.results.length >= 3 ? 'strong' : input.results.length > 0 ? 'weak' : 'missing'; + + const patternExamples = input.preferredPatterns + .slice(0, 3) + .map((p) => (p.example ? `${p.pattern} (${p.example})` : p.pattern)); + const patternsStrength: EvidenceStrength = + input.preferredPatterns.length >= 2 + ? 'strong' + : input.preferredPatterns.length === 1 + ? 'weak' + : 'missing'; + + const activeMemories = input.relatedMemories.filter((m) => !m.stale); + const memoryExamplesFromFailures = input.failureWarnings + .slice(0, 2) + .map((w) => truncate(w.memory)); + const memoryExamplesFromMemories = activeMemories.slice(0, 2).map((m) => truncate(m.memory)); + const memoryExamples = [...memoryExamplesFromFailures, ...memoryExamplesFromMemories].slice(0, 3); + const memoryCount = activeMemories.length; + const memoriesStrength: EvidenceStrength = + memoryCount >= 2 || input.failureWarnings.length > 0 + ? 'strong' + : memoryCount === 1 + ? 'weak' + : 'missing'; + + const sources: EvidenceSource[] = [ + { + source: 'code', + strength: codeStrength, + count: input.results.length, + examples: codeExamples + }, + { + source: 'patterns', + strength: patternsStrength, + count: input.preferredPatterns.length, + examples: patternExamples + }, + { + source: 'memories', + strength: memoriesStrength, + count: memoryCount, + examples: memoryExamples + } + ]; + + const strongSources = sources.filter((s) => s.strength === 'strong').length; + const weakSources = sources.filter((s) => s.strength === 'weak').length; + + const baseScore = + 45 * strengthFactor(codeStrength) + + 30 * strengthFactor(patternsStrength) + + 25 * strengthFactor(memoriesStrength); + const score = Math.min(100, Math.round(baseScore)); + + let status: 'pass' | 'warn' | 'block' = 'block'; + if (codeStrength === 'strong' && strongSources >= 2) { + status = 'pass'; + } else if (codeStrength !== 'missing' && (strongSources >= 1 || weakSources >= 2)) { + status = 'warn'; + } + + const gaps: string[] = []; + if (codeStrength === 'missing') gaps.push('No matching code hits for this intent'); + if (patternsStrength === 'missing') gaps.push('No preferred team pattern evidence found'); + if (memoriesStrength === 'missing') gaps.push('No active team memory evidence found'); + + let nextAction: string | undefined; + if (status === 'block') { + nextAction = + 'Broaden the query or run refresh_index, then retry with intent="edit" to collect stronger evidence.'; + } else if (status === 'warn') { + nextAction = 'Proceed cautiously and confirm at least one golden file before editing.'; + } + + // --- Epistemic stress: detect when evidence is contradictory, stale, or too thin --- + const stressTriggers: string[] = []; + + // Trigger: pattern conflicts (team hasn't converged) + if (input.patternConflicts && input.patternConflicts.length > 0) { + for (const c of input.patternConflicts.slice(0, 3)) { + stressTriggers.push( + `Conflicting patterns in ${c.category}: ${c.primary.name} (${c.primary.adoption}) vs ${c.alternative.name} (${c.alternative.adoption})` + ); + } + } + + // Trigger: high stale memory ratio (most knowledge is outdated) + const totalMemories = input.relatedMemories.length; + const staleMemories = input.relatedMemories.filter((m) => m.stale).length; + if (totalMemories > 0 && staleMemories / totalMemories > 0.5) { + stressTriggers.push( + `${staleMemories}/${totalMemories} related memories are stale — team knowledge may be outdated` + ); + } + + // Trigger: thin evidence (majority of sources missing or weak) + const missingSources = sources.filter((s) => s.strength === 'missing').length; + if (missingSources >= 2) { + stressTriggers.push('Insufficient evidence: most evidence sources are empty'); + } + + let epistemicStress: EpistemicStress | undefined; + if (stressTriggers.length > 0) { + const level: EpistemicStress['level'] = + stressTriggers.length >= 3 ? 'high' : stressTriggers.length >= 2 ? 'moderate' : 'low'; + const abstain = level === 'high' || (level === 'moderate' && status !== 'pass'); + epistemicStress = { level, triggers: stressTriggers, abstain }; + + // High stress overrides status: don't claim readiness when evidence is contradictory + if (abstain && status === 'pass') { + status = 'warn'; + } + if (abstain && !nextAction) { + nextAction = + 'Evidence is contradictory or insufficient. Resolve pattern conflicts or gather more context before editing.'; + } + } + + return { + mode: 'triangulated', + status, + readyToEdit: status === 'pass' && (!epistemicStress || !epistemicStress.abstain), + score, + sources, + ...(gaps.length > 0 && { gaps }), + ...(nextAction && { nextAction }), + ...(epistemicStress && { epistemicStress }) + }; +} diff --git a/src/storage/lancedb.ts b/src/storage/lancedb.ts index 1e3e3ce..1011fff 100644 --- a/src/storage/lancedb.ts +++ b/src/storage/lancedb.ts @@ -172,6 +172,30 @@ export class LanceDBStorageProvider implements VectorStorageProvider { } } + async deleteByFilePaths(filePaths: string[]): Promise { + if (!this.initialized || !this.table || filePaths.length === 0) { + return 0; + } + + try { + const countBefore = await this.table.countRows(); + + // LanceDB supports SQL-style filter for delete + // Escape single quotes in file paths to prevent SQL injection + const escaped = filePaths.map((p) => p.replace(/'/g, "''")); + const inClause = escaped.map((p) => `'${p}'`).join(', '); + await this.table.delete(`filePath IN (${inClause})`); + + const countAfter = await this.table.countRows(); + const deleted = countBefore - countAfter; + console.error(`Deleted ${deleted} chunks for ${filePaths.length} files from LanceDB`); + return deleted; + } catch (error) { + console.error('Failed to delete chunks by file paths:', error); + throw error; + } + } + async clear(): Promise { if (!this.initialized) return; diff --git a/src/storage/types.ts b/src/storage/types.ts index 9e6340c..7498a2b 100644 --- a/src/storage/types.ts +++ b/src/storage/types.ts @@ -27,6 +27,12 @@ export interface VectorStorageProvider { filters?: SearchFilters ): Promise; + /** + * Delete chunks matching the given file paths. + * Returns the number of deleted rows. + */ + deleteByFilePaths(filePaths: string[]): Promise; + /** * Clear all stored data */ diff --git a/src/types/index.ts b/src/types/index.ts index 10715f4..8aad2a1 100644 --- a/src/types/index.ts +++ b/src/types/index.ts @@ -413,6 +413,12 @@ export interface IndexingStats { errors: IndexingError[]; startedAt: Date; completedAt?: Date; + incremental?: { + added: number; + changed: number; + deleted: number; + unchanged: number; + }; } // ============================================================================ @@ -542,7 +548,8 @@ export type MemoryCategory = export type MemoryType = | 'convention' // Style, naming, component preferences | 'decision' // Architecture/tooling choices with rationale - | 'gotcha'; // Things that break and why + | 'gotcha' // Things that break and why + | 'failure'; // Tried X, failed because Y — prevents repeating mistakes /** * A recorded architectural or design decision @@ -561,4 +568,6 @@ export interface Memory { reason: string; /** ISO 8601 date when decision was recorded */ date: string; + /** Source of the memory: 'user' (default) or 'git' (auto-extracted from commits) */ + source?: 'user' | 'git'; } diff --git a/tests/confidence-decay.test.ts b/tests/confidence-decay.test.ts new file mode 100644 index 0000000..3eb3650 --- /dev/null +++ b/tests/confidence-decay.test.ts @@ -0,0 +1,94 @@ +import { describe, it, expect } from 'vitest'; +import type { Memory } from '../src/types/index.js'; +import { computeConfidence, withConfidence } from '../src/memory/store.js'; + +function makeMemory(overrides: Partial & { type: Memory['type']; date: string }): Memory { + return { + id: 'test123', + category: 'conventions', + memory: 'test memory', + reason: 'test reason', + ...overrides + }; +} + +describe('Memory confidence decay', () => { + const now = new Date('2026-02-07T00:00:00.000Z'); + + it('convention type never decays', () => { + const old = makeMemory({ type: 'convention', date: '2020-01-01T00:00:00.000Z' }); + const result = computeConfidence(old, now); + expect(result.effectiveConfidence).toBe(1.0); + expect(result.stale).toBe(false); + }); + + it('decision type has 180-day half-life', () => { + // Exactly 180 days old = 50% confidence + const d = new Date(now); + d.setDate(d.getDate() - 180); + const memory = makeMemory({ type: 'decision', date: d.toISOString() }); + const result = computeConfidence(memory, now); + expect(result.effectiveConfidence).toBe(0.5); + expect(result.stale).toBe(false); + }); + + it('gotcha type has 90-day half-life', () => { + // Exactly 90 days old = 50% confidence + const d = new Date(now); + d.setDate(d.getDate() - 90); + const memory = makeMemory({ type: 'gotcha', date: d.toISOString() }); + const result = computeConfidence(memory, now); + expect(result.effectiveConfidence).toBe(0.5); + expect(result.stale).toBe(false); + }); + + it('failure type has 90-day half-life', () => { + const d = new Date(now); + d.setDate(d.getDate() - 90); + const memory = makeMemory({ type: 'failure', date: d.toISOString() }); + const result = computeConfidence(memory, now); + expect(result.effectiveConfidence).toBe(0.5); + expect(result.stale).toBe(false); + }); + + it('flags memory as stale below 0.3 confidence', () => { + // ~170 days for gotcha (90-day half-life): 2^(-170/90) ≈ 0.26 + const d = new Date(now); + d.setDate(d.getDate() - 170); + const memory = makeMemory({ type: 'gotcha', date: d.toISOString() }); + const result = computeConfidence(memory, now); + expect(result.effectiveConfidence).toBeLessThan(0.3); + expect(result.stale).toBe(true); + }); + + it('brand new memory has full confidence', () => { + const memory = makeMemory({ type: 'decision', date: now.toISOString() }); + const result = computeConfidence(memory, now); + expect(result.effectiveConfidence).toBe(1.0); + expect(result.stale).toBe(false); + }); + + it('handles invalid date gracefully', () => { + const memory = makeMemory({ type: 'decision', date: 'not-a-date' }); + const result = computeConfidence(memory, now); + expect(result.effectiveConfidence).toBe(0); + expect(result.stale).toBe(true); + }); + + it('withConfidence enriches array of memories', () => { + const memories: Memory[] = [ + makeMemory({ id: '1', type: 'convention', date: '2020-01-01T00:00:00.000Z' }), + makeMemory({ id: '2', type: 'gotcha', date: now.toISOString() }) + ]; + + const enriched = withConfidence(memories, now); + expect(enriched).toHaveLength(2); + expect(enriched[0].effectiveConfidence).toBe(1.0); // convention never decays + expect(enriched[0].stale).toBe(false); + expect(enriched[1].effectiveConfidence).toBe(1.0); // brand new + expect(enriched[1].stale).toBe(false); + // Verify original fields preserved + expect(enriched[0].id).toBe('1'); + expect(enriched[1].type).toBe('gotcha'); + }); +}); diff --git a/tests/evidence-lock.test.ts b/tests/evidence-lock.test.ts new file mode 100644 index 0000000..979993c --- /dev/null +++ b/tests/evidence-lock.test.ts @@ -0,0 +1,200 @@ +import { describe, it, expect } from 'vitest'; +import type { SearchResult } from '../src/types/index.js'; +import type { MemoryWithConfidence } from '../src/memory/store.js'; +import { buildEvidenceLock } from '../src/preflight/evidence-lock.js'; + +function makeResult(filePath: string): SearchResult { + return { + summary: 'test summary', + snippet: 'test snippet', + filePath, + startLine: 10, + endLine: 20, + score: 0.9, + language: 'ts', + metadata: {} + }; +} + +function makeMemory(id: string, overrides?: Partial): MemoryWithConfidence { + return { + id, + type: 'decision', + category: 'architecture', + memory: `memory ${id}`, + reason: 'why', + date: '2026-02-01T00:00:00.000Z', + effectiveConfidence: 0.9, + stale: false, + ...overrides + }; +} + +describe('Evidence lock preflight scoring', () => { + it('passes when evidence is triangulated across code, patterns, and memories', () => { + const lock = buildEvidenceLock({ + results: [makeResult('src/a.ts'), makeResult('src/b.ts'), makeResult('src/c.ts')], + preferredPatterns: [ + { pattern: 'Use service wrapper', example: 'src/services/api.ts' }, + { pattern: 'Inject via constructor' } + ], + relatedMemories: [makeMemory('1'), makeMemory('2')], + failureWarnings: [] + }); + + expect(lock.status).toBe('pass'); + expect(lock.readyToEdit).toBe(true); + expect(lock.score).toBeGreaterThanOrEqual(80); + expect(lock.epistemicStress).toBeUndefined(); + }); + + it('warns when evidence is partial but not empty', () => { + const lock = buildEvidenceLock({ + results: [makeResult('src/a.ts')], + preferredPatterns: [{ pattern: 'Use service wrapper' }], + relatedMemories: [makeMemory('1', { stale: true })], + failureWarnings: [] + }); + + expect(lock.status).toBe('warn'); + expect(lock.readyToEdit).toBe(false); + expect(lock.nextAction).toContain('golden file'); + }); + + it('blocks when there are no code hits for the requested intent', () => { + const lock = buildEvidenceLock({ + results: [], + preferredPatterns: [{ pattern: 'Use service wrapper' }], + relatedMemories: [makeMemory('1')], + failureWarnings: [{ memory: 'Previous direct DB migration broke rollback path' }] + }); + + expect(lock.status).toBe('block'); + expect(lock.readyToEdit).toBe(false); + expect(lock.gaps).toContain('No matching code hits for this intent'); + expect(lock.nextAction).toContain('refresh_index'); + }); +}); + +describe('Epistemic stress detection', () => { + it('flags stress from pattern conflicts', () => { + const lock = buildEvidenceLock({ + results: [makeResult('src/a.ts'), makeResult('src/b.ts'), makeResult('src/c.ts')], + preferredPatterns: [ + { pattern: 'inject()' }, + { pattern: 'signals' } + ], + relatedMemories: [makeMemory('1'), makeMemory('2')], + failureWarnings: [], + patternConflicts: [ + { + category: 'dependency-injection', + primary: { name: 'inject()', adoption: '65%' }, + alternative: { name: 'constructor injection', adoption: '35%' } + } + ] + }); + + expect(lock.epistemicStress).toBeDefined(); + expect(lock.epistemicStress!.triggers).toHaveLength(1); + expect(lock.epistemicStress!.triggers[0]).toContain('Conflicting patterns'); + expect(lock.epistemicStress!.triggers[0]).toContain('dependency-injection'); + }); + + it('flags stress when majority of memories are stale', () => { + const lock = buildEvidenceLock({ + results: [makeResult('src/a.ts'), makeResult('src/b.ts'), makeResult('src/c.ts')], + preferredPatterns: [{ pattern: 'inject()' }, { pattern: 'signals' }], + relatedMemories: [ + makeMemory('1', { stale: true }), + makeMemory('2', { stale: true }), + makeMemory('3', { stale: false }) + ], + failureWarnings: [] + }); + + expect(lock.epistemicStress).toBeDefined(); + expect(lock.epistemicStress!.triggers.some((t) => t.includes('stale'))).toBe(true); + }); + + it('flags stress when most evidence sources are empty', () => { + const lock = buildEvidenceLock({ + results: [makeResult('src/a.ts')], + preferredPatterns: [], + relatedMemories: [], + failureWarnings: [] + }); + + expect(lock.epistemicStress).toBeDefined(); + expect(lock.epistemicStress!.triggers.some((t) => t.includes('Insufficient evidence'))).toBe(true); + }); + + it('abstains and downgrades readyToEdit when stress is high', () => { + const lock = buildEvidenceLock({ + results: [makeResult('src/a.ts'), makeResult('src/b.ts'), makeResult('src/c.ts')], + preferredPatterns: [{ pattern: 'inject()' }, { pattern: 'signals' }], + relatedMemories: [ + makeMemory('1', { stale: true }), + makeMemory('2', { stale: true }), + makeMemory('3', { stale: true }), + makeMemory('4', { stale: false }) + ], + failureWarnings: [], + patternConflicts: [ + { + category: 'di', + primary: { name: 'inject()', adoption: '55%' }, + alternative: { name: 'constructor', adoption: '45%' } + }, + { + category: 'state', + primary: { name: 'signals', adoption: '60%' }, + alternative: { name: 'rxjs', adoption: '40%' } + } + ] + }); + + expect(lock.epistemicStress).toBeDefined(); + expect(lock.epistemicStress!.level).toBe('high'); + expect(lock.epistemicStress!.abstain).toBe(true); + expect(lock.readyToEdit).toBe(false); + }); + + it('does not flag stress when evidence is clean and consistent', () => { + const lock = buildEvidenceLock({ + results: [makeResult('src/a.ts'), makeResult('src/b.ts'), makeResult('src/c.ts')], + preferredPatterns: [ + { pattern: 'inject()' }, + { pattern: 'signals' } + ], + relatedMemories: [makeMemory('1'), makeMemory('2')], + failureWarnings: [], + patternConflicts: [] + }); + + expect(lock.epistemicStress).toBeUndefined(); + expect(lock.status).toBe('pass'); + expect(lock.readyToEdit).toBe(true); + }); + + it('moderate stress with single conflict does not abstain when status is pass', () => { + const lock = buildEvidenceLock({ + results: [makeResult('src/a.ts'), makeResult('src/b.ts'), makeResult('src/c.ts')], + preferredPatterns: [{ pattern: 'inject()' }, { pattern: 'signals' }], + relatedMemories: [makeMemory('1'), makeMemory('2')], + failureWarnings: [], + patternConflicts: [ + { + category: 'di', + primary: { name: 'inject()', adoption: '70%' }, + alternative: { name: 'constructor', adoption: '30%' } + } + ] + }); + + expect(lock.epistemicStress).toBeDefined(); + expect(lock.epistemicStress!.level).toBe('low'); + expect(lock.epistemicStress!.abstain).toBe(false); + expect(lock.readyToEdit).toBe(true); + }); +}); diff --git a/tests/failure-memory.test.ts b/tests/failure-memory.test.ts new file mode 100644 index 0000000..27e264a --- /dev/null +++ b/tests/failure-memory.test.ts @@ -0,0 +1,85 @@ +import { describe, it, expect } from 'vitest'; +import { normalizeMemory, normalizeMemories, filterMemories } from '../src/memory/store.js'; + +describe('Failure memory type', () => { + it('normalizes a failure memory', () => { + const raw = { + id: 'fail001', + type: 'failure', + category: 'architecture', + memory: 'Tried direct PrimeNG usage, broke wrapper abstraction', + reason: 'Team uses @company/ui-toolkit wrapper', + date: '2026-02-01T00:00:00.000Z', + source: 'user' + }; + + const result = normalizeMemory(raw); + expect(result).not.toBeNull(); + expect(result!.type).toBe('failure'); + expect(result!.memory).toContain('PrimeNG'); + }); + + it('failure type is filterable', () => { + const memories = normalizeMemories([ + { + id: '1', + type: 'convention', + category: 'conventions', + memory: 'Use inject()', + reason: 'Team standard', + date: '2026-01-01T00:00:00.000Z' + }, + { + id: '2', + type: 'failure', + category: 'architecture', + memory: 'Direct HTTP calls failed', + reason: 'Must use ApiService wrapper', + date: '2026-01-15T00:00:00.000Z' + }, + { + id: '3', + type: 'gotcha', + category: 'testing', + memory: 'Jest timer mocks break signals', + reason: 'Use fakeAsync instead', + date: '2026-01-20T00:00:00.000Z' + } + ]); + + const failures = filterMemories(memories, { type: 'failure' }); + expect(failures).toHaveLength(1); + expect(failures[0].id).toBe('2'); + }); + + it('normalizes git-sourced memory with source field', () => { + const raw = { + id: 'git001', + type: 'decision', + category: 'architecture', + memory: 'refactor: migrate auth to standalone components', + reason: 'Auto-extracted from git commit history', + date: '2026-02-05T00:00:00.000Z', + source: 'git' + }; + + const result = normalizeMemory(raw); + expect(result).not.toBeNull(); + expect(result!.source).toBe('git'); + }); + + it('omits source field when not git', () => { + const raw = { + id: 'user001', + type: 'convention', + category: 'conventions', + memory: 'Use CSS variables for theming', + reason: 'Design system consistency', + date: '2026-02-05T00:00:00.000Z' + }; + + const result = normalizeMemory(raw); + expect(result).not.toBeNull(); + expect(result!.source).toBeUndefined(); + }); +}); diff --git a/tests/git-memory-patterns.test.ts b/tests/git-memory-patterns.test.ts new file mode 100644 index 0000000..8a62b1b --- /dev/null +++ b/tests/git-memory-patterns.test.ts @@ -0,0 +1,66 @@ +import { describe, it, expect } from 'vitest'; +import { parseGitLogLineToMemory, parseGitLogToMemories } from '../src/memory/git-memory.js'; + +describe('Git memory extraction pipeline', () => { + it('extracts memory from a valid refactor commit with commit timestamp', () => { + const line = '2026-02-01T10:20:30+00:00\tabc1234 refactor(auth): extract login state machine'; + + const parsed = parseGitLogLineToMemory(line); + + expect(parsed).not.toBeNull(); + expect(parsed).toMatchObject({ + type: 'decision', + category: 'architecture', + memory: 'refactor(auth): extract login state machine', + reason: 'Auto-extracted from git commit history', + date: '2026-02-01T10:20:30.000Z', + source: 'git' + }); + expect(parsed!.id).toHaveLength(12); + }); + + it('extracts memory from fix and migrate commit types', () => { + const fix = parseGitLogLineToMemory( + '2026-02-02T10:20:30+00:00\tdef5678 fix(api): add null guard for profile cache' + ); + const migrate = parseGitLogLineToMemory( + '2026-02-03T10:20:30+00:00\t0123abc migrate: move websocket client to shared transport' + ); + + expect(fix).toMatchObject({ type: 'gotcha', category: 'conventions' }); + expect(migrate).toMatchObject({ type: 'decision', category: 'dependencies' }); + }); + + it('rejects unsupported commit prefixes', () => { + const parsed = parseGitLogLineToMemory( + '2026-02-04T10:20:30+00:00\tabc1234 feat: add new dashboard widget' + ); + expect(parsed).toBeNull(); + }); + + it('rejects invalid commit dates instead of using current time', () => { + const parsed = parseGitLogLineToMemory( + 'not-a-date\tabc1234 refactor: simplify settings loader' + ); + expect(parsed).toBeNull(); + }); + + it('rejects trivially short commit messages', () => { + const parsed = parseGitLogLineToMemory('2026-02-05T10:20:30+00:00\tabc1234 fix: x'); + expect(parsed).toBeNull(); + }); + + it('extracts multiple memories from git log text', () => { + const log = [ + '2026-02-05T10:20:30+00:00\tabc1234 refactor: split auth adapter', + '2026-02-05T10:21:30+00:00\tdef5678 docs: update readme', + '2026-02-05T10:22:30+00:00\t9876abc fix(cache): guard stale token path' + ].join('\n'); + + const parsed = parseGitLogToMemories(log); + + expect(parsed).toHaveLength(2); + expect(parsed[0].type).toBe('decision'); + expect(parsed[1].type).toBe('gotcha'); + }); +}); diff --git a/tests/incremental-indexing.test.ts b/tests/incremental-indexing.test.ts new file mode 100644 index 0000000..85cc00b --- /dev/null +++ b/tests/incremental-indexing.test.ts @@ -0,0 +1,242 @@ +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import { promises as fs } from 'fs'; +import path from 'path'; +import os from 'os'; +import { CodebaseIndexer } from '../src/core/indexer.js'; +import { readManifest } from '../src/core/manifest.js'; +import { CODEBASE_CONTEXT_DIRNAME, MANIFEST_FILENAME, KEYWORD_INDEX_FILENAME } from '../src/constants/codebase-context.js'; + +describe('Incremental Indexing', () => { + let tempDir: string; + + beforeEach(async () => { + tempDir = await fs.mkdtemp(path.join(os.tmpdir(), 'incremental-test-')); + // Create a minimal project + await fs.writeFile( + path.join(tempDir, 'package.json'), + JSON.stringify({ name: 'test-project', dependencies: {} }) + ); + }); + + afterEach(async () => { + await fs.rm(tempDir, { recursive: true, force: true }); + }); + + it('should create manifest file after full index', async () => { + await fs.writeFile(path.join(tempDir, 'index.ts'), 'export const x = 1;'); + + const indexer = new CodebaseIndexer({ + rootPath: tempDir, + config: { skipEmbedding: true } + }); + + await indexer.index(); + + const manifestPath = path.join(tempDir, CODEBASE_CONTEXT_DIRNAME, MANIFEST_FILENAME); + const manifest = await readManifest(manifestPath); + expect(manifest).not.toBeNull(); + expect(manifest!.version).toBe(1); + expect(Object.keys(manifest!.files).length).toBeGreaterThan(0); + }); + + it('should return early with no changes in incremental mode', async () => { + await fs.writeFile(path.join(tempDir, 'index.ts'), 'export const x = 1;'); + + // Full index first + const indexer1 = new CodebaseIndexer({ + rootPath: tempDir, + config: { skipEmbedding: true } + }); + await indexer1.index(); + + // Incremental index — nothing changed + const indexer2 = new CodebaseIndexer({ + rootPath: tempDir, + config: { skipEmbedding: true }, + incrementalOnly: true + }); + const stats = await indexer2.index(); + + expect(stats.incremental).toBeDefined(); + expect(stats.incremental!.added).toBe(0); + expect(stats.incremental!.changed).toBe(0); + expect(stats.incremental!.deleted).toBe(0); + expect(stats.incremental!.unchanged).toBeGreaterThan(0); + // Duration should be very fast since nothing happened + expect(stats.duration).toBeLessThan(5000); + }); + + it('should detect changed files in incremental mode', async () => { + await fs.writeFile(path.join(tempDir, 'index.ts'), 'export const x = 1;'); + + // Full index first + const indexer1 = new CodebaseIndexer({ + rootPath: tempDir, + config: { skipEmbedding: true } + }); + await indexer1.index(); + + // Modify a file + await fs.writeFile(path.join(tempDir, 'index.ts'), 'export const x = 2; // changed'); + + // Incremental index + const indexer2 = new CodebaseIndexer({ + rootPath: tempDir, + config: { skipEmbedding: true }, + incrementalOnly: true + }); + const stats = await indexer2.index(); + + expect(stats.incremental).toBeDefined(); + expect(stats.incremental!.changed).toBeGreaterThanOrEqual(1); + }); + + it('should detect new files in incremental mode', async () => { + await fs.writeFile(path.join(tempDir, 'index.ts'), 'export const x = 1;'); + + // Full index first + const indexer1 = new CodebaseIndexer({ + rootPath: tempDir, + config: { skipEmbedding: true } + }); + await indexer1.index(); + + // Add a new file + await fs.writeFile(path.join(tempDir, 'utils.ts'), 'export function add(a: number, b: number) { return a + b; }'); + + // Incremental index + const indexer2 = new CodebaseIndexer({ + rootPath: tempDir, + config: { skipEmbedding: true }, + incrementalOnly: true + }); + const stats = await indexer2.index(); + + expect(stats.incremental).toBeDefined(); + expect(stats.incremental!.added).toBeGreaterThanOrEqual(1); + }); + + it('should detect deleted files in incremental mode', async () => { + await fs.writeFile(path.join(tempDir, 'index.ts'), 'export const x = 1;'); + await fs.writeFile(path.join(tempDir, 'delete-me.ts'), 'export const y = 2;'); + + // Full index first + const indexer1 = new CodebaseIndexer({ + rootPath: tempDir, + config: { skipEmbedding: true } + }); + await indexer1.index(); + + // Delete a file + await fs.unlink(path.join(tempDir, 'delete-me.ts')); + + // Incremental index + const indexer2 = new CodebaseIndexer({ + rootPath: tempDir, + config: { skipEmbedding: true }, + incrementalOnly: true + }); + const stats = await indexer2.index(); + + expect(stats.incremental).toBeDefined(); + expect(stats.incremental!.deleted).toBeGreaterThanOrEqual(1); + }); + + it('should fall back to full-like behavior when no manifest exists', async () => { + await fs.writeFile(path.join(tempDir, 'index.ts'), 'export const x = 1;'); + + // Run incremental without prior full index (no manifest) + const indexer = new CodebaseIndexer({ + rootPath: tempDir, + config: { skipEmbedding: true }, + incrementalOnly: true + }); + const stats = await indexer.index(); + + // Should treat all files as "added" since there's no old manifest + expect(stats.incremental).toBeDefined(); + expect(stats.incremental!.added).toBeGreaterThan(0); + expect(stats.incremental!.unchanged).toBe(0); + expect(stats.incremental!.deleted).toBe(0); + + // Manifest should be created + const manifestPath = path.join(tempDir, CODEBASE_CONTEXT_DIRNAME, MANIFEST_FILENAME); + const manifest = await readManifest(manifestPath); + expect(manifest).not.toBeNull(); + }); + + it('should update manifest after incremental index', async () => { + await fs.writeFile(path.join(tempDir, 'index.ts'), 'export const x = 1;'); + + // Full index + const indexer1 = new CodebaseIndexer({ + rootPath: tempDir, + config: { skipEmbedding: true } + }); + await indexer1.index(); + + const manifestPath = path.join(tempDir, CODEBASE_CONTEXT_DIRNAME, MANIFEST_FILENAME); + const manifest1 = await readManifest(manifestPath); + + // Add a new file + await fs.writeFile(path.join(tempDir, 'new.ts'), 'export const y = 2;'); + + // Incremental index + const indexer2 = new CodebaseIndexer({ + rootPath: tempDir, + config: { skipEmbedding: true }, + incrementalOnly: true + }); + await indexer2.index(); + + const manifest2 = await readManifest(manifestPath); + expect(manifest2).not.toBeNull(); + + // New manifest should have more files + expect(Object.keys(manifest2!.files).length).toBeGreaterThan( + Object.keys(manifest1!.files).length + ); + }); + + it('should regenerate keyword index with all chunks during incremental', async () => { + await fs.writeFile(path.join(tempDir, 'a.ts'), 'export const a = 1;'); + await fs.writeFile(path.join(tempDir, 'b.ts'), 'export const b = 2;'); + + // Full index + const indexer1 = new CodebaseIndexer({ + rootPath: tempDir, + config: { skipEmbedding: true } + }); + await indexer1.index(); + + const indexPath = path.join(tempDir, CODEBASE_CONTEXT_DIRNAME, KEYWORD_INDEX_FILENAME); + const fullIndex = JSON.parse(await fs.readFile(indexPath, 'utf-8')); + + // Modify one file + await fs.writeFile(path.join(tempDir, 'a.ts'), 'export const a = 999;'); + + // Incremental index + const indexer2 = new CodebaseIndexer({ + rootPath: tempDir, + config: { skipEmbedding: true }, + incrementalOnly: true + }); + await indexer2.index(); + + // Keyword index should still contain chunks from ALL files (not just changed ones) + const incrementalIndex = JSON.parse(await fs.readFile(indexPath, 'utf-8')); + expect(incrementalIndex.length).toBeGreaterThanOrEqual(fullIndex.length); + }); + + it('should not include incremental stats in full index', async () => { + await fs.writeFile(path.join(tempDir, 'index.ts'), 'export const x = 1;'); + + const indexer = new CodebaseIndexer({ + rootPath: tempDir, + config: { skipEmbedding: true } + }); + const stats = await indexer.index(); + + expect(stats.incremental).toBeUndefined(); + }); +}); diff --git a/tests/manifest.test.ts b/tests/manifest.test.ts new file mode 100644 index 0000000..71d31e7 --- /dev/null +++ b/tests/manifest.test.ts @@ -0,0 +1,192 @@ +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import { promises as fs } from 'fs'; +import path from 'path'; +import { fileURLToPath } from 'url'; +import { + hashFileContent, + readManifest, + writeManifest, + computeFileHashes, + diffManifest, + type FileManifest +} from '../src/core/manifest.js'; + +const __filename = fileURLToPath(import.meta.url); +const __dirname = path.dirname(__filename); + +describe('Manifest System', () => { + const testDir = path.join(__dirname, 'test-workspace-manifest'); + const manifestPath = path.join(testDir, 'manifest.json'); + + beforeAll(async () => { + await fs.mkdir(testDir, { recursive: true }); + }); + + afterAll(async () => { + await fs.rm(testDir, { recursive: true, force: true }); + }); + + describe('hashFileContent', () => { + it('should produce consistent SHA-256 prefix', () => { + const hash1 = hashFileContent('hello world'); + const hash2 = hashFileContent('hello world'); + expect(hash1).toBe(hash2); + expect(hash1).toHaveLength(16); + expect(hash1).toMatch(/^[0-9a-f]{16}$/); + }); + + it('should produce different hashes for different content', () => { + const hash1 = hashFileContent('file A content'); + const hash2 = hashFileContent('file B content'); + expect(hash1).not.toBe(hash2); + }); + + it('should be sensitive to whitespace changes', () => { + const hash1 = hashFileContent('const x = 1;'); + const hash2 = hashFileContent('const x = 1;'); + expect(hash1).not.toBe(hash2); + }); + }); + + describe('readManifest / writeManifest', () => { + it('should round-trip a valid manifest', async () => { + const manifest: FileManifest = { + version: 1, + generatedAt: '2026-02-08T00:00:00.000Z', + files: { + 'src/index.ts': 'abcdef1234567890', + 'src/utils.ts': '1234567890abcdef' + } + }; + + await writeManifest(manifestPath, manifest); + const read = await readManifest(manifestPath); + expect(read).toEqual(manifest); + }); + + it('should return null for missing file', async () => { + const result = await readManifest(path.join(testDir, 'nonexistent.json')); + expect(result).toBeNull(); + }); + + it('should return null for corrupt JSON', async () => { + const corruptPath = path.join(testDir, 'corrupt.json'); + await fs.writeFile(corruptPath, 'not valid json{{{'); + const result = await readManifest(corruptPath); + expect(result).toBeNull(); + }); + + it('should return null for wrong version', async () => { + const wrongVersionPath = path.join(testDir, 'wrong-version.json'); + await fs.writeFile(wrongVersionPath, JSON.stringify({ version: 99, files: {} })); + const result = await readManifest(wrongVersionPath); + expect(result).toBeNull(); + }); + + it('should return null for missing files field', async () => { + const noFilesPath = path.join(testDir, 'no-files.json'); + await fs.writeFile(noFilesPath, JSON.stringify({ version: 1 })); + const result = await readManifest(noFilesPath); + expect(result).toBeNull(); + }); + }); + + describe('computeFileHashes', () => { + it('should compute hashes for files relative to root', async () => { + const fileA = path.join(testDir, 'a.ts'); + const fileB = path.join(testDir, 'b.ts'); + await fs.writeFile(fileA, 'export const a = 1;'); + await fs.writeFile(fileB, 'export const b = 2;'); + + const hashes = await computeFileHashes([fileA, fileB], testDir); + expect(Object.keys(hashes)).toHaveLength(2); + expect(hashes['a.ts']).toMatch(/^[0-9a-f]{16}$/); + expect(hashes['b.ts']).toMatch(/^[0-9a-f]{16}$/); + expect(hashes['a.ts']).not.toBe(hashes['b.ts']); + }); + + it('should skip unreadable files', async () => { + const hashes = await computeFileHashes(['/nonexistent/file.ts'], testDir); + expect(Object.keys(hashes)).toHaveLength(0); + }); + + it('should accept custom readFile function', async () => { + const mockRead = async () => 'mock content'; + const hashes = await computeFileHashes( + [path.join(testDir, 'virtual.ts')], + testDir, + mockRead + ); + expect(hashes['virtual.ts']).toBe(hashFileContent('mock content')); + }); + }); + + describe('diffManifest', () => { + it('should treat all files as added when old manifest is null', () => { + const currentHashes = { + 'src/a.ts': 'aaaa000000000000', + 'src/b.ts': 'bbbb000000000000' + }; + const diff = diffManifest(null, currentHashes); + expect(diff.added).toEqual(['src/a.ts', 'src/b.ts']); + expect(diff.changed).toEqual([]); + expect(diff.deleted).toEqual([]); + expect(diff.unchanged).toEqual([]); + }); + + it('should correctly categorize added, changed, deleted, unchanged', () => { + const oldManifest: FileManifest = { + version: 1, + generatedAt: '2026-01-01T00:00:00.000Z', + files: { + 'src/unchanged.ts': 'aaaa000000000000', + 'src/changed.ts': 'bbbb000000000000', + 'src/deleted.ts': 'cccc000000000000' + } + }; + + const currentHashes = { + 'src/unchanged.ts': 'aaaa000000000000', // same hash + 'src/changed.ts': 'dddd111111111111', // different hash + 'src/added.ts': 'eeee222222222222' // new file + }; + + const diff = diffManifest(oldManifest, currentHashes); + expect(diff.added).toEqual(['src/added.ts']); + expect(diff.changed).toEqual(['src/changed.ts']); + expect(diff.deleted).toEqual(['src/deleted.ts']); + expect(diff.unchanged).toEqual(['src/unchanged.ts']); + }); + + it('should handle empty manifests', () => { + const oldManifest: FileManifest = { + version: 1, + generatedAt: '2026-01-01T00:00:00.000Z', + files: {} + }; + + const diff = diffManifest(oldManifest, {}); + expect(diff.added).toEqual([]); + expect(diff.changed).toEqual([]); + expect(diff.deleted).toEqual([]); + expect(diff.unchanged).toEqual([]); + }); + + it('should handle all files deleted', () => { + const oldManifest: FileManifest = { + version: 1, + generatedAt: '2026-01-01T00:00:00.000Z', + files: { + 'a.ts': 'aaaa000000000000', + 'b.ts': 'bbbb000000000000' + } + }; + + const diff = diffManifest(oldManifest, {}); + expect(diff.added).toEqual([]); + expect(diff.changed).toEqual([]); + expect(diff.deleted).toEqual(['a.ts', 'b.ts']); + expect(diff.unchanged).toEqual([]); + }); + }); +}); diff --git a/tests/memory.test.ts b/tests/memory.test.ts index 360243e..e92313d 100644 --- a/tests/memory.test.ts +++ b/tests/memory.test.ts @@ -46,7 +46,7 @@ describe('Memory System', () => { it('should support all decision categories and types', () => { const validCategories = ['tooling', 'architecture', 'testing', 'dependencies', 'conventions']; - const validTypes = ['convention', 'decision', 'gotcha']; + const validTypes = ['convention', 'decision', 'gotcha', 'failure']; validCategories.forEach((category) => { validTypes.forEach((type) => {