perf: make cache identities portable and streamline hot paths

[lzehrung]

Summary

• Store persisted index identities relative to the active project root, migrate legacy absolute-path artifacts, and confine cache paths before reuse.
• Preserve resolver fingerprints, bloom-sidecar validation, duplicate consumer paths, and external reexport specifiers across portable cache hydration.
• Collapse duplicate search scans, batch cache writes, hydrate validated bloom sidecars, and avoid unnecessary content hashes while retaining diagnostics.

Verification

• npm run check
• Focused cache, duplicate, query-index, search, discovery, confinement, and CLI-startup suites

[Copilot]

Pull request overview

This PR refactors the disk cache to persist project-relative identities (and migrate legacy absolute-path artifacts), adds stricter confinement/validation when hydrating cached state, and streamlines several hot paths (batch writes, reduced scans, and improved query-index candidate selection). It also expands CLI/config surfaces for cache placement and updates documentation and tests accordingly.

Changes:

• Make persisted cache artifacts portable by storing project-relative paths and migrating legacy absolute-path entries (module cache, manifests, project snapshots, duplicate unit cache, bloom sidecars).
• Harden cache hydration by confining rehydrated paths/handles to the active project root and rejecting tampered/out-of-root persisted data.
• Improve performance on hot paths (batch module-cache/duplicate-cache writes, reduce redundant scans, improved query-index candidate selection and fallback parallelism).

Reviewed changes

Copilot reviewed 40 out of 40 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/query-index.test.ts	Adds diagnostics coverage for query-index candidate selection (files vs chunks).
tests/project-file-discovery.test.ts	Adds coverage for safe symlink traversal when explicitly including otherwise-ignored directories.
tests/node-modules-and-paths.test.ts	Adds incremental-cache coverage for node-modules edge refresh when package targets change.
tests/disk-cache-sqlite.test.ts	Expands sqlite cache tests for schema bumps, batch rollback, and portable duplicate identities/handles.
tests/cli-options-validation.test.ts	Adds parsing coverage for Git range value options.
tests/cache-path-confinement.test.ts	New suite asserting cache hydration confinement for module cache, snapshots, manifests, bloom sidecars, and reexports.
tests/cache-modes.test.ts	Updates cache row expectations to reflect relative keys and new cache versioning.
tests/cache-invalidation.test.ts	Updates invalidation expectations and adds confinement + stale snapshot/bloom recovery scenarios.
tests/agent-session.test.ts	Updates sidecar/snapshot versions and adds epoch drift invalidation coverage.
tests/agent-search.test.ts	Adds phrase-boost regression coverage for deduped rank-token joins.
src/util/projectFiles.ts	Improves safe-symlink discovery handling with include overrides and reduces repeated matcher construction.
src/indexer/types.ts	Adds languageExtensions to ProjectIndex and introduces cacheLocation in BuildOptions.
src/indexer/parse-context.ts	Threads languageExtensions through parsing preparation.
src/indexer/navigation.ts	Ensures navigation parsing respects languageExtensions.
src/indexer/finalize.ts	Normalizes/stores language extensions on the finalized index and updates cacheRoot import.
src/indexer/build-manifest.ts	Persists manifest entries as project-relative and records symlink directories/transients as relative.
src/indexer/build-index.ts	Batches module-cache writes, adds snapshot-module reuse, improves bloom hydration checks, and adjusts node-modules cache reuse behavior.
src/indexer/build-cache/project-snapshot.ts	Migrates snapshots, stores relative paths + signatures, adds bloom sidecar format/versioning, and tightens bloom validation.
src/indexer/build-cache/options.ts	Introduces CORE_ALGORITHM_EPOCH in build-option fingerprints and diffs.
src/indexer/build-cache/module-cache.ts	Stores module cache keys as project-relative, adds path rehydration transforms, and batches disk writes in a transaction.
src/indexer/build-cache/manifest.ts	Migrates v3 manifests to relative entries, transforms entries on load, and rehydrates symlink directories across moved roots.
src/indexer/build-cache/location.ts	New cache-root resolution logic supporting repo/user anchors and explicit overrides.
src/indexer/build-cache.ts	Re-exports updated cache APIs (new location + snapshot helpers).
src/graphs/symbol-graph-detailed.ts	Ensures detailed symbol graph build respects languageExtensions.
src/duplicates/units.ts	Introduces symbol-aware chunking path and batches duplicate-unit cache writes.
src/duplicates/unitCache.ts	Stores duplicate-unit cache entries/handles as project-relative and validates confinement during hydration.
src/config.ts	Adds cache location config (project + user config) and plumbs cache config into loaded config.
src/cli/viewer.ts	Updates cacheRoot import to the new location module.
src/cli/options.ts	Treats --cache-dir as a valued option and shares it across commands.
src/cli/invocationContext.ts	Plumbs --cache-dir and config cache.location into BuildOptions.
src/cli/inspect.ts	Uses computed cacheRoot for default cache index path.
src/cli/help.ts	Documents --cache-dir and cache location precedence/semantics.
src/cli/doctor.ts	Adds cache path/anchor diagnostics to doctor output.
src/chunking/chunkFile.ts	Adds symbol-chunk output while preserving existing chunkFile() behavior.
src/agent/search.ts	Improves query-index candidate scoring/snippets and parallelizes fallback scanning; avoids queue shift hot path.
src/agent/query-index/store.ts	Adds a unified candidate-chunk query and sets sqlite synchronous mode.
src/agent/query-index/candidates.ts	Reworks candidate scoring to return richer candidate data (score + matched line).
docs/cli.md	Documents cache location behavior and precedence.
codegraph-skill/codegraph/SKILL.md	Updates skill docs to reflect project-boundary vs cache-location behavior.
AGENTS.md	Updates repo guidance to reflect cache anchoring while keeping --root as project boundary.

Suppressed comments (1)

src/config.ts:207

• loadCodegraphConfig() always returns cache.location (defaulting to "project") when codegraph.config.json exists. That makes the CLI always set BuildOptions.cacheLocation = "project", preventing the documented/default repository-anchor behavior (repo metadata -> project fallback) for any project that has a config file but no explicit cache config. Consider only setting cache when it’s explicitly configured (project or user), and otherwise leaving it undefined so the normal anchor resolution can apply.

  const languageExtensions = normalizeConfigLanguageExtensions(parsed.data.languages?.extensions);
  const discovery = normalizeDiscoveryConfig(parsed.data.discovery);
  const resolutionHints = normalizeResolutionHints(parsed.data.graph?.resolutionHints);
  const graph = resolutionHints.length ? { resolutionHints } : undefined;
  return {
    cache: { location: parsed.data.cache?.location ?? userCacheLocation ?? "project" },
    ...(discovery ? { discovery } : {}),
    ...(graph ? { graph } : {}),
    ...(languageExtensions ? { languages: { extensions: languageExtensions } } : {}),
  };

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 41 out of 41 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/indexer/build-cache/location.ts:80

• cacheRoot() falls back to the project root when the resolved cache anchor directory does not already exist. This breaks --cache-dir / CODEGRAPH_CACHE_DIR / cache.location: "user" on first use because those anchors are expected to be created later by the code that writes cache artifacts, not to be silently ignored.

export function cacheRoot(projectRoot: string, opts?: BuildOptions): string {
  const root = path.resolve(projectRoot);
  const resolution = resolveCacheAnchor(root, opts);
  const anchor = isWritableDirectory(resolution.anchor) ? resolution.anchor : root;
  const sameRoot = fileIdentityKey(anchor) === fileIdentityKey(root);

[Copilot]

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated no new comments.

Suppressed comments (9)

src/indexer/build-cache/project-snapshot.ts:512

• A current portable snapshot keeps the absolute root where it was written. After moving a project, any changed file prevents whole-snapshot reuse and this check then rejects the new per-file snapshot hydration for all unchanged modules, despite their relative paths and signatures being valid. Remove the provenance-root equality check here; confinement is already enforced while paths are transformed.

      !projectRootMatches(projectRoot, payload.projectRoot) ||

src/indexer/build-cache/project-snapshot.ts:566

• The bloom sidecar is also keyed by relative file paths/signatures, but this absolute provenance check rejects it after the project is moved; the fallback snapshot repeats the same check at line 595. As a result, a partial rebuild after a move regenerates bloom filters for every unchanged file. Validate portable identity/signatures and confinement instead of requiring the old absolute root in both paths.

    !projectRootMatches(projectRoot, payload.projectRoot) ||

src/cli/options.ts:131

• --cache-dir is now accepted as a shared build option, but core handlers such as index, graph, review, navigation, impact, and inspect build their own options and never read it (for example, src/cli/index.ts:88-116). Those commands therefore silently ignore both this flag and the new configured cache location. Thread cacheDir/cacheLocation through every shared build path or centralize them on buildAgentOptions().

  "--cache-dir",

src/indexer/build-cache/project-snapshot.ts:238

• Legacy v4 snapshots predate the newly required fileSignatures field, so Object.entries(copy.fileSignatures) throws during the migration and the catch turns every real legacy snapshot into a cache miss. The move test only changes a current payload's version, so it retains this field and does not exercise the actual old schema.

This issue also appears in the following locations of the same file:

• line 512
• line 566

  for (const [file, signature] of Object.entries(copy.fileSignatures)) {

src/indexer/build-cache/location.ts:61

• The namespace still hashes the absolute project path. For a subproject using the new repository anchor, moving the repository carries the cache directory along but changes this hash, so the moved cache is no longer discoverable; the existing move test only covers a project-root cache. Derive repository-anchored namespaces from a stable identity such as the project path relative to the anchor.

export function projectCacheNamespace(projectRoot: string): string {
  const rootIdentity = fileIdentityKey(path.resolve(projectRoot));
  const hash = crypto.createHash("sha256").update(rootIdentity).digest("hex");
  return `project-${hash}`;

src/cli/doctor.ts:325

• The new cache diagnostics are present only in JSON: formatDoctorSummary never reads report.cache, so the default human-readable codegraph doctor output hides the path, anchor, and layer. Add a Cache section to the formatter and a pretty-output assertion.

    cache: {
      path: normalizePathForDisplay(cacheResolution.path),
      anchor: normalizePathForDisplay(cacheResolution.anchor),
      layer: cacheResolution.layer,
    },

src/config.ts:214

• Library/agent sessions do not consume this new config value: CodeReviewSession.currentBuildOptions (src/session.ts:242-256) merges discovery, graph, and language settings but not config.cache.location. Consequently createAgentSession({ root }) ignores cache.location unless a caller separately copies it into buildOptions; merge it there and include it in session identity normalization.

  const cacheLocation = parsed.data.cache?.location ?? userCacheLocation;
  return {
    ...(cacheLocation ? { cache: { location: cacheLocation } } : {}),

src/agent/query-index/store.ts:366

• This consolidated query has no candidate bound, whereas the replaced FTS helpers capped prefetching. A common term can now materialize and deserialize every matching chunk in every eligible file, then sort the entire set before QUERY_INDEX_CANDIDATE_ROW_LIMIT is applied, causing memory and latency to scale with the whole index. Preserve a bounded prefetch strategy while batching paths.

      const rows = this.db
        .prepare(
          `
          ${prefix}
          SELECT files.path AS path, chunks.ordinal, chunks.kind, chunks.name,
                 chunks.start_line, chunks.end_line, chunks.text, chunks.normalized_text
          FROM chunks
          JOIN files ON files.file_id = chunks.file_id
          WHERE files.path IN (${placeholders})
            AND (${conditions.join(" OR ")})
          ORDER BY files.path, chunks.ordinal
        `,
        )
        .all(
          ...(ftsTerms.length ? [ftsQuery, ...batch, ...directParameters] : [...batch, ...directParameters]),
        ) as Array<Record<string, unknown>>;

docs/cli.md:27

• This heading is inserted between Default workflow: and its list, so the workflow bullets are now rendered as part of the Cache location section and the original label is orphaned. Move the cache section after the workflow list.

## Cache location

Index caches store project-relative paths, so a cache can be moved with its project. Cache selection precedence is `--cache-dir`, `CODEGRAPH_CACHE_DIR`, `cache.location` in project config (then user config), repository metadata, then the project root. `cache.location` accepts `project`, `repo`, `user`, or an absolute path; `--root` remains the project scope boundary.

[Copilot]

Pull request overview

Copilot reviewed 53 out of 53 changed files in this pull request and generated no new comments.

Suppressed comments (5)

src/indexer/finalize.ts:35

• Keep ProjectIndex.projectRoot normalized. Callers may pass . or another relative root, so this now makes cold indexes expose a relative root while snapshot-hydrated and empty incremental indexes expose an absolute root. It also makes later duplicate-cache/path resolution depend on the process working directory at query time.
  src/duplicates/units.ts:567
• The batched writer drops options.projectRoot, even though reads above use that override. When duplicate analysis is scoped to a project root different from index.projectRoot (a supported public option), rows are written relative to the index root but subsequently queried relative to the override, so the disk cache can never hit. Pass the effective collection root through the batch writer and use it for signatures, keys, and payload transforms.

  if (pendingWrites.length) {
    writeDuplicateUnitsBatchToCache(index, pendingWrites);

src/indexer/build-cache/location.ts:90

• Nonexistent user and environment cache directories are still the actual write targets (lines 104-111), but this marks their anchor as the project root and layer as project. This makes the exported resolution and the new doctor diagnostics report metadata that contradicts the returned path on first use. Treat user and environment anchors as creatable, like explicit, or return metadata for the configured target.

  const anchorWritable = resolution.layer === "explicit" || isWritableDirectory(resolution.anchor);
  const anchor = anchorWritable ? resolution.anchor : root;
  const effectiveLayer = anchorWritable ? resolution.layer : "project";

src/cli/doctor.ts:329

• The new cache diagnostic ignores both project and user cache.location configuration because it resolves with no build options; only environment/default anchoring is reflected. Thus doctor can report a different cache path from every project command. Load/pass the effective cache configuration for the target project before constructing this section (while preserving the installer call site's non-project behavior).

  const cacheResolution = resolveCacheLocation(process.cwd());

src/cli/inspect.ts:123

• inspect and hotspots accept the shared --cache-dir option, and project config can now select cache.location, but this helper always resolves only the default location. Their index loaders likewise receive neither option, so these commands can silently build/read a different cache and report/recommend the wrong manifest path. Thread cacheDir and configured cacheLocation through InspectCommandContext, the loader options, and these metadata helpers.

function defaultCacheIndexPath(projectRoot: string): string {
  return cacheRoot(projectRoot, { cache: "disk" });
}

[Copilot]

Pull request overview

Copilot reviewed 54 out of 54 changed files in this pull request and generated 1 comment.

Suppressed comments (4)

src/config.ts:214

• loadCodegraphConfig() now exposes cache.location, but createAgentSession() only merges discovery, graph, and language settings from that result (src/agent/session.ts:114-143). A direct library/MCP agent session with useConfig enabled therefore still reads and writes the default cache while CLI and review sessions honor the configured location. Propagate the resolved cache location through the agent file plan and all index/sidecar build options.

  const cacheLocation = parsed.data.cache?.location ?? userCacheLocation;
  return {
    ...(cacheLocation ? { cache: { location: cacheLocation } } : {}),

src/cli/doctor.ts:347

• The new Cache report is not the effective cache location when only the user config sets cache.location. Builds call loadCodegraphConfig(), which falls back to the platform user config, but doctor reads only ./codegraph.config.json; in that supported setup it reports a different path/anchor/layer than commands actually use. Apply the same project-over-user lookup semantics here while preserving the startup constraint.

  const cacheLocation = readProjectCacheLocation(process.cwd());
  const cacheResolution = resolveCacheLocation(process.cwd(), cacheLocation ? { cacheLocation } : undefined);

src/indexer/parse-context.ts:176

• The extension map is only forwarded by the updated navigation call sites. Other reparsing paths such as workspace-symbols.ts:170, agent/renamePreview.ts:528, impact/map.ts:52, and navigation-references.ts:132 still call this without index.languageExtensions; because ordinary indexes default to keepParsed: false, those features fail or silently lose semantic results for configured custom extensions. Thread the index mapping through every index-backed ensureParsedContext call.
  src/indexer/types.ts:146
• cacheLocation is a new public library build option, but the canonical library API documentation is not updated to explain its accepted values, precedence relative to cacheDir/the environment, or that absolute locations are anchors rather than final cache directories. Add this contract to docs/library-api.md; the CLI-only paragraph does not document TypeScript API behavior.

[Copilot]

Pull request overview

Copilot reviewed 65 out of 65 changed files in this pull request and generated no new comments.

Suppressed comments (7)

src/util/projectFiles.ts:344

• When any include glob is present, this drops every default ignore from the full fg(["**/*"]) symlink probe, even when the include only targets src/**. Repositories with large node_modules, vendor, or build trees will now walk all of those directories before the later include filter removes them. Keep default ignores that are not actually reopened by an include glob, and remove only the intersecting ignored roots.
  src/agent/query-index/store.ts:356
• The global, path-ordered limit can be exhausted by a common term in early path batches before chunks matching rarer terms are read. For example, with more than 8,000 early alpha chunks, a later chunk matching both alpha and beta is omitted entirely, so the in-memory scorer cannot retain the best result. Prefetch a bounded set per term and union it, or rank term coverage in SQL before applying the global cap.

    for (let offset = 0; offset < paths.length && candidates.size < normalizedLimit; offset += batchSize) {
      const batch = paths.slice(offset, offset + batchSize);
      const placeholders = batch.map(() => "?").join(", ");
      const remaining = normalizedLimit - candidates.size;

src/indexer/build-index.ts:654

• Disabling manifest-edge reuse does not disable the per-file module cache, whose ImportBinding.resolved values also contain node-module targets. A warm buildProjectIndex() or buildProjectIndexFromFiles() can therefore return stale module imports after a package target changes, even though its graph edges are freshly resolved; only the incremental path currently forces every file changed. Bypass module/snapshot reuse for this mode or include resolver-environment state in the module cache signature.
  src/indexer/build-cache/module-cache.ts:140
• ensureSqliteVersionedTableSchema invokes this migration callback for every valid schema version, including version 2, so every cache open now selects and loops over the entire module cache. Since build entry points close the database after each build, this adds an O(project files) startup scan to every warm invocation. Run the key backfill only when upgrading from a pre-v2 schema.
  src/duplicates/unitCache.ts:215
• This full-table key migration also runs on every opening of an already-v2 duplicate cache because the shared schema helper always calls migrateTable for valid versions. Duplicate caches can contain many rows per source file, so repeated invocations pay an unnecessary O(cache rows) startup scan. Gate this backfill on an actual pre-v2 upgrade.

  const rows = db.prepare("SELECT file FROM duplicate_unit_cache").all() as Array<{ file: string }>;
  const update = db.prepare("UPDATE duplicate_unit_cache SET file = ? WHERE file = ?");
  for (const row of rows) {
    const relative = cacheRelativePath(projectRoot, row.file);
    if (relative !== row.file) update.run(relative, row.file);

src/cli/doctor.ts:338

• This lightweight reader accepts any nonempty string, while the real config schema rejects relative locations. Consequently doctor reports a relative invalid config as an effective explicit cache path even though every real build fails validation. Apply the same project/repo/user/absolute-path check before returning the value.

function readCacheLocationField(configPath: string): string | undefined {
  try {
    const raw = fs.readFileSync(configPath, "utf8");
    const parsed = JSON.parse(raw) as { cache?: { location?: unknown } };
    const location = parsed.cache?.location;
    return typeof location === "string" && location.trim() ? location.trim() : undefined;

docs/library-api.md:639

• This describes cacheDir as the final cache directory, but resolveCacheLocation normally appends project-<hash> to it (and the public type documentation also calls it an anchor). Users following this API documentation will look for artifacts at the wrong path. Document cacheDir, CODEGRAPH_CACHE_DIR, and absolute cacheLocation consistently as namespaced anchors.

Anchor selection precedence: `cacheDir` (an explicit final directory), then `CODEGRAPH_CACHE_DIR`,
then `cacheLocation`, then repository metadata (nearest ancestor `.git`/`.codegraph`), then the
project root. `cacheLocation` accepts `"project"` (anchor at `projectRoot`), `"user"` (anchor at
the platform user cache directory), `"repo"` (the default repository-metadata search), or an
absolute path. Unlike `cacheDir`, an absolute `cacheLocation` is an anchor, not the final cache

[Copilot]

Pull request overview

Copilot reviewed 67 out of 67 changed files in this pull request and generated 1 comment.

Suppressed comments (2)

src/util/sqliteSchema.ts:89

• A current schema marker no longer guarantees that the table exists. If cache_schema_metadata survives while the table is deleted or partially restored, this branch skips migrateTable; callers then fail while creating indexes or preparing statements. Preserve the fast path but run the idempotent createTable callback for the current-version case so a missing table is repaired.
  src/indexer/build-cache/project-snapshot.ts:1127
• The newly persisted languageExtensions field is not validated by isProjectIndexSnapshotPayload. A malformed cache can therefore pass this guard and later throw in normalizeLanguageExtensions (for example when a mapping value is not a string), turning cache corruption into command failure instead of a clean cache miss. Validate the record's keys and string values before hydrating it.

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[Copilot]

Pull request overview

Copilot reviewed 67 out of 67 changed files in this pull request and generated 1 comment.

Suppressed comments (2)

src/indexer/build-cache/project-snapshot.ts:1271

• Incremental builds finalize manifestEntries from ManifestFileEntry, which does not carry cacheSig, so this serializer writes only the weak mtime:size signature for non-Git files even though prepareFileSignatures computed a content hash. Subsequent partial snapshot and bloom-sidecar hydration then falls back to sig and can reuse stale data after a same-size edit with restored mtime. Preserve the current FileSignature.cacheSig in the incremental index entries before writing these artifacts.
  src/indexer/build-cache/location.ts:74
• The public BuildOptions.cacheLocation contract accepts only project, repo, user, or an absolute path, but any typo or relative value reaches this branch and is silently resolved against the process working directory. Validate path.isAbsolute(location) here and throw the same actionable validation error used for config input, so library callers cannot accidentally write to an unintended cache anchor.