Skip to content

perf(navigation): replace legacy reverse-caller scans after v0.4 #342

Description

@styler-ai

Why

ProjectAtlas v0.4 reduces the large-corpus atlas_file_summary path from about 1.396 GB of process reads and 2.23 seconds to 52-55 MiB and 0.41-0.44 seconds without changing output. That passes the preregistered v0.4 ceiling, but the remaining read cost is concentrated in legacy import-alias candidate discovery instead of normalized graph traversal.

Measured on the pinned VS Code 1.130.0 corpus (16,117 indexed files, 538,047 symbols, 1,285,073 legacy relations):

  • Two valid but non-matching full module-name probes account for about 43 MiB.
  • The matching basename probe plus 453 exact caller-path reads account for about 5.5 MiB.
  • Repeated exact summaries remain below the v0.4 64 MiB bound, but a graph-native path should need materially less work.

This is post-v0.4 optimization work. It must not delay or destabilize the v0.4.0 release.

What Changes

Evaluate and implement the smallest proven replacement for legacy reverse-caller alias scans after v0.4.0 stabilizes:

  1. Prefer a graph-native reverse-caller lookup using normalized resolved call relations when it preserves the public called_by contract.
  2. Evaluate language-aware import candidate terms only if they retain uncommon Rust, Python, TypeScript/JavaScript, Fortran, COBOL, ABAP, and fallback-language behavior.
  3. Consider a substring-specific SQLite index only if query and persistent-size measurements justify its migration and storage cost.
  4. Keep the current bounded indexed path as the compatibility fallback until back-to-back output proof shows it can be removed.

Select one measured design; do not implement all three.

Capabilities

  • Faster bounded atlas_file_summary and corresponding CLI/MCP summary routes.
  • Graph-native reverse relationship use for called_by when exact and compatible.
  • Exact work accounting for database rows/bytes and caller hydration.
  • Preserved ambiguity rejection and parser-backed import-alias correctness.

Architecture Diagrams

Release Scope

First suitable post-v0.4.0 minor release, after #308 and the v0.4.0 release evidence are complete. Expected surface: projectatlas-db query ownership, projectatlas-service file-summary caller projection, CLI/MCP compatibility tests, system-scale benchmarks, and relevant architecture documentation.

Affected surface: Performance, MCP tools, CLI, and agent workflow.

Agent workflow

overview -> folders -> files -> file_summary -> resolved caller relationship -> smallest trustworthy slice

The optimization must reduce internal work without requiring agents to issue a new command or learn a different summary schema.

Acceptance criteria

  • On the pinned large corpus, the exact packaged candidate materially improves first-call atlas_file_summary read bytes and latency over the v0.4.0 baseline of 52-55 MiB and 0.41-0.44 seconds.
  • The chosen target and pass/fail threshold are preregistered before implementation measurements are opened.
  • called_by output is identical for the frozen compatibility corpus, or any intentional additive contract change is separately specified and versioned.
  • Positive, negative, ambiguous, unresolved, external, fallback-parser, and cross-language import/call cases remain covered.
  • Real SQLite tests prove the selected query plan, bounded rows/decoded bytes, reopen behavior, and no authoritative mutation from read-only routes.
  • MCP and CLI E2E tests prove identical accepted inputs, output schema, generation/freshness state, truncation, and failure behavior.
  • Persistent database size, scan time, startup, peak RSS, and unchanged refresh do not regress beyond preregistered bounds.
  • On the same paired agent workload, report answer correctness, total input/output tokens, observed and modeled ProjectAtlas tokens avoided, MCP/tool-call count, context bytes, wrong folder/file/relation visits, broad reads, and backtracking against the frozen v0.4.0 baseline; treat any improvement as unproven unless the raw traces support it.
  • The benchmark separates observed measurements from modeled savings and publishes raw reusable artifacts.

Non-Goals

  • No v0.4.0 scope expansion.
  • No weakening of freshness, ambiguity, compatibility, or bounded-output checks.
  • No new semantic model, network dependency, provider credential, or lifecycle scaffolding.
  • No new SQLite table or index without measured benefit exceeding its migration and persistent-size cost.
  • No removal of legacy caller behavior until exact back-to-back compatibility proof exists.

Pre-Mortem

Likely failure modes:

  • Language-aware terms miss uncommon but valid import spellings.
  • Graph-native callers lose the legacy source-name portion of path::source_name.
  • A new index saves reads but pushes large repositories over the storage ceiling.
  • A cache improves repeated calls while leaving the first agent call expensive or stale.
  • Optimization claims conflate OS process I/O, SQLite decoded work, and modeled token savings.

Mitigation checkboxes and owning task IDs will be added when the post-v0.4 OpenSpec change is created; no task IDs are invented before that change exists.

OpenSpec plan and task checklist

OpenSpec change: to be assigned after v0.4.0

OpenSpec Tasks

The future change must cover query ownership, compatibility behavior, real SQLite plan/size proof, CLI/MCP E2E, system-scale regression gates, and documentation synchronization.

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions