feat(symgraph): improve ingestion pipeline for large repositories.

[Zenithar]

Summary

This changes full reindexing from an in-place incremental update into a clean rebuild of a shadow database followed by an atomic swap.

It also hardens indexing and extraction in a few places:

• keeps full rebuilds from leaving stale symbols behind
• keeps the live DB usable if a rebuild fails
• improves bulk indexing behavior for FTS/WAL handling
• avoids stack overflows on deeply nested ASTs
• makes CLI indexing show progress

Problem

Before this change, the system treated “reindex everything” too much like “incrementally update the existing DB.”

That caused a few concrete issues:

1. Full reindexes could leave stale rows behind after file deletes or symbol renames.
2. A failed rebuild risked interfering with the live on-disk database state.
3. Bulk indexing did unnecessary per-file/per-row maintenance work.
4. Deeply nested syntax trees could overflow the extractor’s recursive traversal.
5. The CLI and MCP surface described reindexing in ways that no longer matched the behaviour we actually want.

What changed

1. Full rebuilds now use a shadow database and atomic swap

• Added build_full_index for building a complete index into an empty target DB.
• Added shadow DB helpers:
  • open_shadow_database
  • rebuild_project_database
  • swap_shadow_database
• Added DB swap support:
  • prepare_for_swap
  • replace_with_shadow
  • sidecar cleanup / reopen helpers

Why:

• Solves stale data after deletes/renames by rebuilding from scratch.
• Gives us a clean cutover instead of mutating the live DB in place.

2. CLI index now performs a real rebuild

• index_command now calls rebuild_project_database(...) instead of incremental indexing.
• Added show_progress to IndexConfig.
• Added indicatif and wired progress bars for scanning/parsing/storing/resolving.

Why:

• Solves the mismatch where the symgraph index sounded like a fresh rebuild but behaved incrementally.
• Makes large indexing runs easier to understand from the CLI.

3. MCP reindex semantics are now explicit

• files: Some([...]) still performs targeted in-place reindexing.
• Omitting files now performs a full shadow rebuild.
• Updated MCP descriptions and output text to reflect that.

Why:

• Solves API ambiguity.
• Preserves the cheap targeted path for specific files while making full rebuilds correct.

4. Bulk indexing path is split by mode

• Introduced IndexMode::{Incremental, FullBuild}.
• Incremental mode:
  • disables FTS automerge during bulk inserts
  • checkpoints WAL every CHECKPOINT_INTERVAL files
  • optimises FTS after inserts
• Full-build mode:
  • inserts canonical rows first
  • skips per-insert FTS maintenance
  • rebuilds FTS once at the end

Why:

• Solves unnecessary bulk-write overhead.
• Keeps WAL size bounded during large incremental updates.
• Makes full rebuilds behave like full rebuilds rather than repeated incremental updates.

5. Extraction traversal is now iterative

• Replaced recursive symbol traversal with an explicit work stack.
• Kept contains-edge / nested symbol behaviour intact through explicit push/pop handling.

Why:

• Solves stack overflow risk on deeply nested ASTs.

6. Signature truncation is UTF-8 safe

• Long signatures are now truncated at a valid character boundary.

Why:

• Solves potential invalid slicing on non-ASCII signatures.

Tests added / updated

Added regression coverage for:

• full build populating an empty DB
• full rebuild replacing stale rows
• keeping the live DB intact on rebuild failure
• MCP full rebuild, reopening the live handle correctly
• file-scoped reindex staying in place
• periodic checkpoint behaviour in incremental indexing
• deeply nested extraction without stack overflow

Verification

Ran targeted tests:

• cargo test test_build_full_index_populates_empty_target_db
• cargo test test_rebuild_project_database
• cargo test test_incremental_index_codebase_periodic_checkpoint
• cargo test test_handle_reindex_
• cargo test test_deeply_nested_does_not_overflow

All passed.

Notes for reviewers

The main design decision here is a clean cutover:

• full rebuilds are now correctness-first via shadow DB swap
• targeted file reindex remains incremental and scoped

[Zenithar]

@grahambrooks, is there any chance of getting a review of this PR?

[grahambrooks]

@Zenithar

Sorry about the delay in review and merge. I'll push a new release tomorrow.