Skip to content

Reuse bounded sessions for parallel batch queries - #4972

Merged
Widthdom merged 4 commits into
mainfrom
fix-issue4872
Jul 29, 2026
Merged

Reuse bounded sessions for parallel batch queries#4972
Widthdom merged 4 commits into
mainfrom
fix-issue4872

Conversation

@Widthdom

Copy link
Copy Markdown
Owner

Summary

  • Reuse at most one isolated query-only SQLite session per active cdidx batch --parallel worker slot instead of reopening and reproving the database for every item.
  • Preserve input-order envelopes, cancellation, failure isolation, output/input bounds, and the configured worker limit without sharing a reader across threads.
  • Verify source SQLite header/WAL generation and file identity between items for both detached snapshots and direct read sessions; replace a session whenever freshness cannot be proved.
  • Dispose detached snapshots on validation or reader-construction failures.
  • Add bilingual developer/testing documentation and changelog.d/unreleased/4872.fixed.md.

Root cause and performance evidence

Profiling used a dedicated approximately 717 MB CodeIndex database and the locally built Debug cdidx.dll.

Phase / workload Serial --parallel 4 Evidence
Baseline, 12 lightweight mixed reads 28.77 s 62.13 s Parallel was 2.16x slower.
Empty parallel validation/setup 2.21 s Isolated fixed setup cost.
12 parse/serialize/order failures 3.05 s Coordination and envelope work was comparatively small.
12 rejected commands that still forced per-item DB context/schema setup 0.60 s 27.89 s Repeated open/schema work accounted for about 45% of the valid parallel run.
Warm run after bounded session reuse 17.78 s 14.76 s Parallel/serial ratio improved to 0.83.

A heavier mixed workload also remained faster in parallel (144.84 s serial versus 130.84 s parallel). The deterministic regression guard uses warmup and a generous ratio (12-item / 3-item <= 3.0) rather than an absolute duration.

Validation

  • dotnet build tests/CodeIndex.Tests/CodeIndex.Tests.csproj --no-restore -p:UseSharedCompilation=false — passed with 0 warnings and 0 errors after syncing current origin/main.
  • Issue Avoid overhead that makes parallel batch slower than sequential #4872 focused tests — 6/6 passed on both net8.0 and net9.0 after review fixes and main sync.
  • All RunBatch_* contract tests — 41/41 passed on both net8.0 and net9.0 after review fixes.
  • Existing query-only snapshot regressions (Issue4557) — 13/13 passed on both net8.0 and net9.0.
  • Full Release suite run during implementation:
    • net8.0: 10,809 passed, 7 skipped, 0 failed (10,816 total; 48m 7s).
    • net9.0: 10,335 passed, 420 skipped, 0 failed (10,755 total; 53m 39s).
  • dotnet run --project tools/CodeIndex.Changelog -- check — validated 48 fragments after main sync.
  • dotnet format CodeIndex.sln --no-restore --verify-no-changes — passed.
  • git diff --check origin/main..HEAD — passed.

Adversarial review

Two mandatory codex review --base origin/main rounds were completed (the workflow maximum). The first round identified detached-snapshot staleness and reader-construction cleanup; the second identified checkpointed/direct-session freshness proof and initial-validation cleanup. All findings were addressed with signal-driven regressions for hot/checkpointed WAL, direct journal mode, and failure cleanup.

Documentation and changelog

  • Updated the English and Japanese sections of DEVELOPER_GUIDE.md and TESTING_GUIDE.md.
  • Added changelog.d/unreleased/4872.fixed.md.

Follow-up candidates

None.

Fixes #4872

@Widthdom
Widthdom marked this pull request as ready for review July 29, 2026 07:04
@Widthdom
Widthdom merged commit ee4117e into main Jul 29, 2026
12 checks passed
@Widthdom
Widthdom deleted the fix-issue4872 branch July 29, 2026 07:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Avoid overhead that makes parallel batch slower than sequential

1 participant