feat: pool worker threads in full indexing (avoid 1 Worker per file)

[kapral18]

Context

Incremental indexing already uses a reusable worker pool (src/commands/incremental_index_command.ts) to avoid per-file worker startup/teardown overhead.

Full indexing (src/commands/full_index_producer.ts) still spawns one Worker per file, which can be a major perf and stability hit on large repos.

Historical context / likely rationale

I searched the repo's merged PR history for explicit rationale around producer worker lifecycle (worker pools vs per-file workers) and did not find a PR/issue that directly explains this choice.

The closest related production-history signals are about consumer-side overload when Elasticsearch is slow/timeouting:

• Backpressure to avoid OOM / unlimited dequeuing: Apply backpressure to indexer worker to prevent OOM errors #21
• Follow-up hardening with detailed prod symptoms when ES is slow (timeout death spirals): fix: resolve critical IndexerWorker backpressure bug and migrate to Vitest #113

Those fixes are about not overwhelming memory when ES is slow. They don’t directly require per-file producer workers; producer concurrency is already bounded by p-queue.

Plausible reasons for per-file workers (no historical ticket found):

• Simplicity: spawn a worker, parse one file, terminate.
• Defensive memory reset: if tree-sitter/native parsing accumulates memory over time, terminating per file forces a reset of native allocations.

Why this matters

• Worker startup/teardown overhead dominates when indexing large codebases.
• High worker churn increases memory pressure and can trigger OS limits / slowdowns.

Where in code

• src/commands/full_index_producer.ts: creates a new Worker inside the per-file loop.

Suggested fix

Pool worker threads for full indexing while keeping safety under load:

• Create N workers upfront (N = min(CPU_CORES, configured pool size, file count)).
• Maintain an idle-worker queue and assign jobs.
• Ensure event listeners are cleaned up per job.
• Terminate workers only once after queue drain.

Optional safety guardrail (to preserve the likely “memory reset” benefit of per-file workers):

• Add a worker recycle policy (terminate/recreate a worker after N files or after a memory threshold).

Config knob:

• If PR fix: stabilize queue and move file locations to _locations index #135 lands, reuse PRODUCER_WORKER_POOL_SIZE and keep the same clamp-to-CPU behavior.
• If fix: stabilize queue and move file locations to _locations index #135 does not land, introduce an equivalent pool-size config (env var or CLI flag).

Test plan

• Unit test asserting Worker is instantiated at most poolSize times during full indexing for many files.