Reduce CPU semantic-search embedding overhead without degrading retrieval

[Acters]

Summary

Reduce first-build semantic-search indexing time on CPU without assuming a particular batching algorithm or silently degrading retrieval quality. The current q8 embedding pipeline batches documents by count, but real code chunks vary widely in token length, so padding and dynamic quantization make both performance and output dependent on batch composition.

Current Behavior

Embedder.embedDocuments() processes sequential fixed-count batches. The upstream default is 32 documents, using Xenova/all-MiniLM-L6-v2 with dtype: "q8" and device: "cpu".

On a measured 14,576-chunk local corpus:

• Actual truncated input: 2.89 million tokens
• Fixed-32 padded input: 7.34 million tokens
• Estimated padding waste: 60.7%

A stratified 1,024-document CPU/q8 benchmark on an Intel i7-9750H produced:

Fixed batch size	Time	Throughput	Final RSS
1	15.9s	64.3 docs/s	310 MiB
2	27.3s	37.6 docs/s	376 MiB
4	35.5s	28.9 docs/s	533 MiB
8	44.9s	22.8 docs/s	777 MiB
16	52.8s	19.4 docs/s	1,211 MiB
32	55.7s	18.4 docs/s	2,089 MiB
64	54.4s	18.8 docs/s	4,006 MiB
128	49.8s	20.6 docs/s	7,620 MiB

Concurrent single-document calls at concurrency 1, 2, 8, and 32 remained within approximately 63-66 docs/s, suggesting that extra JavaScript concurrency does not improve this CPU workload. Progress reporting can be coalesced independently from the inference batch size.

Batch composition also changes q8 vectors. Repeating the same fixed-32 schedule was deterministic, but alternate schedules produced mean cosine near 0.993 versus fixed-32, minimum cosine near 0.98, and 17-19/20 overlap for one vector-only top-20 query. This establishes schedule-dependent output, not which schedule has better retrieval quality.

Proposed Behavior

Evaluate the CPU embedding policies against the same end-to-end workloads, then implement the smallest policy that improves indexing time and memory without degrading final search retrieval. Candidate policies include single-document inference with coalesced progress, length-aware fixed-count batches, and padded-token-budget batches; none is prescribed before integration and retrieval testing.

Acceptance Criteria

• Add a reproducible benchmark that includes tokenizer work, ONNX inference, production progress callbacks, and Embedder/IndexManager integration.
• Compare the current fixed-32 baseline with at least single-document inference and one length-aware policy on representative mixed-length corpora.
• Record runtime, progress cadence, and memory methodology; do not infer transient peak RSS from post-batch samples alone.
• Preserve the positional chunk-to-vector contract through IndexManager.embedChunks() for any policy that reorders work.
• Add integration coverage proving vectors are stored under the correct chunk IDs.
• Build a small, reviewable retrieval benchmark from mechanically grounded identifier, path, and natural-language queries; exclude disputed labels after independent review.
• Compare final search results using retrieval metrics such as Recall@5 and first-relevant rank, not vector cosine alone.
• Coalesce progress updates independently from inference calls if the selected policy uses small batches.
• Rebuild or version existing embeddings if the selected policy changes q8 document vectors.
• Keep GPU acceleration, model replacement, dtype changes, ranking redesign, and incremental persistence out of scope.

Technical Notes

Relevant files:

• packages/semantic-search/src/embedder.ts
• packages/semantic-search/src/index-manager.ts
• packages/semantic-search/src/search.ts
• packages/semantic-search/test/index-manager.test.ts
• packages/semantic-search/test/search-engine.test.ts

Exact token-length preprocessing took 13.4 seconds for the measured 14,576-document corpus and must be included when evaluating token-aware policies. Prototype scheduling tests covered empty input, oversized singletons, budget and count limits, stable ties, exactly-once scheduling, and output-order restoration, but production integration remains untested.

The original semantic-search design discussion is available at #108.

Suggested Labels

bug, P1

[Acters]

Issue Assessment

Summary

The issue has a demonstrated CPU/q8 performance problem: fixed-count batches mix widely different sequence lengths, amplify padding, increase memory, and change dynamically quantized document vectors based on batch composition. The requested outcome is to select a smaller, evidence-backed CPU policy without assuming that vector cosine alone proves retrieval quality.

Current State

• Embedder defaults to fixed batches of 32 using Xenova/all-MiniLM-L6-v2, q8, and CPU.
• Embedding progress is emitted only after each inference batch.
• IndexManager.embedChunks() pairs chunks and vectors positionally.
• Existing search/index tests mock embeddings, so they do not exercise real padding, q8 batch dependence, model initialization, dimensions, or retrieval quality.
• Embeddings are keyed only by model name. Dtype, pooling, normalization, preprocessing revision, and batching behavior are not represented in cache identity.
• There is no existing semantic-search benchmark or golden retrieval fixture.

Confirmed Evidence

On the reported host and corpus, fixed single-document inference materially outperformed fixed sizes 2-128 while using less observed memory. JavaScript concurrency from 1 to 32 simultaneous single-document calls did not materially improve throughput. This makes single-document inference with independently coalesced progress the leading candidate, but not a retrieval-quality conclusion.

The q8 model is deterministic for a repeated schedule but produces different vectors under different batch compositions. A one-query top-k comparison is evidence of changed inputs, not evidence that either schedule is more relevant.

Gaps and Ambiguities

1. The real-model benchmark must be opt-in or a separate command; ordinary CI should not download a model or depend on host timing.
2. The retrieval corpus, relevance grades, metric cutoffs, and re-baseline policy are not yet defined.
3. Exact token preprocessing adds measurable work and must be included for token-aware candidates.
4. Production progress/RPC overhead for single-document inference remains unmeasured; progress should be coalesced separately from inference calls.
5. Existing embedding invalidation cannot detect a changed q8 batching policy.
6. Any reordering policy must explicitly restore chunk/vector association. The current SQL retrieval path does not document an ordering guarantee.

Recommended Scope

The issue currently combines benchmark infrastructure, retrieval evaluation, production scheduling, progress behavior, cache invalidation, and integration correctness. That is larger than a normal focused implementation PR.

Use two focused stages:

1. Benchmark/evaluation: add an opt-in real-model benchmark, small mechanically grounded retrieval corpus, metrics, and fixed-32/fixed-1/one length-aware comparison through the production pipeline.
2. Implementation: open a focused follow-up issue selecting one policy from the recorded evidence, coalescing progress, preserving vector association, and handling existing-index invalidation.

Normal CI should test benchmark parsing/metrics and positional contracts with mocks. The real-model benchmark should run only by explicit command or environment opt-in.

Risks

• Choosing a policy from timing alone can trade speed for retrieval quality.
• Exact-order assertions will be brittle because hybrid ranking includes several signals; prefer file/symbol relevance with Recall@K and first-relevant rank.
• Timing assertions in CI will be hardware-sensitive.
• Updating batching without cache invalidation can mix old and new q8 vector distributions.
• A general policy framework would add unnecessary complexity; the implementation stage should select one measured policy.

Estimate

• Benchmark/evaluation stage: medium.
• Focused implementation after evidence: small-to-medium.
• Combined as one PR: large and unnecessarily difficult to review.

---

Automated assessment by mach6

[Acters]

Tested Batching Alternatives: Evidence and Trade-offs

No policy is selected by this comment. These measurements document the alternatives tested before planning an implementation.

Environment and limitations

• Intel i7-9750H, 6 cores / 12 threads
• Xenova/all-MiniLM-L6-v2
• ONNX Runtime Node 1.24.3
• CPU with q8 model
• Warm local model cache
• Two deterministic 1,024-document samples from a 14,576-chunk mixed-length corpus
• Direct Transformers/ONNX measurements; production IndexManager, database, RPC/MCP progress, and final hybrid retrieval still require an opt-in integration benchmark
• Most policy timings are single runs; small differences should not be treated as statistically significant

Fixed-size comparison

A lightweight harness over the same stratified 1,024-document sample produced:

Fixed size	Time	Throughput	Final RSS	Main observation
1	15.9s	64.3 docs/s	310 MiB	Fastest, no cross-document padding
2	27.3s	37.6 docs/s	376 MiB	Already loses substantial throughput
4	35.5s	28.9 docs/s	533 MiB	Padding begins to dominate
8	44.9s	22.8 docs/s	777 MiB	No compensating CPU utilization gain
16	52.8s	19.4 docs/s	1,211 MiB	Near fixed-32 throughput
32	55.7s	18.4 docs/s	2,089 MiB	Current upstream default
64	54.4s	18.8 docs/s	4,006 MiB	Similar speed, much more memory
128	49.8s	20.6 docs/s	7,620 MiB	Coarse progress and highest memory

Observed RSS is not a guaranteed transient high-water mark.

Scheduling-policy comparison

The same instrumented 1,024-document workload was used for all policies below:

Policy	Time	Padding waste	Observed RSS	Main trade-off
Fixed 32	60.9s	60.6%	2,164 MiB	Simple and regular progress, but heavy padding
Character-sort + fixed 32	25.0s	14.2%	2,072 MiB	No token prepass; approximate grouping and output remapping required
Token-sort + fixed 32	22.1s	3.8%	2,157 MiB	Low padding; exact token prepass, sorting, remapping, and uneven batch times
Capped token buckets + fixed 32	27.9s	19.5%	2,098 MiB	Can round-robin length classes; arbitrary boundaries and worse throughput than sorting
Dynamic budget, original order	54.6s	58.2%	1,244 MiB	More regular time cadence and lower memory, but does not solve mixed-length padding
Dynamic budget, token sorted	22.4s	5.1%	1,217 MiB	Fast and lower-memory; exact prepass, tuning, remapping, and irregular document-count progress

Per-batch instrumentation penalizes policies with many small batches. Separate lightweight and stratified runs confirmed the large fixed-32 versus fixed-1/token-aware difference, but exact sub-second rankings between the faster policies are not established.

Dynamic padded-token budget sensitivity

Sorted dynamic batches were tested across several budgets:

Budget	Time	Observed RSS	Progress behavior
512	21.3s	357 MiB	Many very small batches
1,024	22.5s	393 MiB	Frequent updates
2,048	21.9s	516 MiB	Frequent updates
4,096	22.5s	756 MiB	Moderate batch sizes
8,192	22.4s	1,217 MiB	8-128 documents per update
16,384	23.2s	2,178 MiB	Coarser updates
32,768	23.9s	4,745 MiB	Highest memory and longest intervals

No tested value was uniquely optimal. Budgets from 512 through 8,192 had similar CPU throughput, while memory increased with the budget.

Exact versus approximate length costs

• Exact token-length preprocessing over all 14,576 documents took 13.4 seconds.
• Exact token sorting reduced scheduled full-corpus padding waste below 1%.
• Character sorting avoided the second tokenizer pass but left approximately 19% full-corpus padding waste.
• Exact preprocessing must be included in end-to-end timing rather than treated as free.

Single-document concurrency

Single-document calls were tested at JavaScript concurrency 1, 2, 8, and 32. Repeated runs remained approximately 63-66 documents/second, with identical output checksums. Extra JavaScript concurrency did not materially improve this CPU workload, likely because ONNX already uses CPU threads internally or schedules the calls over the same resources.

Inference batch size and reporting frequency are independent. Single-document inference can collect or report progress every 32 completions without creating a 32-document model batch.

Retrieval and q8 batch dependence

The q8 model was exactly deterministic when the same fixed-32 schedule was repeated. Changing peers or scheduling changed document vectors:

• Mean cosine versus fixed-32: approximately 0.993
• Minimum observed cosine: approximately 0.98
• One vector-only query retained 17-19 of the fixed-32 top 20 results
• A targeted document embedded alone versus with different peers produced cosine near 0.989

This proves batch-composition dependence, not which policy retrieves better results. Fixed-1 is attractive because documents and queries are both embedded independently, but retrieval quality must be evaluated through the complete hybrid search pipeline before calling it superior.

Prototype implementation costs

Prototype scheduler sizes, excluding integration and tests:

• Fixed-count loop: 5 nonblank lines
• Stable length sort around fixed-count batches: 4 additional lines plus output remapping
• Capped round-robin buckets: 15 lines plus boundary policy
• Dynamic padded-token budget: 22 lines plus budget policy, token preprocessing, and output remapping

Twelve pure scheduler tests passed for exactly-once scheduling, stable ties, budget/count bounds, empty input, oversized singletons, and output restoration. They do not replace real Embedder/IndexManager integration tests.

Current conclusion

Fixed single-document inference is the leading CPU/q8 candidate because it is currently the fastest, simplest, lowest-memory option and removes peer-dependent batching. It is not yet the selected production solution. Required remaining gates are:

1. Repeated end-to-end production benchmarks including tokenizer, database, and progress consumers.
2. Progress coalescing usability tests.
3. Mechanically grounded retrieval evaluation through final POEM-ranked results.
4. Existing-index invalidation/versioning if the chosen policy changes stored q8 vectors.

[Acters]

Local fixed1 end-to-end indexing result

A fresh production-path semantic search was run after installing the local CPU/q8 batchSize: 1 patch and removing the incomplete prior index.

• Files: 1,743
• Chunks: 14,576
• Embeddings: 14,576
• Wall time: 268.821 seconds (4m 28.821s)
• Effective end-to-end throughput: 54.2 chunks/second

Compared with the fixed32 projection of approximately 792 seconds derived from the 1,024-document benchmark, this is:

• 523.2 seconds less wall time
• 2.95x faster
• 66.1% less time

The 66.1% wall-time reduction is close to the measured 60.6% fixed32 padding waste and is consistent with padding being a dominant cost. It does not establish a direct numerical correlation: the fixed32 value is projected from a smaller benchmark, while this fixed1 wall time also includes scanning, chunking, model setup, database writes, progress callbacks, and final ranking.

Fixed1 will remain the reporter's local default while upstream retrieval-quality and reproducibility work continues. This local decision is not evidence by itself to change the upstream default.

[m-aebrer]

interesting! Definitely going to dig into this later, thanks Acters!