Evaluate optional CUDA acceleration for semantic search

[Acters]

Summary

The CPU embedding investigation in #407 found that batching policy dominates much of the current semantic-search indexing cost. CUDA remains a possible complementary acceleration path after CPU batching is corrected.

This issue asks the maintainer to decide whether optional CUDA support justifies its packaging, platform, configuration, and maintenance complexity. CPU should remain the universal default. Newer CUDA hardware may plausibly provide a 2-3x improvement for suitable batched workloads, but the local GTX 1660 Ti measurements below ranged from 1.2x to approximately 2x and did not demonstrate 3x.

Current Behavior

packages/semantic-search/src/embedder.ts currently uses:

dtype: "q8",
device: "cpu",

Transformers.js supports device: "cuda", but dreb currently receives the exact dependency onnxruntime-node@1.24.3, and the installed package does not contain CUDA provider libraries.

Benchmark Results

Hardware:

• Intel i7-9750H
• NVIDIA GTX 1660 Ti, 6 GiB
• CUDA 13.3
• cuDNN 9.24
• 1,024-document stratified corpus
• Warm model cache

CUDA proof-of-concept stack:

• onnxruntime-node@1.28.0-dev.20260624-ba45260eed
• ONNX Runtime 1.28.0 CUDA 13 provider libraries
• Provider libraries manually colocated with the development Node binding
• This is a feasibility test, not a supported installation

Policy	CPU q8	CUDA q8	CUDA FP16	CPU to CUDA FP16
Fixed1	16.16s	14.34s	13.36s	1.21x
Fixed32	51.06s	30.60s	29.34s	1.74x
Fixed128	52.66s	29.93s	29.67s	1.77x
Token-sort32	18.76s	12.82s	11.98s	1.57x
Dynamic sorted budget	18.98s	-	11.68s	1.62x

An earlier mixed-length workload measured approximately 2x for CUDA FP16. No 3x result was demonstrated locally; newer GPUs and workloads with enough parallel work may perform differently.

GPU Memory

Total GPU memory included a baseline of approximately 550 MiB:

Policy	Peak total GPU memory	Approximate benchmark increase
Fixed1 FP16	780 MiB	230 MiB
Dynamic sorted FP16	1,160 MiB	610 MiB
Fixed128 FP16	5,268 MiB	4,718 MiB

Large batches can approach the limit of a 6 GiB GPU.

Portability and Installation Complexity

CPU execution remains preferable as the universal default because it:

• works without vendor-specific hardware;
• requires no CUDA toolkit, driver, or cuDNN coordination;
• supports more operating systems and architectures;
• uses the normal Transformers.js dependency path;
• avoids downloading hundreds of MiB of provider binaries.

CUDA introduces several complications:

1. ONNX Runtime's prebuilt Node CUDA provider is primarily for Linux x64.
2. Older stable Node packages retrieve CUDA 12 provider artifacts.
3. ONNX Runtime 1.28 is the first Node publishing pipeline intended to consume CUDA 13 artifacts through Switch NPM publishing to consume from CUDA 13 pipeline microsoft/onnxruntime#28773.
4. onnxruntime-node@1.28.0 is not currently published on npm.
5. Transformers.js 4.0.1 pins onnxruntime-node exactly to 1.24.3, requiring a dependency override or upstream update.
6. The CUDA provider library alone is approximately 267 MiB.
7. Provider libraries are downloaded by an NPM post-install script, which can be blocked by modern script-approval policies.
8. The Node binding, common package, core runtime, and provider libraries must use compatible versions.
9. CUDA, driver, and cuDNN compatibility differs across distributions.
10. Library discovery may require runtime path configuration.
11. CUDA FP16 uses a different model artifact and changes embeddings, requiring index invalidation.
12. CPU-only users should not inherit CUDA dependencies, downloads, or failure modes.

Decision Requested

Possible outcomes:

1. Keep CPU-only support.

   • Lowest maintenance and widest compatibility.
   • Users can maintain local CUDA patches.

2. Expose expert device/dtype settings.

   • CPU/q8 remains default.
   • Advanced users supply a compatible CUDA runtime.
   • dreb validates the provider and reports actionable errors.

3. Provide a separately installable CUDA package or extension.

   • Isolates native dependencies from the normal installation.
   • Adds release and platform maintenance.

4. Wait for stable upstream packaging.

   • Revisit after onnxruntime-node@1.28.0 is published and Transformers.js supports it.

Acceptance Criteria if CUDA Is Accepted

• CPU/q8 remains the default.
• CUDA activation is explicit rather than silently automatic.
• Missing or incompatible providers produce a clear error or documented fallback.
• Device, dtype, model revision, and preprocessing profile participate in embedding-cache identity.
• Switching CPU/q8 to CUDA/FP16 triggers a safe index rebuild.
• Supported operating systems, architectures, CUDA versions, and drivers are documented.
• CPU-only installation does not download CUDA provider libraries.
• CUDA tests are opt-in and never required by normal CI.
• Performance claims identify hardware, runtime, dtype, batching policy, and corpus.
• The implementation remains separate from the batching work tracked in Reduce CPU semantic-search embedding overhead without degrading retrieval #407.

Suggested Labels

enhancement, question, P2