feat: add hybrid search for Chroma

[mattfreshwaters]

Description

Adds opt-in hybrid search (vector + BM25 keyword, fused via weighted Reciprocal Rank Fusion) to ChromaVectorDatabaseEngine, plus the supporting work to make Chroma usable on current versions: the v2 REST API and metadata filter support. Controlled by an SMSS flag — engines without it are unaffected, fully backwards compatible.

Requires Chroma v2 (the v1 API is removed in Chroma 1.x). Consolidates the intent of #1481 (v2 connection) and #179 (metadata filters) into one change.

Changes Made

• v2 API: create/add/remove/query now use the v2 tenant/database endpoints (/api/v2/tenants/{tenant}/databases/{db}/collections/...). New optional SMSS keys TENANT / DB_NAME (default default_tenant / default_database).
• Metadata filters: filters / metaFilters are translated to a Chroma where clause via the new ChromaVectorQueryFilterTranslationHelper — recursive $and/$or with $eq/$in/$ne/$nin/$gt/$gte/$lt/$lte.
• Hybrid search: new SMSS key USE_HYBRID_SEARCH=true (defaults false). Fetches a vector candidate pool, scores each candidate's stored content with BM25 (Lucene k1=1.2, b=0.75), and fuses vector + keyword ranks with weighted RRF (k=60). Tunable via HYBRID_VECTOR_WEIGHT (default 0.5) and HYBRID_KEYWORD_GATE_THRESHOLD (default 0.0); falls back to pure vector when the query matches no candidate text. Respects filters.
• nearestNeighborCall now returns Score + Distance (vector distances were previously discarded); all non-hybrid behavior is unchanged when the flag is off.

How to Test

1. Start Chroma v2

The SEMOSS engine connects on creation, so Chroma must be running before step 2:

docker run -d --name semoss-chroma -p 8000:8000 chromadb/chroma:1.0.0
curl http://localhost:8000/api/v2/version

Expect version "1.0.0".

2. Create a Chroma vector engine

Set these SMSS properties on a new vector engine, then ingest a small doc set through the catalog's document upload:

VECTOR_TYPE              CHROMA
HOSTNAME                 http://localhost:8000
CHROMA_COLLECTION_NAME   <collection name>
EMBEDDER_ENGINE_ID       <an existing embedder model engine>

TENANT / DB_NAME are optional (default to default_tenant / default_database).

3. Baseline query (vector-only)

Leave USE_HYBRID_SEARCH unset or false, then run:

VectorDatabaseQuery(engine=["<engineId>"], command=["diabetes blood sugar"], limit=[5]);

Each result includes Score and Distance.

4. Hybrid query

Add to the engine SMSS and reload the engine:

USE_HYBRID_SEARCH   true

Optionally tune HYBRID_VECTOR_WEIGHT (default 0.5) and HYBRID_KEYWORD_GATE_THRESHOLD (default 0.0). Re-run the same query:

VectorDatabaseQuery(engine=["<engineId>"], command=["diabetes blood sugar"], limit=[5]);

Results are re-ranked by weighted RRF (vector + BM25 keyword); a query with a strong keyword match surfaces documents pure vector search ranked lower.

5. (Optional) Metadata filters

Pass filters / metaFilters on the query and confirm results are restricted to matching metadata (supports $and/$or with $eq/$in/$ne/$nin/$gt/$gte/$lt/$lte).

Notes

• Hybrid is strictly opt-in; unset/false leaves the existing vector-only behavior untouched.
• HYBRID_VECTOR_WEIGHT sets the vector weight in RRF; keyword weight is 1 - HYBRID_VECTOR_WEIGHT.
• Falls back to pure vector when the query matches no candidate text.

[snyk-io]

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.