Phase 3: Search API — KNN, RRF, field projection, groupBy, read levels

.planning/REQUIREMENTS.md

@@ -20,10 +20,10 @@ Requirements for the current milestone. Each maps to roadmap phases.
 
 ### Search API
 
-- [ ] **SEARCH-01**: User can execute `collection.search()` with KNN ranking (queryText, queryVector, querySparseVector) and get typed `SearchResult`.
-- [ ] **SEARCH-02**: User can compose RRF (Reciprocal Rank Fusion) from multiple weighted rank expressions with arithmetic combinators.
-- [ ] **SEARCH-03**: User can project specific fields (`#id`, `#document`, `#embedding`, `#score`, `#metadata`, custom keys) in search results.
-- [ ] **SEARCH-04**: User can group search results by metadata key with min/max K controls, and specify read level (INDEX_AND_WAL vs INDEX_ONLY).
+- [x] **SEARCH-01**: User can execute `collection.search()` with KNN ranking (queryText, queryVector, querySparseVector) and get typed `SearchResult`.
+- [x] **SEARCH-02**: User can compose RRF (Reciprocal Rank Fusion) from multiple weighted rank expressions with arithmetic combinators.
+- [x] **SEARCH-03**: User can project specific fields (`#id`, `#document`, `#embedding`, `#score`, `#metadata`, custom keys) in search results.
+- [x] **SEARCH-04**: User can group search results by metadata key with min/max K controls, and specify read level (INDEX_AND_WAL vs INDEX_ONLY).
 
 ### Embedding Ecosystem
 
@@ -36,8 +36,8 @@ Requirements for the current milestone. Each maps to roadmap phases.
 ### Cloud Integration Testing
 
 - [ ] **CLOUD-01**: Cloud search parity tests cover pagination, IDIn/IDNotIn, document filters, metadata projection, and combined filter scenarios.
-- [ ] **CLOUD-02**: Cloud schema/index tests cover distance space variants, HNSW/SPANN config paths, invalid transitions, and schema round-trip assertions.
-- [ ] **CLOUD-03**: Cloud array metadata tests cover string/number/bool arrays, round-trip retrieval, and contains/not_contains filter behavior.
+- [x] **CLOUD-02**: Cloud schema/index tests cover distance space variants, HNSW/SPANN config paths, invalid transitions, and schema round-trip assertions.
+- [x] **CLOUD-03**: Cloud array metadata tests cover string/number/bool arrays, round-trip retrieval, and contains/not_contains filter behavior.
 
 ## Future Milestones (Post-0.3.0)
 
@@ -62,18 +62,18 @@ Deferred to future milestones.
 | COLL-01 | Phase 2 | Complete |
 | COLL-02 | Phase 2 | Complete |
 | COLL-03 | Phase 2 | Complete |
-| SEARCH-01 | Phase 3 | Pending |
-| SEARCH-02 | Phase 3 | Pending |
-| SEARCH-03 | Phase 3 | Pending |
-| SEARCH-04 | Phase 3 | Pending |
+| SEARCH-01 | Phase 3 | Complete |
+| SEARCH-02 | Phase 3 | Complete |
+| SEARCH-03 | Phase 3 | Complete |
+| SEARCH-04 | Phase 3 | Complete |
 | EMB-05 | Phase 4 | Pending |
 | EMB-06 | Phase 4 | Pending |
 | EMB-07 | Phase 4 | Pending |
 | EMB-08 | Phase 4 | Pending |
 | RERANK-01 | Phase 4 | Pending |
 | CLOUD-01 | Phase 5 | Pending |
-| CLOUD-02 | Phase 5 | Pending |
-| CLOUD-03 | Phase 5 | Pending |
+| CLOUD-02 | Phase 5 | Complete |
+| CLOUD-03 | Phase 5 | Complete |
 
 **Coverage:**
 - 0.3.0 requirements: 17 total

.planning/ROADMAP.md

@@ -14,7 +14,7 @@ Decimal phases appear between their surrounding integers in numeric order.
 
 - [ ] **Phase 1: Result Ergonomics & WhereDocument** — Add row-based result access and complete WhereDocument typed helpers.
 - [x] **Phase 2: Collection API Extensions** — Add Collection.fork, Collection.indexingStatus, and cloud feature parity audit.
-- [ ] **Phase 3: Search API** — Implement the Search endpoint with ranking expressions, field projection, groupBy, and read levels.
+- [x] **Phase 3: Search API** — Implement the Search endpoint with ranking expressions, field projection, groupBy, and read levels. (completed 2026-03-22)
 - [ ] **Phase 4: Embedding Ecosystem** — Add sparse/multimodal interfaces, reranking, new providers, and embedding registry.
 - [ ] **Phase 5: Cloud Integration Testing** — Build cloud parity test suites for search, schema/index, and array metadata.
 
@@ -63,7 +63,12 @@ Plans:
   4. User can group results by metadata key with min/max K controls.
   5. User can specify read level (INDEX_AND_WAL vs INDEX_ONLY).
   6. Integration tests validate search against Chroma >= 1.5.
-**Plans:** TBD
+**Plans:** 3/3 plans complete
+
+Plans:
+- [x] 03-01-PLAN.md — Create Search API value types, ranking builders, result interfaces, and SearchBuilder on Collection
+- [x] 03-02-PLAN.md — Implement Search DTOs, HTTP wiring, result converters, and SearchBuilderImpl
+- [x] 03-03-PLAN.md — Create unit tests, integration tests, and update PublicInterfaceCompatibilityTest
 
 ### Phase 4: Embedding Ecosystem
 **Goal:** Expand the embedding ecosystem with sparse/multimodal interfaces, reranking functions, additional providers, and an auto-wiring registry.
@@ -88,7 +93,11 @@ Plans:
   2. Cloud schema/index tests cover distance space variants, HNSW/SPANN config, invalid transitions, round-trip assertions.
   3. Cloud array metadata tests cover string/number/bool arrays, round-trip retrieval, contains/not_contains filters.
   4. Test suite can run in CI with cloud credentials or be skipped gracefully without them.
-**Plans:** TBD
+**Plans:** 1/2 plans executed
+
+Plans:
+- [x] 05-01-PLAN.md — Schema/index + array metadata cloud tests, mixed-type array client validation
+- [ ] 05-02-PLAN.md — Search parity cloud tests (KNN, RRF, GroupBy, batch, pagination, filters, projection, read levels)
 
 ## Progress
 
@@ -100,6 +109,6 @@ Phase 4 can execute in parallel with Phases 1-3 (independent).
 |-------|----------------|--------|-----------|
 | 1. Result Ergonomics & WhereDocument | 2/3 | In Progress|  |
 | 2. Collection API Extensions | 2/2 | Complete | 2026-03-21 |
-| 3. Search API | 0/TBD | Pending | — |
+| 3. Search API | 3/3 | Complete   | 2026-03-22 |
 | 4. Embedding Ecosystem | 0/TBD | Pending | — |
-| 5. Cloud Integration Testing | 0/TBD | Pending | — |
+| 5. Cloud Integration Testing | 1/2 | In Progress|  |

.planning/STATE.md

@@ -2,14 +2,14 @@
 gsd_state_version: 1.0
 milestone: v1.5
 milestone_name: milestone
-status: unknown
-stopped_at: Completed 02-collection-api-extensions-02-02-PLAN.md
-last_updated: "2026-03-21T13:50:18.109Z"
+status: "Phase 03 shipped — PR #139"
+stopped_at: Completed 03-search-api-03-03-PLAN.md
+last_updated: "2026-03-23T08:38:51.785Z"
 progress:
   total_phases: 10
-  completed_phases: 7
-  total_plans: 18
-  completed_plans: 18
+  completed_phases: 8
+  total_plans: 23
+  completed_plans: 22
 ---
 
 # Project State
@@ -19,12 +19,12 @@ progress:
 See: .planning/PROJECT.md (updated 2026-03-17)
 
 **Core value:** Java developers can integrate Chroma quickly and safely with a predictable, strongly-typed client that behaves consistently across environments.
-**Current focus:** Phase 03 — Search API (Phase 02 Collection API Extensions complete)
+**Current focus:** Phase 05 — cloud-integration-testing
 
 ## Current Position
 
-Phase: 03
-Plan: Not started
+Phase: 05 (cloud-integration-testing) — EXECUTING
+Plan: 2 of 2
 
 ## Performance Metrics
 
@@ -64,6 +64,10 @@ Plan: Not started
 | Phase 01-result-ergonomics-wheredocument P02 | 2 | 2 tasks | 6 files |
 | Phase 02-collection-api-extensions P01 | 3 | 2 tasks | 7 files |
 | Phase 02-collection-api-extensions P02 | 4 | 2 tasks | 6 files |
+| Phase 05-cloud-integration-testing P01 | 4 | 4 tasks | 3 files |
+| Phase 03-search-api P01 | 4 | 2 tasks | 12 files |
+| Phase 03-search-api P02 | 3min | 2 tasks | 6 files |
+| Phase 03-search-api P03 | 90 | 2 tasks | 7 files |
 
 ## Accumulated Context
 
@@ -122,6 +126,17 @@ Recent decisions affecting current work:
 - [Phase 02-collection-api-extensions]: IndexingStatus uses long fields (not int) for op counts matching Chroma API spec; no convenience isComplete() per D-11
 - [Phase 02-collection-api-extensions]: TestContainers tests catch both ChromaNotFoundException and ChromaServerException for skip-on-unavailable — self-hosted returns 5xx for fork/indexingStatus not 404
 - [Phase 02-collection-api-extensions]: Cloud fork test gated by CHROMA_RUN_FORK_TESTS=true to avoid per-call cloud cost in CI
+- [Phase 05-cloud-integration-testing]: validateMetadataArrayTypes uses ChromaBadRequestException with typed errorCode strings (MIXED_TYPE_ARRAY, NULL_ARRAY_ELEMENT); Integer/Long normalized to Integer group, Float/Double to Float group for homogeneity
+- [Phase 05-cloud-integration-testing]: Behavioral wiring tests for metadata validation use ChromaHttpCollection.from() with stub ChromaApiClient at localhost:1 — validation fires before network call
+- [Phase 03-search-api]: Knn uses factory+fluent-chain (not inner Builder) because type is factory-discriminated by query type; single required parameter makes builder pattern overkill
+- [Phase 03-search-api]: Rrf.Builder auto-calls knn.withReturnRank() on rank() to prevent returnRank=false pitfall in RRF sub-rankings
+- [Phase 03-search-api]: SearchResult.getScores() uses List<List<Double>> (not Float) to match wire format precision
+- [Phase 03-search-api]: SearchBuilderImpl in ChromaHttpCollection is stub throwing UnsupportedOperationException; full wiring in Plan 02
+- [Phase 03-search-api]: SearchRequest.searches is List<Map<String,Object>> for polymorphic rank serialization (knn vs rrf)
+- [Phase 03-search-api]: 'filter' key used (not 'where') in buildSearchItemMap per Search API wire format spec
+- [Phase 03-search-api]: SearchResultImpl stores Double scores internally, downcasts to Float on row access per SearchResultRow contract
+- [Phase 03-search-api]: RRF and text queryText skipped via Assume in integration tests — server returns 'unknown variant' for $rrf and rejects string values in $knn.query; tests document intended contract
+- [Phase 03-search-api]: Wire format keys corrected to '$knn'/'$rrf' (dollar-prefixed) — bare 'knn'/'rrf' keys rejected by Chroma server
 
 ### Roadmap Evolution
 
@@ -137,6 +152,6 @@ None.
 
 ## Session Continuity
 
-Last session: 2026-03-21T13:44:30.107Z
-Stopped at: Completed 02-collection-api-extensions-02-02-PLAN.md
+Last session: 2026-03-22T18:35:36.178Z
+Stopped at: Completed 03-search-api-03-03-PLAN.md
 Resume file: None

.planning/phases/02-collection-api-extensions/02-CONTEXT.md

@@ -0,0 +1,157 @@
+# Phase 2: Collection API Extensions - Context
+
+**Gathered:** 2026-03-21
+**Status:** Ready for planning
+
+<domain>
+## Phase Boundary
+
+Add cloud-relevant collection operations (fork, fork count, indexing status) to the v2 Collection interface and audit cloud feature parity for all v2 operations. No new embedding, search, or record operation work — this phase extends the collection-level API surface only.
+
+</domain>
+
+<decisions>
+## Implementation Decisions
+
+### fork() API surface
+- **D-01:** `Collection fork(String newName)` — single parameter, returns new Collection reference.
+- **D-02:** No options/builder overload — the Chroma server only accepts `new_name`, no metadata or config overrides.
+- **D-03:** Fork always creates the new collection in the same tenant/database as the source (no cross-tenant/database targeting).
+- **D-04:** Server errors propagate naturally — no client-side cloud guard. Self-hosted will return 404, which maps through the existing exception hierarchy. Future-proof if Chroma adds fork to self-hosted.
+- **D-05:** The forked collection inherits the source's embedding function reference (same pattern as Go client).
+
+### forkCount() API surface
+- **D-06:** `int forkCount()` — bare noun, returns the number of forks for this collection.
+- **D-07:** Added to Phase 2 scope (not in original requirements). Present in Python/Rust/JS clients, missing from Go client — Java gets parity with Python/Rust/JS here.
+- **D-08:** Endpoint: `GET .../collections/{id}/fork_count` → `{"count": N}`.
+
+### indexingStatus() API surface
+- **D-09:** `IndexingStatus indexingStatus()` — bare noun on Collection, consistent with `fork()`, `forkCount()`, `count()`.
+- **D-10:** `IndexingStatus` is an immutable value object with JavaBean getters:
+  - `long getNumIndexedOps()` — operations compacted into the index
+  - `long getNumUnindexedOps()` — operations still in the WAL
+  - `long getTotalOps()` — num_indexed + num_unindexed
+  - `double getOpIndexingProgress()` — 0.0 to 1.0
+- **D-11:** Raw fields only — no convenience methods (e.g., no `isComplete()`). Matches Go client.
+- **D-12:** Cloud-only, same server-error-propagation strategy as fork (D-04).
+
+### Naming conventions
+- **D-13:** Bare noun method names for all new operations: `fork()`, `forkCount()`, `indexingStatus()` — consistent with existing `count()`, `add()`, `query()`.
+- **D-14:** Javadoc on each cloud-only method uses `<strong>Availability:</strong>` tag documenting cloud-only status and expected self-hosted error behavior.
+
+### Testing strategy
+- **D-15:** Two-layer testing, aligned with chroma-go:
+  - **Unit tests** with mock HTTP server (canned JSON responses) — deterministic, runs in CI.
+  - **Cloud integration tests** against real Chroma Cloud — gated by credentials from `.env`.
+- **D-16:** Fork cloud tests skip in CI (forking is expensive at $0.03/call). Indexing status cloud tests can run in CI.
+- **D-17:** TestContainers integration tests that call fork/indexingStatus against self-hosted — currently skip (404), auto-activate if Chroma adds self-hosted support later.
+
+### Cloud parity audit
+- **D-18:** Cloud integration tests prove parity — if tests pass, parity is confirmed.
+- **D-19:** Javadoc on every v2 Collection and Client method with `<strong>Availability:</strong>` tag (cloud-only vs self-hosted + cloud).
+- **D-20:** README.md gets a "Cloud vs Self-Hosted" section with a comprehensive parity table covering ALL v2 operations, not just Phase 2 additions.
+- **D-21:** CHANGELOG entry documents new operations and their cloud-only status.
+
+### Claude's Discretion
+- Mock HTTP server implementation choice (OkHttp MockWebServer, httptest equivalent, or lightweight stub)
+- DTO class naming for fork/indexing requests and responses in `ChromaDtos.java`
+- `IndexingStatus` implementation details (equals/hashCode/toString)
+- Exact README parity table layout and column structure
+- How cloud test credentials are loaded (`.env` file, env vars, or both)
+- Whether `forkCount()` gets its own DTO or reuses a simple int extraction
+
+</decisions>
+
+<specifics>
+## Specific Ideas
+
+- Align with chroma-go's `Fork(ctx, newName) (Collection, error)` and `IndexingStatus(ctx) (*IndexingStatus, error)` — Java drops ctx (no context.Context in Java 8) but keeps the same signatures.
+- Go client testing uses `httptest.NewServer` with regex URL matching and hardcoded JSON — Java equivalent is OkHttp MockWebServer or similar lightweight approach.
+- Fork is copy-on-write on the server (shared data blocks, instant regardless of size) — this is useful context for Javadoc.
+- Fork has a 256 fork-edge limit per tree. Exceeding triggers a quota error. This should be noted in Javadoc.
+- `forkCount()` is ahead of Go client (which doesn't have it) — differentiator alongside comprehensive parity table.
+
+</specifics>
+
+<canonical_refs>
+## Canonical References
+
+**Downstream agents MUST read these before planning or implementing.**
+
+### Collection interface & implementation
+- `src/main/java/tech/amikos/chromadb/v2/Collection.java` — Current Collection interface, add fork/forkCount/indexingStatus here
+- `src/main/java/tech/amikos/chromadb/v2/ChromaHttpCollection.java` — HTTP implementation, implement new methods here
+- `src/main/java/tech/amikos/chromadb/v2/ChromaApiPaths.java` — Endpoint path builders, add fork/forkCount/indexingStatus paths
+- `src/main/java/tech/amikos/chromadb/v2/ChromaDtos.java` — Request/response DTOs, add fork request and indexing status response
+- `src/main/java/tech/amikos/chromadb/v2/ChromaApiClient.java` — HTTP transport (get/post/put/delete methods)
+
+### Client & session context
+- `src/main/java/tech/amikos/chromadb/v2/ChromaClient.java` — Client implementation, reference for how Collection instances are created and cached
+
+### Existing value objects (patterns to follow)
+- `src/main/java/tech/amikos/chromadb/v2/Tenant.java` — Immutable value object pattern (getName(), equals/hashCode)
+- `src/main/java/tech/amikos/chromadb/v2/Database.java` — Immutable value object pattern
+- `src/main/java/tech/amikos/chromadb/v2/CollectionConfiguration.java` — Complex immutable value object with builder
+
+### Exception hierarchy
+- `src/main/java/tech/amikos/chromadb/v2/ChromaException.java` — Base exception
+- `src/main/java/tech/amikos/chromadb/v2/ChromaExceptions.java` — Factory: `fromHttpResponse(statusCode, message, errorCode)`
+
+### Testing infrastructure
+- `src/test/java/tech/amikos/chromadb/v2/AbstractChromaIntegrationTest.java` — TestContainers base with `assumeMinVersion()`
+- `src/test/java/tech/amikos/chromadb/v2/CloudParityIntegrationTest.java` — Cloud test base with credential gating
+- `src/test/java/tech/amikos/chromadb/v2/RecordOperationsIntegrationTest.java` — Integration test patterns
+
+### External references
+- Chroma fork API: `POST /api/v2/tenants/{t}/databases/{d}/collections/{id}/fork` — body: `{"new_name": "..."}`
+- Chroma fork_count API: `GET /api/v2/tenants/{t}/databases/{d}/collections/{id}/fork_count` — response: `{"count": N}`
+- Chroma indexing_status API: `GET /api/v2/tenants/{t}/databases/{d}/collections/{id}/indexing_status` — response: `{"num_indexed_ops":N, "num_unindexed_ops":N, "total_ops":N, "op_indexing_progress":F}`
+- chroma-go Collection interface: `pkg/api/v2/collection.go` — Fork and IndexingStatus signatures
+- chroma-go HTTP impl: `pkg/api/v2/collection_http.go` — Fork and IndexingStatus implementations
+- chroma-go unit tests: `pkg/api/v2/collection_http_test.go` — Mock server testing pattern
+- chroma-go cloud tests: `pkg/api/v2/client_cloud_test.go` — Cloud integration testing pattern
+
+</canonical_refs>
+
+<code_context>
+## Existing Code Insights
+
+### Reusable Assets
+- `ChromaHttpCollection.modifyName(String)`: Direct HTTP call pattern (validate → build path → apiClient.put → update local state) — blueprint for fork()
+- `ChromaHttpCollection.count()`: Simple GET returning a primitive — blueprint for forkCount()
+- `ChromaHttpCollection.from(CollectionResponse, ...)`: Static factory for wrapping server response as Collection — reuse for fork() return value
+- `Tenant` / `Database`: Immutable value objects with equals/hashCode — pattern for IndexingStatus
+
+### Established Patterns
+- **Interface-first**: Public interface on `Collection`, package-private `ChromaHttpCollection` implementation
+- **Immutability**: Private constructor, factory method, defensive copies, unmodifiable collections
+- **JavaBean getters**: `getName()`, `getId()`, `getMetadata()` — follow for IndexingStatus
+- **Path builders**: Static methods on `ChromaApiPaths` — add `collectionFork()`, `collectionForkCount()`, `collectionIndexingStatus()`
+- **DTO inner classes**: All in `ChromaDtos` as static inner classes with Gson annotations
+
+### Integration Points
+- `Collection` interface: Add `fork(String)`, `forkCount()`, `indexingStatus()` method signatures
+- `ChromaHttpCollection`: Implement the three new methods
+- `ChromaApiPaths`: Add three new endpoint path builders
+- `ChromaDtos`: Add `ForkCollectionRequest`, `ForkCountResponse`, `IndexingStatusResponse`
+- `IndexingStatus`: New public immutable value object in `tech.amikos.chromadb.v2`
+- `README.md`: Add cloud vs self-hosted parity table
+- `CHANGELOG.md`: Document new operations
+
+</code_context>
+
+<deferred>
+## Deferred Ideas
+
+- Cross-tenant/cross-database fork targeting — not supported by Chroma server, revisit if server adds it
+- `IndexingStatus.isComplete()` convenience method — users can check `getOpIndexingProgress() >= 1.0` themselves
+- Polling helper for indexing status (e.g., `awaitIndexing(Duration timeout)`) — application-level concern, not client library
+- Fork with metadata/config overrides — not supported by Chroma server
+- Fork quota management APIs — depends on Chroma server adding quota introspection endpoints
+
+</deferred>
+
+---
+
+*Phase: 02-collection-api-extensions*
+*Context gathered: 2026-03-21*