feat(server,cli): bulk document ingestion across gRPC, HTTP, MCP, and CLI (#865)#870
Open
mosuka wants to merge 1 commit into
Open
feat(server,cli): bulk document ingestion across gRPC, HTTP, MCP, and CLI (#865)#870mosuka wants to merge 1 commit into
mosuka wants to merge 1 commit into
Conversation
… CLI (#865) Expose the Engine::put_documents / add_documents batch API (#551) on every server-side surface: gRPC: DocumentService.PutDocuments / AddDocuments take repeated DocumentEntry { id, document } and return the applied count - unary, mirroring the SearchBatch precedent. Entries without a document are rejected up front naming the offending position. LaurusError::BatchIngest is now classified in convert/error.rs by its source (caller mistakes surface as INVALID_ARGUMENT, storage failures as INTERNAL) while the status message keeps the full batch context (failed_index, failed_id, applied). HTTP gateway: POST /v1/documents:bulk?mode=put|add (default put) takes {"documents": [{"id", "document"}, ...]} and responds {"applied": N}; a router smoke test pins axum 0.8 accepting the literal-colon path. MCP: put_documents / add_documents tools taking a documents array. CLI: laurus put docs / add docs stream a JSONL file (--file, --batch-size, --commit-every) through the batch API. Each line is {"id": ..., "document": {"fields": ...}} - the same document JSON shape as --data. Unlike the singular commands, bulk ingestion commits automatically; a mid-file failure names the offending line, commits the applied prefix, and re-running the remaining lines is idempotent. Also fixes the pre-existing CLI docs bug where the --data examples showed a flat JSON object that Document's serde shape never accepted; the corrected examples use {"fields": {name: {Type: value}}}. Closes #865. Refs #551
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Exposes the batch ingestion API (#551 PR-3, on top of the merged
Engine::put_documents/add_documents) across every server-side surface:DocumentService.PutDocuments/AddDocumentswithrepeated DocumentEntry { id, document }and anappliedcount response — unary, mirroring theSearchBatchprecedent (client-side chunking handles the 4 MB default cap; a streaming RPC remains a future non-breaking addition). Entries without a document are rejected up front with the offending position.LaurusError::BatchIngestis now mapped inconvert/error.rsby its source (schema violation at position k →INVALID_ARGUMENT, storage failure →INTERNAL), with the full batch context (failed_index,failed_id,applied) preserved in the status message.POST /v1/documents:bulk?mode=put|add(defaultput) taking{"documents": [{"id", "document"}, ...]}and responding{"applied": N}. A router-build smoke test pins that axum 0.8 accepts the literal-colon path.put_documents/add_documentstools taking adocumentsarray (same entry shape), registered in the server instructions list.laurus put docs/add docs— streaming JSONL bulk ingest (--file,--batch-sizedefault 1000,--commit-everydefault 0 = final-only). Each line is{"id": "...", "document": {"fields": {...}}}(the same document JSON shape as--data). Unlike the singular commands it commits automatically; on a mid-file failure the error names the offending line, the applied prefix is committed, and re-running the remaining lines (or the whole file, under put) is idempotent.--commit-everyprototypes the futureCommitPolicyshape client-side, per the perf(lexical/index): batch ingestion API (add_documents/commit_batch) #551 plan.Also fixed in passing
The CLI docs'
--dataexamples showed a flat JSON object ('{"title":"Hello World"}'), whichserde_json::from_str::<Document>has never accepted — the working shape is{"fields": {"title": {"Text": "..."}}}(verified with a probe; the flat form fails today). Both singular examples and the format description are corrected in en+ja alongside the new bulk sections.Verification
GetDocuments, chunk accumulation, empty batch →applied: 0, entry-without-document →INVALID_ARGUMENTnamingdocuments[1], andBatchIngestsource-based status classification (caller fault vs storage fault).--features laurus/embeddings-all) clean / markdownlint clean across all updated docs (en+ja: grpc_api, http_gateway, mcp tools, cli commands).Docs
docs/{src,ja/src}/laurus-server/{grpc_api.md, http_gateway.md},laurus-mcp/tools.md,laurus-cli/commands.md: new bulk sections with fail-fast/no-rollback semantics, plus the corrected singular examples.Closes #865. Refs #551.