Skip to content

api-reference.md

Latest commit

 

History

History
2550 lines (1955 loc) · 109 KB

api-reference.md

File metadata and controls

2550 lines (1955 loc) · 109 KB
Raw

API Reference

SwarnDB exposes two API surfaces: a REST API on port 8080 and a gRPC API on port 50051. This document covers every endpoint, every parameter, and every response field.

  • REST base URL: http://localhost:8080/api/v1
  • gRPC endpoint: localhost:50051
  • Proto definitions: proto/swarndb/v1/
  • Authentication: X-API-Key header or Authorization: Bearer <token> header (only enforced when SWARNDB_API_KEYS is set)
  • Error format: {"error": "message", "code": 400}

All REST endpoints return JSON. All request bodies are JSON (Content-Type: application/json).

Table of Contents

Health and Readiness

GET /health

Basic health check. Always returns 200 while the server is running.

Response:

Field Type Description
status string Always "ok"
version string Server version (e.g., "1.1.0") 
curl http://localhost:8080/health
{
  "status": "ok",
  "version": "1.1.0"
}

GET /ready

Readiness check. Returns collection count and total vector count.

Response:

Field Type Description
ready bool Always true when server is up
collections uint Number of loaded collections
total_vectors uint64 Total vectors across all collections 
curl http://localhost:8080/ready
{
  "ready": true,
  "collections": 3,
  "total_vectors": 150000
}

GET /healthz

Kubernetes liveness probe. Returns 200 as long as the process is alive.

Response:

Field Type Description
status string Always "alive" 
curl http://localhost:8080/healthz

GET /readyz

Kubernetes readiness probe. Returns 200 when all checks pass, 503 otherwise.

Response:

Field Type Description
status string "ready" or "not_ready"
checks object Map of check names to status strings

Checks performed:

  • collections_accessible: can acquire a read lock on the collections map
  • collections_loaded: at least one collection exists, or server just started
curl http://localhost:8080/readyz
{
  "status": "ready",
  "checks": {
    "collections_accessible": "ok",
    "collections_loaded": "ok"
  }
}

GET /startupz

Kubernetes startup probe. Returns 200 once server initialization is complete, 503 while still starting.

Response:

Field Type Description
status string "started" or "starting" 
curl http://localhost:8080/startupz

GET /metrics

Returns Prometheus-format metrics for monitoring. No authentication required.

Response: Plain text in Prometheus exposition format.

curl http://localhost:8080/metrics

GET /recovery_status

Returns the server's boot-recovery snapshot: the overall recovery path taken at startup, the elapsed time since recovery began, and the recovery path for each collection. Useful for orchestrators that need to gate traffic until recovery completes.

Response:

Field Type Description
path string Recovery path taken at boot: "none", "snapshot_only", "snapshot_plus_wal", or "full_wal_replay"
elapsed_secs uint64 Seconds since recovery began
collections object Map of collection name to its recovery path (same set of strings) 
curl http://localhost:8080/recovery_status
{
  "path": "snapshot_plus_wal",
  "elapsed_secs": 3,
  "collections": {
    "docs": "snapshot_plus_wal",
    "images": "snapshot_only"
  }
}

Collections

Create Collection

POST /api/v1/collections

Creates a new vector collection.

Request Body:

Parameter Type Required Default Description
name string Yes Collection name. Must be alphanumeric with underscores/hyphens.
dimension uint32 Yes Vector dimensionality. Must be greater than 0.
distance_metric string No "cosine" One of: "cosine", "euclidean", "dot_product", "manhattan"
default_threshold float No 0.0 Default similarity threshold for the virtual graph. 0 means no graph edges are auto-computed.
max_vectors uint64 No 0 Maximum number of vectors. 0 means unlimited.
mode string No "vector_only" Collection mode: "vector_only", "auto_similarity", or "hybrid". Omitting it defaults to vector-only. See Typed Graph: Overview.

Response:

Field Type Description
name string Name of the created collection
success bool true on success

Status Codes: 200 (success), 400 (invalid parameters), 409 (collection already exists), 500 (storage error)

curl -X POST http://localhost:8080/api/v1/collections \
  -H "Content-Type: application/json" \
  -d '{
    "name": "documents",
    "dimension": 1536,
    "distance_metric": "cosine",
    "default_threshold": 0.8
  }'
{
  "name": "documents",
  "success": true
}

List Collections

GET /api/v1/collections

Returns all collections and their metadata.

Response:

Field Type Description
collections array Array of collection info objects

Each collection object:

Field Type Description
name string Collection name
dimension uint32 Vector dimensionality
distance_metric string Distance metric in use
vector_count uint64 Number of vectors stored
default_threshold float Default similarity threshold
status string One of: "ready", "pending_optimization", "optimizing" 
curl http://localhost:8080/api/v1/collections
{
  "collections": [
    {
      "name": "documents",
      "dimension": 1536,
      "distance_metric": "cosine",
      "vector_count": 50000,
      "default_threshold": 0.8,
      "status": "ready"
    }
  ]
}

Get Collection

GET /api/v1/collections/{name}

Returns metadata for a single collection.

Path Parameters:

Parameter Type Description
name string Collection name

Response: Same fields as a single collection object in the list response.

Status Codes: 200 (success), 404 (not found)

curl http://localhost:8080/api/v1/collections/documents
{
  "name": "documents",
  "dimension": 1536,
  "distance_metric": "cosine",
  "vector_count": 50000,
  "default_threshold": 0.8,
  "status": "ready"
}

Delete Collection

DELETE /api/v1/collections/{name}

Permanently deletes a collection and all its vectors.

Path Parameters:

Parameter Type Description
name string Collection name

Response:

Field Type Description
success bool true on success

Status Codes: 200 (success), 404 (not found), 500 (storage error)

curl -X DELETE http://localhost:8080/api/v1/collections/documents
{
  "success": true
}

Vectors

Insert Vector

POST /api/v1/collections/{collection}/vectors

Inserts a single vector into the collection.

Path Parameters:

Parameter Type Description
collection string Collection name

Request Body:

Parameter Type Required Default Description
id uint64 No 0 Vector ID. 0 means auto-assign.
values float array Yes Vector values. Length must match collection dimension.
metadata object No null Key-value metadata (see Metadata Types).

Response:

Field Type Description
id uint64 Assigned or provided vector ID
success bool true on success

Status Codes: 200 (success), 400 (dimension mismatch, invalid data), 404 (collection not found), 409 (duplicate ID), 500 (storage error)

curl -X POST http://localhost:8080/api/v1/collections/documents/vectors \
  -H "Content-Type: application/json" \
  -d '{
    "id": 1,
    "values": [0.1, 0.2, 0.3],
    "metadata": {
      "title": "Introduction to Vector Databases",
      "category": "technology",
      "year": 2024
    }
  }'
{
  "id": 1,
  "success": true
}

Get Vector

GET /api/v1/collections/{collection}/vectors/{id}

Retrieves a vector by ID.

Path Parameters:

Parameter Type Description
collection string Collection name
id uint64 Vector ID

Response:

Field Type Description
id uint64 Vector ID
values float array Vector values
metadata object Metadata (omitted if none set)

Status Codes: 200 (success), 404 (collection or vector not found)

curl http://localhost:8080/api/v1/collections/documents/vectors/1
{
  "id": 1,
  "values": [0.1, 0.2, 0.3],
  "metadata": {
    "title": "Introduction to Vector Databases",
    "category": "technology",
    "year": 2024
  }
}

Update Vector

PUT /api/v1/collections/{collection}/vectors/{id}

Updates a vector's values, metadata, or both. At least one of values or metadata must be provided.

Path Parameters:

Parameter Type Description
collection string Collection name
id uint64 Vector ID

Request Body:

Parameter Type Required Description
values float array No New vector values. Must match dimension.
metadata object No New metadata. Replaces existing metadata entirely.

Response:

Field Type Description
success bool true on success

Status Codes: 200 (success), 400 (neither values nor metadata provided, dimension mismatch), 404 (not found), 500 (storage error)

curl -X PUT http://localhost:8080/api/v1/collections/documents/vectors/1 \
  -H "Content-Type: application/json" \
  -d '{
    "metadata": {
      "title": "Updated Title",
      "category": "tech",
      "year": 2025
    }
  }'
{
  "success": true
}

Delete Vector

DELETE /api/v1/collections/{collection}/vectors/{id}

Deletes a vector by ID.

Path Parameters:

Parameter Type Description
collection string Collection name
id uint64 Vector ID

Response:

Field Type Description
success bool true on success

Status Codes: 200 (success), 404 (not found), 500 (storage error)

curl -X DELETE http://localhost:8080/api/v1/collections/documents/vectors/1
{
  "success": true
}

Bulk Insert

POST /api/v1/collections/{collection}/vectors/bulk

Inserts multiple vectors in a single request with configurable performance options.

Path Parameters:

Parameter Type Description
collection string Collection name

Request Body:

Parameter Type Required Default Description
vectors array Yes Array of vector objects, each with id, values, and optional metadata.
batch_lock_size uint32 No 1 Number of vectors to lock and insert at once. Max: 10000.
defer_graph bool No false Skip graph computation during insert. Rebuild later via optimize().
wal_flush_every uint32 No 1 Flush WAL every N vectors. 0 disables WAL for this batch.
ef_construction uint32 No 0 Override HNSW ef_construction for this bulk insert. 0 uses collection default.
index_mode string No "immediate" "immediate" indexes vectors during insert. "deferred" indexes later via optimize().
skip_metadata_index bool No false Skip per-vector metadata indexing. Rebuild later via optimize().
parallel_build bool No false Use parallel HNSW construction. Only effective with index_mode: "deferred".
checkpoint_every uint32 No 0 Write a resume checkpoint every N batches. 0 disables checkpointing. When non-zero, a failed bulk insert returns a resume_token that lets a subsequent call pick up from the next batch.
resume_token string No "" Opaque token from a prior partial bulk insert response. When set, the server resumes ingestion from the next batch after the one referenced by the token; vectors from the already-committed prefix are not re-inserted.

Each vector object in the vectors array:

Field Type Required Description
id uint64 No Vector ID. 0 for auto-assign.
values float array Yes Vector values matching collection dimension.
metadata object No Key-value metadata.

Response:

Field Type Description
inserted_count uint64 Number of successfully inserted vectors
errors string array One error message per failed row, empty on full success
last_completed_batch_idx uint64 Index of the last batch fully committed before a partial failure (0 on full success or non-batched inserts)
last_committed_lsn uint64 LSN of the last committed write in this operation
resume_token string Opaque token for resuming this bulk insert from the next batch (empty on full success or non-resumable failures)
assigned_ids uint64 array IDs assigned to inserted vectors, in input order

Status Codes: 200 (success, possibly partial), 400 (invalid options), 404 (collection not found), 500 (storage error)

curl -X POST http://localhost:8080/api/v1/collections/documents/vectors/bulk \
  -H "Content-Type: application/json" \
  -d '{
    "vectors": [
      {"id": 1, "values": [0.1, 0.2, 0.3], "metadata": {"title": "Doc A"}},
      {"id": 2, "values": [0.4, 0.5, 0.6], "metadata": {"title": "Doc B"}},
      {"id": 3, "values": [0.7, 0.8, 0.9], "metadata": {"title": "Doc C"}}
    ],
    "batch_lock_size": 100,
    "defer_graph": true,
    "wal_flush_every": 500,
    "index_mode": "deferred",
    "parallel_build": true
  }'
{
  "inserted_count": 3,
  "errors": [],
  "last_completed_batch_idx": 0,
  "last_committed_lsn": 42,
  "resume_token": "",
  "assigned_ids": [1, 2, 3]
}

Bulk Insert From Path

POST /api/v1/collections/{collection}/vectors/bulk-from-path

Ingests vectors from a .npy or flat .f32 file on the server's local filesystem. The server reads the file via memory mapping, so the working memory for the load is bounded by the index being built rather than by the input file size. Suited to large loads where staging the data as a file is acceptable.

Path Parameters:

Parameter Type Description
collection string Collection name

Request Body:

Parameter Type Required Default Description
path string Yes Absolute path to the vector file on the server's filesystem. The server process must have permission to read it.
dim uint32 No 0 Vector dimensionality. 0 defers to the collection's configured dimension.
expected_count uint64 No 0 Expected number of vectors. 0 lets the server infer from file size.
total_count_hint uint64 No 0 Hint for the total vector count across multiple bulk inserts; used for arena capacity planning.
id_start uint64 No 1 Starting ID for auto-assigned IDs.
ids_path string No "" Optional path to a sidecar file containing explicit IDs (one per vector). Empty string uses auto-assigned IDs starting at id_start. The server process must have permission to read it.
skip_metadata_index bool No false Skip metadata indexing during insert. Rebuild later via optimize().
index_mode string No "immediate" "immediate" builds the index during insert. "deferred" builds it after, via optimize().
ef_construction uint32 No 0 HNSW ef_construction override for this batch. 0 uses the collection default.
chunk_size uint32 No 0 0 loads the file in a single pass. A positive value processes the load in chunks of that many rows, snapshotting and releasing scratch memory between chunks; trades wall-clock for a lower peak resident memory footprint.

Supported wire formats:

  • .npy: auto-detected via the NumPy magic bytes (\x93NUMPY).
  • Flat little-endian float32 (.f32): the file is expected_count * dim * 4 bytes.

Response: same fields as the bulk insert response (inserted_count, errors, last_completed_batch_idx, last_committed_lsn, resume_token, assigned_ids).

Status Codes: 200 (success, possibly partial), 400 (invalid options, dimension mismatch, file size mismatch), 403 (server denied the path), 404 (collection not found, file not found), 500 (storage error)

Resume mid-call is not supported for bulk-from-path; on failure, restart the full call. For resumable bulk loads, use streaming BulkInsertWithOptions with checkpoint_every and resume_token.

curl -X POST http://localhost:8080/api/v1/collections/docs/vectors/bulk-from-path \
  -H "Content-Type: application/json" \
  -d '{
    "path": "/data/ingest/embeddings.npy",
    "dim": 1536,
    "expected_count": 1000000,
    "total_count_hint": 1000000,
    "index_mode": "immediate"
  }'
{
  "inserted_count": 1000000,
  "errors": [],
  "last_completed_batch_idx": 0,
  "last_committed_lsn": 1000000,
  "resume_token": "",
  "assigned_ids": [1, 2, 3, "... 1000000 ids total"]
}

Optimize

POST /api/v1/collections/{collection}/optimize

Triggers index rebuilding, graph recomputation, and metadata re-indexing for a collection. Call this after bulk inserts with deferred options.

Path Parameters:

Parameter Type Description
collection string Collection name

Request Body: Empty ({})

Response:

Field Type Description
status string "completed", "already_optimized", or "error"
message string Human-readable description of what was done
duration_ms uint64 Time taken in milliseconds
vectors_processed uint64 Number of vectors processed during optimization

Status Codes: 200 (success), 404 (collection not found), 409 (already optimizing), 500 (optimization error)

curl -X POST http://localhost:8080/api/v1/collections/documents/optimize \
  -H "Content-Type: application/json" \
  -d '{}'
{
  "status": "completed",
  "message": "rebuilt HNSW index, recomputed graph edges, re-indexed metadata",
  "duration_ms": 1250,
  "vectors_processed": 50000
}

Storage Operations

Operations that act on a collection's on-disk state: forced snapshots, persistence and LSN inspection, write-ahead log pruning, segment compaction, and per-collection lock-contention metrics.

Force Snapshot

POST /api/v1/collections/{collection}/snapshot

Forces a synchronous snapshot of the collection and returns the LSN written. Useful before deliberate restarts or for taking snapshots on a schedule outside the server's auto-snapshot policy.

Path Parameters:

Parameter Type Description
collection string Collection name

Response:

Field Type Description
last_snapshot_lsn uint64 LSN of the snapshot just written 
curl -X POST http://localhost:8080/api/v1/collections/docs/snapshot
{ "last_snapshot_lsn": 1024 }

Persistence Status

GET /api/v1/collections/{collection}/persistence_status

Returns the snapshot and WAL LSN state for a single collection. Useful for confirming a write has been durably committed.

Path Parameters:

Parameter Type Description
collection string Collection name

Response:

Field Type Description
last_snapshot_lsn uint64 LSN of the last persisted snapshot
current_lsn uint64 LSN of the most recently durably committed write
next_lsn uint64 LSN that the next write will receive 
curl http://localhost:8080/api/v1/collections/docs/persistence_status
{
  "last_snapshot_lsn": 1024,
  "current_lsn": 1040,
  "next_lsn": 1041
}

Prune WAL

POST /api/v1/collections/{collection}/prune-wal

Deletes write-ahead log segments older than the latest snapshot for the collection, reclaiming disk space.

Path Parameters:

Parameter Type Description
collection string Collection name

Response:

Field Type Description
status string Operation status
files_deleted uint64 Number of WAL files deleted
bytes_freed uint64 Bytes reclaimed
duration_ms uint64 Duration in milliseconds 
curl -X POST http://localhost:8080/api/v1/collections/docs/prune-wal

Compact

POST /api/v1/collections/{collection}/compact

Merges on-disk segments, optionally removing tombstoned vectors. Reduces the number of segments and reclaims space from deletes.

Path Parameters:

Parameter Type Description
collection string Collection name

Request Body:

Parameter Type Required Default Description
min_segments uint32 No 4 Minimum number of segments to trigger compaction. 0 uses the default.
remove_deleted bool No true Remove deleted vectors during compaction.

Response:

Field Type Description
status string Operation status
segments_merged uint64 Number of segments merged
vectors_written uint64 Number of vectors written to the merged set
vectors_removed uint64 Number of deleted vectors removed
duration_ms uint64 Duration in milliseconds 
curl -X POST http://localhost:8080/api/v1/collections/docs/compact \
  -H "Content-Type: application/json" \
  -d '{ "min_segments": 8, "remove_deleted": true }'

Collection Metrics

GET /metrics/collection/{collection}

Returns lock-contention counters for a single collection, useful for diagnosing contention under concurrent load.

Path Parameters:

Parameter Type Description
collection string Collection name

Response:

Field Type Description
map_lock_acquisitions uint64 Acquisitions of the collection map read lock
collection_read_acquisitions uint64 Read-lock acquisitions on this collection
collection_write_acquisitions uint64 Write-lock acquisitions on this collection
total_blocked_microseconds uint64 Total time waited for these locks, in microseconds 
curl http://localhost:8080/metrics/collection/docs

Search

Search

POST /api/v1/collections/{collection}/search

Performs vector similarity search with optional filtering and graph enrichment.

Path Parameters:

Parameter Type Description
collection string Collection name

Request Body:

Parameter Type Required Default Description
query float array Yes Query vector. Must match collection dimension.
k uint32 Yes Number of results to return.
filter object No null Filter expression (see Filter Expressions).
strategy string No "auto" Filter strategy: "auto", "pre_filter", or "post_filter".
include_metadata bool No false Include metadata in results.
include_graph bool No false Include virtual graph edges for each result.
graph_threshold float No 0.0 Minimum similarity for graph edges. 0.0 uses the collection default threshold.
max_graph_edges uint32 No 10 Maximum graph edges to return per result.
ef_search uint32 No null Override HNSW ef_search for this query. Higher values improve recall at the cost of latency.

Response:

Field Type Description
results array Array of scored results
search_time_us uint64 Search time in microseconds
warning string Warning message (e.g., stale results during optimization). Omitted if empty.

Each result object:

Field Type Description
id uint64 Vector ID
score float Distance score (lower = more similar)
metadata object Metadata (only if include_metadata: true)
graph_edges array Related edges (only if include_graph: true)

Each graph edge:

Field Type Description
target_id uint64 ID of the related vector
similarity float Similarity score of the edge

Status Codes: 200 (success), 400 (dimension mismatch, invalid filter), 404 (collection not found)

Basic search:

curl -X POST http://localhost:8080/api/v1/collections/documents/search \
  -H "Content-Type: application/json" \
  -d '{
    "query": [0.1, 0.2, 0.3],
    "k": 10,
    "include_metadata": true
  }'
{
  "results": [
    {
      "id": 42,
      "score": 0.0012,
      "metadata": {"title": "Closest Document", "category": "tech"}
    },
    {
      "id": 17,
      "score": 0.0089,
      "metadata": {"title": "Second Closest", "category": "science"}
    }
  ],
  "search_time_us": 245
}

Filtered search:

curl -X POST http://localhost:8080/api/v1/collections/documents/search \
  -H "Content-Type: application/json" \
  -d '{
    "query": [0.1, 0.2, 0.3],
    "k": 5,
    "filter": {
      "and": [
        {"field": "category", "op": "eq", "value": "technology"},
        {"field": "year", "op": "gte", "value": 2023}
      ]
    },
    "strategy": "pre_filter",
    "include_metadata": true
  }'

Graph-enriched search:

curl -X POST http://localhost:8080/api/v1/collections/documents/search \
  -H "Content-Type: application/json" \
  -d '{
    "query": [0.1, 0.2, 0.3],
    "k": 10,
    "include_metadata": true,
    "include_graph": true,
    "graph_threshold": 0.85,
    "max_graph_edges": 5
  }'
{
  "results": [
    {
      "id": 42,
      "score": 0.0012,
      "metadata": {"title": "Closest Document"},
      "graph_edges": [
        {"target_id": 43, "similarity": 0.95},
        {"target_id": 99, "similarity": 0.88}
      ]
    }
  ],
  "search_time_us": 312
}

Batch Search

POST /api/v1/search/batch

Executes multiple search queries in a single request. Each query can target a different collection.

Request Body:

Parameter Type Required Description
queries array Yes Array of search query objects (same fields as the single search request, plus a collection field)

Each query object:

Parameter Type Required Default Description
collection string Yes Target collection name
query float array Yes Query vector
k uint32 Yes Number of results
filter object No null Filter expression
strategy string No "auto" Filter strategy
include_metadata bool No false Include metadata
include_graph bool No false Include graph edges
graph_threshold float No 0.0 Minimum graph edge similarity
max_graph_edges uint32 No 10 Max graph edges per result
ef_search uint32 No null Override HNSW ef_search

Response:

Field Type Description
results array Array of search responses (one per query, same format as single search response)
total_time_us uint64 Total time for the entire batch 
curl -X POST http://localhost:8080/api/v1/search/batch \
  -H "Content-Type: application/json" \
  -d '{
    "queries": [
      {
        "collection": "documents",
        "query": [0.1, 0.2, 0.3],
        "k": 5,
        "include_metadata": true
      },
      {
        "collection": "images",
        "query": [0.4, 0.5, 0.6],
        "k": 3
      }
    ]
  }'
{
  "results": [
    {
      "results": [
        {"id": 42, "score": 0.0012, "metadata": {"title": "Doc A"}}
      ],
      "search_time_us": 180
    },
    {
      "results": [
        {"id": 7, "score": 0.0034}
      ],
      "search_time_us": 95
    }
  ],
  "total_time_us": 290
}

Filter Expressions

Filters narrow search results based on vector metadata. They are passed in the filter field of search requests.

Field Filters

A field filter compares a metadata field against a value.

{"field": "category", "op": "eq", "value": "technology"}

Structure:

Field Type Description
field string Metadata field name
op string Comparison operator (see table below)
value any Single comparison value as plain JSON (for most operators)
values array Array of plain JSON values (for in and between operators)

Operators

Operator Description Value Type Example
eq Equal to single {"field": "status", "op": "eq", "value": "active"}
ne Not equal to single {"field": "status", "op": "ne", "value": "deleted"}
gt Greater than single {"field": "price", "op": "gt", "value": 9.99}
gte Greater than or equal to single {"field": "year", "op": "gte", "value": 2020}
lt Less than single {"field": "count", "op": "lt", "value": 100}
lte Less than or equal to single {"field": "rating", "op": "lte", "value": 5.0}
in Value is in the provided list multi {"field": "category", "op": "in", "values": ["a", "b"]}
between Value is between two bounds (inclusive) multi (2) {"field": "price", "op": "between", "values": [10.0, 50.0]}
exists Field exists (value is ignored) none {"field": "thumbnail", "op": "exists", "value": true}
contains String contains substring single {"field": "title", "op": "contains", "value": "vector"}

Value Types

In REST API filter expressions, values are plain JSON. The type is inferred automatically:

JSON Type Example Description
string "hello" String comparison
number (int) 42 Integer comparison
number (float) 3.14 Float comparison
boolean true Boolean comparison
array ["a", "b"] Used with in operator

Note: The gRPC API uses typed wrappers (string_value, int_value, float_value, bool_value, string_list_value) as defined in the protobuf schema. The REST API does not require these wrappers.

Logical Operators

Combine multiple filters using and, or, and not.

AND (all must match):

{
  "and": [
    {"field": "category", "op": "eq", "value": "tech"},
    {"field": "year", "op": "gte", "value": 2023}
  ]
}

OR (at least one must match):

{
  "or": [
    {"field": "category", "op": "eq", "value": "tech"},
    {"field": "category", "op": "eq", "value": "science"}
  ]
}

NOT (negate a filter):

{
  "not": {"field": "status", "op": "eq", "value": "archived"}
}

Nested example (AND with nested OR):

{
  "and": [
    {"field": "year", "op": "gte", "value": 2020},
    {
      "or": [
        {"field": "category", "op": "eq", "value": "tech"},
        {"field": "category", "op": "eq", "value": "science"}
      ]
    }
  ]
}

Graph

The virtual graph (SwarnDB's automatic similarity graph) connects vectors that exceed a similarity threshold. The graph is computed automatically during insertion (if a threshold is set) or rebuilt during optimize().

Get Related

GET /api/v1/collections/{collection}/graph/related/{id}?threshold=0.8&max_results=10

Returns vectors directly connected to the given vector in the virtual graph.

Path Parameters:

Parameter Type Description
collection string Collection name
id uint64 Vector ID

Query Parameters:

Parameter Type Required Default Description
threshold float No Collection default Minimum similarity for returned edges
max_results uint32 No All edges Maximum number of edges to return

Response:

Field Type Description
edges array Array of graph edges

Each edge:

Field Type Description
target_id uint64 ID of the related vector
similarity float Similarity score

Status Codes: 200 (success), 404 (collection or vector not found)

curl "http://localhost:8080/api/v1/collections/documents/graph/related/42?threshold=0.85&max_results=5"
{
  "edges": [
    {"target_id": 43, "similarity": 0.95},
    {"target_id": 99, "similarity": 0.91},
    {"target_id": 17, "similarity": 0.87}
  ]
}

Traverse

POST /api/v1/collections/{collection}/graph/traverse

Performs a multi-hop traversal through the virtual graph starting from a given vector.

Path Parameters:

Parameter Type Description
collection string Collection name

Request Body:

Parameter Type Required Default Description
start_id uint64 Yes Starting vector ID
depth uint32 No 2 Maximum traversal depth (number of hops)
threshold float No Collection default Minimum similarity threshold for traversal
max_results uint32 No All reachable Maximum number of nodes to return

Response:

Field Type Description
nodes array Array of traversal nodes

Each node:

Field Type Description
id uint64 Vector ID
depth uint32 Hop distance from start
path_similarity float Cumulative similarity along the path
path uint64 array Ordered list of vector IDs from start to this node

Status Codes: 200 (success), 404 (collection or start vector not found)

curl -X POST http://localhost:8080/api/v1/collections/documents/graph/traverse \
  -H "Content-Type: application/json" \
  -d '{
    "start_id": 42,
    "depth": 3,
    "threshold": 0.8,
    "max_results": 20
  }'
{
  "nodes": [
    {"id": 43, "depth": 1, "path_similarity": 0.95, "path": [42, 43]},
    {"id": 99, "depth": 1, "path_similarity": 0.91, "path": [42, 99]},
    {"id": 101, "depth": 2, "path_similarity": 0.86, "path": [42, 43, 101]}
  ]
}

Set Threshold

POST /api/v1/collections/{collection}/graph/threshold

Sets the similarity threshold for graph edge computation. Can be set at the collection level (affects all vectors) or per-vector.

Path Parameters:

Parameter Type Description
collection string Collection name

Request Body:

Parameter Type Required Default Description
vector_id uint64 No 0 Target vector ID. 0 sets the collection-level default threshold.
threshold float Yes Similarity threshold (0.0 to 1.0 for cosine).

Response:

Field Type Description
success bool true on success

Status Codes: 200 (success), 400 (invalid threshold), 404 (collection or vector not found)

curl -X POST http://localhost:8080/api/v1/collections/documents/graph/threshold \
  -H "Content-Type: application/json" \
  -d '{
    "vector_id": 0,
    "threshold": 0.85
  }'
{
  "success": true
}

Typed Graph (hybrid mode)

These routes are available on hybrid collections only. They manage the first-class typed graph of nodes and edges. See Typed Graph: Overview for the concepts and field meanings.

Method Route Description
POST /api/v1/collections/{collection}/graph/nodes Create a typed node. Returns the node id.
GET /api/v1/collections/{collection}/graph/nodes/{node_id} Get a typed node.
PATCH /api/v1/collections/{collection}/graph/nodes/{node_id} Update a node's property bag (records an audit entry).
DELETE /api/v1/collections/{collection}/graph/nodes/{node_id} Delete a node and its incident edges.
GET /api/v1/collections/{collection}/graph/nodes/{node_id}/edges List edges incident to a node (query params: direction, edge_type).
POST /api/v1/collections/{collection}/graph/edges Create a typed edge. Returns the edge id. Accepts optional temporal fields.
GET /api/v1/collections/{collection}/graph/edges/{edge_id} Get a typed edge.
PATCH /api/v1/collections/{collection}/graph/edges/{edge_id} Update a manual edge's properties, confidence, or verified flag.
DELETE /api/v1/collections/{collection}/graph/edges/{edge_id} Delete a typed edge.
POST /api/v1/collections/{collection}/graph/edges/{edge_id}/verify Verify an edge (locks it against re-extraction).
POST /api/v1/collections/{collection}/graph/edges/{edge_id}/reject Reject an edge (deletes it and remembers the pattern).
POST /api/v1/collections/{collection}/graph/bulk-import-edges Bulk import edges from CSV or JSONL.
POST /api/v1/collections/{collection}/hybrid_query Run a composable hybrid query plan.

A node body carries kind (content or entity), label, properties, optional embedding, source, and created_by. An edge body carries source, target, edge_type, properties, provenance, confidence, verified, is_manual, and the three optional temporal fields valid_from, valid_until, and temporal_context (see Temporal Edges). The PATCH .../graph/nodes/{node_id} body carries an optional properties object (replaces the property bag; only the property bag is mutable, provenance and embedding stay immutable) and an optional actor string for the audit trail; the response is the updated node. The bulk-import body carries format (csv or jsonl), data (the raw payload), and optional auto_add_edge_types; the response reports total_rows, imported, failed, and a per-row errors list. The bulk-import payload may also carry the optional temporal columns/keys described under Temporal Edges.

Update Node (hybrid mode)

PATCH /api/v1/collections/{collection}/graph/nodes/{node_id}

Updates a typed node's property bag and appends an entry to its audit trail. Only the property bag is mutable; provenance and the embedding are immutable so the node-to-vector bridge cannot desync.

Path Parameters:

Parameter Type Description
collection string Collection name
node_id uint64 Node ID

Request Body:

Parameter Type Required Description
properties object No New property bag. When present, replaces the existing properties. Omitted or null leaves them unchanged.
actor string No Audit actor recorded in the node's history.

Response: The updated node object under a node field, carrying id, kind, properties, embedding, source, created_at, created_by, updated_at, and the history audit trail. kind is "content" for content nodes, or {"entity": {"label": "<label>"}} for entity nodes.

Status Codes: 200 (success), 400 (bad properties payload), 404 (collection or node not found), 409 (collection not in hybrid mode), 500 (storage error)

curl -X PATCH http://localhost:8080/api/v1/collections/kb/graph/nodes/42 \
  -H "Content-Type: application/json" \
  -d '{
    "properties": {"status": "reviewed", "owner": "alice"},
    "actor": "alice"
  }'

The gRPC equivalent is UpdateNode(UpdateNodeRequest) returns (UpdateNodeResponse). UpdateNodeRequest carries collection, node_id, an optional properties_json (replaces the bag when set), and an optional actor. UpdateNodeResponse returns the updated TypedNode.

Temporal Edges

Typed edges can carry an optional validity window and an optional regime/context label. These fields are additive and default-off: an edge created without them behaves exactly as before.

The three fields appear on PutEdgeRequest (gRPC field numbers 10, 11, 12), on the REST POST .../graph/edges body, and on every returned TypedEdge (gRPC field numbers 12, 13, 14):

Field Type Default Description
valid_from uint64 absent (null) Validity window start, unix-epoch milliseconds. Absent means an unbounded start.
valid_until uint64 absent (null) Validity window end, unix-epoch milliseconds, EXCLUSIVE. Absent means an unbounded end.
temporal_context string absent (null) Regime, version, or scenario label. Absent (or empty on the REST/gRPC surface) means no context.

An edge is valid at instant t when (valid_from is absent OR t >= valid_from) AND (valid_until is absent OR t < valid_until). An edge with both bounds absent is unbounded in time (always valid). On a returned TypedEdge these fields serialize as valid_from, valid_until, and temporal_context, each null when not set.

curl -X POST http://localhost:8080/api/v1/collections/kb/graph/edges \
  -H "Content-Type: application/json" \
  -d '{
    "source": 10,
    "target": 20,
    "edge_type": "EMPLOYED_AT",
    "valid_from": 1704067200000,
    "valid_until": 1735689600000,
    "temporal_context": "v2024"
  }'

Bulk-import temporal columns/keys. The POST .../graph/bulk-import-edges payload accepts three optional fields per row. For CSV, add the optional header columns valid_from, valid_until, and temporal_context alongside the required source, target, edge_type columns (and the existing optional properties, confidence columns); an empty cell means absent. For JSONL, add the optional keys valid_from, valid_until, and temporal_context to each row object; a missing key or empty temporal_context string means absent. Adding these columns/keys is backward-compatible: payloads that omit them import exactly as before.

source,target,edge_type,confidence,valid_from,valid_until,temporal_context
10,20,EMPLOYED_AT,1.0,1704067200000,1735689600000,v2024
30,40,EMPLOYED_AT,0.9,,,
{"source": 10, "target": 20, "edge_type": "EMPLOYED_AT", "valid_from": 1704067200000, "temporal_context": "v2024"}

Filtered Graph Reads (hybrid mode)

Two gRPC RPCs enumerate the whole typed graph in pages, filtered by kind, label, edge type, and property conditions. They walk a cursor over sorted ids, so a client pages by passing the previous response's next_cursor back as the next after_id. These are gRPC-only; over REST the same scoping is expressed with the scan_by_filter step of a hybrid query plan (see Hybrid Query Plans).

EnumerateNodes (EnumerateNodesRequest -> EnumerateNodesResponse):

Field Type Description
collection string Collection name.
after_id uint64 Cursor. Returns nodes with id strictly greater than this. 0 starts at the beginning.
limit uint32 Page size (server-clamped).
filter_by_kind bool When true, restrict to nodes of kind. When false, kind is ignored.
kind TypedNodeKind TYPED_NODE_CONTENT or TYPED_NODE_ENTITY. Applied only when filter_by_kind is true.
label string Entity-label filter. Empty means no label filter.
predicate HybridPredicate Optional property condition over node properties, including structural incident-edge-count terms. Absent means no property filter.

The response carries nodes (a page of TypedNode), next_cursor (the last id in the page; 0 when exhausted), and has_more.

EnumerateEdges (EnumerateEdgesRequest -> EnumerateEdgesResponse):

Field Type Description
collection string Collection name.
after_id uint64 Cursor. Returns edges with id strictly greater than this. 0 starts at the beginning.
limit uint32 Page size (server-clamped).
edge_type string Edge-type filter. Empty means no type filter.
predicate HybridPredicate Optional property condition over edge properties. Absent means no filter. Node-only structural incident-edge-count terms are rejected here.
endpoint_label string Endpoint-node label constraint. An edge passes when one endpoint (source or target) node matches. Empty means no constraint.
endpoint_kind TypedNodeKind Endpoint-node kind constraint. Applied only when filter_by_endpoint_kind is true.
filter_by_endpoint_kind bool When true, apply endpoint_kind.

The response carries edges (a page of TypedEdge), next_cursor, and has_more.

Hybrid Query Plans (hybrid mode)

POST /api/v1/collections/{collection}/hybrid_query

Runs a composable plan of steps. Each step consumes the current frontier and produces the next. Over REST the request body is the plan itself, and the response carries nodes, edges, and paths. The gRPC equivalent is HybridQuery(HybridQueryRequest) returns (HybridQueryResponse), where the request carries collection, the plan, and an optional rrf_rank ranking spec.

JSON encoding. Over REST, a plan is {"steps": [ ... ], "return_kind": "Nodes"}. The return_kind is one of "Nodes", "Edges", or "Paths" (PascalCase). Each step is an externally-tagged JSON object keyed by the step name in PascalCase, for example {"Traverse": { ... }}. This mirrors the proto HybridStep oneof, whose field names are the snake_case forms (traverse, k_hop, shortest_path, scan_by_filter, vector_math, and so on). The step names and their parameters are below; steps not listed here are unchanged from prior versions.

scan_by_filter (source step)

Seeds the frontier by scanning the graph for nodes matching an optional kind, entity label, and property condition. A source step: it takes no vector and no explicit ids. An all-empty filter yields every node.

REST step shape (ScanByFilter):

Field Type Description
is_entity bool true keeps only entity nodes, false keeps only content nodes. Omitted (null) means no kind filter.
label string Entity-label filter. Omitted (null) means no label filter.
predicate object Optional property condition (see Predicates). Omitted means no property condition. 
{"ScanByFilter": {"is_entity": true, "label": "Person", "predicate": {"Exists": {"field": {"Property": "email"}}}}}

The gRPC step is HybridScanByFilter with filter_by_kind (bool), kind (TypedNodeKind), label (string), and predicate (HybridPredicate). When filter_by_kind is true, kind == TYPED_NODE_ENTITY maps to is_entity: true and TYPED_NODE_CONTENT to is_entity: false.

Predicates

Predicates are externally-tagged JSON objects keyed by PascalCase variant name. A field reference is one of {"Property": "<key>"}, "Label", "Kind", or the node-only structural source {"IncidentEdgeCount": {"edge_type": "<type>", "direction": "<dir>"}}. The comparison operator is one of Eq, Ne, Lt, Le, Gt, Ge.

The structural IncidentEdgeCount reference resolves to the count of a node's incident edges, optionally constrained by edge_type (empty means any type) and direction. direction is one of outgoing, incoming, or both; when omitted it defaults to outgoing. This term is node-only: it is meaningful in node scans and node enumeration, and is rejected on edge enumeration.

Predicate Shape Meaning
Compare {"Compare": {"field": <ref>, "op": "Ge", "value": 3}} field op value
In {"In": {"field": <ref>, "values": [ ... ]}} field value is in the set
NotIn {"NotIn": {"field": <ref>, "values": [ ... ]}} field value is not in the set
Exists {"Exists": {"field": <ref>}} field resolves to a value
And {"And": [ <pred>, ... ]} all match
Or {"Or": [ <pred>, ... ]} at least one matches
Not {"Not": <pred>} negate
Always "Always" always true

A structural example, "entity nodes with at least three outgoing KNOWS edges":

{"Compare": {"field": {"IncidentEdgeCount": {"edge_type": "KNOWS", "direction": "outgoing"}}, "op": "Ge", "value": 3}}

The gRPC mirror is HybridPredicate (a oneof over HybridCompare, HybridInList, HybridPredicateList, and so on), with the field reference as HybridPropertyRef (a oneof over property, label, kind, and HybridIncidentEdgeCount). On the gRPC surface, comparison values are carried as JSON scalar literals in value_json / values_json; over REST they are plain JSON.

Edge weighting (weight, order_by_weight)

KHop and ShortestPath steps accept an optional edge-weight spec. When absent, every edge weighs 1.0 and behavior is unchanged.

The weight spec (weight on a step) carries:

Field Type Description
use_confidence bool Fold edge confidence into the weight.
min_confidence float Drop edges below this confidence.
recency_half_life_ms uint64 Recency half-life in milliseconds. 0 means no recency decay.
use_explicit_weight bool Read a numeric weight from edge properties.
explicit_weight_key string Property key for the explicit weight. Empty falls back to "weight".

KHop also accepts order_by_weight (bool): node membership is unchanged, only the output order shifts to descending accumulated edge weight. ShortestPath accepts weighted (bool): when true, it minimizes total path cost using the weight spec instead of hop count. The gRPC mirror is HybridWeightSpec on HybridKHop.weight / HybridShortestPath.weight, with HybridKHop.order_by_weight and HybridShortestPath.weighted.

Edge weighting can also weight the bridge routes of the optional RRF ranking spec: RrfRankSpec.edge_weight (gRPC field 6) carries the same HybridWeightSpec. Absent means unweighted route counting.

{"KHop": {
  "edge_type": "CITES",
  "max": 2,
  "predicate": null,
  "weight": {"use_confidence": true, "min_confidence": 0.5, "recency_half_life_ms": 0, "use_explicit_weight": false, "explicit_weight_key": ""},
  "order_by_weight": true
}}

Temporal traversal filter (temporal)

Traverse, KHop, and ShortestPath steps accept an optional temporal filter (gRPC: HybridTraverse.temporal field 3, HybridKHop.temporal field 6, HybridShortestPath.temporal field 5). When absent, traversal is unfiltered and byte-identical to prior versions.

The filter (REST temporal object, gRPC HybridTemporalFilter) carries:

Field Type Description
as_of uint64 Instant for the time check, unix-epoch milliseconds. Omitted (null) means the server uses "now".
include_unbounded bool Whether edges with no validity window pass the time check.
context string Required regime/context. An edge must carry a matching temporal_context. Omitted (null) means context is ignored.

Note on the gRPC default: include_unbounded is a plain proto3 bool, so omitting it on the wire yields false, which is stricter than the domain default. Because the whole filter message is optional (absence means no filtering at all), a client that sends a filter should set include_unbounded explicitly.

{"Traverse": {
  "edge_type": "EMPLOYED_AT",
  "direction": "outgoing",
  "temporal": {"as_of": 1717200000000, "include_unbounded": true, "context": "v2024"}
}}

direction is one of outgoing, incoming, or both.

vector_math (frontier-consuming step)

The VectorMath step runs one vector-arithmetic operation over exactly the current node frontier and keeps the top-k. It is graph-then-vector: the graph has already fixed the candidate set, so the op runs over only those nodes.

REST step shape (VectorMath):

Field Type Description
op object The operation, an externally-tagged object (see below).
k uint Number of frontier nodes to keep.
on_missing string Policy for frontier nodes that have no vector: "Skip" (drop and count) or "Error" (fail the query).

The op is one of six externally-tagged objects, matching the proto HybridVectorMath oneof (analogy, diversity, cone, isolation, centroid, interpolate):

op (REST key) Operands Behavior
Analogy a, b, c (float arrays) Rank the frontier by ascending distance to a - b + c; top-k.
Diversity query (float array), lambda (float) MMR over the frontier; returns nodes in selection order, length min(k, eligible).
Cone direction (float array), aperture_radians (float) Keep nodes whose angle to direction is within the aperture; rank by ascending angle; cap at k.
Isolation centroids (array of float arrays) Score each node by its minimum distance to any centroid; return the top-k most isolated.
Centroid (none) Compute the centroid of the frontier's own vectors; rank by ascending distance to it; top-k.
Interpolate a, b (float arrays), t (float) Rank by ascending distance to the interpolated point between a and b at t; top-k. t must be in [0, 1].

Analogy, Centroid, Interpolate, and Isolation rank by the collection's configured distance metric. Diversity and Cone use their own internal cosine, which is inherent to MMR scoring and cone geometry.

For each frontier node the step resolves a vector in this order: an inline node embedding, else the indexed vector, else the on_missing policy. The frontier is bounded by a server cap.

{"VectorMath": {
  "op": {"Analogy": {"a": [0.1, 0.2], "b": [0.3, 0.4], "c": [0.5, 0.6]}},
  "k": 10,
  "on_missing": "Skip"
}}

The gRPC step is HybridVectorMath (HybridStep.vector_math, field 14). Its oneof op selects one of HybridAnalogy, HybridDiversity, HybridCone, HybridIsolation, HybridCentroid, or HybridInterpolate; k (uint32) and on_missing (HybridOnMissingVector, HYBRID_ON_MISSING_SKIP or HYBRID_ON_MISSING_ERROR) follow. HybridIsolation.centroids is a list of HybridVector wrappers, each carrying a values float array, so a list-of-lists of floats can be expressed where proto3 cannot nest repeated fields directly.

Example plan

A plan that scans for entity nodes, expands two weighted hops as of a point in time, then ranks the frontier by an analogy:

curl -X POST http://localhost:8080/api/v1/collections/kb/hybrid_query \
  -H "Content-Type: application/json" \
  -d '{
    "steps": [
      {"ScanByFilter": {"is_entity": true, "label": "Person", "predicate": null}},
      {"KHop": {
        "edge_type": "KNOWS",
        "max": 2,
        "predicate": null,
        "weight": {"use_confidence": true, "min_confidence": 0.5, "recency_half_life_ms": 0, "use_explicit_weight": false, "explicit_weight_key": ""},
        "order_by_weight": true,
        "temporal": {"as_of": 1717200000000, "include_unbounded": true, "context": null}
      }},
      {"VectorMath": {"op": {"Centroid": {}}, "k": 10, "on_missing": "Skip"}}
    ],
    "return_kind": "Nodes"
  }'
{
  "nodes": [
    {"id": 42, "node": {"id": 42, "kind": {"entity": {"label": "Person"}}, "properties": {}, "source": "manual", "created_at": 1717200000000}}
  ],
  "edges": [],
  "paths": []
}

The response nodes carry an id and, when the typed store resolved it, the full node. Returned edges are TypedEdge objects (carrying the temporal fields when set). paths are arrays of node ids.

Extraction (hybrid mode)

These routes are available on hybrid collections only and drive LLM extraction. See LLM Extraction.

Method Route Description
PUT /api/v1/collections/{collection}/llm-config Set the LLM config (api key is write-only).
GET /api/v1/collections/{collection}/llm-config Get the redacted LLM config (never the api key).
POST /api/v1/collections/{collection}/llm-config/rotate Rotate just the api key.
PUT /api/v1/collections/{collection}/ontology Set the ontology from a template and/or extension.
GET /api/v1/collections/{collection}/ontology Get the collection's ontology.
POST /api/v1/collections/{collection}/extraction/cost-preview Estimate token usage and cost for a set of chunks.
POST /api/v1/collections/{collection}/extraction Start an async extraction job. Returns a job_id.
GET /api/v1/collections/{collection}/extraction/{job_id} Get an extraction job's status.
POST /api/v1/collections/{collection}/extraction/{job_id}/cancel Cancel a running extraction job.
GET /api/v1/collections/{collection}/proposals List pending ontology proposals.
POST /api/v1/collections/{collection}/proposals/{id}/approve Approve an ontology proposal.
POST /api/v1/collections/{collection}/proposals/{id}/reject Reject an ontology proposal.
POST /api/v1/collections/{collection}/extraction/diff Diff a document's chunks against the stored extraction state.
POST /api/v1/collections/{collection}/extraction/reextract Re-extract only the changed and new chunks of a document.

The LLM config body carries base_url, api_key, model_name, temperature, max_tokens, and timeout_seconds. Extraction and diff/reextract bodies carry a chunks array (each chunk: doc_id, chunk_id, text, optional embedding); diff and reextract also carry doc_id.

The set-ontology body (PUT .../ontology) carries base_template, the entity_labels and edge_types of the extension, a replace flag, and two optional prompt fields: system_prompt (a full override of the generic extraction framing) and extra_guidance (a domain hint appended on top of the framing). An empty value for either means the default prompt is used. The get-ontology response (GET .../ontology) returns the same shape and echoes both fields. SwarnDB always keeps the JSON output schema and the ontology's allowed labels and edge types in the prompt regardless of these fields, so a custom prompt cannot break parsing or step outside the ontology. Both fields are additive and backward-compatible.

The job-status response (GET .../extraction/{job_id}) reports job_id, collection, state, total_chunks, processed_chunks, entities_written, edges_written, cache_hits, cache_misses, and error, plus two partial-success fields: failed_chunks (the total count of chunks that failed) and chunk_errors (a bounded sample of failures, each with doc_id, chunk_id, and error). The state is one of queued, running, completed, completed_with_errors, failed, or cancelled; a job that extracted from most chunks but had one or more fail ends in completed_with_errors. Both fields are additive and backward-compatible. See LLM Extraction.

Vector Math

Advanced vector operations for analytics, clustering, drift detection, and more.

Detect Ghost Vectors

POST /api/v1/collections/{collection}/math/ghosts

Identifies isolated vectors that are far from all cluster centroids. Ghost vectors may indicate outliers, noise, or data quality issues.

Path Parameters:

Parameter Type Description
collection string Collection name

Request Body:

Parameter Type Required Default Description
threshold float No 0.0 Isolation score threshold. Vectors scoring above this are ghosts. 0.0 returns all with scores.
centroids array of float arrays No null Custom centroid vectors. If omitted, auto-clusters with auto_k.
auto_k uint32 No 8 Number of clusters for auto-centroid computation.
metric string No "euclidean" Distance metric for isolation scoring.

Response:

Field Type Description
ghosts array Array of ghost vector results
compute_time_us uint64 Computation time in microseconds

Each ghost object:

Field Type Description
id uint64 Vector ID
isolation_score float Distance to nearest centroid (higher = more isolated) 
curl -X POST http://localhost:8080/api/v1/collections/documents/math/ghosts \
  -H "Content-Type: application/json" \
  -d '{
    "threshold": 2.5,
    "auto_k": 10,
    "metric": "euclidean"
  }'
{
  "ghosts": [
    {"id": 77, "isolation_score": 4.21},
    {"id": 203, "isolation_score": 3.15}
  ],
  "compute_time_us": 8450
}

Cone Search

POST /api/v1/collections/{collection}/math/cone

Finds all vectors within a directional cone defined by a direction vector and an aperture angle. Useful for directional similarity queries.

Path Parameters:

Parameter Type Description
collection string Collection name

Request Body:

Parameter Type Required Description
direction float array Yes Direction vector defining the cone axis.
aperture_radians float Yes Half-angle of the cone in radians (0 to pi).

Response:

Field Type Description
results array Array of vectors within the cone
compute_time_us uint64 Computation time in microseconds

Each result:

Field Type Description
id uint64 Vector ID
cosine_similarity float Cosine similarity to the direction
angle_radians float Angle from the direction in radians 
curl -X POST http://localhost:8080/api/v1/collections/documents/math/cone \
  -H "Content-Type: application/json" \
  -d '{
    "direction": [1.0, 0.0, 0.0],
    "aperture_radians": 0.5
  }'
{
  "results": [
    {"id": 12, "cosine_similarity": 0.98, "angle_radians": 0.12},
    {"id": 45, "cosine_similarity": 0.92, "angle_radians": 0.38}
  ],
  "compute_time_us": 1230
}

Compute Centroid

POST /api/v1/collections/{collection}/math/centroid

Computes the centroid (mean vector) of a set of vectors. Optionally supports weighted averaging.

Path Parameters:

Parameter Type Description
collection string Collection name

Request Body:

Parameter Type Required Default Description
vector_ids uint64 array No All vectors IDs of vectors to include. Empty means all vectors in the collection.
weights float array No Equal weight Per-vector weights. Must match the length of vector_ids if provided.

Response:

Field Type Description
centroid float array The computed centroid vector
compute_time_us uint64 Computation time in microseconds 
curl -X POST http://localhost:8080/api/v1/collections/documents/math/centroid \
  -H "Content-Type: application/json" \
  -d '{
    "vector_ids": [1, 2, 3, 4, 5],
    "weights": [1.0, 1.0, 2.0, 1.0, 1.0]
  }'
{
  "centroid": [0.25, 0.38, 0.51],
  "compute_time_us": 42
}

Interpolate (SLERP/LERP)

POST /api/v1/math/interpolate

Interpolates between two vectors using linear (LERP) or spherical linear (SLERP) interpolation. Can produce a single result or a sequence of evenly spaced points.

Note: This endpoint does not require a collection. It operates on raw vectors.

Request Body:

Parameter Type Required Default Description
a float array Yes Start vector.
b float array Yes End vector.
t float No 0.0 Interpolation parameter (0.0 = a, 1.0 = b). Ignored if sequence_count > 0.
method string No "lerp" Interpolation method: "lerp" or "slerp".
sequence_count uint32 No 0 If greater than 0, generates a sequence of evenly spaced interpolated points.

Response:

Field Type Description
results array of float arrays One interpolated vector (if sequence_count is 0) or a sequence of vectors.
compute_time_us uint64 Computation time in microseconds 
curl -X POST http://localhost:8080/api/v1/math/interpolate \
  -H "Content-Type: application/json" \
  -d '{
    "a": [1.0, 0.0, 0.0],
    "b": [0.0, 1.0, 0.0],
    "t": 0.5,
    "method": "slerp"
  }'
{
  "results": [[0.707, 0.707, 0.0]],
  "compute_time_us": 5
}

Sequence example:

curl -X POST http://localhost:8080/api/v1/math/interpolate \
  -H "Content-Type: application/json" \
  -d '{
    "a": [1.0, 0.0, 0.0],
    "b": [0.0, 1.0, 0.0],
    "method": "slerp",
    "sequence_count": 5
  }'
{
  "results": [
    [1.0, 0.0, 0.0],
    [0.924, 0.383, 0.0],
    [0.707, 0.707, 0.0],
    [0.383, 0.924, 0.0],
    [0.0, 1.0, 0.0]
  ],
  "compute_time_us": 12
}

Detect Drift

POST /api/v1/collections/{collection}/math/drift

Compares two windows (groups) of vectors to detect distributional drift. Useful for monitoring embedding quality over time.

Path Parameters:

Parameter Type Description
collection string Collection name

Request Body:

Parameter Type Required Default Description
window1_ids uint64 array Yes Vector IDs for the first (baseline) window.
window2_ids uint64 array Yes Vector IDs for the second (comparison) window.
metric string No "euclidean" Distance metric for drift computation.
threshold float No null If set, the has_drifted field indicates whether drift exceeds this threshold.

Response:

Field Type Description
centroid_shift float Distance between the centroids of the two windows
mean_distance_window1 float Average intra-window distance for window 1
mean_distance_window2 float Average intra-window distance for window 2
spread_change float Difference in spread (window2 - window1)
has_drifted bool true if centroid_shift exceeds the provided threshold
compute_time_us uint64 Computation time in microseconds 
curl -X POST http://localhost:8080/api/v1/collections/documents/math/drift \
  -H "Content-Type: application/json" \
  -d '{
    "window1_ids": [1, 2, 3, 4, 5],
    "window2_ids": [101, 102, 103, 104, 105],
    "metric": "cosine",
    "threshold": 0.1
  }'
{
  "centroid_shift": 0.15,
  "mean_distance_window1": 0.32,
  "mean_distance_window2": 0.41,
  "spread_change": 0.09,
  "has_drifted": true,
  "compute_time_us": 520
}

K-Means Clustering

POST /api/v1/collections/{collection}/math/cluster

Performs k-means clustering on vectors in the collection.

Path Parameters:

Parameter Type Description
collection string Collection name

Request Body:

Parameter Type Required Default Description
k uint32 Yes Number of clusters.
max_iterations uint32 No 100 Maximum number of k-means iterations.
tolerance float No 0.0001 Convergence tolerance. Stops when centroid movement is below this.
metric string No "euclidean" Distance metric for clustering.

Response:

Field Type Description
centroids array of float arrays Computed cluster centroid vectors
assignments array Per-vector cluster assignments
iterations uint32 Number of iterations performed
converged bool Whether the algorithm converged
compute_time_us uint64 Computation time in microseconds

Each assignment:

Field Type Description
id uint64 Vector ID
cluster uint32 Assigned cluster index (0-based)
distance_to_centroid float Distance from the vector to its centroid 
curl -X POST http://localhost:8080/api/v1/collections/documents/math/cluster \
  -H "Content-Type: application/json" \
  -d '{
    "k": 5,
    "max_iterations": 200,
    "tolerance": 0.001,
    "metric": "euclidean"
  }'
{
  "centroids": [
    [0.12, 0.34, 0.56],
    [0.78, 0.90, 0.11]
  ],
  "assignments": [
    {"id": 1, "cluster": 0, "distance_to_centroid": 0.23},
    {"id": 2, "cluster": 1, "distance_to_centroid": 0.15}
  ],
  "iterations": 47,
  "converged": true,
  "compute_time_us": 15200
}

PCA (Dimensionality Reduction)

POST /api/v1/collections/{collection}/math/pca

Performs Principal Component Analysis on vectors in the collection, projecting them into a lower-dimensional space.

Path Parameters:

Parameter Type Description
collection string Collection name

Request Body:

Parameter Type Required Default Description
n_components uint32 No 2 Number of principal components to compute.
vector_ids uint64 array No All vectors IDs of vectors to include. Empty means all vectors.

Response:

Field Type Description
components array of float arrays Principal component vectors (eigenvectors)
explained_variance float array Variance explained by each component
mean float array Mean vector (centroid) of the input data
projected array of float arrays Input vectors projected into the reduced space
compute_time_us uint64 Computation time in microseconds 
curl -X POST http://localhost:8080/api/v1/collections/documents/math/pca \
  -H "Content-Type: application/json" \
  -d '{
    "n_components": 3,
    "vector_ids": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
  }'
{
  "components": [
    [0.57, 0.42, 0.71],
    [0.82, -0.38, -0.43],
    [-0.01, 0.82, -0.57]
  ],
  "explained_variance": [12.5, 8.3, 3.1],
  "mean": [0.33, 0.44, 0.55],
  "projected": [
    [1.2, -0.5, 0.3],
    [0.8, 0.9, -0.1]
  ],
  "compute_time_us": 2100
}

Compute Analogy

POST /api/v1/math/analogy

Computes vector analogies using the classic "A is to B as C is to ?" formula (result = B - A + C), or performs general weighted vector arithmetic.

Note: This endpoint does not require a collection. It operates on raw vectors.

Request Body:

Parameter Type Required Default Description
a float array No null Vector A (the "from" in the analogy). Required for analogy mode.
b float array No null Vector B (the "to" in the analogy). Required for analogy mode.
c float array No null Vector C (the "as" in the analogy). Required for analogy mode.
normalize bool No false Normalize the result vector to unit length.
terms array No [] For general arithmetic mode. Array of {vector, weight} terms. Overrides a/b/c if non-empty.

Each term object (for general arithmetic):

Field Type Description
vector float array A vector
weight float Weight multiplier for this vector

Response:

Field Type Description
result float array The computed result vector
compute_time_us uint64 Computation time in microseconds

Analogy mode (A:B :: C:?):

curl -X POST http://localhost:8080/api/v1/math/analogy \
  -H "Content-Type: application/json" \
  -d '{
    "a": [1.0, 0.0, 0.0],
    "b": [0.0, 1.0, 0.0],
    "c": [0.5, 0.0, 0.5],
    "normalize": true
  }'
{
  "result": [-0.408, 0.816, 0.408],
  "compute_time_us": 3
}

General arithmetic mode:

curl -X POST http://localhost:8080/api/v1/math/analogy \
  -H "Content-Type: application/json" \
  -d '{
    "terms": [
      {"vector": [1.0, 0.0, 0.0], "weight": 0.5},
      {"vector": [0.0, 1.0, 0.0], "weight": 0.3},
      {"vector": [0.0, 0.0, 1.0], "weight": 0.2}
    ],
    "normalize": false
  }'
{
  "result": [0.5, 0.3, 0.2],
  "compute_time_us": 2
}

Diversity Sampling (MMR)

POST /api/v1/collections/{collection}/math/diversity

Selects a diverse subset of vectors using Maximal Marginal Relevance (MMR). Balances relevance to the query with diversity among selected results.

Path Parameters:

Parameter Type Description
collection string Collection name

Request Body:

Parameter Type Required Default Description
query float array Yes Query vector.
k uint32 Yes Number of diverse results to select.
lambda float Yes Trade-off parameter. 1.0 = pure relevance, 0.0 = pure diversity.
candidate_ids uint64 array No All vectors IDs of candidate vectors. Empty means all vectors in the collection.

Response:

Field Type Description
results array Array of diverse results
compute_time_us uint64 Computation time in microseconds

Each result:

Field Type Description
id uint64 Vector ID
relevance_score float Similarity to the query vector
mmr_score float Final MMR score (combining relevance and diversity) 
curl -X POST http://localhost:8080/api/v1/collections/documents/math/diversity \
  -H "Content-Type: application/json" \
  -d '{
    "query": [0.1, 0.2, 0.3],
    "k": 5,
    "lambda": 0.7,
    "candidate_ids": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
  }'
{
  "results": [
    {"id": 3, "relevance_score": 0.95, "mmr_score": 0.92},
    {"id": 7, "relevance_score": 0.88, "mmr_score": 0.79},
    {"id": 1, "relevance_score": 0.91, "mmr_score": 0.74},
    {"id": 9, "relevance_score": 0.82, "mmr_score": 0.68},
    {"id": 5, "relevance_score": 0.85, "mmr_score": 0.61}
  ],
  "compute_time_us": 340
}

gRPC API

SwarnDB's gRPC API runs on port 50051 and provides the same functionality as the REST API with the performance benefits of Protocol Buffers and HTTP/2.

Proto definitions are located at proto/swarndb/v1/.

Services and RPCs

CollectionService

Defined in proto/swarndb/v1/collection.proto.

RPC Request Response Description
CreateCollection CreateCollectionRequest CreateCollectionResponse Create a collection
DeleteCollection DeleteCollectionRequest DeleteCollectionResponse Delete a collection
GetCollection GetCollectionRequest GetCollectionResponse Get collection info
ListCollections ListCollectionsRequest ListCollectionsResponse List all collections
GetRecoveryStatus GetRecoveryStatusRequest GetRecoveryStatusResponse Server boot-recovery snapshot (path, elapsed, per-collection)
SnapshotCollection SnapshotCollectionRequest SnapshotCollectionResponse Force a synchronous snapshot for a collection
GetPersistenceStatus GetPersistenceStatusRequest GetPersistenceStatusResponse Per-collection snapshot and WAL LSN state
GetCollectionMetrics GetCollectionMetricsRequest GetCollectionMetricsResponse Per-collection lock-contention counters

VectorService

Defined in proto/swarndb/v1/vector.proto.

RPC Request Response Description
Insert InsertRequest InsertResponse Insert a single vector
Get GetVectorRequest GetVectorResponse Get a vector by ID
Update UpdateRequest UpdateResponse Update a vector
Delete DeleteVectorRequest DeleteVectorResponse Delete a vector
BulkInsert stream InsertRequest BulkInsertResponse Stream-based bulk insert
BulkInsertWithOptions stream BulkInsertStreamMessage BulkInsertResponse Stream-based bulk insert with tuning options (first message is options, rest are vectors)
BulkInsertFromPath BulkInsertFromPathRequest BulkInsertResponse Server-side ingestion from a .npy or .f32 file via memory mapping
Optimize OptimizeRequest OptimizeResponse Trigger index/graph rebuild
PruneWAL PruneWALRequest PruneWALResponse Delete WAL segments older than the last snapshot
Compact CompactRequest CompactResponse Merge segments and optionally remove deleted vectors

SearchService

Defined in proto/swarndb/v1/search.proto.

RPC Request Response Description
Search SearchRequest SearchResponse Single search query
BatchSearch BatchSearchRequest BatchSearchResponse Multiple queries

GraphService

Defined in proto/swarndb/v1/graph.proto.

RPC Request Response Description
GetRelated GetRelatedRequest GetRelatedResponse Get connected vectors
Traverse TraverseRequest TraverseResponse Multi-hop graph traversal
SetThreshold SetThresholdRequest SetThresholdResponse Set similarity threshold
PutNode PutNodeRequest PutNodeResponse Create a typed node (hybrid mode)
GetNode GetNodeRequest GetNodeResponse Get a typed node by id (hybrid mode)
UpdateNode UpdateNodeRequest UpdateNodeResponse Update a node's property bag with an audit entry (hybrid mode)
DeleteNode DeleteNodeRequest DeleteNodeResponse Delete a typed node and its edges (hybrid)
PutEdge PutEdgeRequest PutEdgeResponse Create a typed edge, optionally temporal (hybrid mode)
GetEdge GetEdgeRequest GetEdgeResponse Get a typed edge by id (hybrid mode)
DeleteEdge DeleteEdgeRequest DeleteEdgeResponse Delete a typed edge (hybrid mode)
ListEdges ListEdgesRequest ListEdgesResponse List edges incident to a node (hybrid mode)
UpdateEdge UpdateEdgeRequest UpdateEdgeResponse Update a manual edge's properties/confidence/verified
VerifyEdge VerifyEdgeRequest VerifyEdgeResponse Lock an edge against re-extraction
RejectEdge RejectEdgeRequest RejectEdgeResponse Delete an edge and remember the pattern
BulkImportEdges BulkImportEdgesRequest BulkImportEdgesResponse Bulk import edges from CSV or JSONL
EnumerateNodes EnumerateNodesRequest EnumerateNodesResponse Paged, filtered whole-graph node read (cursor over sorted ids, hybrid mode)
EnumerateEdges EnumerateEdgesRequest EnumerateEdgesResponse Paged, filtered whole-graph edge read (cursor over sorted ids, hybrid mode)
HybridQuery HybridQueryRequest HybridQueryResponse Run a composable hybrid query plan

The RPCs above PutNode are unchanged. Everything from PutNode down is additive and active only on hybrid collections; see Typed Graph: Overview.

ExtractionService

Defined in proto/swarndb/v1/extraction.proto. Every RPC is rejected on non-hybrid collections. See LLM Extraction.

RPC Request Response Description
SetLlmConfig SetLlmConfigRequest SetLlmConfigResponse Set the LLM config (api key is write-only)
GetLlmConfig GetLlmConfigRequest GetLlmConfigResponse Get the redacted LLM config (no api key)
RotateLlmConfig RotateLlmConfigRequest RotateLlmConfigResponse Rotate just the api key
SetOntology SetOntologyRequest SetOntologyResponse Set the ontology from a template/extension
GetOntology GetOntologyRequest GetOntologyResponse Get the collection's ontology
CostPreview CostPreviewRequest CostPreviewResponse Estimate token usage and cost
StartExtraction StartExtractionRequest StartExtractionResponse Start an async extraction job
GetExtractionStatus GetExtractionStatusRequest JobStatusMsg Snapshot of an extraction job's progress
CancelExtraction CancelExtractionRequest CancelExtractionResponse Cancel a running extraction job
ListProposals ListProposalsRequest ListProposalsResponse List pending ontology proposals
ApproveProposal ApproveProposalRequest ApproveProposalResponse Approve an ontology proposal
RejectProposal RejectProposalRequest RejectProposalResponse Reject an ontology proposal
DiffDocument DiffDocumentRequest DiffDocumentResponse Diff a document's chunks against stored state
ReextractDocument ReextractDocumentRequest ReextractDocumentResponse Re-extract only changed/new chunks

SetOntology (on SetOntologyRequest.extension) and GetOntology (on GetOntologyResponse.ontology) both carry an OntologyMsg. Besides the entity labels and edge types, it has two optional prompt fields:

Field Type Description
system_prompt string Full override of the generic extraction task framing (field 3). Empty means use the default prompt.
extra_guidance string Domain guidance appended on top of the framing (field 4). Empty means use the default prompt.

On SetOntology these extend the request; on GetOntology they are echoed back in the response. Whatever they hold, SwarnDB always keeps the JSON output schema and the ontology's allowed labels and edge types in the prompt, so a custom prompt cannot break parsing or step outside the ontology. system_prompt (3) and extra_guidance (4) are additive: a client pinned to an older proto simply does not set or read them and keeps working.

GetExtractionStatus returns a JobStatusMsg snapshot of the job:

Field Type Description
job_id string The job id.
collection string The collection being extracted into.
state string queued, running, completed, completed_with_errors, failed, or cancelled.
total_chunks uint64 Total chunks in the job.
processed_chunks uint64 Chunks processed so far.
entities_written uint64 Entities written to the graph.
edges_written uint64 Edges written to the graph.
cache_hits uint64 Chunks served from the extraction cache.
cache_misses uint64 Chunks that called the provider.
error string Job-level error message, set when the whole job failed.
failed_chunks uint64 Total count of chunks that failed (field 11). Zero when the job completed cleanly.
chunk_errors repeated ChunkErrorMsg A bounded sample (up to 100) of the per-chunk failures (field 12). Empty when the job completed cleanly.

A single chunk's failure no longer fails the whole job: the job finishes in the completed_with_errors state, the graph holds everything that succeeded, and failed_chunks / chunk_errors describe what did not. A reply that was cut off at the provider's token limit is retried once automatically with a raised output budget, so most truncations never surface as a failure. The ChunkErrorMsg carries one failure:

Field Type Description
doc_id string The document the chunk came from.
chunk_id uint64 The chunk that failed.
error string Why the chunk failed.

failed_chunks (11), chunk_errors (12), and ChunkErrorMsg are additive: a client pinned to an older proto simply does not read the new fields and keeps working. See LLM Extraction.

VectorMathService

Defined in proto/swarndb/v1/vector_math.proto.

RPC Request Response Description
DetectGhosts DetectGhostsRequest DetectGhostsResponse Find isolated vectors
ConeSearch ConeSearchRequest ConeSearchResponse Directional cone search
ComputeCentroid ComputeCentroidRequest ComputeCentroidResponse Compute mean vector
Interpolate InterpolateRequest InterpolateResponse LERP/SLERP interpolation
DetectDrift DetectDriftRequest DetectDriftResponse Compare vector distributions
Cluster ClusterRequest ClusterResponse K-means clustering
ReduceDimensions ReduceDimensionsRequest ReduceDimensionsResponse PCA dimensionality reduction
ComputeAnalogy ComputeAnalogyRequest ComputeAnalogyResponse Vector analogy/arithmetic
DiversitySample DiversitySampleRequest DiversitySampleResponse MMR diversity sampling

Python gRPC Example

from swarndb import SwarnDBClient

client = SwarnDBClient("localhost:50051")

# Create a collection
client.collections.create(
    name="documents",
    dimension=1536,
    distance_metric="cosine",
    default_threshold=0.8
)

# Insert vectors
client.vectors.insert(
    collection="documents",
    values=[0.1, 0.2, 0.3, ...],  # 1536-dim vector
    metadata={"title": "My Document", "category": "tech"}
)

# Search
results = client.search.query(
    collection="documents",
    vector=[0.1, 0.2, 0.3, ...],
    k=10,
    include_metadata=True,
    include_graph=True,
    graph_threshold=0.85
)

for r in results.results:
    print(f"ID: {r.id}, Score: {r.score}")
    for edge in r.graph_edges:
        print(f"  Related: {edge.target_id} (sim: {edge.similarity})")

# Graph traversal
nodes = client.graph.traverse(
    collection="documents",
    start_id=42,
    depth=3,
    threshold=0.8
)

# Clustering
clusters = client.math.cluster(
    collection="documents",
    k=5,
    max_iterations=100
)

# Async client
from swarndb import AsyncSwarnDBClient

async_client = AsyncSwarnDBClient("localhost:50051")
results = await async_client.search.query(
    collection="documents",
    vector=[0.1, 0.2, 0.3, ...],
    k=10
)

Metadata Types

SwarnDB supports five metadata value types. When setting metadata via the REST API, values are automatically inferred from JSON types. When using gRPC or filter expressions, values must be explicitly typed.

Supported Types

Type Rust Type JSON Representation gRPC Wrapper
String String "hello" {"string_value": "hello"}
Integer i64 42 {"int_value": 42}
Float f64 3.14 {"float_value": 3.14}
Boolean bool true {"bool_value": true}
String List Vec<String> ["a", "b", "c"] {"string_list_value": {"values": ["a", "b"]}}

REST API Metadata Examples

When inserting or updating via REST, metadata is a plain JSON object. Types are inferred:

{
  "metadata": {
    "title": "Vector Databases 101",
    "year": 2024,
    "rating": 4.8,
    "published": true,
    "tags": ["database", "vectors", "search"]
  }
}

Filter Metadata Values

In REST API filter expressions, values are plain JSON (no typed wrappers needed):

{
  "field": "year",
  "op": "gte",
  "value": 2023
}
{
  "field": "rating",
  "op": "between",
  "values": [4.0, 5.0]
}
{
  "field": "published",
  "op": "eq",
  "value": true
}

Note: The gRPC API uses typed wrappers for filter values (e.g., {"int_value": 2023}, {"string_value": "tech"}). See the gRPC API section and the protobuf definitions for the gRPC filter format.


### gRPC Metadata

In gRPC, metadata is a `map<string, MetadataValue>` where each value uses the `oneof` typed wrapper:

```protobuf
message MetadataValue {
  oneof value {
    string string_value = 1;
    int64 int_value = 2;
    double float_value = 3;
    bool bool_value = 4;
    StringList string_list_value = 5;
  }
}

message StringList {
  repeated string values = 1;
}

Authentication

Authentication is optional and only enforced when the SWARNDB_API_KEYS environment variable is set.

Configuration:

# Comma-separated list of valid API keys
export SWARNDB_API_KEYS="key1,key2,key3"

Usage:

Pass the API key in either header format:

# X-API-Key header
curl -H "X-API-Key: key1" http://localhost:8080/api/v1/collections

# Authorization Bearer header
curl -H "Authorization: Bearer key1" http://localhost:8080/api/v1/collections

When SWARNDB_API_KEYS is not set, all requests are allowed without authentication.

When authentication fails, the server returns:

{
  "error": "unauthorized",
  "code": 401
}