feat(api): delete, external ids, and a JSON HTTP API

docs/docs.json

@@ -48,7 +48,8 @@
           {
             "group": "Reference",
             "pages": [
-              "reference/cli"
+              "reference/cli",
+              "reference/http-api"
             ]
           }
         ]

docs/reference/cli.mdx

@@ -36,6 +36,9 @@ changing dimensions.
 | Command | What it does |
 |---|---|
 | `open-index search <query> [-t doc_type]` | Search from the terminal. |
+| `open-index delete <id>` | Delete one entity, its edges and its file (`--yes` to skip the prompt). |
+| `open-index lookup <external-id>` | Find an entity by the id its source system knows it by. |
+| `open-index trace <trace-id>` | What one turn retrieved: each query, each document, rank and score. |
 | `open-index ui` | Launch the explorer (How to use / Schema / Explore / **Map** / Analytics / Jobs). |
 
 ## Serving over MCP

docs/reference/http-api.mdx

@@ -0,0 +1,131 @@
+---
+title: HTTP API
+description: The JSON API over a brain — the same operations MCP exposes, for everything that is not an agent.
+---
+
+MCP is how agents talk to an index. This is how everything else does: a script,
+a cron job, a service, `curl`.
+
+It mirrors the MCP tools rather than inventing a second vocabulary, and calls the
+same `Brain` methods underneath — so the two cannot disagree about what a search
+means or what a write validates.
+
+The API is served by `open-index ui`, alongside the explorer, so one process and
+one port serve the UI, the API and the MCP endpoint for every index on the host.
+
+```
+https://<host>/<index>/api/v1/...     one of several indexes
+http://localhost:8501/api/v1/...      a single brain
+```
+
+## Reading
+
+### Search
+
+```bash
+curl "$HOST/api/v1/search?q=payment+declined&mode=hybrid&limit=10"
+```
+
+| Parameter | |
+|---|---|
+| `q` | free text; omit it to list |
+| `mode` | `hybrid` (default), `keyword`, `semantic` |
+| `doc_type` | repeatable — `&doc_type=issue&doc_type=product` |
+| `filter.<field>` | exact match, e.g. `filter.tenant_id=acme` |
+| `limit` | default 20 |
+
+Every result carries `match`, saying **why** it came back:
+
+```json
+{
+  "id": "issue:payment-declined",
+  "score": 0.937,
+  "match": { "type": "both", "keyword_score": 1.0, "semantic_score": 0.79 }
+}
+```
+
+`type` is `keyword`, `semantic`, `both`, `filter` or `none`. Read it before
+trusting a result — a semantic-only hit at a low score is a guess, not a fact.
+
+**Filters are hard predicates, not ranking hints.** A document that does not
+match cannot be returned at any score, in any mode. Only fields declared
+`filterable: true` can be filtered, and filtering on any other field returns
+`400` rather than being ignored — so a filter never silently fails open. This
+matters when the filter is carrying a tenant or user boundary.
+
+### Entities
+
+```bash
+curl "$HOST/api/v1/entities/issue:payment-declined"
+curl "$HOST/api/v1/entities?id=issue:a&id=issue:b"        # batch
+curl "$HOST/api/v1/entities/by-external-id/CRM-4471"      # your own id
+curl "$HOST/api/v1/schema"
+```
+
+A single entity comes back with its relationships in **both** directions —
+incoming answers "what else points at this?", which the entity's own document
+cannot.
+
+The batch endpoint reports `missing` rather than padding the list with nulls.
+
+### Traces
+
+```bash
+curl "$HOST/api/v1/traces/turn-8f21c3"
+```
+
+Every read records the `X-Trace-Id` header it arrived with. Send one and you can
+later recover exactly which documents that turn retrieved, at what rank, with
+what score, and on what kind of match — which is how you work out afterwards what
+the index actually fed an agent.
+
+## Writing
+
+```bash
+curl -X PUT "$HOST/api/v1/entities/issue:payment-declined" \
+  -H 'Content-Type: application/json' \
+  -d '{"doc_type": "issue", "name": "Payment declined", "severity": "high",
+       "external_id": "CRM-4471"}'
+
+curl -X DELETE "$HOST/api/v1/entities/issue:payment-declined"
+```
+
+`PUT` is an upsert. The id in the URL wins: a body id that disagrees is a `400`,
+so `PUT /entities/a` can never write entity `b`.
+
+`DELETE` removes the entity, its file (for a `storage: file` doc_type), and every
+edge naming it in either direction. Deleting only the index row would leave the
+file to resurrect it on the next reindex — a pause, not a delete.
+
+Prefer correcting an entity over deleting it: an id that once resolved and now
+404s breaks anything still holding a reference.
+
+## Authentication
+
+Off unless a token is configured. Set `OPEN_INDEX_TOKEN` (or a per-index
+`OPEN_INDEX_TOKEN_<NAME>`) and **writes** require it:
+
+```bash
+curl -X PUT "$HOST/api/v1/entities/issue:x" \
+  -H "Authorization: Bearer $OPEN_INDEX_TOKEN" ...
+```
+
+Reads stay open either way, matching how the MCP endpoint already behaves. A
+deployment holding real data sets a token; one serving public demo data does not,
+and nothing changes for it.
+
+<Warning>
+With no token set, anyone who can reach the endpoint can write to it and delete
+from it. That is the right default for disposable demo data and the wrong one for
+anything you would miss.
+</Warning>
+
+## Status codes
+
+| | |
+|---|---|
+| `400` | your request — bad mode, unfilterable field, malformed body, id mismatch |
+| `401` | a write without a valid token, when one is configured |
+| `404` | unknown index, entity, or external id |
+| `422` | a well-formed entity that failed schema validation |
+| `500` | the write could not complete; the message says what state things are in |

open_index/api.py

@@ -0,0 +1,220 @@
+"""A JSON HTTP API over a brain — the same operations MCP exposes, over HTTP.
+
+MCP is for agents; this is for everything else: a script, a job, a service, a
+`curl`. It deliberately mirrors the MCP tools rather than inventing a second
+vocabulary, and calls the same `Brain` methods, so the two cannot disagree about
+what a search means or what a write validates.
+
+Mounted at `/<index>/api/v1` alongside the explorer, so one process and one port
+serve the UI, the API and the MCP endpoint for every brain on the host.
+
+Auth is off unless a token is configured. `OPEN_INDEX_TOKEN` (or the per-brain
+`OPEN_INDEX_TOKEN_<NAME>`) gates *writes* only — reads stay open, matching how
+the MCP endpoint already behaves. A deployment holding real data sets the token;
+one serving public demo data does not, and nothing changes for it.
+"""
+
+from __future__ import annotations
+
+import os
+from typing import Any, Optional
+
+from open_index.brain import Brain
+
+
+def token_for(name: Optional[str]) -> Optional[str]:
+    """The write token for this brain, if one is configured."""
+    if name:
+        specific = os.environ.get(
+            "OPEN_INDEX_TOKEN_" + name.upper().replace("-", "_").replace(".", "_"))
+        if specific:
+            return specific
+    return os.environ.get("OPEN_INDEX_TOKEN") or None
+
+
+def _entity_payload(brain: Brain, entity) -> dict[str, Any]:
+    """One entity as the API returns it: the document plus both edge directions."""
+    payload = entity.to_json()
+    payload["relationships"] = {
+        "outgoing": [{"target": t, "meaning": m}
+                     for (_s, t, m) in brain.backend.relationships_from(entity.id)],
+        "incoming": [{"source": s, "meaning": m}
+                     for (s, _t, m) in brain.backend.relationships_to(entity.id)],
+    }
+    return payload
+
+
+def build_routes(resolve, prefix: str = ""):
+    """Routes for the JSON API.
+
+    `resolve(request)` returns `(name, Brain)` — supplied by the caller so the
+    API and the explorer agree on which index a path refers to instead of each
+    working it out separately.
+    """
+    from starlette.responses import JSONResponse
+    from starlette.routing import Route
+
+    def error(message: str, status: int, **extra):
+        return JSONResponse({"error": message, **extra}, status_code=status)
+
+    def authorized(request, name: Optional[str]) -> bool:
+        expected = token_for(name)
+        if not expected:
+            return True                     # no token configured: writes are open
+        header = request.headers.get("authorization", "")
+        scheme, _, value = header.partition(" ")
+        return scheme.lower() == "bearer" and value == expected
+
+    def with_brain(handler, *, write: bool = False):
+        async def endpoint(request):
+            name, brain = resolve(request)
+            if brain is None:
+                return error("unknown index", 404)
+            if write and not authorized(request, name):
+                # 401 with a challenge, not 403: the caller can fix this by
+                # presenting a token, and should be told how.
+                return JSONResponse(
+                    {"error": "a bearer token is required to write to this index"},
+                    status_code=401,
+                    headers={"WWW-Authenticate": 'Bearer realm="open-index"'},
+                )
+            return await handler(request, brain)
+        return endpoint
+
+    # -- reads ----------------------------------------------------------------
+
+    async def search(request, brain: Brain):
+        params = request.query_params
+        try:
+            limit = int(params.get("limit", 20))
+        except ValueError:
+            return error("limit must be an integer", 400)
+
+        filters: dict[str, Any] = {}
+        # filter.<field>=<value>, so a filter needs no JSON body on a GET.
+        for key, value in params.multi_items():
+            if key.startswith("filter."):
+                filters[key[len("filter."):]] = value
+
+        try:
+            results = brain.search(
+                query=params.get("q"),
+                doc_types=[t for t in params.getlist("doc_type") if t] or None,
+                limit=limit,
+                mode=params.get("mode", "hybrid"),
+                filters=filters or None,
+                source="api",
+            )
+        except ValueError as exc:
+            # An unknown mode or a filter on an undeclared field. The caller can
+            # fix both, and the message says how — so it is a 400, not a 500.
+            return error(str(exc), 400)
+
+        return JSONResponse({
+            "query": params.get("q"),
+            "mode": params.get("mode", "hybrid"),
+            "filters": filters,
+            "total": results.total,
+            "doc_type_counts": results.doc_type_counts,
+            "limited": results.limited,
+            "results": results.results,
+        })
+
+    async def get_one(request, brain: Brain):
+        entity_id = request.path_params["entity_id"]
+        entity = brain.get_entity(entity_id, source="api")
+        if entity is None:
+            return error(f"no entity '{entity_id}'", 404)
+        return JSONResponse(_entity_payload(brain, entity))
+
+    async def get_many(request, brain: Brain):
+        ids = [i for i in request.query_params.getlist("id") if i]
+        found = brain.get_entities(ids, source="api")
+        return JSONResponse({
+            "requested": len(ids),
+            "found": len(found),
+            "missing": sorted(set(ids) - {e.id for e in found}),
+            "entities": [_entity_payload(brain, e) for e in found],
+        })
+
+    async def by_external(request, brain: Brain):
+        external_id = request.path_params["external_id"]
+        entity = brain.get_by_external_id(external_id, source="api")
+        if entity is None:
+            return error(f"no entity with external_id '{external_id}'", 404)
+        return JSONResponse(_entity_payload(brain, entity))
+
+    async def schema(request, brain: Brain):
+        from open_index.config import doc_type_to_yaml_dict
+
+        counts = brain.counts()
+        return JSONResponse({
+            "name": brain.config.name,
+            "description": brain.config.description,
+            "doc_types": [
+                {**doc_type_to_yaml_dict(dt), "count": counts.get(name, 0)}
+                for name, dt in brain.config.doc_types.items()
+            ],
+        })
+
+    async def trace_lookup(request, brain: Brain):
+        trace_id = request.path_params["trace_id"]
+        return JSONResponse({"trace_id": trace_id,
+                             "reads": brain.analytics_by_trace(trace_id)})
+
+    # -- writes ---------------------------------------------------------------
+
+    async def put_one(request, brain: Brain):
+        from open_index.models import Entity
+
+        entity_id = request.path_params["entity_id"]
+        try:
+            body = await request.json()
+        except Exception:
+            return error("body must be JSON", 400)
+        if not isinstance(body, dict):
+            return error("body must be a JSON object", 400)
+
+        body = dict(body)
+        # The URL is the authority on which entity this is. Accepting a body id
+        # that disagrees would let PUT /entities/a write entity b.
+        if body.get("id") not in (None, entity_id):
+            return error("id in the body does not match the URL", 400,
+                         url_id=entity_id, body_id=body.get("id"))
+        body["id"] = entity_id
+        body.setdefault("doc_type", entity_id.split(":", 1)[0])
+
+        try:
+            entity = Entity.from_dict(body)
+        except Exception as exc:
+            return error(f"invalid entity: {exc}", 400)
+        try:
+            path = brain.put_entity(entity)
+        except ValueError as exc:
+            return error(str(exc), 422)
+        return JSONResponse({"written": entity.id,
+                             "file": str(path) if path else None})
+
+    async def delete_one(request, brain: Brain):
+        entity_id = request.path_params["entity_id"]
+        try:
+            deleted = brain.delete_entity(entity_id, source="api")
+        except RuntimeError as exc:
+            return error(str(exc), 500)
+        if not deleted:
+            return error(f"no entity '{entity_id}'", 404)
+        return JSONResponse({"deleted": entity_id})
+
+    p = prefix
+    return [
+        Route(f"{p}/search", with_brain(search)),
+        Route(f"{p}/schema", with_brain(schema)),
+        Route(f"{p}/entities", with_brain(get_many)),
+        Route(f"{p}/entities/by-external-id/{{external_id:path}}", with_brain(by_external)),
+        Route(f"{p}/traces/{{trace_id}}", with_brain(trace_lookup)),
+        Route(f"{p}/entities/{{entity_id:path}}", with_brain(get_one), methods=["GET"]),
+        Route(f"{p}/entities/{{entity_id:path}}", with_brain(put_one, write=True),
+              methods=["PUT"]),
+        Route(f"{p}/entities/{{entity_id:path}}", with_brain(delete_one, write=True),
+              methods=["DELETE"]),
+    ]