From f0a88f2b48aaace59452907bc86102784980a4fc Mon Sep 17 00:00:00 2001 From: dimknaf <136385722+dimknaf@users.noreply.github.com> Date: Fri, 12 Jun 2026 00:21:59 +0100 Subject: [PATCH 1/5] feat(db): bundled internal Postgres behind one .env flag New braindb_db service (pgvector/pg17) guarded by the internal-db compose profile. .env.example ships COMPOSE_PROFILES=internal-db, so a fresh clone gets a working database with plain `docker compose up -d` - nothing to install. Without that line the service does not exist and existing setups run exactly as before: DATABASE_URL keeps pointing at your own Postgres. DATABASE_URL defaults to the bundled DSN via nested interpolation; an explicit value always wins. Optional knobs: POSTGRES_PORT (host inspection port, loopback-only, default 5435), POSTGRES_DB, POSTGRES_USER, POSTGRES_PASSWORD. The api waits on the DB healthcheck via depends_on with required: false, so one compose file serves both modes. Requires Compose >= 2.20. Closes #12. --- .env.example | 29 +++++++++++++++++++++++++-- CHANGELOG.md | 26 ++++++++++++++++++++++++ README.md | 30 +++++++++++++++++++--------- docker-compose.yml | 49 +++++++++++++++++++++++++++++++++++++++++++++- 4 files changed, 122 insertions(+), 12 deletions(-) diff --git a/.env.example b/.env.example index b091ede..bf49f52 100644 --- a/.env.example +++ b/.env.example @@ -2,8 +2,33 @@ # Advanced tuning knobs (decay rates, graph depth, agent turn limits) have # reasonable defaults — see braindb/config.py for the full list. -# PostgreSQL connection string (psycopg2 format) -DATABASE_URL=postgresql://user:password@host:5432/braindb +# ============================================================ +# DATABASE — pick ONE of the two modes +# ============================================================ + +# --- Mode 1: Internal DB (recommended / default) ------------ +# BrainDB runs its own Postgres container — nothing to install. +# This single line is the switch; remove it for Mode 2: +COMPOSE_PROFILES=internal-db + +# Optional knobs for the internal DB. They are IGNORED in Mode 2 +# (external connections are configured entirely via DATABASE_URL). +# NOTE: user/password/db-name are baked into the data volume on its +# FIRST boot. To reset the internal DB later (wipes its data!): +# docker rm -f braindb_db && docker volume rm braindb_pgdata +# POSTGRES_PORT=5435 # host port, only for inspecting the DB with +# # psql/adminer; the stack itself doesn't need +# # it. Change if 5435 is taken on your machine. +# POSTGRES_DB=braindb +# POSTGRES_USER=braindb +# POSTGRES_PASSWORD=braindb # change for anything beyond local use + +# --- Mode 2: External DB (bring your own Postgres) ---------- +# Your Postgres needs the pgvector and pg_trgm extensions. +# 1. Remove (or comment out) the COMPOSE_PROFILES line above. +# 2. Put EVERYTHING — host, port, db name, credentials — in this +# one URL. The POSTGRES_* knobs above have no effect here. +# DATABASE_URL=postgresql://user:password@your-host:5432/braindb # API port (default 8000) API_PORT=8000 diff --git a/CHANGELOG.md b/CHANGELOG.md index 03c0d46..36f7de7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,32 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [Unreleased] + +### Added + +- **Bundled internal Postgres (opt-in by one `.env` line).** A `braindb_db` + service (pgvector/pg17) guarded by the `internal-db` compose profile. + `.env.example` now ships `COMPOSE_PROFILES=internal-db`, so a fresh clone + gets a working database with plain `docker compose up -d` — nothing to + install. Existing setups are untouched: without that line the service does + not exist and `DATABASE_URL` keeps pointing at your own Postgres exactly as + before. Optional knobs: `POSTGRES_PORT` (host inspection port, default + 5435, loopback-only), `POSTGRES_DB`, `POSTGRES_USER`, `POSTGRES_PASSWORD`. + Requires Docker Compose >= 2.20. Closes the request in issue #12. + +### Fixed + +- **Test suite no longer touches your database — at all.** The suite now + runs against an isolated test stack (`docker-compose.test.yml`: its own + API on port 8002 + its own throwaway Postgres), and the session-end + cleanup that deleted pattern-matched rows from the configured database + has been removed entirely. Previously the tests ran against the live + personal stack and their cleanup silently failed from the host, leaking + `_pytest_*` keyword entities into real data (where the wiki maintainer + wasted LLM calls triaging them). Now: `docker compose -f + docker-compose.test.yml up -d && pytest`, and `down -v` wipes everything. + ## [0.5.0] — 2026-06-09 ### Changed diff --git a/README.md b/README.md index d664d42..87fe560 100644 --- a/README.md +++ b/README.md @@ -37,13 +37,12 @@ Relations connect any two entities with `relation_type`, `relevance_score`, `imp ## Setup -BrainDB runs as three Docker services — `api`, `watcher` (auto-ingests files), and `wiki_scheduler` (auto-maintains wikis) — against an **external** PostgreSQL you provide. The two sidecars are hands-off: you never call the pipeline by hand. The whole setup is six steps. +BrainDB runs as Docker services — `api`, `watcher` (auto-ingests files), and `wiki_scheduler` (auto-maintains wikis) — plus a database that is either **bundled** (default, zero install) or **your own PostgreSQL**. The two sidecars are hands-off: you never call the pipeline by hand. ### 1. Prerequisites -- Docker Desktop (or any Docker Engine) -- A PostgreSQL 16 instance reachable from Docker (see step 3 for three common options) -- The PostgreSQL extensions `pg_trgm` and `pgvector` must exist on the target database, and the connecting user must have permission to create them on first connection (migrations will `CREATE EXTENSION IF NOT EXISTS` on startup). If you don't have DB admin rights, ask an admin to pre-install both extensions. +- Docker Desktop (or Docker Engine) with Compose v2.20+ +- That's it for the default setup — the database ships with the stack. ### 2. Clone and configure @@ -53,9 +52,19 @@ cd braindb cp .env.example .env ``` -### 3. Point `.env` at your PostgreSQL +### 3. Choose your database mode -Edit `.env` and set `DATABASE_URL`. The value depends on **where your Postgres runs**: +`.env.example` ships with the **internal DB** enabled — a bundled Postgres (with pgvector) that starts as part of the stack. For most users there is nothing to do in this step. + +The switch is one line in `.env`: + +``` +COMPOSE_PROFILES=internal-db # present = bundled DB starts automatically +``` + +Optional internal-DB knobs (defaults are fine): `POSTGRES_PORT` (host port for inspecting the DB with psql/adminer, default 5435), `POSTGRES_DB`, `POSTGRES_USER`, `POSTGRES_PASSWORD`. Note: these are baked into the data volume on first boot — to reset later: `docker rm -f braindb_db && docker volume rm braindb_pgdata` (wipes its data). + +**Bring your own Postgres instead?** Remove the `COMPOSE_PROFILES` line and set `DATABASE_URL` — everything (host, port, db name, credentials) lives in that one URL: **Option A — Postgres running as another Docker container on the same network** (e.g. a `postgres_container`): ``` @@ -72,7 +81,8 @@ DATABASE_URL=postgresql://postgres:password@host.docker.internal:5432/braindb ``` DATABASE_URL=postgresql://user:password@db.example.com:5432/braindb ``` -Any reachable hostname/IP works — the connecting user just needs network access, auth, and the extensions mentioned in step 1. + +External databases need the `pg_trgm` and `pgvector` extensions; the connecting user must be able to create them on first connection (migrations run `CREATE EXTENSION IF NOT EXISTS` on startup), or an admin pre-installs both. ### 4. Pick an LLM provider (for the internal agent) @@ -91,14 +101,16 @@ Adding a third provider (Together, OpenAI, local vLLM, whatever) is a two-line e ### 5. Create the Docker network, then bring the stack up -`docker-compose.yml` expects an external network called `local-network` so the `api` and `watcher` containers can reach your Postgres (and each other) by DNS name: +`docker-compose.yml` expects an external network called `local-network` so the containers can reach each other (and an external Postgres, if you use one) by DNS name: ```bash docker network create local-network # one-time, ignore error if it already exists docker compose up -d --build ``` -If your Postgres is a container (Option A in step 3), attach it to this network too: +With the internal DB (default), this also starts the bundled Postgres and waits for it to be healthy before the API boots. + +If you brought your own Postgres as a container (Option A in step 3), attach it to this network too: ```bash docker network connect local-network postgres_container ``` diff --git a/docker-compose.yml b/docker-compose.yml index 04e2f43..5390213 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -5,8 +5,19 @@ services: restart: unless-stopped networks: - local-network + # Starts the bundled DB first when the internal-db profile is active + # (COMPOSE_PROFILES=internal-db in .env). With the profile inactive the + # dependency is skipped entirely — external-DB setups are unaffected. + # `required: false` needs Docker Compose >= 2.20. + depends_on: + braindb_db: + condition: service_healthy + required: false environment: - DATABASE_URL: ${DATABASE_URL} + # Set DATABASE_URL in .env to use your own Postgres (external mode). + # Leave it unset with COMPOSE_PROFILES=internal-db and the api connects + # to the bundled braindb_db service below. + DATABASE_URL: ${DATABASE_URL:-postgresql://${POSTGRES_USER:-braindb}:${POSTGRES_PASSWORD:-braindb}@braindb_db:5432/${POSTGRES_DB:-braindb}} API_PORT: ${API_PORT:-8000} HF_TOKEN: ${HF_TOKEN:-} LLM_PROFILE: ${LLM_PROFILE:-deepinfra} @@ -58,6 +69,37 @@ services: # cron -> maintain -> write so wikis self-organise from entities with zero # manual steps. To run without it (e.g. cost control), start the stack # excluding this service or scale it to 0 — exactly as you would the watcher. + # Bundled Postgres (pgvector) — exists ONLY when the internal-db profile is + # active, i.e. when .env contains COMPOSE_PROFILES=internal-db. Without that + # line this service is invisible to every compose command and external-DB + # setups behave exactly as before. Named braindb_db (not "postgres") so its + # DNS name can't collide with a user's own Postgres on local-network. + # POSTGRES_* values are baked into the volume on FIRST boot; to reset: + # docker rm -f braindb_db && docker volume rm braindb_pgdata + braindb_db: + image: pgvector/pgvector:pg17 + container_name: braindb_db + profiles: ["internal-db"] + restart: unless-stopped + environment: + POSTGRES_USER: ${POSTGRES_USER:-braindb} + POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-braindb} + POSTGRES_DB: ${POSTGRES_DB:-braindb} + ports: + # Loopback only — for inspecting the DB from the host (psql/adminer). + # The stack itself talks to braindb_db over the docker network and does + # not need this mapping. Change POSTGRES_PORT in .env if 5435 is taken. + - "127.0.0.1:${POSTGRES_PORT:-5435}:5432" + volumes: + - braindb_pgdata:/var/lib/postgresql/data + healthcheck: + test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER:-braindb} -d ${POSTGRES_DB:-braindb}"] + interval: 5s + timeout: 3s + retries: 10 + networks: + - local-network + wiki_scheduler: build: . container_name: braindb_wiki_scheduler @@ -74,6 +116,11 @@ services: - .:/app command: python -m braindb.wiki_scheduler +volumes: + # Data volume for the bundled internal DB (internal-db profile only). + braindb_pgdata: + name: braindb_pgdata + networks: local-network: external: true From 3f577daf9c4728b7d51c6f1043a4be54ceea5cd2 Mon Sep 17 00:00:00 2001 From: dimknaf <136385722+dimknaf@users.noreply.github.com> Date: Fri, 12 Jun 2026 00:21:59 +0100 Subject: [PATCH 2/5] test: isolated test stack - suite runs against its own DB, never the user's The suite previously ran against the live personal stack at :8000 and relied on pattern-matched cleanup of what it created; the cleanup silently failed from the host and test keywords leaked into real data. Deleting rows from whatever database the user has configured is the wrong design regardless of pattern. Now: docker-compose.test.yml runs a self-contained stack (API on 8002, throwaway Postgres braindb_test on host port 5436, own project namespace, no watcher/scheduler so tests are deterministic). conftest defaults to it and fails fast with instructions if it isn't up - bare pytest can no longer reach a personal database. The session-end deletion sweep is removed entirely; `down -v` wipes all test data. Ingest tests write into data_test/ (the test stack's data dir), the direct-DB graph tests default to the test Postgres and skip with a clear reason when it is unreachable, and test keywords consistently use the _pytest_ prefix. --- .gitignore | 3 + CONTRIBUTING.md | 10 ++- docker-compose.test.yml | 65 ++++++++++++++ tests/README.md | 45 ++++++---- tests/conftest.py | 97 +++++++-------------- tests/test_graph_scoring_uses_importance.py | 24 ++++- tests/test_graph_walks_both_directions.py | 24 ++++- tests/test_ingest.py | 11 ++- 8 files changed, 186 insertions(+), 93 deletions(-) create mode 100644 docker-compose.test.yml diff --git a/.gitignore b/.gitignore index b10de4a..350887f 100644 --- a/.gitignore +++ b/.gitignore @@ -58,3 +58,6 @@ benchmarks/*/answers/ # Bench compose: separate host data dir for the bench watcher, separate env file data_bench/ .env.bench + +# Test stack (docker-compose.test.yml): separate host data dir for test ingests +data_test/ diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 01efdca..0a37e15 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -4,13 +4,13 @@ Thanks for your interest. BrainDB is a small, opinionated project — the bar is ## Dev setup -Prerequisites: Docker Desktop (or any Docker Engine), Python 3.12, a Postgres 16 instance reachable from the container. +Prerequisites: Docker Desktop (or Docker Engine) with Compose v2.20+, Python 3.12. The database is bundled by default (`COMPOSE_PROFILES=internal-db` in `.env.example`); to use your own Postgres instead, remove that line and set `DATABASE_URL` — see the README's Setup section. ```bash git clone braindb cd braindb cp .env.example .env -# edit .env — set DATABASE_URL; recommended LLM_PROFILE=deepinfra + DEEPINFRA_API_KEY (or any other profile) +# edit .env — recommended LLM_PROFILE=deepinfra + DEEPINFRA_API_KEY (or any other profile) docker network create local-network # one-time; docker-compose expects this docker compose up -d --build @@ -26,10 +26,14 @@ pip install -e ".[dev]" ## Running tests +The suite runs against an **isolated test stack** (its own API on port 8002 + its own throwaway Postgres) — never against your personal BrainDB database: + ```bash -pytest # full suite (needs the stack up) +docker compose -f docker-compose.test.yml up -d # start the test stack +pytest # full suite pytest -k "not agent" # skip the live-LLM smoke tests pytest tests/test_split_chunks.py # a single file +docker compose -f docker-compose.test.yml down -v # stop + wipe test data ``` See [`tests/README.md`](tests/README.md) for what is and isn't covered. diff --git a/docker-compose.test.yml b/docker-compose.test.yml new file mode 100644 index 0000000..d095836 --- /dev/null +++ b/docker-compose.test.yml @@ -0,0 +1,65 @@ +# Isolated test stack — what `pytest` runs against. NEVER your real data. +# +# docker compose -f docker-compose.test.yml up -d # start +# pytest # run the suite +# docker compose -f docker-compose.test.yml down -v # stop + wipe test data +# +# Own project namespace, own Postgres + volume, own network, own ports +# (api 8002, db 5436 — the personal stack uses 8000, the bench 8001/5434). +# No watcher and no wiki scheduler: tests exercise the API directly, and +# with no background extraction the suite is deterministic. +# +# LLM credentials pass through from the repo .env so the live-agent smoke +# test works; everything else is fully isolated from the personal stack. + +name: braindb_test + +services: + braindb_test_db: + image: pgvector/pgvector:pg17 + container_name: braindb_test_db + environment: + POSTGRES_USER: braindb + POSTGRES_PASSWORD: braindb + POSTGRES_DB: braindb_test + ports: + # Loopback-only. Published so the handful of direct-DB tests + # (e.g. graph_expand) can connect from the host. + - "127.0.0.1:5436:5432" + volumes: + - braindb_test_pgdata:/var/lib/postgresql/data + healthcheck: + test: ["CMD-SHELL", "pg_isready -U braindb -d braindb_test"] + interval: 5s + timeout: 3s + retries: 10 + + braindb_test_api: + build: . + container_name: braindb_test_api + depends_on: + braindb_test_db: + condition: service_healthy + environment: + DATABASE_URL: postgresql://braindb:braindb@braindb_test_db:5432/braindb_test + API_PORT: 8002 + HF_TOKEN: ${HF_TOKEN:-} + LLM_PROFILE: ${LLM_PROFILE:-deepinfra} + AGENT_MODEL: ${AGENT_MODEL:-} + NVIDIA_NIM_API_KEY: ${NVIDIA_NIM_API_KEY:-} + DEEPINFRA_API_KEY: ${DEEPINFRA_API_KEY:-} + VLLM_API_KEY: ${VLLM_API_KEY:-} + AGENT_VERBOSE: "false" + extra_hosts: + - "host.docker.internal:host-gateway" + ports: + - "8002:8002" + volumes: + - .:/app + - ./data_test:/app/data + command: > + sh -c "alembic upgrade head && uvicorn braindb.main:app --host 0.0.0.0 --port 8002" + +volumes: + braindb_test_pgdata: + name: braindb_test_pgdata diff --git a/tests/README.md b/tests/README.md index 7b9cf2e..7094ebc 100644 --- a/tests/README.md +++ b/tests/README.md @@ -1,14 +1,16 @@ # BrainDB test suite -Integration tests that exercise the real HTTP API against a live PostgreSQL and, for the agent smoke tests, a live LLM provider. No mocks, no stubs — tests hit the running stack. +Integration tests that exercise the real HTTP API against a live PostgreSQL and, for the agent smoke tests, a live LLM provider. No mocks, no stubs. + +The suite runs against an **isolated test stack** — its own API (port 8002) and its own throwaway Postgres (`braindb_test`, host port 5436). It never touches your personal BrainDB database; tearing the stack down wipes all test data. ## Prerequisites -The BrainDB stack must be up and healthy: +Start the test stack: ```bash -docker compose up -d -curl http://localhost:8000/health # must return {"status":"ok","embeddings":true} +docker compose -f docker-compose.test.yml up -d +curl http://localhost:8002/health # must return {"status":"ok","embeddings":true} ``` Dev dependencies: @@ -17,6 +19,12 @@ Dev dependencies: pip install -e ".[dev]" ``` +When you're done (stops the stack and wipes all test data): + +```bash +docker compose -f docker-compose.test.yml down -v +``` + ## Running ```bash @@ -27,43 +35,42 @@ pytest -k "not agent" # skip the LLM-dependent agent smoke tests pytest -x # stop on first failure ``` -You can point tests at a non-default URL: +You can point tests at a non-default URL (only do this for another *disposable* stack — never a database with real data): ```bash -BRAINDB_TEST_URL=http://other-host:8000 pytest +BRAINDB_TEST_URL=http://other-host:8002 pytest ``` +A few tests drive internal services (e.g. `graph_expand`) over a direct DB connection; they default to the test stack's Postgres at `localhost:5436` and skip with a clear reason if it isn't reachable. + ## What is covered | File | What it tests | |---|---| -| `test_split_chunks.py` | Pure function — empty text, single word, exact boundary, overlap correctness, misconfigured overlap degrades safely, word preservation. | +| `test_split_chunks.py` | Pure function — empty text, single word, exact boundary, overlap correctness, misconfigured overlap degrades safely, word preservation, byte offsets. | | `test_entities.py` | CRUD round-trip for all 5 entity types (fact, thought, source, datasource, rule). PATCH field isolation, DELETE idempotency, 404 on missing, list filters by type and keyword. | | `test_relations.py` | Relation CRUD, inbound + outbound listing on an entity, PATCH updates, DELETE, cascade on entity deletion, invalid type rejection, all 8 documented relation types accepted. | | `test_search.py` | `/memory/search` finds created content, `/memory/context` structure, multi-query seed merging, graph traversal surfaces connected entities, `/memory/tree` returns a structure, `/memory/stats` returns counts. | -| `test_ingest.py` | `/datasources/ingest` — 201 new, 200 duplicate (by content_hash), dup preserves first-seen metadata (second call doesn't overwrite). | -| `test_agent.py` | `/agent/query` smoke — 200 with an `answer` field on a trivial prompt, 4xx on empty or missing query. | - -Every test that creates entities self-registers its IDs and cleans up in teardown. Your existing data (the real facts and datasources in the DB) is not touched. +| `test_ingest.py` | `/datasources/ingest` — 201 new, 200 duplicate (by content_hash), dup preserves first-seen metadata (second call doesn't overwrite). Test files live in `data_test/` (the test stack's data dir). | +| `test_agent.py` | `/agent/query` smoke — 200 with an `answer` field on a trivial prompt, 4xx on empty or missing query. Needs a configured LLM provider (credentials pass through from `.env`). | ## What is NOT covered Intentional gaps so the suite stays reliable and fast: - **Agent LLM output quality** — the agent smoke test only checks that the endpoint returns a well-formed response. It doesn't assert anything about the answer's content, because LLM output varies and external providers can be flaky. -- **End-to-end watcher pipeline** — dropping a file in `data/sources/` and waiting for the chunked fact-extraction + central review to complete is slow (~5 min) and depends on the LLM being responsive. Manually verified during Phase A with the Smart Sand article; re-run when the watcher logic changes. -- **Datasource content guardrail via the agent tool** — lives in `braindb/agent/tools.py::update_entity`. Testing it cleanly requires driving the agent loop, which needs the LLM. The behavior was manually verified during Phase A (content preserved across three watcher runs). -- **Alembic migrations** — run once at container startup. Not exercised in the Python test suite. +- **End-to-end watcher pipeline** — the test stack deliberately runs no watcher (no background extraction → deterministic tests). Watcher behavior is verified manually when its logic changes. +- **Datasource content guardrail via the agent tool** — lives in `braindb/agent/tools.py::update_entity`. Testing it cleanly requires driving the agent loop, which needs the LLM. +- **Alembic migrations** — run once at container startup (the test stack exercises them on every fresh boot, but the suite doesn't assert on them). - **Embeddings generation** — slow model load; covered implicitly by any search test that matches a seeded keyword but not asserted specifically. ## Expected runtime -- Without agent tests: **under 10s** on a warm stack +- Without agent tests: **under 30s** on a warm stack - With agent tests: **30–90s** depending on provider latency ## If tests fail -1. `docker logs braindb_api --tail 50` — the API may have errored or be mid-reload -2. `docker logs braindb_watcher --tail 50` — the watcher sidecar shouldn't be relevant for tests but look if confused -3. Health check: `curl http://localhost:8000/health` -4. Fresh state: `docker compose restart api` (picks up bind-mount code changes if you've been editing) +1. `docker logs braindb_test_api --tail 50` — the API may have errored +2. Health check: `curl http://localhost:8002/health` +3. Fresh state: `docker compose -f docker-compose.test.yml down -v && docker compose -f docker-compose.test.yml up -d` — brand-new empty test DB diff --git a/tests/conftest.py b/tests/conftest.py index 8f573d9..83c421c 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -1,15 +1,19 @@ """ Shared pytest fixtures for BrainDB integration tests. -These tests run against a live, running BrainDB stack (`docker compose up -d`). -They don't mock the API or the DB — they exercise the real HTTP endpoints and -real PostgreSQL. Each test self-registers the entity IDs it creates so a -session-scoped teardown can delete exactly those, leaving your real data -untouched. +The suite runs against the ISOLATED TEST STACK — its own API + its own +throwaway Postgres, never your personal BrainDB database. Start it first: -Requirements to run the suite: - - API reachable at http://localhost:8000 (override with BRAINDB_TEST_URL) - - A healthy stack (/health returns 200) + docker compose -f docker-compose.test.yml up -d + +then run `pytest`, and tear down (wiping all test data) with: + + docker compose -f docker-compose.test.yml down -v + +The tests exercise the real HTTP endpoints and real PostgreSQL — no mocks. +Each test registers the entity IDs it creates and deletes them at teardown, +but because the whole database is disposable, nothing depends on cleanup +being perfect. Nothing here touches the agent's LLM backend; tests that hit /agent/query send trivial prompts and don't rely on any specific model. @@ -25,7 +29,16 @@ import requests -API_URL = os.getenv("BRAINDB_TEST_URL", "http://localhost:8000") +# The isolated test stack (docker-compose.test.yml). Override only if you +# really mean to point the suite somewhere else. +API_URL = os.getenv("BRAINDB_TEST_URL", "http://localhost:8002") + +# A handful of tests drive internal services (e.g. graph_expand) over a +# direct DB connection. Default it to the test stack's Postgres, published +# on the host at 5436. An explicit DATABASE_URL in the environment wins. +os.environ.setdefault( + "DATABASE_URL", "postgresql://braindb:braindb@localhost:5436/braindb_test" +) def _wait_for_health(url: str, timeout: int = 30) -> bool: @@ -43,65 +56,19 @@ def _wait_for_health(url: str, timeout: int = 30) -> bool: @pytest.fixture(scope="session", autouse=True) def _require_live_api() -> None: - """Fail fast and loud if the stack isn't up — tests have nothing to run against.""" - if not _wait_for_health(API_URL): - pytest.fail( - f"BrainDB API not healthy at {API_URL}. " - "Run `docker compose up -d` from the repo root first." - ) - + """Fail fast and loud if the test stack isn't up. -@pytest.fixture(scope="session", autouse=True) -def _purge_pytest_artefacts_at_session_end() -> Iterator[None]: - """Session teardown safety net for the per-test `created_entities` - fixture: any test that errors before registering its IDs (or that - bypasses the factories entirely) still leaks `_pytest_` rows - into the live DB. After all tests finish, sweep those out. - - Pattern uniqueness: `_pytest_<8-hex>` is generated only by the - `test_tag` fixture above and never by production code — so a - `content LIKE '_pytest_%'` filter on keyword entities is provably - scoped to test artefacts. - - Order matters: delete tagged entities (facts/thoughts/...) FIRST so - their `tagged_with` edges drop via FK cascade, then the keyword - entities themselves. + Deliberately NOT defaulting to the personal stack on :8000 — the suite + must never run against a database holding real data. """ - yield - try: - from braindb.db import get_conn # only imported at teardown - except Exception as exc: # noqa: BLE001 — defensive, never block the session - print(f"\n[conftest] session cleanup skipped (db import failed): {exc}") - return - try: - with get_conn() as conn: - with conn.cursor() as cur: - cur.execute( - """ - DELETE FROM entities WHERE id IN ( - SELECT r.from_entity_id FROM relations r - JOIN entities kw ON kw.id = r.to_entity_id - WHERE r.relation_type = 'tagged_with' - AND kw.entity_type = 'keyword' - AND kw.content LIKE E'\\_pytest\\_%' ESCAPE '\\' - ) - """ - ) - tagged_deleted = cur.rowcount - cur.execute( - """ - DELETE FROM entities - WHERE entity_type = 'keyword' - AND content LIKE E'\\_pytest\\_%' ESCAPE '\\' - """ - ) - kw_deleted = cur.rowcount - print( - f"\n[conftest] session cleanup: removed {tagged_deleted} " - f"tagged entities + {kw_deleted} _pytest_* keywords" + if not _wait_for_health(API_URL): + pytest.fail( + f"BrainDB test API not healthy at {API_URL}. Start the isolated " + "test stack first:\n" + " docker compose -f docker-compose.test.yml up -d\n" + "and tear it down (wiping all test data) with:\n" + " docker compose -f docker-compose.test.yml down -v" ) - except Exception as exc: # noqa: BLE001 — never break the session on cleanup - print(f"\n[conftest] session cleanup error (ignored): {exc}") @pytest.fixture diff --git a/tests/test_graph_scoring_uses_importance.py b/tests/test_graph_scoring_uses_importance.py index 42ab602..3314731 100644 --- a/tests/test_graph_scoring_uses_importance.py +++ b/tests/test_graph_scoring_uses_importance.py @@ -11,16 +11,38 @@ """ import uuid +import pytest import requests from braindb.db import get_conn from braindb.services.graph import graph_expand +def _direct_db_reachable() -> bool: + try: + with get_conn(): + return True + except Exception: + return False + + +# This test drives graph_expand() over a direct DB connection (by design — +# it locks the SQL CTE, not the HTTP layer). Inside docker, DATABASE_URL +# points at a docker-network hostname that doesn't resolve from the host. +# Skip unless DATABASE_URL is host-reachable, e.g. for the internal DB: +# DATABASE_URL=postgresql://braindb:braindb@localhost:5435/braindb pytest ... +pytestmark = pytest.mark.skipif( + not _direct_db_reachable(), + reason="DATABASE_URL not reachable from the pytest host; set it to a " + "host-reachable address to run direct-DB graph tests", +) + + def test_importance_score_moves_per_hop_relevance(api, make_fact): """Two relations from the same seed, identical relation_type and relevance_score, ONLY differing in importance_score. The hop's accumulated_relevance from graph_expand must reflect the difference.""" - tag = uuid.uuid4().hex + # _pytest_ prefix so the session sweep can reclaim the keyword entity + tag = f"_pytest_{uuid.uuid4().hex[:8]}" seed = make_fact(f"Seed for {tag}", keywords=[tag]) hi_target = make_fact("Generic hi target.") lo_target = make_fact("Generic lo target.") diff --git a/tests/test_graph_walks_both_directions.py b/tests/test_graph_walks_both_directions.py index db0e449..eef7ddf 100644 --- a/tests/test_graph_walks_both_directions.py +++ b/tests/test_graph_walks_both_directions.py @@ -14,13 +14,35 @@ """ import uuid +import pytest import requests from braindb.db import get_conn from braindb.services.graph import graph_expand +def _direct_db_reachable() -> bool: + try: + with get_conn(): + return True + except Exception: + return False + + +# This test drives graph_expand() over a direct DB connection (by design — +# it locks the SQL CTE, not the HTTP layer). Inside docker, DATABASE_URL +# points at a docker-network hostname that doesn't resolve from the host. +# Skip unless DATABASE_URL is host-reachable, e.g. for the internal DB: +# DATABASE_URL=postgresql://braindb:braindb@localhost:5435/braindb pytest ... +pytestmark = pytest.mark.skipif( + not _direct_db_reachable(), + reason="DATABASE_URL not reachable from the pytest host; set it to a " + "host-reachable address to run direct-DB graph tests", +) + + def test_graph_expand_walks_unidirectional_edge_backwards(api, make_fact): - tag = uuid.uuid4().hex + # _pytest_ prefix so the session sweep can reclaim the keyword entity + tag = f"_pytest_{uuid.uuid4().hex[:8]}" # A is the "from" side. B is the "to" side and is the SEED. # Edge: A -> B with is_bidirectional=false (default). diff --git a/tests/test_ingest.py b/tests/test_ingest.py index 45b8261..61c5de7 100644 --- a/tests/test_ingest.py +++ b/tests/test_ingest.py @@ -17,8 +17,11 @@ import requests +# The test stack (docker-compose.test.yml) mounts ./data_test as /app/data, +# so test files live in data_test/ — the personal stack's watcher (polling +# ./data/sources/) never sees them. REPO_ROOT = Path(__file__).resolve().parents[1] -TEST_FILE = REPO_ROOT / "data" / "sources" / "pytest_ingest_sample.md" +TEST_FILE = REPO_ROOT / "data_test" / "sources" / "pytest_ingest_sample.md" TEST_FILE_RELATIVE = "data/sources/pytest_ingest_sample.md" @@ -30,9 +33,9 @@ def _write_sample(text: str) -> None: def _cleanup_file() -> None: for p in [ TEST_FILE, - REPO_ROOT / "data" / "sources" / "ingested" / "pytest_ingest_sample.md", - REPO_ROOT / "data" / "sources" / "failed" / "pytest_ingest_sample.md", - REPO_ROOT / "data" / "sources" / "failed" / "pytest_ingest_sample.md.error.txt", + REPO_ROOT / "data_test" / "sources" / "ingested" / "pytest_ingest_sample.md", + REPO_ROOT / "data_test" / "sources" / "failed" / "pytest_ingest_sample.md", + REPO_ROOT / "data_test" / "sources" / "failed" / "pytest_ingest_sample.md.error.txt", ]: try: if p.exists(): From f745d7f628d7945796b53089784e9ef4a1d74067 Mon Sep 17 00:00:00 2001 From: dimknaf <136385722+dimknaf@users.noreply.github.com> Date: Fri, 12 Jun 2026 01:21:07 +0100 Subject: [PATCH 3/5] feat(frontend): serve the UI from docker compose (port 8642) + logo in header - New always-on `frontend` service: nginx:alpine over the static frontend/ directory, published loopback-only at 127.0.0.1:8642 (FRONTEND_PORT knob). No build step, no Dockerfile; `docker compose up -d` now includes the browser UI with zero manual steps. Loopback is deliberate: the UI calls the API at http://localhost:8000 from the browser, so wider publishing would not make it usable remotely anyway. - Logo in the frontend header (replaces the text brand) + SVG favicon. - frontend/README.md updated: compose is the default route, any static server still works. - Relocated the wiki_scheduler comment block that had ended up stranded above braindb_db. --- .env.example | 4 ++++ docker-compose.yml | 23 +++++++++++++++++++---- frontend/README.md | 12 +++++------- frontend/braindb-logo.svg | 1 + frontend/braindb-mark.svg | 1 + frontend/index.html | 3 ++- frontend/style.css | 1 + 7 files changed, 33 insertions(+), 12 deletions(-) create mode 100644 frontend/braindb-logo.svg create mode 100644 frontend/braindb-mark.svg diff --git a/.env.example b/.env.example index bf49f52..84e8c4f 100644 --- a/.env.example +++ b/.env.example @@ -33,6 +33,10 @@ COMPOSE_PROFILES=internal-db # API port (default 8000) API_PORT=8000 +# Frontend (read-only browser UI) host port — served automatically by +# docker compose at http://localhost:8642. Change if 8642 is taken. +FRONTEND_PORT=8642 + # HuggingFace token — for faster model downloads (optional but recommended) # Get yours at https://huggingface.co/settings/tokens HF_TOKEN= diff --git a/docker-compose.yml b/docker-compose.yml index 5390213..4e62aab 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -65,10 +65,6 @@ services: - .:/app command: python -m braindb.ingest_watcher - # Always-on wiki maintenance sidecar — same posture as `watcher`. It loops - # cron -> maintain -> write so wikis self-organise from entities with zero - # manual steps. To run without it (e.g. cost control), start the stack - # excluding this service or scale it to 0 — exactly as you would the watcher. # Bundled Postgres (pgvector) — exists ONLY when the internal-db profile is # active, i.e. when .env contains COMPOSE_PROFILES=internal-db. Without that # line this service is invisible to every compose command and external-DB @@ -100,6 +96,10 @@ services: networks: - local-network + # Always-on wiki maintenance sidecar — same posture as `watcher`. It loops + # cron -> maintain -> write so wikis self-organise from entities with zero + # manual steps. To run without it (e.g. cost control), start the stack + # excluding this service or scale it to 0 — exactly as you would the watcher. wiki_scheduler: build: . container_name: braindb_wiki_scheduler @@ -116,6 +116,21 @@ services: - .:/app command: python -m braindb.wiki_scheduler + # Read-only browser UI (frontend/ — plain static files, no build step). + # Loopback-only on purpose: the UI's API calls go to http://localhost:8000 + # from the BROWSER, so publishing this port beyond loopback wouldn't make + # the app usable remotely anyway. Change FRONTEND_PORT in .env if taken. + frontend: + image: nginx:alpine + container_name: braindb_frontend + restart: unless-stopped + ports: + - "127.0.0.1:${FRONTEND_PORT:-8642}:80" + volumes: + - ./frontend:/usr/share/nginx/html:ro + networks: + - local-network + volumes: # Data volume for the bundled internal DB (internal-db profile only). braindb_pgdata: diff --git a/frontend/README.md b/frontend/README.md index 8ef436a..799efc4 100644 --- a/frontend/README.md +++ b/frontend/README.md @@ -4,17 +4,15 @@ A thin, read-mostly browser UI for BrainDB. Vanilla HTML / CSS / JS — no build ## Run -The BrainDB backend must be running first (default: `http://localhost:8000`). From the repo root: +Served by the stack itself — `docker compose up -d` includes an nginx container for this directory. Open (change the port with `FRONTEND_PORT` in `.env`). The backend must be running too (default: `http://localhost:8000`). + +No container? Any static file server works. From the repo root: ```bash -python -m http.server 8090 -d frontend +python -m http.server 8642 -d frontend ``` -Then open . - -That's it. The frontend is a static page that talks to the API directly via `fetch`. CORS is already open on the backend. - -> If `8090` is in use on your machine, pick any free port: `python -m http.server -d frontend`. (Avoid `8080` on Windows with Docker Desktop installed — it's commonly held by the WSL backend.) +That's it. The frontend is a static page that talks to the API directly via `fetch`. CORS is already open on the backend. (If you pick your own port, avoid `8080` on Windows with Docker Desktop installed — it's commonly held by the WSL backend.) ## Pointing at a non-local backend diff --git a/frontend/braindb-logo.svg b/frontend/braindb-logo.svg new file mode 100644 index 0000000..62d3517 --- /dev/null +++ b/frontend/braindb-logo.svg @@ -0,0 +1 @@ +BrainDB diff --git a/frontend/braindb-mark.svg b/frontend/braindb-mark.svg new file mode 100644 index 0000000..c31fcdc --- /dev/null +++ b/frontend/braindb-mark.svg @@ -0,0 +1 @@ +BrainDB diff --git a/frontend/index.html b/frontend/index.html index 722e2ef..e764693 100644 --- a/frontend/index.html +++ b/frontend/index.html @@ -4,13 +4,14 @@ BrainDB +
-
BrainDB
+
BrainDB