v0.5.0 — BEAM benchmark + wikis on by default - #11
Merged
Conversation
…lation) Foundation only -- no harness code yet, no benchmark runs. Lays the scaffolding for v0.5.0's BEAM target so the next commit can implement the adapter / runner / judge / score with the safety + trust posture locked in. What ships: - benchmarks/beam/upstream/ as git submodule pinned to mohammadtavakoli78/BEAM@3e12035 (ICLR 2026, arXiv:2510.27246). Our harness will call their eval module unmodified and import the judge prompt (src/prompts.py::unified_llm_judge_base_prompt) at runtime -- zero prompt content in our codebase. - docker-compose.bench.yml: standalone bench stack with strict isolation. Separate Postgres container on host port 5433, separate database (braindb_bench), separate volume, separate BrainDB API on :8001, separate watcher data dir (./data_bench/sources/). The personal braindb stack is never touched. - .env.bench.example + benchmarks/beam/README.md document the bench config knobs, trust model, isolation guarantees, and Qwen-as-judge caveat (Qwen not directly comparable to mem0's GPT-4o or Hindsight's Gemini-judged BEAM scores; cross-judge calibration planned). - Granular .gitignore: personal A/B content stays local-only, benchmarks/beam/ is tracked, generated artefacts ignored.
The personal Postgres in this environment (postgres_container, timescale/timescaledb:latest-pg16) already binds host port 5433. Bench Postgres claimed the same port and would have refused to start. Move bench Postgres to 5434. Verified port 5434 (and 8001 for the bench API) free with netstat before commit. README updated to match. No other isolation properties change.
…mup) Adapter renders BEAM rows to markdown for the existing watcher pipeline. Runner orchestrates per-conversation reset -> ingest -> warmup -> answer -> record, with a hard-coded DATABASE_URL safety assertion (must contain literal "braindb_bench" or refuses to proceed). Judge imports upstream BEAM's unified_llm_judge_base_prompt VERBATIM from the git submodule (zero prompt content in our codebase), calls Qwen via OpenAI-compat, replicates upstream's per-nugget 0/0.5/1.0 scoring. Score reporter aggregates + writes results.md with the Qwen-judge caveat. First smoke run found + fixed two harness bugs (both folded into this commit): - Watcher is non-recursive AND ingest endpoint hardcodes the file path prefix "data/sources/ingested/..." (braindb/ingest_watcher.py:278). Writing adapter output to a beam/ subdir fails ingest with a 404. Adapter now writes top-level data_bench/sources/; bench-conv filenames (e.g. beam_100k_conv_001.md) are unique enough at top level. - Warmup with settle_seconds=30 was too tight for slow chunk processing: declared "settled" between datasource creation and first extracted fact (a 3-5 min gap on big documents). Bumped default to 180s. First smoke also measured BrainDB extraction throughput on workstation Qwen at ~4.5 min/chunk (1M conversation, chunk 1: 5 facts + ~20 relations + per-edge LLM scoring). Implies BEAM-1M ~77h/conv (infeasible in the 2-3 day budget) and BEAM-100K ~9h/conv (at the edge). 100K --limit 1 smoke running now to confirm one full end-to-end pipeline. Status: harness is complete and verified to ingest end-to-end. The extraction throughput is a separate v0.4.0-cost / v0.5.0-evaluation trade-off to surface and decide on next session (e.g. ablate per-edge scoring at bench time, or pivot to LoCoMo with smaller conversations).
Recovery from the 10h timeout that lost the first BEAM-100K run. Two bench-side
problems, both purely on bench code (no BrainDB changes):
1. Warmup blocked on `pending_wiki == 0`. The wiki maintainer queues
candidates faster than the writer drains them on big documents (grew
from 24 pending at t=3min to 1701 at t=10h on the lost run). The queue
never converged. Phase C never ran.
2. Per-conversation isolation via TRUNCATE meant each conversation's
memory state was destroyed when moving to the next. No way to inspect
what happened in conv_007 after the fact.
Fix:
- warmup.py: wait_for_warmup gains block_on_wiki_queue (default False) and
database_url (default falls back to BENCH_DATABASE_URL). The new default
drops `pending == 0` from the `clear` condition. Wikis continue async in
the background while Phase C answers questions; recall queries on
existing entities don't need an empty wiki queue. CLI gains
--wait-for-wikis as the opt-in for the strict variant.
- bench.py: replaces reset_bench_db() with three helpers:
create_conv_db(conv_id) -> CREATE DATABASE braindb_conv_NNN
restart_api_with_db(db_url) -> docker compose up -d --force-recreate api_bench
with BENCH_DATABASE_URL overridden
wait_for_api_healthy() -> poll /health until ready
Each conversation now gets its own Postgres database that persists for
inspection after the run. The api_bench container restarts with a new
DATABASE_URL between conversations (~30-60s embedding reload per restart
× N convs is negligible vs ingest wall-clock).
- config.py: safety sentinel becomes a tuple ("braindb_bench", "braindb_conv_").
assert_bench_database_url accepts either pattern; rejects bare "braindb"
(personal) as before. Adds BENCH_ADMIN_DATABASE_URL for connecting to
Postgres's maintenance DB (CREATE / DROP).
- docker-compose.bench.yml:
* api_bench's DATABASE_URL becomes ${BENCH_DATABASE_URL:-...default}
so the runner can override per conversation
* adds new bench_runner service (profile: "runner") that has the docker
CLI installed + docker socket mounted, so it can recreate api_bench
between conversations. Profile means it doesn't auto-start with
`docker compose up`; explicit:
docker compose -f docker-compose.bench.yml run --rm bench_runner \
python -m benchmarks.beam.bench --split 100K --limit 1
- benchmarks/beam/runner.Dockerfile (new): small image for the bench_runner
service. python:3.12-slim + docker.io + datasets + huggingface_hub +
psycopg2-binary + requests. Repo code itself comes via bind mount.
BrainDB code (braindb/) is completely untouched. Central review will still
fail on large documents (it puts all facts in a single agent query, which
exceeds the 10K char limit). That's a BrainDB design choice, accepted as-is
for this session. Watcher logs a warning, extraction succeeds, facts land,
the bench proceeds. The BrainDB central-review redesign is a separate
focused session — to be validated by re-running the bench and measuring
the delta vs this number.
…stall Two follow-up refinements on top of 9e06aa5, surfaced when actually running the bench on the workstation: 1. restart_api_with_db now uses the Docker Python SDK instead of subprocess(docker compose up -d --force-recreate). The subprocess path broke because `docker compose -f docker-compose.bench.yml` resolves the relative bind-mount paths (`.:/app`, `./data_bench:/app/data`) against the bench_runner container's filesystem, not the host's, leaving the recreated api_bench with empty mounts and alembic failing on missing script_location. The SDK path inspects the existing container's already- host-resolved HostConfig.Binds and preserves them on recreate. 2. runner.Dockerfile: the apt `docker.io` package on python:3.12-slim only ships `docker-init`, not the `docker` CLI itself (FileNotFoundError on `docker` invocation). Switched to downloading the official static CLI binary from download.docker.com plus the compose plugin from the docker/compose GitHub release. Also adds the `docker` Python SDK to the pip install list (used by the new restart_api_with_db). BrainDB code (braindb/) remains completely untouched. Pure bench-harness follow-up.
…point cap Five surgical changes that together eliminate central_review's HTTP 422 on large documents and roughly halve the per-conversation wall-clock. All in one revertible commit; central_review function body kept dormant for easy restore. 1. routers/agent.py — AgentQueryRequest.query.max_length 10000 -> 40000. Endpoint-scoped (HTTP /api/v1/agent/query only); internal callers (run_typed from wiki maintainer/writer/subagent) are unaffected. Qwen 27B runs at max_model_len=40960; 40K input chars ~= 10K tokens leaves ~70% of the window free for system prompt + tool defs + ~35-turn tool-call iteration + final output. No schemas, migrations, or API contract change. 2. ingest_watcher.py — CHUNK_WORDS 600 -> 1200. Halves chunk count on big documents (130 chunks -> 65 chunks on a BEAM-100K conversation), so halves total /agent/query round-trips. Per-chunk prompt grows from ~4 KB to ~7-8 KB, still 5x under the new 40K cap. CHUNK_OVERLAP stays at 75. 3. ingest_watcher.py — extract_facts_from_chunk prompt gains step (c): "AFTER all facts in this chunk are saved, look back at the facts you just created and add cross-fact relations WITHIN this chunk where genuinely meaningful." Relation types restricted to supports / contradicts / elaborates / similar_to / is_example_of. Explicit "do NOT create relations to facts from OTHER chunks" because the agent only sees this chunk's facts. Each per-chunk agent now does the same kinds of work central_review used to do, scoped to its chunk. 4. ingest_watcher.py — central_review call removed from enrich_datasource. The function body is intentionally KEPT in place as dormant code with a pointer comment explaining how to re-enable. Restoring the old behaviour = uncommenting one line. No cross-chunk semantic relations and no whole- document holistic thought get created — both already failed today on big documents (HTTP 422), so this is not a regression for big-doc behaviour; small-doc behaviour gains chunk-scoped relations instead of doc-scoped ones, with the same kinds of writes (same Pydantic tool schemas, same entity types, same downstream consumers). 5. ingest_watcher.py — max_turns 40 -> 35 on the per-chunk call. Math: ~10 facts/chunk * 2 calls (save_fact + derived_from) + ~0-5 cross-fact relations + 1 final_answer = ~21-26 turns, with ~9 turns of margin. config.py and .env / .env.bench audited: nothing of ours is env-driven. tests/test_split_chunks.py references CHUNK_WORDS by symbol (not literal 600), so the bump is test-clean. pytest tests/test_split_chunks.py: 10/10 green. pytest tests/test_ingest.py + test_ingest_watcher_no_dictation.py: 6/6 green. No test references central_review. Rollback: `git revert <this-sha>` puts everything back atomically. Worst-case edge tweaks (drop max_length to 30K, drop CHUNK_WORDS to 1000, bump max_turns to 45) are each single-number edits if the smoke run surfaces an issue.
restart_api_with_db() was using docker SDK containers.run(network=name) to attach the recreated api_bench to bench-network, but run()'s network kwarg does not accept aliases — so the recreated container lost the compose-set service-name alias `api_bench`. The bench_runner uses `BENCH_API_BASE: http://api_bench:8001` for DNS, and the alias loss made the first conversation fail with "Failed to resolve 'api_bench'" inside the 180s healthcheck loop. Fix: switch to containers.create() + networks.get(name).connect(container, aliases=...) + container.start(). Capture aliases from each original network and always include "api_bench" as a fallback (docker inspect sometimes returns null aliases even when DNS works, so always-add is safer). Also disconnect from the default bridge create() picks up by default. Caught by the monitor on the smoke run for the BrainDB big-chunks fix. No BrainDB code touched.
Caught by the v3 smoke run on the new big-chunks code path: warmup
converged after only chunk 1 of 57 had completed, because:
- The new per-chunk extract_facts_from_chunk prompt asks the agent to
(a) save_fact + create_relation(derived_from) for each fact, (b) then
create_relation between facts in the same chunk, and (c) the agent
routinely calls delegate_to_subagent + recall_memory before final_answer.
- Phase (a) bursts into the entities table for the first ~2 minutes of
each chunk. Phases (b) and (c) write to the relations table (or do
pure reasoning) for the next 2-4 minutes — the entities table is
quiet during this whole tail.
- The old _seconds_since_last_entity probe interpreted that quiet
entities-table tail as "extraction is done", so warmup converged at
t=281s while chunk 1's agent was still working and chunks 2..N
had not yet started.
Fix:
- Switch warmup to _seconds_since_last_activity, which is the seconds
since the most recent INSERT in EITHER entities OR relations
(GREATEST in Postgres ignores NULLs, so empty relations falls
through to entities cleanly).
- Bump settle_seconds default from 180s -> 300s. The new probe is
refreshed by every create_relation call, so 5 minutes of no DB
writes truly means the agent loop has finished, not just paused in
its relation-creation tail.
- Update the TimeoutError message + CLI help to reflect the new probe.
No BrainDB code touched. Bench-only fix on feat/benchmark-big-chunks.
The previous warmup.py fix bumped its OWN default 180 -> 300, but bench.py had its own warmup_settle_seconds parameter default at 180, which is the value actually passed to wait_for_warmup() — so the warmup.py default never took effect. Caught on the v4 smoke run: warmup converged at activity_age=183s while chunks 4..57 were still pending; Phase C kicked off against a 37-fact DB (~1/4 of the document). Now both layers default to 300s and the CLI flag matches. No BrainDB code touched. Two-line bench-only fix on feat/benchmark-big-chunks.
Six surgical edits across three files to make the full BEAM run robust enough
that user can leave it for 5 days without supervision.
braindb/ingest_watcher.py:
- AGENT_TIMEOUT 1200s -> 1800s (30 min single-attempt cap; chunks average
~6 min, the cross-fact-relation tail can stretch past 10 min on dense
chunks).
- max_turns 35 -> 40 on the per-chunk call (was deliberately lowered in
c185ee0 thinking 35 was enough; v4 smoke caught 2/57 chunks hitting
"Max turns (3) exceeded" — the agent's Layer-4 retry ran out after
primary budget exhausted at 35).
- Per-chunk retry loop: up to 3 attempts with 5s/10s backoff. Retries
ONLY when call_agent returns None (HTTP error) OR the parse returns
zero fact UUIDs (agent never reached final_answer). If the first
attempt's parse DID return UUIDs, do not retry — those facts are
already committed via tool calls and a retry would dupe them. Catches
the transient-failure class without the duplication risk.
benchmarks/beam/bench.py:
- answer_one_question: 3-attempt retry around the POST with 5s/10s
backoff. Retries on ConnectionError / Timeout / HTTP 5xx. Does NOT
retry on HTTP 4xx (request-shape errors will repeat). After three
failures the last exception bubbles to the caller which records the
question as `[ERROR: ...]` and the run continues to the next question.
- wait_for_api_healthy default timeout 180s -> 300s (api restart needs
alembic + uvicorn + embedding model load; 5 min has comfortable slack
over the observed ~30-60s).
- run_one_conversation warmup_settle_seconds default 300 -> 600 to match
the new warmup.py default (otherwise this layer overrides).
benchmarks/beam/warmup.py:
- settle_seconds default 300 -> 600. With the new per-chunk
cross-fact-relation prompt, in-chunk quiet stretches can reach 3 min
(subagent / recall_memory work). 10 min gives generous slack while
still catching genuinely stuck convergence (the warmup_timeout=1800s
cap fires if extraction itself stalls).
Pytest tests/test_split_chunks.py + tests/test_ingest.py +
tests/test_ingest_watcher_no_dictation.py: 16/16 green.
This is the last set of changes before the full --split 100K run. After
this commit, do not touch the run autonomously.
…tion Caught on the morning run: every conversation timed out at 1800s warmup because 100K-conv extraction takes ~5.7h (57 chunks * ~6 min each at CHUNK_WORDS=1200) — there is no way for warmup to converge within 30 minutes when extraction is actively progressing. When the bench gave up, api_bench was recreated for the next conv mid-extraction, polluting both the old and new conv databases with cross-talk facts. Fix: bump warmup_timeout default from 1800 -> 43200 (12h). This is 2x the projected 100K extraction time and gives generous slack for retry overhead and edge cases. settle_seconds=600s (set in the prior commit) still catches genuine stalls inside the 12h window, so the full timeout only burns when something is truly stuck. Both the warmup.py default + CLI default and the bench.py run_one_conversation default + CLI default are updated together to keep them in lockstep (otherwise the bench layer overrides). For future 500K / 1M splits this knob will need to scale up further (500K convs are ~30h, 1M convs are ~60h). 100K is the first run target; later splits will bump the CLI flag explicitly. No BrainDB code touched. Two-number bench-only fix.
…density Observed during the in-flight 100K run: agent consistently hits the ~5-10 facts/chunk band the prompt suggests, treating it as both target and implicit cap. For 1200-word (~4-5 page) chunks this looks light relative to realistic conversation density. One-line phrase swap inside extract_facts_from_chunk's prompt template. Replaces: "typically 5-10 facts per chunk" with: "typically from 1-2 to 30 depending on density and length of the chunk" Everything else in the prompt is preserved: same "concrete, standalone FACTS — specific claims, numbers, events, named decisions" framing, same "Ignore filler, opinion, and generic statements", same "quality over quantity" anchor, same a/b/c steps, same Do-NOT rules, same final_answer format. No bump to max_turns (currently 40). The agent will likely shift distribution upward but rarely hit 30; if "Max turns exceeded" failures spike we can bump max_turns in a follow-up. Tests: pytest tests/test_split_chunks.py + test_ingest.py + test_ingest_watcher_no_dictation.py = 16/16 green. The no-dictation test specifically checks the prompt doesn't pre-dictate certainty / importance / relevance_score values; this change doesn't touch those. Easy revert: git revert this SHA, push, workstation pulls, restart.
Symptoms: each Phase C question takes 30-60+ min (42 get_entity + 4 recursive delegate_to_subagent calls per question). One subagent crashed with ContextWindowExceededError after 891s. Root cause: facts had notes "Extracted from <doc> chunk X/Y" but get_entity only takes byte offsets. Agent guessed chunk -> offset (36/57 x 526K = 333K) and paged randomly. Fix: - split_chunks_with_offsets returns (text, byte_start, byte_end) per chunk; original whitespace preserved. split_chunks is now a 1-line back-compat wrapper - single source of truth, all 10 existing tests pass unchanged. - extract_facts_from_chunk threads byte_start/byte_end into the prompt so save_fact records notes="Extracted from <doc> chunk X/Y (bytes A-B)". - enrich_datasource uses the offset-aware chunker. - system_prompt.md + skills/braindb/SKILL.md each get ONE descriptive sentence explaining the new notes format. No behavioural tightening. 6 new unit tests for split_chunks_with_offsets cover: offsets recover first/last word via text[byte_start:byte_end], byte_end <= len(text), monotonic byte_starts, back-compat with split_chunks. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds optional --wiki-wait-seconds CLI flag. When > 0, the bench sleeps between [B done] and [C], printing [Bw] / [Bw done] markers. During the sleep the wiki writer + maintainer have vLLM essentially to themselves (no extraction, no question agent competing), so wikis develop more before recall queries hit during Phase C. Recommended value for BEAM 100K: 28800s (8h) — wiki writer at full vLLM speed drains ~120 writes/hour, so 8h captures most of the typical 1600-job queue. Longer waits hit diminishing returns because the maintainer keeps queuing new triage jobs at roughly the rate the writer drains them. Default 0 (no wait) — opt-in. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two coordinated default flips so a fresh clone runs end-to-end with just
a hosted-provider API key, no workstation tunnel needed.
1) LLM profile defaults flipped from workstation-Qwen to deepinfra
in three places that together define the bench's runtime LLM:
- docker-compose.bench.yml LLM_PROFILE_BENCH default
- docker-compose.bench.yml QWEN_BASE_URL/QWEN_MODEL pass-through (empty
default; Python config supplies the real default)
- benchmarks/beam/config.py judge defaults (QWEN_BASE_URL/MODEL/API_KEY)
QWEN_* env-var names kept for backwards compat with existing .env.bench
files; the values can point at any OpenAI-compatible endpoint. Self-hosted
Qwen runs supported via the documented overrides in .env.bench.example.
2) WIKI_ENABLED defaults flipped from false to true in both
braindb/wiki_scheduler.py and docker-compose.yml. The wiki pipeline
(maintainer + writer) is part of what makes BrainDB a knowledge layer
vs. a fact store, so a fresh stack starts it automatically. Idle ticks
are pure SQL (no LLM cost). Set WIKI_ENABLED=false to opt out.
CHANGELOG and .env.example updated to surface both changes.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The README still described the harness as "Step 0 scaffolding" and hard-coded Qwen as the judge. Both are stale after the v0.5.0 work: the harness is shipped (adapter / runner / judge / score / warmup all present), and the judge defaults to deepinfra with self-hosted Qwen as the documented override. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
v0.5.0 ships a public benchmark harness against BEAM (ICLR 2026, the toughest recent long-term memory benchmark), tightens BrainDB's ingestion pipeline for long documents, and flips two repo defaults so a fresh clone runs end-to-end with a hosted LLM provider.
Branch:
feat/beam-benchmark→main. 16 commits, 25 files changed (+2,434 / −37).Scope of changes by area
benchmarks/beam/(new directory)A self-contained BEAM runner: dataset adapter, per-conversation runner, warmup convergence barrier, judge, scoring, plus an isolated bench stack (
docker-compose.bench.yml). The bench has its own Docker project namespace, its own Postgres on host port 5434, its own API on port 8001, its own data directory, and a hard-coded safety sentinel that refuses any destructive op unless the activeDATABASE_URLliterally containsbraindb_benchorbraindb_conv_. The upstream BEAM eval is pinned as a git submodule and its judge prompt is imported at runtime — never paraphrased or copied into our code.A separate orchestrator container (
bench_runner, profile-gated so it doesn't auto-start) drives the loop. It uses the docker-py SDK + a mounted Docker socket to recreate the bench API container with a freshDATABASE_URLper conversation, preserving the network alias across recreates so DNS keeps resolving. The warmup barrier tracks both new entities and new relations independently and is decoupled from the wiki queue — bench timing is bounded by extraction, not by how fast summaries get written.braindb/ingest_watcher.py(extraction pipeline changes)CHUNK_WORDS600 → 1200 (halves chunk count on long sources)notesfield now includes a byte range into the source datasource ((bytes 245760-252960)) so the agent can read the exact slice viaget_entity(id, offset, limit)instead of guessing chunk numberscentral_reviewextraction pass is no longer called (the call site is removed; function body kept in-place for trivial revert)braindb/routers/agent.py—/agent/queryinput cap raised from 10,000 → 40,000 characters. Endpoint-scoped only; internal callers untouched. Lets per-chunk extraction prompts accommodate 1200-word chunks with cross-fact-relation instructions.braindb/wiki_scheduler.pyanddocker-compose.yml—WIKI_ENABLEDdefault flipped fromfalsetotrue. The wiki maintainer + writer are part of what makes BrainDB a knowledge layer rather than a fact store; idle ticks are pure SQL (zero LLM cost). SetWIKI_ENABLED=falsein.envto opt out.benchmarks/beam/config.pyanddocker-compose.bench.yml— benchLLM_PROFILE_BENCHand judge endpoint defaults flipped fromvllm_workstation_qwentodeepinfra(google/gemma-4-31B-it). Self-hosted Qwen via vLLM remains a first-class supported path; see.env.bench.examplefor the override block.Tests —
tests/test_split_chunks.pygets +95 lines covering the new offset-aware splitter.Docs —
CHANGELOG.md,.env.example,.env.bench.example,benchmarks/beam/README.md,skills/braindb/SKILL.md, and the agent system prompt all updated to reflect the new defaults and capabilities.Test plan
docker compose up -dagainst a fresh checkout;/healthreturns 200docker compose -f docker-compose.bench.yml up -d;api_benchhealthy on port 8001answers.json,verdicts.json,category_scores.jsonpytest tests/passes against the personal stackA full BEAM 100K run on this release is being executed independently; the score numbers and a cross-judge calibration sample will land in a follow-up PR.