v0.5.0 — BEAM benchmark + wikis on by default

.env.bench.example

@@ -0,0 +1,53 @@
+# Copy this to .env.bench and fill in the values you need.
+#
+# Bench-only environment. Used by docker-compose.bench.yml + the bench
+# runner under benchmarks/. The personal stack reads .env (separate file)
+# and is unaffected by anything here.
+#
+# DO NOT commit .env.bench — it's gitignored. Only this template is tracked.
+
+# --- LLM provider for the bench BrainDB API ---
+# Public default: deepinfra (hosted, cheap, validated). For self-hosted Qwen
+# runs, comment out the first line and uncomment the workstation line below.
+LLM_PROFILE_BENCH=deepinfra
+# LLM_PROFILE_BENCH=vllm_workstation_qwen
+
+# Optional model override (e.g. to try a smaller Qwen variant for bench).
+AGENT_MODEL=
+
+# --- Provider keys (only fill in what your LLM_PROFILE_BENCH needs) ---
+NVIDIA_NIM_API_KEY=
+DEEPINFRA_API_KEY=
+
+# Self-hosted vLLM: leave empty unless your vLLM was started with --api-key.
+# The resolver supplies the literal "EMPTY" placeholder if blank.
+VLLM_API_KEY=
+
+# HuggingFace token — for the bench dataset cache + the BEAM dataset download
+HF_TOKEN=
+
+# --- Bench-specific tuning knobs (defaults match docker-compose.bench.yml) ---
+# How fresh new entities must be before the wiki maintainer considers them
+# eligible (prod default: 30 min). Bench: 1 min so warmup stays short.
+WIKI_FRESHNESS_MINUTES_BENCH=1
+
+# Wiki scheduler tick (prod default: 60s). Bench: 5s so per-conversation
+# warmup is dominated by LLM time, not scheduler wait.
+WIKI_INTERVAL_BENCH=5
+
+# Ingest watcher poll interval (prod default: 7s). Bench: 3s for snappy smoke.
+INGEST_POLL_INTERVAL_BENCH=3
+
+# Agent verbose logging (every tool call printed). Bench default: true.
+AGENT_VERBOSE_BENCH=true
+
+# --- Judge LLM (scores answers against the BEAM rubric) ---
+# Independent of LLM_PROFILE_BENCH. The judge speaks OpenAI-compatible REST,
+# so any provider works. Names are QWEN_* for backwards compatibility; the
+# values can point anywhere (deepinfra, OpenAI, self-hosted vLLM, ...).
+#
+# Defaults (in benchmarks/beam/config.py) target deepinfra.
+# For self-hosted Qwen via vLLM, uncomment and adjust:
+# QWEN_BASE_URL_BENCH=http://host.docker.internal:8010/v1
+# QWEN_MODEL_BENCH=cyankiwi/Qwen3.6-27B-AWQ-INT4
+# QWEN_API_KEY=EMPTY

.env.example

@@ -35,6 +35,14 @@ AGENT_MODEL=
 # (visible via `docker logs braindb_api -f`). Response payload unchanged.
 AGENT_VERBOSE=false
 
+# Wiki pipeline (maintainer + writer) — ON by default. The wiki layer is
+# what turns BrainDB's fact graph into navigable summary pages. Each tick
+# (every WIKI_INTERVAL seconds) the scheduler runs cheap SQL bookkeeping
+# and only fires LLM-heavy /wiki/maintain or /wiki/write when work is
+# pending — idle ticks cost zero. Set to false if you want a pure
+# fact-store stack with no LLM-heavy background work.
+# WIKI_ENABLED=true
+
 # Agent turn budget — how many tool-call turns the general /agent/query
 # is allowed before the SDK forces termination. Default 20. Lowering
 # this below ~15 degrades deep-research models (notably local Qwen via

.gitignore

@@ -39,3 +39,22 @@ data/sources/*
 
 # Wiki review exports — generated, read-only inspection output
 data/wiki_review/
+
+# --- Benchmarks ---
+# Personal A/B bench content (5-Q personal recall + run traces) — local only,
+# tied to the user's own BrainDB content, not for the public repo.
+benchmarks/README.md
+benchmarks/questions.json
+benchmarks/run_agent.sh
+benchmarks/runs/
+
+# Generated artefacts inside any public benchmark subdir (e.g. benchmarks/beam/)
+# — full LLM traces, dataset caches, per-question JSONLs. Only results/*.md
+# gets committed.
+benchmarks/*/runs/
+benchmarks/*/dataset_cache/
+benchmarks/*/answers/
+
+# Bench compose: separate host data dir for the bench watcher, separate env file
+data_bench/
+.env.bench

.gitmodules

@@ -0,0 +1,3 @@
+[submodule "benchmarks/beam/upstream"]
+	path = benchmarks/beam/upstream
+	url = https://github.com/mohammadtavakoli78/BEAM.git

CHANGELOG.md

@@ -5,6 +5,24 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.5.0] — 2026-06-09
+
+### Changed
+
+- **`WIKI_ENABLED` now defaults to `true`** in both `braindb/wiki_scheduler.py`
+  and `docker-compose.yml`. A fresh `docker compose up` starts the wiki
+  maintainer + writer automatically. The pipeline is part of what makes
+  BrainDB a knowledge layer; idle ticks cost zero LLM (cheap SQL only),
+  and the previous opt-in default was a friction point in onboarding.
+  Set `WIKI_ENABLED=false` in `.env` to opt out (e.g. pure fact-store
+  deployments).
+- **Bench repo defaults flipped from workstation Qwen to deepinfra.**
+  `docker-compose.bench.yml`'s `LLM_PROFILE_BENCH` default is now
+  `deepinfra`; the judge endpoint defaults in `benchmarks/beam/config.py`
+  also point at deepinfra. Self-hosted Qwen runs are still supported —
+  set `LLM_PROFILE_BENCH=vllm_workstation_qwen` plus the judge overrides
+  in `.env.bench` (see `.env.bench.example`).
+
 ## [0.4.0] — 2026-06-03
 
 Headline: a focused pass on recall quality and the `view_tree` tool. The

benchmarks/beam/README.md

@@ -0,0 +1,116 @@
+# BrainDB on BEAM (Beyond a Million Tokens)
+
+Public benchmark harness running BrainDB against the **BEAM** memory benchmark
+([arXiv:2510.27246](https://arxiv.org/abs/2510.27246), ICLR 2026, Tavakoli et al.
+of U Alberta + UMass Amherst).
+
+**Status**: harness shipped. Dataset adapter, per-conversation runner,
+warmup barrier, Qwen-as-judge, scoring, and the isolated bench stack
+(`docker-compose.bench.yml`) are all in this directory. See
+`adapter.py`, `bench.py`, `judge.py`, `score.py`, `warmup.py`.
+
+---
+
+## Trust model
+
+The integrity bar is: **the parts that score us are theirs, used unmodified**.
+Nothing in our code paraphrases, copies, or reinterprets the benchmark.
+
+| Component | Where it lives | Why this matters |
+|---|---|---|
+| BEAM dataset | `Mohammadta/BEAM` on HuggingFace; pinned by sha256 in our harness | Same SHA → same dataset; anyone can verify |
+| BEAM eval script | Git submodule `upstream/` pinned to a specific commit SHA of `mohammadtavakoli78/BEAM` | We call `python -m src.evaluation.run_evaluation` unmodified |
+| Judge prompt | `from src.prompts import unified_llm_judge_base_prompt` — loaded at runtime from the submodule | Zero prompt content in our codebase; no risk of paraphrase or typo |
+| Adapter (dataset → BrainDB ingest) | `adapter.py` (ours, ~150 LOC) | Transparent: read the code |
+| Bench runner (per-conversation reset + ingest + warmup + answer) | `bench.py` (ours, ~120 LOC) | Same |
+| Judge runner (calls the configured LLM with the upstream prompt) | `judge.py` (ours, ~80 LOC) | Anyone can re-judge our `answers.json` with any model |
+| Eval wrapper (invokes upstream eval) | `score.py` (ours, ~50 LOC) | Just a thin caller |
+
+**The "no copy" rule** is satisfied at the strongest level: the judge prompt
+text never leaves the upstream submodule. We import it as a Python string
+and pass it to our chosen judge model. Hindsight and mem0 both do the same
+(they wire it into their own judge runners); the upstream eval code itself
+already imports it from `src/prompts.py::unified_llm_judge_base_prompt`.
+
+---
+
+## Isolation from your personal BrainDB
+
+The bench runs in `docker-compose.bench.yml` — a completely separate stack
+from your personal `docker-compose.yml`. Layered isolation:
+
+- Separate Docker project namespace: `name: braindb_bench`
+- **Separate Postgres container** (`braindb_bench_postgres`) on host port 5434
+  (host port 5433 is already used by the personal `postgres_container` in
+  this environment; bench takes 5434 to avoid a collision)
+- Separate Postgres database: `braindb_bench` (never `braindb`)
+- Separate Postgres data volume: `braindb_bench_pgdata`
+- **Separate BrainDB API on port 8001** (personal stays on 8000)
+- **Separate host data directory**: `./data_bench/sources/` — the bench
+  watcher polls this; personal watcher continues polling `./data/sources/`
+  and never sees bench files
+- Hard-coded safety assertion in `bench.py`: the active `DATABASE_URL`
+  MUST contain `braindb_bench` literally, or the runner refuses to start
+- Explicit invocation: bench requires
+  `docker compose -f docker-compose.bench.yml --env-file .env.bench up`;
+  a plain `docker compose up` runs the personal stack as normal
+
+Before the first benchmark run, snapshot your personal Postgres volume as a
+paranoia tarball:
+
+```bash
+docker run --rm -v braindb_pgdata:/data -v "$(pwd)":/backup alpine \
+  tar czf /backup/braindb_personal_backup_$(date +%Y%m%d).tar.gz /data
+```
+
+One-command restore if anything ever goes wrong.
+
+---
+
+## Caveats on the published number
+
+The judge is an OpenAI-compatible LLM endpoint. The repo default is
+**deepinfra** (`google/gemma-4-31B-it`); self-hosted Qwen 3.6-27B via a
+local vLLM endpoint is supported via `.env.bench` overrides
+(see `.env.bench.example`). Published BEAM scores used GPT-4o (mem0)
+or Gemini-2.5-flash-lite (Hindsight); **our judge will systematically
+differ from theirs**, so the headline is not directly comparable to those
+published numbers without a calibration sample.
+
+The mitigations:
+
+1. We publish `answers.json` (our raw answers, judge-independent) so anyone
+   with another model's API access can re-judge our work and verify.
+2. We run a 30-question stratified Claude Sonnet calibration sample (~$15–30
+   Anthropic credit) so the delta between Qwen and Claude scoring is
+   published alongside the headline.
+3. BEAM's ceiling is ~73% (Hindsight @ 1M, Gemini judge) / ~64% (mem0 @ 1M,
+   GPT-4o judge) — lower than LongMemEval's ~95%. Less headroom → small judge
+   biases shift the number more. This makes the calibration sample
+   **mandatory for credibility**, not optional.
+
+Bench-mode config tunes the cadence of the wiki maintenance pipeline (5s
+tick instead of 60s; writer concurrency 5 instead of 1) so per-conversation
+warmup completes in minutes, not 30+ min. **The pipeline itself is
+identical to production** — same extraction prompts, same wiki maintainer,
+same writer. Only the throughput knobs differ, and they are listed
+explicitly in `docker-compose.bench.yml` so reviewers see them.
+
+---
+
+## Citations
+
+If you reference BrainDB's BEAM numbers, please also cite the original BEAM
+paper — it's their benchmark, we just ran on it:
+
+```
+@inproceedings{tavakoli2026beam,
+  title={Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs},
+  author={Tavakoli, Mohammad and Salemi, Alireza and Ye, Carrie and Abdalla, Mohamed and Zamani, Hamed and Mitchell, J. Ross},
+  booktitle={ICLR 2026},
+  year={2026}
+}
+```
+
+BEAM dataset and code are CC BY-SA 4.0 / MIT respectively; see
+`upstream/LICENSE`.

benchmarks/beam/__init__.py

@@ -0,0 +1,9 @@
+"""BrainDB on BEAM benchmark harness.
+
+Standards-compliant: upstream BEAM (dataset + eval script + judge prompt)
+is invoked verbatim via the git submodule at ``benchmarks/beam/upstream/``.
+Our code (this package) is the adapter only: dataset -> BrainDB ingest,
+question -> BrainDB /agent/query, our answers -> upstream eval.
+
+See ``benchmarks/beam/README.md`` for the full trust model.
+"""