feat(custom-profiles): opt-in domain profiles — Hacker News chronicle + Hermes situational tracker

.env.example

@@ -103,15 +103,15 @@ AGENT_VERBOSE=false
 INGEST_POLL_INTERVAL=7
 
 # Wiki scheduler HTTP read-timeout (seconds) on /wiki/maintain and
-# /wiki/write calls. Default 1200 (20 min). Local quantised models
+# /wiki/write calls. Default 2400 (40 min). Local quantised models
 # (Qwen 27B AWQ-INT4 on vLLM) routinely take 6-15 min for a full wiki
 # body; setting this below ~600 caused the scheduler to give up while
 # the api kept working — queue drained slower than reality. Raise if
 # you see "Read timed out" in the scheduler log AND the corresponding
 # write actually committed (check `wikis_ext.revision`); lower only if
 # you specifically want quicker scheduler turnover. The api itself is
 # unbounded by this; this only controls the scheduler's patience.
-# WIKI_AGENT_TIMEOUT=1200
+# WIKI_AGENT_TIMEOUT=2400
 
 # Per-wiki cooldown on attach claims (seconds). Default 300 (5 min).
 # Once the OLDEST pending attach for a given wiki is this old, the

.gitignore

@@ -65,3 +65,33 @@ data_test/
 # Hermes sandbox (integrations/hermes/sandbox): throwaway agent profile dir —
 # holds a .env with the LLM key + provider state; never commit it.
 integrations/hermes/sandbox/hermes-data/
+
+# Custom profiles — only the contract docs + the example/ profile are public.
+# Every real profile (its prompt fragments, ingestor.py, and .env secret) is
+# local-only: domain-specific content + credentials that must not be committed.
+custom-profiles/*
+!custom-profiles/README.md
+!custom-profiles/SKILL.md
+!custom-profiles/example/
+!custom-profiles/example/**
+!custom-profiles/hackernews/
+!custom-profiles/hackernews/**
+# ...but never the hackernews ingestor's runtime files (created when it runs)
+custom-profiles/hackernews/.env
+custom-profiles/hackernews/.state/
+custom-profiles/hackernews/__pycache__/
+!custom-profiles/gdrive/
+!custom-profiles/gdrive/**
+# ...but never the gdrive ingestor's runtime files OR its credentials — the
+# service-account JSON key must NEVER be committed (public repo).
+custom-profiles/gdrive/.env
+custom-profiles/gdrive/.state/
+custom-profiles/gdrive/__pycache__/
+custom-profiles/gdrive/*.json
+!custom-profiles/hermes/
+!custom-profiles/hermes/**
+# ...but never the hermes ingestor's runtime files OR its secret (the Hermes bearer
+# token lives in .env, which must never be committed).
+custom-profiles/hermes/.env
+custom-profiles/hermes/.state/
+custom-profiles/hermes/__pycache__/

CHANGELOG.md

@@ -5,6 +5,57 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.9.0] — 2026-06-26
+
+Headline: **custom profiles** — opt-in, self-contained overlays that reshape what BrainDB ingests
+and how its auto-maintained wikis read, with ZERO effect on defaults when inactive. Two ship ready
+to run: a keyless **Hacker News** chronicle that grows company/project/tech pages from the firehose,
+and a **Hermes situational tracker** that asks an external web-search agent about an evolving
+real-world situation and keeps a dated, sourced, dashboard-shaped status page. Also in this release:
+the wiki writer now rejects blank/garbage bodies before persist, and the scheduler write-timeout
+default is raised for slower local models.
+
+### Added
+
+- **Custom profiles framework.** An opt-in composition layer (`braindb/custom_profile.py`,
+  `braindb/profile_runner.py`) that shapes ingestion + wiki output from a single self-contained
+  folder under `custom-profiles/<name>/`, activated by `CUSTOM_PROFILE` — defaults are byte-for-byte
+  unchanged when unset. A profile may carry `wiki_maintainer.add.md` / `wiki_writer.add.md` prompt
+  fragments (appended after the base prompt; a `*.replace.md` swaps it wholesale) and an optional
+  supervised `ingestor.py` (plus its own gitignored `.env`) that feeds a custom source into the
+  existing file-watcher. Profiles compose (comma-separated). Core API, schema, and base prompts are
+  untouched. Ships with an `example/` template and a `gdrive/` source ingestor alongside the two
+  profiles below.
+- **`hackernews` profile — keyless tech chronicle.** Polls the public Hacker News API, drops each new
+  story into `data/sources/`, and shapes the resulting entity pages as tech-entity chronicles
+  (type / homepage / category, a dated `current-developments` feed that ages into `background`, every
+  claim cited). No API key; runs immediately.
+- **`hermes` profile — situational tracker.** On a schedule, asks an external
+  [Hermes Agent](https://github.com/NousResearch/hermes-agent) (web-search, OpenAI-compatible gateway)
+  a dated, stateless question and folds each answer into a central situational wiki shaped as a
+  dashboard — status + confidence + most-credible-source, then `now` / `recent-days` / `recent-weeks`
+  / `forecast` / `history`, every line dated and `[[ref:UUID]]`-cited, conflicts resolved by
+  credibility or flagged low-confidence — while the other entities each report mentions still get
+  their own cross-linked wikis. Dormant by default (no key, so it idles and never calls); a
+  `HERMES_SAMPLE_FILE` dry-run exercises the whole pipeline offline.
+- **Current date in wiki prompts.** The maintainer and writer prompts now carry today's UTC date, so
+  recency / "as of" / dated-bucket reasoning has an anchor.
+
+### Changed
+
+- **Wiki scheduler write-timeout default raised 1200 -> 2400s (20 -> 40 min).** `WIKI_AGENT_TIMEOUT`;
+  smaller/slower quantised local models (e.g. Gemma 12B) can exceed 20 min on a full-body write, so
+  the old default abandoned slow-but-completing writes and forced retries. Client-patience only — the
+  API is not bounded by it and a write past the deadline still commits.
+
+### Fixed
+
+- **Writer rejects blank/garbage bodies before persist.** A weak model sometimes emits serialization
+  junk (`""`, `body=""`, `{}`) instead of a page; the previous whitespace-only gate let it through to
+  be written as content and bump the revision, self-perpetuating into empty, high-revision pages. A
+  single blank-body check now routes such writes to the existing retry path instead of persisting —
+  good pages can't be overwritten by junk, and the entity stays a healable orphan for the next cycle.
+
 ## [0.8.0] — 2026-06-16
 
 Headline: the wiki maintainer now batches related orphan triage into ONE LLM call —

README.md

@@ -352,6 +352,24 @@ automatically. To run without it, bring the stack up excluding the service or
 scale it to 0 (`docker compose up -d --scale wiki_scheduler=0`), exactly as
 you would for the watcher; or point `LLM_PROFILE` at a local model.
 
+### Custom profiles — shape ingestion and wiki structure
+
+A **custom profile** is an opt-in, self-contained folder under [`custom-profiles/`](custom-profiles/)
+that reshapes what BrainDB ingests and how its auto-maintained wikis read — without touching core. A
+profile can append (or replace) the maintainer/writer prompts via `*.add.md` fragments and run its own
+supervised `ingestor.py` to feed a custom source into the watcher. Activate with `CUSTOM_PROFILE=<name>`
+(comma-separated to compose); when unset, defaults are byte-for-byte unchanged.
+
+Two ship ready to run:
+
+- **[`hackernews`](custom-profiles/hackernews/)** — keyless: polls Hacker News and grows
+  company/project/tech pages as dated chronicles.
+- **[`hermes`](custom-profiles/hermes/)** — asks an external web-search agent about an evolving
+  situation and keeps a dated, sourced, dashboard-shaped status page; dormant until configured.
+
+See [`custom-profiles/README.md`](custom-profiles/README.md) for the contract and
+[`SKILL.md`](custom-profiles/SKILL.md) to author your own.
+
 ## Frontend (read-only browser UI)
 
 A small vanilla-JS frontend ships under `frontend/` — no build step — and is served by the stack itself: after `docker compose up -d`, open **`http://localhost:8642`** (change the port with `FRONTEND_PORT` in `.env`). Three views (Reader for browsing wikis, Graph for visual exploration, Ops for watching the maintainer/writer pipeline) plus an Ask drawer that talks to the agent endpoint. The UI calls the BrainDB API at `http://localhost:8000` from your browser.

braindb/custom_profile.py

@@ -0,0 +1,84 @@
+"""
+Optional custom-profile prompt composition. ZERO effect unless CUSTOM_PROFILE is set.
+
+    CUSTOM_PROFILE=cityfalcon_news                # one profile
+    CUSTOM_PROFILE=company_base,cityfalcon_news   # several, composed in order
+    CUSTOM_PROFILE_DIR=custom-profiles            # optional; default shown
+
+A profile is a folder under CUSTOM_PROFILE_DIR holding prompt fragments named
+`<target>.add.md` (appended to the base prompt) and/or `<target>.replace.md`
+(swaps the base template entirely). `target` is the prompt key a call site asks
+for, e.g. "wiki_maintainer" or "wiki_writer".
+
+Two composition primitives, used at each prompt-assembly site:
+
+  replace_or_base(base_template, target) -> the active `<target>.replace.md`
+      (last active profile wins) or `base_template` unchanged. Applied to the
+      TEMPLATE *before* placeholder substitution, so a replace author keeps the
+      base's `{seeds}` / `%%MODE%%` tokens and owns the machine contract.
+
+  additions(target) -> "\n\n" + joined `<target>.add.md` fragments across the
+      active profiles in list order, or "". Appended *after* substitution, so a
+      fragment may contain any characters (including `{}`) with no str.format
+      hazard.
+
+Both degrade to the no-op result (the base, or "") when CUSTOM_PROFILE is unset,
+a folder/file is missing, or anything goes wrong — they never raise into a
+prompt-assembly path. Read-at-call: a dropped-in fragment takes effect on the
+next assembly, and tests can monkeypatch CUSTOM_PROFILE without reimporting.
+"""
+import os
+from pathlib import Path
+
+_REPO_ROOT = Path(__file__).resolve().parent.parent  # braindb/.. -> repo root
+
+
+def _active_profiles() -> list[str]:
+    """Profile names from CUSTOM_PROFILE (comma-separated), in order. [] when unset."""
+    raw = os.getenv("CUSTOM_PROFILE", "").strip()
+    return [p.strip() for p in raw.split(",") if p.strip()]
+
+
+def _profiles_root() -> Path:
+    base = os.getenv("CUSTOM_PROFILE_DIR", "custom-profiles").strip() or "custom-profiles"
+    p = Path(base)
+    return p if p.is_absolute() else _REPO_ROOT / p
+
+
+def _read(path: Path) -> str | None:
+    try:
+        return path.read_text(encoding="utf-8") if path.is_file() else None
+    except Exception:
+        return None
+
+
+def replace_or_base(base_template: str, target: str) -> str:
+    """Return the active `<target>.replace.md` (last profile wins), else
+    `base_template` unchanged. Never raises."""
+    try:
+        root = _profiles_root()
+        chosen = base_template
+        for name in _active_profiles():
+            text = _read(root / name / f"{target}.replace.md")
+            if text is not None:
+                chosen = text
+        return chosen
+    except Exception:
+        return base_template
+
+
+def additions(target: str) -> str:
+    """Return '\\n\\n' + joined `<target>.add.md` fragments across active profiles
+    (list order), or '' when none / on any error. Never raises."""
+    try:
+        root = _profiles_root()
+        frags: list[str] = []
+        for name in _active_profiles():
+            text = _read(root / name / f"{target}.add.md")
+            if text is not None:
+                stripped = text.strip()
+                if stripped:
+                    frags.append(stripped)
+        return ("\n\n" + "\n\n".join(frags)) if frags else ""
+    except Exception:
+        return ""

braindb/main.py

@@ -11,7 +11,7 @@
 app = FastAPI(
     title="BrainDB",
     description="Memory database and REST API for LLM agents",
-    version="0.8.0",
+    version="0.9.0",
 )
 
 app.add_middleware(

braindb/profile_runner.py

@@ -0,0 +1,109 @@
+"""
+Generic custom-profile ingestor supervisor — ONE loop, same posture as
+ingest_watcher.py / wiki_scheduler.py. DORMANT unless CUSTOM_PROFILE is set.
+
+For each active profile (CUSTOM_PROFILE, comma-separated) that ships an
+`ingestor.py`, this launches `python <profile>/ingestor.py` as an ISOLATED
+subprocess and keeps it alive (restart with backoff on exit). It holds no
+profile-specific knowledge and never imports profile code: a broken ingestor
+can only crash its own subprocess, never the api / watcher / wiki.
+
+No active profile, or no active profile ships an ingestor.py -> the runner
+sleeps and makes zero calls (exactly like WIKI_ENABLED=false). This is what
+lets ONE switch (CUSTOM_PROFILE) bring up prompt-shaping AND a profile's
+ingestor together, while the base docker-compose.yml carries only a neutral,
+dormant-by-default service. See custom-profiles/README.md.
+"""
+import logging
+import os
+import subprocess
+import sys
+import time
+from pathlib import Path
+
+_REPO_ROOT = Path(__file__).resolve().parent.parent  # braindb/.. -> repo root
+
+# Liveness-check cadence and the minimum spacing between relaunch attempts for a
+# crashing ingestor (prevents a tight crash-loop). Both are seconds.
+POLL_INTERVAL = int(os.getenv("PROFILE_RUNNER_INTERVAL", "15"))
+RESTART_BACKOFF = int(os.getenv("PROFILE_RUNNER_BACKOFF", "30"))
+
+logging.basicConfig(
+    level=logging.INFO,
+    format="%(asctime)s [profile-runner] %(message)s",
+    datefmt="%H:%M:%S",
+    stream=sys.stdout,
+)
+log = logging.getLogger("profile-runner")
+
+
+def _profiles_root() -> Path:
+    base = os.getenv("CUSTOM_PROFILE_DIR", "custom-profiles").strip() or "custom-profiles"
+    p = Path(base)
+    return p if p.is_absolute() else _REPO_ROOT / p
+
+
+def _active_profiles() -> list[str]:
+    raw = os.getenv("CUSTOM_PROFILE", "").strip()
+    return [p.strip() for p in raw.split(",") if p.strip()]
+
+
+def _ingestors() -> dict[str, Path]:
+    """{profile_name: ingestor.py path} for active profiles that ship one."""
+    root = _profiles_root()
+    out: dict[str, Path] = {}
+    for name in _active_profiles():
+        script = root / name / "ingestor.py"
+        if script.is_file():
+            out[name] = script
+    return out
+
+
+def main() -> None:
+    ingestors = _ingestors()
+    if not ingestors:
+        log.info("no active profile ingestor (CUSTOM_PROFILE=%r). Idle.",
+                 os.getenv("CUSTOM_PROFILE", ""))
+        # Sleep forever — container stays Up, zero calls. Toggle via env + restart.
+        while True:
+            time.sleep(3600)
+
+    log.info("supervising %d profile ingestor(s): %s",
+             len(ingestors), ", ".join(ingestors))
+
+    procs: dict[str, subprocess.Popen] = {}
+    launched_at: dict[str, float] = {}
+
+    def _launch(name: str, script: Path) -> None:
+        log.info("launching ingestor for profile '%s': %s", name, script)
+        # Isolated subprocess; cwd at repo root so the script's relative paths
+        # (its own folder, data/sources/) resolve exactly as in the containers.
+        procs[name] = subprocess.Popen([sys.executable, str(script)], cwd=str(_REPO_ROOT))
+        launched_at[name] = time.time()
+
+    for name, script in ingestors.items():
+        _launch(name, script)
+
+    while True:
+        try:
+            for name, script in ingestors.items():
+                proc = procs.get(name)
+                if proc is None:
+                    _launch(name, script)
+                    continue
+                if proc.poll() is None:
+                    continue  # still running
+                # Exited — relaunch once the backoff window (since last launch)
+                # has elapsed, so a fast crash-loop is throttled while a
+                # long-lived process that dies restarts promptly.
+                if time.time() - launched_at[name] >= RESTART_BACKOFF:
+                    log.warning("ingestor '%s' exited (code=%s); restarting",
+                                name, proc.returncode)
+                    _launch(name, script)
+        except Exception as e:
+            log.exception("supervisor loop error: %s", e)
+        time.sleep(POLL_INTERVAL)
+
+
+if __name__ == "__main__":
+    main()