From b4cf08e39484cd1170578e1eb4456de294136cb6 Mon Sep 17 00:00:00 2001 From: "Christian M. Todie" Date: Sat, 18 Jul 2026 07:27:19 -0400 Subject: [PATCH] docs(system-prompting): methodology corpus + leaks analysis + hermes register case study New docs/system-prompting/ (sibling to docs/eval, docs/specs) collating the estate's system- and user-prompt methodology: - methodology synthesis: grounds the house agent-prompting-manual.md in empirical structure + adds the user-prompt half the manual omits - leaks-corpus analysis: 7 findings from todie/system_prompts_leaks (438 prompts, 18 vendors; analysed against current upstream since the fork is ~1yr stale) - hermes ops-register case study: methodology applied to cerebral-agent, incl. a live-pod charter/tool-mismatch finding (no kubectl/helm/MCP wired) Reference only; no code. Register text itself delivered out-of-band per Principle 9. --- ...026-07-18-hermes-ops-register-casestudy.md | 79 +++++++++++++ .../2026-07-18-leaks-corpus-analysis.md | 94 +++++++++++++++ ...2026-07-18-system-prompting-methodology.md | 111 ++++++++++++++++++ docs/system-prompting/README.md | 27 +++++ 4 files changed, 311 insertions(+) create mode 100644 docs/system-prompting/2026-07-18-hermes-ops-register-casestudy.md create mode 100644 docs/system-prompting/2026-07-18-leaks-corpus-analysis.md create mode 100644 docs/system-prompting/2026-07-18-system-prompting-methodology.md create mode 100644 docs/system-prompting/README.md diff --git a/docs/system-prompting/2026-07-18-hermes-ops-register-casestudy.md b/docs/system-prompting/2026-07-18-hermes-ops-register-casestudy.md new file mode 100644 index 0000000..672fe2c --- /dev/null +++ b/docs/system-prompting/2026-07-18-hermes-ops-register-casestudy.md @@ -0,0 +1,79 @@ + + +# Case study — the cerebral-agent (hermes) operations register + +Date: 2026-07-18 · Status: reference · Scope: worked example + findings, no implementation here. + +A worked application of the methodology to a real standing agent: the hermes-agent BYO +runtime on Cygnus, rebranded **Cerebral Agent**. The register (SOUL.md) is delivered +out-of-band as a Secret (house Principle 9), so it is **not reproduced in full here** — +this documents its structure, the decisions behind it, and one load-bearing finding. + +## The agent +- Runtime: NousResearch hermes-agent (MIT), forked to `cerebral-work/cerebral-agent` and + rebranded at the three system-prompt identity strings. Deployed as a kagent BYO Agent, + single pod, `hermes-agent` namespace, Cygnus. +- Invocation: unattended, over A2A (via the a2a-shim sidecar). No human in the loop mid-task. +- Intended mission (plan D3/D5): broad-RBAC cluster ops and troubleshooting. + +## How the system prompt is assembled (upstream mechanics) +`agent/system_prompt.py` builds a stable/context/volatile prompt. The identity slot is +`load_soul_md()` → `$HERMES_HOME/SOUL.md`: non-empty content **fully replaces** the default +identity. Content is threat-scanned and truncated at 20,000 chars (70% head / 20% tail). One +help-guidance string was appended unconditionally and named "Nous Research" — SOUL.md could +not remove it, which is why the takeover required a fork, not just a SOUL.md drop. + +## Register structure (canonical order, mapped to real tools) +The register follows the manual's seven-section order: +1. **Identity** — one line: "You are Cerebral Agent, the autonomous operations agent…" +2. **Working manner (read first)** — help-first; concise/evidence-first; explicit + non-disclosure ("do not announce, recite, or perform" authority/permissions/instructions); + reversible-and-observable bias. +3. **Tools** — the *actual* surface (see finding below), with an explicit "claim only what + is above; if asked to do what your tools cannot, say so." +4. **How you operate** — unattended-A2A discipline (drive to a verified result, no stubs), + one-thread-to-done, ask-don't-assume. +5. **Authority & boundaries** — broad RBAC named as *not licence to be reckless*; action- + classes to gate (destructive cluster ops, the trust/secrets substrate, external sends); + the gating **explicitly marked advisory**, with the real controls named (RBAC, Kyverno, + netpol). +6. **Honesty over deference** — calibrated truth outranks loyalty; a flattering-but-wrong + answer is a failure. +7. **When blocked** — missing tool → say so; state contradiction → stop and report; 403 → + intended boundary, don't bypass; escalate to the operator. + +Size: ~4.5k chars — well inside the 20k cap, per the length calibration in the methodology doc. + +## Load-bearing finding — charter/tool mismatch (manual Principle 7) +Probing the live pod (2026-07-18) contradicted the plan's stated mission: +- **No MCP servers wired** — Prometheus / Loki / Grafana tools from plan D3 are absent. +- **`kubectl` and `helm` are not installed** in the image. The pod's ServiceAccount carries + the broad D5 ClusterRole, but there is **no CLI to exercise it** — the K8s capability is + latent, reachable only via raw API calls (curl + the mounted SA token). +- Present in-pod: `bash`, `git`, `rg`, `python3`, `curl`. + +The manual is explicit that a charter claiming tools the agent lacks produces *confident +failure and erodes trust*. So the register was written to the **real** surface (K8s API via +the SA token; in-pod terminal; memory/skills) and explicitly disclaims the absent tools — +rather than to the aspirational mission. **This gap is the actual blocker on hermes being a +useful ops agent** and is filed as a follow-up (wire kubectl/helm + the grafana-MCP tools, or +narrow the stated mission). The register is honest today; the capability needs to catch up to +the charter, not the other way round. + +## Takeaways generalizable to other estate registers +- Verify the tool surface on the *live* workload before writing capability claims. The plan + is aspirational; the pod is ground truth. +- Mark advisory constraints as advisory and name the real enforcement — do not let a register + imply the prompt is the boundary. +- For an unattended broad-write agent, the "read-only scope fence" of a QA-review prompt + inverts into an explicit action-class taxonomy plus a dominant honesty mandate. + +## References +- `./2026-07-18-system-prompting-methodology.md`, `./2026-07-18-leaks-corpus-analysis.md` +- `unsigned-paas/docs/specs/agent-prompting-manual.md` +- Register delivery: OpenBao `secret/hermes-agent` property `soul-md` → Secret → + `/home/hermes/.hermes/SOUL.md` (subPath, read-only), wired in the hermes-agent chart. diff --git a/docs/system-prompting/2026-07-18-leaks-corpus-analysis.md b/docs/system-prompting/2026-07-18-leaks-corpus-analysis.md new file mode 100644 index 0000000..52c2e1a --- /dev/null +++ b/docs/system-prompting/2026-07-18-leaks-corpus-analysis.md @@ -0,0 +1,94 @@ + + +# Leaks-corpus analysis — what production system prompts actually look like + +Date: 2026-07-18 · Status: reference · Scope: empirical analysis, no implementation. + +Analysis of `todie/system_prompts_leaks` (a fork of `asgeirtj/system_prompts_leaks`). +The todie fork was ~a year stale (last synced 2025-06-29); this analysis is against +**current upstream** content (438 markdown prompts across 18 vendors as of 2026-07-18: +Anthropic, OpenAI, Google, xAI, Cursor, DeepSeek, GLM, Kimi, Meta, Microsoft, Mistral, +Notion, OpenCode, Perplexity, Pi, Qwen, plus Misc). **Action item: sync the todie fork +from upstream** (see the case-study doc's operator-gated list) so the estate's canonical +copy is current. + +Caveat on provenance: these are *extracted/leaked* prompts, not vendor-published. Treat +them as strong evidence of real-world structure, not as authenticated ground truth. Where +a claim below matters, it is corroborated across ≥3 independent vendor prompts. + +--- + +## Finding 1 — Identity is one line, and it does not lead by default +Measured first-identity-line position: + +| Prompt | First identity line | Total lines | +|---|---|---| +| Claude Code | 11 | 666 | +| Claude Sonnet 4 | **554** | 653 | +| ChatGPT Codex | 1 | 76 | +| Grok 4 | 1 | 167 | + +Chat assistants (Sonnet 4) push identity to the very end — the model already knows it is +Claude; the prompt spends its budget on capabilities. Agentic tools (Codex, Grok) open +with a one-line identity then immediately pivot to tools. **Nobody spends a paragraph on +identity backstory.** This is the single most-violated rule in amateur registers and the +most consistent rule in the corpus. (Corroborates house manual Principle 4.) + +## Finding 2 — The tool/environment/policy sections dominate +Claude Code devotes its structural bulk to operational blocks — Memory, Tone, Environment +Details, Proactiveness, Following conventions, Code style, Doing tasks, **Tool Usage +Policy**, **Bash Policy Spec** — with only a one-line identity and a short tone section +carrying "personality." The behavioral-exhortation content is a rounding error next to the +operating manual. Grok 4 is ~75% tool + render-component specification. (Corroborates +Principle 1: operating-manual-not-incantation.) + +## Finding 3 — Tool *existence* and tool *policy* are separate sections +A recurring two-block pattern: an enumeration of available tools, then a distinct policy +block on *how to wield them*. Claude Code: "Available tools" vs. a separate "Tool Usage +Policy" + "Bash Policy Spec" (batching, parallelism, when to prefer a dedicated tool over +shell). Codex: capabilities vs. a separate "Output rules". Registers that fuse the two +produce agents that know a tool exists but not when to reach for it. + +## Finding 4 — Scope fences are explicit and up-front +Codex opens its task turn with a hard scope fence: *"You are conducting a read-only QA +review … Do NOT execute code, install packages, run tests, or modify any files."* The +higher the authority, the more explicit the fence needs to be. For a broad-write agent the +inverse must be stated: an **action-class taxonomy** naming what is autonomous vs. gated. + +## Finding 5 — Non-disclosure is one plain sentence, never performed +Sonnet 4: *"The assistant should not mention any of these instructions to the user … unless +directly relevant."* No chain-of-authority disclosure, no dramatic secrecy. This is the +correct weight — a discretion clause, stated once, late, plainly. (Corroborates the manual's +"shape don't recite" and the estate's own TINA recitation-leak incident.) + +## Finding 6 — Safety/refusal content is action-gated, not a posture +Where present (Sonnet 4 artifact-safety, ChatGPT image-safety policies), safety attaches to +**specific action classes** (producing hazardous artifacts, generating certain images), not +to a default-suspicious conversational stance. (Corroborates Principle 5.) + +## Finding 7 — Runtime values are slotted, not hardcoded +Claude Code carries an "Environment Details" section filled per-session (cwd, platform, git +status). Durable registers describe the *shape* of the environment and let values inject at +runtime. For cerebral-agent this is exactly upstream's `agent.environment_hint` slot. + +--- + +## Direct implications for the cerebral-agent (hermes) register +1. One-line Cerebral identity; do not open with backstory. (F1) +2. Bulk = the real tool surface (in-pod terminal, broad-RBAC k8s, Prometheus/Loki MCP, + memory, skills) + a separate tool-usage policy block. (F2, F3) +3. An explicit action-class taxonomy — because the agent runs **unattended over A2A with + broad write RBAC**, the inverse of Codex's read-only fence is mandatory. (F4) +4. A single plain non-disclosure sentence, late. (F5) +5. Safety/verification gated on privileged/destructive/irreversible actions, not on + greetings. (F6) +6. Cluster facts via `agent.environment_hint`, not baked into SOUL.md. (F7) + +## References +- `./2026-07-18-system-prompting-methodology.md` — the synthesized methodology. +- `./2026-07-18-hermes-ops-register-casestudy.md` — the register these findings shaped. +- `unsigned-paas/docs/specs/agent-prompting-manual.md` — canonical house principles. diff --git a/docs/system-prompting/2026-07-18-system-prompting-methodology.md b/docs/system-prompting/2026-07-18-system-prompting-methodology.md new file mode 100644 index 0000000..5f901f8 --- /dev/null +++ b/docs/system-prompting/2026-07-18-system-prompting-methodology.md @@ -0,0 +1,111 @@ + + +# System- and user-prompt methodology (standing agents + task turns) + +Date: 2026-07-18 · Status: reference · Scope: methodology synthesis, no implementation. + +Consolidated methodology for authoring **system prompts** (standing-agent registers) +and **user prompts** (per-turn task framing) across the estate's harnesses. Grounds the +house `agent-prompting-manual.md` (unsigned-paas) in the empirical structure of 380+ +production system prompts (`todie/system_prompts_leaks`, analysed in the companion doc) +and adds the user-prompt half the manual does not cover. + +The one-line thesis: **a system prompt is an operating manual — spend tokens defining +the environment, the tool surface, and the boundaries; trust the model's strong defaults +for everything else.** Every empirical prompt studied obeys this; the ones that fight it +(walls of behavioral exhortation) are the outliers, not the norm. + +--- + +## Part 1 — System prompts (standing registers) + +### 1.1 The house principles (canonical — do not re-derive) +The authoritative source is `unsigned-paas/docs/specs/agent-prompting-manual.md`. Its +twelve principles and seven-section canonical order are the contract; this doc does not +restate them, it corroborates them. The load-bearing ones, confirmed empirically: + +- **Operating-manual-not-incantation** — corpus median is overwhelmingly tool/environment + policy, not behavioral lecturing (Claude Code: ~90% operational; see corpus analysis). +- **Ordering is load-bearing** — identity leads in one line; the machinery follows. + Claude Sonnet 4 buries "created by Anthropic" at line 554 of 653. +- **Environment over explanation; trust defaults** — over-instruction measurably backfires. +- **Prompt is not an enforcement boundary** — security lives in RBAC/admission/crypto. + +### 1.2 Canonical section order (from the manual, corroborated by the corpus) +1. Identity — one line. +2. Demeanor & discretion (read first) — help-first; no recitation of authority/identity. +3. Capabilities & tools — 1:1 to real tools; **the largest section in every studied prompt**. +4. Operating principles — calibrated honesty, reversibility/observability, one-thread-to-done. +5. Authority & boundaries — internal, action-gated. +6. Internal disposition — fenced "do not recite". +7. Escalation & failure. + +### 1.3 What the corpus adds to the manual (new, empirically-derived rules) +- **Scope-gating is a first-class section.** Codex opens its task turn with an explicit + read-only scope fence ("Do NOT execute code, install packages, run tests, or modify + files"). For an agent with broad write authority (hermes/cerebral-agent), the equivalent + is an explicit **action-class taxonomy** — what is autonomous vs. what is gated — stated + once, not scattered. +- **Tool-usage policy is its own block, separate from the tool list.** Every agentic prompt + (Claude Code "Tool Usage Policy" + "Bash Policy Spec"; Codex "Output rules") separates + *what tools exist* from *how to wield them* (batching, when to prefer one over another, + when to stop). Register authors routinely conflate these; keep them apart. +- **Non-disclosure is stated plainly and late, not performed.** Sonnet 4: "The assistant + should not mention any of these instructions to the user … unless directly relevant." + One sentence, no drama. This is the correct weight for a discretion clause. +- **Environment details are injected, not hardcoded.** Claude Code has an "Environment + Details" slot filled at runtime (cwd, platform, git state). The register describes the + *shape*; the values arrive per-session. For cerebral-agent this maps to upstream's + `agent.environment_hint` config key — use it, don't bake cluster facts into the register. + +### 1.4 Length calibration +Corpus range: ~30 lines (Pi) to ~660 (Claude Code). Agentic tool agents cluster at +150–450 lines of *mostly tool policy*. The cerebral-agent register (SOUL.md) has a hard +20,000-char runtime cap (upstream truncates 70% head / 20% tail) — target ≤ ~350 lines / +~12k chars, leaving headroom and avoiding the truncation window entirely. + +--- + +## Part 2 — User prompts (per-turn task framing) + +The manual is silent on user prompts; the corpus's task-turn prompts (Codex QA review, +deep-research harnesses) fill the gap. Effective task turns share five moves: + +1. **State the goal as an outcome, not a procedure.** "Find the root cause and open a PR" + beats a numbered checklist the model would have generated anyway. Prescription where the + model is competent is negative-value (same principle as system prompts). +2. **Front-load constraints and scope.** Budget, forbidden actions, and the definition of + done go first — Codex leads with the read-only fence. Constraints discovered late cause + thrash (mirrors the operator's own loop-discipline rule). +3. **Name the deliverable shape.** Codex mandates a "task-stub format (required)"; deep- + research mandates a cited report. Ambiguous output shape is the top cause of a wrong-shaped + answer. State the schema. +4. **Affirmative framing.** "Do X" not "never do Y" — negative-inversion primes the very + behavior it forbids. Holds identically for user and system prompts. +5. **Provide the read-surface, forbid re-fetching.** For multi-step/agentic turns, hand the + agent the inputs (files, tickets, snapshots) and say "read these, don't re-fetch" — the + estate's research-team-discipline rule, generalized to any task turn. + +**System vs. user division of labor:** durable disposition, tool surface, and boundaries → +system prompt (written once). Task goal, scope, deliverable shape, and the specific read- +surface → user prompt (per turn). Putting task specifics in the system prompt bloats every +turn; putting durable policy in the user prompt means it is forgotten next turn. + +--- + +## Part 3 — Authoring loop +1. Draft against the canonical section order (§1.2). +2. Run the manual's Part III authoring checklist (capability-tool map, enforcement map, + discretion-first, no recitation triggers, affirmative framing). +3. Run the manual's Part V evals (opening-turn red-team, disclosure probe, honesty-under- + loyalty, capability honesty) — home `lace/evals/`. +4. Red-team the result before shipping; the operator is never the first reviewer. + +## References +- `unsigned-paas/docs/specs/agent-prompting-manual.md` — canonical house methodology. +- `./2026-07-18-leaks-corpus-analysis.md` — empirical structure from the leaks corpus. +- `./2026-07-18-hermes-ops-register-casestudy.md` — worked example (cerebral-agent SOUL.md). diff --git a/docs/system-prompting/README.md b/docs/system-prompting/README.md new file mode 100644 index 0000000..5930570 --- /dev/null +++ b/docs/system-prompting/README.md @@ -0,0 +1,27 @@ + + +# system-prompting — methodology corpus + +Consolidated reference for authoring system prompts (standing-agent registers) and user +prompts (per-turn task framing) across the estate's harnesses. Grounds the house prompting +manual (`unsigned-paas/docs/specs/agent-prompting-manual.md`) in the empirical structure of +production system prompts, and works one real example end-to-end. + +## Contents + +| Doc | What it is | +|---|---| +| [`2026-07-18-system-prompting-methodology.md`](./2026-07-18-system-prompting-methodology.md) | The synthesized methodology — system-prompt principles (corroborating the house manual) + the user-prompt half the manual omits. | +| [`2026-07-18-leaks-corpus-analysis.md`](./2026-07-18-leaks-corpus-analysis.md) | Empirical analysis of `todie/system_prompts_leaks` (438 prompts, 18 vendors) — seven findings on how real production prompts are structured. | +| [`2026-07-18-hermes-ops-register-casestudy.md`](./2026-07-18-hermes-ops-register-casestudy.md) | The methodology applied to the cerebral-agent (hermes) ops register, including a charter/tool-mismatch finding from the live pod. | + +## Source of truth +- **Canonical principles**: `unsigned-paas/docs/specs/agent-prompting-manual.md`. This + corpus corroborates and extends it; it does not supersede it. +- **Leaks corpus**: `todie/system_prompts_leaks` (fork of `asgeirtj/system_prompts_leaks`). + The fork is stale — **sync it from upstream before mining** (the analysis here used current + upstream content as of 2026-07-18). +- **Evals**: the register evaluation protocol lives in `unsigned-paas` `lace/evals/`.