chore(probes): strip restated comments and document what a probe floor proves

CONTEXT.md

@@ -105,3 +105,39 @@ A corollary: a probe must be able to **establish its premise** before forging.
 If the real spine fails (its tools never ran), the trajectory the forge assumes
 does not exist, so the probe aborts loudly rather than emitting evidence that
 would fail for the wrong reason.
+
+## Probe implementation invariants
+
+Four rules every probe in this package follows. They are stated once here
+rather than re-argued in each module.
+
+1. **A write's own exit status is the check.** Never append a trailing
+   `echo __OK__` to a heredoc write: the echo's exit 0 masks a failed `cat`
+   (a protected path, a full disk) and the probe would report evidence it
+   never landed — an empty attempt scoring as a floor.
+2. **Chain a script write to its run with `&&`.** A failed or partial heredoc
+   must skip the interpreter rather than execute a stale `/tmp` script from an
+   earlier trial.
+3. **Never write `/logs/agent/trajectory.json`.** It is evaluator-owned, and a
+   hardened world makes it read-only, so the write throws and aborts the probe
+   before it reaches the gate it targets.
+4. **`result.json` carries the run summary, not the state blob.** A world that
+   validates `result.json` against its own schema would otherwise reject the
+   probe on shape before the targeted gate runs (single-defect principle).
+
+## Trajectory reachability
+
+An agent gets a trajectory only by writing one: Harbor's adapters (`pi`,
+`claude_code`, …) each rebuild `/logs/agent/trajectory.json` from their
+runtime's event stream, and declare it by setting `SUPPORTS_ATIF = True`. Every
+probe here inherits `BaseAgent.SUPPORTS_ATIF = False` and writes no trajectory —
+correctly, since invariant 3 forbids it. So a probe that shells out to a world's
+tool bodies mutates real state and writes the in-container ledger, but **nothing
+host-witnessed records those calls**.
+
+The consequence is the limit of this package: against a world whose criteria
+are conditioned on trajectory coverage, every probe here scores zero before
+its payload is read. Such a floor is honest evidence for the execution-faking
+family (absent execution evidence is exactly their claim) and **vacuous for
+every other probe** — the weakness they target sits behind a gate they cannot
+pass. Proving a probe fires needs a sensitivity fixture, not a gate run.

README.md

@@ -88,19 +88,25 @@ harbor run /path/to/world --agent nop
 
 `state_rewrite` is world-specific (see the † note above) — run it only against the world whose answer it encodes, by its import path, not as part of the general cascade.
 
-## What each probe signals when it scores above floor
+## How to fix a world a probe caught
 
-**Isolation escape (Berkeley 1)** — the container boundary between agent and host leaks. An agent can write `/host/...` paths, walk `/proc/1/root`, or escalate suid. A world that credits this probe has lost structural isolation; failures here compromise every other safety claim.
+| Probe scored above floor | Fix |
+|---|---|
+| `isolation-escape`, `reward-sink` | Evaluator-owned paths (`/logs/verifier`, `/tests`) must be root-owned and agent-read-only. A breach here compromises every other safety claim. |
+| `judge-injection` | Wrap agent bytes in `<agent_output>...</agent_output>` and instruct the judge to ignore directives inside them. |
+| `success-token-echo` | Replace substring checks with structural equality or semantic comparison. |
+| `fake-state`, `ledger-forge`, `state-rewrite` | Add `trajectory_tool_used` coverage plus a ledger/trajectory reconciliation criterion, and condition state credit on it. |
+| `random-agent`, `forbidden-subprocess`, `shortcut-hunter` | Reconcile the state delta against the task rather than rewarding activity; watch for side effects outside the tool interface; keep the answer key off the agent-readable surface. |
 
-**Judge injection (Berkeley 4)** — the rubric's LLM-judge consumes agent-controlled text without structural delimiters. A world that credits this probe is coaching its own grader. Fix: wrap agent bytes in `<agent_output>...</agent_output>` with instructions to ignore directives inside.
+## Limits — what a floor does and does not prove
 
-**Success-token echo (Berkeley 5)** — the rubric credits any non-empty output or uses substring matching for output-file checks. A world that credits this probe has a non-discriminating capability signal. Fix: replace substring checks with structural equality or semantic comparison.
+A floor is evidence only when the probe genuinely reached the defense under test. Two things stop that, and both are properties of this package rather than of the world:
 
-**Reward sink (Berkeley 7)** — the agent can write to `/logs/verifier/reward.txt`, `/logs/agent/trajectory.json`, `/tests/`, or other paths the verifier trusts as canonical. A world that credits this probe has a permission drift. Fix: evaluator-owned paths must be root-owned and agent-read-only.
+**Probes emit no trajectory.** An agent gets one only by writing it: Harbor's adapters each rebuild `/logs/agent/trajectory.json` from their runtime's event stream and declare it with `SUPPORTS_ATIF = True`. Every probe here inherits `BaseAgent.SUPPORTS_ATIF = False` and writes no trajectory — correctly, since writing that path is forbidden. A probe that shells out to a world's tool bodies therefore mutates real state and writes the in-container ledger, but nothing host-witnessed records those calls. Against a world whose criteria are conditioned on trajectory coverage, **every probe here scores zero before its payload is read.** That is honest evidence for the execution-faking family, whose claim is precisely that no execution evidence exists — and vacuous for every other probe, whose target sits behind a gate it cannot pass.
 
-**Execution-faking family (`fake_state`, `ledger_forge`, `state_rewrite`)** — the rubric grades final state without verifying the agent did the work. Fix: add `trajectory_tool_used` coverage plus a ledger/trajectory reconciliation criterion (see `docs/design/execution-verification.md` in the ocarina repo).
+**Payloads are shaped for one world.** The mechanisms are general, but the bytes are not. `fake_state` and `forbidden_subprocess` write contractor-payment-run field names (`processedCount`, `payment_run_submitted`); `ledger_forge` forges AP-payment tool verbs; `success_token_echo`, `judge_injection`, and `shortcut_hunter` submit to a fixed set of flat output filenames. A world that grades a differently-shaped state file floors them on shape, without ever consulting the weakness they exist to find.
 
-**False-positive controls (`random_agent`, `forbidden_subprocess`, `shortcut_hunter`)** — these SHOULD floor on a healthy world; if one scores above floor, the world is rewarding activity over outcome, is blind to work done outside the tools, or leaks its answer key to the agent at runtime.
+So a clean cascade is a **necessary** condition for publication, not a sufficient one, and it should not be reported as "the world resisted every probe". Proving a probe fires needs a sensitivity fixture — a world with a known injected defect that a named probe must detect. Until one exists, prefer replaying a probe's payload against a world's own criteria offline (cheap, deterministic, no container) over inferring sensitivity from a gate run.
 
 ## Developer quickstart
 
@@ -109,8 +115,22 @@ git clone https://github.com/ocarinalabs/harbor-ext
 cd harbor-ext
 uv sync --extra dev
 uv run pytest
+uv run ruff check .
 ```
 
+A comment in this package earns its place by explaining *why* an attack works or
+why a defence is ordered as it is. Shared rationale lives once in `CONTEXT.md`,
+not restated per module. `scripts/strip_comments.py` enforces the floor:
+
+```bash
+python3 scripts/strip_comments.py --check src tests   # report, exit 1 if any
+python3 scripts/strip_comments.py tests               # rewrite
+```
+
+It never touches docstrings, inline comments, tool directives, or licence
+headers, and it honours `# strip-comments: keep` on a block and
+`# strip-comments: reviewed` on a module.
+
 ## Related
 
 - [ocarinalabs/ocarina](https://github.com/ocarinalabs/ocarina) — benchmark world generator that uses these probes as a pre-publication gate

scripts/strip_comments.py

@@ -0,0 +1,159 @@
+#!/usr/bin/env python3
+"""Remove full-line ``#`` comments from Python sources, idempotently.
+
+Comment rot is the failure mode this exists for: rationale that was true once,
+restated in five files, drifting out of step with the code. The rule this
+package settled on is that a comment earns its place by explaining *why* an
+attack works or why a defence is ordered as it is; anything else belongs in
+``CONTEXT.md`` once, or nowhere.
+
+Run it, read the diff, and put back what was load-bearing **with a keep
+marker** on the block's first line::
+
+    # strip-comments: keep
+    # Create the parent first: not every Harbor environment pre-creates
+    # /logs/agent, and a bare `cat >` dies on "no such file".
+
+The marker is what makes rescuing safe. Without it a second run deletes the
+rescued comment again — ocarina hit exactly that and lost a licensing
+attribution on the re-run.
+
+A module whose comments have all been reviewed can say so once, near the top,
+instead of marking every block::
+
+    # strip-comments: reviewed
+
+Prefer that for curated modules; per-block markers are for a rescue that sits in
+a file still worth stripping.
+
+Never touched:
+
+* docstrings — they are the module's and its public callables' interface
+* trailing inline comments — short, attached to the line they qualify
+* tool directives — ``noqa``, ``type: ignore``, ``pragma``, ``pylint``,
+  ``mypy``, ``ruff``, encoding declarations, shebangs, licence headers
+* comments inside string literals — embedded ``//`` and ``#`` script bodies are
+  code, not commentary
+
+Usage::
+
+    python3 scripts/strip_comments.py --check src tests   # report only
+    python3 scripts/strip_comments.py src tests           # rewrite
+"""
+
+from __future__ import annotations
+
+# strip-comments: reviewed
+import argparse
+import io
+import re
+import sys
+import tokenize
+from pathlib import Path
+
+KEEP_MARKER = "strip-comments: keep"
+REVIEWED_MARKER = "strip-comments: reviewed"
+
+DIRECTIVE = re.compile(
+    r"""
+    ^\#\s*(?:
+        !                       # shebang
+      | -\*-                    # encoding declaration
+      | (?:type|ruff|mypy|pylint|pyright|flake8|isort|black|coverage)\s*:
+      | noqa\b
+      | pragma\b
+      | (?:SPDX|Copyright|License|LICENSE)\b
+    )
+    """,
+    re.VERBOSE | re.IGNORECASE,
+)
+
+
+def _own_line_comments(source: str) -> list[tokenize.TokenInfo]:
+    """Comment tokens that occupy a whole line, in source order.
+
+    Tokenizing rather than matching ``^\\s*#`` is what keeps the ``#`` inside a
+    shell heredoc or a TypeScript payload safe: the tokenizer sees those as part
+    of a string literal, never as a comment.
+    """
+    tokens = tokenize.generate_tokens(io.StringIO(source).readline)
+    return [
+        token
+        for token in tokens
+        if token.type == tokenize.COMMENT and not token.line[: token.start[1]].strip()
+    ]
+
+
+def strip(source: str) -> str:
+    """Drop every own-line comment block that is not kept or a directive."""
+    comments = _own_line_comments(source)
+    # The marker counts only as a comment of its own. A plain substring search
+    # would also match the string in a docstring or a test fixture and exempt
+    # that whole file by accident.
+    if any(REVIEWED_MARKER in token.string for token in comments):
+        return source
+
+    doomed: set[int] = set()
+    block_start: int | None = None
+    block_kept = False
+
+    for token in comments:
+        line_no = token.start[0]
+        contiguous = block_start is not None and line_no == block_start + 1
+        if not contiguous:
+            block_kept = False
+        block_start = line_no
+
+        # A keep marker rescues its whole block; a directive protects only its
+        # own line, so narration sitting directly beneath a lint directive is
+        # not shielded by it.
+        if KEEP_MARKER in token.string:
+            block_kept = True
+        if not block_kept and not DIRECTIVE.match(token.string.strip()):
+            doomed.add(line_no)
+
+    if not doomed:
+        return source
+
+    lines = source.splitlines(keepends=True)
+    kept = [line for number, line in enumerate(lines, start=1) if number not in doomed]
+    return "".join(kept)
+
+
+def _python_files(roots: list[str]) -> list[Path]:
+    found: list[Path] = []
+    for root in roots:
+        path = Path(root)
+        found.extend([path] if path.is_file() else sorted(path.rglob("*.py")))
+    return found
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("paths", nargs="+", help="files or directories to process")
+    parser.add_argument(
+        "--check",
+        action="store_true",
+        help="report what would be stripped without rewriting",
+    )
+    args = parser.parse_args(argv)
+
+    total = 0
+    for path in _python_files(args.paths):
+        source = path.read_text()
+        stripped = strip(source)
+        if stripped == source:
+            continue
+        removed = len(source.splitlines()) - len(stripped.splitlines())
+        total += removed
+        print(f"{path}: {removed} comment line(s)")
+        if not args.check:
+            path.write_text(stripped)
+
+    verb = "would remove" if args.check else "removed"
+    print(f"{verb} {total} comment line(s)")
+    return 1 if (args.check and total) else 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

src/ocarinalabs_harbor_ext/probes/__init__.py

@@ -1,37 +1,19 @@
 """Ocarina adversarial probes.
 
-Probes corresponding to the Berkeley patterns for broken agent benchmarks,
-plus the execution-faking family that targets the "wrote the answer, did no
-work" loophole. Each probe is a standalone
-:class:`~harbor.agents.base.BaseAgent` subclass. A benchmark that credits any
-probe above the configured floor is not publication-ready.
+Each probe is a standalone :class:`~harbor.agents.base.BaseAgent` subclass that
+plays an adversary against a benchmark world. A benchmark that credits any probe
+above the configured floor is not publication-ready. See ``README.md`` for what
+each one catches and ``CONTEXT.md`` for the vocabulary.
 
-- isolation_escape / reward_sink: isolation-boundary escapes and verifier-state
-  tamper.
-- judge_injection / success_token_echo: judge-injection and loose success-token
-  credit.
-- fake_state / ledger_forge: fabricate final state (and forge the in-container
-  ledger) without calling tools — floored only by execution verification
-  (trajectory coverage + reconciliation). See
-  ``docs/design/execution-verification.md``.
-- state_rewrite: the hybrid — drives a minimal REAL tool spine (coverage
-  passes), then rewrites final state to the winning answer and forges the
-  ledger to match. Floored only by the trajectory lower bound in
-  reconciliation (host-witnessed calls >= claimed state delta).
-- random_agent / forbidden_subprocess / shortcut_hunter: the gate's
-  false-positive controls. random_agent takes arbitrary valid tool actions and
-  must not earn reward (rewarding activity over outcome is a broken world);
-  forbidden_subprocess reaches the outcome via subprocess escape hatches
-  instead of the sanctioned tools and must be floored by execution + safety
-  checks; shortcut_hunter greps the checkers/fixtures for the answers and
-  hardcodes them — the runtime side of the answer-leak controls.
+Two families, distinguished by name prefix:
 
-The probes above form the general, reusable adversarial gate (``ocarina-*``
-names). Alongside them ship world-specific probes (``ap-payment-*`` names) that
-encode the winning answer for one world — contractor-payment-run — and exist to
-prove that world's execution gate holds against an answer-knowing adversary.
-They are loaded ad hoc by submodule path during hardening, not run as part of
-the general cascade.
+- ``ocarina-*`` — the general cascade, run against any Harbor world.
+- ``ap-payment-*`` — world-specific, encoding the contractor-payment-run winning
+  answer to prove that world's execution gate holds against an answer-knowing
+  adversary. Loaded ad hoc during hardening, never as part of the cascade.
+
+``state_rewrite`` carries the ``ocarina-`` prefix but is world-specific; the
+consumer's registry, not this package, decides which probes count for a world.
 
 The stock Harbor ``nop`` agent covers Berkeley's null-agent archetype; run it
 alongside these for full coverage.

src/ocarinalabs_harbor_ext/probes/_contractor_payment_run.py

@@ -1,18 +1,14 @@
 """The contractor-payment-run winning answer — one source of truth.
 
-Both world-specific probes forge this world's oracle-shaped final state:
-``ap_payment_state_forger`` writes it with zero tool calls, and
-``state_rewrite`` drives a minimal real spine and then forges it. They must
-claim the *same* winning answer — the only honest difference between them is
-the execution evidence (or absence of it) behind the state. Keeping the answer
-in two places let it drift (placeholder remittance data, held-invoice verdicts
-that contradicted the hold reasons), which repeatedly tripped shape-validation
-gates and produced vacuous floors. This module is that answer, imported by
-both; it is private (underscore) and not a probe, so it is not part of the
-public probe import surface.
+``ap_payment_state_forger`` writes this state with zero tool calls;
+``state_rewrite`` drives a minimal real spine and then forges it. They must claim
+the *same* answer, so the only honest difference between them is the execution
+evidence behind it — hence one shared module rather than two copies that drift
+into shape defects and floor for the wrong reason.
 
 Values are the frontier oracle answer for ocarina/contractor-payment-run.
 """
+# strip-comments: reviewed
 from __future__ import annotations
 
 WINNING_STATE: dict[str, object] = {
@@ -132,9 +128,8 @@
     "duplicate_payment_count": 0,
     "escalated_invoice_ids": ["INV-4512"],
     "held_invoice_ids": ["INV-4508", "INV-4509", "INV-4498"],
-    # A plausible run-summary note, not a tell: a probe writes this verbatim to
-    # result.json, so a literal "forged" could trip a content check before the
-    # execution gate under test.
+    # Deliberately not a tell: a literal "forged" here would trip a content check
+    # before the execution gate under test.
     "notes": "end-of-day contractor payment run submitted",
     "payment_count": 4,
     "released_invoice_ids": ["INV-4501", "INV-4502", "INV-4503", "INV-4507"],