Skip to content

chore(probes): strip restated comments and document what a probe floor proves - #7

Open
faw01 wants to merge 6 commits into
mainfrom
ws/strip-slop
Open

chore(probes): strip restated comments and document what a probe floor proves#7
faw01 wants to merge 6 commits into
mainfrom
ws/strip-slop

Conversation

@faw01

@faw01 faw01 commented Jul 26, 2026

Copy link
Copy Markdown
Member

Why

Two things, from an audit of whether these probes are general or world-coupled.

The slop. 670 own-line # comments across 19 files, now 161. The prior sweep judged them "design rationale, not slop" and it was half right — the content was rationale, but the same four rules were re-argued in five modules apiece, and the tests carried 348 comment lines restating their own descriptive names. Deduplicating is the fix, not deleting.

The limitation. The audit found that a clean probe cascade proves much less than it reads as, and nothing in the repo said so. That is now documented, because it changes how a gate result should be reported.

What changed

  • Shared rationale moved to CONTEXT.md, once. Four probe implementation invariants (a write's own exit status is the check; &&-chain a script write to its run; never write the host-owned trajectory; result.json is a summary, not the state blob) were each restated in 3-5 modules. They are now stated once and cited.
  • Comment volume: 670 -> 161 own-line # lines. Removed: restatement of the adjacent line, process archaeology (review-thread cross-references, "verified argc=3"), and triplicated rationale — forbidden_subprocess argued hermeticity three times in one file. Kept: why an attack works, why a defence is ordered as it is, and the constraints someone would otherwise "fix" (the deliberately inert loopback targets, pip --isolated, the proxy clearing).
  • scripts/strip_comments.py — idempotent, tokenizer-based so a # inside a heredoc or a // inside a TypeScript payload is never touched. Preserves docstrings, inline comments, and directives (noqa, type: ignore, pragma, licence headers) structurally rather than by convention. Honours # strip-comments: keep on a block and # strip-comments: reviewed on a module, so a rescued comment survives a re-run — the failure mode that cost ocarina a licensing attribution.
  • Two test comments rescued with keep markers, both explaining a trap: that the oracle constants are transcribed independently and a DRY pass would make the assertions tautological, and that one test is the pin preventing the spine script from drifting away from the module constants.
  • Corrected a false claim in five docstrings. They stated that a probe's real tool calls are witnessed by Harbor's host-built ATIF trajectory. They are not — see below.
  • New README section, "Limits — what a floor does and does not prove."

The limitation, in short

A probe is a BaseAgent, and BaseAgent.SUPPORTS_ATIF is False; only Harbor's agent adapters rebuild /logs/agent/trajectory.json from a runtime event stream. So a probe that drives a world's real tool bodies mutates state and writes the in-container ledger, but no host-witnessed trajectory records those calls. Against a world whose criteria are conditioned on trajectory coverage, every probe scores zero before its payload is read — honest for the execution-faking family, vacuous for the rest.

Separately, the payloads are shaped for contractor-payment-run (processedCount, payment_run_submitted, AP-payment ledger verbs, a fixed set of flat output filenames), so a world grading a differently-shaped state file floors them on shape.

A clean cascade is therefore a necessary condition for publication, not a sufficient one, and should not be reported as "the world resisted every probe."

Test plan

  • uv run pytest — 116 passed, unchanged from the pre-strip baseline
  • uv run ruff check . — clean
  • Stripper is idempotent: a second run removes 0 lines; --check src tests scripts exits 0
  • All 10 probe import paths declared in ocarina's validation-gate.ts still resolve, with name() unchanged — a rename here silently breaks ocarina's gate
  • No behaviour touched: every change is a comment, a docstring, or a doc file, plus the new script

Note

Low Risk
No executable probe logic changes—only docs, comments, and a maintenance script. Risk is limited to accidentally stripping load-bearing comments on future edits if keep/reviewed markers are omitted.

Overview
This PR is documentation and comment hygiene only: probe run() behavior, probe names, and tests are unchanged.

Shared rationale moves into CONTEXT.md — four probe implementation invariants (write exit status as the check, &&-chain script writes, never write host trajectory, result.json as summary) plus trajectory reachability (probes inherit SUPPORTS_ATIF = False, so real tool calls are not host-witnessed). Probe modules drop hundreds of restated # lines and cite CONTEXT.md instead; several docstrings correct the claim that Harbor rebuilds ATIF from probe activity.

README adds a fix table per probe family, a Limits section (trajectory-gated worlds floor most probes before payload; payloads are contractor-payment-run–shaped), and developer notes for ruff and scripts/strip_comments.py.

New scripts/strip_comments.py removes own-line narration idempotently (tokenizer-safe for heredocs), preserving docstrings, directives, and # strip-comments: keep / reviewed markers. Tests assert the shipped tree is already stripped; two test comments are rescued with keep markers where DRY would make assertions tautological.

Reviewed by Cursor Bugbot for commit a901f05. Bugbot is set up for automated code reviews on this repo. Configure here.

Comment thread scripts/strip_comments.py
@faw01

faw01 commented Jul 26, 2026

Copy link
Copy Markdown
Member Author

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

1 issue from previous review remains unresolved.

Fix All in Cursor

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 1c7e214. Configure here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant