chore(probes): strip restated comments and document what a probe floor proves - #7
Open
faw01 wants to merge 6 commits into
Open
chore(probes): strip restated comments and document what a probe floor proves#7faw01 wants to merge 6 commits into
faw01 wants to merge 6 commits into
Conversation
… test the stripper
Member
Author
|
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
1 issue from previous review remains unresolved.
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 1c7e214. Configure here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Why
Two things, from an audit of whether these probes are general or world-coupled.
The slop. 670 own-line
#comments across 19 files, now 161. The prior sweep judged them "design rationale, not slop" and it was half right — the content was rationale, but the same four rules were re-argued in five modules apiece, and the tests carried 348 comment lines restating their own descriptive names. Deduplicating is the fix, not deleting.The limitation. The audit found that a clean probe cascade proves much less than it reads as, and nothing in the repo said so. That is now documented, because it changes how a gate result should be reported.
What changed
CONTEXT.md, once. Four probe implementation invariants (a write's own exit status is the check;&&-chain a script write to its run; never write the host-owned trajectory;result.jsonis a summary, not the state blob) were each restated in 3-5 modules. They are now stated once and cited.#lines. Removed: restatement of the adjacent line, process archaeology (review-thread cross-references, "verified argc=3"), and triplicated rationale —forbidden_subprocessargued hermeticity three times in one file. Kept: why an attack works, why a defence is ordered as it is, and the constraints someone would otherwise "fix" (the deliberately inert loopback targets,pip --isolated, the proxy clearing).scripts/strip_comments.py— idempotent, tokenizer-based so a#inside a heredoc or a//inside a TypeScript payload is never touched. Preserves docstrings, inline comments, and directives (noqa,type: ignore,pragma, licence headers) structurally rather than by convention. Honours# strip-comments: keepon a block and# strip-comments: reviewedon a module, so a rescued comment survives a re-run — the failure mode that cost ocarina a licensing attribution.The limitation, in short
A probe is a
BaseAgent, andBaseAgent.SUPPORTS_ATIFisFalse; only Harbor's agent adapters rebuild/logs/agent/trajectory.jsonfrom a runtime event stream. So a probe that drives a world's real tool bodies mutates state and writes the in-container ledger, but no host-witnessed trajectory records those calls. Against a world whose criteria are conditioned on trajectory coverage, every probe scores zero before its payload is read — honest for the execution-faking family, vacuous for the rest.Separately, the payloads are shaped for contractor-payment-run (
processedCount,payment_run_submitted, AP-payment ledger verbs, a fixed set of flat output filenames), so a world grading a differently-shaped state file floors them on shape.A clean cascade is therefore a necessary condition for publication, not a sufficient one, and should not be reported as "the world resisted every probe."
Test plan
uv run pytest— 116 passed, unchanged from the pre-strip baselineuv run ruff check .— clean--check src tests scriptsexits 0validation-gate.tsstill resolve, withname()unchanged — a rename here silently breaks ocarina's gateNote
Low Risk
No executable probe logic changes—only docs, comments, and a maintenance script. Risk is limited to accidentally stripping load-bearing comments on future edits if keep/reviewed markers are omitted.
Overview
This PR is documentation and comment hygiene only: probe
run()behavior, probe names, and tests are unchanged.Shared rationale moves into
CONTEXT.md— four probe implementation invariants (write exit status as the check,&&-chain script writes, never write host trajectory,result.jsonas summary) plus trajectory reachability (probes inheritSUPPORTS_ATIF = False, so real tool calls are not host-witnessed). Probe modules drop hundreds of restated#lines and citeCONTEXT.mdinstead; several docstrings correct the claim that Harbor rebuilds ATIF from probe activity.README adds a fix table per probe family, a Limits section (trajectory-gated worlds floor most probes before payload; payloads are contractor-payment-run–shaped), and developer notes for
ruffandscripts/strip_comments.py.New
scripts/strip_comments.pyremoves own-line narration idempotently (tokenizer-safe for heredocs), preserving docstrings, directives, and# strip-comments: keep/reviewedmarkers. Tests assert the shipped tree is already stripped; two test comments are rescued with keep markers where DRY would make assertions tautological.Reviewed by Cursor Bugbot for commit a901f05. Bugbot is set up for automated code reviews on this repo. Configure here.