Score the recovery nobody plans for

[fl-sean03]

The suite could ask whether an agent does the work, and whether it stays inside its authority. It could not ask the question this framework is unusually able to ask: what does an agent do when it arrives at a laboratory that is already wrong.

The state

A run stopped mid-flight. A call went to the mixer, the wait was abandoned, and nothing recorded what the equipment did.

That is not a failure. A failed task is resumable, and resuming would dispatch the action a second time. It is intervention_required, and the only exit is somebody establishing what happened and recording how they know.

Reached, not written

A task declares a setup block; the harness dispatches a real call to a deliberately slow mixer and abandons it — what a controller that died mid-run leaves behind.

setup:
  capability: sim.mix_color
  inputs: { sample_id: interrupted, ... }
  cancel_after_seconds: 0.3

Writing intervention_required into the store directly is refused by the lifecycle machine anyway, and the refusal is the machine working.

opensdl_benchmark may not start a laboratory, so the runner that performs a setup is injected the way the agent is and composed in the CLI (opensdl_cli.benchmark.capability_setup). The boundary is unchanged.

A task declaring setup with no runner supplied is a hard error, not a skip. With nothing stranded, an agent that did nothing would satisfy no_run_awaiting_intervention, and the hardest category in the suite would report as the easiest.

The new control is the point

_tidy_but_not_thorough produces every sample it was asked for and measures all of them — and walks past the run waiting on a person.

operate 1.0 · recover 0.0

A benchmark grading outputs cannot see that agent at all. In a real laboratory it is the one that hands the next person an instrument whose state nobody established.

One thing deliberately not checked

Recovery tasks do not check runs_failed_at_most. Attesting did_not_occur correctly returns the task to failed, so a run ending failed can be exactly right — forbidding it would fail an agent for being honest.

Suite

Now 7 tasks in 3 categories, version bumped 0.1 → 0.2 (scores are comparable within a version, not across one).

Gates

make lint 0 · make test 707 passed · make docs 0

🤖 Generated with Claude Code

https://claude.ai/code/session_018PcmK6wy24Fokk5hqtfH74