Skip to content

fix: enforce max_steps on codex/claudecode/hermes harnesses - #12

Merged
Wanli-Lee merged 1 commit into
mainfrom
fix/harness-max-steps
Jul 1, 2026
Merged

fix: enforce max_steps on codex/claudecode/hermes harnesses#12
Wanli-Lee merged 1 commit into
mainfrom
fix/harness-max-steps

Conversation

@Wanli-Lee

Copy link
Copy Markdown
Contributor

Summary

Only the openclaw harness actually enforced max_steps (via a step-cap watchdog). codex/claudecode/hermes accepted the parameter but never used it — codex's "translated to model_max_turns" comment was never implemented, hermes's --effort path doesn't exist in 0.14.0, and claudecode hardcoded MAX_STEPS_REACHED=0. Those three runtimes were bounded only by the wall-clock timeout, breaking cross-harness comparability.

  • claudecode: pass the native --max-turns N flag on both the first-turn and resume invocations; detect the error_max_turns result subtype to emit a real MAX_STEPS_REACHED marker.
  • codex: background step-cap watchdog (parity with openclaw) counting "type":"function_call" events across rollout.jsonl, pkills codex at max_steps * (1 + max_refusal_retries).
  • hermes: same watchdog, counting "role":"assistant" in the incrementally-written session_*.json — the same unit openclaw counts.

All three guard the retry loop with [ -f $STEPS_CAPPED ] && break so a watchdog kill isn't misread as a transient failure and retried. A step-capped task keeps its deliverables and is still judged (parity with openclaw): hitting the cap is a real "model looping" signal, not a fake zero.

Verification (done on real VMs, not just unit tests)

  • claudecode --max-turns: verified accepted by claude 2.1.76 in-VM. It's undocumented in --help but functional — a real flag returns "Not logged in" (parsed OK) while a fake flag returns "error: unknown option".
  • hermes counting: verified in-VM that hermes writes session_*.json (NOT request_dump_*.json — an earlier assumption, disproven and corrected). The "role":"assistant" grep count matched the JSON-parsed assistant count exactly (29 == 29) on a live session.
  • codex counting: find … -exec grep "type":"function_call" cross-file accumulation + empty-dir/no-file edge cases tested.

Test plan

  • pytest tests/ — 73 passed (added tests/test_harness_max_steps.py: cap formula, --max-turns injection on both branches, function_call/assistant counting, anti-re-retry break, step markers)
  • bash -n on all three rendered runner scripts
  • claude step-decision python snippet unit-checked against real stream-json result events (error_max_turns / normal / boundary / empty / malformed)

🤖 Generated with Claude Code

Only the openclaw harness actually enforced max_steps (via a step-cap
watchdog). codex/claudecode/hermes accepted the parameter but never used it:
codex's "translated to model_max_turns" comment was never implemented, hermes's
"--effort" path doesn't exist in 0.14.0, and claudecode hardcoded
MAX_STEPS_REACHED=0. As a result those three runtimes were bounded only by the
wall-clock timeout, breaking cross-harness comparability.

- claudecode: pass the native `--max-turns N` flag on both the first-turn and
  resume invocations (verified accepted by claude 2.1.76); detect the
  error_max_turns result subtype to emit a real MAX_STEPS_REACHED marker.
- codex: add a background step-cap watchdog (parity with openclaw) that counts
  "type":"function_call" events across the rollout.jsonl files and pkills codex
  once the count reaches max_steps * (1 + max_refusal_retries).
- hermes: same watchdog, counting "role":"assistant" occurrences in the
  incrementally-written session_*.json — the same unit openclaw counts
  (verified on a real VM: hermes writes session_*.json, not request_dump_*.json).

All three guard the retry loop with `[ -f $STEPS_CAPPED ] && break` so a
watchdog kill is not misread as a transient failure and retried. A step-capped
task keeps its deliverables and is still judged (parity with openclaw): hitting
the cap is a real "model looping" signal, not a fake zero.

Co-Authored-By: Claude <noreply@anthropic.com>
@Wanli-Lee
Wanli-Lee merged commit 000a9d2 into main Jul 1, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant