fix: enforce max_steps on codex/claudecode/hermes harnesses - #12
Merged
Conversation
Only the openclaw harness actually enforced max_steps (via a step-cap watchdog). codex/claudecode/hermes accepted the parameter but never used it: codex's "translated to model_max_turns" comment was never implemented, hermes's "--effort" path doesn't exist in 0.14.0, and claudecode hardcoded MAX_STEPS_REACHED=0. As a result those three runtimes were bounded only by the wall-clock timeout, breaking cross-harness comparability. - claudecode: pass the native `--max-turns N` flag on both the first-turn and resume invocations (verified accepted by claude 2.1.76); detect the error_max_turns result subtype to emit a real MAX_STEPS_REACHED marker. - codex: add a background step-cap watchdog (parity with openclaw) that counts "type":"function_call" events across the rollout.jsonl files and pkills codex once the count reaches max_steps * (1 + max_refusal_retries). - hermes: same watchdog, counting "role":"assistant" occurrences in the incrementally-written session_*.json — the same unit openclaw counts (verified on a real VM: hermes writes session_*.json, not request_dump_*.json). All three guard the retry loop with `[ -f $STEPS_CAPPED ] && break` so a watchdog kill is not misread as a transient failure and retried. A step-capped task keeps its deliverables and is still judged (parity with openclaw): hitting the cap is a real "model looping" signal, not a fake zero. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Only the openclaw harness actually enforced
max_steps(via a step-cap watchdog). codex/claudecode/hermes accepted the parameter but never used it — codex's "translated tomodel_max_turns" comment was never implemented, hermes's--effortpath doesn't exist in 0.14.0, and claudecode hardcodedMAX_STEPS_REACHED=0. Those three runtimes were bounded only by the wall-clock timeout, breaking cross-harness comparability.--max-turns Nflag on both the first-turn and resume invocations; detect theerror_max_turnsresult subtype to emit a realMAX_STEPS_REACHEDmarker."type":"function_call"events acrossrollout.jsonl, pkills codex atmax_steps * (1 + max_refusal_retries)."role":"assistant"in the incrementally-writtensession_*.json— the same unit openclaw counts.All three guard the retry loop with
[ -f $STEPS_CAPPED ] && breakso a watchdog kill isn't misread as a transient failure and retried. A step-capped task keeps its deliverables and is still judged (parity with openclaw): hitting the cap is a real "model looping" signal, not a fake zero.Verification (done on real VMs, not just unit tests)
--max-turns: verified accepted by claude 2.1.76 in-VM. It's undocumented in--helpbut functional — a real flag returns "Not logged in" (parsed OK) while a fake flag returns "error: unknown option".session_*.json(NOTrequest_dump_*.json— an earlier assumption, disproven and corrected). The"role":"assistant"grep count matched the JSON-parsed assistant count exactly (29 == 29) on a live session.find … -exec grep "type":"function_call"cross-file accumulation + empty-dir/no-file edge cases tested.Test plan
pytest tests/— 73 passed (addedtests/test_harness_max_steps.py: cap formula, --max-turns injection on both branches, function_call/assistant counting, anti-re-retry break, step markers)bash -non all three rendered runner scripts🤖 Generated with Claude Code