Support OpenHands MAX and complete trainer trajectories

[bingran-you]

Summary

• propagate OpenHands reasoning effort after PR Fix OpenHands Azure reasoning effort propagation #911
• pass MAX through litellm_extra_body.reasoning.effort, bypassing the OpenHands SDK typed enum
• repair duplicate tool-output fragments before Prime-SFT validation
• fail closed if any completed LLM exchange is omitted from results.jsonl
• exclude recovered HTTP/provider status=incomplete responses from training while retaining raw provenance
• require usage and exact exchange indices for every completed response
• correct nested truncation semantics and union tool definitions
• strengthen benchflow-experiment-review validator against dropped exchanges and nested truncation

Production evidence

MAX canaries proved provider support and OpenHands routing. A 50-exchange citation pass exposed silent trainer-row loss and false truncation; an integrated 25-exchange canary exposed two recovered content-filter responses. Rebuilding unchanged raw trajectories now retains every completed exchange, excludes only incomplete/error responses from training, records zero truncated steps, and passes the strengthened validator.

Validation

• exporter + trainer artifact + experiment-review validator suites: 63 passed
• agent registry suite: 27 passed
• live manifest parity against benchflow-ai/agents PR Unnatural Surge in GitHub Stars – Please Address This Issue #54: 12 passed
• ruff and diff checks pass
• fresh integrated no-skill MAX canary emitted a publishable row end-to-end at commit 324a8f40b
• integrated with-skill raw trace is under final semantic audit after regeneration at 4968f0a85

Official MAX rows remain blocked until the final integrated with-skill audit passes.

Latest automation update (2026-07-12)

• Pushed ed531a2bd to address current-line OpenHands settings feedback: typed reasoning_effort is now emitted only for OpenHands-supported values (none, low, medium, high, xhigh); max and provider-only values such as minimal stay in litellm_extra_body so the OpenHands SDK schema does not reject the settings file.
• Local validation passed: uv sync --extra dev --locked; uv run pytest -q tests/test_agent_registry.py (30 passed); uv run ruff check src/benchflow/agents/registry.py tests/test_agent_registry.py; uv run ruff format --check src/benchflow/agents/registry.py tests/test_agent_registry.py; uv run ty check src/benchflow/agents/registry.py; git diff --check.
• Current-head GitHub validation on ed531a2bd: test, pip-audit, both detect-scope jobs, and integration-light / rollout-smoke passed. manifest-parity / parity still fails against benchflow-ai/agents main because external manifests drift for opencode (install_cmd) and openhands (launch_cmd).
• Current-head rollout-smoke artifact validated healthy with benchflow-experiment-review: 1/1 rollout healthy, 227,853 tokens, 13 tool calls, reward present, 14/14 LLM responses with usage, and training_ready=1.

Remaining blocker: keep status:blocked / review:changes-requested until the external canonical benchflow-ai/agents manifests are synchronized and parity is green.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2fbb526e1b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Clamp OpenHands reasoning_effort to its accepted values

When LLM_REASONING_EFFORT is set to a BenchFlow-supported value that OpenHands does not accept in its LLM.reasoning_effort field, such as max or minimal, this writes that value directly into agent_settings.json. The pinned OpenHands SDK schema only accepts low/medium/high/xhigh/none there, so those runs will fail to load the settings (or fall back after treating them as corrupted) before the proxy can use the otherwise valid provider-side effort.

Useful? React with 👍 / 👎.

[bingran-you]

Fixed in ed531a2bd.

OpenHands now writes the typed reasoning_effort field only for the SDK-accepted enum values (none, low, medium, high, xhigh). max stays on litellm_extra_body.reasoning.effort, and provider-only values such as minimal stay in litellm_extra_body.reasoning_effort without corrupting ~/.openhands/agent_settings.json.

Validation: uv run pytest -q tests/test_agent_registry.py (30 passed), focused Ruff check/format, uv run ty check src/benchflow/agents/registry.py, and git diff --check.

[bingran-you]

Canonical manifest sync opened as benchflow-ai/agents#54. Local live parity against that branch: 12 passed. MAX Daytona canaries are running under this commit; official rows remain blocked pending their provider-response audit.

[bingran-you]

Latest production validation: integrated no-skill MAX canary is publishable end-to-end (9/9 MAX responses, exact 9-step trainer row). Integrated with-skill canary had 23 completed MAX responses plus two recovered content-filter incomplete responses; commit 4968f0a excludes only those incomplete responses and the strengthened validator passes. Final semantic audit is in progress. Canonical manifest dependency remains agents#54.

[bingran-you]

MAX experiment follow-up: OpenHands SDK 1.22.1 defaults each LLM HTTP request to 300 seconds. In the 16-row GPT-5.6-SOL MAX ramp, the eight unfinished rows repeatedly logged litellm.Timeout: Connection timed out after 300 seconds even though the BenchFlow agent and Daytona PTY limits were already 115,200 seconds.

Commit c1378af54 adds an explicit numeric LLM_TIMEOUT field to generated ~/.openhands/agent_settings.json; production MAX launches will use LLM_TIMEOUT=115200.

Validation:

• uv run pytest -q tests/test_agent_registry.py — 29 passed
• targeted Ruff and ty checks — passed
• OpenHands SDK 1.22.1 schema validation — timeout=115200 and litellm_extra_body.reasoning.effort=max both preserved
• live Daytona MAX timeout canary — running before refill resumes

[bingran-you]

MAX experiment follow-up: semantic review found a second retry-race edge in adaptive-cruise-control.

Raw exchange 59 completed after its 300s client timeout, but the agent had already resumed from the timeout error. Its nonterminal tool call was never consumed or executed. The old fallback still selected it because its request body was unique.

Commit a2e5802e8 now keeps an unconsumed completed response only when it is plain text, a finish action, or the final response with no later provider request. Unconsumed nonterminal tool calls are dropped when later requests prove the agent moved on. The deterministic review validator uses the same rule.

Evidence:

• Regenerated adaptive results.jsonl: 56 consumed responses; exchange 59 excluded
• Selected tokens 2,954,590 + excluded retry tokens 263,448 = 3,218,038 total billed
• Deterministic and independent semantic audit: publishable healthy fail
• 113 targeted exporter/validator/launcher tests passed; Ruff, ty, and full format check passed
• f77de79aa is formatting-only follow-up

[bingran-you]

Runtime refresh for the GPT-5.6-SOL MAX experiment:

• merged current main (#922 live LLM trajectory streaming)
• incorporated the already-green companion #924 OpenCode Pythonless proxy fix
• updated benchflow-ai/agents#54 to current agents main and final OpenHands typed-effort logic

Local validation on head aef06709b:

• full suite: 4,871 passed, 16 skipped, 7 deselected
• focused MAX/export/validator/streaming regressions: 115 passed
• live manifest parity against agents Unnatural Surge in GitHub Stars – Please Address This Issue #54 branch: 12/12 passed
• Ruff and ty: passed

Remote parity will remain externally blocked until agents #54 lands; the exact runtime code and local cross-repo parity are green.

This update was generated by Codex on behalf of Bingran You.

[bingran-you]

Users Simulation automation review (2026-07-12): blocked on current head aef06709bbbb7294862212789df19f7122940c17.

What passed:

• Focused PR tests passed in an isolated worktree: 99 tests over agent registry, trainer artifact emission, validator, and Prime-SFT export.
• Focused ruff and ty on changed source/test surfaces passed.
• Current-head integration-light-jobs artifact from run 29201051018 passed benchflow-experiment-review: 1/1 healthy, with result.json, results.jsonl, ACP + LLM trajectories, and training-ready output present.
• Artifact telemetry: 386,923 total tokens, 14 tool calls, 33/33 LLM responses with usage, reward present, total timing 410.4s.

Blocker:

• manifest-parity / parity is still red for openhands. The job cloned benchflow-ai/agents@9c5bb1b7572316c8a2f12d9a6281371b4e3a7065; its OpenHands launch_cmd is still missing this PR's LLM_TIMEOUT handling and MAX/reasoning passthrough. Core BenchFlow behavior looks healthy, but the external agents manifest must be synced before this can merge.

Thermo notes:

• tests/test_train_mode_artifact_emission.py grows from 959 to 1291 lines. Since the PR is blocked anyway, please split or justify this test growth before final review.
• The successful-exchange selection logic now exists in both production export and the validator; it is covered, but should be centralized if it continues to evolve.

Cost metadata caveat: token/timing telemetry is complete, but GLM per-rollout cost_usd / price_source remain null while the summary reports zero/unpriced cost. Keep this visible until cost-accounting policy is explicit.

Labels remain status:blocked + review:changes-requested.

[bingran-you]

Follow-up runtime fix in 66bc73bbc for the GPT-5.6-SOL MAX SkillsBench run:

• Reproduced a Daytona live-capture leak: the one-second callback.jsonl poll retained one exec session/wrapper shell per poll. The affected long rollout accumulated 3,622 orphaned sleeping bash processes and wedged a pending terminal result.
• Added a transient Daytona exec path that deletes only child-free polling sessions; ordinary exec semantics remain unchanged for detached services.
• Real Daytona stress probe: 100 transient commands, sessions_before=0, sessions_after=0, probe VM deleted.
• Real OpenHands + GPT-5.6-SOL MAX + Daytona canary (citation-check, with skill): healthy pass, complete ACP/LLM/results artifacts, all provider responses reasoning.effort=max.
• Validation: 4873 passed, 16 skipped, 7 deselected; relevant focused suite 99 passed, 6 skipped; repo-wide Ruff and format checks passed.

This fix is required before resuming long MAX rollouts such as paratransit-routing.

[bingran-you]

Added opt-in Daytona SSH ACP fallback in c3864ac01 after two long OpenHands/PTy runs repeatedly completed their final tool work but lost controller completion.

The default remains PTY. Set BENCHFLOW_DAYTONA_ACP_TRANSPORT=ssh to select the existing Daytona SSH transport for a targeted retry.

Validation:

• 4888 passed, 16 skipped, 7 deselected
• focused ACP/process suite: 107 passed
• repo-wide Ruff + format checks passed
• real GPT-5.6-SOL MAX + OpenHands + Daytona SSH canaries:
  • with-skill citation-check: reward 1, validator healthy, complete ACP/LLM/results, task_skills_loading=1
  • no-skill 3d-scan-calc: reward 1, validator healthy, complete ACP/LLM/results, task_skills_loading=0
• both canaries independently passed benchflow-experiment-review

This is intentionally opt-in rather than automatic failover; the canaries prove the direct-Daytona SSH happy path, not DinD or interrupted-session recovery.

[bingran-you]

Users Simulation automation review (2026-07-13): still blocked on current head c3864ac012877fa6f9ddfbf272ff80da84ae890a.

What passed:

• Shared base smoke passed: uv sync --extra dev --extra sandbox-daytona --locked, repo ruff, ty check src, CLI help/list/check surfaces, and SDK Scene.single(agent="oracle") + RolloutConfig.
• PR worktree checks passed: uv sync --extra dev --extra sandbox-daytona --locked; pytest tests/test_acp.py tests/test_acp_setup_failure_propagation.py tests/test_agent_registry.py tests/test_benchflow_experiment_review_validator.py tests/test_config_override.py tests/test_daytona_command_polling.py tests/test_live_llm_trajectory.py tests/test_sandbox_setup.py tests/test_sdk_internals.py tests/test_sdk_lockdown.py tests/test_train_mode_artifact_emission.py tests/trajectories/test_export_prime_sft.py -q -> 338 passed; focused ruff; ty check src; CLI + SDK smoke.
• Current-head integration-light artifact run 29231430374 validates healthy: 1/1 healthy, ACP + LLM trajectories present, results.jsonl training-ready, reward 1.0, 18/18 LLM exchanges with usage, 309,996 tokens, 16 tool calls, timing total 296.1s.
• The core OpenHands launch command now includes LLM_TIMEOUT handling and MAX reasoning body emission.

Blocker:

• GitHub current-head manifest-parity / parity is still failing. The job clones benchflow-ai/agents@9c5bb1b7572316c8a2f12d9a6281371b4e3a7065 and fails tests/agents/test_manifest_parity.py::test_manifest_byte_identical_to_core[openhands] because the external OpenHands manifest still has the old launch_cmd without LLM_TIMEOUT and reasoning/MAX fields.
• Local tests/agents/test_manifest_parity.py passes only when the external agents repo is absent/skipped, so the failing GitHub check is the authoritative merge gate.

Thermo-nuclear review: the broad 25-file surface is tolerable for this trajectory/proxy/sandbox change only because focused tests and the artifact validator cover the boundaries; no file crosses from under 1k to over 1k lines. The merge blocker remains manifest parity, not a local unit failure.

Keeping status:blocked + review:changes-requested. Next action: sync the OpenHands manifest in benchflow-ai/agents (or otherwise make the parity source match this core launch command), then rerun checks.

[bingran-you]

MAX long-run recovery update (2026-07-13): pushed 4d8cf1f4f after the targeted SSH batch produced 0/2 scored long runs.

Root causes addressed:

• BenchFlow pinned OpenHands CLI from 3ca17446c... and forcibly held SDK/tools at 1.22.1; the matching current CLI pair is 2df8a283... + SDK/tools 1.28.1, which includes later long-run ACP/terminal/tool-executor/event-log fixes.
• Daytona SSH access used the service's 60-minute default, with no protocol keepalive.
• Rollout artifacts did not record the selected ACP transport, so an otherwise healthy canary could not prove it used SSH.

Changes:

• pin OpenHands CLI 2df8a283..., SDK/tools 1.28.1
• request 2,880-minute SSH access and enable ServerAliveInterval=30, ServerAliveCountMax=12, TCPKeepAlive=yes
• record config.json.acp_transport from the same pure selector used by runtime launch

Validation:

• full suite: 4,891 passed, 16 skipped, 7 deselected
• repo-wide Ruff, format, and ty check src: passed
• scored MAX + OpenHands + Daytona SSH with-skill citation-check: reward 1, deterministic validator pass, independent semantic audit publishable, 10/10 provider responses reasoning.effort=max, task_skills_loading=1, and config.json.acp_transport=ssh
• canary uploaded to HF PR17 at progress/skillsbench-v1.1/gpt-5-6-sol__openhands-max/canaries/with-skill/citation-check__241d49a9
• one bounded long no-skill paratransit-routing recovery is now running; wider refill remains paused until it yields an audited verifier-scored pass/fail

Cross-repo manifest sync is pushed to benchflow-ai/agents#54; local parity against that branch is 12/12. Remote parity remains red until that PR lands. The current pip-audit failure is the newly reported Click PYSEC-2026-2132 on the unchanged lockfile, separate from this runtime change.

This update was generated by Codex on behalf of Bingran You.

[bingran-you]

Daytona cross-operator cleanup follow-up (2026-07-13): pushed a38e0fc9f.

The 105-response, 120.8-minute MAX paratransit-routing recovery did not fail from SSH/OpenHands. Daytona organization audit event e5e40c22-3ede-4b62-8f93-dcb529927877 shows another operator SDK deleted sandbox a80ce1ef-... at 2026-07-13T21:13:26.888Z together with a group of BenchFlow sandboxes. The old shared ownership marker, benchflow.managed=1, did not distinguish operators/experiments.

Fix:

• optional BENCHFLOW_DAYTONA_OWNER
• scoped managed value benchflow.managed=1:<owner>
• explicit benchflow.owner=<owner> label
• current reaper deletes only its exact scoped value
• scoped labels from other operators are treated as foreign, not orphaned

Validation:

• full suite: 4,895 passed, 16 skipped, 7 deselected
• Ruff, format, and ty check src: passed
• real Daytona label probe accepted managed=1:gpt56-sol-openhands-max; current reaper matched it; the older broad managed == 1 predicate did not
• fresh bounded paratransit recovery is running with that scoped label; wider refill remains paused pending an audited verifier score

The failed 105-response rollout was independently audited not publishable: task_skills_loading=0, all responses MAX, no reward hacking, genuine optimization progress, external deletion before verifier.

This update was generated by Codex on behalf of Bingran You.

[bingran-you]

Daily scan maintenance update (2026-07-14): pushed 77f647206 to clear the fresh pip-audit blocker on this PR.

What changed:

• litellm[proxy] pin moved from 1.89.0 to 1.91.0, allowing cryptography 48.0.1.
• lock refreshed from click 8.3.1 to 8.4.2 and cryptography 46.0.7 to 48.0.1.
• the removed-short-option CLI regression now checks the semantic Click error (No such option + rejected alias) so it remains stable across Click 8.4 punctuation changes.

Local validation on the PR branch:

• uv run --extra dev python -m pytest tests/test_trace_import_cli.py -q -> 6 passed
• uv run --extra dev python -m pytest tests/test_litellm_config.py tests/test_litellm_hardening.py tests/test_litellm_logging.py tests/test_litellm_smoke.py tests/test_usage_litellm.py tests/test_agent_registry.py tests/test_bedrock_thinking.py -q -> 156 passed, 2 skipped
• uv export --locked --extra dev --format requirements-txt --no-hashes --output-file /tmp/req-audit-921 && uv run --with pip-audit pip-audit -r /tmp/req-audit-921 --no-deps --disable-pip -> no known vulnerabilities found
• uv run ruff check . -> passed
• uv run ruff format --check pyproject.toml tests/test_trace_import_cli.py -> passed
• uv run ty check src/ -> passed
• git diff --check -> passed

Remaining known blocker is still manifest parity against external benchflow-ai/agents OpenHands manifest drift (install_cmd / launch_cmd). Keeping status:blocked / review:changes-requested until that external source is synchronized and GitHub checks are green.

[bingran-you]

Users Simulation automation review (2026-07-14): still blocked on current head 77f6472060ebdef31c53aafd74606e558f56d20b.

What passed:

• Shared base gates passed in the main checkout: uv sync --extra dev --extra sandbox-daytona --locked, repo ruff check ., repo ty check src, CLI help/list/check surfaces, and SDK Scene.single(agent="oracle") + RolloutConfig smoke.
• Current-head isolated worktree gates passed: uv sync --extra dev --extra sandbox-daytona --locked; focused OpenHands/MAX/trajectory/sandbox suite -> 468 passed; focused ruff; ty check src; CLI + SDK smoke.
• Current-head integration-light artifact run 29318912142 validated healthy with benchflow-experiment-review: ACP trajectory, LLM trajectory, one training-ready results.jsonl row, reward present, 20/20 LLM responses with usage, 363,952 tokens, 19 tool calls, timing total 375.3s.

Blocking finding:

• manifest-parity / parity still fails. The external agents manifest for openhands is not byte-identical to core: it still has the old OpenHands CLI install pin (CLI 3ca174..., SDK/tools 1.22.1) and old launch command, while core now uses CLI 2df8a..., SDK/tools 1.28.1, LLM_TIMEOUT, and MAX/reasoning launch fields. This is a real cross-repo source-of-truth drift, so the PR should stay status:blocked / review:changes-requested until benchflow-ai/agents is synced.

Thermo-nuclear review:

• No new local behavior blocker from the focused checks, but the diff remains broad and extends already-large files/tests. The manifest parity drift is the merge blocker; keep future cleanup focused on reconciling the canonical agent config instead of adding another local shim.

Cost caveat:

• The GLM user-endpoint artifact has complete token/timing telemetry, but result.json.agent_result.cost_usd and price_source are still null.