Merge main into the 0.7 line + land OSWorld vendored evaluator#827
Merge main into the 0.7 line + land OSWorld vendored evaluator#827xdotli wants to merge 55 commits into
Conversation
Decouple two I_eval axes from the task, the way --agent/--model/--sandbox
already are ("decouple the task from the harness"):
- S (environment): load_manifest resolves a `name@version` spec via
$BENCHFLOW_ENV_REGISTRY (content-addressed) in addition to a file path,
so --environment-manifest binds a registry environment per run. Mirrors
resolve_dataset.
- C (config): --config-override deep-merges a config patch (inline
JSON/YAML/TOML or @file) into each task's resolved TaskConfig. Parsed
once at plan time and threaded as typed data through EvalCreateRequest
-> EvalPlan -> EvaluationConfig -> RolloutConfig (incl. the sharded
worker payload), applied at the rollout layer (not the Task ctor),
re-validated, allowlist-scoped away from scoring fields, and persisted
to config.json by content hash for replay.
Adds _utils/{env_registry,config_override,content_address}.py and unit
tests for both axes.
Resolve conflicts in the eval pipeline where main's --loop-strategy and this branch's --config-override / --environment both thread a new field through the same sites (EvalPlan, EvaluationConfig, RolloutConfig, the sharded worker payload, _write_config). Kept both — they are independent axes. Verified: 36 unit tests, ruff clean, and an oracle run with --config-override still scores 1.0 with provenance persisted.
Make PR #790's env-axis usable on real env0, reproducibly: - benchmarks/_environments/{env0@prod,env0@outage}.toml — a git-tracked env registry so `--environment-manifest env0@prod` resolves from the repo (BENCHFLOW_ENV_REGISTRY=benchmarks/_environments) instead of a /tmp dir. env0@prod = 7 services (auth/gmail/slack/gcal/gdoc/gdrive/stripe); env0@outage = the same minus gmail+slack (tool-outage perturbation). - _utils/build_context_stage.py — adapter that stages tasks authored with a repo-root build context (smolclaws/env0: `COPY tasks/<name>/data`) into benchflow's environment/ context so they run via --tasks-dir without hand-staging: `python -m benchflow._utils.build_context_stage <src> <out>`. Dogfooded: 59/60 env0 oracles score 1.0 on Daytona via env0@prod.
…e agent-mixing (#789) * fix(cli): harden error handling across the CLI surface Surface clean errors (not tracebacks) and correct exit codes for malformed-but-plausible input across the command surface, found by a full command/flag audit: - hub env: suppress prime's tty upgrade banner (PRIME_DISABLE_VERSION_CHECK) - sandbox list/cleanup: degrade cleanly (exit 0) when the Daytona SDK is absent - eval create: fix resume progress denominator (was 11/1 1100%) - agent create / adopt init: guard create_benchmark mkdir (NotADirectoryError) - agent verify / adopt verify: guard --issue-out write (FileNotFoundError) - tasks digest: skip unreadable subdirs in the scan (PermissionError) - tasks generate --from-file: reject non-file inputs (IsADirectoryError) - tasks generate --outcome: reject values outside success/failure/unknown - continue-batch: fail fast on a nonexistent/non-directory ROOT - task loader: name task.md/task.toml, not the legacy instruction.md fallback - eval view: fail fast on an empty dir instead of writing a blank trajectory.html - print_error: emoji=False so user input with :tokens: echoes verbatim Adds regression tests for each. * fix(eval): refuse resume on agent mismatch instead of blending scores A jobs_dir holds one (agent, model) run. Resuming it with a different agent silently folded the prior agent's cached rollouts into the new run's summary, publishing a blended Score: X/N that belonged to neither (the symptom that looked like 'scores appear without Docker running'). _warn_on_resume_mismatch -> _check_resume_mismatch: an agent mismatch now raises ResumeMismatchError (surfaced cleanly by the eval-create handler, which already catches ValueError) directing the user to a fresh --jobs-dir, which also preserves the existing results. A loop_strategy-only mismatch (same agent) still just warns. * fix(cli): second-pass error-handling hardening from deep fix-hunt Ten more defects found by an adversarially-verified hunt over the merged tree (5 P0 tracebacks, 2 P1, 1 P2, 2 P3): - tasks normalize --output <dir>: IsADirectoryError -> clean error (OSError catch) - skills eval: evals.json being a directory -> clean 'No evals' (is_file guard) - eval adopt --verify: parity_experiment.json being a directory -> clean (is_file) - sandbox create: task.md/task.toml being a directory -> clean (OSError catch) - tasks generate --output <file>: NotADirectoryError -> clean pre-check - eval adopt --verify --roundtrip-task: malformed task.toml TOMLDecodeError -> clean - eval create --config: git clone failure -> clean message (catch CalledProcessError) - tasks generate: 0 results now exits 1 instead of green 'Generated 0 tasks' - eval list <explicit nonexistent>: exits 1 (default jobs dir still exit 0) - sandbox create: clean up the stray jobs/environment/ dir on env-create failure Also updates the resume agent-mismatch test for the refuse behavior (follow-up to the ResumeMismatchError change). Regression tests added for the tractable cases. ruff clean. * style: ruff format test_cli_edge_case_hardening.py Fixes the failing `test` CI job (ruff format --check). Pure auto-formatting (line-wrapping of multi-arg invoke calls); no logic change. 52 tests in the file still pass. * fix(cli): address PR #789 review + fix CI format check - sandbox._daytona_sdk_available: apply the anyio compat shim before importing daytona, so a real install that needs the shim is not reported absent (codex P1) - task.py: instruction_path.is_file() not exists(), so a directory named instruction.md gives a clean error, not IsADirectoryError (greptile P2) - eval list: distinguish an omitted arg (None -> benign 'No jobs yet') from an explicit nonexistent path (error) via a None sentinel default (codex/greptile P2) - tasks generate: neutral 0-results message covering no-input and all-filtered (greptile P2) - unreadable-dir digest test: skip under uid 0 (chmod 000 is a no-op for root) (codex P2) - name PR #789 in the new regression-test docstrings per AGENTS.md (codex P2) - ruff format (fixes the CI format-check failure) --------- Co-authored-by: Bingran You <bingran@benchflow.ai>
…ntellect) (#791) * feat(hub): bench hub env list browses Harbor too, not just PrimeIntellect hub env list was hard-gated to --provider primeintellect. Make it a clean provider dispatch: - primeintellect — hosted environments (unchanged) - harbor — the benchmark registry, listing its 80 datasets (name / version / task-count / description) --search/--limit/--json work for both hubs; --owner is primeintellect-only and is rejected cleanly for harbor; an unknown --provider gives a clean error (supported: primeintellect, harbor). show/inspect remain primeintellect for now. * refactor(hub): flatten 'hub env list' to 'hub list' (the env nesting was redundant) 'hub' already means environment hubs, so the 'env' level was duplicative. Promote the browsing verbs to 'bench hub list|show|inspect' (alongside 'bench hub check'). 'bench hub env *' stays as a hidden back-compat alias of the flattened verbs, so existing scripts keep working. Also updates the 'environment' deprecation targets, docs/reference/cli.md, and the affected tests. * Update src/benchflow/cli/_hosted_env.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
The task / run / job configs are YAML, so a TOML-only env manifest is
inconsistent. load_manifest + the env registry now resolve and parse both
`.toml` and `.yaml`/`.yml`: EnvironmentManifest.model_validate_path picks the
parser by extension, and resolve_environment finds `<name>@<version>.{toml,
yaml,yml}`. YAML is canonical for new manifests; TOML stays supported for
back-compat (the committed env0@prod.toml + Harbor heritage keep working).
Additive — no breaking change.
Make the S/C axes bindable inline, the way Han sketched on the whiteboard:
- --state: S-axis environment binding, decoupled from the task. Inline JSON
with a tool subset, e.g. {"name":"env0","tools":["gmail","gcal"]} — resolves
env0 from the registry and starts ONLY the listed services — OR a
name@version spec, OR a manifest path. Wins over --environment-manifest
(kept for back-compat). New resolve_state() in env_registry.
- --config / --config-override: the C-axis overlay (inline JSON / @file),
unchanged plumbing, now reachable under --config.
- The old job-config --config (file) is renamed to --run-config, which frees
--config for the overlay — matching the parallel eval-run branch's direction.
So `bench eval create --state '{"name":"env0","tools":["gmail","gcal"]}'
--config '{"agent":{"timeout_sec":30}}'` resolves the right env (subset) and
overlay. The `eval run` verb + `--task <dir>` + `--run-config <file>` are owned
by the parallel branch; these axes attach to the shared request/plan.
…cated alias) (#795) * feat(cli): rename `bench eval create` to `bench eval run` The verb now matches what the command does — it runs an evaluation (single task or batch). `bench eval create` stays as a deprecated alias that prints a Click deprecation notice on use, so existing scripts, YAML configs, and downstream repos (e.g. benchflow-ai/skillsbench) keep working unchanged. - main.py: `@eval_app.command("run")` / `def eval_run`; register `eval_app.command("create", deprecated=True)(eval_run)` as the alias. - Rename the `eval_create` symbol to `eval_run` (incl. `__all__` and callers). - Drift test repointed to `run` (bidirectional flag/doc check) + a new guard asserting `create` is a deprecated alias sharing `run`'s callback. - Sweep `eval create` -> `eval run` across docs, src docstrings/strings, the CI integration workflow, benchmarks/ + .agents/ docs, and CHANGELOG. Existing test argv invocations that still pass `["eval", "create"]` are kept intentionally: they exercise the deprecated alias end-to-end (the back-compat guarantee), and pass via the alias. * fix(cli): address review — keep eval_create import alias + fix stale docstring - Restore `eval_create` as a deprecated import alias of `eval_run` (back in __all__) so `from benchflow.cli.main import eval_create` keeps working, matching the CLI's deprecated `create` alias (Codex review). - eval_plan.py: fix a stale `eval_create` docstring reference → `eval_run` (Greptile review).
Public release cut from main. Headlines since 0.6.2: - `bench eval create` renamed to `bench eval run` (deprecated alias kept) (#795) - `task.md` is now the sole task authoring format - CLI surface flattening + error-handling hardening (#787, #789, #791) - MiMo Code (mimo) ACP agent (#679) - numerous fixes Bumps pyproject + CITATION.cff (version + date-released) + uv.lock; promotes the CHANGELOG [Unreleased] section to 0.6.3.
chore(release): 0.6.3
Rollout.disconnect() runs `pkill -f {pattern}` to kill the agent between
scenes while keeping the Environment plane alive. The pattern was derived
from the first launch token, so a `python <shim>` launcher (deepagents,
harvey-lab) produced `(^|[ /])python( |$)` — and Environment-plane mock
services are console scripts whose argv is `/usr/bin/python /usr/local/bin/
<svc> ...` (their shebang interpreter). The teardown then SIGTERMed the
service too; the verifier later read its live state over HTTP, found it
dead (`could not reach .../_admin/state`), and scored 0.0 on an otherwise
correct rollout — breaking every stateful Daytona task with a live-service
verifier.
Resolve the agent token robustly: reduce to the last shell-command segment
(so `export ... && <agent>` launches like openhands key on the agent, not
the `export` builtin), then walk it skipping FOO=bar env-assignments,
generic interpreters, package-runner subcommands (`uv run <agent>`), and
flags. Return None rather than fire a sandbox-wide pkill when nothing
specific is found.
Verified end-to-end on Daytona: archive-amazon-shipping went 0.0 -> 1.0.
Tests assert every real agent's pattern matches its own argv and none match
a co-resident python service argv.
OpenCode and its MiMo fork (acp_model_format="provider/model") validate model ids against the models.dev catalog and reject the synthetic openai/benchflow-<alias> that BenchFlow's LiteLLM proxy serves the model under, so they never issue a request through the proxy and produce no trajectory/llm_trajectory.jsonl -- which benchflow-experiment-review requires and which every other proxy-routed agent (openhands, pi-acp, openclaw) emits. Install an <agent>-proxy wrapper under /opt/benchflow/bin that, in proxy mode (usage_tracking != off), registers the gateway alias under the agent's openai provider (with the gateway baseURL) before exec'ing the isolated binary, bypassing the catalog check so the agent routes through the proxy. No-op outside proxy mode; idempotent; launch entrypoint stays under /opt/benchflow/bin so the isolated-node invariant is preserved.
pi-acp (acp_model_format="registered-provider/model") was sent the bare gateway alias `benchflow-<...>` via session/set_model, but its launcher registers that model under the `litellm` provider in models.json — so Pi could not resolve the bare id and its model calls bypassed the LiteLLM usage proxy, completing the task (tool calls happen) but capturing no trajectory/llm_trajectory.jsonl (usage_source=unavailable). Send the registered-provider-qualified `litellm/<alias>` so Pi hits the gateway route. TDD: tests/test_pi_acp_proxy_routing.py.
…oxy wrapper opencode-ai 1.17.x ships its bin as a native ELF (bin/opencode.exe), but the proxy wrapper exec'd it through the isolated node launcher (`node <bin>`), which parsed the ELF as JavaScript and crashed at startup with a SyntaxError before the agent ran. Detect non-shebang bins and exec them directly (node-shim bins still go through the node launcher). Lets opencode actually start so its registered gateway model routes through the proxy.
…proxy
OpenCode and its MiMo fork validate model ids against the models.dev catalog and
hard-code the OpenAI *Responses API* for the built-in `openai` provider id
(`getModel` calls `provider.responses(id)`). BenchFlow's LiteLLM gateway and the
openai-completions upstreams it fronts (e.g. DeepSeek) only serve chat
completions, so the gateway alias never produced a usable request:
- registered bare under `openai` -> Responses API -> 404/500, agent idles
- `openai.npm` overridden to @ai-sdk/openai-compatible -> OpenCode still calls
`.responses()` -> 'provider.responses is not a function' -> zero requests
Register the gateway alias under a dedicated provider id (`benchflow`) using
@ai-sdk/openai-compatible, which OpenCode routes through the chat-completions
path, and send set_model `benchflow/<alias>` for provider/model agents in proxy
mode. Forward the gateway master key as options.apiKey and pin small_model to the
alias so the title/summary helper stops hitting the hard-coded gpt-5-nano the
gateway cannot serve.
Validated on docker (citation-check, deepseek-v4-flash):
- opencode: reward 1.0, 11 tool calls, 37/37 LLM reqs with provider_response usage
- mimo: reward 1.0, 14 tool calls, 9/9 LLM reqs with provider_response usage
all via /v1/chat/completions, so trajectory/llm_trajectory.jsonl now carries raw
request bodies + per-request tokens + timestamps for experiment-review.
Drop a placeholderless f-string (F541) and apply ruff format to the touched test files; refresh the now-stale module docstring.
…vider test_opencode/mimo_litellm_alias_formats_* asserted the superseded openai/<alias> set_model route; opencode/mimo proxy mode now routes via the dedicated OPENCODE_PROXY_PROVIDER_ID (chat completions). Assert that and that it is not the built-in openai/ id (whose Responses-API hard-coding the gateway cannot serve).
Address Greptile review on the dedicated-provider proxy wrapper: - P1 fail-loud: drop `|| true`; in proxy mode a registration failure (malformed pre-existing config, unwritable path) now aborts the launch with an explanatory stderr line instead of silently launching an agent whose gateway alias is missing from its config (which would hit ProviderModelNotFoundError with no reason). - P2 agent-home: resolve the config file against $BENCHFLOW_AGENT_HOME (falling back to ~) — the same home `disallow_web_tools_setup_cmd`/`credential_files` write to — so every writer targets one file even if the sandbox home differs from $HOME. Call sites now pass a relative config path. - Refresh the stale mimo comment that still described the old openai/<alias> routing this PR replaced. Tests cover fail-loud (exit 1, no `|| true`) and $BENCHFLOW_AGENT_HOME resolution. Re-validated opencode on docker: reward 1.0, 29 tool calls, 15/15 LLM reqs with provider_response usage via /v1/chat/completions.
…xy-tracking fix(agents): route OpenCode-family + pi-acp agents through the LLM usage proxy
Register `continue` / `continue-batch` on the `eval` group so the canonical spelling is `bench eval continue`, discoverable in `bench eval --help` next to `run` and `adopt`. Keep the original top-level `bench continue` / `bench continue-batch` as hidden, deprecated aliases (they print a deprecation notice) so existing scripts and docs keep working. Update docs (continue-runs, CLI reference) and CHANGELOG, and add a regression test pinning the canonical eval location plus the alias.
- Command-agnostic error prefix (`benchflow:`) in continue/continue-batch so deprecated-alias users (`bench continue`) don't see `bench eval continue:` in errors. - Parametrize the continue exit-code + concurrency-floor regression tests over BOTH spellings (canonical `eval continue` and the deprecated alias) so a broken eval_app registration can't hide behind the alias path. - Add regression-test docstrings naming the guarded behavior + PR #800 per AGENTS.md.
feat(eval): bind the environment (S) and config (C) axes at the CLI
feat(cli): move `continue` under `bench eval continue` (keep deprecated top-level alias)
Score OSWorld tasks with the same logic as upstream, toward the whole benchmark: - Vendor xlang-ai/OSWorld's evaluator suite verbatim into adapters/_osworld_vendor/ (15 metric + 12 getter modules, Apache-2.0) and resolve metrics from it via a lazy, dependency-resilient registry (adapters/osworld_vendor.py): local hand-ports first (proven reward-identical to upstream), then upstream for the rest. Metric coverage 6 -> 338/369 funcs. - Extend the native evaluator: file getters (vm_file/cloud_file/cache_file/local/ content_from_vm_file), evaluator options, and the sleep postconfig step; add faithful ports of check_direct_json_object/check_list/compare_text_file. - Guard docs.py easyocr/skimage so docx metrics work without the torch-class deps. - Declare 'osworld' (medium) + 'osworld-cv' (cv2/easyocr/librosa) extras; exclude the vendored tree from ruff/ty. - Bump anyio>=4.9 so the daytona SDK imports (httpx-ws needs AsyncContextManagerMixin; anyio 4.8 broke it). Tests: 37 osworld unit tests green; metric parity harness benchflow==upstream 56/56; ruff clean.
Wire the vendored OSWorld getters into the evaluator via an in-guest controller shim that routes execute_python_command (76 uses) + get_file (20) through the verifier's run_command channel, so OSWorld's own get_<type> code runs in whatever sandbox the verifier runs in. osworld_eval._get_state falls back to the vendored getter for any type the native path doesn't cover; ShimEnv stands in for DesktopEnv. Live-state getters (accessibility_tree, chrome CDP) raise — they need the real controller server (route A). Scorable coverage (metric + getter resolvable): 248 -> 340/369 (92%). Tests: 45 osworld unit tests green; ruff clean.
…ute correctly (#805) A bare model id like `deepseek-v4-pro` (no `provider/` prefix) never resolved to a provider in resolve_provider_env / _normalize_openhands_model, because both only called find_provider(), which matches an explicit prefix. So BENCHFLOW_ PROVIDER_NAME/BASE_URL/API_KEY were never emitted and provider-driven harnesses misrouted: - openhands: litellm got LLM_MODEL=deepseek-v4-pro with no provider -> `LLM Provider NOT provided`, 0 tool calls. - openclaw: its shim defaulted the provider to anthropic -> `FailoverError: Unknown model: anthropic/deepseek-v4-pro`, 0 tool calls. Route bare family ids through find_provider_for_bare_model() (already used by acp/runtime.py) in both sites. To avoid regressing the harnesses that DO work today (deepagents/pi-acp/opencode/mimo, which rely on their shims' hardcoded deepseek default), give the deepseek provider a default base_url via a new ProviderConfig.url_param_defaults field — so resolve_base_url no longer requires DEEPSEEK_BASE_URL, while an explicit DEEPSEEK_BASE_URL still overrides. The default (https://api.deepseek.com/v1) matches the shim default. Verified non-regressing: every relevant agent declares api_protocol='' so the protocol-compat check passes; prefixed ids and non-deepseek models are unchanged. Adds regression tests for the bare-deepseek openhands/openclaw paths and the DEEPSEEK_BASE_URL override. harvey-lab-harness needs a separate shim change (its _create_adapter only dispatches claude/gpt/o*/gemini) and is handled separately.
* fix(eval): expose context-root on eval run * fix(eval): validate context root during planning
Mirror BENCHFLOW_PROVIDER_MODELS metadata (maxTokens/contextWindow) onto the LiteLLM proxy alias that pi-acp sees, so model limits survive the mandatory proxy routing introduced in #820. Rebased onto main after #820 restructured litellm_runtime into _apply/_wire_litellm_agent_env. - Add _provider_models_for_proxy_alias + _provider_model_id helpers. - Clone the source model entry under route.model_alias in the pi-acp branch of _wire_litellm_agent_env. - Regression test asserts vLLM/Qwen alias carries maxTokens/contextWindow. Local: tests/test_litellm_runtime.py 23 passed; ruff + ty clean.
* fix(integration): avoid file-editor judge false positives * fix(integration): parse file-editor titles as json
… (#822) * fix(eval): reject .git and file --source-path with a clean error Closes the remaining half of #548. The --tasks-dir <file> crash was already fixed; this handles the source-resolver path. _resolve_repo_path validated existence and root-escape but not that the target is a usable task directory, so: - --source-path .git resolved to the clone's VCS metadata (a real dir inside the root) and produced zero task hashes downstream. - a regular-file --source-path (e.g. README.md) passed existence and resolved to a non-directory that failed later. Reject both at the resolver boundary with a clear ValueError. The eval-create CLI already catches resolver ValueError/OSError (cli/main.py) and surfaces a clean 'Could not resolve --source-repo ...' message + exit 1, so no traceback reaches the user. Tests: two regression tests in test_task_download.py for the .git and file cases. Local: test_task_download.py 23 passed; source-resolution suite (provenance/dataset_registry/cli_edge_case/cli_arg_validation) 145 passed; ruff + ruff format + ty clean. * fix(eval): harden source path git metadata guard
Brings main's 49 commits (L0–L3 integration gates, MLE-bench adapter, adapter skill, eval create→run rename, environment→sandbox/hub deprecation, eval fixes) onto the 0.7 universal-environment line while keeping the 0.7-only OSWorld/CUA work (use_computer_cookbook, osworld_*, _osworld_vendor, computer_use, browser_use, stagehand, iosworld). Conflict resolutions: - agents/registry.py: keep both (_BENCHFLOW_PYTHONPATH + OPENCODE_PROXY_PROVIDER_ID). - cli/main.py: keep both — 0.7 eval-payload helpers (still used by the eval command) + main's `eval create`→`run` deprecated alias. The eval command itself is main's eval_run. - cli/environment.py: adopt main's deprecation (environment → sandbox/hub), resolving the long-standing env-CLI revival-vs-deprecation decision. - uv.lock: regenerated against the merged pyproject (keeps anyio>=4.9). Test alignment: - Removed tests/test_cli_environment.py (covered the deprecated rich env CLI; main moved coverage to test_cli_hub_env.py). - Ported the 0.7 daytona snapshot-reaper (bf-snap-*) into main's canonical `sandbox cleanup` so the feature survives the env→sandbox move; the deprecated `environment cleanup` alias inherits it. - Made the oracle providers-phrase assertion wrap-robust (the phrase grew to docker, daytona, modal, cua, cua-cloud and now wraps in dense help tables). Full suite: 4882 passed, 15 skipped. The 3 remaining test_eval_inbound_adapter foreign_adapter failures are pre-existing (fail identically pre-merge; need the sandbox-cua extra in the test env), not introduced here.
|
Too many files changed for review. ( Bypass the limit by tagging |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4900567c93
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # pdf/json/text metrics). The desktop + OSWorld apps come from the desktop | ||
| # backend (Daytona computer-use, or the OSWorld VM), not this base image. | ||
| osworld_deps = ( | ||
| "pandas openpyxl python-docx python-pptx pdfplumber PyPDF2 pymupdf " |
There was a problem hiding this comment.
For OSWorld tasks that use the vendored PDF metrics such as check_pdf_pages, osworld_vendor imports _osworld_vendor/desktop_env/evaluators/metrics/pdf.py, which does from pypdf import PdfReader. This generated verifier image installs PyPDF2 but not the separate pypdf distribution, so those metrics raise VendoredMetricUnavailable and the verifier records 0/error instead of scoring PDFs. Please install pypdf here and in the osworld extra rather than relying on PyPDF2 alone.
Useful? React with 👍 / 👎.
bingran-you
added
enhancement
|
Automation triage (2026-06-24): not ready to merge. Blockers I found in the live PR state:
Recommended fix before re-review: add I set |
|
Automation user-simulation + thermo-nuclear review (2026-06-26): still not ready to merge. A separate subagent reviewed the 0.7 merge/OSWorld stack, and I verified its highest-risk findings on the current head. The targeted local slices are green: But there are still merge blockers:
Concrete user-simulation commands that should pass before re-review: uv run bench eval run --help
uv run bench hub list --provider harbor --json
uv run python benchmarks/mle-bench/main.py --help
uv run pytest tests/test_env_registry.py tests/test_config_override.py tests/test_eval_sharding.py -q
uv run pytest tests/test_osworld_eval.py tests/test_osworld_metrics.py tests/test_osworld_getters.py -q
uv run pytest tests/test_integration_matrix.py tests/test_build_review_pack.py tests/test_codex_review.py tests/test_rubric_checks.py -qPlease fix the env-registry semantics, split the planner, add the missing |
|
Users Simulation review (2026-06-26): blocked. The release-line merge is mostly coherent and the new CI/review scripts have useful direct tests, but the artifact-health contract is inconsistent with a supported BenchFlow auth path. Blocker: This matches the shared live canary attempted on current Checks recorded by the PR-specific simulation: Please align the validator with the same subscription-auth split used by |
|
Users Simulation automation review (2026-06-27T12:21Z): blocked. Blockers:
Commands/evidence:
Please make getter failures hard evaluator errors, fix the fallback-import typing, format the touched files, and rerun the focused pytest plus artifact-validator smoke. |
|
Users Simulation automation follow-up (2026-06-27T23:15Z): still blocked. I rechecked current head New blocker evidence:
Passing coverage from the same isolated worktree: Artifact-health note: the Verdict: keep |
|
Users Simulation review (2026-07-03): blocked. CLI smoke and a large targeted unit slice passed (
No live benchmark trajectory was generated, so ACP/LLM artifact health and Prime-RL readiness remain unverified. |
|
Users Simulation automation review (2026-07-04): still blocked on unchanged head Fresh cheap checks refined, but did not clear, the blocker set:
Shared sweep canary produced healthy ACP+LLM/results artifacts on the current code path, but this release-line OSWorld PR remains blocked on reviewability and dependency-surface grounds. Keep |
|
Users Simulation automation review (2026-07-05): blocked on head Fresh isolated validation found current blockers:
The base BenchFlow CLI/SDK smoke and 178 focused tests passed, but the install/static/structure blockers above keep this unmergeable. Labels: keep |
|
Users Simulation automation review (2026-07-06): still blocked on head Blockers reproduced:
Keep |
|
Users Simulation automation review (2026-07-07): still blocked on head Fresh PR-scoped validation reproduced current blockers:
Keep |
|
Users Simulation automation review (2026-07-08): blocked on head Fresh PR-head vs control repro:
Passing evidence:
Thermo-nuclear review: the vendored OSWorld integration still has too much runtime risk for a release-line PR. The control proving verifier reach is the key regression signal: fix the verifier mount path first, then rerun the OSWorld smoke and artifact validator. Labels: kept |
|
Users Simulation automation review (2026-07-10): blocked / process-gated on head Local smoke in an isolated worktree was good:
Merge blockers remain:
Keeping |
|
Users Simulation automation review (2026-07-11): blocked on head Passing evidence:
Blockers remain:
Thermo review: blocked. This is a very large release-line adoption PR, and BenchFlow-owned 1k+ additions such as Please fix the static gates, split or justify the new monolithic CI/rubric modules, align OSWorld dependency declarations with the vendored imports, and provide release-line rollout/review evidence before re-review. |
|
Users Simulation automation review (2026-07-12): still blocked on head Current-run blocker:
Existing blockers still apply:
This remains |
|
Codex cleanup update (2026-07-15): pushed commit Fixed the deterministic code/config blockers I could safely address in this PR:
Validation run locally in an isolated worktree: Live PR state after push: new head is Remaining blockers: I did not clear the release/process/reviewability gates. This still targets |
4 participants
Brings
main(+49 commits) onto the 0.7 release line and lands the (previously unpushed) OSWorld vendored evaluator work, per themain → 0.7direction.What's in here
origin/mainmerged in — L0–L3 integration gates, MLE-bench adapter, adapter skill,eval create→runrename,environment→sandbox/hubdeprecation, eval/integration fixes._osworld_vendor/,osworld_eval.py,osworld_getters.py,osworld_metrics.py,osworld_vendor.py+ getter shim + verifier-package carry (92% scorable, metric-function parity 56/56). These never touched main's adapter files, so no collision.Architectural note
mainand the 0.7 line have forked into two adapter frameworks — main hasinbound.py/harbor.py/inspect_ai.py/ors.pyand none of the OSWorld/CUA work; 0.7 hasuse_computer_cookbook/computer_use/browser_use/stagehand/iosworld/osworld_*. This merge keeps both (different files, they coexist). A deeper consolidation of the two frameworks is deliberately out of scope here.Conflict resolutions
agents/registry.py— kept both (_BENCHFLOW_PYTHONPATH+OPENCODE_PROXY_PROVIDER_ID).cli/main.py— kept both: 0.7's eval-payload helpers (still used by the eval command) + main'seval create→rundeprecated alias. Eval command itself = main'seval_run.cli/environment.py— adopted main's deprecation (environment→sandbox/hub), resolving the long-standing env-CLI revival-vs-deprecation decision toward main.uv.lock— regenerated against the mergedpyproject(keepsanyio>=4.9).Test alignment
tests/test_cli_environment.py(covered the now-deprecated rich env CLI; main moved coverage totest_cli_hub_env.py).bf-snap-*snapshot-reaper now runs in main's canonicalsandbox cleanup(_cleanup_daytona_snapshots), so the deprecatedenvironment cleanupalias inherits it.docker, daytona, modal, cua, cua-cloud— wraps in dense help tables).Verification
Full suite: 4882 passed, 15 skipped. The 3 remaining
test_eval_inbound_adapterforeign_adapter failures are pre-existing (fail identically on the pre-merge branch; need thesandbox-cuaextra in the test env), not introduced here.Note on #826
This branch also carries the
anyio>=4.9bump (bundled in the OSWorld vendor commit). PR #826 is the standalone version of that same fix — if this lands first, #826 becomes redundant and can be closed.