fix(eval): clean Linear issue UX paths

[bingran-you]

Purpose

Fixes three repo-owned Linear issues that had no GitHub mirror before today's daily scan:

• Fixes ProgramBench YAML eval should fail cleanly when generated tasks are missing #915 / Linear ENG-162: ProgramBench YAML config no longer reaches a raw missing-.cache/programbench-benchflow failure; missing YAML tasks_dir is rejected at config load with the ProgramBench generation hint.
• Fixes bench eval create should expose --max-retries alias #916 / Linear ENG-169: bench eval create now accepts --max-retries as the expected alias for --retry-attempts, including YAML --config eval runs, and rejects both aliases when passed together.
• Fixes Missing agent credentials should be clean non-retryable rollout errors #917 / Linear ENG-257: missing agent/model credentials now use a typed MissingAgentCredentialError, surface as clean rollout errors, and classify as non-retryable provider_auth instead of falling into the generic traceback logger.

Changes

• Adds --max-retries to bench eval create and threads the normalized retry override through both batch and config-file eval paths.
• Validates native YAML tasks_dir existence during Evaluation.from_yaml() and adds a targeted hint for .cache/programbench-benchflow.
• Adds MissingAgentCredentialError, catches it explicitly in Rollout.run(), and classifies missing credential strings as provider_auth.
• Adds regression coverage for all three issues.

Validation

• uv run --extra dev python -m pytest tests/test_agent_idle_timeout_cli.py tests/test_cli_edge_case_hardening.py tests/test_provider_auth_detection.py -q -> 86 passed
• uv run --extra dev python -m pytest tests/test_agent_idle_timeout_cli.py tests/test_cli_edge_case_hardening.py::test_from_yaml_rejects_missing_tasks_dir_with_programbench_hint tests/test_cli_edge_case_hardening.py::test_eval_create_config_missing_tasks_dir_no_raw_traceback tests/test_provider_auth_detection.py::test_run_missing_credential_error_is_clean_provider_auth -q -> 15 passed
• uv run --extra dev ruff check src/benchflow/cli/main.py src/benchflow/evaluation.py src/benchflow/agents/env.py src/benchflow/_utils/scoring.py src/benchflow/rollout/__init__.py tests/test_agent_idle_timeout_cli.py tests/test_cli_edge_case_hardening.py tests/test_provider_auth_detection.py
• uv run --extra dev ruff format --check src/benchflow/cli/main.py src/benchflow/evaluation.py src/benchflow/agents/env.py src/benchflow/_utils/scoring.py src/benchflow/rollout/__init__.py tests/test_agent_idle_timeout_cli.py tests/test_cli_edge_case_hardening.py tests/test_provider_auth_detection.py
• uv run --extra dev ty check src/benchflow/cli/main.py src/benchflow/evaluation.py src/benchflow/agents/env.py src/benchflow/_utils/scoring.py src/benchflow/rollout/__init__.py
• git diff --check

Merge Gate

Keep this on status:in-progress until GitHub CI completes, then move to status:ready / review:pending. Per repo policy, do not merge without non-author human review.

[bingran-you]

Daily scan implementation update (2026-07-11): pushed follow-up 0d17cdab2 so the new regression docstrings name PR #918 per repo convention.

Current local validation on this branch:

• uv run --extra dev python -m pytest tests/test_agent_idle_timeout_cli.py tests/test_cli_edge_case_hardening.py tests/test_provider_auth_detection.py -q -> 86 passed
• uv run --extra dev python -m pytest tests/test_agent_idle_timeout_cli.py::test_eval_create_max_retries_alias_lands_in_config tests/test_agent_idle_timeout_cli.py::test_eval_create_max_retries_alias_overrides_yaml_config tests/test_agent_idle_timeout_cli.py::test_eval_create_rejects_duplicate_retry_aliases tests/test_cli_edge_case_hardening.py::test_from_yaml_rejects_missing_tasks_dir_with_programbench_hint tests/test_cli_edge_case_hardening.py::test_eval_create_config_missing_tasks_dir_no_raw_traceback tests/test_provider_auth_detection.py::test_run_missing_credential_error_is_clean_provider_auth -q -> 6 passed
• uv run --extra dev ruff check ... on touched source/tests -> passed
• uv run --extra dev ruff format --check ... on touched source/tests -> passed
• uv run --extra dev ty check src/benchflow/cli/main.py src/benchflow/evaluation.py src/benchflow/agents/env.py src/benchflow/_utils/scoring.py src/benchflow/rollout/__init__.py -> passed
• git diff --check -> clean

Keeping status:in-progress while GitHub CI runs; review:pending remains the human review gate before merge.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 956c3fcdc3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Defer or rebase native YAML tasks_dir validation

This new existence check evaluates relative tasks_dir values against the current process directory during config loading. Existing Evaluation.from_yaml() callers and tests create self-contained configs outside the cwd with tasks_dir: tasks and load them to inspect/override config fields before running; those now raise YAML tasks_dir not found even though the loader previously accepted the config. Please either keep from_yaml() as a parser and defer this validation to the run path, or resolve relative paths consistently before rejecting them.

Useful? React with 👍 / 👎.

[bingran-you]

Daily scan CI follow-up (2026-07-11): pushed 2558f2205 after the first full test job failed.

Root cause:

• The initial ProgramBench YAML guard rejected every missing tasks_dir in Evaluation.from_yaml(). That was too broad because many unit tests intentionally parse YAML before materializing task dirs.
• The new --max-retries flag also needed to be added to docs/reference/cli.md for the CLI docs drift gate.

Fix:

• Narrowed the YAML guard to the reported ProgramBench cache path .cache/programbench-benchflow, preserving existing parse-before-materialization behavior for other YAML configs.
• Documented --max-retries in the CLI reference.

Local validation after the fix:

• uv run --extra dev python -m pytest tests/test_agent_router_cli_e2e.py::test_cli_create_writes_full_scaffold_that_compiles_and_parses tests/test_cli_docs_drift.py::test_eval_run_flags_match_cli_md_bidirectional tests/test_config_override.py::test_cli_config_override_applies_on_run_config_file_path tests/test_job_sequential_shared.py::test_native_yaml_parses_job_mode tests/test_oracle_chokepoint.py::TestOracleYamlLoaders::test_native_yaml_oracle_no_model tests/test_oracle_chokepoint.py::TestOracleYamlLoaders::test_native_yaml_non_oracle_keeps_default_when_omitted tests/test_yaml_config.py -q -> 38 passed
• uv run --extra dev python -m pytest tests/test_agent_idle_timeout_cli.py tests/test_cli_edge_case_hardening.py tests/test_provider_auth_detection.py -q -> 86 passed
• focused ruff check, ruff format --check on touched Python files, ty check, and git diff --check passed.

Keeping status:in-progress while refreshed GitHub CI runs.

[bingran-you]

Users Simulation automation review (2026-07-11): blocked on head 2558f22053e10db4de4ee748aa2ff6c6f75c3dfa.

The focused tests, static checks, CLI/SDK oracle smokes, GitHub checks, and integration-light artifact are green, but user simulation found a real ProgramBench regression: tasks_dir: .cache/programbench-benchflow is rejected even when that directory already exists.

Repro:

tmp=/tmp/pr-918-existing-programbench
rm -rf "$tmp"
mkdir -p "$tmp/.cache/programbench-benchflow"
printf 'tasks_dir: .cache/programbench-benchflow\nagent: oracle\n' > "$tmp/config.yaml"
(cd "$tmp" && uv run --project /tmp/users-simulation-20260711/pr-918 bench eval create --config config.yaml --jobs-dir "$tmp/jobs")

Actual: exits 1 with YAML tasks_dir not found ... .cache/programbench-benchflow.

Root cause: src/benchflow/evaluation.py checks elif str(ref) == ".cache/programbench-benchflow" without also requiring not tasks_dir.exists(). This fixes the missing-path UX but breaks the valid generated ProgramBench config path. Please guard that branch with the existence check and add a regression test proving an existing .cache/programbench-benchflow is accepted.

Other validation passed:

• pytest tests/test_agent_idle_timeout_cli.py tests/test_cli_edge_case_hardening.py tests/test_provider_auth_detection.py -q -> 86 passed.
• Focused ruff, ty check src, git diff --check.
• bench eval run --help shows --max-retries.
• Missing credential classification smoke maps generated credential errors to provider_auth while idle timeout still maps to idle_timeout.
• CLI oracle run + artifact checker passed; SDK oracle smoke passed.
• Current-head integration-light run 29146787636 validated healthy with benchflow-experiment-review: 1/1 healthy, ACP + LLM trajectories present, training-ready results.jsonl, reward present, 369,660 tokens, 18 tool calls, 306.9s timing.

Artifact caveat: this GLM user-endpoint artifact still has null agent_result.cost_usd / price_source; token, timing, and trajectory metadata are complete.

Thermo review: blocked. A ProgramBench-specific special case was inserted into the already-large Evaluation._from_native_yaml path, and the condition is wrong. At minimum this needs elif not tasks_dir.exists() and str(ref) == ".cache/programbench-benchflow":; cleaner would be extracting missing-tasks_dir diagnostics into a focused helper instead of growing bespoke branches inside the 1.9k-line evaluation module.

[bingran-you]

Follow-up for the Users Simulation blocker on head 2558f22053e10db4de4ee748aa2ff6c6f75c3dfa: pushed ccb8284cd62e37ab898393fd74e07bbd6b805238.

Fix:

• Guarded the ProgramBench missing-path diagnostic with not tasks_dir.exists(), so an existing generated .cache/programbench-benchflow path is accepted instead of being rejected as missing.
• Added test_from_yaml_accepts_existing_programbench_tasks_dir, with a PR fix(eval): clean Linear issue UX paths #918 regression docstring, to cover the generated-cache path.

Validation:

• uv run --extra dev python -m pytest tests/test_cli_edge_case_hardening.py::test_from_yaml_rejects_missing_tasks_dir_with_programbench_hint tests/test_cli_edge_case_hardening.py::test_from_yaml_accepts_existing_programbench_tasks_dir tests/test_cli_edge_case_hardening.py::test_eval_create_config_missing_tasks_dir_no_raw_traceback -q -> 3 passed.
• uv run --extra dev python -m pytest tests/test_cli_edge_case_hardening.py -q -> 59 passed.
• uv run --extra dev ruff check src/benchflow/evaluation.py tests/test_cli_edge_case_hardening.py -> passed.
• uv run --extra dev ruff format --check src/benchflow/evaluation.py tests/test_cli_edge_case_hardening.py -> passed.
• uv run --extra dev ty check src/benchflow/evaluation.py -> passed.
• git diff --check -> clean.
• The original synthetic CLI repro no longer fails with YAML tasks_dir not found; it now reaches the normal empty-task-selection guard because that repro creates an empty directory.

Current GitHub checks on ccb8284cd at the time of this comment: detect-scope, parity, pip-audit, and test are green; rollout-smoke is still in progress. I am removing status:blocked and setting status:ready; review:pending remains the human review gate.

[bingran-you]

Users Simulation automation review (2026-07-12): ready by simulation on head ccb8284cd62e37ab898393fd74e07bbd6b805238.

Validated focused regressions and adjacent YAML/docs surface:

• Main-thread focused checks: pytest tests/test_agent_idle_timeout_cli.py tests/test_cli_edge_case_hardening.py tests/test_provider_auth_detection.py -q -> 87 passed; focused ruff and ty passed.
• Subagent adjacent checks: YAML/docs/config/oracle slice -> 38 passed; ruff format --check, ty, and git diff --check passed.
• CLI/SDK probes confirmed --max-retries propagation, duplicate alias rejection, clean missing ProgramBench hint, and existing generated .cache/programbench-benchflow no longer being rejected as missing.

Current-head artifact 29153730886 passed benchflow-experiment-review: 1/1 healthy, with result.json, results.jsonl, ACP + LLM trajectories, and training-ready output present. Telemetry: 302,280 total tokens, 17 tool calls, 20/20 LLM responses with usage, reward present, total timing 224.1s.

Cost metadata caveat: per-rollout agent_result.cost_usd and price_source are null for this GLM user-endpoint artifact while token/timing telemetry is complete and summary cost is zero/unpriced.

Thermo review: no blocking structural regression. The ProgramBench guard is now narrow to the generated-path case, the retry alias is normalized once before planning, and the credential failure path uses a typed exception rather than generic traceback handling. Existing large files are touched, but no new 1k-line crossing was introduced.

Labels stay status:ready + review:pending.