Skip to content

Fix AA-next OpenHands timeout guidance #2094

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Edwardf0t1 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Fix AA-next OpenHands timeout guidance #2094

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .agents/skills/evaluation/recipes/examples/example_eval_next.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ cluster:
account: ???
walltime: "04:00:00" # auto_resume chains across windows
shards: 1 # N = N nodes (each redeploys vLLM)
eval_image: ${NEL_NEXT_EVAL_IMAGE} # from eval-config (TB2.1 needs ≥0.3.1.1-harbor, multi-arch; enroot creds per SKILL Step 7.5)
eval_image: ${NEL_NEXT_EVAL_IMAGE} # from eval-config (current AA-next source of truth: 0.5.0.1-harbor, multi-arch; enroot creds per SKILL Step 7.5)
sbatch_comment: '{"OccupiedIdleGPUsJobReaper":{"exemptIdleTimeMins":"480","reason":"benchmarking","description":"nel-next agentic eval"}}'
sbatch_extra_flags: {switches: 1, exclusive: true}
container_env: # AWS creds reach the eval container ONLY via here
Expand Down
16 changes: 14 additions & 2 deletions .agents/skills/evaluation/recipes/tasks/aa_next/swebench_verified.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,8 @@ run flow). Same harbor/ECS-Fargate flow as Terminal-Bench; the deltas are the
**OpenHands agent**, a larger problem set, longer timeouts, and a different
ECR/region. Start from `recipes/examples/example_eval_next.yaml`.

> **Source of truth:** `configs/benchmarks/nel_next/swebench_verified/bench.yaml`
> **Source of truth:** `configs/benchmarks/swe-bench-verified/bench.yaml` and
> `configs/shared/nel_next_containers.yaml`
> in nvidia-eval-factory-benchmarking — match its values for a reference run.

## Task-specific values (canonical `bench.yaml`)
Expand All @@ -19,7 +20,7 @@ ECR/region. Start from `recipes/examples/example_eval_next.yaml`.
| `solver` | `timeout_strategy: max`, `run_timeout: 10800` (3h), `agent_kwargs.llm_kwargs.timeout: 3600` |
| `sandbox.region` | `us-east-2` |
| `sandbox.ecr_repository` | `${HARBOR_SWEBENCH_ECR_REPOSITORY}` (dedicated `harbor-swebench` repo, **us-west-2**, regardless of sandbox region) |
| `cluster.eval_image` | `${NEL_NEXT_EVAL_IMAGE}` (needs **≥ `0.3.1.1-harbor`** — FEP-1085 reasoning fix) |
| `cluster.eval_image` | `${NEL_NEXT_EVAL_IMAGE}` (current internal source of truth: **`0.5.0.1-harbor`**) |

@cjluo-nv cjluo-nv Aug 7, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bot comment.

This drops the previous rationale (FEP-1085 reasoning fix) for the image floor. 0.5.0.1-harbor presumably includes it, but readers who need to substitute a different tag now have no record of that constraint — and the tag ordering here isn't obviously monotonic (the reference file describes 0.17.x/0.18.x-harbor as older than 0.3.1.1-harbor), so "newer tag ⇒ includes the fix" isn't safe to infer. Suggest listing both required fixes, e.g. "must include the FEP-1085 reasoning fix and the OpenHands timeout propagation fix (Evaluator#1083)".

| `cluster.container_env.AWS_DEFAULT_REGION` | `us-east-2` (match `sandbox.region`) |
| `instruction_template` | **must be MOUNTED** — the harbor image doesn't bundle the built-in (gotcha below) |

Expand All @@ -41,6 +42,17 @@ benchmarks:
log_stream_prefix: swebench-verified-<model>-<cluster>
```

### Canary gate — verify the OpenHands request timeout

`0.3.1.1-harbor` accepts the `llm_kwargs.timeout` field above but does not pass
it to the OpenHands `LLM` client, which silently retains its 300-second default.
Use an eval image containing
[NVIDIA-NeMo/Evaluator#1083](https://github.com/NVIDIA-NeMo/Evaluator/pull/1083)
(`0.5.0.1-harbor` in the current internal source of truth). After the first
canary task starts, inspect the OpenHands startup/agent log and require an
active timeout of `3600`; treat `timeout: 300` as an infrastructure failure and
do not launch the scored run.

### Gotcha — mount the instruction template

The playbook defaults `instruction_template: swebench-instruction.md`, but the
Expand Down
8 changes: 5 additions & 3 deletions .agents/skills/evaluation/recipes/tasks/aa_next/terminal_bench_2_1.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,10 +24,11 @@ pinned via a vendored registry override in nemo-evaluator-next.
| `cluster.container_env.AWS_DEFAULT_REGION` | match `sandbox.region` |
| `max_concurrent` / `sandbox.concurrency` | `50` (canonical bench.yaml) |
| timeout_strategy | `max` (canonical bench.yaml) + `agent_kwargs.llm_kwargs.timeout: 3600`; use `task` for leaderboard-comparable |
| `cluster.eval_image` requirement | **≥ `0.3.1.1-harbor`** — TB 2.1's task set is pinned via a vendored registry override in that image (`${NEL_NEXT_EVAL_IMAGE}`, multi-arch) |
| `cluster.eval_image` requirement | current internal source of truth: **`0.5.0.1-harbor`** (`${NEL_NEXT_EVAL_IMAGE}`, multi-arch) |

These values mirror the canonical TB2.1 config — re-check it before a scored run:
`configs/benchmarks/nel_next/terminal_bench_21/bench.yaml` in
`configs/benchmarks/terminal-bench-2.1/bench.yaml` and
`configs/shared/nel_next_containers.yaml` in
nvidia-eval-factory-benchmarking (see `references/nel-next.md` + the eval-config
"source of truth" note). The `benchmarks:` block (drop into the example template):

Expand All @@ -50,7 +51,8 @@ benchmarks:
log_stream_prefix: terminalbench21-<model>-<cluster>
```

`cluster.eval_image: ${NEL_NEXT_EVAL_IMAGE}` (≥ `0.3.1.1-harbor`) and the AWS creds
`cluster.eval_image: ${NEL_NEXT_EVAL_IMAGE}` (`0.5.0.1-harbor` in the current
internal source of truth) and the AWS creds
come from `modelopttools:eval-config` (run it first) + the workspace `.env`.

## Score Extraction
Expand Down
16 changes: 13 additions & 3 deletions .agents/skills/evaluation/references/nel-next.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -131,9 +131,14 @@ with its own `run_id`, copying the shared `services:` block.

## Rules & gotchas

- **`eval_image`** = `${NEL_NEXT_EVAL_IMAGE}`. `0.3.1.1-harbor` is multi-arch and is
the minimum for **TB 2.1**; older `0.17.x/0.18.x-harbor-<arch>` are arch-suffixed.
Private gitlab-master image → cluster needs enroot creds (SKILL Step 7.5).
- **`eval_image`** = `${NEL_NEXT_EVAL_IMAGE}`. The current internal source of

@cjluo-nv cjluo-nv Aug 7, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bot comment.

Compatibility check: this now pins the eval runtime to 0.5.0.1-harbor, but .agents/scripts/nel-next.sh still installs the submitting CLI as nemo-evaluator[harbor,export]==0.3.*, and this file's header ("nel-next (nemo-evaluator 0.3.x)") plus the "Separate venv" section describe 0.3.x. If the harbor image tag tracks the nemo-evaluator version, users would be running a 0.3.x launcher against a 0.5.0.1 runtime. Since the config schemas are extra="forbid", a config that validates under --dry-run (client side, 0.3.x) could hard-fail inside the container, or vice versa. Please either (a) note explicitly that the image tag is independent of the client package version, or (b) bump/loosen NEL_NEXT_SPEC and the surrounding 0.3.x prose in the same PR.

truth pins `0.5.0.1-harbor` (multi-arch) for TB 2.1 and SWE-bench. Do not use
`0.3.1.1-harbor` for OpenHands: although it accepts
`agent_kwargs.llm_kwargs.timeout`, the client still uses its 300-second
default. The runtime must include
[NVIDIA-NeMo/Evaluator#1083](https://github.com/NVIDIA-NeMo/Evaluator/pull/1083),
which propagates that setting into OpenHands. Private gitlab-master images
need enroot creds (SKILL Step 7.5).
- **Mount sources must pre-exist** — pyxis won't create the host side of a bind
mount (invisible to `--dry-run`, fails at canary). `ssh <login> 'mkdir -p
<lustre>/<user>/.cache/{vllm,huggingface}'`.
Expand All @@ -160,6 +165,11 @@ $NEL eval {status|logs -f|report -f markdown|merge} -r <run_id> # lifecycle
$NEL mlflow-push -r <run_id> -c <cfg>.yaml # post-run: push merged bundle(s) to MLflow
```

For a SWE-bench canary, inspect the OpenHands startup/agent log before scaling
out. The configured `agent_kwargs.llm_kwargs.timeout` must appear as the active
LLM timeout (for the recipe below, `3600`); `timeout: 300` means the eval image

@cjluo-nv cjluo-nv Aug 7, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bot comment.

"for the recipe below, 3600" — there is no recipe below in this file (the next paragraph is the eval run / SLURM description); the agent_kwargs.llm_kwargs.timeout: 3600 value lives in recipes/tasks/aa_next/swebench_verified.md. Suggest rewording to e.g. "the value configured in the SWE-bench recipe (recipes/tasks/aa_next/swebench_verified.md, currently 3600)". Also, this paragraph largely restates the new "Canary gate" section in swebench_verified.md; consider keeping the detail in one place and cross-linking so the two can't drift.

does not contain the propagation fix and the canary is invalid.

`eval run` on a slurm cluster scp's the sbatch + redacted `.secrets.env` and
submits via SSH; a built-in afternotok chain auto-resumes across walltime windows;
sharded runs auto-merge. **SLURM does not auto-export** — `mlflow-push` is the final
Expand Down
Loading