From 5a51296c610d9dc911d85a06202f2b5cf1e696f6 Mon Sep 17 00:00:00 2001 From: Chenjie Luo Date: Tue, 4 Aug 2026 17:06:27 +0000 Subject: [PATCH 1/4] [skill] evaluation: align nel-next TB2.1/SWE-bench with golden toolchain Terminal-Bench 2.1 configs generated from this skill drifted from the canonical eval-factory config (configs/benchmarks/terminal-bench-2.1/ bench.yaml). The scoring contract (playbook, repeats=8, timeout_strategy, run_timeout, concurrency) already matched; the toolchain and several proxy-level defaults did not. - nel-next.sh: install from the public upstream repo (github.com/NVIDIA-NeMo/Evaluator, default branch -> 0.4.0) instead of PyPI. PyPI nemo-evaluator tops out at 0.3.0 and cannot reach the 0.4.x toolchain golden runs on. NEL_NEXT_SPEC becomes the PyPI escape hatch and now takes precedence when explicitly set; NEL_NEXT_ORIGIN stays overridable from .env so internal mirrors stay out of this repo. - eval_image: document the golden pin 0.5.0.1-harbor (single source of truth: configs/shared/nel_next_containers.yaml) rather than 0.3.1.1-harbor as a floor. The TB2.1 task set is unchanged across those versions -- the vendored registry override has not moved since 2026-06-03 and both score 89 samples -- so this is a toolchain fix, not a benchmark change. - proxy.request_timeout 1800 -> 3600: it must be >= the solver's llm_kwargs.timeout (3600), otherwise the proxy truncates long agent turns the harness is still waiting on. - drop_params: add max_input_tokens_per_task and no_rebuild, which the 0.5.x harbor eval image sends and vLLM rejects unless stripped. - exclude_patterns: add model_traffic.jsonl so captured request bodies stay in the run dir and never reach MLflow. - add the http_pairs_dump interceptor (last in chain) for HTTP diagnostics. - document sharding: max_concurrent/sandbox.concurrency are per shard, so shards:N multiplies both serving capacity and live sandboxes. - .gitignore: broaden .env/.env-* to .env* so secret backups such as .env.bak-tb21 cannot be staged. Verified: nel-next.sh --version reports 0.4.0 built from Evaluator.git@9dcca2ae; both TB2.1 run configs pass `eval run --dry-run` on that CLI with 0.5.0.1-harbor in the generated sbatch. Signed-off-by: Chenjie Luo --- .agents/scripts/nel-next.sh | 34 ++++++++++++++----- .../recipes/examples/example_eval_next.yaml | 11 +++--- .../tasks/aa_next/swebench_verified.md | 2 +- .../tasks/aa_next/terminal_bench_2_1.md | 26 +++++++++++--- .../skills/evaluation/references/nel-next.md | 32 +++++++++++++---- .gitignore | 3 +- 6 files changed, 81 insertions(+), 27 deletions(-) diff --git a/.agents/scripts/nel-next.sh b/.agents/scripts/nel-next.sh index 92f3f18f35d..748ad413f5b 100755 --- a/.agents/scripts/nel-next.sh +++ b/.agents/scripts/nel-next.sh @@ -32,21 +32,37 @@ # using the config's export_config.mlflow (resolves ${MLFLOW_TRACKING_URI}, forces # emit_traces=false to avoid the per-sample hang). Run after `source .env`. # -# Install source (env overrides): NEL_NEXT_SPEC (PyPI default, "nemo-evaluator[harbor,export]==0.3.*" -# — [export] pulls mlflow for mlflow-push; pin an exact 0.3.x here for reproducibility), or -# NEL_NEXT_ORIGIN [+ NEL_NEXT_REF] for the internal git build. uv caches the resolved env and -# refreshes it when the spec changes. +# Install source (env overrides): NEL_NEXT_ORIGIN [+ NEL_NEXT_REF] selects a git build +# (the DEFAULT — see below); NEL_NEXT_SPEC forces a plain PyPI release instead. +# uv caches the resolved env and refreshes it when the spec changes. +# +# Default is the PUBLIC upstream repo (github.com/NVIDIA-NeMo/Evaluator — the `Repository` +# URL in nemo-evaluator's own pyproject). No auth, nothing internal. Its default branch is +# version 0.4.0 and ships the vendored `harbor_datasets/registry_overrides` (the TB 2.1 +# task-set pin), which is what the golden TB2.1/SWE-bench toolchain runs on. +# +# Why not plain PyPI: `nemo-evaluator` on PyPI tops out at **0.3.0**, so a version pin +# cannot reach 0.4.x. Set NEL_NEXT_SPEC (e.g. "nemo-evaluator[harbor,export]==0.3.*") to +# fall back to it deliberately — older toolchain, not golden-comparable. +# +# NEL_NEXT_REF: there is no v0.4.0 tag (public tags stop at v0.3.0); 0.4.0 is an untagged +# default-branch build. Leave REF empty to track the branch, or pin a commit SHA for +# reproducibility (9dcca2ae33362b4236ed4027f1516628d24c2cc2 was HEAD at 2026-08-04). +# +# To build from an internal mirror instead, set NEL_NEXT_ORIGIN in `.env` — internal URLs +# stay out of this repo, same convention as ${NEL_NEXT_EVAL_IMAGE}. set -euo pipefail # [harbor] = agentic/sandbox deps; [export] pulls mlflow for `mlflow-push`. -NEL_NEXT_SPEC="${NEL_NEXT_SPEC:-nemo-evaluator[harbor,export]==0.3.*}" -NEL_NEXT_ORIGIN="${NEL_NEXT_ORIGIN:-}" +NEL_NEXT_SPEC="${NEL_NEXT_SPEC:-}" +NEL_NEXT_ORIGIN="${NEL_NEXT_ORIGIN:-git+https://github.com/NVIDIA-NeMo/Evaluator.git}" NEL_NEXT_REF="${NEL_NEXT_REF:-}" -if [[ -n "$NEL_NEXT_ORIGIN" ]]; then - INSTALL_SPEC="nemo-evaluator[harbor,export] @ ${NEL_NEXT_ORIGIN}${NEL_NEXT_REF:+@${NEL_NEXT_REF}}" -else +# NEL_NEXT_SPEC wins when explicitly set (PyPI escape hatch); otherwise use the git origin. +if [[ -n "$NEL_NEXT_SPEC" ]]; then INSTALL_SPEC="$NEL_NEXT_SPEC" +else + INSTALL_SPEC="nemo-evaluator[harbor,export] @ ${NEL_NEXT_ORIGIN}${NEL_NEXT_REF:+@${NEL_NEXT_REF}}" fi _log() { printf '\033[2m %s\033[0m\n' "$*" >&2; } diff --git a/.agents/skills/evaluation/recipes/examples/example_eval_next.yaml b/.agents/skills/evaluation/recipes/examples/example_eval_next.yaml index c61908907d4..d11c001b8d9 100644 --- a/.agents/skills/evaluation/recipes/examples/example_eval_next.yaml +++ b/.agents/skills/evaluation/recipes/examples/example_eval_next.yaml @@ -43,12 +43,15 @@ services: - ???:/cache/huggingface generation: {temperature: 1.0, top_p: 0.95} # from model card (reasoning mode); adjust per card — mandatory lookup (references/model-card-research.md), same as 0.2.6 proxy: - request_timeout: 1800 + request_timeout: 3600 # canonical; MUST be >= benchmarks[].solver.agent_kwargs.llm_kwargs.timeout extra_body: {skip_special_tokens: false} # add model-card sampling extras here if the card specifies them; mirror them in the export tags below interceptors: - name: drop_params # agents send max_tokens; many servers reject it - config: {params: [max_tokens, max_completion_tokens]} + # last two are sent by the 0.5.x harbor eval image; vLLM 400s on them unless stripped + config: {params: [max_tokens, max_completion_tokens, max_input_tokens_per_task, no_rebuild]} # SWE-bench (OpenHands, multi-turn) adds turn_counter + consolidate_system + a system_message — see swebench_verified.md + - name: http_pairs_dump # FEP-1104/1120 diagnostics; canonical LAST in the chain + config: {dump_path: "$${NEL_OUTPUT_DIR}/http_pairs_metrics.json", first_n: 50} # $$ defers expansion to run time node_pool: gpu benchmarks: @@ -72,7 +75,7 @@ cluster: account: ??? walltime: "04:00:00" # auto_resume chains across windows shards: 1 # N = N nodes (each redeploys vLLM) - eval_image: ${NEL_NEXT_EVAL_IMAGE} # from eval-config (TB2.1 needs ≥0.3.1.1-harbor, multi-arch; enroot creds per SKILL Step 7.5) + eval_image: ${NEL_NEXT_EVAL_IMAGE} # from eval-config (golden pin 0.5.0.1-harbor, multi-arch; enroot creds per SKILL Step 7.5) sbatch_comment: '{"OccupiedIdleGPUsJobReaper":{"exemptIdleTimeMins":"480","reason":"benchmarking","description":"nel-next agentic eval"}}' sbatch_extra_flags: {switches: 1, exclusive: true} container_env: # AWS creds reach the eval container ONLY via here @@ -100,7 +103,7 @@ output: experiment_name: ??? # /- (hardcode; ${USER}=root in-container) log_config_params: true copy_logs: true - exclude_patterns: ["shard*"] + exclude_patterns: ["shard*", "model_traffic.jsonl"] # captured request bodies (FEA-224) stay in the run dir description: ??? # ' | T=1.0 top_p=0.95 | (timeout_strategy=…) | r8' # model/checkpoint_path/benchmark drive dashboard attribution (engine logs only a generic metric key); temperature/top_p mirror generation above. tags: {framework: vllm, model: "???", checkpoint_path: "???", benchmark: "???", temperature: '1.0', top_p: '0.95'} diff --git a/.agents/skills/evaluation/recipes/tasks/aa_next/swebench_verified.md b/.agents/skills/evaluation/recipes/tasks/aa_next/swebench_verified.md index 00287bc97e1..8e51432de55 100644 --- a/.agents/skills/evaluation/recipes/tasks/aa_next/swebench_verified.md +++ b/.agents/skills/evaluation/recipes/tasks/aa_next/swebench_verified.md @@ -19,7 +19,7 @@ ECR/region. Start from `recipes/examples/example_eval_next.yaml`. | `solver` | `timeout_strategy: max`, `run_timeout: 10800` (3h), `agent_kwargs.llm_kwargs.timeout: 3600` | | `sandbox.region` | `us-east-2` | | `sandbox.ecr_repository` | `${HARBOR_SWEBENCH_ECR_REPOSITORY}` (dedicated `harbor-swebench` repo, **us-west-2**, regardless of sandbox region) | -| `cluster.eval_image` | `${NEL_NEXT_EVAL_IMAGE}` (needs **≥ `0.3.1.1-harbor`** — FEP-1085 reasoning fix) | +| `cluster.eval_image` | `${NEL_NEXT_EVAL_IMAGE}` — golden pin **`0.5.0.1-harbor`** (same single source as TB2.1: `configs/shared/nel_next_containers.yaml`). `0.3.1.1-harbor` is the bare minimum (FEP-1085 reasoning fix) but is two minors behind | | `cluster.container_env.AWS_DEFAULT_REGION` | `us-east-2` (match `sandbox.region`) | | `instruction_template` | **must be MOUNTED** — the harbor image doesn't bundle the built-in (gotcha below) | diff --git a/.agents/skills/evaluation/recipes/tasks/aa_next/terminal_bench_2_1.md b/.agents/skills/evaluation/recipes/tasks/aa_next/terminal_bench_2_1.md index d5b976513c7..c67503550c4 100644 --- a/.agents/skills/evaluation/recipes/tasks/aa_next/terminal_bench_2_1.md +++ b/.agents/skills/evaluation/recipes/tasks/aa_next/terminal_bench_2_1.md @@ -24,12 +24,21 @@ pinned via a vendored registry override in nemo-evaluator-next. | `cluster.container_env.AWS_DEFAULT_REGION` | match `sandbox.region` | | `max_concurrent` / `sandbox.concurrency` | `50` (canonical bench.yaml) | | timeout_strategy | `max` (canonical bench.yaml) + `agent_kwargs.llm_kwargs.timeout: 3600`; use `task` for leaderboard-comparable | -| `cluster.eval_image` requirement | **≥ `0.3.1.1-harbor`** — TB 2.1's task set is pinned via a vendored registry override in that image (`${NEL_NEXT_EVAL_IMAGE}`, multi-arch) | +| `cluster.eval_image` requirement | **`0.5.0.1-harbor`** — the golden pin (`${NEL_NEXT_EVAL_IMAGE}`, multi-arch). `0.3.1.1-harbor` is the bare minimum that runs TB 2.1 but is two minors behind | +| `proxy.request_timeout` | `3600` — must be **≥** `agent_kwargs.llm_kwargs.timeout` | +| `drop_params` | `max_tokens`, `max_completion_tokens`, `max_input_tokens_per_task`, `no_rebuild` | +| `output.export_config.mlflow.exclude_patterns` | `["shard*", "model_traffic.jsonl"]` | + +TB 2.1's task set is pinned by a vendored registry override in the eval image, but that +override (`harbor_datasets/registry_overrides/terminal_bench_2_1.json`) has not changed since +2026-06-03 — **89 tasks on both 0.3.1.1 and 0.5.0.1**. So the image bump is a toolchain fix and +does not move the benchmark; scores stay comparable across it. These values mirror the canonical TB2.1 config — re-check it before a scored run: -`configs/benchmarks/nel_next/terminal_bench_21/bench.yaml` in -nvidia-eval-factory-benchmarking (see `references/nel-next.md` + the eval-config -"source of truth" note). The `benchmarks:` block (drop into the example template): +`configs/benchmarks/terminal-bench-2.1/bench.yaml` (+ `manifest.yaml`) in +nvidia-eval-factory-benchmarking (`dl/JoC/competitive_evaluation/…`), with the image pin in +`configs/shared/nel_next_containers.yaml`. See `references/nel-next.md` + the eval-config +"source of truth" note. The `benchmarks:` block (drop into the example template): ```yaml benchmarks: @@ -50,9 +59,16 @@ benchmarks: log_stream_prefix: terminalbench21-- ``` -`cluster.eval_image: ${NEL_NEXT_EVAL_IMAGE}` (≥ `0.3.1.1-harbor`) and the AWS creds +`cluster.eval_image: ${NEL_NEXT_EVAL_IMAGE}` (`0.5.0.1-harbor`) and the AWS creds come from `modelopttools:eval-config` (run it first) + the workspace `.env`. +**Sharding.** `max_concurrent`/`sandbox.concurrency` are **per shard**, and each shard gets +its own vLLM instance on its own node — so `shards: N` multiplies both serving capacity and +live Fargate sandboxes (`N × concurrency`). Golden runs 89 tasks × r8 = 712 trials on +`shards: 4` (~178 trials/shard). Purely a wall-clock/throughput lever: trials are partitioned +and merged, so the score is unaffected. Before raising it, check `N × concurrency` against the +harbor Fargate quota and `N × gpus_per_node` against your allocation. + ## Score Extraction Report **`pass@1`** only — benchmark `terminal-bench@2.1`, scorer `pass@1` (0–1): diff --git a/.agents/skills/evaluation/references/nel-next.md b/.agents/skills/evaluation/references/nel-next.md index 5d1975f4e2f..21cd063e7a8 100644 --- a/.agents/skills/evaluation/references/nel-next.md +++ b/.agents/skills/evaluation/references/nel-next.md @@ -27,8 +27,13 @@ Installing 0.3.x into the 0.2.6 env clobbers `nel`, so it lives in its own venv: .agents/scripts/nel-next.sh eval run --dry-run | --submit | … ``` -Default install is public PyPI `nemo-evaluator[harbor]==0.3.*`; set -`NEL_NEXT_ORIGIN`/`NEL_NEXT_REF` for the internal git build (see script header). +Default install is a git build from the **public upstream repo** +(`github.com/NVIDIA-NeMo/Evaluator`, no auth) via `NEL_NEXT_ORIGIN` — its default branch is +**0.4.0** and ships the vendored TB 2.1 registry override, matching the golden toolchain. +Plain PyPI `nemo-evaluator` tops out at **0.3.0**, so a version pin can't reach 0.4.x; set +`NEL_NEXT_SPEC` only to deliberately fall back to it (older, not golden-comparable). There is +no `v0.4.0` tag — pin `NEL_NEXT_REF` to a commit SHA for reproducibility. To build from an +internal mirror, set `NEL_NEXT_ORIGIN` in `.env`; internal URLs stay out of this repo. ## Credentials + internal infra (`.env`) @@ -75,7 +80,7 @@ services: extra_env: {...} # VLLM_* backend env (e.g. NVFP4 MoE flags) container_mounts: [/.cache/vllm:/cache/vllm, ...] generation: {temperature: 1.0, top_p: 0.95} - proxy: {request_timeout: 1800, extra_body: {...}, interceptors: [...]} + proxy: {request_timeout: 3600, extra_body: {...}, interceptors: [...]} # >= llm_kwargs.timeout node_pool: gpu benchmarks: # EXACTLY ONE entry — one benchmark per config (see "One benchmark per config") - playbook: # per recipe @@ -131,9 +136,24 @@ with its own `run_id`, copying the shared `services:` block. ## Rules & gotchas -- **`eval_image`** = `${NEL_NEXT_EVAL_IMAGE}`. `0.3.1.1-harbor` is multi-arch and is - the minimum for **TB 2.1**; older `0.17.x/0.18.x-harbor-` are arch-suffixed. - Private gitlab-master image → cluster needs enroot creds (SKILL Step 7.5). +- **`eval_image`** = `${NEL_NEXT_EVAL_IMAGE}`. Golden **pins `0.5.0.1-harbor`** (single + source of truth: `configs/shared/nel_next_containers.yaml` in the eval-factory repo) — + track that pin, don't treat it as a floor. `0.3.1.1-harbor` is the bare minimum that can + run **TB 2.1** at all but is two minors behind golden; older `0.17.x/0.18.x-harbor-` + are arch-suffixed. The TB2.1 task set itself is stable across these versions (the + vendored `registry_overrides/terminal_bench_2_1.json` has not changed since 2026-06-03, + and both 0.3.1.1 and 0.5.0.1 score 89 samples), so the bump is a toolchain fix, not a + benchmark change. Private gitlab-master image → cluster needs enroot creds (SKILL Step 7.5). +- **`proxy.request_timeout` must be >= `agent_kwargs.llm_kwargs.timeout`** (both 3600 in + canonical TB2.1). A smaller proxy timeout silently truncates long agent turns. +- **`drop_params` takes four params** for harbor agentic benchmarks: `max_tokens`, + `max_completion_tokens`, `max_input_tokens_per_task`, `no_rebuild`. The last two are sent + by the 0.5.x eval image; vLLM 400s on them if they aren't stripped. +- **`exclude_patterns`** = `["shard*", "model_traffic.jsonl"]` — captured request bodies + (FEA-224) stay in the run dir and must never be exported to MLflow. +- **`http_pairs_dump`** (FEP-1104/1120) goes **last** in the interceptor chain: + `config: {dump_path: $${NEL_OUTPUT_DIR}/http_pairs_metrics.json, first_n: 50}`. Note the + `$$` — it defers expansion to run time. Diagnostics only; does not affect scoring. - **Mount sources must pre-exist** — pyxis won't create the host side of a bind mount (invisible to `--dry-run`, fails at canary). `ssh 'mkdir -p //.cache/{vllm,huggingface}'`. diff --git a/.gitignore b/.gitignore index 89a4a8540dd..72089b06e7e 100644 --- a/.gitignore +++ b/.gitignore @@ -29,8 +29,7 @@ docs/source/reference/generated **/.ipynb_checkpoints # Environments -.env -.env-* +.env* .venv env/ venv/ From 821ed9229713611c33148e67058fb5ce9ff355e0 Mon Sep 17 00:00:00 2001 From: Chenjie Luo Date: Tue, 4 Aug 2026 17:17:29 +0000 Subject: [PATCH 2/4] [skill] evaluation: align SWE-bench Verified recipe with golden Same golden cross-check as the TB2.1 change, against the canonical configs/benchmarks/swe-bench-verified/bench.yaml and a reference oci-hsg run (Qwen3.6-35B-A3B, pass@1 0.7012 over 500 tasks x r5). - fix the stale source-of-truth path (configs/benchmarks/nel_next/ swebench_verified/ -> configs/benchmarks/swe-bench-verified/) and point at the shared eval-image pin. - correct the interceptor ORDER: SWE-bench differs from TB2.1 -- http_pairs_dump is FIRST, and drop_params comes BEFORE consolidate_system. - turn_counter gains position: system_message. - drop_params: add max_input_tokens_per_task and no_rebuild. - exclude_patterns: add model_traffic.jsonl; document proxy.model_traffic.capture_request_body. - document that the instruction_template CONTENT is scoring-relevant, and that the public built-in in nemo_evaluator/templates/ is a DIFFERENT prompt whose results are not comparable to the canonical one. - state plainly that omitting the system_message interceptor is a scoring change, not a simplification. - reasoning_replay.mode is per MODEL, not per benchmark: think_tags (Qwen), native (GLM), omitted (MiniMax). Taking it from another model's config is a silent output-parsing bug. - max_concurrent: bench.yaml sets 15, but per-model leaves override it. - document sharding: concurrency is per shard; the reference run uses shards:10 for 500 x r5 = 2500 trials. Signed-off-by: Chenjie Luo --- .../tasks/aa_next/swebench_verified.md | 58 +++++++++++++++---- 1 file changed, 47 insertions(+), 11 deletions(-) diff --git a/.agents/skills/evaluation/recipes/tasks/aa_next/swebench_verified.md b/.agents/skills/evaluation/recipes/tasks/aa_next/swebench_verified.md index 8e51432de55..15730b5be5b 100644 --- a/.agents/skills/evaluation/recipes/tasks/aa_next/swebench_verified.md +++ b/.agents/skills/evaluation/recipes/tasks/aa_next/swebench_verified.md @@ -5,8 +5,9 @@ run flow). Same harbor/ECS-Fargate flow as Terminal-Bench; the deltas are the **OpenHands agent**, a larger problem set, longer timeouts, and a different ECR/region. Start from `recipes/examples/example_eval_next.yaml`. -> **Source of truth:** `configs/benchmarks/nel_next/swebench_verified/bench.yaml` -> in nvidia-eval-factory-benchmarking — match its values for a reference run. +> **Source of truth:** `configs/benchmarks/swe-bench-verified/bench.yaml` in +> nvidia-eval-factory-benchmarking (`dl/JoC/competitive_evaluation/…`), with the eval-image +> pin in `configs/shared/nel_next_containers.yaml` — match its values for a reference run. ## Task-specific values (canonical `bench.yaml`) @@ -15,13 +16,17 @@ ECR/region. Start from `recipes/examples/example_eval_next.yaml`. | `playbook` | `swebench_verified` (`harbor://swebench-verified@1.0`) | | agent | `openhands-sdk` (playbook; `agent_kwargs: {max_iterations: 200, version: "1.17.0"}`) | | scope | 500 Python tasks × `repeats: 5` | -| `max_concurrent` / `sandbox.concurrency` | `15` | +| `max_concurrent` / `sandbox.concurrency` | `15` in `bench.yaml`; **per-model leaves override it** (e.g. MiniMax-M2.7 uses `20`) — check the model's own golden leaf | | `solver` | `timeout_strategy: max`, `run_timeout: 10800` (3h), `agent_kwargs.llm_kwargs.timeout: 3600` | | `sandbox.region` | `us-east-2` | | `sandbox.ecr_repository` | `${HARBOR_SWEBENCH_ECR_REPOSITORY}` (dedicated `harbor-swebench` repo, **us-west-2**, regardless of sandbox region) | | `cluster.eval_image` | `${NEL_NEXT_EVAL_IMAGE}` — golden pin **`0.5.0.1-harbor`** (same single source as TB2.1: `configs/shared/nel_next_containers.yaml`). `0.3.1.1-harbor` is the bare minimum (FEP-1085 reasoning fix) but is two minors behind | | `cluster.container_env.AWS_DEFAULT_REGION` | `us-east-2` (match `sandbox.region`) | -| `instruction_template` | **must be MOUNTED** — the harbor image doesn't bundle the built-in (gotcha below) | +| `instruction_template` | `/configs/prompts/swebench_instruction.md`, **must be MOUNTED** — the harbor image doesn't bundle a built-in, and the template **content is scoring-relevant** (gotcha below) | +| `proxy.request_timeout` | `3600` (FEP-1104 paired HTTP timeout; leaves mirror it on the service proxy) | +| `drop_params` | `max_tokens`, `max_completion_tokens`, `max_input_tokens_per_task`, `no_rebuild` | +| `output.export_config.mlflow.exclude_patterns` | `["shard*", "model_traffic.jsonl"]` | +| `system_message` | `strategy: replace` + the canonical OpenHands prompt — **scoring-relevant**, copy verbatim from `bench.yaml` | ```yaml benchmarks: @@ -45,9 +50,15 @@ benchmarks: The playbook defaults `instruction_template: swebench-instruction.md`, but the harbor image doesn't ship that built-in → run dies at finalize with -`FileNotFoundError: instruction_template not found`. Mount it (the canonical -config uses the compeval OpenHands prompt; the public built-in from the host venv -works too): +`FileNotFoundError: instruction_template not found`. So it must be mounted. + +**Which file you mount changes the score.** The canonical runs mount the internal compeval +prompt (`swebench_instruction.md` — note the underscore) at +`/configs/prompts/swebench_instruction.md`; take it from the canonical `bench.yaml` / +reference run dir, not from this repo. The public built-in shipped in the +`nemo_evaluator/templates/` venv directory is a **different prompt** (`swebench-instruction.md`, +hyphen) — it will run, but the result is not comparable to golden. Use the canonical one for +any scored or baseline-vs-quantized run, and keep it fixed across both sides of a comparison. ```bash VENV="${NEL_NEXT_VENV:-$HOME/.local/share/nel/venvs/nel-next}" # same default as nel-next.sh (NEL_NEXT_VENV may be unset) @@ -65,21 +76,46 @@ cluster: OpenHands runs ~200 turns/task. The canonical config adds a `system_message` interceptor (a large OpenHands system prompt — copy it verbatim from `bench.yaml`) -plus `turn_counter`. Full stack on `services..proxy.interceptors`: +plus `turn_counter`. + +**Order matters and differs from TB2.1.** Here `http_pairs_dump` is **first** (not last) +and `drop_params` comes **before** `consolidate_system`. This is the canonical order, +verified against a reference oci-hsg run: ```yaml proxy: request_timeout: 3600 extra_body: {skip_special_tokens: false} # add model-card sampling extras if the card sets them + model_traffic: {capture_request_body: true} # FEA-224; pair with the exclude_patterns entry interceptors: + - {name: http_pairs_dump, config: {dump_path: "$${NEL_OUTPUT_DIR}/http_pairs_metrics.json", first_n: 50}} - {name: system_message, config: {strategy: replace, system_message: ""}} - - {name: turn_counter, config: {max_turns: 200}} + - {name: turn_counter, config: {max_turns: 200, position: system_message}} + - {name: drop_params, config: {params: [max_tokens, max_completion_tokens, max_input_tokens_per_task, no_rebuild]}} - {name: consolidate_system} - - {name: drop_params, config: {params: [max_tokens, max_completion_tokens]}} - {name: reasoning} # reasoning models: normalize reasoning field … - - {name: reasoning_replay} # … and replay it across turns (drop both for instruct) + - {name: reasoning_replay} # … and replay across turns. Drop both for instruct models. ``` +**`reasoning_replay.mode` is per model, not per benchmark — take it from that model's own +golden leaf, never from another model's.** Known values: `think_tags` (Qwen-style), +`native` (GLM), and **omitted entirely** (MiniMax — the default is correct). Copying another +model's mode is a silent output-parsing bug, not a config nit. + +**Omitting `system_message` is a scoring change, not a simplification** — without it the +agent runs on the openhands-sdk default prompt instead of the canonical one, and results +are not comparable to golden or to other models evaluated with it. + +### Sharding + +`max_concurrent`/`sandbox.concurrency` are **per shard**, and each shard redeploys the model +on its own node — so `shards: N` multiplies serving capacity *and* live Fargate sandboxes +(`N × concurrency`). SWE-bench is the heaviest AA benchmark here: 500 tasks × `repeats: 5` += 2500 trials, and the reference run uses `shards: 10` (~250 trials/shard) at +`concurrency: 15`. Trials are partitioned and merged, so the score is unaffected — it is +purely a wall-clock lever. Check `N × concurrency` against the harbor Fargate quota and +`N × gpus_per_node` against your allocation before raising it. + ## Score Extraction Report **`pass@1`** only — benchmark `swebench-verified@1.0`, scorer `pass@1` (0–1): From 595cf48f259708b4b230f3718b76dbe8ddbaa5d8 Mon Sep 17 00:00:00 2001 From: Chenjie Luo Date: Thu, 6 Aug 2026 23:36:00 +0000 Subject: [PATCH 3/4] [skill] evaluation: trim nel-next TB2.1/SWE-bench notes to config values Follow-up to the two prior commits on this branch. They carried the reasoning behind each value (version history, task-set stability, sample counts) into the skill files. The skill's job is producing a correct config, and that rationale does not change any value it emits -- it belongs in the PR description, not in five files that will drift apart. Also drop the term "golden" throughout. It is internal shorthand for the reference eval-factory configs and means nothing to an external reader of this repo; the files now name the config or state the value directly. - nel-next.sh: 19-line install-source essay -> 5 lines. - references/nel-next.md: keep the value + the rejection rule for stale arch-suffixed tags; drop the version archaeology. - terminal_bench_2_1.md / swebench_verified.md: move settings into the existing task-values tables; keep only the non-obvious constraints (proxy timeout >= llm timeout, per-benchmark interceptor order, per-model reasoning_replay.mode, scoring impact of instruction_template and system_message). - swebench_verified.md: make the snippets use the canonical /configs/prompts/swebench_instruction.md consistently; the table and prose had been updated but the bash/yaml examples still showed the old hyphenated path. Net 128 -> 99 added lines. No config value changed. Signed-off-by: Chenjie Luo --- .agents/scripts/nel-next.sh | 24 ++------ .../recipes/examples/example_eval_next.yaml | 2 +- .../tasks/aa_next/swebench_verified.md | 57 ++++++++----------- .../tasks/aa_next/terminal_bench_2_1.md | 20 +++---- .../skills/evaluation/references/nel-next.md | 41 ++++++------- 5 files changed, 55 insertions(+), 89 deletions(-) diff --git a/.agents/scripts/nel-next.sh b/.agents/scripts/nel-next.sh index 748ad413f5b..06a46e59dfe 100755 --- a/.agents/scripts/nel-next.sh +++ b/.agents/scripts/nel-next.sh @@ -32,25 +32,11 @@ # using the config's export_config.mlflow (resolves ${MLFLOW_TRACKING_URI}, forces # emit_traces=false to avoid the per-sample hang). Run after `source .env`. # -# Install source (env overrides): NEL_NEXT_ORIGIN [+ NEL_NEXT_REF] selects a git build -# (the DEFAULT — see below); NEL_NEXT_SPEC forces a plain PyPI release instead. -# uv caches the resolved env and refreshes it when the spec changes. -# -# Default is the PUBLIC upstream repo (github.com/NVIDIA-NeMo/Evaluator — the `Repository` -# URL in nemo-evaluator's own pyproject). No auth, nothing internal. Its default branch is -# version 0.4.0 and ships the vendored `harbor_datasets/registry_overrides` (the TB 2.1 -# task-set pin), which is what the golden TB2.1/SWE-bench toolchain runs on. -# -# Why not plain PyPI: `nemo-evaluator` on PyPI tops out at **0.3.0**, so a version pin -# cannot reach 0.4.x. Set NEL_NEXT_SPEC (e.g. "nemo-evaluator[harbor,export]==0.3.*") to -# fall back to it deliberately — older toolchain, not golden-comparable. -# -# NEL_NEXT_REF: there is no v0.4.0 tag (public tags stop at v0.3.0); 0.4.0 is an untagged -# default-branch build. Leave REF empty to track the branch, or pin a commit SHA for -# reproducibility (9dcca2ae33362b4236ed4027f1516628d24c2cc2 was HEAD at 2026-08-04). -# -# To build from an internal mirror instead, set NEL_NEXT_ORIGIN in `.env` — internal URLs -# stay out of this repo, same convention as ${NEL_NEXT_EVAL_IMAGE}. +# Install source: NEL_NEXT_ORIGIN [+ NEL_NEXT_REF] git build (default), or NEL_NEXT_SPEC +# to force a PyPI release. Default branch of the upstream repo is 0.4.0 and ships the +# vendored TB 2.1 registry override; PyPI stops at 0.3.0, so a 0.4.x pin there resolves to +# nothing. No v0.4.0 tag exists — set NEL_NEXT_REF to a commit SHA to pin. Override +# NEL_NEXT_ORIGIN in `.env` to build from a mirror. set -euo pipefail # [harbor] = agentic/sandbox deps; [export] pulls mlflow for `mlflow-push`. diff --git a/.agents/skills/evaluation/recipes/examples/example_eval_next.yaml b/.agents/skills/evaluation/recipes/examples/example_eval_next.yaml index d11c001b8d9..4175323af14 100644 --- a/.agents/skills/evaluation/recipes/examples/example_eval_next.yaml +++ b/.agents/skills/evaluation/recipes/examples/example_eval_next.yaml @@ -75,7 +75,7 @@ cluster: account: ??? walltime: "04:00:00" # auto_resume chains across windows shards: 1 # N = N nodes (each redeploys vLLM) - eval_image: ${NEL_NEXT_EVAL_IMAGE} # from eval-config (golden pin 0.5.0.1-harbor, multi-arch; enroot creds per SKILL Step 7.5) + eval_image: ${NEL_NEXT_EVAL_IMAGE} # from eval-config (0.5.0.1-harbor, multi-arch; enroot creds per SKILL Step 7.5) sbatch_comment: '{"OccupiedIdleGPUsJobReaper":{"exemptIdleTimeMins":"480","reason":"benchmarking","description":"nel-next agentic eval"}}' sbatch_extra_flags: {switches: 1, exclusive: true} container_env: # AWS creds reach the eval container ONLY via here diff --git a/.agents/skills/evaluation/recipes/tasks/aa_next/swebench_verified.md b/.agents/skills/evaluation/recipes/tasks/aa_next/swebench_verified.md index 15730b5be5b..3e0ea6104f6 100644 --- a/.agents/skills/evaluation/recipes/tasks/aa_next/swebench_verified.md +++ b/.agents/skills/evaluation/recipes/tasks/aa_next/swebench_verified.md @@ -16,24 +16,24 @@ ECR/region. Start from `recipes/examples/example_eval_next.yaml`. | `playbook` | `swebench_verified` (`harbor://swebench-verified@1.0`) | | agent | `openhands-sdk` (playbook; `agent_kwargs: {max_iterations: 200, version: "1.17.0"}`) | | scope | 500 Python tasks × `repeats: 5` | -| `max_concurrent` / `sandbox.concurrency` | `15` in `bench.yaml`; **per-model leaves override it** (e.g. MiniMax-M2.7 uses `20`) — check the model's own golden leaf | +| `max_concurrent` / `sandbox.concurrency` | `15` in `bench.yaml`; per-model configs override it (MiniMax-M2.7 uses `20`) | | `solver` | `timeout_strategy: max`, `run_timeout: 10800` (3h), `agent_kwargs.llm_kwargs.timeout: 3600` | | `sandbox.region` | `us-east-2` | | `sandbox.ecr_repository` | `${HARBOR_SWEBENCH_ECR_REPOSITORY}` (dedicated `harbor-swebench` repo, **us-west-2**, regardless of sandbox region) | -| `cluster.eval_image` | `${NEL_NEXT_EVAL_IMAGE}` — golden pin **`0.5.0.1-harbor`** (same single source as TB2.1: `configs/shared/nel_next_containers.yaml`). `0.3.1.1-harbor` is the bare minimum (FEP-1085 reasoning fix) but is two minors behind | +| `cluster.eval_image` | `${NEL_NEXT_EVAL_IMAGE}` → **`0.5.0.1-harbor`** (same pin as TB2.1: `configs/shared/nel_next_containers.yaml`) | | `cluster.container_env.AWS_DEFAULT_REGION` | `us-east-2` (match `sandbox.region`) | -| `instruction_template` | `/configs/prompts/swebench_instruction.md`, **must be MOUNTED** — the harbor image doesn't bundle a built-in, and the template **content is scoring-relevant** (gotcha below) | +| `instruction_template` | `/configs/prompts/swebench_instruction.md`, **must be MOUNTED**; content is scoring-relevant (gotcha below) | | `proxy.request_timeout` | `3600` (FEP-1104 paired HTTP timeout; leaves mirror it on the service proxy) | | `drop_params` | `max_tokens`, `max_completion_tokens`, `max_input_tokens_per_task`, `no_rebuild` | | `output.export_config.mlflow.exclude_patterns` | `["shard*", "model_traffic.jsonl"]` | -| `system_message` | `strategy: replace` + the canonical OpenHands prompt — **scoring-relevant**, copy verbatim from `bench.yaml` | +| `system_message` | `strategy: replace` + the OpenHands prompt from `bench.yaml` (verbatim) — scoring-relevant | ```yaml benchmarks: - playbook: swebench_verified repeats: 5 max_concurrent: 15 - instruction_template: /configs/swebench-instruction.md # mounted (see gotcha) + instruction_template: /configs/prompts/swebench_instruction.md # mounted (see gotcha) solver: service: timeout_strategy: max # canonical; "task" = leaderboard-comparable @@ -52,24 +52,21 @@ The playbook defaults `instruction_template: swebench-instruction.md`, but the harbor image doesn't ship that built-in → run dies at finalize with `FileNotFoundError: instruction_template not found`. So it must be mounted. -**Which file you mount changes the score.** The canonical runs mount the internal compeval -prompt (`swebench_instruction.md` — note the underscore) at -`/configs/prompts/swebench_instruction.md`; take it from the canonical `bench.yaml` / -reference run dir, not from this repo. The public built-in shipped in the -`nemo_evaluator/templates/` venv directory is a **different prompt** (`swebench-instruction.md`, -hyphen) — it will run, but the result is not comparable to golden. Use the canonical one for -any scored or baseline-vs-quantized run, and keep it fixed across both sides of a comparison. +**Which file you mount changes the score.** Mount the canonical `swebench_instruction.md` +(underscore) at `/configs/prompts/swebench_instruction.md`, taken from the reference config or +run dir. The built-in in the `nemo_evaluator/templates/` venv directory is a **different +prompt** (`swebench-instruction.md`, hyphen) — it runs, but results are not comparable. Keep +whichever you use fixed across both sides of a comparison. ```bash -VENV="${NEL_NEXT_VENV:-$HOME/.local/share/nel/venvs/nel-next}" # same default as nel-next.sh (NEL_NEXT_VENV may be unset) -cp "$VENV/lib/python3.12/site-packages/nemo_evaluator/templates/swebench-instruction.md" /tmp/ -ssh 'mkdir -p //prompts' && scp /tmp/swebench-instruction.md ://prompts/ +ssh 'mkdir -p //prompts' +scp swebench_instruction.md ://prompts/ # canonical file; verify sha256 against the source ``` ```yaml -benchmarks: [{playbook: swebench_verified, instruction_template: /configs/swebench-instruction.md}] +benchmarks: [{playbook: swebench_verified, instruction_template: /configs/prompts/swebench_instruction.md}] cluster: - container_mounts: ["//prompts/swebench-instruction.md:/configs/swebench-instruction.md:ro"] + container_mounts: ["//prompts/swebench_instruction.md:/configs/prompts/swebench_instruction.md:ro"] ``` ### Deployment proxy (multi-turn agentic) @@ -78,9 +75,8 @@ OpenHands runs ~200 turns/task. The canonical config adds a `system_message` interceptor (a large OpenHands system prompt — copy it verbatim from `bench.yaml`) plus `turn_counter`. -**Order matters and differs from TB2.1.** Here `http_pairs_dump` is **first** (not last) -and `drop_params` comes **before** `consolidate_system`. This is the canonical order, -verified against a reference oci-hsg run: +**Order differs from TB2.1**: `http_pairs_dump` is **first** (not last) and `drop_params` +comes **before** `consolidate_system`. ```yaml proxy: @@ -97,24 +93,19 @@ proxy: - {name: reasoning_replay} # … and replay across turns. Drop both for instruct models. ``` -**`reasoning_replay.mode` is per model, not per benchmark — take it from that model's own -golden leaf, never from another model's.** Known values: `think_tags` (Qwen-style), -`native` (GLM), and **omitted entirely** (MiniMax — the default is correct). Copying another -model's mode is a silent output-parsing bug, not a config nit. +**`reasoning_replay.mode` is per model, not per benchmark.** `think_tags` (Qwen-style), +`native` (GLM), omitted (MiniMax). Copying another model's mode is a silent output-parsing bug. -**Omitting `system_message` is a scoring change, not a simplification** — without it the -agent runs on the openhands-sdk default prompt instead of the canonical one, and results -are not comparable to golden or to other models evaluated with it. +**Omitting `system_message` is a scoring change**: without it the agent runs the +openhands-sdk default prompt instead of the canonical one. ### Sharding `max_concurrent`/`sandbox.concurrency` are **per shard**, and each shard redeploys the model -on its own node — so `shards: N` multiplies serving capacity *and* live Fargate sandboxes -(`N × concurrency`). SWE-bench is the heaviest AA benchmark here: 500 tasks × `repeats: 5` -= 2500 trials, and the reference run uses `shards: 10` (~250 trials/shard) at -`concurrency: 15`. Trials are partitioned and merged, so the score is unaffected — it is -purely a wall-clock lever. Check `N × concurrency` against the harbor Fargate quota and -`N × gpus_per_node` against your allocation before raising it. +on its own node — `shards: N` multiplies serving capacity *and* live Fargate sandboxes +(`N × concurrency`). 500 tasks × `repeats: 5` = 2500 trials; `shards: 10` at `concurrency: 15` +suits it. Score is unaffected — purely a wall-clock lever. Check `N × concurrency` against the +Fargate quota and `N × gpus_per_node` against your allocation. ## Score Extraction diff --git a/.agents/skills/evaluation/recipes/tasks/aa_next/terminal_bench_2_1.md b/.agents/skills/evaluation/recipes/tasks/aa_next/terminal_bench_2_1.md index c67503550c4..ffe3e17d11d 100644 --- a/.agents/skills/evaluation/recipes/tasks/aa_next/terminal_bench_2_1.md +++ b/.agents/skills/evaluation/recipes/tasks/aa_next/terminal_bench_2_1.md @@ -24,15 +24,12 @@ pinned via a vendored registry override in nemo-evaluator-next. | `cluster.container_env.AWS_DEFAULT_REGION` | match `sandbox.region` | | `max_concurrent` / `sandbox.concurrency` | `50` (canonical bench.yaml) | | timeout_strategy | `max` (canonical bench.yaml) + `agent_kwargs.llm_kwargs.timeout: 3600`; use `task` for leaderboard-comparable | -| `cluster.eval_image` requirement | **`0.5.0.1-harbor`** — the golden pin (`${NEL_NEXT_EVAL_IMAGE}`, multi-arch). `0.3.1.1-harbor` is the bare minimum that runs TB 2.1 but is two minors behind | +| `cluster.eval_image` | **`0.5.0.1-harbor`** (`${NEL_NEXT_EVAL_IMAGE}`, multi-arch) | | `proxy.request_timeout` | `3600` — must be **≥** `agent_kwargs.llm_kwargs.timeout` | | `drop_params` | `max_tokens`, `max_completion_tokens`, `max_input_tokens_per_task`, `no_rebuild` | | `output.export_config.mlflow.exclude_patterns` | `["shard*", "model_traffic.jsonl"]` | - -TB 2.1's task set is pinned by a vendored registry override in the eval image, but that -override (`harbor_datasets/registry_overrides/terminal_bench_2_1.json`) has not changed since -2026-06-03 — **89 tasks on both 0.3.1.1 and 0.5.0.1**. So the image bump is a toolchain fix and -does not move the benchmark; scores stay comparable across it. +| `http_pairs_dump` | **last** in the interceptor chain | +| scope | 89 tasks × `repeats: 8` | These values mirror the canonical TB2.1 config — re-check it before a scored run: `configs/benchmarks/terminal-bench-2.1/bench.yaml` (+ `manifest.yaml`) in @@ -62,12 +59,11 @@ benchmarks: `cluster.eval_image: ${NEL_NEXT_EVAL_IMAGE}` (`0.5.0.1-harbor`) and the AWS creds come from `modelopttools:eval-config` (run it first) + the workspace `.env`. -**Sharding.** `max_concurrent`/`sandbox.concurrency` are **per shard**, and each shard gets -its own vLLM instance on its own node — so `shards: N` multiplies both serving capacity and -live Fargate sandboxes (`N × concurrency`). Golden runs 89 tasks × r8 = 712 trials on -`shards: 4` (~178 trials/shard). Purely a wall-clock/throughput lever: trials are partitioned -and merged, so the score is unaffected. Before raising it, check `N × concurrency` against the -harbor Fargate quota and `N × gpus_per_node` against your allocation. +**Sharding.** `max_concurrent`/`sandbox.concurrency` are **per shard**, and each shard runs +its own vLLM on its own node — `shards: N` multiplies both serving capacity and live Fargate +sandboxes (`N × concurrency`). Trials are partitioned and merged, so the score is unaffected; +it is purely a wall-clock lever. `shards: 4` suits 89 × r8 = 712 trials. Check +`N × concurrency` against the Fargate quota and `N × gpus_per_node` against your allocation. ## Score Extraction diff --git a/.agents/skills/evaluation/references/nel-next.md b/.agents/skills/evaluation/references/nel-next.md index 21cd063e7a8..9c05bb30041 100644 --- a/.agents/skills/evaluation/references/nel-next.md +++ b/.agents/skills/evaluation/references/nel-next.md @@ -27,13 +27,10 @@ Installing 0.3.x into the 0.2.6 env clobbers `nel`, so it lives in its own venv: .agents/scripts/nel-next.sh eval run --dry-run | --submit | … ``` -Default install is a git build from the **public upstream repo** -(`github.com/NVIDIA-NeMo/Evaluator`, no auth) via `NEL_NEXT_ORIGIN` — its default branch is -**0.4.0** and ships the vendored TB 2.1 registry override, matching the golden toolchain. -Plain PyPI `nemo-evaluator` tops out at **0.3.0**, so a version pin can't reach 0.4.x; set -`NEL_NEXT_SPEC` only to deliberately fall back to it (older, not golden-comparable). There is -no `v0.4.0` tag — pin `NEL_NEXT_REF` to a commit SHA for reproducibility. To build from an -internal mirror, set `NEL_NEXT_ORIGIN` in `.env`; internal URLs stay out of this repo. +Default install is a git build from `github.com/NVIDIA-NeMo/Evaluator` via `NEL_NEXT_ORIGIN` +(default branch → **0.4.0**). PyPI `nemo-evaluator` stops at **0.3.0**, so a version pin +can't reach 0.4.x. No `v0.4.0` tag exists — pin `NEL_NEXT_REF` to a commit SHA for +reproducibility. Set `NEL_NEXT_ORIGIN` in `.env` to build from a mirror. ## Credentials + internal infra (`.env`) @@ -136,24 +133,20 @@ with its own `run_id`, copying the shared `services:` block. ## Rules & gotchas -- **`eval_image`** = `${NEL_NEXT_EVAL_IMAGE}`. Golden **pins `0.5.0.1-harbor`** (single - source of truth: `configs/shared/nel_next_containers.yaml` in the eval-factory repo) — - track that pin, don't treat it as a floor. `0.3.1.1-harbor` is the bare minimum that can - run **TB 2.1** at all but is two minors behind golden; older `0.17.x/0.18.x-harbor-` - are arch-suffixed. The TB2.1 task set itself is stable across these versions (the - vendored `registry_overrides/terminal_bench_2_1.json` has not changed since 2026-06-03, - and both 0.3.1.1 and 0.5.0.1 score 89 samples), so the bump is a toolchain fix, not a - benchmark change. Private gitlab-master image → cluster needs enroot creds (SKILL Step 7.5). -- **`proxy.request_timeout` must be >= `agent_kwargs.llm_kwargs.timeout`** (both 3600 in - canonical TB2.1). A smaller proxy timeout silently truncates long agent turns. -- **`drop_params` takes four params** for harbor agentic benchmarks: `max_tokens`, - `max_completion_tokens`, `max_input_tokens_per_task`, `no_rebuild`. The last two are sent - by the 0.5.x eval image; vLLM 400s on them if they aren't stripped. +- **`eval_image`** = `${NEL_NEXT_EVAL_IMAGE}` → `0.5.0.1-harbor` (multi-arch). Re-check + against `configs/shared/nel_next_containers.yaml` in the eval-factory repo, which is the + pin and does move. Arch-suffixed `0.17.x/0.18.x-harbor-` are too old for TB 2.1. + Private gitlab-master image → cluster needs enroot creds (SKILL Step 7.5). +- **`proxy.request_timeout` must be >= `agent_kwargs.llm_kwargs.timeout`** (both 3600). A + smaller proxy timeout silently truncates long agent turns. +- **`drop_params`** for harbor agentic benchmarks: `max_tokens`, `max_completion_tokens`, + `max_input_tokens_per_task`, `no_rebuild`. The last two are sent by the 0.5.x eval image; + vLLM 400s on them if they aren't stripped. - **`exclude_patterns`** = `["shard*", "model_traffic.jsonl"]` — captured request bodies - (FEA-224) stay in the run dir and must never be exported to MLflow. -- **`http_pairs_dump`** (FEP-1104/1120) goes **last** in the interceptor chain: - `config: {dump_path: $${NEL_OUTPUT_DIR}/http_pairs_metrics.json, first_n: 50}`. Note the - `$$` — it defers expansion to run time. Diagnostics only; does not affect scoring. + stay in the run dir, never MLflow. +- **`http_pairs_dump`** — `config: {dump_path: $${NEL_OUTPUT_DIR}/http_pairs_metrics.json, + first_n: 50}`. The `$$` defers expansion to run time. Chain position is per benchmark + (last for TB2.1, first for SWE-bench). Diagnostics only. - **Mount sources must pre-exist** — pyxis won't create the host side of a bind mount (invisible to `--dry-run`, fails at canary). `ssh 'mkdir -p //.cache/{vllm,huggingface}'`. From a6ddca20e6918b871210c0c210cc10787b61475d Mon Sep 17 00:00:00 2001 From: Zhiyu Cheng Date: Sun, 9 Aug 2026 23:45:22 -0700 Subject: [PATCH 4/4] [skill] evaluation: pin the nel-next install, scope http_pairs_dump to canaries Two review findings on the nel-next toolchain change: 1. NEL_NEXT_REF defaulted to empty, so the install resolved to whatever NVIDIA-NeMo/Evaluator main pointed at. Baseline and candidate runs submitted days apart (or from a cold uv cache) would install different harness commits, folding a harness change into the pass@1 delta that drives the ACCEPT/REGRESSION call, with no recorded SHA to detect it. Default NEL_NEXT_REF to the current main SHA (0.4.0, verified to install); NEL_NEXT_REF=main still tracks HEAD for dev/canary work. --version now also prints the resolved spec so it can be recorded with scored results. 2. http_pairs_dump was enabled in the shared template and both recipes. first_n caps only successful pairs -- the keep rule is (total_seen <= first_n) or (status != 200) -- so every error pair is held in memory with full request and response bodies for the life of the run, and the whole list is re-serialized on each write. On the very failures it diagnoses (vLLM 400s, sustained 429/5xx) a long agentic run grows the proxy without bound. Comment it out of the template and the SWE-bench chain, and document it as canary/diagnostic-only. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Zhiyu Cheng --- .agents/scripts/nel-next.sh | 18 +++++++++------- .../recipes/examples/example_eval_next.yaml | 7 +++++-- .../tasks/aa_next/swebench_verified.md | 6 ++++-- .../tasks/aa_next/terminal_bench_2_1.md | 2 +- .../skills/evaluation/references/nel-next.md | 21 +++++++++++++------ 5 files changed, 36 insertions(+), 18 deletions(-) diff --git a/.agents/scripts/nel-next.sh b/.agents/scripts/nel-next.sh index 06a46e59dfe..8b8f1130b9a 100755 --- a/.agents/scripts/nel-next.sh +++ b/.agents/scripts/nel-next.sh @@ -32,17 +32,21 @@ # using the config's export_config.mlflow (resolves ${MLFLOW_TRACKING_URI}, forces # emit_traces=false to avoid the per-sample hang). Run after `source .env`. # -# Install source: NEL_NEXT_ORIGIN [+ NEL_NEXT_REF] git build (default), or NEL_NEXT_SPEC -# to force a PyPI release. Default branch of the upstream repo is 0.4.0 and ships the -# vendored TB 2.1 registry override; PyPI stops at 0.3.0, so a 0.4.x pin there resolves to -# nothing. No v0.4.0 tag exists — set NEL_NEXT_REF to a commit SHA to pin. Override -# NEL_NEXT_ORIGIN in `.env` to build from a mirror. +# Install source: NEL_NEXT_ORIGIN + NEL_NEXT_REF git build (default), or NEL_NEXT_SPEC +# to force a PyPI release. Upstream `main` is 0.4.0 and ships the vendored TB 2.1 registry +# override; PyPI stops at 0.3.0, so a 0.4.x pin there resolves to nothing. No v0.4.0 tag +# exists, so NEL_NEXT_REF defaults to a commit SHA — an unpinned branch HEAD would install +# a different harness for the baseline and the candidate run, folding a harness change into +# the pass@1 delta. NEL_NEXT_REF=main tracks HEAD (dev/canary only); bump the default below +# after re-validating a canary. Override NEL_NEXT_ORIGIN in `.env` to build from a mirror. +# `--version` prints the resolved spec — record it alongside scored results. set -euo pipefail # [harbor] = agentic/sandbox deps; [export] pulls mlflow for `mlflow-push`. NEL_NEXT_SPEC="${NEL_NEXT_SPEC:-}" NEL_NEXT_ORIGIN="${NEL_NEXT_ORIGIN:-git+https://github.com/NVIDIA-NeMo/Evaluator.git}" -NEL_NEXT_REF="${NEL_NEXT_REF:-}" +# Reproducibility pin — NVIDIA-NeMo/Evaluator main @ 2026-08-04 (nemo-evaluator 0.4.0). +NEL_NEXT_REF="${NEL_NEXT_REF:-4d081325170aababd0c8f27c58bed31a81ce82ac}" # NEL_NEXT_SPEC wins when explicitly set (PyPI escape hatch); otherwise use the git origin. if [[ -n "$NEL_NEXT_SPEC" ]]; then @@ -130,7 +134,7 @@ command -v uvx >/dev/null 2>&1 || { echo "ERROR: 'uvx' not found (curl -LsSf htt case "${1:-}" in --setup-only) _uvx nel --version >/dev/null 2>&1 && _log "nel-next ready — ${INSTALL_SPEC}"; exit 0 ;; --which) echo "uvx --python 3.12 --from \"${INSTALL_SPEC}\" nel"; exit 0 ;; - --version) _uvx python -c 'import nemo_evaluator; print(nemo_evaluator.__version__)'; exit 0 ;; + --version) _uvx python -c 'import nemo_evaluator; print(nemo_evaluator.__version__)'; _log "from ${INSTALL_SPEC}"; exit 0 ;; mlflow-push) _mlflow_push "${@:2}"; exit $? ;; "") echo "ERROR: no args. Try: nel-next.sh eval run [--dry-run] (or --help)" >&2; exit 2 ;; esac diff --git a/.agents/skills/evaluation/recipes/examples/example_eval_next.yaml b/.agents/skills/evaluation/recipes/examples/example_eval_next.yaml index 4175323af14..4aeb4390859 100644 --- a/.agents/skills/evaluation/recipes/examples/example_eval_next.yaml +++ b/.agents/skills/evaluation/recipes/examples/example_eval_next.yaml @@ -50,8 +50,11 @@ services: # last two are sent by the 0.5.x harbor eval image; vLLM 400s on them unless stripped config: {params: [max_tokens, max_completion_tokens, max_input_tokens_per_task, no_rebuild]} # SWE-bench (OpenHands, multi-turn) adds turn_counter + consolidate_system + a system_message — see swebench_verified.md - - name: http_pairs_dump # FEP-1104/1120 diagnostics; canonical LAST in the chain - config: {dump_path: "$${NEL_OUTPUT_DIR}/http_pairs_metrics.json", first_n: 50} # $$ defers expansion to run time + # FEP-1104/1120 diagnostics — uncomment for a CANARY/debug run, drop it for the scored run: + # first_n caps only 200s, so every error pair (full req+res bodies) is retained in memory for + # the whole run and re-serialized on each write — unbounded growth exactly when the server errors. + # - name: http_pairs_dump # canonical LAST in the chain (SWE-bench: first) + # config: {dump_path: "$${NEL_OUTPUT_DIR}/http_pairs_metrics.json", first_n: 50} # $$ defers expansion to run time node_pool: gpu benchmarks: diff --git a/.agents/skills/evaluation/recipes/tasks/aa_next/swebench_verified.md b/.agents/skills/evaluation/recipes/tasks/aa_next/swebench_verified.md index 3e0ea6104f6..399ebceef5e 100644 --- a/.agents/skills/evaluation/recipes/tasks/aa_next/swebench_verified.md +++ b/.agents/skills/evaluation/recipes/tasks/aa_next/swebench_verified.md @@ -76,7 +76,9 @@ interceptor (a large OpenHands system prompt — copy it verbatim from `bench.ya plus `turn_counter`. **Order differs from TB2.1**: `http_pairs_dump` is **first** (not last) and `drop_params` -comes **before** `consolidate_system`. +comes **before** `consolidate_system`. `http_pairs_dump` is canary/diagnostic-only — it +retains every error pair in memory for the whole run (`references/nel-next.md`); drop it +from the scored config. ```yaml proxy: @@ -84,7 +86,7 @@ proxy: extra_body: {skip_special_tokens: false} # add model-card sampling extras if the card sets them model_traffic: {capture_request_body: true} # FEA-224; pair with the exclude_patterns entry interceptors: - - {name: http_pairs_dump, config: {dump_path: "$${NEL_OUTPUT_DIR}/http_pairs_metrics.json", first_n: 50}} + # - {name: http_pairs_dump, config: {dump_path: "$${NEL_OUTPUT_DIR}/http_pairs_metrics.json", first_n: 50}} # canary only - {name: system_message, config: {strategy: replace, system_message: ""}} - {name: turn_counter, config: {max_turns: 200, position: system_message}} - {name: drop_params, config: {params: [max_tokens, max_completion_tokens, max_input_tokens_per_task, no_rebuild]}} diff --git a/.agents/skills/evaluation/recipes/tasks/aa_next/terminal_bench_2_1.md b/.agents/skills/evaluation/recipes/tasks/aa_next/terminal_bench_2_1.md index ffe3e17d11d..114203cf5ed 100644 --- a/.agents/skills/evaluation/recipes/tasks/aa_next/terminal_bench_2_1.md +++ b/.agents/skills/evaluation/recipes/tasks/aa_next/terminal_bench_2_1.md @@ -28,7 +28,7 @@ pinned via a vendored registry override in nemo-evaluator-next. | `proxy.request_timeout` | `3600` — must be **≥** `agent_kwargs.llm_kwargs.timeout` | | `drop_params` | `max_tokens`, `max_completion_tokens`, `max_input_tokens_per_task`, `no_rebuild` | | `output.export_config.mlflow.exclude_patterns` | `["shard*", "model_traffic.jsonl"]` | -| `http_pairs_dump` | **last** in the interceptor chain | +| `http_pairs_dump` | **last** in the interceptor chain — canary/diagnostic only, drop it for a scored run (unbounded error-pair retention) | | scope | 89 tasks × `repeats: 8` | These values mirror the canonical TB2.1 config — re-check it before a scored run: diff --git a/.agents/skills/evaluation/references/nel-next.md b/.agents/skills/evaluation/references/nel-next.md index 9c05bb30041..59efcfa7c68 100644 --- a/.agents/skills/evaluation/references/nel-next.md +++ b/.agents/skills/evaluation/references/nel-next.md @@ -28,9 +28,13 @@ Installing 0.3.x into the 0.2.6 env clobbers `nel`, so it lives in its own venv: ``` Default install is a git build from `github.com/NVIDIA-NeMo/Evaluator` via `NEL_NEXT_ORIGIN` -(default branch → **0.4.0**). PyPI `nemo-evaluator` stops at **0.3.0**, so a version pin -can't reach 0.4.x. No `v0.4.0` tag exists — pin `NEL_NEXT_REF` to a commit SHA for -reproducibility. Set `NEL_NEXT_ORIGIN` in `.env` to build from a mirror. +(`main` → **0.4.0**). PyPI `nemo-evaluator` stops at **0.3.0**, so a version pin can't reach +0.4.x. No `v0.4.0` tag exists, so `NEL_NEXT_REF` **defaults to a pinned commit SHA** in +`nel-next.sh` — a floating branch HEAD installs a different harness for the baseline and the +candidate, so the pass@1 delta no longer isolates the model. `NEL_NEXT_REF=main` tracks HEAD +for dev/canary work only; to move the pin, canary against the new SHA and bump the default in +`nel-next.sh`. Record `nel-next.sh --version` (prints version + resolved spec) with scored +results. Set `NEL_NEXT_ORIGIN` in `.env` to build from a mirror. ## Credentials + internal infra (`.env`) @@ -144,9 +148,14 @@ with its own `run_id`, copying the shared `services:` block. vLLM 400s on them if they aren't stripped. - **`exclude_patterns`** = `["shard*", "model_traffic.jsonl"]` — captured request bodies stay in the run dir, never MLflow. -- **`http_pairs_dump`** — `config: {dump_path: $${NEL_OUTPUT_DIR}/http_pairs_metrics.json, - first_n: 50}`. The `$$` defers expansion to run time. Chain position is per benchmark - (last for TB2.1, first for SWE-bench). Diagnostics only. +- **`http_pairs_dump` — canary/diagnostic runs only, leave it OUT of scored runs.** + `config: {dump_path: "$${NEL_OUTPUT_DIR}/http_pairs_metrics.json", first_n: 50}` (the `$$` + defers expansion to run time). `first_n` caps only the *successful* pairs: the keep rule is + `(total_seen <= first_n) or (status != 200)`, so **every** non-200 pair is held in memory + for the life of the run, full request + response bodies, and the whole list is re-serialized + on each write. A long agentic run that is 400ing or rate-limiting (the failure this dumps + diagnose) grows the proxy without bound — exactly the run you can least afford to lose. + Chain position is per benchmark (last for TB2.1, first for SWE-bench). - **Mount sources must pre-exist** — pyxis won't create the host side of a bind mount (invisible to `--dry-run`, fails at canary). `ssh 'mkdir -p //.cache/{vllm,huggingface}'`.