[skill] evaluation: align nel-next TB2.1/SWE-bench with golden toolchain - #2063
[skill] evaluation: align nel-next TB2.1/SWE-bench with golden toolchain#2063cjluo-nv wants to merge 3 commits into
Conversation
Terminal-Bench 2.1 configs generated from this skill drifted from the canonical eval-factory config (configs/benchmarks/terminal-bench-2.1/ bench.yaml). The scoring contract (playbook, repeats=8, timeout_strategy, run_timeout, concurrency) already matched; the toolchain and several proxy-level defaults did not. - nel-next.sh: install from the public upstream repo (github.com/NVIDIA-NeMo/Evaluator, default branch -> 0.4.0) instead of PyPI. PyPI nemo-evaluator tops out at 0.3.0 and cannot reach the 0.4.x toolchain golden runs on. NEL_NEXT_SPEC becomes the PyPI escape hatch and now takes precedence when explicitly set; NEL_NEXT_ORIGIN stays overridable from .env so internal mirrors stay out of this repo. - eval_image: document the golden pin 0.5.0.1-harbor (single source of truth: configs/shared/nel_next_containers.yaml) rather than 0.3.1.1-harbor as a floor. The TB2.1 task set is unchanged across those versions -- the vendored registry override has not moved since 2026-06-03 and both score 89 samples -- so this is a toolchain fix, not a benchmark change. - proxy.request_timeout 1800 -> 3600: it must be >= the solver's llm_kwargs.timeout (3600), otherwise the proxy truncates long agent turns the harness is still waiting on. - drop_params: add max_input_tokens_per_task and no_rebuild, which the 0.5.x harbor eval image sends and vLLM rejects unless stripped. - exclude_patterns: add model_traffic.jsonl so captured request bodies stay in the run dir and never reach MLflow. - add the http_pairs_dump interceptor (last in chain) for HTTP diagnostics. - document sharding: max_concurrent/sandbox.concurrency are per shard, so shards:N multiplies both serving capacity and live sandboxes. - .gitignore: broaden .env/.env-* to .env* so secret backups such as .env.bak-tb21 cannot be staged. Verified: nel-next.sh --version reports 0.4.0 built from Evaluator.git@9dcca2ae; both TB2.1 run configs pass `eval run --dry-run` on that CLI with 0.5.0.1-harbor in the generated sbatch. Signed-off-by: Chenjie Luo <chenjiel@nvidia.com>
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
📝 WalkthroughWalkthroughThe PR changes ChangesAA-next evaluation configuration
Environment file ignore rules
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 6✅ Passed checks (6 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2063 +/- ##
==========================================
- Coverage 67.00% 66.28% -0.73%
==========================================
Files 520 522 +2
Lines 59545 62009 +2464
==========================================
+ Hits 39900 41101 +1201
- Misses 19645 20908 +1263
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Same golden cross-check as the TB2.1 change, against the canonical configs/benchmarks/swe-bench-verified/bench.yaml and a reference oci-hsg run (Qwen3.6-35B-A3B, pass@1 0.7012 over 500 tasks x r5). - fix the stale source-of-truth path (configs/benchmarks/nel_next/ swebench_verified/ -> configs/benchmarks/swe-bench-verified/) and point at the shared eval-image pin. - correct the interceptor ORDER: SWE-bench differs from TB2.1 -- http_pairs_dump is FIRST, and drop_params comes BEFORE consolidate_system. - turn_counter gains position: system_message. - drop_params: add max_input_tokens_per_task and no_rebuild. - exclude_patterns: add model_traffic.jsonl; document proxy.model_traffic.capture_request_body. - document that the instruction_template CONTENT is scoring-relevant, and that the public built-in in nemo_evaluator/templates/ is a DIFFERENT prompt whose results are not comparable to the canonical one. - state plainly that omitting the system_message interceptor is a scoring change, not a simplification. - reasoning_replay.mode is per MODEL, not per benchmark: think_tags (Qwen), native (GLM), omitted (MiniMax). Taking it from another model's config is a silent output-parsing bug. - max_concurrent: bench.yaml sets 15, but per-model leaves override it. - document sharding: concurrency is per shard; the reference run uses shards:10 for 500 x r5 = 2500 trials. Signed-off-by: Chenjie Luo <chenjiel@nvidia.com>
69c56cf to
821ed92
Compare
Follow-up to the two prior commits on this branch. They carried the reasoning behind each value (version history, task-set stability, sample counts) into the skill files. The skill's job is producing a correct config, and that rationale does not change any value it emits -- it belongs in the PR description, not in five files that will drift apart. Also drop the term "golden" throughout. It is internal shorthand for the reference eval-factory configs and means nothing to an external reader of this repo; the files now name the config or state the value directly. - nel-next.sh: 19-line install-source essay -> 5 lines. - references/nel-next.md: keep the value + the rejection rule for stale arch-suffixed tags; drop the version archaeology. - terminal_bench_2_1.md / swebench_verified.md: move settings into the existing task-values tables; keep only the non-obvious constraints (proxy timeout >= llm timeout, per-benchmark interceptor order, per-model reasoning_replay.mode, scoring impact of instruction_template and system_message). - swebench_verified.md: make the snippets use the canonical /configs/prompts/swebench_instruction.md consistently; the table and prose had been updated but the bash/yaml examples still showed the old hyphenated path. Net 128 -> 99 added lines. No config value changed. Signed-off-by: Chenjie Luo <chenjiel@nvidia.com>
Validation: end-to-end eval runs on the aligned configsConfigs generated from these skill changes were run end-to-end on aws-cmh (4x GB300
Both This exercises the changes in this PR specifically: the 0.5.0.1-harbor eval image, the Note: the checklist item "Did you get Claude approval on this PR?" is still open -- |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| NEL_NEXT_SPEC="${NEL_NEXT_SPEC:-nemo-evaluator[harbor,export]==0.3.*}" | ||
| NEL_NEXT_ORIGIN="${NEL_NEXT_ORIGIN:-}" | ||
| NEL_NEXT_SPEC="${NEL_NEXT_SPEC:-}" | ||
| NEL_NEXT_ORIGIN="${NEL_NEXT_ORIGIN:-git+https://github.com/NVIDIA-NeMo/Evaluator.git}" |
There was a problem hiding this comment.
[IMPORTANT Compatibility] The new default install is an unpinned git branch, which is weaker pinning than what it replaces and undercuts this PR's own goal.
Before: NEL_NEXT_SPEC defaulted to nemo-evaluator[harbor,export]==0.3.* — a version-constrained resolve that a user could tighten to an exact 0.3.x.
After: NEL_NEXT_ORIGIN defaults to git+https://github.com/NVIDIA-NeMo/Evaluator.git with NEL_NEXT_REF empty, so INSTALL_SPEC resolves to whatever the default branch HEAD happens to be.
Why it matters for this specific script:
INSTALL_SPECis now a constant string across upstream commits. The header's premise — "uv caches the resolved env and refreshes it when the spec changes" — no longer holds: the spec never changes while HEAD moves. uv caches the resolved git commit, so machine A (first run in June) and machine B (first run in August) silently run different toolchains under an identical spec, and neither run needs--refreshto notice.--versioncannot disambiguate. The PR body notes there is nov0.4.0tag, sonemo_evaluator.__version__prints0.4.0for every commit on the branch. There is no way for a user (or an agent following the skill) to answer "which build produced this score."- The validated toolchain is a specific SHA. The PR's own testing section cites
Evaluator.git@9dcca2aeas the build that matches golden, and the header tells the user to "setNEL_NEXT_REFto a commit SHA to pin" — but the shipped default does not do that, so the default path is the unreproducible one.
For a script whose entire purpose is producing benchmark numbers that are comparable across runs and across the BF16/NVFP4 sides of a comparison, the default should be the reproducible build.
Suggested fix — default NEL_NEXT_REF to the validated SHA, keeping both override paths intact:
NEL_NEXT_SPEC="${NEL_NEXT_SPEC:-}"
NEL_NEXT_ORIGIN="${NEL_NEXT_ORIGIN:-git+https://github.com/NVIDIA-NeMo/Evaluator.git}"
# Pinned to the build golden runs on (0.4.0, no tag exists). Set to a branch name
# (e.g. main) to track upstream, at the cost of reproducibility.
NEL_NEXT_REF="${NEL_NEXT_REF:-9dcca2ae}"and update the header + references/nel-next.md to state the default is pinned rather than instructing the user to pin it themselves. If tracking HEAD is genuinely intended, then --version / --which should surface the resolved commit so a run is at least attributable after the fact.
Secondary, smaller point in the same hunk: precedence between the two variables inverted (NEL_NEXT_ORIGIN used to win when both were set; now NEL_NEXT_SPEC does). That is intentional per the comment, but anyone with both already in their .env flips from a git build to a PyPI build with no signal — worth a line in references/nel-next.md.
| vLLM 400s on them if they aren't stripped. | ||
| - **`exclude_patterns`** = `["shard*", "model_traffic.jsonl"]` — captured request bodies | ||
| stay in the run dir, never MLflow. | ||
| - **`http_pairs_dump`** — `config: {dump_path: $${NEL_OUTPUT_DIR}/http_pairs_metrics.json, |
There was a problem hiding this comment.
[SUGGESTION] This snippet is not valid YAML if copied verbatim. In a flow mapping ({...}), {, }, [, ], and , are indicator characters and cannot appear in a plain (unquoted) scalar — so dump_path: $${NEL_OUTPUT_DIR}/http_pairs_metrics.json inside the braces is a parse error, not just a style nit.
recipes/examples/example_eval_next.yaml:54 and swebench_verified.md:87 both quote it correctly; only this reference line doesn't. Since these skill docs are copied into generated configs, the unquoted form would fail at config load rather than at --dry-run schema validation.
| - **`http_pairs_dump`** — `config: {dump_path: $${NEL_OUTPUT_DIR}/http_pairs_metrics.json, | |
| - **`http_pairs_dump`** — `config: {dump_path: "$${NEL_OUTPUT_DIR}/http_pairs_metrics.json", |
| # SWE-bench (OpenHands, multi-turn) adds turn_counter + consolidate_system + a system_message — see swebench_verified.md | ||
| - name: http_pairs_dump # FEP-1104/1120 diagnostics; canonical LAST in the chain | ||
| config: {dump_path: "$${NEL_OUTPUT_DIR}/http_pairs_metrics.json", first_n: 50} # $$ defers expansion to run time |
There was a problem hiding this comment.
[SUGGESTION] The comment says http_pairs_dump is "canonical LAST in the chain", which is right for TB2.1 (the benchmark this template's benchmarks: block is configured for), but swebench_verified.md:78 documents that SWE-bench needs it first and drop_params before consolidate_system. This template is the stated starting point for both recipes ("Start from recipes/examples/example_eval_next.yaml"), and the adjacent SWE-bench pointer on line 52 only mentions the added interceptors, not that the ordering changes.
Given the PR body calls "per-benchmark interceptor ordering" one of the things this change gets right, making the template say so avoids someone lifting the TB2.1 order into a SWE-bench config. Suggest extending the existing pointer comment:
# SWE-bench (OpenHands, multi-turn) adds turn_counter + consolidate_system + a
# system_message, AND reorders this chain (http_pairs_dump first, drop_params before
# consolidate_system) — see swebench_verified.md
| # Environments | ||
| .env | ||
| .env-* | ||
| .env* |
There was a problem hiding this comment.
[SUGGESTION] .env* is broader than the stated goal ("secret backups such as .env.bak-tb21 cannot be staged") and will also swallow any future .env.example / .env.template at the repo root — the exact files you want tracked. The skill already relies on a committed template (.agents/skills/evaluation/recipes/env.example, referenced from SKILL.md:84 and four task recipes); that path is safe today only because it lacks the leading dot.
Cheap insurance, since a silently-untracked template is a confusing failure mode:
.env*
!.env.example
|
|
||
| **Which file you mount changes the score.** Mount the canonical `swebench_instruction.md` | ||
| (underscore) at `/configs/prompts/swebench_instruction.md`, taken from the reference config or | ||
| run dir. The built-in in the `nemo_evaluator/templates/` venv directory is a **different | ||
| prompt** (`swebench-instruction.md`, hyphen) — it runs, but results are not comparable. Keep | ||
| whichever you use fixed across both sides of a comparison. | ||
|
|
||
| ```bash | ||
| VENV="${NEL_NEXT_VENV:-$HOME/.local/share/nel/venvs/nel-next}" # same default as nel-next.sh (NEL_NEXT_VENV may be unset) | ||
| cp "$VENV/lib/python3.12/site-packages/nemo_evaluator/templates/swebench-instruction.md" /tmp/ | ||
| ssh <login> 'mkdir -p <lustre>/<user>/prompts' && scp /tmp/swebench-instruction.md <login>:<lustre>/<user>/prompts/ | ||
| ssh <login> 'mkdir -p <lustre>/<user>/prompts' |
There was a problem hiding this comment.
[SUGGESTION] The rewritten gotcha loses the one thing that made the old version actionable: where to get the file.
The previous text gave a concrete cp from the venv ($VENV/lib/python3.12/site-packages/nemo_evaluator/templates/swebench-instruction.md). The new text correctly explains that the built-in is a different prompt and not comparable — good, that's a real scoring trap — but then the scp on line 64 sources a bare swebench_instruction.md from the caller's cwd, with the provenance only as "taken from the reference config or run dir." An agent following this recipe has no path to fetch, and the instruction "verify sha256 against the source" names no source to verify against.
The two other source-of-truth pointers in this file are precise (configs/benchmarks/swe-bench-verified/bench.yaml, configs/shared/nel_next_containers.yaml), so this one stands out. Suggest naming the repo-relative path the same way, e.g.:
# canonical prompt: configs/prompts/swebench_instruction.md in nvidia-eval-factory-benchmarking
# (dl/JoC/competitive_evaluation/…) — record its sha256 alongside the score
ssh <login> 'mkdir -p <lustre>/<user>/prompts'
scp swebench_instruction.md <login>:<lustre>/<user>/prompts/Also worth stating explicitly that both sides of a BF16-vs-quantized comparison must mount the same file — the current "Keep whichever you use fixed across both sides of a comparison" implies it, but this is the highest-leverage sentence in the section.
| | `max_concurrent` / `sandbox.concurrency` | `15` | | ||
| | `max_concurrent` / `sandbox.concurrency` | `15` in `bench.yaml`; per-model configs override it (MiniMax-M2.7 uses `20`) | | ||
| | `solver` | `timeout_strategy: max`, `run_timeout: 10800` (3h), `agent_kwargs.llm_kwargs.timeout: 3600` | | ||
| | `sandbox.region` | `us-east-2` | |
There was a problem hiding this comment.
[SUGGESTION] Two small consistency issues in this table row and the block below it.
-
The row now says
15inbench.yamlbut "per-model configs override it (MiniMax-M2.7 uses20)", while the YAML at line 33 still hardcodesmax_concurrent: 15with no note. Sincesandbox.concurrencymust trackmax_concurrent(the TB2.1 example flagskeep == sandbox.concurrency), a reader who bumps one per the table may not bump the other. A trailing comment on line 33 would close it. -
The new Sharding section recommends
shards: 10atconcurrency: 15→ 150 live Fargate sandboxes, whereas TB2.1's parallel section recommendsshards: 4atconcurrency: 50→ 200. Both then say "checkN × concurrencyagainst the Fargate quota" without giving the quota. Naming the actual limit once (inreferences/nel-next.md, since it's account-level rather than per-benchmark) would make both checks performable instead of advisory.
|
|
||
| Default install is public PyPI `nemo-evaluator[harbor]==0.3.*`; set | ||
| `NEL_NEXT_ORIGIN`/`NEL_NEXT_REF` for the internal git build (see script header). | ||
| Default install is a git build from `github.com/NVIDIA-NeMo/Evaluator` via `NEL_NEXT_ORIGIN` |
There was a problem hiding this comment.
[SUGGESTION] This paragraph now correctly says the default resolves to 0.4.0, but the version labels elsewhere still say 0.3.x and were not updated — so the doc contradicts itself a few lines apart:
references/nel-next.md:14— comparison table:package | `nemo-evaluator-launcher` 0.2.6 | `nemo-evaluator[harbor]` 0.3.xreferences/nel-next.md:23— "Installing 0.3.x into the 0.2.6 env clobbersnel"SKILL.md:39— "(nemo-evaluator[harbor]0.3.x)"scripts/nel-next.sh:17,20,23,55— header and_uvxcomment all say "nel 0.3.x" / "0.3.x environment"
The 0.2.6-vs-next separation argument holds regardless of which minor the "next" side is, so nothing is functionally wrong — but "0.3.x" is the value a reader will quote when sanity-checking --version output against the docs, and it now prints 0.4.0. Since the point of this PR is removing exactly this kind of drift, worth relabeling these to 0.3.x/0.4.x (or just "next") in the same pass.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Warning
CodeRabbit couldn't request changes on this pull request because it doesn't have sufficient GitHub permissions.
Please grant CodeRabbit Pull requests: Read and write permission and re-run the review.
Actionable comments posted: 2
🧹 Nitpick comments (2)
.agents/skills/evaluation/references/nel-next.md (1)
30-33: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick winAlign the
nel-nextversion contract. The installer uses the upstream default branch whenNEL_NEXT_REFis unset, while the reference and skill files still describe a 0.3.x environment. Pin the installer to the documented 0.3.x release, or update the CLI, schema, and templates to 0.4.0.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.agents/skills/evaluation/references/nel-next.md around lines 30 - 33, Align the nel-next version contract across the installer and its documentation: either pin the installer’s default NEL_NEXT_REF to the documented 0.3.x release, or consistently update the CLI, schema, templates, and reference/skill files to 0.4.0. Ensure the default branch is not used implicitly when the documented version is expected..agents/scripts/nel-next.sh (1)
43-51: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winPreserve the required extras in
NEL_NEXT_SPEC.If
NEL_NEXT_SPEComits[export],mlflow-pushlacks themlflowdependency. Document the requirednemo-evaluator[harbor,export]syntax and reject incomplete overrides, or install the exporter separately.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.agents/scripts/nel-next.sh around lines 43 - 51, Update the NEL_NEXT_SPEC override handling in the installation setup to ensure it includes the required nemo-evaluator[harbor,export] extras, rejecting incomplete overrides with a clear message or installing the exporter dependencies separately. Document the required override syntax while preserving valid custom specifications.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.agents/skills/evaluation/recipes/tasks/aa_next/swebench_verified.md:
- Around line 92-93: Update the reasoning interceptor examples in the model
configuration section so they are model-specific: retain both reasoning and
reasoning_replay for reasoning models, use reasoning_replay.mode "think_tags"
for Qwen-style models, "native" for GLM models, and omit reasoning_replay for
MiniMax; ensure instruct-model examples include neither interceptor.
- Around line 66-70: Update the NEL schema reference in references/nel-next.md
to document cluster.container_mounts under the cluster configuration, noting
that it mounts files into the evaluator container for NEL SLURM jobs. Do not add
a service-level mount or alter the recipe configuration.
---
Nitpick comments:
In @.agents/scripts/nel-next.sh:
- Around line 43-51: Update the NEL_NEXT_SPEC override handling in the
installation setup to ensure it includes the required
nemo-evaluator[harbor,export] extras, rejecting incomplete overrides with a
clear message or installing the exporter dependencies separately. Document the
required override syntax while preserving valid custom specifications.
In @.agents/skills/evaluation/references/nel-next.md:
- Around line 30-33: Align the nel-next version contract across the installer
and its documentation: either pin the installer’s default NEL_NEXT_REF to the
documented 0.3.x release, or consistently update the CLI, schema, templates, and
reference/skill files to 0.4.0. Ensure the default branch is not used implicitly
when the documented version is expected.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 84c9459f-a486-4872-8a1b-e40ebcd7a48d
📒 Files selected for processing (6)
.agents/scripts/nel-next.sh.agents/skills/evaluation/recipes/examples/example_eval_next.yaml.agents/skills/evaluation/recipes/tasks/aa_next/swebench_verified.md.agents/skills/evaluation/recipes/tasks/aa_next/terminal_bench_2_1.md.agents/skills/evaluation/references/nel-next.md.gitignore
| ```yaml | ||
| benchmarks: [{playbook: swebench_verified, instruction_template: /configs/swebench-instruction.md}] | ||
| benchmarks: [{playbook: swebench_verified, instruction_template: /configs/prompts/swebench_instruction.md}] | ||
| cluster: | ||
| container_mounts: ["<lustre>/<user>/prompts/swebench-instruction.md:/configs/swebench-instruction.md:ro"] | ||
| container_mounts: ["<lustre>/<user>/prompts/swebench_instruction.md:/configs/prompts/swebench_instruction.md:ro"] | ||
| ``` |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n -C 8 \
'container_mounts|^services:|^cluster:' \
.agents/skills/evaluation/references/nel-next.md \
.agents/skills/evaluation/recipes/examples/example_eval_next.yaml \
.agents/skills/evaluation/recipes/tasks/aa_next/swebench_verified.mdRepository: NVIDIA/Model-Optimizer
Length of output: 13010
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- relevant files ---'
git ls-files '.agents/skills/evaluation' | sed -n '1,160p'
printf '%s\n' '--- mount and eval-image references ---'
rg -n -C 4 \
'container_mounts|eval_image|instruction_template|eval.image|mount' \
.agents/skills/evaluation \
--glob '*.md' --glob '*.yaml' --glob '*.yml' --glob '*.py' --glob '*.sh' \
| sed -n '1,260p'
printf '%s\n' '--- recipe context ---'
cat -n .agents/skills/evaluation/recipes/tasks/aa_next/swebench_verified.md | sed -n '45,90p'Repository: NVIDIA/Model-Optimizer
Length of output: 34115
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- NEL schema reference ---'
cat -n .agents/skills/evaluation/references/nel-next.md | sed -n '58,125p'
printf '%s\n' '--- all repository references to cluster.container_mounts and evaluation mounts ---'
rg -n -C 5 \
'cluster\.container_mounts|container_mounts|mounts:|eval_image|evaluation:|eval-image|eval image' \
. \
--glob '!*.lock' --glob '!*.jsonl' \
| sed -n '1,360p'
printf '%s\n' '--- possible schema/parser sources ---'
rg -n -C 3 \
'extra="forbid"|extra.*forbid|class .*Config|container_mounts|eval_image' \
. \
--glob '*.py' --glob '*.pyi' --glob '*.yaml' --glob '*.yml' --glob '*.toml' \
| sed -n '1,360p'Repository: NVIDIA/Model-Optimizer
Length of output: 50380
🌐 Web query:
NEL next configuration schema eval_image container_mounts evaluation container mounts nemo evaluator
💡 Result:
In the NVIDIA NeMo Evaluator (including configurations used with NeMo Evaluator Launcher/NEL), the eval_image and container_mounts parameters are primary configuration options used when deploying evaluation jobs to containerized environments, such as SLURM clusters or Docker-based executors [1][2]. 1. eval_image: This parameter specifies the container image (e.g., Apptainer, Enroot, or Docker) used to execute the evaluation harness [1][2]. It provides the isolated environment where the evaluation code runs [3]. In some advanced workflows, this can be dynamically set via environment variables (e.g., ${NEL_NEXT_EVAL_IMAGE}) to ensure the correct architecture-matched or version-pinned image is used [4]. 2. container_mounts: This parameter accepts a list of host paths to mount into the evaluation container [1][2]. It is essential for providing the containerized environment access to required data on the host machine, such as: - Datasets (if they are not automatically managed by the task's dataset directory mounting feature) [5][6] - Caching directories (e.g., for vLLM or HuggingFace) [4] - Configuration files [4] - Results output directories [7] Usage Contexts: - SLURM Deployment: Both parameters are typically defined under the cluster configuration block (e.g., cluster.eval_image and cluster.container_mounts) [1][2]. - NeMo Run: When using NeMo Run, these are set as properties on the executor object (e.g., executor.container_image and executor.container_mounts) [8]. - Dataset-Specific Mounting: Beyond general container mounts, tasks that require specific local datasets can also use the dataset_dir and optional dataset_mount_path fields within the evaluation task configuration [5][6]. The system automatically handles these mounts and sets the corresponding NEMO_EVALUATOR_DATASET_DIR environment variable [5][6]. For further details on specific schema structures, users should refer to the official documentation for the respective version of the NeMo Evaluator SDK, as configurations may vary between stable releases and development branches [5][9][10].
Citations:
- 1: https://docs.nvidia.com/nemo/evaluator/deployment/slurm
- 2: https://docs.nvidia.com/nemo/evaluator/nightly/deployment/slurm
- 3: https://docs.nvidia.com/nemo/evaluator/architecture
- 4: a05850b
- 5: https://docs.nvidia.com/nemo/evaluator/0.2.5/libraries/nemo-evaluator-launcher/configuration/evaluation/index.html
- 6: https://docs.nvidia.com/nemo/evaluator/0.2.5/deployment/launcher-orchestrated/local.html
- 7: https://docs.nvidia.com/nemo/evaluator/0.2.5/get-started/quickstart/container.html
- 8: https://docs.nvidia.com/nemo/evaluator/latest/tutorials/nemo-fw/nemo-run.html
- 9: https://docs.nvidia.com/nemo/microservices/latest/evaluate/evaluation-configs/schema.html
- 10: https://docs.nvidia.com/nemo/evaluator/latest/libraries/nemo-evaluator-launcher/configuration/executors/slurm.html
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- official SLURM documentation ---'
for url in \
'https://docs.nvidia.com/nemo/evaluator/deployment/slurm' \
'https://docs.nvidia.com/nemo/evaluator/nightly/deployment/slurm' \
'https://docs.nvidia.com/nemo/evaluator/latest/libraries/nemo-evaluator-launcher/configuration/executors/slurm.html'
do
printf '\nURL: %s\n' "$url"
curl -L --fail --silent --show-error "$url" |
tr '\n' ' ' |
sed 's/<[^>]*>/ /g' |
grep -oE '.{0,180}(container_mounts|eval_image|eval image|mounts).{0,260}' |
head -n 12 || true
done
printf '%s\n' '--- referenced repository commit ---'
if git cat-file -e a05850bffad6f254f265d7e30b46823a6064d8f7^{commit} 2>/dev/null; then
git show --stat --oneline a05850bffad6f254f265d7e30b46823a6064d8f7
git show a05850bffad6f254f265d7e30b46823a6064d8f7 -- \
'.agents/skills/evaluation/references/nel-next.md' \
'.agents/skills/evaluation/recipes/tasks/aa_next/swebench_verified.md' \
| sed -n '1,260p'
else
echo 'commit not present locally'
fiRepository: NVIDIA/Model-Optimizer
Length of output: 18435
Document cluster.container_mounts in the NEL schema reference.
cluster.container_mounts is supported for NEL SLURM jobs and mounts files into the evaluator container. The recipe does not need a service-level mount. Update references/nel-next.md to list this field under cluster and clarify its target.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.agents/skills/evaluation/recipes/tasks/aa_next/swebench_verified.md around
lines 66 - 70, Update the NEL schema reference in references/nel-next.md to
document cluster.container_mounts under the cluster configuration, noting that
it mounts files into the evaluator container for NEL SLURM jobs. Do not add a
service-level mount or alter the recipe configuration.
| - {name: reasoning} # reasoning models: normalize reasoning field … | ||
| - {name: reasoning_replay} # … and replay it across turns (drop both for instruct) | ||
| - {name: reasoning_replay} # … and replay across turns. Drop both for instruct models. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -eu
file='.agents/skills/evaluation/recipes/tasks/aa_next/swebench_verified.md'
printf '%s\n' '--- target excerpt ---'
sed -n '70,115p' "$file"
printf '%s\n' '--- related reasoning terms ---'
rg -n -C 3 'reasoning_replay|think_tags|MiniMax|Qwen|GLM|instruct|native' .agents/skills/evaluation/recipes/tasks/aa_next .agents/skills/evaluation 2>/dev/null | head -n 240Repository: NVIDIA/Model-Optimizer
Length of output: 33100
🏁 Script executed:
#!/bin/bash
set -eu
printf '%s\n' '--- interceptor definitions and usage ---'
rg -n -C 4 'reasoning_replay|name:\s*reasoning|think_tags|reasoning.*mode|native.*GLM' . --glob '!*.md' --glob '!*.lock' --glob '!package-lock.json' 2>/dev/null | head -n 320
printf '%s\n' '--- all likely proxy/config files ---'
git ls-files | rg '(^|/)(bench\.yaml|.*(proxy|interceptor|nel.*config|config).*\.(ya?ml|json|py|ts|js|md))$' | head -n 240Repository: NVIDIA/Model-Optimizer
Length of output: 18764
🌐 Web query:
NeMo Evaluator Launcher reasoning_replay interceptor mode think_tags native MiniMax
💡 Result:
In the NVIDIA NeMo Evaluator, reasoning_replay is a specialized interceptor designed to manage chain-of-thought (CoT) reasoning across multi-turn agentic interactions [1][2]. It addresses the issue where certain agent SDKs drop reasoning_content during replay, which can lead to a loss of CoT context and increased turn counts in agentic rollouts [1][3]. The reasoning_replay interceptor caches reasoning_content from upstream model responses (keyed by tool call ID or content hash) and re-injects it into subsequent outbound requests [1]. Regarding the requested modes and features: 1. Interceptor Mode (Native): The reasoning_replay interceptor supports a native mode [3]. In this mode, it ensures compatibility with backends (such as vLLM versions >=0.19) that may require specific reasoning field representations [3]. Recent updates ensure that in native (and both) modes, the interceptor preserves and writes both reasoning and reasoning_content fields to accommodate differences in how various backends parse chat input [3]. 2. Think Tags: The interceptor integrates with the configured reasoning pipeline to wrap cached hidden chains of thought with configured think tokens (e.g., ...) [3][4]. While NeMo Evaluator generally allows customization of these tokens (e.g., in reasoning interceptor configuration) [4], the reasoning_replay interceptor uses these settings to maintain consistent reasoning formatting during re-injection [3]. 3. MiniMax: There is no direct mention of native MiniMax integration specifically within the NeMo Evaluator reasoning_replay interceptor documentation or commit history [1][3]. NeMo Evaluator interceptors are modular and typically built to be backend-agnostic by using standard OpenAI-compatible interfaces [4][5], meaning they would process reasoning content based on standard fields (like reasoning_content) regardless of the specific underlying model provider, provided the provider follows that protocol. In summary, the reasoning_replay interceptor is a tool to ensure continuity of reasoning in long-running agentic evaluations, with native modes specifically tuned to handle backend-specific field requirements (like vLLM) and standard think-tag wrapping [1][3].
Citations:
- 1: NVIDIA-NeMo/Evaluator@751d1f7
- 2: NVIDIA-NeMo/Evaluator@fce9256
- 3: NVIDIA-NeMo/Evaluator@e4cc20c
- 4: https://docs.nvidia.com/nemo/evaluator/latest/about/concepts/interceptors.html
- 5: https://docs.nvidia.com/nemo/evaluator/latest/libraries/nemo-evaluator/interceptors/index.html
🌐 Web query:
site:github.com/NVIDIA/NeMo-Evaluator-Launcher reasoning_replay think_tags
💡 Result:
I couldn’t find an indexed match for reasoning_replay or think_tags in NVIDIA/NeMo-Evaluator-Launcher.
The relevant repository appears to be NVIDIA-NeMo/Evaluator, whose launcher package is located under packages/nemo-evaluator-launcher/. (github.com)
Note: GitHub’s current repository path is likely:
NVIDIA-NeMo/Evaluator
Citations:
🏁 Script executed:
#!/bin/bash
set -eu
for sha in 751d1f7cc2b078ac90a161e49db361da487d741a fce9256690e99c5905e74ba03fea482f2e9f9a75 e4cc20c94e4dd9dbdcd3c5a2d05b878bbfc22907; do
printf '\n--- commit %s ---\n' "$sha"
curl -fsSL "https://api.github.com/repos/NVIDIA-NeMo/Evaluator/commits/$sha" |
jq -r '.commit.message, (.files[]? | "FILE: \(.filename)\n\(.patch // "")")' |
rg -n -C 5 'reasoning_replay|think_tags|native|MiniMax|mode|reasoning_content' | head -n 240 || true
done
printf '\n--- current upstream interceptor docs ---\n'
curl -fsSL 'https://raw.githubusercontent.com/NVIDIA-NeMo/Evaluator/main/docs/about/concepts/interceptors.md' |
rg -n -C 5 'reasoning_replay|think_tags|native|MiniMax|reasoning_content' | head -n 240 || trueRepository: NVIDIA/Model-Optimizer
Length of output: 570
🌐 Web query:
site:docs.nvidia.com/nemo/evaluator reasoning_replay mode think_tags
💡 Result:
In the context of LLM evaluation frameworks, specifically the NVIDIA NeMo Evaluator SDK, the concepts of reasoning traces and think tags are utilized to process and analyze the step-by-step thinking processes of reasoning-capable models [1][2]. Think tags (e.g., and ) are special tokens used by many modern LLMs to encapsulate their chain-of-thought reasoning process—the intermediate logical steps, calculations, and deliberations performed before arriving at a final answer [1][2]. Reasoning replay or reasoning interception is a functional component within the NeMo Evaluator that manages these traces [2][3]. Because many benchmarks and evaluation pipelines expect only the final answer from a model, the Reasoning Interceptor serves several key roles: - Extraction and Removal: It identifies the reasoning trace within the model's output (using the specified start and end tokens), removes it from the content to ensure the evaluation receives only the final answer, and optionally isolates it for further analysis [1][2][4]. - Statistics and Metrics Tracking: The interceptor tracks metrics related to the reasoning process, such as the number of responses that included reasoning, whether the reasoning completed (i.e., reached the end token), and the average length of the reasoning trace (in tokens or words) [2][4][5]. This data is typically saved to an evaluation metrics file (e.g., eval_factory_metrics.json), allowing developers to analyze model behavior, compute costs, or evaluate the quality of the chain-of-thought [1][4][5]. While "reasoning_replay" is not a singular standard term, it appears in this technical ecosystem as part of the operational workflow for handling, logging, and re-evaluating the reasoning traces generated during model inference [2][6]. Configuration of these interceptors allows users to specify custom tokens if their model uses different markers than the standard tags [2][7].
Citations:
- 1: https://docs.nvidia.com/nemo/evaluator/nightly/evaluation/run-evals/reasoning.html
- 2: https://docs.nvidia.com/nemo/evaluator/nightly/libraries/nemo-evaluator/interceptors/reasoning.html
- 3: https://docs.nvidia.com/nemo/evaluator/0.2.5/deployment/adapters/configuration.html
- 4: https://docs.nvidia.com/nemo/evaluator/0.2.5/tutorials/how-to/reasoning.html
- 5: https://docs.nvidia.com/nemo/evaluator/latest/tutorials/how-to/reasoning.html
- 6: https://docs.nvidia.com/nemo/evaluator/0.2.5/about/concepts/interceptors.html
- 7: https://docs.nvidia.com/nemo/evaluator/0.2.5/_modules/nemo_evaluator/adapters/interceptors/reasoning_interceptor.html
🌐 Web query:
site:github.com/NVIDIA-NeMo/Evaluator "reasoning_replay" "think_tags"
💡 Result:
In the context of the NVIDIA NeMo Evaluator framework, reasoning_replay and think_tags are components designed to manage and preserve Chain-of-Thought (CoT) reasoning processes across multi-turn agent interactions [1][2]. reasoning_replay is an interceptor designed to solve the problem where certain agent SDKs (such as OpenHands or pi-ai) drop reasoning_content during conversation replays, causing the model to lose its reasoning context in subsequent turns [2]. The interceptor functions by caching the reasoning_content from upstream model responses and automatically re-injecting it into the next outbound request [3][2]. This ensures that the model's intermediate "thinking" remains available throughout the session [2]. think_tags refer to the configuration used to wrap this cached reasoning content [3]. Because different model backends and APIs may use different fields or formats for reasoning (e.g., specific tags or normalized fields like reasoning_content), the system uses think_tags to standardize how the hidden chain of thought is presented when re-injected [3]. The interceptor can be configured to operate in different modes (such as native or both), ensuring compatibility across various backends like vLLM and SGLang by reading and writing the appropriate reasoning fields [3].
Citations:
- 1: https://github.com/NVIDIA-NeMo/Evaluator/blob/main/README.md
- 2: NVIDIA-NeMo/Evaluator@751d1f7
- 3: NVIDIA-NeMo/Evaluator@e4cc20c
🌐 Web query:
site:github.com/NVIDIA-NeMo/Evaluator "reasoning_replay" "MiniMax"
💡 Result:
The term reasoning_replay refers to a specific interceptor component within the NVIDIA NeMo Evaluator framework, not a standalone model or feature from MiniMax [1][2]. It was developed to solve the issue where certain agent SDKs (such as OpenHands or older implementations) would drop an LLM's reasoning content—its Chain-of-Thought (CoT)—during multi-turn interactions [1][2]. When agentic workflows replay previous turns, the loss of this reasoning can cause models to lose context, leading to redundant or inefficient rollouts [2]. Key technical aspects of the reasoning_replay interceptor include: Caching and Re-injection: It intercepts model responses to cache reasoning content (typically identified by fields like reasoning_content or reasoning) and re-injects that content into subsequent outbound requests [1][2]. Cross-Backend Compatibility: The interceptor manages differences in how various inference backends (such as vLLM versions or SGLang) represent reasoning fields [2]. For example, it ensures compatibility with vLLM ≥0.19 by writing both reasoning and reasoning_content fields, as different backends rely on different keys [2]. Deterministic Operation: It is designed to compose deterministically with other framework components, such as reasoning normalizers and caching interceptors, ensuring the CoT is preserved consistently across long agentic benchmark runs [1]. While NeMo Evaluator is used to evaluate various models, including those with reasoning capabilities, reasoning_replay is a utility within the evaluation infrastructure designed to maintain data integrity during testing, rather than a feature of any specific model provider like MiniMax [1][3][4].
Citations:
- 1: NVIDIA-NeMo/Evaluator@751d1f7
- 2: NVIDIA-NeMo/Evaluator@e4cc20c
- 3: https://github.com/NVIDIA-NeMo/Evaluator/blob/main/README.md
- 4: https://github.com/NVIDIA-NeMo/Evaluator
Make the reasoning interceptor configuration model-specific.
Reasoning models require reasoning and reasoning_replay; instruct models require neither. Set reasoning_replay.mode per model: think_tags for Qwen-style models, native for GLM models, and omit it for MiniMax. Update the examples at lines 92–93 and 96–100.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.agents/skills/evaluation/recipes/tasks/aa_next/swebench_verified.md around
lines 92 - 93, Update the reasoning interceptor examples in the model
configuration section so they are model-specific: retain both reasoning and
reasoning_replay for reasoning models, use reasoning_replay.mode "think_tags"
for Qwen-style models, "native" for GLM models, and omit reasoning_replay for
MiniMax; ensure instruct-model examples include neither interceptor.
There was a problem hiding this comment.
Claude review — [skill] evaluation: align nel-next TB2.1/SWE-bench with golden toolchain
Scope applied: full review per the standard procedure. The triggering comment supplied end-to-end validation evidence but no scoping instruction, so nothing was narrowed. All 6 changed files reviewed (99 additions / 43 deletions) — no modelopt/ source is touched, so the mode-registration, config-schema, and export-compatibility categories don't apply here; the review focused on the agent-skill contract: are the documented values self-consistent, reproducible, and copy-pasteable.
Note on the diff: a .agents/skills/deployment/references/trtllm.md hunk shows up in the two-dot diff against origin/main. That is main-side drift (the checkout is shallow, so no merge base is available), not part of this PR's file list — I excluded it.
Findings: CRITICAL: 0 · IMPORTANT: 1 · SUGGESTION: 5
The one blocking item
nel-next.sh:44 — the new default install is an unpinned git branch. This is the only finding I'd hold the PR for, because it works against the PR's own thesis.
The change is correct in its diagnosis: PyPI tops out at 0.3.0 and cannot reach the 0.4.x toolchain, so a git install is the right move. But NEL_NEXT_REF defaults to empty, so INSTALL_SPEC resolves to whatever the default branch HEAD happens to be. Three consequences specific to this script:
INSTALL_SPECis now a constant string while HEAD moves, which breaks the header's own stated invariant that "uv caches the resolved env and refreshes it when the spec changes." Two machines first running this in different months get different toolchains under an identical spec, and neither needs--refreshto notice.--versioncan't disambiguate them — the PR body confirms nov0.4.0tag exists, so every commit on the branch prints0.4.0. A score becomes unattributable to a build.- The PR's own testing section names
Evaluator.git@9dcca2aeas the build matching golden, and the new header tells the user to "setNEL_NEXT_REFto a commit SHA to pin" — but the shipped default doesn't, so the default path is the unreproducible one.
Fix is one line (NEL_NEXT_REF="${NEL_NEXT_REF:-9dcca2ae}"), details inline. If tracking HEAD is deliberate, then --version/--which should surface the resolved commit so runs stay attributable after the fact.
Also flagged there: the precedence between NEL_NEXT_SPEC and NEL_NEXT_ORIGIN inverted. That's intentional per the new comment, but anyone with both already in .env silently flips from a git build to a PyPI build — worth a line in the reference doc.
Suggestions (non-blocking)
references/nel-next.md:147— thehttp_pairs_dumpsnippet isn't valid YAML as written: unquoted$${NEL_OUTPUT_DIR}/…inside a flow mapping terminates the plain scalar at{. The two other copies of this snippet (example_eval_next.yaml:54,swebench_verified.md:87) quote it correctly; only the reference doesn't. Since these get copied into generated configs, it'd fail at config load rather than at--dry-run.references/nel-next.md:30— the doc now says the default resolves to 0.4.0, but the version labels four lines up (:14,:23), inSKILL.md:39, and in the script header (:17,20,23,55) still say "0.3.x". Nothing is functionally wrong, but that's the value a reader quotes when checking--versionoutput — and removing exactly this kind of drift is the point of the PR.swebench_verified.md:54-62— the rewritten instruction-template gotcha adds the genuinely valuable warning (the venv built-in is a different prompt and not comparable) but drops the concrete path to fetch the canonical file, so thescpnow sources a bare filename from cwd and "verify sha256 against the source" names no source. The other two source-of-truth pointers in the same file are precise; this one should be too.example_eval_next.yaml:52-54— the template is the documented starting point for both recipes and sayshttp_pairs_dumpis "canonical LAST", which is TB2.1-only; the adjacent SWE-bench pointer mentions the added interceptors but not that the ordering itself changes..gitignore:32—.env*is broader than the stated goal and would also swallow a future root.env.example; a!.env.examplenegation is cheap insurance. (Today's committed template,.agents/skills/evaluation/recipes/env.example, is safe only because it lacks the leading dot.)swebench_verified.md:21— themax_concurrentrow now documents a per-model override (MiniMax uses 20) but the YAML below still hardcodes 15 with no note, andsandbox.concurrencyhas to track it. Separately, the two new sharding sections recommend fan-outs that differ by 25% in live sandboxes (150 vs 200) while both say "check against the Fargate quota" without naming it.
Risk assessment
Low. Docs and one shell script; no library code, no public API, no modelopt_state schema, no export path — the checkpoint/config backward-compatibility concerns in CONTRIBUTING.md don't come into play, and the PR's own N/A answers on tests and CHANGELOG are appropriate.
The substantive value-check items are well supported. I confirmed the internal consistency of the aligned values that carry scoring weight: proxy.request_timeout: 3600 >= llm_kwargs.timeout: 3600 holds in the example, both task recipes, and the reference; the four-param drop_params list is identical across all four sites; exclude_patterns is consistent across all five; and the model_traffic: {capture_request_body: true} addition is correctly paired with its exclude_patterns entry, which is the pairing that keeps captured request bodies out of MLflow. The end-to-end evidence in the triggering comment (TB2.1 delta 0.0000, SWE-bench +0.40%, overlapping CIs on full task sets) is the right validation for a change of this kind and covers the scoring-relevant surface — the interceptor ordering, sampling preset, and instruction template — better than any static review could.
The remaining reproducibility gap is the unpinned ref: that eval run pinned a specific build implicitly, but the config as shipped doesn't let the next person reproduce it. Worth closing before merge, since the whole purpose here is scores that stay comparable across runs and across both sides of a quantization comparison.
1 participant
What does this PR do?
Type of change: Documentation / tooling (agent skill)
Terminal-Bench 2.1 configs generated from the
evaluationskill had drifted from thecanonical eval-factory config (
configs/benchmarks/terminal-bench-2.1/bench.yaml). Thescoring contract already matched golden exactly — playbook,
repeats: 8,timeout_strategy: max,run_timeout: 7200,llm_kwargs.timeout: 3600, concurrency. Whathad drifted was the toolchain and a few proxy-level defaults.
The drift was in the skill, not in individual configs: a config generated fresh from the
skill reproduced every stale value, so patching configs alone would not have held.
nel-next.shinstalls from the public upstream repo(
github.com/NVIDIA-NeMo/Evaluator, default branch →0.4.0) instead of PyPI. PyPInemo-evaluatortops out at0.3.0and cannot reach the 0.4.x toolchain golden runs on.NEL_NEXT_SPECbecomes the PyPI escape hatch and now takes precedence when explicitly set;NEL_NEXT_ORIGINstays overridable from.envso internal mirrors stay out of this repo.eval_image: document the golden pin0.5.0.1-harbor(single source of truth:configs/shared/nel_next_containers.yaml) rather than0.3.1.1-harboras a floor.proxy.request_timeout1800 → 3600 — must be>=the solver'sllm_kwargs.timeout,else the proxy truncates long agent turns the harness is still awaiting.
drop_params: addmax_input_tokens_per_task,no_rebuild— sent by the 0.5.x harboreval image; vLLM returns 400 unless stripped.
exclude_patterns: addmodel_traffic.jsonlso captured request bodies stay in the rundir and never reach MLflow.
http_pairs_dumpinterceptor (last in chain) for HTTP diagnostics.max_concurrent/sandbox.concurrencyare per shard, soshards: Nmultiplies both serving capacity and live sandboxes (N x concurrency)..gitignore: broaden.env/.env-*to.env*so secret backups such as.env.bak-tb21cannot be staged.This does not move the benchmark. The TB2.1 task set is pinned by a vendored registry
override that has not changed since 2026-06-03, and both
0.3.1.1-harborand0.5.0.1-harborscore 89 samples — so the image bump is a toolchain fix and scores staycomparable across it.
Usage
Testing
nel-next.sh --version→0.4.0, built fromEvaluator.git@9dcca2ae(matches golden'stoolchain; golden
manifest.yamlevidence SHAs resolve in the public repo).eval run --dry-runon that CLI with no schema errors — theseschemas are
extra="forbid", sohttp_pairs_dumpand the newdrop_paramswould hard-failif unsupported.
0.5.0.1-harborconfirmed in the generatednel_eval.sbatchfor both.pre-commit run --files <changed>— all hooks pass, no file modifications.dl/JoC/competitive_evaluation/nvidia-eval-factory-benchmarking@
mainand a reference oci-hsg run (Qwen3.6-35B-A3B TB2.1,pass@10.4438 over 89x8).Not yet run: a live 2-problem canary on the new toolchain.
--dry-rundoes not exerciseenroot credentials for the new image, the harbor agent handshake, or request-shape acceptance.
Before your PR is "Ready for review"
NEL_NEXT_SPECrestores the previous PyPI install.CONTRIBUTING.md: N/A--dry-run+ pre-commit.Additional Information
Personal run configs under
.agents/skills/evaluation/runs/are deliberately not included:they carry internal cluster hostnames, lustre paths, account names and an AWS account id, which
do not belong in this public repo.
Summary by CodeRabbit
Documentation
Configuration
nemo-evaluatorinstallation with optional version pinning..env-prefixed name are now ignored.