[skill] evaluation: align nel-next TB2.1/SWE-bench with golden toolchain #2063
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -5,8 +5,9 @@ run flow). Same harbor/ECS-Fargate flow as Terminal-Bench; the deltas are the | |
| **OpenHands agent**, a larger problem set, longer timeouts, and a different | ||
| ECR/region. Start from `recipes/examples/example_eval_next.yaml`. | ||
|
|
||
| > **Source of truth:** `configs/benchmarks/swe-bench-verified/bench.yaml` in | ||
| > nvidia-eval-factory-benchmarking (`dl/JoC/competitive_evaluation/…`), with the eval-image | ||
| > pin in `configs/shared/nel_next_containers.yaml` — match its values for a reference run. | ||
|
|
||
| ## Task-specific values (canonical `bench.yaml`) | ||
|
|
||
|
|
@@ -15,20 +16,24 @@ ECR/region. Start from `recipes/examples/example_eval_next.yaml`. | |
| | `playbook` | `swebench_verified` (`harbor://swebench-verified@1.0`) | | ||
| | agent | `openhands-sdk` (playbook; `agent_kwargs: {max_iterations: 200, version: "1.17.0"}`) | | ||
| | scope | 500 Python tasks × `repeats: 5` | | ||
| | `max_concurrent` / `sandbox.concurrency` | `15` in `bench.yaml`; per-model configs override it (MiniMax-M2.7 uses `20`) | | ||
| | `solver` | `timeout_strategy: max`, `run_timeout: 10800` (3h), `agent_kwargs.llm_kwargs.timeout: 3600` | | ||
| | `sandbox.region` | `us-east-2` | | ||
|
There was a problem hiding this comment. [SUGGESTION] Two small consistency issues in this table row and the block below it.
|
||
| | `sandbox.ecr_repository` | `${HARBOR_SWEBENCH_ECR_REPOSITORY}` (dedicated `harbor-swebench` repo, **us-west-2**, regardless of sandbox region) | | ||
| | `cluster.eval_image` | `${NEL_NEXT_EVAL_IMAGE}` → **`0.5.0.1-harbor`** (same pin as TB2.1: `configs/shared/nel_next_containers.yaml`) | | ||
| | `cluster.container_env.AWS_DEFAULT_REGION` | `us-east-2` (match `sandbox.region`) | | ||
| | `instruction_template` | `/configs/prompts/swebench_instruction.md`, **must be MOUNTED**; content is scoring-relevant (gotcha below) | | ||
| | `proxy.request_timeout` | `3600` (FEP-1104 paired HTTP timeout; leaves mirror it on the service proxy) | | ||
| | `drop_params` | `max_tokens`, `max_completion_tokens`, `max_input_tokens_per_task`, `no_rebuild` | | ||
| | `output.export_config.mlflow.exclude_patterns` | `["shard*", "model_traffic.jsonl"]` | | ||
| | `system_message` | `strategy: replace` + the OpenHands prompt from `bench.yaml` (verbatim) — scoring-relevant | | ||
|
|
||
| ```yaml | ||
| benchmarks: | ||
| - playbook: swebench_verified | ||
| repeats: 5 | ||
| max_concurrent: 15 | ||
| instruction_template: /configs/prompts/swebench_instruction.md # mounted (see gotcha) | ||
| solver: | ||
| service: <svc-name> | ||
| timeout_strategy: max # canonical; "task" = leaderboard-comparable | ||
|
|
@@ -45,41 +50,65 @@ benchmarks: | |
|
|
||
| The playbook defaults `instruction_template: swebench-instruction.md`, but the | ||
| harbor image doesn't ship that built-in → run dies at finalize with | ||
| `FileNotFoundError: instruction_template not found`. So it must be mounted. | ||
|
|
||
| **Which file you mount changes the score.** Mount the canonical `swebench_instruction.md` | ||
| (underscore) at `/configs/prompts/swebench_instruction.md`, taken from the reference config or | ||
| run dir. The built-in in the `nemo_evaluator/templates/` venv directory is a **different | ||
| prompt** (`swebench-instruction.md`, hyphen) — it runs, but results are not comparable. Keep | ||
| whichever you use fixed across both sides of a comparison. | ||
|
|
||
| ```bash | ||
| ssh <login> 'mkdir -p <lustre>/<user>/prompts' | ||
|
Comment on lines
+54
to
+62
There was a problem hiding this comment. [SUGGESTION] The rewritten gotcha loses the one thing that made the old version actionable: where to get the file. The previous text gave a concrete The two other source-of-truth pointers in this file are precise ( # canonical prompt: configs/prompts/swebench_instruction.md in nvidia-eval-factory-benchmarking
# (dl/JoC/competitive_evaluation/…) — record its sha256 alongside the score
ssh <login> 'mkdir -p <lustre>/<user>/prompts'
scp swebench_instruction.md <login>:<lustre>/<user>/prompts/Also worth stating explicitly that both sides of a BF16-vs-quantized comparison must mount the same file — the current "Keep whichever you use fixed across both sides of a comparison" implies it, but this is the highest-leverage sentence in the section. |
||
| scp swebench_instruction.md <login>:<lustre>/<user>/prompts/ # canonical file; verify sha256 against the source | ||
| ``` | ||
|
|
||
| ```yaml | ||
| benchmarks: [{playbook: swebench_verified, instruction_template: /configs/prompts/swebench_instruction.md}] | ||
| cluster: | ||
| container_mounts: ["<lustre>/<user>/prompts/swebench_instruction.md:/configs/prompts/swebench_instruction.md:ro"] | ||
| ``` | ||
|
Comment on lines
66
to
70
Contributor
There was a problem hiding this comment. 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
rg -n -C 8 \
'container_mounts|^services:|^cluster:' \
.agents/skills/evaluation/references/nel-next.md \
.agents/skills/evaluation/recipes/examples/example_eval_next.yaml \
.agents/skills/evaluation/recipes/tasks/aa_next/swebench_verified.mdRepository: NVIDIA/Model-Optimizer Length of output: 13010 🏁 Script executed: #!/bin/bash
set -euo pipefail
printf '%s\n' '--- relevant files ---'
git ls-files '.agents/skills/evaluation' | sed -n '1,160p'
printf '%s\n' '--- mount and eval-image references ---'
rg -n -C 4 \
'container_mounts|eval_image|instruction_template|eval.image|mount' \
.agents/skills/evaluation \
--glob '*.md' --glob '*.yaml' --glob '*.yml' --glob '*.py' --glob '*.sh' \
| sed -n '1,260p'
printf '%s\n' '--- recipe context ---'
cat -n .agents/skills/evaluation/recipes/tasks/aa_next/swebench_verified.md | sed -n '45,90p'Repository: NVIDIA/Model-Optimizer Length of output: 34115 🏁 Script executed: #!/bin/bash
set -euo pipefail
printf '%s\n' '--- NEL schema reference ---'
cat -n .agents/skills/evaluation/references/nel-next.md | sed -n '58,125p'
printf '%s\n' '--- all repository references to cluster.container_mounts and evaluation mounts ---'
rg -n -C 5 \
'cluster\.container_mounts|container_mounts|mounts:|eval_image|evaluation:|eval-image|eval image' \
. \
--glob '!*.lock' --glob '!*.jsonl' \
| sed -n '1,360p'
printf '%s\n' '--- possible schema/parser sources ---'
rg -n -C 3 \
'extra="forbid"|extra.*forbid|class .*Config|container_mounts|eval_image' \
. \
--glob '*.py' --glob '*.pyi' --glob '*.yaml' --glob '*.yml' --glob '*.toml' \
| sed -n '1,360p'Repository: NVIDIA/Model-Optimizer Length of output: 50380 🌐 Web query:
💡 Result: In the NVIDIA NeMo Evaluator (including configurations used with NeMo Evaluator Launcher/NEL), the eval_image and container_mounts parameters are primary configuration options used when deploying evaluation jobs to containerized environments, such as SLURM clusters or Docker-based executors [1][2]. 1. eval_image: This parameter specifies the container image (e.g., Apptainer, Enroot, or Docker) used to execute the evaluation harness [1][2]. It provides the isolated environment where the evaluation code runs [3]. In some advanced workflows, this can be dynamically set via environment variables (e.g., ${NEL_NEXT_EVAL_IMAGE}) to ensure the correct architecture-matched or version-pinned image is used [4]. 2. container_mounts: This parameter accepts a list of host paths to mount into the evaluation container [1][2]. It is essential for providing the containerized environment access to required data on the host machine, such as: - Datasets (if they are not automatically managed by the task's dataset directory mounting feature) [5][6] - Caching directories (e.g., for vLLM or HuggingFace) [4] - Configuration files [4] - Results output directories [7] Usage Contexts: - SLURM Deployment: Both parameters are typically defined under the cluster configuration block (e.g., cluster.eval_image and cluster.container_mounts) [1][2]. - NeMo Run: When using NeMo Run, these are set as properties on the executor object (e.g., executor.container_image and executor.container_mounts) [8]. - Dataset-Specific Mounting: Beyond general container mounts, tasks that require specific local datasets can also use the dataset_dir and optional dataset_mount_path fields within the evaluation task configuration [5][6]. The system automatically handles these mounts and sets the corresponding NEMO_EVALUATOR_DATASET_DIR environment variable [5][6]. For further details on specific schema structures, users should refer to the official documentation for the respective version of the NeMo Evaluator SDK, as configurations may vary between stable releases and development branches [5][9][10]. Citations:
🏁 Script executed: #!/bin/bash
set -euo pipefail
printf '%s\n' '--- official SLURM documentation ---'
for url in \
'https://docs.nvidia.com/nemo/evaluator/deployment/slurm' \
'https://docs.nvidia.com/nemo/evaluator/nightly/deployment/slurm' \
'https://docs.nvidia.com/nemo/evaluator/latest/libraries/nemo-evaluator-launcher/configuration/executors/slurm.html'
do
printf '\nURL: %s\n' "$url"
curl -L --fail --silent --show-error "$url" |
tr '\n' ' ' |
sed 's/<[^>]*>/ /g' |
grep -oE '.{0,180}(container_mounts|eval_image|eval image|mounts).{0,260}' |
head -n 12 || true
done
printf '%s\n' '--- referenced repository commit ---'
if git cat-file -e a05850bffad6f254f265d7e30b46823a6064d8f7^{commit} 2>/dev/null; then
git show --stat --oneline a05850bffad6f254f265d7e30b46823a6064d8f7
git show a05850bffad6f254f265d7e30b46823a6064d8f7 -- \
'.agents/skills/evaluation/references/nel-next.md' \
'.agents/skills/evaluation/recipes/tasks/aa_next/swebench_verified.md' \
| sed -n '1,260p'
else
echo 'commit not present locally'
fiRepository: NVIDIA/Model-Optimizer Length of output: 18435 Document
🤖 Prompt for AI Agents |
||
|
|
||
| ### Deployment proxy (multi-turn agentic) | ||
|
|
||
| OpenHands runs ~200 turns/task. The canonical config adds a `system_message` | ||
| interceptor (a large OpenHands system prompt — copy it verbatim from `bench.yaml`) | ||
| plus `turn_counter`. | ||
|
|
||
| **Order differs from TB2.1**: `http_pairs_dump` is **first** (not last) and `drop_params` | ||
| comes **before** `consolidate_system`. `http_pairs_dump` is canary/diagnostic-only — it | ||
| retains every error pair in memory for the whole run (`references/nel-next.md`); drop it | ||
| from the scored config. | ||
|
|
||
| ```yaml | ||
| proxy: | ||
| request_timeout: 3600 | ||
| extra_body: {skip_special_tokens: false} # add model-card sampling extras if the card sets them | ||
| model_traffic: {capture_request_body: true} # FEA-224; pair with the exclude_patterns entry | ||
|
Contributor
There was a problem hiding this comment. The "pair with the In 0.4.0 capture_tool_calls: bool = True
capture_reasoning: bool = True
capture_messages: bool = True
capture_request_body: bool = False # <- the only opt-in
max_content_chars: int = 0 # 0 = no truncationSo Suggest making the exclude unconditional in |
||
| interceptors: | ||
| # - {name: http_pairs_dump, config: {dump_path: "$${NEL_OUTPUT_DIR}/http_pairs_metrics.json", first_n: 50}} # canary only | ||
| - {name: system_message, config: {strategy: replace, system_message: "<the OpenHands prompt from bench.yaml>"}} | ||
| - {name: turn_counter, config: {max_turns: 200, position: system_message}} | ||
| - {name: drop_params, config: {params: [max_tokens, max_completion_tokens, max_input_tokens_per_task, no_rebuild]}} | ||
| - {name: consolidate_system} | ||
| - {name: reasoning} # reasoning models: normalize reasoning field … | ||
| - {name: reasoning_replay} # … and replay across turns. Drop both for instruct models. | ||
|
Comment on lines
94
to
+95
Contributor
There was a problem hiding this comment. 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -eu
file='.agents/skills/evaluation/recipes/tasks/aa_next/swebench_verified.md'
printf '%s\n' '--- target excerpt ---'
sed -n '70,115p' "$file"
printf '%s\n' '--- related reasoning terms ---'
rg -n -C 3 'reasoning_replay|think_tags|MiniMax|Qwen|GLM|instruct|native' .agents/skills/evaluation/recipes/tasks/aa_next .agents/skills/evaluation 2>/dev/null | head -n 240Repository: NVIDIA/Model-Optimizer Length of output: 33100 🏁 Script executed: #!/bin/bash
set -eu
printf '%s\n' '--- interceptor definitions and usage ---'
rg -n -C 4 'reasoning_replay|name:\s*reasoning|think_tags|reasoning.*mode|native.*GLM' . --glob '!*.md' --glob '!*.lock' --glob '!package-lock.json' 2>/dev/null | head -n 320
printf '%s\n' '--- all likely proxy/config files ---'
git ls-files | rg '(^|/)(bench\.yaml|.*(proxy|interceptor|nel.*config|config).*\.(ya?ml|json|py|ts|js|md))$' | head -n 240Repository: NVIDIA/Model-Optimizer Length of output: 18764 🌐 Web query:
💡 Result: In the NVIDIA NeMo Evaluator, reasoning_replay is a specialized interceptor designed to manage chain-of-thought (CoT) reasoning across multi-turn agentic interactions [1][2]. It addresses the issue where certain agent SDKs drop reasoning_content during replay, which can lead to a loss of CoT context and increased turn counts in agentic rollouts [1][3]. The reasoning_replay interceptor caches reasoning_content from upstream model responses (keyed by tool call ID or content hash) and re-injects it into subsequent outbound requests [1]. Regarding the requested modes and features: 1. Interceptor Mode (Native): The reasoning_replay interceptor supports a native mode [3]. In this mode, it ensures compatibility with backends (such as vLLM versions >=0.19) that may require specific reasoning field representations [3]. Recent updates ensure that in native (and both) modes, the interceptor preserves and writes both reasoning and reasoning_content fields to accommodate differences in how various backends parse chat input [3]. 2. Think Tags: The interceptor integrates with the configured reasoning pipeline to wrap cached hidden chains of thought with configured think tokens (e.g., ...) [3][4]. While NeMo Evaluator generally allows customization of these tokens (e.g., in reasoning interceptor configuration) [4], the reasoning_replay interceptor uses these settings to maintain consistent reasoning formatting during re-injection [3]. 3. MiniMax: There is no direct mention of native MiniMax integration specifically within the NeMo Evaluator reasoning_replay interceptor documentation or commit history [1][3]. NeMo Evaluator interceptors are modular and typically built to be backend-agnostic by using standard OpenAI-compatible interfaces [4][5], meaning they would process reasoning content based on standard fields (like reasoning_content) regardless of the specific underlying model provider, provided the provider follows that protocol. In summary, the reasoning_replay interceptor is a tool to ensure continuity of reasoning in long-running agentic evaluations, with native modes specifically tuned to handle backend-specific field requirements (like vLLM) and standard think-tag wrapping [1][3]. Citations:
🌐 Web query:
💡 Result: I couldn’t find an indexed match for The relevant repository appears to be NVIDIA-NeMo/Evaluator, whose launcher package is located under Note: GitHub’s current repository path is likely:
Citations: 🏁 Script executed: #!/bin/bash
set -eu
for sha in 751d1f7cc2b078ac90a161e49db361da487d741a fce9256690e99c5905e74ba03fea482f2e9f9a75 e4cc20c94e4dd9dbdcd3c5a2d05b878bbfc22907; do
printf '\n--- commit %s ---\n' "$sha"
curl -fsSL "https://api.github.com/repos/NVIDIA-NeMo/Evaluator/commits/$sha" |
jq -r '.commit.message, (.files[]? | "FILE: \(.filename)\n\(.patch // "")")' |
rg -n -C 5 'reasoning_replay|think_tags|native|MiniMax|mode|reasoning_content' | head -n 240 || true
done
printf '\n--- current upstream interceptor docs ---\n'
curl -fsSL 'https://raw.githubusercontent.com/NVIDIA-NeMo/Evaluator/main/docs/about/concepts/interceptors.md' |
rg -n -C 5 'reasoning_replay|think_tags|native|MiniMax|reasoning_content' | head -n 240 || trueRepository: NVIDIA/Model-Optimizer Length of output: 570 🌐 Web query:
💡 Result: In the context of LLM evaluation frameworks, specifically the NVIDIA NeMo Evaluator SDK, the concepts of reasoning traces and think tags are utilized to process and analyze the step-by-step thinking processes of reasoning-capable models [1][2]. Think tags (e.g., and ) are special tokens used by many modern LLMs to encapsulate their chain-of-thought reasoning process—the intermediate logical steps, calculations, and deliberations performed before arriving at a final answer [1][2]. Reasoning replay or reasoning interception is a functional component within the NeMo Evaluator that manages these traces [2][3]. Because many benchmarks and evaluation pipelines expect only the final answer from a model, the Reasoning Interceptor serves several key roles: - Extraction and Removal: It identifies the reasoning trace within the model's output (using the specified start and end tokens), removes it from the content to ensure the evaluation receives only the final answer, and optionally isolates it for further analysis [1][2][4]. - Statistics and Metrics Tracking: The interceptor tracks metrics related to the reasoning process, such as the number of responses that included reasoning, whether the reasoning completed (i.e., reached the end token), and the average length of the reasoning trace (in tokens or words) [2][4][5]. This data is typically saved to an evaluation metrics file (e.g., eval_factory_metrics.json), allowing developers to analyze model behavior, compute costs, or evaluate the quality of the chain-of-thought [1][4][5]. While "reasoning_replay" is not a singular standard term, it appears in this technical ecosystem as part of the operational workflow for handling, logging, and re-evaluating the reasoning traces generated during model inference [2][6]. Configuration of these interceptors allows users to specify custom tokens if their model uses different markers than the standard tags [2][7]. Citations:
🌐 Web query:
💡 Result: In the context of the NVIDIA NeMo Evaluator framework, reasoning_replay and think_tags are components designed to manage and preserve Chain-of-Thought (CoT) reasoning processes across multi-turn agent interactions [1][2]. reasoning_replay is an interceptor designed to solve the problem where certain agent SDKs (such as OpenHands or pi-ai) drop reasoning_content during conversation replays, causing the model to lose its reasoning context in subsequent turns [2]. The interceptor functions by caching the reasoning_content from upstream model responses and automatically re-injecting it into the next outbound request [3][2]. This ensures that the model's intermediate "thinking" remains available throughout the session [2]. think_tags refer to the configuration used to wrap this cached reasoning content [3]. Because different model backends and APIs may use different fields or formats for reasoning (e.g., specific tags or normalized fields like reasoning_content), the system uses think_tags to standardize how the hidden chain of thought is presented when re-injected [3]. The interceptor can be configured to operate in different modes (such as native or both), ensuring compatibility across various backends like vLLM and SGLang by reading and writing the appropriate reasoning fields [3]. Citations:
🌐 Web query:
💡 Result: The term reasoning_replay refers to a specific interceptor component within the NVIDIA NeMo Evaluator framework, not a standalone model or feature from MiniMax [1][2]. It was developed to solve the issue where certain agent SDKs (such as OpenHands or older implementations) would drop an LLM's reasoning content—its Chain-of-Thought (CoT)—during multi-turn interactions [1][2]. When agentic workflows replay previous turns, the loss of this reasoning can cause models to lose context, leading to redundant or inefficient rollouts [2]. Key technical aspects of the reasoning_replay interceptor include: Caching and Re-injection: It intercepts model responses to cache reasoning content (typically identified by fields like reasoning_content or reasoning) and re-injects that content into subsequent outbound requests [1][2]. Cross-Backend Compatibility: The interceptor manages differences in how various inference backends (such as vLLM versions or SGLang) represent reasoning fields [2]. For example, it ensures compatibility with vLLM ≥0.19 by writing both reasoning and reasoning_content fields, as different backends rely on different keys [2]. Deterministic Operation: It is designed to compose deterministically with other framework components, such as reasoning normalizers and caching interceptors, ensuring the CoT is preserved consistently across long agentic benchmark runs [1]. While NeMo Evaluator is used to evaluate various models, including those with reasoning capabilities, reasoning_replay is a utility within the evaluation infrastructure designed to maintain data integrity during testing, rather than a feature of any specific model provider like MiniMax [1][3][4]. Citations:
Make the reasoning interceptor configuration model-specific. Reasoning models require 🤖 Prompt for AI Agents |
||
| ``` | ||
|
|
||
| **`reasoning_replay.mode` is per model, not per benchmark.** `think_tags` (Qwen-style), | ||
|
Contributor
There was a problem hiding this comment.
If |
||
| `native` (GLM), omitted (MiniMax). Copying another model's mode is a silent output-parsing bug. | ||
|
|
||
| **Omitting `system_message` is a scoring change**: without it the agent runs the | ||
| openhands-sdk default prompt instead of the canonical one. | ||
|
|
||
| ### Sharding | ||
|
|
||
| `max_concurrent`/`sandbox.concurrency` are **per shard**, and each shard redeploys the model | ||
| on its own node — `shards: N` multiplies serving capacity *and* live Fargate sandboxes | ||
| (`N × concurrency`). 500 tasks × `repeats: 5` = 2500 trials; `shards: 10` at `concurrency: 15` | ||
| suits it. Score is unaffected — purely a wall-clock lever. Check `N × concurrency` against the | ||
| Fargate quota and `N × gpus_per_node` against your allocation. | ||
|
|
||
| ## Score Extraction | ||
|
|
||
| Report **`pass@1`** only — benchmark `swebench-verified@1.0`, scorer `pass@1` (0–1): | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[IMPORTANT Compatibility] The new default install is an unpinned git branch, which is weaker pinning than what it replaces and undercuts this PR's own goal.
Before:
NEL_NEXT_SPECdefaulted tonemo-evaluator[harbor,export]==0.3.*— a version-constrained resolve that a user could tighten to an exact0.3.x.After:
NEL_NEXT_ORIGINdefaults togit+https://github.com/NVIDIA-NeMo/Evaluator.gitwithNEL_NEXT_REFempty, soINSTALL_SPECresolves to whatever the default branch HEAD happens to be.Why it matters for this specific script:
INSTALL_SPECis now a constant string across upstream commits. The header's premise — "uv caches the resolved env and refreshes it when the spec changes" — no longer holds: the spec never changes while HEAD moves. uv caches the resolved git commit, so machine A (first run in June) and machine B (first run in August) silently run different toolchains under an identical spec, and neither run needs--refreshto notice.--versioncannot disambiguate. The PR body notes there is nov0.4.0tag, sonemo_evaluator.__version__prints0.4.0for every commit on the branch. There is no way for a user (or an agent following the skill) to answer "which build produced this score."Evaluator.git@9dcca2aeas the build that matches golden, and the header tells the user to "setNEL_NEXT_REFto a commit SHA to pin" — but the shipped default does not do that, so the default path is the unreproducible one.For a script whose entire purpose is producing benchmark numbers that are comparable across runs and across the BF16/NVFP4 sides of a comparison, the default should be the reproducible build.
Suggested fix — default
NEL_NEXT_REFto the validated SHA, keeping both override paths intact:and update the header +
references/nel-next.mdto state the default is pinned rather than instructing the user to pin it themselves. If tracking HEAD is genuinely intended, then--version/--whichshould surface the resolved commit so a run is at least attributable after the fact.Secondary, smaller point in the same hunk: precedence between the two variables inverted (
NEL_NEXT_ORIGINused to win when both were set; nowNEL_NEXT_SPECdoes). That is intentional per the comment, but anyone with both already in their.envflips from a git build to a PyPI build with no signal — worth a line inreferences/nel-next.md.