fix(evaluator): keep a Fabric trial completed when only Relay teardown failed - #1205
fix(evaluator): keep a Fabric trial completed when only Relay teardown failed#1205SandyChapman wants to merge 1 commit into
Conversation
…n failed
NVBug 6562846: a DeepAgents invocation finishes its turn — model calls, tool
calls, workspace changes, final assistant response — and is then reported as
failed because NeMo Relay's telemetry teardown raises
RuntimeError: invalid argument: scope handle is not at the top of the stack
Relay keeps one process-global LIFO scope stack in a ContextVar. LangGraph
schedules child tasks with copy_context(), which shares that same mutable stack
rather than cloning it, so overlapping chain callbacks close out of LIFO order;
the Fabric DeepAgents adapter catches the resulting teardown error in the same
try that guards the invocation and rewrites a successful run into an adapter
failure. QA saw 31 such trials in the 2026-08-05 regression, at parallelism 1 as
well as 3/5/10, so it is not an Evaluator concurrency problem.
The root cause is upstream in NeMo-Relay and NeMo-Fabric, neither of which is
vendored here. Until that lands, stop scoring the false negative: when a Fabric
failure carries the Relay scope-stack signature in the adapter's own
output.error *and* a non-empty final response, treat the trial as completed and
record the fault (recovered_from_telemetry_fault, fabric_status,
telemetry_error) so the recovery is auditable and the possibly-truncated
trajectory is not read as complete. This mirrors the existing
recovered_from_mcp_binding path.
The predicate deliberately needs both halves of the evidence: a run with no
final response, or a failure that is not the telemetry one, still fails.
Signed-off-by: Sandy Chapman <schapman@nvidia.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThe Fabric runtime now recovers runs affected only by a known Relay telemetry teardown failure when a non-empty final response exists. Tests verify metadata preservation and rejection of incomplete or unrelated failures. ChangesRelay teardown recovery
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
|
| if telemetry_error is None: | ||
| return self._failed_trial(task, evidence_dir, _result_error(result), extra_metadata=base_metadata) | ||
| # The agent finished its turn and produced a final response; only Relay's telemetry teardown | ||
| # failed, and the Fabric adapter's single try/except rewrote that into an invocation failure |
There was a problem hiding this comment.
nit: repeated comments at _RELAY_SCOPE_STACK_ERROR as well as here
| if not isinstance(output, Mapping): | ||
| return None | ||
| error = output.get("error") | ||
| if not isinstance(error, str) or _RELAY_SCOPE_STACK_ERROR not in error: |
There was a problem hiding this comment.
I will reluctantly approve it.
But this smells like encoding Relay/Fabric workarounds in our code.
Ideally relay fixes it and we just bump up the version. (I did same with last ATIF nvbug).
At worst, the issue is moved to next release(which is okay since it is P1)
I shall however leave it your judgement whether to ship it versus, let relay fix this bug and bump version
|
Closing in favour of the upstream fix. This treated the Relay teardown error as non-fatal in the evaluator, which worked around the symptom rather than the cause. The actual defect was in Relay's LangChain callback handler dropping a scope close that arrived out of LIFO order, and it is now fixed upstream in NeMo-Relay #755, merged to That fix cannot reach the platform immediately — it needs a Relay 0.7.x release, and NeMo Fabric currently pins |
Pull request was closed
2 participants
Summary
NVBug 6562846: a Fabric DeepAgents invocation completes its turn — model calls, tool calls, workspace changes, a final assistant response — and is then reported as failed because NeMo Relay's telemetry teardown raises
RuntimeError: invalid argument: scope handle is not at the top of the stack. QA saw 31 such trials in the 2026-08-05 full Evaluator regression, including atparallelism=1, so it is not an Evaluator concurrency problem. Before: those trials score as adapter invocation failures. After: when the failure is provably telemetry-only, the trial stayscompletedwith the fault recorded on it.Related Issue
AALGO-495. NVBug 6562846.
Changes
FabricAgentRuntime._to_trialno longer fails a trial whose only failure is a Relay telemetry-teardown fault. It falls through to the normal success path, so the recovered trial carries the same output and evidence (workspace,fabric_result.json, any promoted ATIF/ATOF artifacts) as any other completed trial._telemetry_teardown_errorpredicate. It requires both halves of the evidence before recovering: the adapter's ownoutput.errorcarries the Relay scope-stack signature, andoutput.responseholds a non-empty final assistant message. It readsoutput.errorrather thanresult.errorbecause Fabric normalizes every adapter-reported failure to the same top-leveladapter_reported_failurecode, so only the adapter's own error string distinguishes a telemetry teardown from a real agent failure.recovered_from_telemetry_fault,fabric_status, andtelemetry_errorland on the trial metadata (and a warning is logged), so it is auditable and the possibly-truncated trajectory is not read as complete. This mirrors the existingrecovered_from_mcp_bindingpath.make vendor).Why this is a mitigation, not the fix
The root cause is upstream and not vendored in this repo:
NVIDIA/NeMo-Relay—nemo_relay/integrations/langchain/callbacks.pymaps every chain run onto one process-global LIFOScopeStackheld in aContextVar. LangGraph schedules child tasks withcopy_context(), which preserves that same mutable stack rather than cloning it, so overlapping chain callbacks close out of LIFO order._pop_scopealso drops the handle from_scope_handlesbefore attempting the Relay pop, stranding an untracked live scope. Relay shipsfork_asyncio_context()for exactly this case; the integration never uses it.NVIDIA/NeMo-Fabric—adapters/deepagents/adapter.pycatches the teardown exception in the sametrythat guards the agent invocation, so an observability fault rewrites a successful functional outcome.Verified still live on 2026-08-10 by re-running the bug's minimal repro (two overlapping async chain lifecycles inside one outer Relay scope): reproduces on
nemo-relay0.6.0 (the version pinned here) and on 0.7.2, the newest release —callbacks.pyis byte-identical between the two. Upstream issues are being filed separately.Type of Change
Quality Gates
Verification
Signed-off-by:traileruv run pre-commit run -apasses, or any blocked checks are identified belowTargeted validation:
uv run --frozen pytest packages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_runtime.py -q— 37 passed (34 before, 3 new)._telemetry_teardown_errorto returnNonefailstest_fabric_runtime_recovers_trial_when_only_relay_teardown_failed— the new test genuinely exercises the new code.uv run --frozen pytest packages/nemo_evaluator_sdk/tests/ -q— 1451 passed.uv run ruff check packages/nemo_evaluator_sdk/— clean.uv run ruff format --checkon both changed source files — clean.tools/lint/lint-python-types.sh(the exact CI type-check entrypoint) — exit 0; no diagnostics in the changed files.tools/lint/lint-sdk-vendored.shandtools/lint/lint-cli.sh— pass after committing the regenerated mirror.Blocked locally, not marked as passed:
uv run pre-commit run -a— all Python hooks pass (ruff, ruff format, ty, config-reference docs, uv.lock drift, copyright headers, merge conflicts, plugin-import check). Two hooks fail for local-environment reasons unrelated to this change and are left to CI:uv-lockrequires uv 0.9.14 and this machine has 0.9.30 (nopyproject.toml/uv.lockchange in this PR), andstudio-lint-stagedcannot run withoutweb/node_modules(noweb/files changed).tools/lint/lint-openapi.shfails locally withmapfile: command not found(macOS bash 3.2) andtools/lint/lint-web-sdk.shfails on missingweb/node_modules— both environmental, neither touched by this PR.Summary by CodeRabbit
Bug Fixes
Tests