From 77f03e6dff322dd444e1b4d731e32bac8d7334a9 Mon Sep 17 00:00:00 2001 From: Sandy Chapman Date: Wed, 12 Aug 2026 15:57:32 -0300 Subject: [PATCH 1/3] docs: record the Fabric agent-eval false-failure known issue A Fabric agent-eval trial can be scored as an adapter failure after the agent completed its work, because NeMo Relay's LangChain callback handler leaves a scope open when concurrent LangGraph sibling runs finish out of LIFO order. The defect is fixed upstream in NeMo-Relay #755 but cannot reach the platform until NeMo Relay publishes a release carrying it and NeMo Fabric moves off its nemo-relay<0.7 pin, so document it as a known issue for this release. Signed-off-by: Sandy Chapman --- docs/about/release-notes/current-release.mdx | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/docs/about/release-notes/current-release.mdx b/docs/about/release-notes/current-release.mdx index a47e073a04..436c8ddff2 100644 --- a/docs/about/release-notes/current-release.mdx +++ b/docs/about/release-notes/current-release.mdx @@ -205,6 +205,24 @@ workflows against the upgraded checkout. - **Python support.** Python 3.11 is no longer part of the supported source checkout or `nemo-platform` distribution matrix. +## Known issues + +- **Fabric agent-eval trials can be scored as failures after successful runs.** + An agent-eval trial run through a NeMo Fabric adapter can be recorded as an + adapter failure even though the agent completed its work and produced the + expected output. It affects agents whose graph runs nodes concurrently, and + it can occur at any parallelism, including one. NeMo Relay closes telemetry + scopes in strict last-in-first-out order, but concurrent LangGraph sibling + runs finish in an order that can leave a scope open; the enclosing request + scope then fails to close with `scope handle is not at the top of the stack`, + and the Fabric adapter reports that teardown error as an invocation failure. + Affected trials are scored as failures, so evaluation results can understate + an agent's accuracy. There is no workaround in this release. The underlying + defect is fixed upstream in + [NeMo-Relay #755](https://github.com/NVIDIA/NeMo-Relay/pull/755); the fix + reaches NeMo Platform once NeMo Relay publishes a release containing it and + NeMo Fabric updates its NeMo Relay dependency. + ## Links - Repository: [https://github.com/NVIDIA-NeMo/nemo-platform](https://github.com/NVIDIA-NeMo/nemo-platform) From c69716df01a0b0730f3ebdb50779daa3c99c964c Mon Sep 17 00:00:00 2001 From: Sandy Chapman Date: Wed, 12 Aug 2026 16:00:36 -0300 Subject: [PATCH 2/3] docs: trim the Deep Agents known-issue entry to the user-facing facts Signed-off-by: Sandy Chapman --- docs/about/release-notes/current-release.mdx | 18 +++--------------- 1 file changed, 3 insertions(+), 15 deletions(-) diff --git a/docs/about/release-notes/current-release.mdx b/docs/about/release-notes/current-release.mdx index 436c8ddff2..058579a9b8 100644 --- a/docs/about/release-notes/current-release.mdx +++ b/docs/about/release-notes/current-release.mdx @@ -207,21 +207,9 @@ workflows against the upgraded checkout. ## Known issues -- **Fabric agent-eval trials can be scored as failures after successful runs.** - An agent-eval trial run through a NeMo Fabric adapter can be recorded as an - adapter failure even though the agent completed its work and produced the - expected output. It affects agents whose graph runs nodes concurrently, and - it can occur at any parallelism, including one. NeMo Relay closes telemetry - scopes in strict last-in-first-out order, but concurrent LangGraph sibling - runs finish in an order that can leave a scope open; the enclosing request - scope then fails to close with `scope handle is not at the top of the stack`, - and the Fabric adapter reports that teardown error as an invocation failure. - Affected trials are scored as failures, so evaluation results can understate - an agent's accuracy. There is no workaround in this release. The underlying - defect is fixed upstream in - [NeMo-Relay #755](https://github.com/NVIDIA/NeMo-Relay/pull/755); the fix - reaches NeMo Platform once NeMo Relay publishes a release containing it and - NeMo Fabric updates its NeMo Relay dependency. +- **Deep Agents trials can be scored as failures.** Agent-eval trials run + through the NeMo Fabric Deep Agents adapter can be recorded as failures even + when the agent completed successfully. Fixed in a future release. ## Links From 64e503b3f2db5e9ea488e2c9350c52b5fa9338ea Mon Sep 17 00:00:00 2001 From: Sandy Chapman Date: Wed, 12 Aug 2026 16:03:01 -0300 Subject: [PATCH 3/3] docs: commit to the next release for the Deep Agents known issue Signed-off-by: Sandy Chapman --- docs/about/release-notes/current-release.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/about/release-notes/current-release.mdx b/docs/about/release-notes/current-release.mdx index 058579a9b8..e0bf499055 100644 --- a/docs/about/release-notes/current-release.mdx +++ b/docs/about/release-notes/current-release.mdx @@ -209,7 +209,7 @@ workflows against the upgraded checkout. - **Deep Agents trials can be scored as failures.** Agent-eval trials run through the NeMo Fabric Deep Agents adapter can be recorded as failures even - when the agent completed successfully. Fixed in a future release. + when the agent completed successfully. This will be fixed in the next release. ## Links