Skip to content

fix(e2e): remove scenario advisor yaml dependency #4081

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
cv merged 1 commit into from
May 22, 2026
+85 −36
Merged

fix(e2e): remove scenario advisor yaml dependency #4081

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
121 changes: 85 additions & 36 deletions tools/e2e-advisor/scenarios.mts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,10 @@
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0

import { createRequire } from "node:module";
import fs from "node:fs";
import path from "node:path";
import { pathToFileURL } from "node:url";

const require = createRequire(import.meta.url);
const yaml: { load(input: string): unknown } = require("js-yaml");

import { getChangedFiles } from "../advisors/git.mts";
import { parseArgs, writeJson } from "../advisors/io.mts";

Expand Down Expand Up @@ -50,23 +46,6 @@ export type ScenarioAdvisorResult = {
type ScenarioEntry = {
suites?: unknown;
runner_requirements?: unknown;
dimensions?: {
platform?: unknown;
onboarding?: unknown;
};
};

type ScenariosFile = {
setup_scenarios?: Record<string, ScenarioEntry>;
test_plans?: Record<string, ScenarioEntry>;
};

type SuiteEntry = {
steps?: Array<{ script?: unknown }>;
};

type SuitesFile = {
suites?: Record<string, SuiteEntry>;
};

if (
Expand Down Expand Up @@ -305,27 +284,97 @@ function loadScenarios(root: string): Record<string, ScenarioEntry> {
"test/e2e/nemoclaw_scenarios/scenarios.yaml",
);
if (!fs.existsSync(filePath)) return {};
const doc = yaml.load(fs.readFileSync(filePath, "utf8")) as
| ScenariosFile
| undefined;
return { ...(doc?.test_plans ?? {}), ...(doc?.setup_scenarios ?? {}) };
const text = fs.readFileSync(filePath, "utf8");
return {
...parseScenarioSection(text, "test_plans"),
...parseScenarioSection(text, "setup_scenarios"),
};
}

function loadSuiteScriptMap(root: string): Record<string, string[]> {
const filePath = path.join(root, "test/e2e/validation_suites/suites.yaml");
if (!fs.existsSync(filePath)) return {};
const doc = yaml.load(fs.readFileSync(filePath, "utf8")) as
| SuitesFile
| undefined;
const output: Record<string, string[]> = {};
for (const [suiteId, suite] of Object.entries(doc?.suites ?? {})) {
output[suiteId] = Array.isArray(suite.steps)
? suite.steps
.map((step) => step.script)
.filter((script): script is string => typeof script === "string")
: [];
return parseSuiteScripts(fs.readFileSync(filePath, "utf8"));
}

function parseScenarioSection(
text: string,
sectionName: string,
): Record<string, ScenarioEntry> {
const section = extractTopLevelSection(text, sectionName);
const scenarios: Record<string, ScenarioEntry> = {};
let currentId: string | undefined;
let inSuites = false;
let inRunnerRequirements = false;

for (const line of section.split(/\r?\n/)) {
const entryMatch = line.match(
/^ ([A-Za-z0-9_.-]+(?:__[A-Za-z0-9_.-]+)?):\s*$/,
);
if (entryMatch) {
currentId = entryMatch[1];
scenarios[currentId] = { suites: [], runner_requirements: [] };
inSuites = false;
inRunnerRequirements = false;
continue;
}
if (!currentId) continue;
if (/^ suites:\s*(?:\[\])?\s*$/.test(line)) {
inSuites = true;
inRunnerRequirements = false;
continue;
}
if (/^ runner_requirements:\s*$/.test(line)) {
inSuites = false;
inRunnerRequirements = true;
continue;
}
if (/^ [A-Za-z0-9_-]+:/.test(line)) {
inSuites = false;
inRunnerRequirements = false;
continue;
}
const listItem = line.match(/^ - ([A-Za-z0-9_.-]+)\s*$/);
if (listItem && inSuites) {
(scenarios[currentId].suites as string[]).push(listItem[1]);
} else if (listItem && inRunnerRequirements) {
(scenarios[currentId].runner_requirements as string[]).push(listItem[1]);
}
}
return output;

return scenarios;
}

function parseSuiteScripts(text: string): Record<string, string[]> {
const section = extractTopLevelSection(text, "suites");
const suites: Record<string, string[]> = {};
let currentId: string | undefined;

for (const line of section.split(/\r?\n/)) {
const suiteMatch = line.match(/^ ([A-Za-z0-9_.-]+):\s*$/);
if (suiteMatch) {
currentId = suiteMatch[1];
suites[currentId] = [];
continue;
}
if (!currentId) continue;
const scriptMatch = line.match(/^ script:\s*([A-Za-z0-9_./-]+)\s*$/);
if (scriptMatch) suites[currentId].push(scriptMatch[1]);
}

return suites;
}
Comment on lines +348 to +366

@coderabbitai coderabbitai Bot May 22, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Verify custom parser correctly extracts suite scripts

SUITES_FILE="test/e2e/validation_suites/suites.yaml"

if [ ! -f "$SUITES_FILE" ]; then
  echo "ERROR: $SUITES_FILE not found"
  exit 1
fi

echo "=== Checking for tabs ==="
if grep -P '^\t' "$SUITES_FILE"; then
  echo "ERROR: Found tabs. Parser expects spaces only."
  exit 1
fi
echo "✓ No tabs found"

echo ""
echo "=== Sample suite entries (should match pattern: 2 spaces + name + colon) ==="
grep -P '^  [A-Za-z0-9_.-]+:\s*$' "$SUITES_FILE" | head -5

echo ""
echo "=== Sample script entries (should match pattern: 6 spaces + 'script:' + path) ==="
grep -P '^      script:\s*[A-Za-z0-9_./-]+\s*$' "$SUITES_FILE" | head -10

echo ""
echo "=== Checking for YAML anchors on steps (parser won't follow aliases) ==="
if grep -P '^    steps:.*[&*]' "$SUITES_FILE"; then
  echo "WARNING: Found anchors/aliases on steps. Parser won't follow alias references."
fi

echo ""
echo "=== Count of suite definitions vs script extractions ==="
SUITE_COUNT=$(grep -cP '^  [A-Za-z0-9_.-]+:\s*$' "$SUITES_FILE")
SCRIPT_COUNT=$(grep -cP '^      script:\s*[A-Za-z0-9_./-]+\s*$' "$SUITES_FILE")
echo "Suites: $SUITE_COUNT"
echo "Scripts: $SCRIPT_COUNT"
echo "✓ Verification complete"

Repository: NVIDIA/NemoClaw

Length of output: 1293

🏁 Script executed:

#!/bin/bash
set -euo pipefail

SUITES_FILE="test/e2e/validation_suites/suites.yaml"

python3 - <<'PY'
import re, sys, collections
path="test/e2e/validation_suites/suites.yaml"
text=open(path,"r",encoding="utf-8").read()

# Extract top-level 'suites:' section (best-effort)
m=re.search(r'(?ms)^suites:\s*\n(.*)$', text)
if not m:
    print("Could not find 'suites:' top-level key")
    sys.exit(1)
section=m.group(1)

suite_re=re.compile(r'^  ([A-Za-z0-9_.-]+):\s*$')
script_re=re.compile(r'^      script:\s*([A-Za-z0-9_./-]+)\s*$')
steps_alias_re=re.compile(r'^    steps:\s*\*')

suites_scripts=collections.defaultdict(list)
suites_has_steps_alias=set()
current=None

for line in section.splitlines():
    sm=suite_re.match(line)
    if sm:
        current=sm.group(1)
        continue
    if current is None:
        continue
    if steps_alias_re.match(line):
        suites_has_steps_alias.add(current)
    sm2=script_re.match(line)
    if sm2:
        suites_scripts[current].append(sm2.group(1))

# Summarize
all_suites=sorted(set(suite_re.findall(section)))
zero=[s for s in all_suites if len(suites_scripts.get(s,[]))==0]
alias_only=[s for s in sorted(suites_has_steps_alias) if len(suites_scripts.get(s,[]))==0]

print("Total suites in section:", len(all_suites))
print("Suites with 0 extracted scripts:", len(zero))
if zero:
    print("First 15 zero-script suites:", zero[:15])

print("Suites that use steps aliases and have 0 extracted scripts:", len(alias_only))
if alias_only:
    print("First 15 alias-only suites:", alias_only[:15])

# Show a few non-zero samples for sanity
nonzero=[s for s in all_suites if len(suites_scripts.get(s,[]))>0]
print("Sample non-zero suites (up to 10):")
for s in nonzero[:10]:
    scripts=suites_scripts[s]
    print(f"- {s}: {len(scripts)} scripts; first: {scripts[0]}")
PY

Repository: NVIDIA/NemoClaw

Length of output: 366

🏁 Script executed:

#!/bin/bash
set -euo pipefail

FILE="test/e2e/validation_suites/suites.yaml"

# Show the suite blocks for a few suites that were "alias-only" (0 extracted scripts by the indentation parser)
for suite in cloud-inference diagnostics docs-validation gateway-health openai-compatible-inference sandbox-shell; do
  echo "================ ${suite} ================"
  # Print from the suite key line until the next top-level suite key (2 spaces + name + colon)
  awk -v s="$suite" '
    $0 ~ "^  " s ":\\s*$" {inblock=1}
    inblock {print}
    inblock && $0 ~ "^  [A-Za-z0-9_.-]+:\\s*$" && $0 !~ "^  " s ":\\s*$" {inblock=0}
  ' "$FILE" | head -n 120
done

# Also confirm what "script:" lines exist (and their indentation) within these suites
echo "================ script lines (with leading whitespace counts) ================"
python3 - <<'PY'
import re
file="test/e2e/validation_suites/suites.yaml"
suites=["cloud-inference","diagnostics","docs-validation","gateway-health","openai-compatible-inference","sandbox-shell"]
lines=open(file,'r',encoding='utf-8').read().splitlines()
suite_re=re.compile(r'^  ([A-Za-z0-9_.-]+):\s*$')
script_re=re.compile(r'^(\s*)script:\s*(.+?)\s*$')

current=None
wanted=set(suites)
out={s:[] for s in suites}
for line in lines:
    m=suite_re.match(line)
    if m:
        current=m.group(1)
    if current in wanted:
        sm=script_re.match(line)
        if sm:
            indent=len(sm.group(1).replace('\t','    '))
            out[current].append((indent, sm.group(2)))
for s in suites:
    print(s, "script lines:", len(out[s]))
    if out[s]:
        print("  indent samples:", out[s][:5])
PY

Repository: NVIDIA/NemoClaw

Length of output: 1157

🏁 Script executed:

#!/bin/bash
set -euo pipefail

FILE="tools/e2e-advisor/scenarios.mts"

# 1) Locate parseSuiteScripts + extractTopLevelSection implementation
rg -n "function parseSuiteScripts|extractTopLevelSection" "$FILE"
# Show surrounding code (avoid huge output by limiting lines around matches)
python3 - <<'PY'
import subprocess, re
file="tools/e2e-advisor/scenarios.mts"
# get line numbers for parseSuiteScripts and extractTopLevelSection
text=open(file,'r',encoding='utf-8').read().splitlines()
targets=[]
for i,l in enumerate(text,1):
    if "function parseSuiteScripts" in l or "extractTopLevelSection" in l:
        targets.append(i)
for t in sorted(set(targets)):
    start=max(1,t-20); end=min(len(text),t+60)
    print("\n### Snippet around line",t,"###\n")
    for j in range(start,end+1):
        print(f"{j}:{text[j-1]}")
PY

# 2) Re-run the parser logic in Python using the same regex expectations:
#    - take the "suites" top-level section the way extractTopLevelSection likely does (between "suites:" and next top-level key)
python3 - <<'PY'
import re, sys, collections

text=open("test/e2e/validation_suites/suites.yaml","r",encoding="utf-8").read()

# Best-effort mimic: extract from "^suites:" to next "^\\S" (top-level key)
m=re.search(r'(?ms)^suites:\s*\n(.*?)(?:\n^[^ \t].*?:\s*$|\Z)', text)
if not m:
    # fallback: from suites: to EOF
    m=re.search(r'(?ms)^suites:\s*\n(.*)$', text)
    if not m:
        print("Could not locate suites: section")
        sys.exit(1)

section=m.group(1)

suite_re=re.compile(r'^  ([A-Za-z0-9_.-]+):\s*$')
script_re=re.compile(r'^      script:\s*([A-Za-z0-9_./-]+)\s*$')

suites=collections.defaultdict(list)
current=None
for line in section.splitlines():
    sm=suite_re.match(line)
    if sm:
        current=sm.group(1)
        suites[current]=[]
        continue
    if not current:
        continue
    sm2=script_re.match(line)
    if sm2:
        suites[current].append(sm2.group(1))

all_suites=sorted(suites.keys())
zero=[s for s in all_suites if len(suites[s])==0]

print("Parsed suites:", len(all_suites))
print("Suites with 0 extracted scripts (would fail to map scripts for alias-based suites):", len(zero))
print("First 30 zero-script suites:", zero[:30])

# Count how many script lines exist in file under exact indentation (for sanity)
file_script_re=re.compile(r'^      script:\s*[A-Za-z0-9_./-]+\s*$', re.M)
print("Total inline 'script:' lines matching exact indentation:", len(file_script_re.findall(text)))
PY

Repository: NVIDIA/NemoClaw

Length of output: 12268

Fix parseSuiteScripts to handle YAML aliases (it currently misses 6 suites)

  • In tools/e2e-advisor/scenarios.mts:348-366, the regexes correctly match inline suite/script entries in test/e2e/validation_suites/suites.yaml ( <suite>: and script:).
  • However, 6 suites define steps: *id... without any inline script: keys (cloud-inference, diagnostics, docs-validation, gateway-health, openai-compatible-inference, sandbox-shell), so parseSuiteScripts returns an empty list for them.
  • Downstream mapping that relies on this will treat those suites as having no scripts—use a YAML parser that resolves aliases (instead of indentation-based regex parsing) or explicitly handle/skip alias-based suites.
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tools/e2e-advisor/scenarios.mts` around lines 348 - 366, parseSuiteScripts
currently uses regexes to parse the "suites" section line-by-line so YAML
aliases like "steps: *id" are ignored (causing 6 suites to appear empty); update
parseSuiteScripts to use a proper YAML parser that resolves aliases (or at
minimum detect a "steps:\s*\*[A-Za-z0-9_-]+" pattern and follow the alias
lookup) so script lists are populated. Specifically, replace the
indentation/regex parsing in parseSuiteScripts (and its dependency
extractTopLevelSection if needed) with a YAML parse of the suites block (e.g.,
parse into objects and read suiteId -> steps/script arrays) or add logic to
resolve alias references from the same YAML content before returning suites.


function extractTopLevelSection(text: string, sectionName: string): string {
const lines = text.split(/\r?\n/);
const start = lines.findIndex((line) => line === `${sectionName}:`);
if (start === -1) return "";
const sectionLines: string[] = [];
for (const line of lines.slice(start + 1)) {
if (/^[A-Za-z0-9_-]+:/.test(line)) break;
sectionLines.push(line);
}
return sectionLines.join("\n");
}

function buildSuiteToScenarios(
Expand Down
Loading