-
Notifications
You must be signed in to change notification settings - Fork 2.8k
test(e2e): cover onboard negative paths #3791
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
c513333
test(e2e): cover onboard negative paths
sandl99 2219618
test(e2e): satisfy onboard hook gates
sandl99 048da39
test(e2e): migrate onboard negatives to scenarios
jyaunches 5f7802d
ci(e2e): install scenario runner deps in nightly
jyaunches 127aae3
style(e2e): apply pre-push formatting
jyaunches d89919f
fix(e2e): address negative path review findings
jyaunches 38f9506
test(e2e): source negative-path helpers
ericksoa File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| // SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| /** Testing-only hook for deterministic E2E resume/repair fault injection. */ | ||
| export function maybeForceE2eStepFailure(stepName: string): void { | ||
| if (process.env.NEMOCLAW_E2E_FAILURE_INJECTION !== "1") return; | ||
| const forcedStep = (process.env.NEMOCLAW_E2E_FORCE_FAIL_AT_STEP || "").trim(); | ||
| if (!forcedStep || forcedStep !== stepName) return; | ||
| console.error(` [e2e] Forced onboarding failure at step '${stepName}'.`); | ||
| process.exit(1); | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,34 @@ | ||
| #!/usr/bin/env bash | ||
| # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| # Helpers for expected-failure E2E flows. | ||
|
|
||
| e2e_negative_output_has_stack_trace() { | ||
| local output="$1" | ||
| printf '%s\n' "${output}" | grep -Eq '(^|[[:space:]])(TypeError|ReferenceError|SyntaxError):|^[[:space:]]+at ' | ||
| } | ||
|
|
||
| e2e_negative_assert_failure() { | ||
| local log_file="$1" | ||
| local actual_exit="$2" | ||
| local expected_exit="$3" | ||
| local message_contains="$4" | ||
| local no_stack_trace="${5:-0}" | ||
|
|
||
| if [[ "${actual_exit}" -ne "${expected_exit}" ]]; then | ||
| echo "expected failure exit ${expected_exit}, got ${actual_exit}" >&2 | ||
| cat "${log_file}" >&2 | ||
| return 1 | ||
| fi | ||
| if [[ -n "${message_contains}" ]] && ! grep -Fq "${message_contains}" "${log_file}"; then | ||
| echo "expected failure output to contain: ${message_contains}" >&2 | ||
| cat "${log_file}" >&2 | ||
| return 1 | ||
| fi | ||
| if [[ "${no_stack_trace}" == "1" ]] && e2e_negative_output_has_stack_trace "$(cat "${log_file}")"; then | ||
| echo "expected failure output not to contain a JavaScript stack trace" >&2 | ||
| cat "${log_file}" >&2 | ||
| return 1 | ||
| fi | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,47 @@ | ||
| #!/usr/bin/env bash | ||
| # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| # Assertions for onboard registry/session provider, model, and policy state. | ||
|
|
||
| e2e_onboard_state_assert_registry() { | ||
| local registry_file="$1" | ||
| local sandbox_name="$2" | ||
| local expected_provider="$3" | ||
| local expected_model="$4" | ||
| local expected_presets_csv="$5" | ||
| node - "${registry_file}" "${sandbox_name}" "${expected_provider}" "${expected_model}" "${expected_presets_csv}" <<'NODE' | ||
| const fs = require("node:fs"); | ||
| const [registryPath, sandboxName, expectedProvider, expectedModel, csv] = process.argv.slice(2); | ||
| const registry = JSON.parse(fs.readFileSync(registryPath, "utf8")); | ||
| const sandbox = registry.sandboxes && registry.sandboxes[sandboxName]; | ||
| if (!sandbox) throw new Error(`missing sandbox registry entry: ${sandboxName}`); | ||
| if (sandbox.provider !== expectedProvider) throw new Error(`expected provider ${expectedProvider}, got ${sandbox.provider}`); | ||
| if (sandbox.model !== expectedModel) throw new Error(`expected model ${expectedModel}, got ${sandbox.model}`); | ||
| const policies = Array.isArray(sandbox.policies) ? sandbox.policies : []; | ||
| for (const preset of csv.split(",").filter(Boolean)) { | ||
| if (!policies.includes(preset)) throw new Error(`missing policy preset ${preset}; policies=${JSON.stringify(policies)}`); | ||
| } | ||
| NODE | ||
| } | ||
|
|
||
| e2e_onboard_state_assert_session() { | ||
| local session_file="$1" | ||
| local sandbox_name="$2" | ||
| local expected_provider="$3" | ||
| local expected_model="$4" | ||
| local expected_presets_csv="$5" | ||
| node - "${session_file}" "${sandbox_name}" "${expected_provider}" "${expected_model}" "${expected_presets_csv}" <<'NODE' | ||
| const fs = require("node:fs"); | ||
| const [sessionPath, sandboxName, expectedProvider, expectedModel, csv] = process.argv.slice(2); | ||
| const session = JSON.parse(fs.readFileSync(sessionPath, "utf8")); | ||
| if (session.status !== "complete") throw new Error(`session status ${session.status}`); | ||
| if (session.sandboxName !== sandboxName) throw new Error(`session sandbox ${session.sandboxName}`); | ||
| if (session.provider !== expectedProvider) throw new Error(`session provider ${session.provider}`); | ||
| if (session.model !== expectedModel) throw new Error(`session model ${session.model}`); | ||
| const presets = Array.isArray(session.policyPresets) ? session.policyPresets : []; | ||
| for (const preset of csv.split(",").filter(Boolean)) { | ||
| if (!presets.includes(preset)) throw new Error(`missing session policy preset ${preset}; presets=${JSON.stringify(presets)}`); | ||
| } | ||
| NODE | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,54 @@ | ||
| #!/usr/bin/env bash | ||
| # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| # Local TCP listener helper for deterministic gateway port-conflict tests. | ||
|
|
||
| E2E_PORT_HOLDER_PID="${E2E_PORT_HOLDER_PID:-}" | ||
|
|
||
| e2e_port_holder_start() { | ||
| local port="$1" | ||
| if [[ -n "${E2E_PORT_HOLDER_PID}" ]]; then | ||
| e2e_port_holder_stop | ||
| fi | ||
| E2E_PORT_HOLDER_PID="" | ||
| node - "${port}" <<'NODE' >/tmp/nemoclaw-e2e-port-holder.log 2>&1 & | ||
| const net = require("node:net"); | ||
| const port = Number(process.argv[2]); | ||
| const server = net.createServer((socket) => socket.end()); | ||
| server.on("error", (err) => { | ||
| console.error(err && err.message ? err.message : err); | ||
| process.exit(2); | ||
| }); | ||
| server.listen(port, "127.0.0.1", () => { | ||
| console.log("ready"); | ||
| }); | ||
| setInterval(() => {}, 1000); | ||
| NODE | ||
| E2E_PORT_HOLDER_PID=$! | ||
| local _i | ||
| for _i in $(seq 1 40); do | ||
| if node -e 'const net=require("node:net"); const port=Number(process.argv[1]); const s=net.connect(port,"127.0.0.1"); s.once("connect",()=>{s.destroy(); process.exit(0);}); s.once("error",()=>process.exit(1)); setTimeout(()=>process.exit(1),250);' "${port}" >/dev/null 2>&1; then | ||
| return 0 | ||
| fi | ||
| if ! kill -0 "${E2E_PORT_HOLDER_PID}" >/dev/null 2>&1; then | ||
| E2E_PORT_HOLDER_PID="" | ||
| return 1 | ||
| fi | ||
| sleep 0.25 | ||
| done | ||
| if [[ -n "${E2E_PORT_HOLDER_PID}" ]]; then | ||
| kill "${E2E_PORT_HOLDER_PID}" >/dev/null 2>&1 || true | ||
| wait "${E2E_PORT_HOLDER_PID}" >/dev/null 2>&1 || true | ||
| E2E_PORT_HOLDER_PID="" | ||
| fi | ||
| return 1 | ||
| } | ||
|
|
||
| e2e_port_holder_stop() { | ||
| if [[ -n "${E2E_PORT_HOLDER_PID}" ]]; then | ||
| kill "${E2E_PORT_HOLDER_PID}" >/dev/null 2>&1 || true | ||
| wait "${E2E_PORT_HOLDER_PID}" >/dev/null 2>&1 || true | ||
| E2E_PORT_HOLDER_PID="" | ||
| fi | ||
| } | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.