Skip to content

test(e2e): add environment phase fixture#5002

Merged
cv merged 4 commits into
mainfrom
codex/e2e-4990-stack-01-environment-phase
Jun 9, 2026
Merged

test(e2e): add environment phase fixture#5002
cv merged 4 commits into
mainfrom
codex/e2e-4990-stack-01-environment-phase

Conversation

@cv

@cv cv commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds a typed environment phase fixture for Vitest E2E scenarios so install/runtime readiness can move out of shell-only orchestration. This first stack layer verifies the NemoClaw CLI and Docker runtime modes from typed scenario environment metadata.

Related Issue

Refs #4941
Refs #4990

Changes

  • Adds EnvironmentPhaseFixture with typed install and Docker runtime readiness checks.
  • Exposes environment on the shared Vitest E2E scenario context.
  • Lets HostCliClient report its configured CLI path and run CLI availability checks with a scoped allowlisted host env.
  • Adds framework tests for supported runtime modes, no-Docker expectations, optional Docker, and context typing.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Verification

  • npx prek run --all-files passes
  • npm test passes
  • Tests added or updated for new or changed behavior
  • No secrets, API keys, or credentials committed
  • Docs updated for user-facing behavior changes
  • npm run docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: Carlos Villela cvillela@nvidia.com

Summary by CodeRabbit

  • Tests
    • Added a new environment readiness fixture to e2e tests to validate CLI and Docker availability across scenarios.
    • Strengthened e2e checks for CLI version and Docker probes, including scoped environment variables for probe runs.
    • Exposed host CLI path in test fixtures and improved reporting of Docker availability (required/optional/missing).

@cv cv self-assigned this Jun 9, 2026
@coderabbitai

coderabbitai Bot commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Warning

Review limit reached

@cv, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 29 minutes and 7 seconds. Learn how PR review limits work.

Your organization has run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: f9168310-00ff-4502-bec8-b1ba9cfa1bc2

📥 Commits

Reviewing files that changed from the base of the PR and between 99399d7 and 8fe4353.

📒 Files selected for processing (3)
  • test/e2e-scenario/framework-tests/e2e-phase-environment.test.ts
  • test/e2e-scenario/framework/availability-env.ts
  • test/e2e-scenario/scenarios/orchestrators/redaction.ts
📝 Walkthrough

Walkthrough

Adds an EnvironmentPhaseFixture that checks Nemoclaw CLI and Docker runtime availability, a filtered probe environment helper, HostCliClient API updates, test-framework wiring, re-exports, and comprehensive tests validating install/runtime expectation behavior and probe env scoping.

Changes

Environment Phase Fixture

Layer / File(s) Summary
Environment fixture core implementation
test/e2e-scenario/framework/phases/environment.ts, test/e2e-scenario/framework/availability-env.ts
Defines EnvironmentPhaseFixture, DockerRuntimeExpectation/DockerRuntimeReady/EnvironmentReady types, runtime expectation mapping, probeDocker (30s timeout), and buildAvailabilityProbeEnv helper to filter probe environment variables.
Framework integration and client API
test/e2e-scenario/framework/clients/host.ts, test/e2e-scenario/framework/e2e-test.ts, test/e2e-scenario/framework/phases/index.ts
Adds HostCliClient.commandPath getter, updates expectNemoclawAvailable() to pass env: buildAvailabilityProbeEnv(), wires EnvironmentPhaseFixture into the Vitest environment fixture, and re-exports fixture and types.
Comprehensive environment fixture tests
test/e2e-scenario/framework-tests/e2e-phase-environment.test.ts, test/e2e-scenario/framework-tests/e2e-clients.test.ts
Adds FakeRunner and extensive tests covering CLI/Docker probes, required/optional/missing runtime semantics, probe env scoping, support for launchable/gpu-docker-cdi IDs, unsupported ID rejection, and updates an existing client test to expect probe env.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

  • NVIDIA/NemoClaw#4966: Extends the fixture-friendly e2e client framework and updates HostCliClient probe behavior.
  • NVIDIA/NemoClaw#4965: Earlier fixture-context work establishing Vitest fixture plumbing used here.

Suggested labels

area: e2e, v0.0.62

Suggested reviewers

  • jyaunches
  • prekshivyas

Poem

"I hop and probe where CLIs play,
Docker whispers night and day,
I filter envs, I check the path,
Tests pass lightly — a rabbit's craft.
Cheers to fixtures, small and spry!" 🐰

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title 'test(e2e): add environment phase fixture' directly and specifically describes the main change—adding a new environment phase fixture to the e2e test framework. It is concise, clear, and accurately reflects the primary objective of the PR.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/e2e-4990-stack-01-environment-phase

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions

github-actions Bot commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

E2E Advisor Recommendation

Required E2E: None
Optional E2E: e2e-scenarios:ubuntu-repo-cloud-openclaw, e2e-vitest-scenarios:ubuntu-repo-cli-smoke

Dispatch hint: scenarios=ubuntu-repo-cloud-openclaw

Workflow run

Full advisor summary

E2E Recommendation Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E

  • None. No merge-blocking E2E is required because the PR changes only the E2E scenario harness and its framework tests, not NemoClaw runtime/user-flow code. Unit/framework tests should cover the new environment fixture and env-redaction behavior; optional E2E can be run if maintainers want confidence that the harness still drives real scenarios.

Optional E2E

  • e2e-scenarios:ubuntu-repo-cloud-openclaw (medium): Optional smoke coverage for the typed scenario runner because buildChildEnv is shared by scenario phase action spawns. This verifies the env allowlist still permits install/onboarding/runtime actions and secretEnv passthrough in a real Ubuntu OpenClaw flow.
  • e2e-vitest-scenarios:ubuntu-repo-cli-smoke (low): Optional live Vitest fixture smoke to ensure the E2E fixture stack and HostCliClient still execute the built repo CLI correctly after fixture changes.

New E2E recommendations

  • None.

Dispatch hint

  • Workflow: .github/workflows/e2e-scenarios.yaml
  • jobs input: scenarios=ubuntu-repo-cloud-openclaw

@github-actions

github-actions Bot commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

E2E Scenario Advisor Recommendation

Required scenario E2E: e2e-scenarios-all
Optional scenario E2E: None

Dispatch required scenario E2E:

  • gh workflow run e2e-scenarios-all.yaml --ref <pr-head-ref>

Workflow run

Full scenario advisor summary

E2E Scenario Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required scenario E2E

  • e2e-scenarios-all: Changes modify shared scenario E2E framework/orchestration behavior: environment readiness probing, host CLI availability env scoping, fixture wiring, Docker runtime checks, and child-env redaction allowlisting. These shared paths can affect every scenario regardless of suite or platform, so run the full scenario fan-out.
    • Dispatch: gh workflow run e2e-scenarios-all.yaml --ref <pr-head-ref>

Optional scenario E2E

  • None.

Relevant changed files

  • test/e2e-scenario/framework-tests/e2e-clients.test.ts
  • test/e2e-scenario/framework-tests/e2e-phase-environment.test.ts
  • test/e2e-scenario/framework/availability-env.ts
  • test/e2e-scenario/framework/clients/host.ts
  • test/e2e-scenario/framework/e2e-test.ts
  • test/e2e-scenario/framework/phases/environment.ts
  • test/e2e-scenario/framework/phases/index.ts
  • test/e2e-scenario/scenarios/orchestrators/redaction.ts

@github-actions

github-actions Bot commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor

Findings: 0 needs attention, 0 worth checking, 0 nice ideas
Since last review: 3 prior items resolved, 0 still apply, 0 new items found

Consider writing more tests for

Workflow run details

This is an automated advisory review. A human maintainer must make the final merge decision.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
test/e2e-scenario/framework-tests/e2e-phase-environment.test.ts (1)

163-180: 💤 Low value

Consider adding test coverage for optional Docker succeeding.

The test suite comprehensively covers required/missing Docker scenarios, but the case where an optional Docker runtime probe succeeds (e.g., macos-docker-optional with Docker available) is not explicitly tested. The logic is straightforward (would return with available: true), but explicit coverage would complete the matrix.

Optional test case
+  it("records optional Docker as available when present", async () => {
+    const runner = new FakeRunner();
+    runner.enqueue(shellResult(0, "nemoclaw v0.0.0\n"));
+    runner.enqueue(shellResult(0, "Docker is available\n"));
+    const environment = new EnvironmentPhaseFixture(new HostCliClient(runner));
+
+    const ready = await environment.assertReady({
+      ...cloudOpenClawEnvironment,
+      platform: "macos-local",
+      runtime: "macos-docker-optional",
+    });
+
+    expect(ready.docker).toMatchObject({
+      id: "macos-docker-optional",
+      expectation: "optional",
+      available: true,
+    });
+  });
+
   it("rejects unsupported install and runtime IDs", async () => {
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/e2e-scenario/framework-tests/e2e-phase-environment.test.ts` around lines
163 - 180, Add a new test that verifies an optional Docker runtime probe can
succeed: create a FakeRunner and EnvironmentPhaseFixture (using HostCliClient)
with the scenario config spread from cloudOpenClawEnvironment but set runtime to
"macos-docker-optional", enqueue a shellResult that simulates a successful
docker response (e.g., version output), call environment.assertReady(...) and
assert it resolves and the returned environment/runtime probe indicates
available: true; reference EnvironmentPhaseFixture, HostCliClient, FakeRunner,
environment.assertReady and cloudOpenClawEnvironment to locate where to add the
assertion.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@test/e2e-scenario/framework-tests/e2e-phase-environment.test.ts`:
- Around line 163-180: Add a new test that verifies an optional Docker runtime
probe can succeed: create a FakeRunner and EnvironmentPhaseFixture (using
HostCliClient) with the scenario config spread from cloudOpenClawEnvironment but
set runtime to "macos-docker-optional", enqueue a shellResult that simulates a
successful docker response (e.g., version output), call
environment.assertReady(...) and assert it resolves and the returned
environment/runtime probe indicates available: true; reference
EnvironmentPhaseFixture, HostCliClient, FakeRunner, environment.assertReady and
cloudOpenClawEnvironment to locate where to add the assertion.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 36e898e7-7081-4592-bf47-e0c68a844dfc

📥 Commits

Reviewing files that changed from the base of the PR and between ab76d0b and 32ad96a.

📒 Files selected for processing (6)
  • test/e2e-scenario/framework-tests/e2e-clients.test.ts
  • test/e2e-scenario/framework-tests/e2e-phase-environment.test.ts
  • test/e2e-scenario/framework/clients/host.ts
  • test/e2e-scenario/framework/e2e-test.ts
  • test/e2e-scenario/framework/phases/environment.ts
  • test/e2e-scenario/framework/phases/index.ts

@wscurran wscurran added the chore Build, CI, dependency, or tooling maintenance label Jun 9, 2026
@cv cv merged commit d9db199 into main Jun 9, 2026
63 of 87 checks passed
@cv cv deleted the codex/e2e-4990-stack-01-environment-phase branch June 9, 2026 03:31
@coderabbitai coderabbitai Bot mentioned this pull request Jun 9, 2026
12 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

chore Build, CI, dependency, or tooling maintenance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants