Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
6c9d212
ci(e2e): publish parity coverage report
jyaunches May 15, 2026
0626819
fix(sandbox): recover stale inference route on connect (#3444)
yimoj May 15, 2026
ce42e53
fix(docker): force-enable BuildKit in dockerBuild (#3585)
laitingsheng May 15, 2026
a28f365
refactor(e2e): enhance test of cloudflared tunnel and test of backup/…
hunglp6d May 15, 2026
2534db4
fix(onboard): honor NEMOCLAW_POLICY_PRESETS on recreate (#2682)
rluo8 May 15, 2026
b41687f
fix(sandbox): auto-respawn gateway when it exits unexpectedly (#2757)…
cjagwani May 15, 2026
73d30c9
fix(onboard): tighten UFW reachability remediation (#3533)
stevenrick May 15, 2026
88e3936
fix(onboard): suppress sandbox-base local build log on success (#3586)
laitingsheng May 15, 2026
8bc805f
feat(messaging): add WeChat (personal) channel for OpenClaw (#3186)
sandl99 May 15, 2026
03d1248
fix(ci): mark parity report script executable
jyaunches May 15, 2026
c3d3da9
Merge remote-tracking branch 'origin/main' into docs/e2e-parity-report
jyaunches May 15, 2026
fd4ad17
fix(ci): escape markdown backslashes
jyaunches May 15, 2026
b28b139
fix(e2e): run scenario workflow for real (#3493)
jyaunches May 15, 2026
aed7c39
Merge remote-tracking branch 'origin/main' into docs/e2e-parity-report
jyaunches May 15, 2026
2efc7f4
fix(rebuild): preserve disabledChannels across destroy/recreate (#3532)
sandl99 May 15, 2026
df582cc
Merge remote-tracking branch 'origin/main' into docs/e2e-parity-report
jyaunches May 15, 2026
c596a01
ci(nightly): restore Brev E2E workflow (#3401)
jyaunches May 15, 2026
750689e
Merge remote-tracking branch 'origin/main' into docs/e2e-parity-report
jyaunches May 15, 2026
ecd2708
fix(verify): retry gateway and dashboard probes; correct gateway-log …
laitingsheng May 15, 2026
943e546
Merge remote-tracking branch 'origin/main' into docs/e2e-parity-report
jyaunches May 15, 2026
41a10a8
Merge remote-tracking branch 'origin/fix/e2e-scenarios-live-execution…
jyaunches May 15, 2026
dc8b57d
fix(e2e): remove unused state backup skip helper
jyaunches May 15, 2026
40a99e8
fix(onboard): handle Ollama unified-memory probe failures on DGX Spar…
tiaz-hh May 15, 2026
da0ed94
Merge remote-tracking branch 'origin/main' into docs/e2e-parity-report
jyaunches May 15, 2026
56a987c
fix(e2e): refresh parity inventory
jyaunches May 15, 2026
17b06a4
feat(messaging): add WeChat (personal) channel for Hermes (#3512)
sandl99 May 15, 2026
308e838
Merge remote-tracking branch 'origin/main' into docs/e2e-parity-report
jyaunches May 15, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 38 additions & 6 deletions .coderabbit.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -183,6 +183,8 @@ reviews:
- `cloud-e2e` — full onboard + cloud inference
- `sandbox-operations-e2e` — multi-sandbox lifecycle
- `rebuild-openclaw-e2e` — workspace state survives rebuild
- `channels-stop-start-e2e` — channel disable/enable lifecycle across
rebuild with cached Telegram credentials
- `messaging-compatible-endpoint-e2e` — Telegram + compatible endpoint
inference.local regression
- `hermes-discord-e2e` — Hermes Discord config schema + placeholder
Expand All @@ -193,9 +195,37 @@ reviews:

To run selectively:
```
gh workflow run nightly-e2e.yaml --ref <branch> -f jobs=cloud-e2e,sandbox-operations-e2e,rebuild-openclaw-e2e,messaging-compatible-endpoint-e2e,hermes-discord-e2e,hermes-slack-e2e,openshell-gateway-upgrade-e2e
gh workflow run nightly-e2e.yaml --ref <branch> -f jobs=cloud-e2e,sandbox-operations-e2e,rebuild-openclaw-e2e,channels-stop-start-e2e,messaging-compatible-endpoint-e2e,hermes-discord-e2e,hermes-slack-e2e,openshell-gateway-upgrade-e2e
```

- path: "src/lib/onboard/channel-state.ts"
instructions: &e2e-channel-stop-start |
This file controls disabled channel resolution used during onboard
and rebuild. Changes affect whether `channels stop` persists across
sandbox destroy/recreate and whether `channels start` reattaches
cached credentials.

**E2E test recommendation:**
- `channels-stop-start-e2e` — Telegram channel stop/start lifecycle
across rebuild with cached credentials

To run selectively:
```
gh workflow run nightly-e2e.yaml --ref <branch> -f jobs=channels-stop-start-e2e
```

- path: "src/lib/state/onboard-session.ts"
instructions: *e2e-channel-stop-start

- path: "src/lib/actions/sandbox/rebuild.ts"
instructions: *e2e-channel-stop-start

- path: "src/lib/commands/sandbox/channels/**"
instructions: *e2e-channel-stop-start

- path: "test/e2e/test-channels-stop-start.sh"
instructions: *e2e-channel-stop-start

- path: "src/lib/actions/inference-set.ts"
instructions: |
This file switches the OpenShell inference route and patches the
Expand Down Expand Up @@ -245,15 +275,16 @@ reviews:

- path: "src/lib/deploy/**"
instructions: |
This file contains deployment lifecycle logic (start/stop,
cloudflared tunnel, uninstall).
This file contains deployment lifecycle logic (start/stop
cloudflared tunnel).

**E2E test recommendation:**
- `deployment-services-e2e` — backup/restore, start/stop, uninstall
- `tunnel-lifecycle-e2e` — start/stop cloudflared tunnel
- `state-backup-restore-e2e` — backup/restore workspace state

To run selectively:
```
gh workflow run nightly-e2e.yaml --ref <branch> -f jobs=deployment-services-e2e
gh workflow run nightly-e2e.yaml --ref <branch> -f jobs=tunnel-lifecycle-e2e,state-backup-restore-e2e
```

- path: "src/lib/state/sandbox.ts"
Expand All @@ -263,12 +294,13 @@ reviews:
lifecycle operations.

**E2E test recommendation:**
- `state-backup-restore-e2e` — backup/restore workspace state
- `snapshot-commands-e2e` — snapshot create/list/restore lifecycle
- `rebuild-openclaw-e2e` — workspace state survives rebuild

To run selectively:
```
gh workflow run nightly-e2e.yaml --ref <branch> -f jobs=snapshot-commands-e2e,rebuild-openclaw-e2e
gh workflow run nightly-e2e.yaml --ref <branch> -f jobs=state-backup-restore-e2e,snapshot-commands-e2e,rebuild-openclaw-e2e
```

- path: "src/lib/shields/**"
Expand Down
58 changes: 58 additions & 0 deletions .github/workflows/brev-nightly-e2e.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

name: brev-nightly-e2e

# Runs Brev launchable validation as a standalone nightly workflow while the
# main nightly-e2e workflow remains isolated from Brev platform flakiness.
#
# Suites:
# all credential-sanitization + telegram-injection
# messaging-providers Telegram + Discord provider/L7 proxy validation
# full install/onboard/inference/CLI path

on:
schedule:
- cron: "0 6 * * *"
workflow_dispatch:
inputs:
branch:
description: "Branch to test (default: ref used for this dispatch; schedule always tests main)"
required: false
default: ""
launchable_id:
description: "Published launchable ID override (empty = workflow/test default)"
required: false
default: ""
keep_alive:
description: "Keep Brev instances alive after tests (for SSH debugging)"
required: false
type: boolean
default: false

permissions:
contents: read

concurrency:
group: brev-nightly-e2e-${{ github.event_name }}-${{ github.event_name == 'workflow_dispatch' && github.ref || 'schedule' }}
cancel-in-progress: true

jobs:
brev-nightly-e2e:
if: github.repository == 'NVIDIA/NemoClaw'
strategy:
fail-fast: false
matrix:
test_suite: [all, messaging-providers, full]
uses: ./.github/workflows/e2e-branch-validation.yaml
with:
branch: ${{ github.event_name == 'schedule' && 'main' || inputs.branch || github.ref_name }}
test_suite: ${{ matrix.test_suite }}
use_launchable: true
use_published_launchable: true
launchable_id: ${{ github.event_name == 'workflow_dispatch' && inputs.launchable_id || '' }}
keep_alive: ${{ github.event_name == 'workflow_dispatch' && inputs.keep_alive || false }}
secrets:
BREV_API_KEY: ${{ secrets.BREV_API_KEY }}
BREV_ORG_ID: ${{ secrets.BREV_ORG_ID }}
NVIDIA_API_KEY: ${{ secrets.NVIDIA_API_KEY }}
7 changes: 6 additions & 1 deletion .github/workflows/e2e-parity-compare.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -144,11 +144,16 @@ jobs:
--report "$REPORT" \
"${STRICT_ARGS[@]}"

- name: Render coverage report
- name: Render parity and coverage reports
if: always()
run: |
mkdir -p .e2e/parity
bash test/e2e/runtime/coverage-report.sh > .e2e/parity/coverage-report.md
npx --no-install tsx scripts/e2e/render-parity-report.ts \
--parity-json .e2e/parity/parity-report.json \
--coverage-report .e2e/parity/coverage-report.md \
--output .e2e/parity/e2e-parity-report.md
cat .e2e/parity/e2e-parity-report.md >> "$GITHUB_STEP_SUMMARY"

- name: Upload parity artifacts
if: always()
Expand Down
105 changes: 89 additions & 16 deletions .github/workflows/nightly-e2e.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ on:
hermes-inference-switch-e2e, hermes-discord-e2e,
hermes-slack-e2e, sandbox-operations-e2e, inference-routing-e2e,
openclaw-inference-switch-e2e,
network-policy-e2e, deployment-services-e2e, diagnostics-e2e,
network-policy-e2e, state-backup-restore-e2e, tunnel-lifecycle-e2e, diagnostics-e2e,
credential-migration-e2e,
snapshot-commands-e2e, shields-config-e2e, rebuild-openclaw-e2e,
upgrade-stale-sandbox-e2e, rebuild-hermes-e2e,
Expand All @@ -83,7 +83,7 @@ on:
credential-sanitization-e2e, telegram-injection-e2e,
overlayfs-autofix-e2e, device-auth-health-e2e,
launchable-smoke-e2e, gpu-e2e, gpu-double-onboard-e2e,
brave-search-e2e
channels-stop-start-e2e, brave-search-e2e
required: false
type: string
default: ""
Expand Down Expand Up @@ -371,6 +371,45 @@ jobs:
path: /tmp/nemoclaw-e2e-messaging-compatible-endpoint-install.log
if-no-files-found: ignore

# ── Channels stop/start lifecycle E2E (#3462 Test 1) ─────────
# Regression coverage for #3453 (channels stop must actually disable the
# channel across rebuild) and #3381 (channels start must re-attach from
# the cached credential). Telegram-only — Discord/Slack walk the same
# disabledChannels filter; telegram is the cheapest regression gate.
channels-stop-start-e2e:
if: >-
github.repository == 'NVIDIA/NemoClaw' &&
(github.event_name != 'workflow_dispatch' ||
inputs.jobs == '' ||
contains(format(',{0},', inputs.jobs), ',channels-stop-start-e2e,'))
runs-on: ubuntu-latest
timeout-minutes: 60
steps:
- name: Checkout
uses: actions/checkout@v6
with:
ref: ${{ inputs.target_ref || github.ref }}

- name: Run channels stop/start lifecycle E2E test
env:
NVIDIA_API_KEY: ${{ secrets.NVIDIA_API_KEY }}
NEMOCLAW_NON_INTERACTIVE: "1"
NEMOCLAW_ACCEPT_THIRD_PARTY_SOFTWARE: "1"
NEMOCLAW_POLICY_TIER: "open"
NEMOCLAW_SANDBOX_NAME: "e2e-channels-stop-start"
GITHUB_TOKEN: ${{ github.token }}
TELEGRAM_BOT_TOKEN: "test-fake-telegram-token-stop-start-e2e"
TELEGRAM_ALLOWED_IDS: "123456789"
run: bash test/e2e/test-channels-stop-start.sh

- name: Upload install log on failure
if: failure()
uses: actions/upload-artifact@v4
with:
name: install-log-channels-stop-start
path: /tmp/nemoclaw-e2e-install.log
if-no-files-found: ignore

# ── Brave Search E2E (#2687) ─────────────────────────────────
# Validates the full Brave Search path with a real BRAVE_API_KEY:
# non-interactive onboard auto-enables web search, the brave network
Expand Down Expand Up @@ -1086,16 +1125,45 @@ jobs:
path: test-network-policy-*.log
if-no-files-found: ignore

# ── Deployment & Services E2E ────────────────────────────────
# TC-STATE-02: backup-workspace.sh lifecycle (backup → destroy → restore)
# TC-DEPLOY-01: nemoclaw start/stop (cloudflared tunnel)
# TC-DEPLOY-03: uninstall --keep-openshell (destructive, runs last in script)
deployment-services-e2e:
# ── Workspace Backup & Restore E2E ───────────────────────────
# TC-STATE-01: backup-workspace.sh lifecycle (backup → destroy → restore)
state-backup-restore-e2e:
if: >-
github.repository == 'NVIDIA/NemoClaw' &&
(github.event_name != 'workflow_dispatch' ||
inputs.jobs == '' ||
contains(format(',{0},', inputs.jobs), ',state-backup-restore-e2e,'))
runs-on: ubuntu-latest
timeout-minutes: 60
steps:
- name: Checkout
uses: actions/checkout@v6
with:
ref: ${{ inputs.target_ref || github.ref }}

- name: Run state backup/restore E2E test
env:
NVIDIA_API_KEY: ${{ secrets.NVIDIA_API_KEY }}
NEMOCLAW_NON_INTERACTIVE: "1"
NEMOCLAW_ACCEPT_THIRD_PARTY_SOFTWARE: "1"
run: bash test/e2e/test-state-backup-restore.sh

- name: Upload test log on failure
if: failure()
uses: actions/upload-artifact@v4
with:
name: state-backup-restore-test-log
path: test-state-backup-restore-*.log
if-no-files-found: ignore

# ── Tunnel Lifecycle E2E ─────────────────────────────────────
# TC-DEPLOY-01a/b/c: nemoclaw tunnel start / probe / stop (cloudflared tunnel)
tunnel-lifecycle-e2e:
if: >-
github.repository == 'NVIDIA/NemoClaw' &&
(github.event_name != 'workflow_dispatch' ||
inputs.jobs == '' ||
contains(format(',{0},', inputs.jobs), ',deployment-services-e2e,'))
contains(format(',{0},', inputs.jobs), ',tunnel-lifecycle-e2e,'))
runs-on: ubuntu-latest
timeout-minutes: 60
steps:
Expand All @@ -1104,20 +1172,19 @@ jobs:
with:
ref: ${{ inputs.target_ref || github.ref }}

- name: Run deployment services E2E test
- name: Run tunnel lifecycle E2E test
env:
NVIDIA_API_KEY: ${{ secrets.NVIDIA_API_KEY }}
NEMOCLAW_NON_INTERACTIVE: "1"
NEMOCLAW_ACCEPT_THIRD_PARTY_SOFTWARE: "1"
NEMOCLAW_SANDBOX_NAME: "e2e-deploy-svc"
run: bash test/e2e/test-deployment-services.sh
run: bash test/e2e/test-tunnel-lifecycle.sh

- name: Upload test log on failure
if: failure()
uses: actions/upload-artifact@v4
with:
name: deployment-services-test-log
path: test-deployment-services-*.log
name: tunnel-lifecycle-test-log
path: test-tunnel-lifecycle-*.log
if-no-files-found: ignore

# ── Diagnostics E2E ─────────────────────────────────────────
Expand Down Expand Up @@ -1948,6 +2015,7 @@ jobs:
docs-validation-e2e,
messaging-providers-e2e,
messaging-compatible-endpoint-e2e,
channels-stop-start-e2e,
brave-search-e2e,
kimi-inference-compat-e2e,
token-rotation-e2e,
Expand All @@ -1961,7 +2029,8 @@ jobs:
inference-routing-e2e,
openclaw-inference-switch-e2e,
network-policy-e2e,
deployment-services-e2e,
state-backup-restore-e2e,
tunnel-lifecycle-e2e,
diagnostics-e2e,
credential-migration-e2e,
snapshot-commands-e2e,
Expand Down Expand Up @@ -2039,6 +2108,7 @@ jobs:
docs-validation-e2e,
messaging-providers-e2e,
messaging-compatible-endpoint-e2e,
channels-stop-start-e2e,
brave-search-e2e,
kimi-inference-compat-e2e,
token-rotation-e2e,
Expand All @@ -2052,7 +2122,8 @@ jobs:
inference-routing-e2e,
openclaw-inference-switch-e2e,
network-policy-e2e,
deployment-services-e2e,
state-backup-restore-e2e,
tunnel-lifecycle-e2e,
diagnostics-e2e,
credential-migration-e2e,
snapshot-commands-e2e,
Expand Down Expand Up @@ -2187,6 +2258,7 @@ jobs:
docs-validation-e2e,
messaging-providers-e2e,
messaging-compatible-endpoint-e2e,
channels-stop-start-e2e,
brave-search-e2e,
kimi-inference-compat-e2e,
token-rotation-e2e,
Expand All @@ -2200,7 +2272,8 @@ jobs:
inference-routing-e2e,
openclaw-inference-switch-e2e,
network-policy-e2e,
deployment-services-e2e,
state-backup-restore-e2e,
tunnel-lifecycle-e2e,
diagnostics-e2e,
credential-migration-e2e,
snapshot-commands-e2e,
Expand Down
Loading
Loading