Skip to content

Add VS Code extension E2E tests with Playwright + Docker#15407

Draft
mitchdenny wants to merge 32 commits intomicrosoft:mainfrom
mitchdenny:extension/e2e-tests
Draft

Add VS Code extension E2E tests with Playwright + Docker#15407
mitchdenny wants to merge 32 commits intomicrosoft:mainfrom
mitchdenny:extension/e2e-tests

Conversation

@mitchdenny
Copy link
Member

@mitchdenny mitchdenny commented Mar 19, 2026

Description

Adds end-to-end UI testing infrastructure for the Aspire VS Code extension using Playwright browser automation inside Docker containers.

What it does

A single comprehensive test (ExtensionShowsResourcesFromRunningAppHost) validates the full extension workflow:

  1. Launches VS Code via code serve-web in a Docker container with Docker-in-Docker
  2. Installs the locally-built Aspire CLI (native AOT) and VS Code extension (VSIX)
  3. Creates a new Aspire Starter App interactively using the CLI's hive mechanism with local packages
  4. Runs aspire start and verifies the extension's resource tree shows all expected resources (apiservice, cache, webfrontend)

Architecture

  • Docker container: .NET 10 SDK + Node.js 22 + Docker Engine (DinD) + VS Code + Hex1b terminal tool
  • Playwright: Automates the VS Code web UI (clicking, typing, screenshots, video recording)
  • Hex1b WebSocket: Remote terminal automation for interactive CLI commands inside the container
  • PROMPT_COMMAND trick: Structured prompt detection (same pattern as CLI E2E tests) for reliable command success/failure detection
  • Hive mechanism: Uses localhive.sh packages via ~/.aspire/hives/local/ for correct version testing

Prerequisites to run

# Build packages + native AOT CLI with matching version suffixes
./localhive.sh --native-aot

# Build the VS Code extension
./build.sh --build-extension

# Run the test (from worktree or main repo)
ASPIRE_ARTIFACTS_ROOT=/path/to/aspire dotnet test tests/Aspire.Extension.EndToEndTests/

Test artifacts produced

Each test run generates: screenshots at key milestones (01-08), video recording (WebM), terminal recording (asciinema .cast), and Playwright trace (.zip).

New files

File Purpose
ExtensionEndToEndTests.cs The E2E test with 6 phases
Infrastructure/AspireBuildArtifacts.cs Host artifact detection (CLI, VSIX, packages)
Infrastructure/Hex1b/RemoteTerminalSession.cs WebSocket terminal automation wrapper
Infrastructure/VsCodeContainer.cs Docker container lifecycle + DinD
Infrastructure/VsCodeWebFixture.cs xUnit fixture (container + Playwright)
Dockerfile.e2e-vscode Container image with VS Code + DinD
entrypoint-e2e-vscode.sh Container entrypoint (dockerd + VSIX install + code serve-web)

Checklist

  • Is this feature complete?
    • Yes. Ready to ship.
    • No. Follow-up changes expected.
  • Are you including unit tests for the changes and scenario tests if relevant?
    • Yes
    • No
  • Did you add public API?
    • Yes
      • If yes, did you have an API Review for it?
        • Yes
        • No
      • Did you add <remarks /> and <code /> elements on your triple slash comments?
        • Yes
        • No
    • No
  • Does the change make any security assumptions or guarantees?
    • Yes
      • If yes, have you done a threat model and had a security review?
        • Yes
        • No
    • No
  • Does the change require an update in our Aspire docs?

@github-actions
Copy link
Contributor

github-actions bot commented Mar 19, 2026 •
edited
Loading

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 15407

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 15407"

Mitch Denny and others added 24 commits March 20, 2026 10:37
Add VS Code extension E2E test scaffolding with Playwright
4ce9292 
- Dockerfile.e2e-vscode: Docker image running VS Code via 'code serve-web'
  with --without-connection-token for auth-free browser access
- VsCodeContainer.cs: Docker lifecycle (build, run, wait for ready, dispose)
- VsCodeWebFixture.cs: xUnit fixture combining Docker + Playwright with
  video recording, tracing, and screenshot support
- SmokeTests.cs: Smoke test verifying VS Code renders in headless Chromium
- Test project configured as Linux-only, excluded from Helix/AzDO

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add Aspire CLI install test using aspire.dev install script
c75ff43 
- Install GA Aspire CLI via 'curl -sSL https://aspire.dev/install.sh | bash'
- Use docker exec to poll for completion (xterm.js canvas can't be read from DOM)
- Verify aspire --version runs successfully inside container
- Screenshots captured: terminal-opened, install-complete, aspire-version

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add three install quality tests and Node.js to Docker image
6f73826 
- Theory-based test: release (GA), dev (daily), staging (prerelease)
- Each test cleans previous install, runs curl aspire.dev/install.sh
  with --quality flag, verifies aspire --version via docker exec
- Dockerfile: add Node.js 22 LTS (needed for VS Code extension host)
- Dockerfile: add curl explicitly for install script

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add Hex1b remote terminal support for E2E tests
2bea382 
Milestone 1 & 2: Docker + Hex1b diagnostics socket + workload adapter

- Install Hex1b.Tool in Docker image and expose socket directory via volume mount
- Implement DiagnosticsWorkloadAdapter: connects to hex1b diagnostics socket,
  performs attach handshake, streams o:/i:/r: frames
- Implement RemoteTerminalSession: wraps adapter + headless Hex1bTerminal for
  convenient test use (SendTextAsync, WaitForTextAsync, GetScreenText)
- Add socket permission fix (chmod 777 via docker exec) for host access
- Add two new tests:
  - Hex1bTerminalCreatesSocketViaMountedVolume: proves socket appears
  - RemoteTerminalSessionCanSendAndReceiveText: full round-trip echo test

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Refactor install tests to use Hex1b remote terminal
c96663c 
Replace docker exec + marker file approach with RemoteTerminalSession:
- Tests connect via diagnostics socket to read real terminal output
- WaitForAnyTextAsync detects install completion signals
- aspire --version verified through both terminal session and docker exec
- Asciinema recordings captured via hex1b --record flag and copied from container
- Each quality level produces: .webm video, .zip trace, .cast recording

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add fully interactive 'aspire new' project creation test
a966e19 
Milestone 5: Drive Spectre.Console prompts through Hex1b remote terminal

- New AspireNewCreatesProjectInteractively test: installs Aspire CLI, runs
  'aspire new' with no arguments, drives all interactive prompts
- Template selection, project name, output path, and template-specific
  prompts (localhost TLD, Redis cache, test project) all handled
- Additional prompts auto-detected and answered with defaults
- Project creation verified via docker exec (files exist)
- Full artifact suite: screenshots per prompt step, Playwright trace,
  asciinema recording from container
- Add scrollback support to RemoteTerminalSession (GetFullText,
  includeScrollback parameter on wait methods) — critical for long
  command output that scrolls past the visible 30-line screen

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Maximize terminal and linger 5s before test exit
3491f60 
- Add MaximizeTerminalPanelAsync helper: opens command palette, runs
  'View: Toggle Maximized Panel' to fill editor area with terminal
- Called before hex1b launch in all 4 terminal-based tests
- Add 5-second delay after final assertion so video captures show
  the completed result

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add build artifact detection and Docker infrastructure for integratio…
7253f51 
…n tests

M6: Follow the CLI E2E pattern — build on host, mount read-only into container.

- AspireBuildArtifacts: detects CLI binary, VSIX, NuGet packages, and
  nuget config from prior ./build.sh --bundle --build-extension --pack
- Dockerfile: add Docker CLI (for DCP), entrypoint script that installs
  VSIX via 'code --install-extension' before starting serve-web
- VsCodeContainer: accept optional artifacts + Docker socket mount,
  volume-mount CLI at /opt/aspire-cli, VSIX, packages, nuget config
- VsCodeWebFixture: auto-detect artifacts, skip integration tests when
  no local build is available

Existing tests continue to work without artifacts.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add full integration test: extension shows resources from running App…
3648263 
…Host

- Install CLI from local build artifacts, create project with aspire new
- Patch SDK version from GA to dev to enable backchannel sockets
- Restore with local NuGet packages, run aspire start
- Verify Aspire extension tree view shows apiservice, cache, webfrontend
- Add ASPIRE_ARTIFACTS_ROOT env var for worktree artifact detection
- Fix VSIX install: use correct --extensions-dir for serve-web
- All 8 tests passing (smoke + 3 install + interactive new + integration)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Upgrade Hex1b to 0.119.0 and increase terminal timeouts
239d838 
- Update Hex1b, Hex1b.McpServer, Hex1b.Tool from 0.116.0 to 0.119.0
- Add nuget.org package source for Hex1b* packages
- Update Dockerfile to install Hex1b.Tool 0.119.0
- Increase terminal-wrapper wait timeout from 30s to 60s to reduce
  flakiness under concurrent Docker load

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Migrate from UDS socket attach to WebSocket transport (--passthru --p…
2ba2108 
…ort)

Replace custom DiagnosticsWorkloadAdapter with Hex1b's built-in
RemoteTerminalWorkloadAdapter, connecting via WebSocket instead of
Unix domain sockets. This eliminates the socket volume mount and
simplifies the Docker setup.

Key changes:
- Delete DiagnosticsWorkloadAdapter.cs (replaced by built-in adapter)
- RemoteTerminalSession.ConnectAsync now takes a URI and uses
  WithRemoteTerminal() with retry logic for connection resilience
- VsCodeContainer: replace socket mount with port allocation and
  socat bridge (workaround for hex1b binding to loopback only)
- All tests updated: --attach → --passthru --port
- Dockerfile: add socat for port bridging

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Upgrade Hex1b to 0.120.0 and use --bind 0.0.0.0
e2ffee9 
Hex1b 0.120.0 adds --bind flag for the WebSocket listener, allowing
it to bind to 0.0.0.0 in containers. This removes the need for the
socat port bridge workaround.

Changes:
- Hex1b packages: 0.119.0 → 0.120.0 (Directory.Packages.props)
- Hex1b.Tool: 0.119.0 → 0.120.0 (Dockerfile)
- Remove socat from Dockerfile and VsCodeContainer
- All hex1b commands now use --bind 0.0.0.0
- Simplify AllocateHex1bPort (no more internal/exposed port split)

All 7 non-integration tests pass.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Consolidate to single E2E test with per-test output directories
addc56e 
- Remove 6 scaffolding/smoke tests (VsCodeLaunchesAndRendersWorkbench,
  Hex1bTerminalStartsWithWebSocketPort, RemoteTerminalSessionCanSendAndReceiveText,
  InstallAspireCliViaTerminal, AspireNewCreatesProjectInteractively)
- Keep only ExtensionShowsResourcesFromRunningAppHost (full integration test)
- Rename class SmokeTests → ExtensionEndToEndTests
- Refactor fixture: each test gets its own output directory under
  artifacts/testresults/extension-e2e/{TestName}/ with all artifacts
  (screenshots, traces, recordings, videos) in one place
- Number screenshot filenames (01-cli-installed, 02-project-created, etc.)
  for clear chronological ordering
- Each test creates its own browser context (videos/traces scoped per-test)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use 'code -a' for continuous VS Code session recording
c3ae761 
Replace the ?folder= URL approach (which causes full page reload and
terminal disconnection) with 'code -a /path' to add the project folder
to the workspace. This keeps the terminal, hex1b session, and video
recording running continuously throughout the entire test.

Key changes:
- All phases run in a single VS Code session without page reloads
- CLI install, aspire new, restore, and aspire start all happen in
  the same hex1b terminal session
- Workspace trust dialog dismissed automatically after code -a
- Default apphost notification from Aspire extension handled
- Late trust dialog safety check before Phase 6 sidebar interaction
- Test passes in ~115 seconds with full artifact suite

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use Docker official repo for CLI + Compose + BuildX
3739aa7 
Replace the docker.io Debian/Ubuntu package with Docker's official apt
repository, providing docker-ce-cli, docker-compose-plugin, and
docker-buildx-plugin. This ensures templates that use Docker containers
(e.g., Redis, PostgreSQL) work correctly inside the test container.

The base image is Ubuntu (noble), so we use Docker's Ubuntu repo URL.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Switch to Docker-in-Docker for container resources
ee2adf6 
Replace the host Docker socket mount with a full Docker-in-Docker setup
so that DCP-managed containers (e.g., Redis cache) run inside the test
container and share its network namespace. This fixes the cache resource
failing to start (red icon in the extension tree view).

Changes:
- Dockerfile: install docker-ce + containerd.io (full daemon), add
  VOLUME /var/lib/docker for overlay storage
- Entrypoint: start dockerd in background, wait up to 30s for ready
- Container runner: use --privileged flag, remove socket mount option
- VsCodeContainer: simplify constructor (remove mountDockerSocket param)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove artificial delays, add command palette safety and terminal focus
bf8b75e 
- Remove unnecessary Task.Delay calls and replace with WaitForSelectorAsync
- Reduce polling loop intervals (2000ms→500ms for creation, 5000ms→2000ms for resources)
- Add WaitForSelectorState.Hidden after MaximizeTerminalPanelAsync to ensure command palette closes
- Click terminal area before typing hex1b command to ensure focus
- Add hex1b process check diagnostic after WaitForHex1bAsync
- Replace fragile .split-view-view wait with simple delay
- Test passes in ~87s

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adopt PROMPT_COMMAND trick from CLI E2E tests
f9b643e 
Use the same PROMPT_COMMAND trick as Aspire.Cli.EndToEnd.Tests:
bash PROMPT_COMMAND sets PS1 to [N OK] $ or [N ERR:code] $
after each command, making command completion deterministic.

Changes:
- Add SequenceCounter class to track prompt sequence numbers
- Add SetupPromptAsync to install PROMPT_COMMAND in the shell
- Add WaitForSuccessPromptAsync, WaitForAnyPromptAsync,
  WaitForSuccessPromptFailFastAsync to RemoteTerminalSession
- Replace all WaitForTextAsync("~#") / WaitForAnyTextAsync(["~#",
  "root@"]) with prompt-based detection
- Remove echo marker test (prompt trick proves session works)
- Use WaitForSuccessPromptFailFastAsync for critical commands
  (CLI install) to fail immediately on error
- Use WaitForAnyPromptAsync for restore (allows failure to continue)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Reduce ConnectAsync delays: 1s→300ms fault check, 2s→500ms backoff, 1…
9289552 
…0s→3s content wait

Total test time dropped from ~110s to ~98s by reducing conservative
retry delays in the WebSocket connection loop. Also added timing
instrumentation to the hex1b connection phase for diagnosing lag.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Fix hex1b readiness check: use in-container ss instead of host TCP probe
d59de35 
Docker's userspace proxy accepts TCP connections even when hex1b isn't
listening yet, causing the WebSocket handshake to hang for ~30s. Now
WaitForHex1bAsync uses 'docker exec ss -tlnp' to verify hex1b is
actually listening inside the container before attempting the WebSocket
connection from the host. Added iproute2 to the Docker image for ss.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Fix 30s prompt delay: send Enter to trigger initial bash prompt rende…
8b40fb4 
…ring

The shell was running but its initial PS1 prompt wasn't rendered to
hex1b's virtual terminal. WaitForAnyTextAsync polled for 30s finding
nothing. Sending a single Enter keystroke nudges bash into displaying
its prompt immediately (0.2s vs 30s). Total test time: 65s (was 98s).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use local 13.2.0-dev template packages instead of GA 13.1.3
f6b3553 
Pass --source /opt/aspire/packages --version {version} to aspire new
so the generated project references the same dev packages that are
mounted into the container. Previously aspire new resolved templates
from nuget.org (13.1.3 GA) making the test validate the wrong thing.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use localhive.sh hive mechanism for package discovery
59c8e94 
Instead of --source/--version flags, use the CLI's built-in hive
mechanism (same as get-aspire-cli-pr.sh for PR builds):
- Set up hive symlink: ~/.aspire/hives/local/packages → /opt/aspire/packages
- Use --channel local to auto-select local packages
- CLI auto-generates NuGet.config from hive channel

Key changes:
- AspireBuildArtifacts: prefer Release config (localhive.sh output),
  search Aspire.Cli.Tool before Aspire.Cli for framework-dependent CLI
- Copy entire CLI directory (cp -a) since Cli.Tool is framework-dependent
- localhive.sh produces properly versioned prerelease packages
  (13.2.0-local.TIMESTAMP) that dotnet package search can discover

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Prefer native AOT CLI over framework-dependent
bbbcbfd 
Reorder FindCliPublishDirectory to search Aspire.Cli (native AOT)
before Aspire.Cli.Tool (framework-dependent), and prefer the native
subdirectory. The native AOT binary is self-contained so cp -a still
works (just copies 2 files instead of 80+).

Prerequisite: run ./localhive.sh --native-aot to produce a native AOT
CLI with matching version suffix.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mitchdenny mitchdenny force-pushed the extension/e2e-tests branch from deb2b0f to bbbcbfd Compare March 19, 2026 23:37
Mitch Denny and others added 4 commits March 20, 2026 11:41
Set up extension E2E tests for CI with per-test jobs
fc77a5e 
- Add SplitTestsOnCI, RequiresCliArchive, RequiresNugets, EnablePlaywrightInstall
  to the extension E2E test project
- Add IncludeExtensionE2ETests skip condition in TestEnumerationRunsheetBuilder
- Pass IncludeExtensionE2ETests flag in tests.yml enumerate step
- Pass enablePlaywrightInstall from matrix to run-tests.yml
- Propagate enablePlaywrightInstall through build-test-matrix.ps1

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove custom nuget.org source, upgrade Hex1b to 0.121.0
c311d44 
Hex1b packages resolve through the existing dotnet-public/dotnet-eng
wildcard mappings, so the explicit nuget.org source is unnecessary.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Fix Playwright browser install on CI
e6533bc 
Import Playwright.targets which provides the ProvisionBrowsersForPlaywright
build target that downloads chromium during build. Set
InstallBrowsersForPlaywright=true on CI builds.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Set PLAYWRIGHT_BROWSERS_PATH before launching browser
6374b60 
Browsers are installed to artifacts/bin/playwright-deps/ during build,
but at runtime Playwright defaults to ~/.cache/ms-playwright/. Use the
shared PlaywrightProvider.DetectAndSetInstalledPlaywrightDependenciesPath()
to auto-discover the repo-local browser install path.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Mitch Denny and others added 4 commits March 20, 2026 15:13
Upload extension E2E test artifacts (screenshots, videos, traces, casts)
9c8c314 
The extension tests output to artifacts/testresults/extension-e2e/ but
the CI upload step only captured testresults/**. Add artifacts/testresults/**
to the upload path and search it for .cast recordings too.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Wire up CLI archive + VSIX download for extension E2E tests on CI
a97e6fa 
The extension E2E test self-skipped because the CI job didn't have the
locally-built CLI binary or VSIX available. Add requiresExtensionArtifacts
flag that flows through the test matrix pipeline and triggers download +
extraction of:
  - cli-native-archives-linux-x64 → artifacts/bin/Aspire.Cli/.../publish/
  - aspire-extension VSIX → artifacts/packages/Debug/vscode/

This matches what AspireBuildArtifacts.Detect() expects to find.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Fix job name: extension_tests_win, not vscode_tests
7dcf624 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Wait for VS Code reload after 'code -a' folder add
fb9db5f 
When 'code -a' adds a folder to an empty workspace, VS Code reloads
the window (full navigation). The old execution context is destroyed,
causing QuerySelectorAsync to throw. Fix by:
1. WaitForLoadStateAsync after code -a to let navigation complete
2. WaitForSelectorAsync for .monaco-workbench to confirm reload done
3. Use WaitForSelector with timeout for notification handling instead
   of QuerySelector which races with async DOM updates

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@radical radical Awaiting requested review from radical radical will be requested when the pull request is marked ready for review radical is a code owner

@eerhardt eerhardt Awaiting requested review from eerhardt eerhardt will be requested when the pull request is marked ready for review eerhardt is a code owner

Copilot code review Copilot Awaiting requested review from Copilot Copilot will automatically review once the pull request is marked ready for review

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@mitchdenny