[cli] Add E2E sample upgrade tests for aspire-samples#15421
Draft
mitchdenny wants to merge 14 commits intomainfrom
Draft
[cli] Add E2E sample upgrade tests for aspire-samples#15421mitchdenny wants to merge 14 commits intomainfrom
mitchdenny wants to merge 14 commits intomainfrom
Conversation
Add infrastructure for testing that PR/CI builds of Aspire can upgrade and run external repos from dotnet/aspire-samples. Includes: - SampleUpgradeHelpers.cs: Shared helpers for clone, update, run, and verify workflows (CloneSampleRepoAsync, AspireUpdateInSampleAsync, AspireRunSampleAsync, StopAspireRunAsync, VerifyHttpEndpointAsync) - SampleUpgradeAspireWithNodeTests.cs: First sample test targeting the aspire-with-node sample (Node.js + .NET + Redis) Each test class becomes its own CI job via SplitTestsOnCI. The flow is: install CLI from PR → git clone aspire-samples → aspire update → aspire run → verify → Ctrl+C stop. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
|
🚀 Dogfood this PR with:
curl -fsSL https://raw.githubusercontent.com/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 15421Or
iex "& { $(irm https://raw.githubusercontent.com/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 15421" |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds end-to-end test infrastructure to validate that Aspire CLI PR/CI builds can upgrade and run external samples from dotnet/aspire-samples, establishing a reusable pattern starting with the aspire-with-node sample.
Changes:
- Added reusable
Hex1bTerminalAutomatorextension helpers for cloning sample repos, runningaspire update, running/stoppingaspire run, and basic HTTP verification. - Added the first sample upgrade E2E test covering
aspire-with-nodeupgrade + run flow inside the existing Docker-based E2E harness.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| tests/Aspire.Cli.EndToEnd.Tests/SampleUpgradeAspireWithNodeTests.cs | New E2E test that installs the CLI, clones aspire-samples, upgrades aspire-with-node, runs it, and stops it. |
| tests/Aspire.Cli.EndToEnd.Tests/Helpers/SampleUpgradeHelpers.cs | New shared helpers to support sample-upgrade E2E scenarios (clone/update/run/stop/endpoint checks). |
| .RightText(" ERR:"); | ||
| if (errorSearcher.Search(snapshot).Count > 0) | ||
| { | ||
| return true; |
There was a problem hiding this comment.
In AspireUpdateInSampleAsync, an error prompt ([N ERR:...]) causes the wait loop to return, but the method then increments the counter and continues as if the update succeeded. This can let failing aspire update runs pass silently and lead to confusing downstream failures. Consider tracking whether an ERR prompt was observed (similar to WaitForSuccessPromptFailFastAsync) and throwing an exception when it occurs (ideally including the sequence number / guidance to check the terminal recording).
Suggested change
| return true; | |
| throw new InvalidOperationException( | |
| $"aspire update reported an error at sequence [{expectedCounter}]. " + | |
| "Check the terminal recording for details."); |
Comment on lines
+24
to
+29
| var workspace = TemporaryWorkspace.Create(output); | ||
|
|
||
| using var terminal = CliE2ETestHelpers.CreateDockerTestTerminal( | ||
| repoRoot, installMode, output, | ||
| mountDockerSocket: true, | ||
| workspace: workspace); |
There was a problem hiding this comment.
TemporaryWorkspace implements IDisposable and deletes its temp directory on disposal. This test creates a workspace but never disposes it, which can leak temp directories over repeated runs (local dev and CI). Consider changing this to using var workspace = TemporaryWorkspace.Create(output); so cleanup happens after the Docker terminal is disposed.
- Handle "Perform updates? [y/n]" confirmation prompt (was causing timeout) - Add VerifySampleWasUpgradedAsync helper that checks the csproj no longer contains the original version and dumps the updated csproj to the recording - Call verification step in the aspire-with-node test Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Pass --channel pr-{N} when in PullRequest install mode so aspire update
resolves packages from the PR hive instead of stable nuget.org
- Add handlers for NuGet.config prompts that appear with explicit channels:
"Which directory for NuGet.config file?" and "Apply these changes to NuGet.config?"
- These prompts only appear when using explicit (non-implicit) channels
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…hase The aspire update --channel command uses a temp NuGet config for the search phase but the apply phase (dotnet add package) needs the PR hive source in the project NuGet config. Pre-creating NuGet.config ensures packages can be resolved during apply. Also navigate to sample directory before update and increase update timeout. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
aspire update --channel has issues with dotnet package add failing to resolve PR hive packages during the apply phase. Instead: 1. Run aspire update (implicit channel) for structural migration 2. In PR mode, detect the PR version from the hive and use sed to replace all Aspire version strings in the csproj 3. Set up NuGet.config with the PR hive source for restore/build 4. Verify dotnet restore succeeds before running Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The nupkg filename pattern was case-sensitive and failed to match. Use grep -oE with a version regex pattern instead of sed on a specific package name. Also list hive packages for diagnostics and add a guard that fails fast if version detection returns empty. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
aspire run already restores as part of building. The explicit restore was a debugging checkpoint that is no longer needed and wastes ~30s. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Instead of parsing terminal output with grep to verify upgrades, mount a host temp directory into the Docker container. This lets the test read the csproj via normal file I/O and use Assert.DoesNotContain / Assert.Contains for cleaner, more reliable verification. Removes the VerifySampleWasUpgradedAsync terminal helper in favor of direct File.ReadAllTextAsync on the mounted path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Instead of upgrading to stable then using sed to replace versions, pass --channel directly to aspire update so it resolves packages from the PR hive. This tests the actual intended upgrade path. Removes SetupPrHiveNuGetConfigAsync and UpgradeToPrVersionAsync helpers since aspire update handles NuGet.config creation itself. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
aspire update --channel pr-{N} correctly updates the SDK version
and creates the NuGet.config, but the apply phase (dotnet package add)
fails for individual PackageReference entries due to a known issue
where --configfile is not passed to the underlying NuGet restore.
After aspire update, read the csproj from the mounted volume,
extract the PR version from the SDK attribute (which was set correctly),
and use sed to fix any remaining package references still on old versions.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
When using an explicit channel (e.g., a PR hive), aspire update creates a NuGet.config with the channel's package source for the search phase, but the apply phase calls 'dotnet package add' without making that config discoverable. Since 'dotnet package add' doesn't support --configfile, use the NuGet config directory as the working directory (same pattern as InstallTemplateAsync). Changes: - DotNetCliRunner: Add AddPackageAsync overload with nugetConfigDirectory parameter; use it as working directory when provided - ProjectUpdater: Store _nugetConfigDirectory field during explicit channel config creation; pass it through to UpdatePackageReferenceInProject - Remove sed workaround from E2E test now that the fix is in place - Update test mocks to implement new interface method Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
When using an explicit channel (e.g., a PR hive), the NuGet.config's package source mapping routes Aspire* packages exclusively to the hive source. During 'dotnet package add', NuGet does an implicit restore that fails because other Aspire packages still at old versions don't exist in the hive source. Fix by: 1. Setting --no-restore on each 'dotnet package add' when using an explicit channel to avoid restore failures from partial upgrades 2. Running a single 'dotnet restore' after all packages are updated, when all versions are consistent Also default TestDotNetCliRunner.RestoreAsync to succeed when no callback is set, matching the pattern used by GetNuGetConfigPathsAsync. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…pgrade tests - Add useHostNetwork parameter to CreateDockerTestTerminal (sets c.Network = "host") - Modify AspireRunSampleAsync to capture and return dashboard URL from terminal output - Add DashboardVerificationHelpers with Playwright-based dashboard verification - Add PollEndpointAsync for host-side HTTP endpoint polling with retry - Add Microsoft.Playwright package reference for browser-based dashboard testing - Update SampleUpgradeAspireWithNodeTests to verify dashboard shows expected resources - Dashboard screenshots saved to testresults/screenshots/ for CI artifact upload - Update CLI E2E testing skill with sample upgrade test documentation Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…re only The NuGetConfigMerger already creates the NuGet.config in a discoverable ancestor directory, so the working directory change on AddPackageAsync was unnecessary. The only real bug was the implicit restore conflict from package source mappings during dotnet package add for explicit channels. Simplified to: - _isExplicitChannel boolean flag (no DirectoryInfo field) - Pass noRestore: true for explicit channels - Final dotnet restore after all packages updated - No IDotNetCliRunner interface changes Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
|
🎬 CLI E2E Test Recordings — 53 recordings uploaded (commit View recordings
📹 Recordings uploaded automatically from CI run #23334857945 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR introduces E2E sample upgrade tests that validate
aspire updatecan upgrade external repos to PR/CI builds of Aspire, and fixes two bugs inaspire update --channelthat were blocking this scenario.Bug Fixes in
aspire update --channelNuGet config directory not passed to
dotnet package add: When using an explicit channel (e.g.,--channel pr-15421), the NuGet config with the channel's hive package source was created in a specific directory, butdotnet package addwas invoked from the project directory where it couldn't discover that config. Fixed by passing the NuGet config directory as the working directory fordotnet package add.Package source mapping causes restore failures during individual package adds: The NuGet config's
<packageSourceMapping>routesAspire*packages exclusively to the hive source. Duringdotnet package add's implicit restore, OTHER Aspire packages still at their old versions can't be found in the hive (which only has the new channel versions). Fixed by using--no-restoreon eachdotnet package addwhen using an explicit channel, then running a singledotnet restoreafter all packages are updated.Sample Upgrade E2E Test Infrastructure
SampleUpgradeHelpers.cs: Shared helpers for cloning repos, runningaspire updatewith interactive prompt handling, runningaspire runwith dashboard URL capture, and stopping the apphost.DashboardVerificationHelpers.cs: Playwright-based dashboard verification — navigates to the dashboard, verifies expected resources are present, takes a screenshot for CI artifacts. Also providesPollEndpointAsyncfor host-side HTTP endpoint polling with retry logic.--network hostmode so services started byaspire runinside the container are directly accessible from the test process for Playwright and HttpClient verification.SampleUpgradeAspireWithNodeTests.cs: First sample test — clonesdotnet/aspire-samples, upgradesaspire-with-nodeto the PR build, verifies the dashboard showscache,weatherapi, andfrontendresources, and takes a dashboard screenshot.Validation
aspire runstarts successfullyFixes # (issue)
Checklist