Fix flaky SubLevelCommandNameShouldBeSentToTelemetry test on macOS#54747
Conversation
The test was intermittently failing on macOS arm64 because it used the static Parser.RootCommand which can be mutated by other tests. InstantiateCommand.ExecuteAsync dynamically adds template names (e.g. 'console') as subcommands to the static parser tree. When this mutation occurs before the telemetry test runs, 'console' is classified as TokenType.Command instead of TokenType.Argument, causing the test to fail. Fix by parsing against a fresh DotNetCommandDefinition instance which is guaranteed to not have dynamically-added template subcommands. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR fixes intermittent failures in TelemetryFilterTests.SubLevelCommandNameShouldBeSentToTelemetry on macOS by avoiding reliance on the process-wide static CLI parser tree, which can be mutated by other tests (notably via template instantiation adding dynamic dotnet new subcommands).
Changes:
- Switch the test to parse using a freshly constructed
DotNetCommandDefinition()instead of the staticMicrosoft.DotNet.Cli.Parser.RootCommand. - Keep the existing
Parser.ParserConfigurationso parsing behavior (POSIX bundling/response file handling) remains consistent with other tests.
dsplaisted
left a comment
There was a problem hiding this comment.
So Parser.Parse isn't thread-safe for us? Are there other tests that call the same API?
In production this is fine because each CLI invocation is a separate process. But in-process during tests, if any test executes a Other tests that call
This specific test is the only one that parses |
joeloff
left a comment
There was a problem hiding this comment.
Are there other tests that follow a similar pattern that could be at risk?
Remove the built-in VersionOption from the fresh DotNetCommandDefinition before parsing. DotNetCommandDefinition extends RootCommand which adds a default --version option, and the DotNetCommandDefinition constructor adds its own custom one. In production, Parser.NormalizeRootOptions (private) handles this removal, so the test must replicate that step. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
I did ask copilot and this was the only occurrence it found. |
Summary
Fixes #54746
The
SubLevelCommandNameShouldBeSentToTelemetrytest was intermittently failing on the macOS arm64 CI leg (osx.15.arm64.open).Root Cause
InstantiateCommand.ExecuteAsyncdynamically adds template names (e.g., "console") as subcommands to the staticParser.RootCommand'sNewCommand. This mutation persists for the process lifetime. When another test in the same xUnit process triggers template instantiation before this test runs, "console" becomes a registered subcommand — causing System.CommandLine to classify it asTokenType.Commandinstead ofTokenType.Argument. TheAllowListToSendVerbSecondVerbFirstArgumenttelemetry rule then reportssubcommand="CONSOLE", argument=""instead of the expectedargument="CONSOLE".The intermittent nature is due to non-deterministic xUnit test ordering, which varies by platform/scheduler.
Fix
Parse against a fresh
DotNetCommandDefinition()instance instead of the staticParser.RootCommandsingleton. This guarantees the parser tree has not been mutated by prior template instantiations.Testing