Skip to content

Fix flaky SubLevelCommandNameShouldBeSentToTelemetry test on macOS#54747

Merged
MichaelSimons merged 2 commits into
mainfrom
michaelsimons/fix-telemetry-test-macos-sublevel-comman
Jun 15, 2026
Merged

Fix flaky SubLevelCommandNameShouldBeSentToTelemetry test on macOS#54747
MichaelSimons merged 2 commits into
mainfrom
michaelsimons/fix-telemetry-test-macos-sublevel-comman

Conversation

@MichaelSimons

Copy link
Copy Markdown
Member

Summary

Fixes #54746

The SubLevelCommandNameShouldBeSentToTelemetry test was intermittently failing on the macOS arm64 CI leg (osx.15.arm64.open).

Root Cause

InstantiateCommand.ExecuteAsync dynamically adds template names (e.g., "console") as subcommands to the static Parser.RootCommand's NewCommand. This mutation persists for the process lifetime. When another test in the same xUnit process triggers template instantiation before this test runs, "console" becomes a registered subcommand — causing System.CommandLine to classify it as TokenType.Command instead of TokenType.Argument. The AllowListToSendVerbSecondVerbFirstArgument telemetry rule then reports subcommand="CONSOLE", argument="" instead of the expected argument="CONSOLE".

The intermittent nature is due to non-deterministic xUnit test ordering, which varies by platform/scheduler.

Fix

Parse against a fresh DotNetCommandDefinition() instance instead of the static Parser.RootCommand singleton. This guarantees the parser tree has not been mutated by prior template instantiations.

Testing

  • Build verified locally
  • The fix ensures deterministic behavior regardless of test execution order

The test was intermittently failing on macOS arm64 because it used the
static Parser.RootCommand which can be mutated by other tests.
InstantiateCommand.ExecuteAsync dynamically adds template names (e.g.
'console') as subcommands to the static parser tree. When this mutation
occurs before the telemetry test runs, 'console' is classified as
TokenType.Command instead of TokenType.Argument, causing the test to
fail.

Fix by parsing against a fresh DotNetCommandDefinition instance which
is guaranteed to not have dynamically-added template subcommands.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings June 12, 2026 15:09

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes intermittent failures in TelemetryFilterTests.SubLevelCommandNameShouldBeSentToTelemetry on macOS by avoiding reliance on the process-wide static CLI parser tree, which can be mutated by other tests (notably via template instantiation adding dynamic dotnet new subcommands).

Changes:

  • Switch the test to parse using a freshly constructed DotNetCommandDefinition() instead of the static Microsoft.DotNet.Cli.Parser.RootCommand.
  • Keep the existing Parser.ParserConfiguration so parsing behavior (POSIX bundling/response file handling) remains consistent with other tests.

@dsplaisted dsplaisted left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So Parser.Parse isn't thread-safe for us? Are there other tests that call the same API?

@MichaelSimons

MichaelSimons commented Jun 12, 2026

Copy link
Copy Markdown
Member Author

So Parser.Parse isn't thread-safe for us? Are there other tests that call the same API?

Parser.Parse itself is thread-safe — it doesn't mutate the parser tree. The issue is that InstantiateCommand.ExecuteAsync (line 250) mutates the static tree after parsing by calling args.NewOrInstantiateCommand.Subcommands.Add(templateCommandToRun). This adds template short names (like "console") as subcommands to the NewCommand.

In production this is fine because each CLI invocation is a separate process. But in-process during tests, if any test executes a dotnet new <template> command (not just parses), the static tree is permanently mutated for that process.

Other tests that call Parser.Parse (workload, tool, build, sdk check, etc.) are not affected because:

  1. Their commands' subcommands are never dynamically mutated
  2. The "new" command is the only one where InstantiateCommand adds subcommands at execution time

This specific test is the only one that parses ["new", "<template-name>"] and relies on the template name being classified as an Argument token. The workload/tool tests work because "install" is a statically-registered subcommand regardless of mutation.

@joeloff joeloff left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are there other tests that follow a similar pattern that could be at risk?

Remove the built-in VersionOption from the fresh DotNetCommandDefinition
before parsing. DotNetCommandDefinition extends RootCommand which adds a
default --version option, and the DotNetCommandDefinition constructor adds
its own custom one. In production, Parser.NormalizeRootOptions (private)
handles this removal, so the test must replicate that step.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@MichaelSimons

Copy link
Copy Markdown
Member Author

Are there other tests that follow a similar pattern that could be at risk?

I did ask copilot and this was the only occurrence it found.

@MichaelSimons MichaelSimons merged commit 6a30209 into main Jun 15, 2026
25 checks passed
@MichaelSimons MichaelSimons deleted the michaelsimons/fix-telemetry-test-macos-sublevel-comman branch June 15, 2026 12:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SubLevelCommandNameShouldBeSentToTelemetry test is flaky on macOS

4 participants