Add CI-friendly textual progress heartbeat to MTP terminal reporter (raw log progress signal)

[Evangelink]

Summary

When running dotnet test (MTP) in AzDO (and any non-ANSI CI), users currently have a binary choice in the raw console log:

Mode	Per-test "passed" lines	Progress in raw log
--output normal (default)	❌ none (good)	❌ silent until summary (bad)
--output detailed	✅ one per test (noisy)	❌ still no time-based progress

--report-azdo-progress only updates the AzDO timeline view via ##vso[task.logdetail …] commands; it does not print anything human-readable to the raw build log, which is what you stare at when investigating a CI run.

We currently work around this in our own pipelines with --output detailed, which produces hundreds of lines like:

passed UnsupportedRunSettingsEntriesAreFlagged ("net10.0") (3s 768ms)
  from D:\a\_work\1\s\artifacts\bin\MSTest.Acceptance.IntegrationTests\Debug\net11.0\MSTest.Acceptance.IntegrationTests.dll (net11.0|x64)
passed UnsupportedRunSettingsEntriesAreFlagged_Localization ("fr-FR","it-IT",null,"fr-FR") (3s 756ms)
…

That's a lot of noise (failures get buried) and it still does not convey time-based progress — only completion order.

Motivation

The maintainer experience in CI is:

• "Did the test run hang?" requires scrolling 10k lines of passed-test names, or staring at the timeline view in a separate panel.
• Dropping --output detailed gives a clean log but no signal at all between the banner and the summary — for a 15‑minute test run that feels broken.
• The cursor-redraw progress (SimpleAnsi/AnsiIfPossible) is intentionally disabled in CI because raw log files can't display cursor movement, so today there is no progress at all in raw logs.

Proposed feature: CI-friendly textual progress heartbeat

Emit a single-line plain-text status line periodically (default ~30s) in SimpleAnsi and NoAnsi modes. Each line is appended to the log (no cursor magic), so it's readable in AzDO/GitHub Actions/Jenkins raw logs.

Example shape (subject to bikeshed):

[00:30] running … 124/? completed, 2 failed (active: MyClass.MyTest, MyOther.OtherTest)
[01:00] running … 312/? completed, 2 failed (active: SlowTest.Foo)
[01:30] running … 451/? completed, 5 failed (active: SlowTest.Foo, Bar.Baz)

Behavior:

• Default: enabled in CI (SimpleAnsi) and when --no-progress is not specified.
• Configurable: --ci-progress-interval <seconds> (or env var); --no-progress continues to suppress everything.
• Independent of --output: works whether or not passed tests are shown.
• Single line per emission so log filters / regex grep stay clean.
• Per-assembly variant when running multiple assemblies (matches existing ShowAssemblyStartAndComplete pattern).

Scope

This needs coordinated changes in two repos:

1. microsoft/testfx — MTP's TerminalTestReporter / SimpleAnsiTerminal / TerminalOutputDevice for the single-exe --no-build and dotnet run scenarios. Touchpoints:
   • src/Platform/Microsoft.Testing.Platform/OutputDevice/TerminalOutputDevice.cs:212-216 (currently forces ShowProgress=false for SimpleAnsi/NoAnsi).
   • src/Platform/Microsoft.Testing.Platform/OutputDevice/Terminal/TerminalTestReporterOptions.cs (new option).
   • src/Platform/Microsoft.Testing.Platform/OutputDevice/Terminal/SimpleAnsiTerminal.cs (heartbeat timer + line emission).
   • TerminalTestReporterCommandLineOptionsProvider + acceptance-test expectations (HelpInfoTests, HelpInfoAllExtensionsTests, MSBuild.KnownExtensionRegistration).
2. dotnet/sdk — the forked reporter under src/Cli/dotnet/Commands/Test/MTP/Terminal/ for the dotnet test (build → MTP) path. The SDK's TerminalTestReporter ctor force-disables showProgress when AnsiMode == SimpleAnsi; same logic gap.

Short-term mitigation (this repo only)

We can drop --output detailed from _CommonReleaseTestArgs in eng/pipelines/variables/test-args.yml:67 (plus the inline copies in azure-pipelines.yml:100 and eng/pipelines/steps/test-non-windows.yml:19) after the heartbeat lands, so our CI logs immediately benefit from the new signal without losing visibility.

Related work / prior art

• [MTP] CI logs unnecessarily long and hard to read #5712 — [MTP] CI logs unnecessarily long and hard to read (closed; addressed the 3‑second active-test spam, not the no-progress-in-CI gap).
• Report interesting state in progress bar #3639 — Report interesting state in progress bar (closed; about live progress, not CI heartbeat).
• [Parent] Terminal Reporter improvements and output control #8398 / [parent] Terminal Reporter Output and Progress Display Issues #8606 — [parent] Terminal Reporter improvements (closed parents).
• [RFC] Agent/LLM-efficient test output for Microsoft.Testing.Platform #8824 — [RFC] Agent/LLM-efficient test output for MTP (open; complementary).
• Add "no progress" and "no ANSI" environment variables #7647 — Add no-progress and no-ANSI env vars (closed).
• The SDK's --report-azdo-progress extension only writes ##vso[task.logdetail …] (timeline panel), not a readable line.

Open questions

1. Default emission interval: 15s, 30s, 60s?
2. Include active-test names in the line? (Privacy/security implications minimal here, but worth confirming.)
3. New CLI flag name vs. reusing --no-progress semantics: a single --no-progress should still suppress heartbeats.
4. Should the heartbeat be enabled by default outside CI when ANSI cursor progress is unavailable (e.g., output redirected to a file)? Probably yes.
5. Should each assembly emit its own heartbeat in multi-assembly runs (SDK path), or one aggregate heartbeat?

cc @Evangelink @nohwnd

[Evangelink]

Design update (2026-06-15) — converged proposal

After offline brainstorm, the original periodic 30s heartbeat proposal in the issue body has been superseded by a smaller, event-driven design that avoids the fixed-cadence trap of #6753. Capturing here so the parent reflects the current direction; this issue stays open as the tracking umbrella.

Core principle

Speak only when something is interesting. Three event-driven emitters, never a fixed-cadence ticker:

• E1 — silence-driven heartbeat: emit one line only after N seconds with no test completions (default 30s). Healthy fast suite emits zero lines.
• E2 — slow-test surfacing: when a single test crosses a threshold (default 60s) with exponential backoff (60s → 2m → 4m …).
• E3 — failures inline (verify already works in SimpleAnsi/NoAnsi).

Architectural call

CI-specific surfaces (collapsible groups, native annotations, GitHub \, AzDO timeline mirror) do not live in the platform — each CI gets its own extension package, symmetric with the existing Microsoft.Testing.Extensions.AzureDevOpsReport. Core only sees one bit: "are we in any CI" (already detected today at TerminalOutputDevice.cs:191).

Sub-issues (recommended PR order)

1. Add --progress {auto|on|off} and deprecate --no-progress in MTP terminal reporter #9138 — Add --progress {auto|on|off}; deprecate --no-progress. Prerequisite for everything else.
2. Silence-driven progress heartbeat renderer for SimpleAnsi/NoAnsi terminal modes #9139 — Silence-driven heartbeat renderer for SimpleAnsi/NoAnsi + new IProgressEnricher hook surface. Depends on Add --progress {auto|on|off} and deprecate --no-progress in MTP terminal reporter #9138.
3. Azure DevOps history-driven slow-test threshold enricher #9140 — AzDO Test Analytics history → per-test thresholds + p95/p99 annotation. Depends on Silence-driven progress heartbeat renderer for SimpleAnsi/NoAnsi terminal modes #9139, can run in parallel with 4 & 5.
4. Extend AzureDevOpsReport extension: log groups, failure annotations, raw-log progress mirror #9141 — Extend AzureDevOpsReport with ##[group], ##vso[task.issue], raw-log mirror. Depends on Silence-driven progress heartbeat renderer for SimpleAnsi/NoAnsi terminal modes #9139, parallel with 3 & 5.
5. New extension: Microsoft.Testing.Extensions.GitHubActionsReport (groups, annotations, step summary) #9142 — New Microsoft.Testing.Extensions.GitHubActionsReport package. Depends on Silence-driven progress heartbeat renderer for SimpleAnsi/NoAnsi terminal modes #9139, parallel with 3 & 4.

Reminders for implementers (not separate issues)

• IProgressEnricher (introduced in Silence-driven progress heartbeat renderer for SimpleAnsi/NoAnsi terminal modes #9139) is a public API addition → needs entries in the relevant PublicAPI.Unshipped.txt per the repo convention.
• --help preamble in Add --progress {auto|on|off} and deprecate --no-progress in MTP terminal reporter #9138 should briefly explain the relationship between --ansi / --progress / --output (three look-alike flags).
• Coordinate the --no-progress deprecation warning with the matching dotnet/sdk fork port to avoid users seeing the warning before the SDK has --progress.
• After Silence-driven progress heartbeat renderer for SimpleAnsi/NoAnsi terminal modes #9139 lands, drop --output detailed from eng/pipelines/variables/test-args.yml:67, azure-pipelines.yml:100, eng/pipelines/steps/test-non-windows.yml:19 — short-term mitigation from this issue's body.
• All five children update the four help/info acceptance test expectations per AGENTS.md (HelpInfoTests, HelpInfoAllExtensionsTests, MSBuild.KnownExtensionRegistration, MSTest's HelpInfoTests) and run /t:UpdateXlf on the owning project for any new resource strings.