FluentValidation real-repo diagnostics test: timeout budget is 5 minutes short, and the 18-error baseline is a brittle pin

[MelbourneDeveloper]

Two separate problems in real-repo-fluentvalidation.test.ts › diagnostics round-trip: a broken generic constraint surfaces and clears

1. The test's own timeout is smaller than the waits it performs

this.timeout(180_000);                                  //  3 minutes allowed
await waitForDocumentSymbols(uri, 120_000);             //  2 minutes
const baseline = await waitForStableErrorBaseline(uri, 120_000, 18);  // +2
await waitForError(uri, 120_000, ...);                  // +2
await waitForErrorBaseline(uri, baseline, 120_000);     // +2

Four sequential waits of 120s each is a worst case of 480 seconds inside a 180-second budget. Under load, mocha kills the test before any inner wait can reach its own deadline, and the resulting failure is an opaque mocha timeout rather than the specific assertion that would identify which stage stalled.

We were lucky in the observed failure: the second wait hit its own 120s deadline first, so we got the clear Error diagnostic baseline never stabilized message instead of a generic timeout.

Either raise the test budget above the sum of its waits, or shorten the individual waits so they fit.

2. The 18-error baseline encodes current imperfect behaviour

const baseline = await waitForStableErrorBaseline(uri, 120_000, 18);
assert.strictEqual(baseline.length, 18, 'pinned FluentValidation baseline must be complete');

FluentValidation is pinned at tag 12.1.1 — a released library. It should not have 18 genuine compile errors in src/FluentValidation/IValidator.cs. That 18 is the count of diagnostics our LSP currently produces, i.e. a snapshot of what it fails to resolve, not a property of the source.

That makes the number sensitive to restore state, SDK version, and how much of the solution Roslyn has finished loading. waitForStableErrorBaseline only returns once current.length >= minimumErrors and the set has been unchanged for 2s, so if the real figure settles at 17 the loop can never return and the test fails on timeout with no indication that the count, not the timing, was wrong.

Worth either deriving the expected set from the source rather than pinning a magic count, or asserting on the specific diagnostics that matter and their identity, rather than on a total.

Observed

First full-suite run to complete on the cleanup branch (earlier ones were cancelled by pushes): 766 passing, this one failing. All other 23 CI checks green.

Notes

waitForStableErrorBaseline is new on that branch, so it has no track record to distinguish flake from systematic failure.