CI silently skips ~23% of the test suite, including the absolute-path regression guards

[juemerson-at-purestorage]

Summary

The tooling test suite — drift report, capability map, field/cmdlet map, value-enum map, param tools — gates itself on the presence of tools/specs/. That directory is a build cache and is gitignored, so it never exists on a CI runner. Every test behind that gate reports as skipped, the job reports success, and the coverage loss is invisible in the run summary.

Roughly 23% of the suite does not execute in CI. Measured on main at 34170e2 against CI run 30331643319:

Environment	Passed	Failed	Skipped
Local (tools/specs/ populated)	650	5	2
CI (tools/specs/ absent)	503	0	154

Every failure visible locally is inside that skipped set, so a branch can be locally red and CI-green at the same time. The absolute-path regression guards added in PR #62 are among the skipped tests, on both matrix legs.

Mechanism

Tests/Build-PfbApiDriftReport.Tests.ps1:398-399:

$script:hasRealArtifacts = (Test-Path $realCapabilityMapPath) -and (Test-Path $realFieldCmdletMapPath) -and
    (Test-Path $realSpecsDir) -and (Get-ChildItem $realSpecsDir -Filter 'fb*.json' -ErrorAction SilentlyContinue)

Of the three inputs, two are tracked in git (Data/PfbCapabilityMap.json, Reports/PfbFieldCmdletMap.json). Only tools/specs/ is missing — and it is excluded twice, by .gitignore:36 (tools/specs/) and again by the broader .gitignore:46 (tools/).

The same pattern repeats at Tests/Build-PfbApiDriftReport.Tests.ps1:515-516 and across five test files:

File	Specs-gated guard sites
Tests/Build-PfbApiDriftReport.Tests.ps1	22
Tests/Build-PfbFieldCmdletMap.Tests.ps1	8
Tests/Build-PfbCapabilityMap.Tests.ps1	3
Tests/Build-PfbValueEnumMap.Tests.ps1	3
Tests/PfbApiDriftTools.Tests.ps1	3

The graceful-skip behaviour is correct in intent — a contributor without the cache should not see a wall of spurious failures. The defect is that CI inherits the contributor's escape hatch and reports the result as a pass.

Recommendation

1. Decouple the assertions that do not need the spec cache

The three absolute-path guards assert against a freshly regenerated report, which is why they need specs. They should instead assert against the committed Reports/PfbApiDriftReport.json, which is tracked and needs no cache at all.

This is both cheaper and more honest: the standing requirement protects what is committed to the repository, so the test should read what is committed to the repository. Re-pointing them removes the specs dependency entirely and makes them run on every push, on every runner.

Audit the remaining gated tests on the same question — an assertion that only reads tracked artifacts does not belong behind a specs gate.

2. Restore tools/specs/ in CI so the rest of the tooling tests run

Populate the cache on at least one matrix leg, via actions/cache keyed on the spec-set version, seeded from the capability-map workflow's artifact. Keep the graceful skip for local contributors; the point is that CI stops being one of them.

Rejected alternative: committing the cache. 29 files, 49.4 MB raw. It compresses well, but it grows with every REST release and undoes the deliberate decision to treat specs as a build input rather than source.

3. Fail loudly on regression (small, worth doing with the above)

Add a CI assertion that the skip count stays under a threshold. The failure mode here was not that skipping is wrong — it is that skipping was unobservable. A ceiling makes the next occurrence a red build instead of a silent hole.

Acceptance

• The three absolute-path guards execute (not skip) in CI on both matrix legs.
• CI skip count drops to single digits on the leg with the cache restored.
• Deliberately breaking ConvertTo-PfbRepoRelativePath produces a red CI build, not a green one.
• Skip count exceeding the agreed ceiling fails the job.

Notes

The failures visible locally are separate pre-existing stale acceptance pins, not in scope here — but they are a second illustration of the same problem, since CI has never once reported them.

Blocked on: nothing.
Owner: Justin.