ops: escalate a workflow that has failed on consecutive runs

[JSONbored]

Closes #10146

selfhost.yml caught migration 0209's SQLite-only AUTOINCREMENT (#10138) correctly, on the very first push — the real-Postgres "applies every migration" test failed exactly as designed. It then stayed red across five consecutive runs while PRs kept merging, and the breakage was ultimately found by deploying the resulting image and watching the Orb crash-loop.

Nothing was wrong with the test. The gap is that selfhost.yml is push-to-main only, so its failure blocks nothing and pages no one — a red run is indistinguishable from one somebody is already handling.

Second instance of a solved class

#9951 hit exactly this with the publish workflows (MCP and miner failing on every main commit for as far back as the run history went, from a one-line missing build step). Its own comment says why it went unnoticed:

nobody noticed because the only signal was a ::warning:: nobody reads and a red check that looks like release noise

That fix landed as ~30 lines of inline bash inside mcp-release-please.yml. Needing the same thing in a second workflow is the point at which it stops being one workflow's business — so this moves it to scripts/escalate-workflow-outage.ts, and mcp-release-please.yml now calls that instead of carrying its own copy. One implementation, not two that drift.

Behaviour

• Consecutive failures at the head of the run history, threshold 3. A deterministic failure fails identically every time, so one red run is a flake; below threshold this stays silent deliberately, because an alert that fires on every transient red is how an alert gets muted.
• Once per outage — an open tracking issue is reused rather than a new one filed per commit.
• Never fails its caller. The workflow it runs in has already failed; turning "couldn't read the run history" into a second red is noise on top of the real problem.
• selfhost.yml gets an escalate-persistent-failure job gated on failure() — never always(), a success must file nothing — with issues: write, the one permission build-boot deliberately lacks.

The arithmetic, and why it is unit-tested

leadingNonSuccessCount has one case that is easy to get exactly backwards: a window with no success anywhere. indexOf("success") returns -1, and -1 treated as a count reports "no failures" for a workflow that has never once succeeded in its recorded history — precisely the shape #9951 found and the shape selfhost.yml was in for five runs. Returning runs.length there is what makes the worst state escalate instead of reading as healthy.

Also pinned: cancelled / timed_out / null count as non-successes, so a cancelled run cannot silently reset the streak and suppress the alert; and a success at the head reports 0 however bad the history behind it, so a fixed workflow stops alerting immediately.

Verification

• Full suite green (26,257 passed, 0 failed); typecheck, dead-exports, dead-source-files, checkers-wired, actionlint all clean.
• 7 unit tests over the pure helpers.

Behaviour change worth flagging

The tracking issue title becomes workflow-generic (workflow outage: X has failed on consecutive runs) rather than publish-specific. I checked before changing it: no open issue currently uses the old title, so no reuse is broken.

What this does not do

It does not make selfhost.yml pre-merge. That was a deliberate cost decision (#9951-era, ~100 runs/week at ~5 min) and CI runtime is still the binding constraint. This makes the post-merge signal actionable rather than moving the check — the pre-merge counterpart for the specific dialect class is the 11ms invariant added in #10143.

[superagent-security]

Superagent found 1 supply chain issue(s)

[loopover-orb]

Important

🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨

⏳ LoopOver is waiting…

LoopOver has seen this pull request and is waiting on CI checks to finish before reviewing it. This comment will update once the review runs.

_{🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed · 🟨 Waiting}

[superagent-security]

Superagent didn't find any vulnerabilities or security issues in this PR.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.99%. Comparing base (07df5ca) to head (e0aa785).
⚠️ Report is 7 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@           Coverage Diff           @@
##             main   #10147   +/-   ##
=======================================
  Coverage   91.99%   91.99%           
=======================================
  Files         931      931           
  Lines      114009   114009           
  Branches    27524    27524           
=======================================
  Hits       104886   104886           
  Misses       7823     7823           
  Partials     1300     1300           

Flag	Coverage Δ
backend	95.67% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.