feat(status): publish per-component service status from the existing alerting stack - #9984
Merged
Conversation
…alerting stack #9747's backend slice, taking option C: source the public board from the deployment's OWN Grafana-managed rules -> Alertmanager -> PagerDuty stack rather than adding a status-page-only probe. A second probe would be a second opinion about the same components, free to disagree with the one that actually pages a human -- and when they disagree, the public page is the one nobody is watching. NO NEW INFRASTRUCTURE, verified on the box rather than assumed. The app container already reaches Alertmanager on the compose network (`fetch( "http://alertmanager:9093/api/v2/alerts")` returns `[]` from inside the running container), and /etc/cloudflared/config.yml already routes `^/v1/public/.*$` on shots.loopover.ai to this app -- so the route is publicly reachable the moment it ships, with no tunnel edit and no new hostname. That last part matters: there is no cert.pem on the box, so a new hostname could not be minted there. UNREACHABLE IS `unknown`, NEVER `operational`. This is the module's whole correctness content. A status page that renders green because it could not reach its source is worse than no status page -- it says the thing is fine at exactly the moment nobody can confirm it. Every failure path lands on `unknown`: unset URL, network error, timeout, non-200, unparseable body, and a body that parses but is not an array. That last one is the subtle case: `{}` is valid JSON with no alerts in it, and reading it as "nothing firing" would publish green off a payload we failed to understand. `unknown` also outranks `operational` in the rollup, so a component we could not read is never averaged away by one we could. An unrecognised severity rounds UP to degraded. A firing alert whose severity we do not know is still firing, and rounding down would hide it. PUBLIC-SAFE BY CONSTRUCTION, not by filtering. Alert payloads routinely carry `job`, `instance`, `pod` and other host identifiers, so alerts are mapped onto a fixed component vocabulary and the originals dropped -- an unmapped `service` label is ignored rather than published verbatim. Whitelisting the shape is what keeps a new alert label from leaking through a filter nobody updated. The failure reason is a category, never the configured URL, and a test asserts the serialized response contains no host, port, instance or capacity token. 404s where no alerting source is configured (the hosted Worker) instead of publishing a board that reads "unknown" for every component forever. Cached for 15s, not the sibling surfaces' 60s: this is the endpoint people refresh DURING an incident, and a minute of stale "operational" is the wrong failure. Uptime percentages and incident history are deliberately NOT here -- Alertmanager serves active alerts only, there is no Prometheus on the box, and no sample table, so there is no honest source for "has it been healthy?" yet. Fabricating one from a single live sample is the option this declines. Closes #9983
Contributor
|
Important 🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨 ⏳ LoopOver is waiting…LoopOver has seen this pull request and is waiting on CI checks to finish before reviewing it. This comment will update once the review runs. 🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed · 🟨 Waiting |
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
loopover-ui | ac6c046 | Commit Preview URL Branch Preview URL |
Jul 31 2026, 03:28 AM |
Bundle ReportChanges will increase total bundle size by 1.89kB (0.02%) ⬆️. This is within the configured threshold ✅ Detailed changes
Affected Assets, Files, and Routes:view changes for bundle: loopover-uiAssets Changed:
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #9984 +/- ##
==========================================
- Coverage 91.88% 91.01% -0.88%
==========================================
Files 928 929 +1
Lines 113722 113772 +50
Branches 27436 27455 +19
==========================================
- Hits 104489 103545 -944
- Misses 7936 9125 +1189
+ Partials 1297 1102 -195
Flags with carried forward coverage won't be shown. Click here to find out more.
|
codecov/patch flagged 94% against the 99% gate. Branch coverage on
service-status.ts was 92.3%, with three arms of the per-component fold
unexercised -- all of them in the paths that decide what a malformed or
partial alert does:
• a `service` label that is not a string. Alertmanager's contract says
Record<string, string>, but this reads a parsed JSON body from a source that
can ship anything, and a non-string must take the same ignore path an
unmapped label takes rather than throwing on .toLowerCase();
• a firing alert with no usable `startsAt`. Still degraded -- an alert with no
timestamp is still an alert -- but no invented `since`;
• the KEEP arms of the earliest-wins comparison. The existing test put the
earlier alert second, which only exercises the replace arm; the mirror case
(earlier first, and a later untimestamped one) leaves half the comparison
free to invert unnoticed.
Branch coverage is now 100% (39/39), statements and lines already were.
No production code changed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The backend slice of #9747, taking option C — source the board from the deployment's own PagerDuty-integrated alerting stack.
Why C is cheap — verified on the box, not assumed
I'd previously called this "the largest change, needs a new egress path." That was wrong, and checking corrected it:
loopover-loopover-1container:fetch("http://alertmanager:9093/api/v2/alerts")→[]. Same compose network. Grafana's/api/healthanswers too./etc/cloudflared/config.yml:/v1/public/*route is publicly reachable the moment it ships — no tunnel edit, no new hostname. That last part matters: there's nocert.pemon the box, so a new hostname couldn't be minted there anyway.alertmanager.ymlroutes tooncall-pagerandteam-slack, rules managed in Grafana. Exactly the instrumentation site(status): public service-status surface — ORB and AMS fleet heartbeats and uptime #9747 says to reuse.Why the alerting stack rather than a new probe
A status-page-only probe is a second opinion about the same components, free to disagree with the one that actually pages a human — and when they disagree, the public page is the one nobody is watching. Reading the alerting stack means the page says what the on-call rotation already believes.
The correctness content: never report green when you couldn't check
A status page that renders "operational" because it couldn't reach its source is worse than no status page — it says the thing is fine at precisely the moment nobody can confirm it. Same false-green class as #9724's all-skip run. Every failure path lands on
unknown:unknown— "not configured"unknown— "unreachable"unknown— "returned an error"unknown— "unreadable"unknown— "unexpected shape"That last one is the subtle case:
{}is valid JSON containing no alerts, and reading it as "nothing firing" would publish green off a payload we failed to understand.Two more one-way choices:
unknownoutranksoperationalin the rollup, so a component we couldn't read is never averaged away by one we could.degraded. A firing alert with a severity we don't know is still firing; rounding down would hide it.Public-safe by construction, not by filtering
Alert payloads routinely carry
job,instance,podand other host identifiers. Rather than filtering those out, alerts are mapped onto a fixed component vocabulary and the originals dropped — an unmappedservicelabel is ignored, never published verbatim. Whitelisting the shape is what stops a new alert label leaking through a filter nobody updated. The failurereasonis a category, never the configured URL.A test asserts the serialized response contains none of
alertmanager,9093,instance,job,pod,localhost,cpu,memory,disk.Deliberately out of scope
Uptime percentages and incident history. Alertmanager's
/api/v2/alertsreturns active alerts only; there's no Prometheus on the box (metrics go via otel-collector) and no sample table — so there is no honest source for "has it been healthy?" yet. Fabricating an uptime figure from a single live sample is the option this declines. The follow-up is aservice_status_samplestable on the existing cron, with incidents derived from contiguous non-operational runs.Also: 404 where no alerting source is configured (the hosted Worker), rather than a board reading "unknown" forever. And a 15s cache, not the sibling surfaces' 60s — this is the endpoint people refresh during an incident.
Verification
test/unit+test/integration— 1334 files, 25,890 tests, all greentypecheck,selfhost:env-reference:check,cf-typegen:check,ui:openapi:check,docs:drift-check,dead-source-files:check,dead-exports:check,import-specifiers:check,coverage-boltons:check,server-manifest:check,ui-derived-types:check,checkers-wired:check,control-plane:openapi:check,contract:api-schemas:check,validate:mcpgreenroute-spec-ratchetsatisfied — the route is registered in the OpenAPI document, androute-authexempts it in the same PR as the route so the two can't driftopenapi.json,selfhost-env-reference.tsCloses #9983