feat(status): uptime and incident history, derived from persisted samples - #9990
Merged
Conversation
…ples #9983 shipped the live board and deliberately published no uptime figure: there was no source for one. Alertmanager serves ACTIVE alerts only, resolved ones are gone, there is no Prometheus on the box, and nothing persisted a history. Deriving a percentage from a single live sample would have been fabrication. This is that source -- one sample per component per cron tick -- and the aggregation over it. Incidents are DERIVED from contiguous non-operational runs rather than posted by hand, which is what #9747 means by "incidents appear without manual posting". Three ways a plausible implementation lies here, and what this does instead: THE WINDOW LYING ABOUT ITSELF. A "30-day uptime" over three days of samples is the same falsehood as a verifier reporting green after checking nothing. Every window carries `since`, `measured`, `unmeasured` and `partial`, so "99.9% over a full month" is distinguishable from "99.9% over the two days we have been recording". Partiality is decided by the oldest sample OVERALL, not the oldest one inside the window -- an in-window sample can never predate the window start, so reading it from there marks every window partial forever. That was a real bug in the first cut, caught by its own test. UNMEASURED TIME COUNTED AS UPTIME. An `unknown` sample means the source could not be read. Counting it as up inflates the figure exactly when we were blind; counting it as down invents an outage nobody saw. It is excluded from the percentage and reported separately, and stored rather than skipped -- dropping those rows would silently shrink the window. BLINDNESS MANUFACTURING INCIDENTS. An `unknown` tick neither opens nor closes an incident. Letting it close one would split a single outage in two and double the incident count on any night the alerting source was flaky. A run's severity is the worst status within it, so an escalation is one incident, not two. A run still open at the newest sample reports a null end rather than being closed at "now", which would assert a recovery nobody observed. An empty or unreadable history yields a null uptime -- no data must render as no claim, never 0% (reads as a total outage) or 100% (reads as perfect). The sampler rides every cron tick: a handful of INSERTs, no fan-out, no GitHub budget. Coarser sampling would make short incidents invisible. It reads through the SAME loader the public route uses, so the history and the live board are the same measurement rather than two read paths free to disagree. Retention is 35 days, deliberately EXCEEDING the widest reported window (30), so the 30-day figure never loses its own tail to the prune. That inequality is asserted in a test, not just commented, because widening the windows past it would break it silently. Two things the checkers caught and taught: the table is registered as raw-SQL-only rather than declared in the drizzle schema (a declaration nothing imports is a dead export), and it carries a second index leading with `sampled_at` because the prune deletes by age across all components while the read is per-component over a window. Closes #9985
Contributor
|
Important 🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨 ⏳ LoopOver is waiting…LoopOver has seen this pull request and is waiting on CI checks to finish before reviewing it. This comment will update once the review runs. 🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed · 🟨 Waiting |
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #9990 +/- ##
==========================================
- Coverage 91.88% 91.88% -0.01%
==========================================
Files 929 930 +1
Lines 113772 113827 +55
Branches 27455 27466 +11
==========================================
+ Hits 104539 104588 +49
- Misses 7936 7940 +4
- Partials 1297 1299 +2
Flags with carried forward coverage won't be shown. Click here to find out more.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The second slice of #9747, completing it. #9983 shipped the live board and deliberately published no uptime figure — Alertmanager serves active alerts only, resolved ones are gone, there is no Prometheus on the box, and nothing persisted a history. Deriving a percentage from a single live sample would have been fabrication.
This adds the source: one sample per component per cron tick, and the aggregation over it. Incidents are derived from contiguous non-operational runs rather than posted by hand — #9747's "incidents appear without manual posting".
Three ways this could lie, and what it does instead
The window lying about itself. A "30-day uptime" over three days of samples is the same falsehood as a verifier reporting green after checking nothing. Every window carries
since,measured,unmeasuredandpartial, so "99.9% over a full month" is distinguishable from "99.9% over the two days we've been recording".Unmeasured time counted as uptime. An
unknownsample means the source could not be read. Counting it as up inflates the figure exactly when we were blind; counting it as down invents an outage nobody saw. It is excluded from the percentage, reported separately, and stored rather than skipped — dropping those rows would silently shrink the window.Blindness manufacturing incidents. An
unknowntick neither opens nor closes an incident. Letting it close one would split a single outage in two and double the incident count on any night the alerting source was flaky. A run's severity is the worst status within it, so an escalation is one incident. A run still open at the newest sample reports a null end rather than being closed at "now", which would assert a recovery nobody observed.An empty or unreadable history yields a null uptime — no data must render as no claim, never
0%(reads as a total outage) or100%(reads as perfect).Wiring
try/catcharound the loop rather than.catch()on the promise —prepare()throws synchronously when the binding is unusable, and a promise-only guard lets that escape and fail the tick it rides on. Also caught by a test.Two things the checkers taught me
db:schema-drift:checkwanted a declaration;dead-exports:checkthen rejected it, correctly — nothing imports it. The right answer wasRAW_SQL_ONLY_TABLES, the same posture as the siblingdecision_*tables.retention.test.tsrequires an index leading with the retention column. Mine led withcomponent. The prune deletes by age across all components while the read is per-component over a window, so both indexes are genuinely needed.Verification
test/unit+test/integration: 25,963 passed, 0 failedtypecheck,db:schema-drift:check,db:migrations:check,db:migrations:immutable:check,ui:openapi:check,dead-exports:check,checkers-wired:check,coverage-boltons:check,ui:lintgreentest/unit/index.test.ts's per-tick job-set expectations updated — the new job legitimately runs every tickCloses #9985