Skip to content

feat(status): uptime and incident history, derived from persisted samples - #9990

Merged
JSONbored merged 1 commit into
mainfrom
site/service-status-history-9985
Jul 31, 2026
Merged

feat(status): uptime and incident history, derived from persisted samples#9990
JSONbored merged 1 commit into
mainfrom
site/service-status-history-9985

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

The second slice of #9747, completing it. #9983 shipped the live board and deliberately published no uptime figure — Alertmanager serves active alerts only, resolved ones are gone, there is no Prometheus on the box, and nothing persisted a history. Deriving a percentage from a single live sample would have been fabrication.

This adds the source: one sample per component per cron tick, and the aggregation over it. Incidents are derived from contiguous non-operational runs rather than posted by hand — #9747's "incidents appear without manual posting".

Three ways this could lie, and what it does instead

The window lying about itself. A "30-day uptime" over three days of samples is the same falsehood as a verifier reporting green after checking nothing. Every window carries since, measured, unmeasured and partial, so "99.9% over a full month" is distinguishable from "99.9% over the two days we've been recording".

Partiality is decided by the oldest sample overall, not the oldest one inside the window. In-window samples are filtered to >= startMs, so they can never predate the window start — reading partiality from there marks every window partial forever, including ones with years of history. That was a real bug in my first cut, caught by its own test.

Unmeasured time counted as uptime. An unknown sample means the source could not be read. Counting it as up inflates the figure exactly when we were blind; counting it as down invents an outage nobody saw. It is excluded from the percentage, reported separately, and stored rather than skipped — dropping those rows would silently shrink the window.

Blindness manufacturing incidents. An unknown tick neither opens nor closes an incident. Letting it close one would split a single outage in two and double the incident count on any night the alerting source was flaky. A run's severity is the worst status within it, so an escalation is one incident. A run still open at the newest sample reports a null end rather than being closed at "now", which would assert a recovery nobody observed.

An empty or unreadable history yields a null uptime — no data must render as no claim, never 0% (reads as a total outage) or 100% (reads as perfect).

Wiring

  • Sampler rides every cron tick: a handful of INSERTs, no fan-out, no GitHub budget. Coarser sampling makes short incidents invisible.
  • It reads through the same loader the public route uses, so the history and the live board are one measurement rather than two read paths free to disagree.
  • Best-effort write, with try/catch around the loop rather than .catch() on the promise — prepare() throws synchronously when the binding is unusable, and a promise-only guard lets that escape and fail the tick it rides on. Also caught by a test.
  • Retention is 35 days, deliberately exceeding the widest window (30), so the 30-day figure never loses its own tail to the prune. That inequality is asserted in a test, not just commented, because widening the windows past it would break it silently.

Two things the checkers taught me

  • db:schema-drift:check wanted a declaration; dead-exports:check then rejected it, correctly — nothing imports it. The right answer was RAW_SQL_ONLY_TABLES, the same posture as the sibling decision_* tables.
  • retention.test.ts requires an index leading with the retention column. Mine led with component. The prune deletes by age across all components while the read is per-component over a window, so both indexes are genuinely needed.

Verification

  • 25 new tests, weighted toward the dishonest-output paths
  • Full test/unit + test/integration: 25,963 passed, 0 failed
  • typecheck, db:schema-drift:check, db:migrations:check, db:migrations:immutable:check, ui:openapi:check, dead-exports:check, checkers-wired:check, coverage-boltons:check, ui:lint green
  • test/unit/index.test.ts's per-tick job-set expectations updated — the new job legitimately runs every tick

Closes #9985

…ples

#9983 shipped the live board and deliberately published no uptime figure: there
was no source for one. Alertmanager serves ACTIVE alerts only, resolved ones are
gone, there is no Prometheus on the box, and nothing persisted a history.
Deriving a percentage from a single live sample would have been fabrication.

This is that source -- one sample per component per cron tick -- and the
aggregation over it. Incidents are DERIVED from contiguous non-operational runs
rather than posted by hand, which is what #9747 means by "incidents appear
without manual posting".

Three ways a plausible implementation lies here, and what this does instead:

  THE WINDOW LYING ABOUT ITSELF. A "30-day uptime" over three days of samples is
  the same falsehood as a verifier reporting green after checking nothing. Every
  window carries `since`, `measured`, `unmeasured` and `partial`, so "99.9% over
  a full month" is distinguishable from "99.9% over the two days we have been
  recording". Partiality is decided by the oldest sample OVERALL, not the oldest
  one inside the window -- an in-window sample can never predate the window
  start, so reading it from there marks every window partial forever. That was a
  real bug in the first cut, caught by its own test.

  UNMEASURED TIME COUNTED AS UPTIME. An `unknown` sample means the source could
  not be read. Counting it as up inflates the figure exactly when we were blind;
  counting it as down invents an outage nobody saw. It is excluded from the
  percentage and reported separately, and stored rather than skipped -- dropping
  those rows would silently shrink the window.

  BLINDNESS MANUFACTURING INCIDENTS. An `unknown` tick neither opens nor closes
  an incident. Letting it close one would split a single outage in two and double
  the incident count on any night the alerting source was flaky. A run's severity
  is the worst status within it, so an escalation is one incident, not two. A run
  still open at the newest sample reports a null end rather than being closed at
  "now", which would assert a recovery nobody observed.

An empty or unreadable history yields a null uptime -- no data must render as no
claim, never 0% (reads as a total outage) or 100% (reads as perfect).

The sampler rides every cron tick: a handful of INSERTs, no fan-out, no GitHub
budget. Coarser sampling would make short incidents invisible. It reads through
the SAME loader the public route uses, so the history and the live board are the
same measurement rather than two read paths free to disagree.

Retention is 35 days, deliberately EXCEEDING the widest reported window (30), so
the 30-day figure never loses its own tail to the prune. That inequality is
asserted in a test, not just commented, because widening the windows past it
would break it silently.

Two things the checkers caught and taught: the table is registered as
raw-SQL-only rather than declared in the drizzle schema (a declaration nothing
imports is a dead export), and it carries a second index leading with
`sampled_at` because the prune deletes by age across all components while the
read is per-component over a window.

Closes #9985
@loopover-orb

loopover-orb Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Important

🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨

⏳ LoopOver is waiting…

LoopOver has seen this pull request and is waiting on CI checks to finish before reviewing it. This comment will update once the review runs.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed · 🟨 Waiting

@JSONbored JSONbored self-assigned this Jul 31, 2026
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@JSONbored
JSONbored merged commit a7673e2 into main Jul 31, 2026
4 checks passed
@JSONbored
JSONbored deleted the site/service-status-history-9985 branch July 31, 2026 06:00
@codecov

codecov Bot commented Jul 31, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 89.28571% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.88%. Comparing base (b73347a) to head (5a12f45).
⚠️ Report is 1 commits behind head on main.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
src/queue/job-dispatch.ts 0.00% 4 Missing ⚠️
src/selfhost/service-status-history.ts 95.74% 0 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #9990      +/-   ##
==========================================
- Coverage   91.88%   91.88%   -0.01%     
==========================================
  Files         929      930       +1     
  Lines      113772   113827      +55     
  Branches    27455    27466      +11     
==========================================
+ Hits       104539   104588      +49     
- Misses       7936     7940       +4     
- Partials     1297     1299       +2     
Flag Coverage Δ
backend 95.67% <89.28%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
src/api/routes.ts 95.62% <100.00%> (+<0.01%) ⬆️
src/db/retention.ts 98.87% <ø> (ø)
src/index.ts 98.24% <100.00%> (+0.01%) ⬆️
src/selfhost/queue-common.ts 98.65% <ø> (ø)
src/types.ts 100.00% <ø> (ø)
src/selfhost/service-status-history.ts 95.74% <95.74%> (ø)
src/queue/job-dispatch.ts 97.50% <0.00%> (-1.99%) ⬇️

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

site(status): uptime and incident history — sample the status board so 'has it been healthy?' has a source

1 participant