Skip to content

fix(incidents): a membership row outliving its incident must not gate the next one - #483

Merged
passcod merged 1 commit into
mainfrom
fix/stranded-incident-membership
Aug 4, 2026
Merged

fix(incidents): a membership row outliving its incident must not gate the next one#483
passcod merged 1 commit into
mainfrom
fix/stranded-incident-membership

Conversation

@passcod

@passcod passcod commented Aug 4, 2026

Copy link
Copy Markdown
Member

🤖 A server can sit red with no incident open and nothing paging, permanently.

is_issue_in_open_incident counted incident_issues rows by left_at IS NULL alone, never consulting the incident's closed_at. Closing an incident doesn't stamp left_at on the members that never left — a sub-failure contributor is held attached for context, and the close paths only touch the incidents row — so those rows outlive their incident and the count read them as live membership.

An issue that already looks attached never opens an incident. Its next effective failure arrives with was_in = true and should_leave = false, so it misses both the join arm and the leave arm and lands in the catch-all, whose only action is un-lingering incidents with closed_at IS NULL — which excludes the closed one. Zero rows updated, nothing opens. No later event can clear the stale row, so it never recovers on its own; reconcile_open_incidents calls the same helper and repairs nothing.

The leave arm selected its membership row the same way, so it comes along. An issue can hold a stranded row and a live row at the same time, and stamping the stranded one weighs remaining_open against a long-closed incident, abandoning the open one with no live members and no Slack resolve.

Impact

At the time of diagnosis prod had 207 stranded issues across 51 servers and 107 closed incidents, accruing continuously since early June. One server had a disk_free check effectively failing for seven weeks — monitored, grouped, never paged.

This is the read-side fix, deployable on its own: it makes every stranded row inert immediately, since membership is re-read on each evaluation. Stamping left_at when an incident closes, and backfilling the existing rows, follow separately.

… the next one

is_issue_in_open_incident counted incident_issues rows by left_at IS NULL
alone, never consulting the incident's closed_at. Closing an incident
doesn't stamp left_at on the members that never left — a sub-failure
contributor is held attached for context, and the close paths only touch
the incidents row — so those rows outlive their incident and the count
read them as live membership.

An issue that already looks attached never opens an incident: its next
effective failure arrives with was_in = true and should_leave = false,
lands in the no-op arm, and the only thing that arm does is un-linger
incidents where closed_at IS NULL, which excludes the closed one. Nothing
opens, and no later event can clear the stale row, so the server stays
red with nothing paging. In production this had stranded 207 issues
across 51 servers, one of them failing unattended for seven weeks.

The leave arm picked its membership row the same way, so it comes along:
an issue can hold a stranded row and a live row at once, and stamping the
stranded one weighs remaining_open against a long-closed incident and
abandons the open one with no live members and no Slack resolve.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@passcod
passcod enabled auto-merge August 4, 2026 22:13
@passcod
passcod disabled auto-merge August 4, 2026 22:14
@passcod
passcod merged commit d82be4a into main Aug 4, 2026
6 checks passed
@passcod
passcod deleted the fix/stranded-incident-membership branch August 4, 2026 22:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant