fix(incidents): a membership row outliving its incident must not gate the next one - #483
Merged
Merged
Conversation
… the next one is_issue_in_open_incident counted incident_issues rows by left_at IS NULL alone, never consulting the incident's closed_at. Closing an incident doesn't stamp left_at on the members that never left — a sub-failure contributor is held attached for context, and the close paths only touch the incidents row — so those rows outlive their incident and the count read them as live membership. An issue that already looks attached never opens an incident: its next effective failure arrives with was_in = true and should_leave = false, lands in the no-op arm, and the only thing that arm does is un-linger incidents where closed_at IS NULL, which excludes the closed one. Nothing opens, and no later event can clear the stale row, so the server stays red with nothing paging. In production this had stranded 207 issues across 51 servers, one of them failing unattended for seven weeks. The leave arm picked its membership row the same way, so it comes along: an issue can hold a stranded row and a live row at once, and stamping the stranded one weighs remaining_open against a long-closed incident and abandons the open one with no live members and no Slack resolve. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 A server can sit red with no incident open and nothing paging, permanently.
is_issue_in_open_incidentcountedincident_issuesrows byleft_at IS NULLalone, never consulting the incident'sclosed_at. Closing an incident doesn't stampleft_aton the members that never left — a sub-failure contributor is held attached for context, and the close paths only touch theincidentsrow — so those rows outlive their incident and the count read them as live membership.An issue that already looks attached never opens an incident. Its next effective failure arrives with
was_in = trueandshould_leave = false, so it misses both the join arm and the leave arm and lands in the catch-all, whose only action is un-lingering incidents withclosed_at IS NULL— which excludes the closed one. Zero rows updated, nothing opens. No later event can clear the stale row, so it never recovers on its own;reconcile_open_incidentscalls the same helper and repairs nothing.The leave arm selected its membership row the same way, so it comes along. An issue can hold a stranded row and a live row at the same time, and stamping the stranded one weighs
remaining_openagainst a long-closed incident, abandoning the open one with no live members and no Slack resolve.Impact
At the time of diagnosis prod had 207 stranded issues across 51 servers and 107 closed incidents, accruing continuously since early June. One server had a
disk_freecheck effectively failing for seven weeks — monitored, grouped, never paged.This is the read-side fix, deployable on its own: it makes every stranded row inert immediately, since membership is re-read on each evaluation. Stamping
left_atwhen an incident closes, and backfilling the existing rows, follow separately.