Skip to content

pr_outcome missing for bot-closed PRs — 40 of 66 detected reversals never reach fleet calibration #8823

Description

@JSONbored

Context

Found while fixing #8820. The fleet-calibration export (FLEET_QUERY, src/selfhost/orb-collector.ts) inner-joins gate_decision to pr_outcome in review_audit — a PR with no pr_outcome row is never exported at all, so it contributes to neither the numerator nor the denominator of any published accuracy metric.

Measured on the live self-host (2026-07-26):

This is not a stale-data artifact. Spot-checked two of the missing targets against GitHub:

PR GitHub state pr_outcome row
#8728 CLOSED 2026-07-26T01:19:47Z none
#8748 CLOSED 2026-07-26T02:36:34Z none

Both were genuinely bot-closed (they have agent.action.close audit events — that is exactly how recordSupersededReversals found them), yet no pr_outcome was ever written. So the outcome writer is missing cases the close actuator completes.

Because the dropped rows are disproportionately errors, their absence biases the published accuracy upward. With #8822 the number reads 93.6%; folding in the missing 40 would lower it further.

Requirements

  • Determine why pr_outcome is not written for these bot-closed PRs. Start at recordPrOutcome/the pull_request closed webhook path in src/review/outcomes-wire.ts (the writer is documented as firing "on a pull_request closed webhook") and compare against agent.action.close completions — a self-closed PR may not deliver a webhook the instance processes, or the handler may bail before the write.
  • Fix the gap so every terminal gate action records its realized outcome. A bot-initiated close must not depend on a webhook round-trip the instance might never see.
  • Backfill the missing pr_outcome rows for targets that have a completed agent.action.close/agent.action.merge but no outcome, so historical calibration is corrected (the fleet export re-sends on cursor rewind, and ingest upserts, so a backfill propagates safely).
  • Add an invariant test: a completed terminal gate action always yields a pr_outcome row.
  • Regression test for the specific shape above (bot-closed PR, no inbound webhook → outcome still recorded).

Test Coverage Requirements

99%+ patch coverage, branch-counted, on all changed src/** lines — both arms of every new conditional.

Expected Outcome

Every bot-closed and bot-merged PR carries a pr_outcome row; the fleet export stops dropping ~61% of detected reversals; published decision accuracy reflects the full error population rather than the subset that happened to get an outcome row.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions