Skip to content

orb(merge-train): a blocked head-of-line PR stalls every PR behind it — 5 PRs blocked 4h behind one manual-review label #9039

Description

@JSONbored

The merge train has no bypass for a head-of-line PR that is itself blocked, so one stuck PR stalls every mergeable PR behind it until a human intervenes. This is the cascade amplifier for #9009.

Confirmed production incident (2026-07-26, ledger-verified)

PR #8735 (contributor bitfathers94, opened 01:32Z) was labeled manual-review by the bot. Per #9009 that label makes the executor deny merge — and nothing removes it autonomously. #8735 then sat as the head of the merge train for four hours, and every mergeable PR behind it was denied:

For contrast, the only other train-wait episode in the same 48h (behind #8925) cleared in 20 seconds — that is the train working correctly. The pathology is specifically head-of-line PR blocked by a non-self-clearing hold.

Why it matters

The blocking condition and the train are independent subsystems, so every absorbing state in the system (#9009 manual-review, #9012 merge_blocked_sha, #9018 pause-stall, #9031 pending-closure) is silently multiplied by the number of PRs queued behind it. Throughput collapses to zero for the whole train, and nothing reports that the train is wedged — the denial events are the only trace.

Fix

  1. Skip, do not wait, when the head is not actually mergeable. A sibling that is held/labeled/merge-blocked is not "an older mergeable sibling" — it should be evicted from the train rather than treated as the thing to wait for. Re-check the head's live mergeability each pass and pass over it if it cannot proceed.
  2. Bound the wait. Cap how long any PR may sit behind one sibling; on expiry, evict the head and re-evaluate the train.
  3. Alert on a wedged train — N consecutive denials on the same blocking sibling should page, not just accumulate audit rows. Today a four-hour stall was invisible until a human happened to look.

Acceptance

  • A head-of-line PR carrying a blocking label is skipped and the PRs behind it merge normally.
  • A train wedged beyond the cap emits an operator alert.

Refs #9009 (root cause), #9012, #9003 (silent-decision invariant).

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.orbGittensory Orb related - maintainer self-hosting analytics.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions