Skip to content

orb(ux): ORB publishes nothing for a median 8min / p90 22min while awaiting CI — invisible waits caused a failing-CI PR to be hand-merged #9042

Description

@JSONbored

While a PR waits for CI, ORB publishes nothing at all — no comment, no label, no check. prReadyForReview returns false and maybePublishPrPublicSurface is never reached, so the only trace is a low-signal github_app.review_deferred_ci_pending audit row nobody sees. To a maintainer the PR looks ignored.

Measured (live ledger, JSONbored/loopover, 2026-07-26, n=161 PRs)

Time from ORB first seeing a PR to its first published surface:

median p90 max avg
8.3 min 21.9 min 50.9 min 10.6 min

Most of that is legitimately waiting for validate-tests (~14 min). The problem is not the wait — it is that the wait is invisible. review_deferred_ci_pending fired 1,800 times across 342 distinct PRs in 48h, which also means a genuinely stuck PR (#9011) is indistinguishable from a healthy one.

Concrete harm, not just perception

PR #8944: opened 13:58, ORB deferred silently from 14:10 to 14:26. The maintainer, seeing no ORB output, merged it manually at 14:28:30. Its CI then completed: validate-tests FAILED at 14:47:14 and validate FAILED at 14:49:47. A PR with failing CI reached main specifically because ORB had not yet said anything.

That is the real cost of the silent window: it trains the maintainer to merge without waiting, which defeats the gate entirely. It is also the direct cause of most of the 29 contributor PRs the maintainer resolved by hand today.

Fix

  1. Publish an immediate "review pending" surface on first sighting — a panel comment stating ORB has the PR and is waiting on named checks, updated (not duplicated) as checks land. The panel upsert machinery already exists; this just needs to run before readiness rather than after.
  2. Make the defer audit event carry a distinguishable reason and elapsed time (ci_running vs missing_required_context — see orb(review): prReadyForReview defers forever and silently when a required CI context never reports — no surface, no label, no alert #9011), so healthy waits and stuck PRs are separable.
  3. Consider a check-run in in_progress state as the pending signal, which also gives branch protection something to hold on.

Acceptance

  • Every PR gets an ORB surface within one pass of being seen, stating what it is waiting for.
  • A maintainer can tell at a glance whether ORB is waiting or wedged.

Refs #9011 (the stuck case this silence hides), #9003.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.orbGittensory Orb related - maintainer self-hosting analytics.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions