fix(review): finalize CI-stuck silence, post a waiting placeholder, and name every silent AI-review skip by JSONbored · Pull Request #9180 · JSONbored/loopover

JSONbored · 2026-07-27T06:15:26Z

Summary

orb(review): prReadyForReview defers forever and silently when a required CI context never reports — no surface, no label, no alert #9011 — prReadyForReview deferred a missing-required-context PR forever past its 2-minute surfacing cap instead of falling through to the existing finalize-past-cap block. A required context that never appears keeps ciState at pending (never passed), so finalizing here can never produce a would-merge disposition — it's safe to stop deferring silently. A stuckReason (missing_required_context vs ci_running) is threaded through the audit/log calls so the two causes are distinguishable.
orb(ux): ORB publishes nothing for a median 8min / p90 22min while awaiting CI — invisible waits caused a failing-CI PR to be hand-merged #9042 — the silent wait before a review runs (measured live: 8.3min median, 21.9min p90, up to 50.9min) now gets an immediate "waiting" panel comment upserted through the same sticky-panel marker the final verdict later replaces, instead of leaving the PR with no visible signal until CI settles.
orb(review): panel re-run checkbox is unreliable end-to-end — clicks silently lost, forced passes can silently skip the review #9000 (partial) — root-causes the #8972 incident where a forced ("Re-run LoopOver review") retrigger completed with none of the events a forced pass should ever emit anywhere in the audit trail. shouldStartAiReviewForAdvisory folds in shouldRequirePublicAiReviewForAdvisory's hard gate (AI review mode off, an ineligible author, no head SHA, an AI kill-switch off, no AI binding) — a forced retrigger does not bypass this gate, and every one of its branches previously returned false with zero audit trail anywhere. A new reason-resolving function names the exact cause, and a catch-all audit event fires whenever the AI review doesn't run for a reason none of the other (already-audited) paths cover. Two adjacent silent branches in the frozen/paused/one-shot reuse logic — a hold condition true but nothing published to actually reuse — get their own named "unavailable" events for the same reason.

This PR only closes the exhaustive-auditing and root-cause portion of #9000; the receipt-feedback / lost-webhook-click recovery sweep it also asks for is a separate, larger piece of work and is not implemented here.

Test plan

npx tsc --noEmit -p tsconfig.json --incremental false — clean
npx vitest run test/unit/queue.test.ts test/unit/queue-2.test.ts test/unit/queue-3.test.ts test/unit/queue-4.test.ts test/unit/queue-5.test.ts test/unit/queue-lifecycle-guards.test.ts test/unit/ai-review-advisory.test.ts test/unit/auto-review-wiring.test.ts test/unit/reputation-wiring.test.ts test/unit/pr-panel-retrigger-pending-force-review.test.ts test/unit/pr-labeled-public-surface.test.ts — 1126 passed
New regression tests cover: the orb(review): prReadyForReview defers forever and silently when a required CI context never reports — no surface, no label, no alert #9011 finalize-past-cap policy reversal, the orb(ux): ORB publishes nothing for a median 8min / p90 22min while awaiting CI — invisible waits caused a failing-CI PR to be hand-merged #9042 waiting placeholder, and all four new orb(review): panel re-run checkbox is unreliable end-to-end — clicks silently lost, forced passes can silently skip the review #9000 named-skip audit events (frozen/paused/one-shot "unavailable" + the catch-all hard-gate skip on a forced retrigger)
Rebased onto latest main after landing (picked up fix(selfhost): rescue orphaned dispositions, break the freeze-deadlock, and release AI-review locks before a hard kill can #9174/fix(webhook): scope the label public-surface trigger to disposition labels only #9176/fix(orb): bind identity to credentials instead of renameable logins or trusted request bodies #9177/fix(trust): commit the decision record to what actually decided, record the holdout draw, and fix the attestation envelope's width and freshness #9178/fix(orb): stop four silent alarm gaps in relay-drain, crash handling, DLQ, and reversal signals #9179 with no conflicts)

Closes #9011
Closes #9042

…nd name every silent AI-review skip #9011: prReadyForReview used to defer a missing-required-context PR forever past the 2-minute cap instead of falling through to the existing finalize-past-cap block; a required context that never appears keeps ciState pending (never "passed"), so finalizing here can never produce a would-merge disposition. A new stuckReason ("missing_required_context" vs "ci_running") is threaded through the audit/log calls so the two causes are distinguishable. #9042: the silent wait before a review runs (measured live: 8.3min median, 21.9min p90, up to 50.9min) now gets an immediate "waiting" panel comment (renderWaitingForCiPlaceholder) upserted through the same PR_PANEL_COMMENT_MARKER the final verdict replaces, instead of leaving the PR blank until CI settles. #9000: root-causes the #8972 incident where a forced ("Re-run LoopOver review") retrigger completed with none of the events a forced pass should emit anywhere in the audit trail. shouldStartAiReviewForAdvisory folds in shouldRequirePublicAiReviewForAdvisory's hard gate (aiReviewMode off, an ineligible author, no head SHA, an AI kill-switch off, no AI binding) - a forced retrigger does not bypass this gate, and every one of its branches previously returned false with zero audit trail. resolvePublicAiReviewGateSkipReason names the exact reason, and a new catch-all audit event fires whenever aiReviewWillRun ends up false for a cause none of the other (already-audited) paths cover. The two adjacent silent branches in the frozen/paused/one-shot reuse chain - a hold condition true but nothing published to actually reuse - get their own named "unavailable" events for the same reason. This only closes the exhaustive-auditing and root-cause portion of #9000; the receipt-feedback / lost-webhook-click recovery sweep it also asks for is not yet implemented. Closes #9011 Closes #9042

superagent-security · 2026-07-27T06:16:45Z

Superagent didn't find any vulnerabilities or security issues in this PR.

#9173 introduced a comment referencing the deprecated pre-rename product name, which the branding-drift check on main was not catching -- breaking CI for every PR based on current main, including this one.

codecov · 2026-07-27T06:41:05Z

❌ 5 Tests Failed:

Tests completed	Failed	Passed	Skipped
22088	5	22083	21

View the top 3 failed test(s) by shortest run time

test/unit/setup-wizard-docs-parity.test.ts > self-host GitHub App manifest <-> docs parity (#2542) > the docs page's events sentence names every buildManifest default_event, and only those

Stack Traces | 0.0155s run time

AssertionError: expected [ 'check run', 'check suite', …(5) ] to deeply equal [ 'check run', 'check suite', …(10) ]

- Expected
+ Received

  [
    "check run",
    "check suite",
-   "deployment status",
-   "issue comment",
    "issues",
    "pull request",
    "pull request review",
-   "pull request review thread",
    "push",
-   "repository",
    "status",
-   "workflow run",
  ]

 ❯ test/unit/setup-wizard-docs-parity.test.ts:65:31

test/unit/backfill.test.ts > GitHub backfill > refreshes installation health from live GitHub App metadata

Stack Traces | 0.0423s run time

AssertionError: expected [ { installationId: 123, …(19) } ] to deeply equal ArrayContaining{…}

- Expected
+ Received

- ArrayContaining [
-   ObjectContaining {
+ [
+   {
      "accountLogin": "JSONbored",
+     "authMode": "local",
+     "checkedAt": "2026-07-27T06:27:01.609Z",
+     "errorSummary": undefined,
+     "eventRemediation": [
+       {
+         "action": "No change needed.",
+         "event": "issues",
+         "ok": true,
+       },
+       {
+         "action": "No change needed.",
+         "event": "issue_comment",
+         "ok": true,
+       },
+       {
+         "action": "No change needed.",
+         "event": "pull_request",
+         "ok": true,
+       },
+       {
+         "action": "Subscribe to the pull_request_review webhook event.",
+         "event": "pull_request_review",
+         "ok": false,
+       },
+       {
+         "action": "No change needed.",
+         "event": "repository",
+         "ok": true,
+       },
+       {
+         "action": "Subscribe to the check_run webhook event.",
+         "event": "check_run",
+         "ok": false,
+       },
+       {
+         "action": "Subscribe to the check_suite webhook event.",
+         "event": "check_suite",
+         "ok": false,
+       },
+     ],
+     "events": [
+       "issues",
+       "issue_comment",
+       "pull_request",
+       "repository",
+       "installation_repositories",
+     ],
      "installationId": 123,
-     "missingEvents": [],
+     "installedReposCount": 0,
+     "missingEvents": [
+       "pull_request_review",
+       "check_run",
+       "check_suite",
+     ],
      "missingPermissions": [],
-     "status": "healthy",
+     "optionalPermissions": {
+       "checks": "write",
+     },
+     "optionalVisibleEvents": [
+       "installation_target",
+       "installation_repositories",
+     ],
+     "permissionRemediation": [
+       {
+         "action": "No change needed.",
+         "currentAccess": "read",
+         "ok": true,
+         "permission": "metadata",
+         "requiredAccess": "read",
+       },
+       {
+         "action": "No change needed.",
+         "currentAccess": "write",
+         "ok": true,
+         "permission": "pull_requests",
+         "requiredAccess": "read",
+       },
+       {
+         "action": "No change needed.",
+         "currentAccess": "write",
+         "ok": true,
+         "permission": "issues",
+         "requiredAccess": "write",
+       },
+     ],
+     "permissions": {
+       "checks": "write",
+       "issues": "write",
+       "metadata": "read",
+       "pull_requests": "write",
+     },
+     "registeredInstalledCount": 0,
+     "repairSteps": [
+       "Update the GitHub App permissions and subscribed events.",
+       "Approve the changed permissions or reinstall the app on the target account.",
+       "Run refresh-installation-health after GitHub sends the updated installation payload.",
+       "Recheck /v1/readiness and this installation health endpoint.",
+     ],
+     "repositorySelection": "selected",
+     "requiredEvents": [
+       "issues",
+       "issue_comment",
+       "pull_request",
+       "pull_request_review",
+       "repository",
+       "check_run",
+       "check_suite",
+     ],
+     "requiredPermissions": {
+       "issues": "write",
+       "metadata": "read",
+       "pull_requests": "read",
+     },
+     "status": "needs_attention",
    },
  ]

 ❯ test/unit/backfill.test.ts:1369:37

test/unit/backfill.test.ts > GitHub backfill > uses installation source for queued segment jobs and sparse live installation fallback metadata

Stack Traces | 0.152s run time

AssertionError: expected [ { installationId: 123, …(19) } ] to deeply equal ArrayContaining{…}

- Expected
+ Received

- ArrayContaining [
-   ObjectContaining {
+ [
+   {
      "accountLogin": "JSONbored",
+     "authMode": "local",
+     "checkedAt": "2026-07-27T06:27:07.120Z",
+     "errorSummary": undefined,
+     "eventRemediation": [
+       {
+         "action": "Subscribe to the issues webhook event.",
+         "event": "issues",
+         "ok": false,
+       },
+       {
+         "action": "Subscribe to the issue_comment webhook event.",
+         "event": "issue_comment",
+         "ok": false,
+       },
+       {
+         "action": "Subscribe to the pull_request webhook event.",
+         "event": "pull_request",
+         "ok": false,
+       },
+       {
+         "action": "Subscribe to the pull_request_review webhook event.",
+         "event": "pull_request_review",
+         "ok": false,
+       },
+       {
+         "action": "Subscribe to the repository webhook event.",
+         "event": "repository",
+         "ok": false,
+       },
+       {
+         "action": "Subscribe to the check_run webhook event.",
+         "event": "check_run",
+         "ok": false,
+       },
+       {
+         "action": "Subscribe to the check_suite webhook event.",
+         "event": "check_suite",
+         "ok": false,
+       },
+     ],
      "events": [],
      "installationId": 123,
+     "installedReposCount": 1,
      "missingEvents": [
        "issues",
        "issue_comment",
        "pull_request",
+       "pull_request_review",
        "repository",
+       "check_run",
+       "check_suite",
      ],
      "missingPermissions": [
        "metadata",
        "pull_requests",
        "issues",
      ],
+     "optionalPermissions": {
+       "checks": "write",
+     },
+     "optionalVisibleEvents": [
+       "installation_target",
+       "installation_repositories",
+     ],
+     "permissionRemediation": [
+       {
+         "action": "Set repository permission metadata to read.",
+         "currentAccess": "missing",
+         "ok": false,
+         "permission": "metadata",
+         "requiredAccess": "read",
+       },
+       {
+         "action": "Set repository permission pull_requests to read.",
+         "currentAccess": "missing",
+         "ok": false,
+         "permission": "pull_requests",
+         "requiredAccess": "read",
+       },
+       {
+         "action": "Set repository permission issues to write.",
+         "currentAccess": "missing",
+         "ok": false,
+         "permission": "issues",
+         "requiredAccess": "write",
+       },
+     ],
      "permissions": {},
+     "registeredInstalledCount": 1,
+     "repairSteps": [
+       "Update the GitHub App permissions and subscribed events.",
+       "Approve the changed permissions or reinstall the app on the target account.",
+       "Run refresh-installation-health after GitHub sends the updated installation payload.",
+       "Recheck /v1/readiness and this installation health endpoint.",
+     ],
      "repositorySelection": "selected",
+     "requiredEvents": [
+       "issues",
+       "issue_comment",
+       "pull_request",
+       "pull_request_review",
+       "repository",
+       "check_run",
+       "check_suite",
+     ],
+     "requiredPermissions": {
+       "issues": "write",
+       "metadata": "read",
+       "pull_requests": "read",
+     },
      "status": "needs_attention",
    },
  ]

 ❯ test/unit/backfill.test.ts:4724:34

test/unit/backfill.test.ts > GitHub backfill > reports installation health from stored permissions and events

Stack Traces | 0.179s run time

AssertionError: expected { installationId: 123, …(19) } to match object { status: 'needs_attention', …(3) }
(22 matching properties omitted from actual)

- Expected
+ Received

@@ -1,10 +1,13 @@
  {
    "missingEvents": [
      "issues",
      "issue_comment",
+     "pull_request_review",
      "repository",
+     "check_run",
+     "check_suite",
    ],
    "missingPermissions": [
      "pull_requests",
      "issues",
    ],

 ❯ test/unit/backfill.test.ts:686:37

test/integration/api.test.ts > api routes > serves installation repair diagnostics and refreshes installation health

Stack Traces | 0.328s run time

AssertionError: expected { …(12) } to match object { refreshed: true, …(2) }
(37 matching properties omitted from actual)

- Expected
+ Received

@@ -1,10 +1,14 @@
  {
    "installation": {
-     "missingEvents": [],
+     "missingEvents": [
+       "pull_request_review",
+       "check_run",
+       "check_suite",
+     ],
      "missingPermissions": [],
-     "status": "healthy",
+     "status": "needs_attention",
    },
    "refreshed": true,
    "requiredPermissions": {
      "checks": "write",
      "issues": "write",

 ❯ test/integration/api.test.ts:2364:35

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

JSONbored self-assigned this Jul 27, 2026

fix(review): correct stale 'gittensory' brand reference in a comment

b834112

#9173 introduced a comment referencing the deprecated pre-rename product name, which the branding-drift check on main was not catching -- breaking CI for every PR based on current main, including this one.

JSONbored merged commit b3f29f5 into main Jul 27, 2026
3 of 4 checks passed

JSONbored deleted the fix/9000-9011-9042-silent-pipeline branch July 27, 2026 06:34

JSONbored mentioned this pull request Jul 27, 2026

fix(gate): defer the screenshot-table close on a capture-pipeline blip instead of treating it as missing evidence #9207

Merged

5 tasks

github-actions Bot mentioned this pull request Jul 27, 2026

chore(release): cut orb-v3.5.0 #9419

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(review): finalize CI-stuck silence, post a waiting placeholder, and name every silent AI-review skip - #9180

fix(review): finalize CI-stuck silence, post a waiting placeholder, and name every silent AI-review skip#9180
JSONbored merged 2 commits into
mainfrom
fix/9000-9011-9042-silent-pipeline

JSONbored commented Jul 27, 2026

Uh oh!

superagent-security Bot commented Jul 27, 2026

Uh oh!

Uh oh!

codecov Bot commented Jul 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

JSONbored commented Jul 27, 2026

Summary

Test plan

Uh oh!

superagent-security Bot commented Jul 27, 2026

Uh oh!

Uh oh!

codecov Bot commented Jul 27, 2026

❌ 5 Tests Failed:

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant