Skip to content

fix(gate): never close a PR for visual evidence the pipeline cannot produce - #9934

Merged
JSONbored merged 1 commit into
mainfrom
fix/screenshot-gate-unobtainable-9881
Jul 30, 2026
Merged

fix(gate): never close a PR for visual evidence the pipeline cannot produce#9934
JSONbored merged 1 commit into
mainfrom
fix/screenshot-gate-unobtainable-9881

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Closes #9881.

The defect

screenshotTableGate.action: "close" closes a PR when no visual evidence is found. That is correct when evidence is merely absent. It is wrong when evidence was never obtainable.

A repo with review.visual.enabled: true but no preview deployments at all — JSONbored/awesome-claude is the live case — exhausts its preview-poll budget, never sets visualCaptureSatisfiedSha, and every contributor PR touching a visual-scoped path is then closed one-shot. No contributor action other than hand-authoring the table could ever change that, and nothing had told the maintainer their configuration was unsatisfiable.

Current blast radius: latent, not firing. I checked the six most recently closed-unmerged PRs on awesome-claude — all touch zero visual-scoped files, so the gate did not close them. It triggers on the next PR that touches apps/web/src/{components,routes}/** without a hand-authored table.

The proof was already being computed

capture.ts distinguishes preview build states, and absent means no preview check-run was found at all — the file already notes this was "confirmed live on a repo whose UI has no preview-deploy CI configured: every PR's 'after' cell spun forever."

absent plus a spent poll budget (5 polls over 7.5 minutes) is conclusive: not a late deploy, not a renderer blip, no pipeline. That pairing is surfaced as CaptureResult.previewUnobtainable, persisted per head SHA as visual_capture_unobtainable_sha (migration 0207), and read by the gate.

What changes, and what deliberately does not

  • The violation is unchanged. The PR genuinely lacks visual evidence, and the finding still says so. Suppressing it would hide a real gap from the maintainer.
  • Only enforcement degrades. action: "close" becomes advisory for that head, with an audit event (screenshot_table_close_degraded_capture_unobtainable) recording it.
  • The reason is contributor-visible and names the maintainer-side remedy — enable preview deploys, or set requireScreenshotTable.action: advisory — because no contributor action can resolve it.
  • Scoped to head SHA, like visualCaptureSatisfiedSha and visualCaptureRetryPendingSha. A later push re-arms the requirement; a repo that gains preview deploys stops matching on its very next commit; a successful capture clears the marker.

Decisions I made

The issue offered three directions and raised a fourth question; I took the narrow, reversible ones:

  • Per-PR detection, not repo-level "repeatedly." The per-head signal is already conclusive and needs no new cross-PR state, no tuning threshold, and no window during which early PRs still get closed.
  • Degrade to advisory rather than hold. On a one-shot contributor pipeline a hold just relocates the dead end.
  • No new ScreenshotTableGateAction tier. The issue notes "close" | "advisory" has no middle setting, which is a fair point — but it is full config-as-code parity work (engine type → default → GATE key → normalizer → resolver → types.tsopenapi/schemas.ts → both example YAMLs → docs drift) and this fix does not need it. Left for its own issue.
  • Maintainer surfacing (the ops-anomaly bullet) not included here. The audit event makes the condition queryable now; a readiness surface is a separate change.

Validation

  • 5 new unit tests on the pure degrade, covering the four cases that matter: the violation still reports, the degrade is marked with an actionable reason, a merely-absent capture is not degraded (that close stays legitimate), and omitting the flag is byte-identical to before. Verified by mutation — neutering the degrade condition fails exactly one test.
  • 286 tests pass across the gate and queue suites; the pre-existing 110 screenshot-gate tests are untouched and green.
  • tsc --noEmit, db:migrations:check (contiguous 0001..0207), db:schema-drift:check, engine-parity:drift-check, dead-exports:check, docs:drift-check, git diff --check all clean.

…roduce (#9881)

`screenshotTableGate.action: "close"` closed PRs when no visual evidence was found.
Correct when evidence is merely absent; wrong when it was never obtainable. A repo
with `review.visual.enabled` but no preview deploys at all (JSONbored/awesome-claude)
exhausts its poll budget, never sets visualCaptureSatisfiedSha, and every contributor
PR touching a visual-scoped path is closed one-shot -- for evidence no contributor
action could ever supply, against a misconfiguration nobody was told about.

The proof was already in the capture: build state `absent` means no preview check-run
was found AT ALL, and the file already noted this was "confirmed live on a repo whose
UI has no preview-deploy CI configured". Paired with a spent poll budget, that
distinguishes "no pipeline" from a late deploy or a renderer blip, which is exactly
the distinction the close was missing.

The violation is UNCHANGED -- the PR really does lack visual evidence and the finding
still says so. Only enforcement degrades, and the contributor-visible reason names
the maintainer-side remedy, since no contributor action can resolve it.

Marked per head SHA like its siblings, so a later push re-arms the requirement and a
repo that gains preview deploys stops matching on its very next commit.
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Jul 30, 2026

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
loopover-ui 2ea08bb Commit Preview URL

Branch Preview URL
Jul 30 2026, 05:07 PM

@loopover-orb

loopover-orb Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Caution

🛑 LoopOver review result - fixes required

Review updated: 2026-07-30 17:30:24 UTC

9 files · 1 AI reviewer · no blockers · CI failing · unstable

🛑 Suggested Action - Manual Review

Review summary
This PR closes #9881 by distinguishing "no visual evidence found" from "visual evidence structurally unobtainable" (no preview pipeline at all): capture.ts sets previewUnobtainable when build state is absent and the poll budget is exhausted, that gets persisted per-head-SHA via a new visual_capture_unobtainable_sha column (schema.ts + migration 0207, contiguous, D1-safe ALTER TABLE ADD COLUMN with no default needed since it's nullable), and screenshot-table-gate.ts wraps the pure evaluator so a violation still surfaces but enforcement degrades from close to advisory with a contributor-visible reason. processors.ts correctly gates both the close-match and evidence-unresolved paths on the new degraded flag, and markPullRequestVisualCaptureSatisfied properly clears the unobtainable marker on a later successful capture. The gate-level unit tests (screenshot-table-gate.test.ts) are solid and directly exercise the new degrade branch with the real evaluator.

Nits — 6 non-blocking
  • The diff shows no new integration test exercising `capture.previewUnobtainable === true` driving `markPullRequestVisualCaptureUnobtainable` or the resulting audit event in src/queue/processors.ts — queue-3.test.ts only updates existing mocks to add `previewUnobtainable: false`, so the new persistence/audit branches appear uncovered by anything visible in this diff.
  • I can't verify from the visible diff/context whether the row→`PullRequestRecord` mapping in src/db/repositories.ts actually includes the new `visualCaptureUnobtainableSha` column when loading `pr` for the gate — if that mapping doesn't select it, `processors.ts`'s `captureUnobtainable` computation silently stays `false` forever; worth a quick grep to confirm it's wired alongside the sibling `visualCaptureSatisfiedSha` field.
  • The `screenshot_table_close_degraded_capture_unobtainable` audit event (src/queue/processors.ts) fires on every maintenance pass while the head SHA stays unobtainable, matching the existing `screenshot_table_close_deferred_capture_retry` pattern, but consider whether repeat-audit spam across many passes is worth deduping — not new to this PR's pattern, so low priority.
  • CAPTURE_UNOBTAINABLE_REASON in screenshot-table-gate.ts is a long inline string; consider whether it belongs alongside DEFAULT_SCREENSHOT_CONTRACT_MESSAGE for consistency, purely a style nit.
  • Add a queue-processor-level test that seeds `pr.visualCaptureUnobtainableSha === pr.headSha` and asserts the CLOSE is suppressed end-to-end (not just at the pure-evaluator level), to guard the actual wiring this PR depends on.
  • PR author also opened the linked issue — Link an issue that was opened by a different contributor, or provide a rationale for why this self-authored issue represents genuine discovery work.

CI checks failing

  • codecov/patch — 64.58% of diff hit (target 99.00%)

Decision drivers

  • ✅ Code review — No blockers (1 reviewer)
  • ✅ Gate result — Passing (No configured blocker found.)
Context & advisory signals — never blocks the verdict
Signal Result Evidence
Linked issue ✅ Linked #9881
Related work ✅ No active overlap found No same-issue or scoped active PR overlap found.
Change scope ✅ 20/20 Low review scope from cached public metadata (1 linked issue).
Validation posture ✅ 25/25 PR body includes validation/test evidence.
Contributor workload ✅ 10/10 Author activity: 9 registered-repo PR(s), 8 merged, 303 issue(s).
Contributor context ✅ Confirmed Gittensor contributor JSONbored; Gittensor profile; 9 PR(s), 303 issue(s).
Improvement ✅ Minor risk: clean · value: minor
Linked issue satisfaction

Addressed
The PR directly implements the issue's suggested direction: it detects the exhausted-poll-budget/no-deployment condition (previewUnobtainable), persists it per head SHA, degrades the gate's close action to advisory with a contributor/maintainer-visible reason, and adds audit logging plus unit tests covering the new behavior without weakening legitimate closes.

Review context
  • Author: JSONbored
  • Role context: owner (maintainer lane)
  • Public audience mode: oss maintainer
  • Lane context: Repository is configured for direct PR review.
  • Public profile languages: Python, TypeScript, Ruby, Go, MDX, Shell, Solidity, JavaScript
  • Official Gittensor activity: 9 PR(s), 303 issue(s).
  • PR-specific overlap: none found.
Contributor next steps
  • Start here: Treat this as maintainer-lane context rather than normal contributor-lane activity.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.
🧪 Chat with LoopOver

Ask LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

  • @loopover ask <question> answers contribution-quality Q&A with source citations and freshness.
  • @loopover chat <question> answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
  • A plain-language @loopover mention with a real question is routed to the closest matching read-only command automatically — no exact syntax required.

Full command reference: https://loopover.ai/docs/loopover-commands

🧪 Experimental — new and may change.

Decision record
  • action: hold · clause: success
  • config: b354db8928f22eed7df3f29fe176d767a8079f5d89139c62a29eda5bbf1bc181 · pack: oss-anti-slop · ci: failed
  • record: d7c0aead0d118456cf17218d76b0995914c9427869431b9b33ead87af4fd3708 (schema v6, head 2ea08bb)

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.

  • Re-run LoopOver review

@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@codecov

codecov Bot commented Jul 30, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 64.58333% with 17 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.87%. Comparing base (8248b88) to head (2ea08bb).
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
src/queue/processors.ts 33.33% 6 Missing and 2 partials ⚠️
...oopover-engine/src/review/screenshot-table-gate.ts 77.41% 7 Missing ⚠️
src/db/repositories.ts 0.00% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #9934      +/-   ##
==========================================
- Coverage   91.88%   91.87%   -0.02%     
==========================================
  Files         923      923              
  Lines      113407   113451      +44     
  Branches    27332    27338       +6     
==========================================
+ Hits       104205   104232      +27     
- Misses       7915     7930      +15     
- Partials     1287     1289       +2     
Flag Coverage Δ
backend 95.67% <52.38%> (-0.02%) ⬇️
engine 72.35% <67.74%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
src/db/schema.ts 74.87% <ø> (ø)
src/review/visual/capture.ts 96.32% <100.00%> (+0.02%) ⬆️
src/types.ts 100.00% <ø> (ø)
src/db/repositories.ts 96.78% <0.00%> (-0.10%) ⬇️
...oopover-engine/src/review/screenshot-table-gate.ts 70.25% <77.41%> (+0.40%) ⬆️
src/queue/processors.ts 94.60% <33.33%> (-0.22%) ⬇️

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

manual-review Gittensor contributor context

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The screenshot-table gate can close PRs for evidence the bot is structurally unable to produce

1 participant