fix(review): persist AI review diagnostics into cache metadata on degraded runs - #9447
Merged
Conversation
…raded runs `ai_review_cache.metadata_json` records `inconclusive` — that a review degraded, but never why. The per-attempt `AiReviewDiagnostic` values that carry the answer were only ever sent to PostHog as a capture property, so the 2026-07-27 investigation into missing review summaries could not distinguish the failure classes from the database or the container logs, and reaching PostHog needs an interactive OAuth an incident responder may not have. Persist the compact `model#attempt:status[:error]` strings alongside `inconclusive`, so `missing_assessment` (the model returned no narrative), `unparseable_output`/`empty_output` (it returned something parseModelReview could not read) and `provider_error` are separable at read time — three different bugs with three different fixes. Written on the degraded paths only, never on the healthy one: every byte lands in a D1 row and this database hit its 10 GB ceiling on 2026-07-26, so this buys diagnosability for the reviews that actually failed rather than growing all of them. The strings come from formatReviewDiagnosticsForCapture, whose `error` field is errorMessage() output and never raw provider text, so the public/private boundary is unchanged. Refs #9432
Contributor
|
Important 🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨 ⏳ LoopOver is waiting…LoopOver has seen this pull request and is waiting on CI checks to finish before reviewing it. This comment will update once the review runs. 🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed · 🟨 Waiting |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #9447 +/- ##
==========================================
- Coverage 89.54% 88.64% -0.91%
==========================================
Files 843 843
Lines 110011 110012 +1
Branches 26184 26185 +1
==========================================
- Hits 98511 97521 -990
- Misses 10238 11520 +1282
+ Partials 1262 971 -291
Flags with carried forward coverage won't be shown. Click here to find out more.
|
…vel nullish arm is excluded codecov/patch flagged the `?? []` on the diagnostics spread as a partial: the ignore comment shared a line with the ternary's `?`, where the v8 provider does not register it. Match the working precedent at the PostHog capture site -- comment on its own line directly above the property -- and the branch drops out of the report. No behavior change.
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
ai_review_cache.metadata_jsoncarriesinconclusive— it records that a review degraded, but never why.The per-attempt
AiReviewDiagnosticvalues that hold the answer (missing_assessment,unparseable_output,empty_output,provider_error) were only ever passed tocapturePostHogReviewFailureas a capture property. They reach PostHog and nowhere else — not the database, not the container logs.That is a real dead end during an incident. The 2026-07-27 investigation into intermittently missing AI review summaries could not tell the failure classes apart: a
docker logsgrep over the incident window returns zero matches, and reaching PostHog needs an interactive OAuth an incident responder may not be able to complete.Change
Persist the compact
model#attempt:status[:error]strings intometadata_jsonalongsideinconclusive, so the three classes are separable at read time — they are three different bugs with three different fixes:missing_assessment— the model genuinely returned no narrativeunparseable_output/empty_output— it returned somethingparseModelReviewcould not read (no balanced JSON object)provider_error— the call itself failedScope and cost
Written on the degraded paths only, never on the healthy one. Every byte lands in a D1 row, and this database hit its 10 GB ceiling on 2026-07-26 — so this buys diagnosability for the small minority of reviews that actually failed rather than growing all of them.
No schema change:
metadata_jsonis an existing column, so no migration and no generated-artifact regeneration.Safety
The strings come from
formatReviewDiagnosticsForCapture, whoseerrorfield iserrorMessage()output and never raw provider text — the public/private boundary is unchanged. This metadata is read-only structured output, not a cache-replay trigger.Tests
Both sides of the new branch are covered:
empty_outputPatch lines are fully covered; the changed file's remaining uncovered lines (375, 597, 707, 875) all predate this change.
Refs #9432