Skip to content

fix(review): persist AI review diagnostics into cache metadata on degraded runs - #9447

Merged
JSONbored merged 2 commits into
mainfrom
fix/persist-review-diagnostics
Jul 27, 2026
Merged

fix(review): persist AI review diagnostics into cache metadata on degraded runs#9447
JSONbored merged 2 commits into
mainfrom
fix/persist-review-diagnostics

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Problem

ai_review_cache.metadata_json carries inconclusive — it records that a review degraded, but never why.

The per-attempt AiReviewDiagnostic values that hold the answer (missing_assessment, unparseable_output, empty_output, provider_error) were only ever passed to capturePostHogReviewFailure as a capture property. They reach PostHog and nowhere else — not the database, not the container logs.

That is a real dead end during an incident. The 2026-07-27 investigation into intermittently missing AI review summaries could not tell the failure classes apart: a docker logs grep over the incident window returns zero matches, and reaching PostHog needs an interactive OAuth an incident responder may not be able to complete.

Change

Persist the compact model#attempt:status[:error] strings into metadata_json alongside inconclusive, so the three classes are separable at read time — they are three different bugs with three different fixes:

  • missing_assessment — the model genuinely returned no narrative
  • unparseable_output / empty_output — it returned something parseModelReview could not read (no balanced JSON object)
  • provider_error — the call itself failed

Scope and cost

Written on the degraded paths only, never on the healthy one. Every byte lands in a D1 row, and this database hit its 10 GB ceiling on 2026-07-26 — so this buys diagnosability for the small minority of reviews that actually failed rather than growing all of them.

No schema change: metadata_json is an existing column, so no migration and no generated-artifact regeneration.

Safety

The strings come from formatReviewDiagnosticsForCapture, whose error field is errorMessage() output and never raw provider text — the public/private boundary is unchanged. This metadata is read-only structured output, not a cache-replay trigger.

Tests

Both sides of the new branch are covered:

  • degraded run persists diagnostics, and an empty provider response is recoverable from the row as empty_output
  • healthy run omits them entirely (the D1-cost guarantee)

Patch lines are fully covered; the changed file's remaining uncovered lines (375, 597, 707, 875) all predate this change.

Refs #9432

…raded runs

`ai_review_cache.metadata_json` records `inconclusive` — that a review degraded, but never why.
The per-attempt `AiReviewDiagnostic` values that carry the answer were only ever sent to PostHog
as a capture property, so the 2026-07-27 investigation into missing review summaries could not
distinguish the failure classes from the database or the container logs, and reaching PostHog
needs an interactive OAuth an incident responder may not have.

Persist the compact `model#attempt:status[:error]` strings alongside `inconclusive`, so
`missing_assessment` (the model returned no narrative), `unparseable_output`/`empty_output`
(it returned something parseModelReview could not read) and `provider_error` are separable at
read time — three different bugs with three different fixes.

Written on the degraded paths only, never on the healthy one: every byte lands in a D1 row and
this database hit its 10 GB ceiling on 2026-07-26, so this buys diagnosability for the reviews
that actually failed rather than growing all of them. The strings come from
formatReviewDiagnosticsForCapture, whose `error` field is errorMessage() output and never raw
provider text, so the public/private boundary is unchanged.

Refs #9432
@loopover-orb

loopover-orb Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Important

🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨

⏳ LoopOver is waiting…

LoopOver has seen this pull request and is waiting on CI checks to finish before reviewing it. This comment will update once the review runs.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed · 🟨 Waiting

@codecov

codecov Bot commented Jul 27, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 88.64%. Comparing base (fb8e2eb) to head (9c3672c).
⚠️ Report is 8 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #9447      +/-   ##
==========================================
- Coverage   89.54%   88.64%   -0.91%     
==========================================
  Files         843      843              
  Lines      110011   110012       +1     
  Branches    26184    26185       +1     
==========================================
- Hits        98511    97521     -990     
- Misses      10238    11520    +1282     
+ Partials     1262      971     -291     
Flag Coverage Δ
backend 93.61% <100.00%> (-1.65%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
src/queue/ai-review-orchestration.ts 98.78% <100.00%> (+<0.01%) ⬆️

... and 3 files with indirect coverage changes

@loopover-orb loopover-orb Bot added the manual-review Gittensor contributor context label Jul 27, 2026
…vel nullish arm is excluded

codecov/patch flagged the `?? []` on the diagnostics spread as a partial: the ignore comment
shared a line with the ternary's `?`, where the v8 provider does not register it. Match the
working precedent at the PostHog capture site -- comment on its own line directly above the
property -- and the branch drops out of the report. No behavior change.
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@JSONbored
JSONbored merged commit 776c414 into main Jul 27, 2026
5 checks passed
@JSONbored
JSONbored deleted the fix/persist-review-diagnostics branch July 27, 2026 19:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

manual-review Gittensor contributor context

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant