fix(qa): de-flake behavioral A8 — recovered transient rejection is WARN, not RED (#897) - #1156
Conversation
…RN, not RED (#897) The `no_rejected_tool_calls` (A8) gate flipped RED on ANY single tool rejection carrying an extra_forbidden / validation-error payload. Behavioral is computed from ONE stochastic duo, so a single RECOVERED transient flub (the DM emits one malformed call, immediately retries the same tool correctly, the session completes cleanly — invisible to the player) RED-capped EVERY lens to 2.5 and swung the headline RRI by ~1.0. Observed twice (#897). Make A8 reflect a PATTERN, not a recovered blip — mirroring the #1030 discriminator-aware severity (WARN below threshold, FATAL only for the genuine defect). A schema rejection counts toward FATAL only when it is: - UNRECOVERED — the offending tool was NEVER successfully called anywhere in the run (the DM's intent silently never took effect — the real version-skew defect), OR - REPEATED — the same tool was rejected >=2x (systematic skew); stays FATAL even if a later call eventually succeeds. A single rejection the DM then successfully retried -> WARN, never RED. This is a PRECISION improvement, not a leniency hack: the unrecovered + repeated classes the gate was built for still flip RED. The existing corpus fixture (an unrecovered update_character rejection) still REDs; the new green corpus fixture (case_no_rejected_tool_calls_recovered_warn) locks the FATAL->WARN scope guard so a future re-promotion to FATAL is caught (#1036-style green-case discipline). TDD: qa/test_behavioral_gate_a8_deflake.py (7 cases) — recovered/two-distinct transients GREEN+WARN; unrecovered/repeated/mixed still RED+FAIL; benign engine-guard rejection unchanged; clean run emits no A8 check. Verified: - qa/test_behavioral_gate_a8_deflake.py 7 passed - qa/test_behavioral_gate_corpus.py 22 passed (incl. new green case) - qa/test_qa_gate.py + test_root_cause_analyzer 45 passed - bash qa/fast_gate.sh 241 passed, GREEN
|
Warning Review limit reached
More reviews will be available in 39 minutes and 6 seconds. Learn how PR review limits work. Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file). ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits. 🚦 How do rate limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan refill rate. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, the refill rate gradually slows as usage increases. The highest same-day bursts are limited more strictly. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (7)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| import sys | ||
| from pathlib import Path | ||
|
|
||
| import pytest |
1 participant
WS0b — Behavioral-gate false-RED de-flake (closes #897)
The bug
The
no_rejected_tool_calls(A8) gate inqa/assert_behavioral.pyflipped RED on ANY single tool rejection carrying anextra_forbidden/validation errorpayload. Behavioral is computed from ONE stochastic duo, so a single recovered transient flub — the DM emits one malformed call, immediately retries the same tool correctly, the session completes cleanly (invisible to the player) — RED-capped EVERY lens to 2.5 and swung the headline RRI by ~1.0. Observed twice (the 27d8002 sweep read RRI 4.5 purely because the DM emitted one barepersist_beat()and recovered).The fix (precision, not leniency)
Make A8 reflect a pattern, not a recovered blip — mirroring the #1030 discriminator-aware severity (WARN below threshold, FATAL only for the genuine defect). A schema rejection counts toward FATAL only when it is:
is_error=False) anywhere in the run → the DM's intent silently never took effect → the real version-skew defect. (this is the existing corpus fixture's shape)A single rejection of a tool the DM then successfully retried → WARN, never RED (still surfaced, never silently dropped).
This does not weaken any true integrity gate: the unrecovered + repeated classes the gate was built for still flip RED. Benign non-schema engine-guard rejections (
engine_guards_hit) are unchanged.TDD
New
qa/test_behavioral_gate_a8_deflake.py(7 cases), written failing-first:Corpus regression lock
no_rejected_tool_callsis now explicitly unrecovered (commented so) and still REDs — the genuine version-skew signal is preserved.no_rejected_tool_calls_recovered_warnlocks the FATAL→WARN scope guard (structural_completeness still FATAL-caps authored-campaign runs (campaign-arc quest can't resolve in one session) #1036-style green-case discipline): a future edit re-promoting recovered transients to FATAL flips it RED andtest_green_case_warns_but_stays_greenfails.Verification (single-process, focused)
Invariants
Additive + default-on for the new precision; reads only DM-stream tool results (engine-mutated ground truth, never fiction); QA-only (no engine/Eva/gateway touch). The behavioral corpus still REDs every genuine failure (player-seated, systematic rejected-tools, dice honesty, dm-output, SRD-correctness, xp).