qa(rri): detect 429 quota-aborts, run VM personas sequentially, never roll up a junk RRI (rc3 lesson) - #844
Merged
Merged
Conversation
… roll up a junk RRI (rc3 lesson) rc3 hit the claude account session limit (HTTP 429) mid-sweep: the canary (run alone) came back clean (sat=7, 0 crit, arc), then all 4 PARALLEL Opus cold-opens 429'd, never seated a PC, and the rollup rolled those quota-corpses up into a MISLEADING RRI 1.8 with no signal that it was infra, not product. Three fixes: - sweep_v2.sh: quota_tripped()/quota_reset_hint() helpers; the persona batch now runs SEQUENTIALLY (cold-open is API-generation-bound per worldos-latency-forensics, so the old 'parallel = 16 vCPUs' premise bought nothing but a 4x quota burst) so the FIRST 429 aborts before the rest spend; a QUOTA_ABORT marker short-circuits the duo + RRI rollup and emits an explicit ABORTED status instead of a number. - release_readiness.py: infra_abort_hint() greps a persona's backend.log for a session limit / HTTP 429; any infra-aborted release persona (or an --abort-marker) forces status=ABORTED, release_ready=False, verdict 'QUOTA-ABORTED', and a loud human line — attribution that distinguishes a quota abort from a genuinely broken build. - tests: 429-backend.log => ABORTED; backend_not_ready WITHOUT a 429 stays a product failure (the discriminator); --abort-marker forces ABORTED. No product code touched. 60/60 release-gate suite + fast_gate green; bash -n clean.
|
Caution Review failedPull request was closed or merged during review 📝 WalkthroughWalkthroughAdds HTTP 429 session-limit quota-abort detection to the QA sweep pipeline. ChangesQuota-Abort Circuit Breaker
Sequence DiagramsequenceDiagram
participant sweep as sweep_v2.sh
participant canary as canary persona
participant personas as remaining personas (sequential)
participant marker as results/QUOTA_ABORT
participant rr as release_readiness.py
sweep->>sweep: rm -f results/QUOTA_ABORT
sweep->>canary: run canary persona
canary-->>sweep: score-newbie.json absent?
sweep->>sweep: quota_tripped(CANARY_BL)?
alt quota tripped
sweep->>marker: write QUOTA_ABORT + exit
else canary scored
sweep->>sweep: quota_tripped post-score?
alt quota tripped
sweep->>marker: write QUOTA_ABORT + exit
else ok
loop each of 4 remaining personas
sweep->>personas: run persona
sweep->>sweep: quota_tripped(persona BL)?
alt quota tripped
sweep->>marker: write QUOTA_ABORT + break
end
end
end
end
sweep->>marker: QUOTA_ABORT present?
alt present
sweep->>sweep: write RRI.json {status:ABORTED, release_ready:false}
sweep-->>sweep: exit (skip duo/behavioral/audit)
else absent
sweep->>rr: release_readiness.py [--abort-marker]
rr->>rr: infra_abort_hint per run_dir
rr->>rr: check --abort-marker file
rr->>rr: populate result {status, aborted, abort_reason, abort_detail}
rr-->>sweep: scorecard with QUOTA-ABORTED or GREEN/RED
end
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related issues
Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cea9dfccf5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if quota_tripped "$CANARY_BL"; then | ||
| note "QUOTA ABORT — the canary scored but its backend 429'd ($(quota_reset_hint "$CANARY_BL")); the account is at its session limit. Not spending the batch." | ||
| echo "newbie $(quota_reset_hint "$CANARY_BL")" > "$RES/QUOTA_ABORT" | ||
| touch "$RES/DONE"; exit 0 |
There was a problem hiding this comment.
When the account is already over quota during the canary, this branch touches DONE and exits before the later QUOTA_ABORT handler runs, so no RRI.json/rri.txt with status=ABORTED is produced. Artifact consumers will see a completed VM sweep without the explicit quota-abort evidence this change is meant to guarantee; route the canary quota exits through the same abort writer or call release_readiness.py --abort-marker before exiting.
Useful? React with 👍 / 👎.
Comment on lines
+175
to
+179
| json.dump({"status": "ABORTED", "abort_reason": "quota_session_limit", | ||
| "detail": detail, "build_sha": sha, "release_ready": False, | ||
| "note": "claude account session limit (HTTP 429) tripped mid-sweep; " | ||
| "this is an INFRA abort, NOT a product RRI. Re-run after the quota resets."}, | ||
| open(out, "w"), indent=2) |
There was a problem hiding this comment.
When a post-canary persona trips the quota marker, this hand-written RRI.json omits the fields produced by release_readiness.py for aborted runs (aborted, abort_detail, rri, gates_passed, failed_gates, artifact sources, etc.) and uses detail instead. Downstream readers/tests that rely on the canonical RRI shape will not classify this artifact the same way as the new rollup contract; use the new --abort-marker path or mirror that schema here.
Useful? React with 👍 / 👎.
100yenadmin
added a commit
that referenced
this pull request
Jun 14, 2026
…ta-aborted); fix sweep header parallel→sequential (#846) - OPERATING-GOAL state block → current truth: main ~347b6c2, the audit fix-wave + the sibling full-engine-audit fork's fixes (mech #832/#834, image_render #829, dead-beat #841, seat #833, spells #830, economy #831, beat-reliability #828) all landed; rc3 QUOTA-ABORTED + invalid (#845, newbie-canary clean); NEXT ACTION = re-measure with the hardened sequential sweep at current main, blocked on claude quota. - sweep_v2.sh header: the top comment + start-line still said PARALLEL after #844 made the persona batch sequential — corrected so the file is internally consistent. Co-authored-by: Eva <arncalso@gmail.com>
100yenadmin
added a commit
that referenced
this pull request
Jun 20, 2026
…ning gaps) (#1042) * fix(qa): quota circuit-breaker + stale-evidence hygiene — a 429 yields QUOTA-ABORT, never a junk RRI/score (#842) PR #844 landed the core (sequential personas, quota_tripped, QUOTA_ABORT sentinel, the rollup ABORTED status). This closes the SIX remaining gaps so a mid-sweep account session-limit 429 can never masquerade as a product measurement, and a quota'd run can never republish a previous run's evidence. Fix A (qa/vm/sweep_v2.sh) — sweep-start cleanup now also `rm -f "$RES/RRI.json"`, so a sweep that quota-aborts before writing a fresh rollup can't leave the PREVIOUS run's RRI.json in place (the rc3 stale-RRI bug). Fix B (qa/vm/sweep_v2.sh) — the canary QUOTA_ABORT path now writes the {"status":"ABORTED",…} RRI.json (it previously touched DONE + exited, leaving a stale RRI behind). Extracted a shared write_aborted_rri() helper and reused it at BOTH canary-abort sites and the post-batch QUOTA_ABORT short-circuit so the ABORTED JSON shape is identical at every quota exit. Fix C (qa/vm/sweep_v2.sh) — wipe this run's stale duo artifacts (duo-{tolkien,angrydm,latency}.json + qa/transcripts/vm2-duo.{tolkien,angrydm}.json) BEFORE the duo runs, so the `[ -f ] && cp` only copies CURRENT-run output (rc3 republished rc2's byte-identical "story 4.0/mech 3.0" verbatim). Fix D (qa/ui_playtest_app.sh) — the backend player-ready poll loop now greps backend.log for a 429/session-limit INSIDE the loop (after the kill -0 check), drops a QUOTA_EXHAUSTED sentinel + breaks early; the readiness-failure path checks the sentinel and emits a `quota_exhausted` bucket instead of mis-bucketing the corpse as backend_not_ready/no_actor. Added quota_exhausted to APP_FAILURE_BUCKETS_JSON. Fix E (qa/run_duo.sh + qa/vm/sweep_v2.sh) — run_duo detects a session-limit 429 in the DM cold-open ($COMBINED / $RUN.dm.err), logs "[duo] QUOTA ABORT", skips scoring, and exits rc=2 (before the empty-reply rc=1 abort). The sweep greps duo.log for that marker before copying duo scores → on a hit it writes QUOTA_ABORT + the ABORTED RRI and skips the rollup. Fix F (qa/score.sh + qa/run_duo.sh) — score.sh adds a 429 fast-fail arm (no 3 retries): on api_error_status==429 (or a session-limit body) it writes the sentinel {"quota_exhausted":true,"api_error_status":429} to $OUT and exits rc=2. run_duo's post-scoring check treats any lens carrying that sentinel as a quota abort (→ "[duo] QUOTA ABORT" + exit rc=2), never a valid scorecard, before the behavioral gate runs. Tests (qa/test_release_gate_static.py) — 6 static grep-the-shell-source contracts mirroring test_release_gate_static.py style (no live runs): RRI.json in the cleanup rm; canary-abort writes an ABORTED RRI; duo-artifact rm precedes the duo call; quota_exhausted in APP_FAILURE_BUCKETS_JSON; score.sh 429 fast-fail arm (sentinel + rc=2); run_duo checks for QUOTA ABORT before scoring. All 20 tests in the file pass (14 pre-existing + 6 new). bash -n clean on all four touched scripts. * fix(#842 review): write_aborted_rri must emit aborted:true + abort_detail (was masking as RELEASE_READY) Adversarial review caught the load-bearing defect: the ABORTED RRI used {status:ABORTED, detail:...} but evidence_audit.py keys on aborted:true + abort_detail — so a quota-aborted sweep read as RELEASE_READY, the exact masking #842 prevents. Mirror release_readiness.py's shape. Lock it with a static key-check + a functional test that runs the shape through evidence_audit.py (21 pass). --------- Co-authored-by: Eva <arncalso@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The rc3 VM sweep (@a245a2c) hit the claude account session limit (HTTP 429) partway through. The newbie canary (run alone, before the limit) came back clean — sat=7, 0 crit, arc complete, no give-up. Then all 4 parallel Opus cold-opens 429'd, never seated a player character (
backend_not_ready), and the rollup folded those quota-corpses into a misleading RRI 1.8 with nothing flagging it as infra-not-product. That is the exact "junk PARTIAL RRI" failure mode we must never ship as a measurement.What
qa/vm/sweep_v2.sh—quota_tripped()/quota_reset_hint()helpers; the persona batch now runs sequentially (cold-open is API-generation-bound perworldos-latency-forensics, so the old "parallel = use the 16 vCPUs" premise bought nothing but a 4× quota burst). The first 429 aborts before the rest spend; aQUOTA_ABORTmarker short-circuits the duo + the RRI rollup and emits an explicit ABORTED status instead of a number. (SWEEP_PERSONA_CONCURRENCYleft as the documented override for a quota-rich window.)qa/release_readiness.py—infra_abort_hint()greps a persona'sbackend.logfor a session-limit / HTTP 429. Any infra-aborted release persona (or an--abort-marker) forcesstatus=ABORTED,release_ready=False, scorecard verdictQUOTA-ABORTED, and a loud human line — attribution that distinguishes a quota abort from a genuinely broken build.backend.log⇒ ABORTED;backend_not_readywithout a 429 stays a product/harness failure (the discriminator);--abort-markerforces ABORTED.Evidence
backend.logof the rc3 failures:HTTP 429 ... You've hit your session limit · resets 3:50pm (UTC).bash -nclean. No product code touched.Refs the rc3 quota-abort (issue filed separately). The hardened harness is what the post-quota-reset re-run uses.
Summary by CodeRabbit
New Features
Tests