fix(rri): scope native_gate build-SHA contract to the canonical release personas - #723
Merged
Conversation
…se personas native_gate's build-SHA gates (single build_sha / same-build / per-run) iterated ALL persona scores, so an EXTRA diagnostic persona (opus-high / lean variants) at a different SHA falsely failed native_gate even when the 5 canonical release personas + the Mac handoff were all same-build. The Tuesday RRI hit exactly this: newbie/veteran/adversarial/narrative/optimizer all at 033e4ba, but 3 narrative variants stamped stale SHAs (8afed3c/f89ce94/eabf2a3) -> 'mixed persona build_sha' gap, despite a 100/100 Mac handoff at 033e4ba. Fix: factor the contract into build_sha_evidence_gaps(persona_scores, build_sha, release_personas), scoped to REQUIRED_RELEASE_PERSONAS. Diagnostic variants no longer trip it; an absent canonical persona is still caught by missing_release_personas; a canonical persona at a wrong SHA still fails. Pure fn, 6 unit tests incl. the exact Tuesday situation. Makes the next full RRI honest re: native_gate (the Mac handoff already proves the .app at the candidate SHA). NOTE: release_readiness.py is part of the scoring ruler — this correctly re-versions it (#722).
📝 WalkthroughWalkthroughThis PR refactors native_gate build-SHA validation in release readiness tooling. A new ChangesBuild-SHA Scoping for Release Personas
Estimated code review effort🎯 2 (Simple) | ⏱️ ~12 minutes Possibly related issues
Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@qa/release_readiness.py`:
- Around line 472-504: The variable build_shas is computed only inside
build_sha_evidence_gaps but main() expects it when assembling the signals (the
"run_build_shas" key), causing a NameError; fix by having
build_sha_evidence_gaps return both gaps and the scoped build_shas (e.g., return
(gaps, build_shas)) or by moving the build_shas computation into main() before
calling build_sha_evidence_gaps, and update the call site in main() to unpack
the new return value (or supply build_shas) and populate "run_build_shas":
build_shas accordingly.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: e8a11958-a0af-47bb-a140-3f9630ee176a
📒 Files selected for processing (2)
qa/release_readiness.pyqa/test_release_readiness_scope.py
Comment on lines
+472
to
+504
| def build_sha_evidence_gaps(persona_scores: list[dict], build_sha: str, | ||
| release_personas: list[str]) -> list[dict]: | ||
| """native_gate's build-SHA contract, SCOPED to the canonical release personas. | ||
|
|
||
| The release verdict is about the canonical five (REQUIRED_RELEASE_PERSONAS) + the Mac handoff, | ||
| all at ONE SHA. Extra DIAGNOSTIC personas (opus-high / lean variants outside the release set) | ||
| may run at other SHAs without invalidating the release — so they must NOT trip the "single | ||
| build_sha" / "same-build" gates. (RRI 2026-06-09: 3 narrative variants stamped stale SHAs while | ||
| newbie/veteran/adversarial/narrative/optimizer were all at the candidate SHA, falsely failing | ||
| native_gate even though the Mac handoff + the 5 release personas were same-build.) | ||
| """ | ||
| release = [p for p in persona_scores if str(p.get("persona") or "") in set(release_personas)] | ||
| build_shas = sorted({str(p["run_build_sha"]) for p in release if p.get("run_build_sha")}) | ||
| missing = [p for p in release if not p.get("run_build_sha")] | ||
| gaps: list[dict] = [] | ||
| if not build_sha: | ||
| gaps.append({"gate": "native_gate", "missing": "--build-sha", | ||
| "detail": "release verdict requires the measured build SHA"}) | ||
| if missing: | ||
| gaps.append({"gate": "native_gate", "missing": "per-run build_sha", | ||
| "detail": "missing run build_sha for: " | ||
| + ", ".join(str(p.get("persona") or p.get("run")) for p in missing)}) | ||
| if build_sha: | ||
| mismatched = [p for p in release | ||
| if p.get("run_build_sha") and not build_sha_matches(str(p.get("run_build_sha")), build_sha)] | ||
| if mismatched: | ||
| gaps.append({"gate": "native_gate", "missing": "same-build persona evidence", | ||
| "detail": "run build_sha mismatch: " | ||
| + ", ".join(f"{p['persona']}={p['run_build_sha']}" for p in mismatched)}) | ||
| if len(build_shas) > 1: | ||
| gaps.append({"gate": "native_gate", "missing": "single build_sha", | ||
| "detail": "mixed release-persona build_sha values: " + ", ".join(build_shas)}) | ||
| return gaps |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Verify that build_shas in main() is defined elsewhere or if there's a scoping issue
rg -n 'build_shas' qa/release_readiness.pyRepository: electricsheephq/WorldOS
Length of output: 347
🏁 Script executed:
#!/bin/bash
# Inspect where build_shas is referenced around line 851 to determine scope/definition.
nl -ba qa/release_readiness.py | sed -n '820,880p'Repository: electricsheephq/WorldOS
Length of output: 108
🏁 Script executed:
#!/bin/bash
# Output qa/release_readiness.py with line numbers for region around line 851.
# Use cat -n because nl is unavailable.
cat -n qa/release_readiness.py | sed -n '820,870p'Repository: electricsheephq/WorldOS
Length of output: 2710
🏁 Script executed:
#!/bin/bash
cat -n qa/release_readiness.py | sed -n '620,720p'Repository: electricsheephq/WorldOS
Length of output: 5857
Fix build_shas scoping before using it in the signals result
build_sha_evidence_gaps()correctly scopes the “single build SHA” checks to the canonical release personas and detects missing/mismatched/multiple SHAs within that set.main()referencesbuild_shasat line 851 ("run_build_shas": build_shas,), butbuild_shasis only defined insidebuild_sha_evidence_gaps()(line 484). This will raiseNameErrorat runtime—either computebuild_shasinmain()or change the helper to also return/passbuild_shasback tomain().
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@qa/release_readiness.py` around lines 472 - 504, The variable build_shas is
computed only inside build_sha_evidence_gaps but main() expects it when
assembling the signals (the "run_build_shas" key), causing a NameError; fix by
having build_sha_evidence_gaps return both gaps and the scoped build_shas (e.g.,
return (gaps, build_shas)) or by moving the build_shas computation into main()
before calling build_sha_evidence_gaps, and update the call site in main() to
unpack the new return value (or supply build_shas) and populate
"run_build_shas": build_shas accordingly.
This was referenced Jun 9, 2026
100yenadmin
added a commit
that referenced
this pull request
Jun 9, 2026
…every RRI rollup (#728) * fix(rri): restore main()'s build_shas — #723 regression that crashed every RRI rollup #723 factored the inline build-SHA gate logic into build_sha_evidence_gaps(), which moved the 'build_shas = sorted(...)' definition into the helper's scope but left main()'s output reference ('run_build_shas': build_shas, line ~851) dangling -> NameError: name 'build_shas' is not defined. This crashed the FINAL RRI rollup of EVERY release_readiness.py run (the VM 5-persona sweep @fa97b34 produced all persona/duo/ui_audit scores, then died on the rollup). Restore the all-persona distinct-SHA set in main()'s scope (the diagnostic output field; the native_gate CONTRACT stays release-persona-scoped via the helper — #723's intent intact). WHY IT SHIPPED: qa/test_release_readiness.py ALREADY has 4 tests that exercise this path (test_mixed_build_sha / test_missing_run_build_sha / test_handoff_json_must_prove_app_status_build_sha / test_short_build_sha_prefix) and they FAIL at fa97b34 — but #723's CI did not gate on them. The code fix makes all 30 pass; a separate follow-up will close the CI gap so this class of regression is caught pre-merge. Tests: qa/test_release_readiness.py 30/30 + test_release_readiness_scope.py + release_gate_static 16/16. * qa(scores): record v1.0.4-rc1 RRI = 3.6/10 (4/11) @fa97b34 — first comparable RC-grade verdict The first 11-gate RRI produced by the FIXED rollup (this PR's build_shas fix). Mac part-A handoff 100/100 + VM part-B 5-persona sweep (lean-ON), ruler-stamped sc_5ac7a1d9103c. Verdict: NOT release-ready. PASS: native_gate(signal), arc, no_give_up, behavioral GREEN, palette. FAIL: sat 4.8<7, zero_critical(4), story 3.9<4.3, mech 3.2<4.5 (one-duo sampling), ui_audit FAIL. vs Tuesday 3.6: no_give_up RED->PASS, sat 4.4->4.8, behavioral GREEN — the #719/#720 reliability fixes show; the genuine product long-poles remain. --------- Co-authored-by: Eva <arncalso@gmail.com>
100yenadmin
added a commit
that referenced
this pull request
Jun 9, 2026
The 'test' job only collected servers/*/tests/ — qa/test_*.py was NEVER run in CI, so a *tested* crash (#723 build_shas, caught by test_release_readiness.py's 4 build_sha tests) merged green and shipped, surfacing only when the live RRI rollup crashed. Add a qa-release-gate-tests job running the release-authority surface: test_release_readiness, test_release_readiness_scope, test_scores_db_comparability, test_scores_db, test_release_gate_static (67 tests, validated locally via the engine venv). A regression in release_readiness.py / scores_db.py can no longer merge green. Closes #729. Co-authored-by: Eva <arncalso@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The Tuesday 11-gate RRI failed
native_gateon "mixed persona build_sha" — but the 5 canonical release personas (newbie/veteran/adversarial/narrative/optimizer) were all at033e4bawith a 100/100 Mac handoff at the same SHA. The failure came from 3 extra diagnostic personas (opus-high / lean variants) that stamped stale SHAs (8afed3c/f89ce94/eabf2a3). The build-SHA gates iterated all personas, so a diagnostic variant artificially capped the release signal.Fix
Factor the build-SHA contract into a pure, testable
build_sha_evidence_gaps(persona_scores, build_sha, release_personas)scoped toREQUIRED_RELEASE_PERSONAS:single build_sha/same-build.missing_release_personasgate).Tests (
qa/test_release_readiness_scope.py, 6)The exact Tuesday situation (5 canonical + 3 stale variants → no gaps), clean set, a release persona at a wrong SHA (still fails), missing build_sha, no
--build-sha, variants-only.Makes the next full RRI re-run honestly count
native_gate(the Mac handoff already proves the built.appat the candidate SHA).release_readiness.pyis part of the scoring ruler, so this correctly re-versions it once #722 lands.Summary by CodeRabbit
New Features
Tests