[codex] Split app playtest failure buckets

[100yenadmin]

Summary

• Recover the old [codex] Split app playtest buckets and release gates #491 gate-bucket slice onto current main@1e148ab without stale pre-squash history.
• Add stable Part A/Part B failure_bucket, failure_detail, and original_result fields to built-app playtest evidence.
• Add explicit RRI gate split metadata so deterministic smoke and short real-provider playtests are evidence inputs, while only the complete five-persona RRI is a release verdict.
• Add score_failed classification so Part B cannot report persona_loop=PASS with score_pass=false and an empty bucket.
• Add fixture tests that execute the bucket classifiers, not just static string checks.

Closes #483.
Progress for #486 and #480.

Evidence

• Recovered from old closed draft [codex] Split app playtest buckets and release gates #491 commit 1f8c366, then rebased onto main@1e148ab as d0f39f6.
• Recovery preserved [codex] Add dev-gated scripted smoke provider #494 scripted-provider static tests and [codex] Add stable OpenWorlds agent UI hooks #495 UI-hook main.
• No full local RRI or heavyweight persona sweep run on the 16GB Mac.

Tests

• bash -n qa/ui_playtest_app.sh
• python3 -m pytest qa/test_macos_app_static.py qa/test_release_readiness.py qa/test_ui_playtest_app_buckets.py -q -> 22 passed

Summary by CodeRabbit

• Tests

  • Enhanced failure classification and tracking during application testing cycles.
  • Expanded release readiness validation across multiple assessment gates.
  • Increased test coverage for scoring failures and application readiness scenarios.

• Chores

  • Improved quality assurance infrastructure for more accurate release verification.

[coderabbitai]

Warning

Review limit reached

@100yenadmin, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 9 minutes and 47 seconds. Learn how PR review limits work.

Your organization has run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: ca824441-3721-4d2c-8af5-9189f39f38a0

📥 Commits

Reviewing files that changed from the base of the PR and between d0f39f6 and a0be596.

📒 Files selected for processing (2)

• qa/release_readiness.py
• qa/test_ui_playtest_app_buckets.py

📝 Walkthrough

Walkthrough

This PR implements structured failure bucketing for the built-app playtest harness (qa/ui_playtest_app.sh), adding agent-readable classification for Part A (native transition) and Part B (persona loop) failures. A gate split contract is introduced in release readiness to track multiple gate verdicts. Fixture and static tests validate the bucketing functions and contract fields.

Changes

Built-app Playtest Failure Bucketing

Layer / File(s)	Summary
Gate split contract definition qa/release_readiness.py	Defines GATE_SPLIT_CONTRACT with three gates (deterministic_built_app_smoke, short_real_provider_playtest, full_five_persona_rri), each containing scope and release_verdict. Includes contract in release-readiness JSON output.
Failure bucketing core functions and Part A transition writer qa/ui_playtest_app.sh	Introduces APP_FAILURE_BUCKETS_JSON constant, bucket_pair helper, four classifier functions (classify_native_failure, classify_part_b_readiness_failure, classify_part_b_failure_from_artifacts, classify_part_b_score_failure), set_bucket_pair accumulator, and write_part_a_transition helper that writes native/transition.json with failure bucket and detail fields.
Part A native transition failure classification qa/ui_playtest_app.sh	On build/launch failure, immediately classifies and writes transition artifact. On early exits (no launcher, app not running), sets bucket/detail and writes transition. On final PASS/FAIL, logs outcome and either clears or computes failure bucket/detail, then writes final transition.
Part B persona loop failure classification qa/ui_playtest_app.sh	When port selection fails, records bucket/detail. When readiness gate fails, classifies via classify_part_b_readiness_failure. When scoring fails, classifies from score.json or falls back to artifact-based classification.
Run.json payload extension with failure buckets qa/ui_playtest_app.sh	Extends run.json generation to include Part A and Part B failure bucket/detail values accumulated during playtest execution.
Bucket classification fixture tests qa/test_ui_playtest_app_buckets.py	New test module with run_classifier helper sourcing ui_playtest_app.sh and executing classification commands. Tests validate score.json-based bucketing (satisfaction 5/10 failure produces score_failed|score.json failed: satisfaction=5/10) and artifact-based classification (prefers console error).
Static contract and gate split validation tests qa/test_macos_app_static.py, qa/test_release_readiness.py	MacOSAppStaticContractTests validates all expected failure bucket identifiers and classifier names present in script. test_release_readiness asserts release_verdict_gate and gate_split_contract verdicts match contract.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related issues

• electricsheephq/WorldOS#483: Directly implements the acceptance criteria for structured failure bucketing in the built-app harness (captures launcher/app-status JSON, emits failure_bucket and failure_detail in native/transition.json, derives buckets from app-status/session-surface/console evidence).

Possibly related PRs

• electricsheephq/WorldOS#374: Both PRs modify qa/ui_playtest_app.sh transition/run JSON artifact generation and failure-bucketing logic; the related PR's transition/run output structure is extended by these changes.
• electricsheephq/WorldOS#413: Both PRs modify the release-readiness/gating contract in qa/release_readiness.py, with this PR adding GATE_SPLIT_CONTRACT and new release_verdict_gate/gate_split_contract fields alongside the related PR's RRI gate-scoring framework.
• electricsheephq/WorldOS#475: Both PRs extend Part A and Part B artifacts in qa/ui_playtest_app.sh (native/transition.json and run.json); the related PR adds kept-alive and first-turn readiness fields that coexist with these failure bucket/detail additions.

Poem

🐰 A bucket for each failure, neat and clean,
No mysteries left for agents to glean,
Part A to Part B, all reasons classified,
Playtest results now crystallified. ✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly describes the main change: introducing split failure buckets for app playtest gates, which is the primary focus of the changeset.
Description check	✅ Passed	The description covers the summary, evidence/rationale, tests run, linked issues, and licensing acknowledgment sections. It provides substantive detail on changes made and validation performed.
Linked Issues check	✅ Passed	The PR fulfills issue #483 requirements: adds failure buckets (no_app, no_launcher, no_provider, console_error, move_rejected, score_failed) [#483], captures app-status JSON, includes failure_bucket and failure_detail in transition.json [#483], derives buckets from app-status/session-surface/console evidence [#483], handles missing app-status fallback [#483], and provides fixture tests [#483]. Also progresses #486 and #480 as stated.
Out of Scope Changes check	✅ Passed	All changes focus on failure bucket classification, RRI gate split metadata, and associated tests directly tied to issue #483 objectives. No unrelated modifications detected.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[100yenadmin]

@coderabbitai review

[coderabbitai]

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[100yenadmin]

@coderabbitai review

[coderabbitai]

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (1)

qa/release_readiness.py (1)

340-341: 💤 Low value

Consider extracting the release verdict gate name to a constant.

The string "full_five_persona_rri" is hardcoded here and appears as a key in GATE_SPLIT_CONTRACT (line 50). To reduce duplication and prevent typos, consider extracting it to a module-level constant.

♻️ Proposed refactor

 REQUIRED_RELEASE_PERSONAS = ["newbie", "veteran", "adversarial", "narrative", "optimizer"]
+RELEASE_VERDICT_GATE = "full_five_persona_rri"
 GATE_SPLIT_CONTRACT = {
     "deterministic_built_app_smoke": {
         "scope": "fast built-app wiring proof with deterministic provider",
         "release_verdict": False,
     },
     "short_real_provider_playtest": {
         "scope": "short built-app proof with a real provider and provider trace evidence",
         "release_verdict": False,
     },
-    "full_five_persona_rri": {
+    RELEASE_VERDICT_GATE: {
         "scope": "non-partial five-persona release readiness verdict",
         "release_verdict": True,
     },
 }

Then use it on line 340:

-        "release_verdict_gate": "full_five_persona_rri",
+        "release_verdict_gate": RELEASE_VERDICT_GATE,

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@qa/release_readiness.py` around lines 340 - 341, Extract the hardcoded gate
name "full_five_persona_rri" into a module-level constant (e.g.,
RELEASE_VERDICT_GATE_NAME) and use that constant both where the dict sets
"release_verdict_gate" and where the same key appears in GATE_SPLIT_CONTRACT;
update references to the literal in release_verdict_gate assignment and the
GATE_SPLIT_CONTRACT definition so both use the new constant to avoid duplication
and typos.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@qa/test_ui_playtest_app_buckets.py`:
- Around line 13-20: The temp-file sourcing currently assumes
qa/ui_playtest_app.sh was truncated correctly by run_classifier; before the line
that does source "$funcs" add a fast-fail check that the generated "$funcs"
contains the expected sentinel patterns (the rewritten ROOT= line and the '#
DRIVE' marker) — if either pattern is missing, print an explicit error and exit
non‑zero. Refer to the variables/commands in this snippet (funcs, source
"$funcs", run_classifier, and the sentinel strings "ROOT=" and "# DRIVE") so the
test fails fast when the sentinels move or change instead of sourcing the wrong
content.

---

Nitpick comments:
In `@qa/release_readiness.py`:
- Around line 340-341: Extract the hardcoded gate name "full_five_persona_rri"
into a module-level constant (e.g., RELEASE_VERDICT_GATE_NAME) and use that
constant both where the dict sets "release_verdict_gate" and where the same key
appears in GATE_SPLIT_CONTRACT; update references to the literal in
release_verdict_gate assignment and the GATE_SPLIT_CONTRACT definition so both
use the new constant to avoid duplication and typos.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: d09f573c-d90d-4141-971a-3c20be56633a

📥 Commits

Reviewing files that changed from the base of the PR and between 1e148ab and d0f39f6.

📒 Files selected for processing (5)

• qa/release_readiness.py
• qa/test_macos_app_static.py
• qa/test_release_readiness.py
• qa/test_ui_playtest_app_buckets.py
• qa/ui_playtest_app.sh

[100yenadmin]

Fixed the CodeRabbit review pass in a0be596: added sentinel fast-fail checks for the bucket classifier fixture loader and extracted RELEASE_VERDICT_GATE so the RRI release-verdict gate name is not duplicated. Validation: bash -n qa/ui_playtest_app.sh; python3 -m pytest qa/test_macos_app_static.py qa/test_release_readiness.py qa/test_ui_playtest_app_buckets.py -q -> 22 passed. @coderabbitai review

[coderabbitai]

@100yenadmin Thanks for the update! The sentinel fast-fail checks and the RELEASE_VERDICT_GATE extraction both sound like solid improvements — good call on eliminating that string duplication. Triggering a fresh review of a0be596 now.

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.