Release harness v2: app-status readiness, scripted smoke, evidence export - #497
Merged
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (5)
🚧 Files skipped from review as they are similar to previous changes (4)
📝 WalkthroughWalkthroughAdds a centralized failure-bucketing CLI, a deterministic scripted-provider multi-beat smoke runner, server-side app-status readiness/health diagnostics, run-dir evidence export, structured scripted-provider telemetry, UI/test/docs updates, and tests coordinating those pieces. ChangesScripted Smoke Testing & Failure Routing
Sequence Diagram(s)sequenceDiagram
participant Test as qa/app_smoke_scripted.py
participant Provider as scripts/play_scripted_dm.sh
participant Status as viewer/server.py (/app-status)
participant UI as OpenWorlds UI
participant Trace as play-state trace files
Test->>Provider: Start with WORLDOS_ENABLE_SCRIPTED_PROVIDER=1
Provider->>Trace: Emit init events and per-move trace.ndjson
Test->>Status: Poll /openworlds/app-status until readiness.ready_for_smoke
Status->>Status: Compute readiness/health from provider/actor/narration/console
loop For each beat (5–8)
Test->>UI: POST /move deterministic intent
Provider->>Trace: Append move_resolved with beat detail
Test->>Status: Poll for narration advancement
Test->>Test: Capture app-status, surface HTML, screenshot, NDJSON logs
end
Provider->>Trace: Write summary.json with resolved_move_count
Test->>Test: Write smoke.json verdict and copy play-state artifacts
Test->>CLI: qa/export_app_evidence.py --run-dir <dir> --out <bundle> (optional)
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
Comment |
This was referenced May 31, 2026
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
docs/AGENT_GRADE_APP_TESTABILITY.md (1)
87-117:⚠️ Potential issue | 🟡 Minor | ⚡ Quick win
blockedstatus referenced but not in the readiness enum.The
readiness.statusenum lists onlyready|degraded(line 88), but the behavioral rules describe ablockedstatus (line 117). Either addblockedto the enum or drop it from the prose so the documented contract is self-consistent.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/AGENT_GRADE_APP_TESTABILITY.md` around lines 87 - 117, The documentation currently lists readiness.status as "ready|degraded" but the behavioral rules mention a third status "blocked"; update the contract to be consistent by adding "blocked" to the readiness.status enum (e.g., change the literal set for readiness.status to "ready|degraded|blocked") and, if needed, add a brief line in the readiness section describing when readiness.status should be set to blocked to match the prose in the behavioral rules (reference readiness.status and the behavioral rules mentioning blocked).
🧹 Nitpick comments (2)
qa/ui_playtest_app.sh (1)
95-97: 💤 Low valueConsider documenting or removing unused variable.
APP_FAILURE_BUCKETS_JSONis no longer used in this script (all classification delegates toqa/app_failure_buckets.py). If it's part of the public contract for external scripts that source this file, add a comment explaining its purpose. Otherwise, consider removing it along with thebucket_pairbash function.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@qa/ui_playtest_app.sh` around lines 95 - 97, APP_FAILURE_BUCKETS_JSON and the bucket_pair() function are unused in qa/ui_playtest_app.sh; either remove both to avoid dead code or document them if they're part of an external sourcing contract: if removing, delete the APP_FAILURE_BUCKETS_JSON declaration and the bucket_pair() function; if keeping for backward compatibility, add a clear comment above APP_FAILURE_BUCKETS_JSON explaining its purpose and that qa/app_failure_buckets.py now handles classification, and mark bucket_pair() as deprecated with a short usage note referencing qa/app_failure_buckets.py.qa/app_failure_buckets.py (1)
247-251: 💤 Low valueClarify browser-probe fallback message.
When
classify_browser_probereturnsNone(indicating app_status is OK), the fallback at line 251 returns"no_provider|browser probe passed". This pairs a failure bucket (no_provider) with a success message (passed), which could confuse callers. Consider either:
- Returning
"ok|browser probe passed"(though"ok"isn't inAPP_FAILURE_BUCKETS)- Documenting that callers should not invoke
browser-probeunless they already know there's a failure- Exiting with a different code or empty output when the probe passes
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@qa/app_failure_buckets.py` around lines 247 - 251, The current fallback coerces a successful browser probe (classify_browser_probe returning None) into a failure bucket by using bucket_pair("no_provider", "browser probe passed"); change this so a successful probe returns None (or an explicit success sentinel) instead of a failure pair: remove the "or bucket_pair(...)" fallback and let result be the direct return value of classify_browser_probe (or return a documented "success" sentinel), updating callers to handle None/sentinel accordingly; references: classify_browser_probe and bucket_pair.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@qa/app_smoke_scripted.py`:
- Around line 357-383: The in-beat polling and final status/surface fetches
(calls to wait_for_status, fetch_json and surface_url inside the beat loop and
before finalization) are unguarded and can raise
RuntimeError/URLError/ValueError; wrap the polling body and the final
wait_for_status/fetch_json blocks in try/except to catch those exceptions, set
verdict.update(...) with an appropriate failure_bucket (e.g. "no_app" or
"console_error") and failure_detail describing the exception, json_dump the
smoke.json via the existing json_dump(out / "smoke.json", verdict) call, and
return 1 — specifically add exception handling around the loop that uses
wait_for_status/fetch_json (referencing wait_for_status, fetch_json,
surface_url, provider_summary, json_dump, write_text_snapshot,
capture_openworlds_screenshot) so any transient provider/viewer drop produces a
bucketed smoke.json and a clean return instead of an unhandled traceback.
In `@viewer/server.py`:
- Around line 5402-5403: The readiness counters network_failures and
console_errors are incorrectly hardcoded to 0; replace those assignments with
computed aggregates from the runtime event data (e.g., count entries in the
console/log buffer and network request records) so the readiness model can
reflect real failures — for example, set console_errors = sum(1 for e in
console_events if e.level in ("error","fatal") or matches error patterns) and
network_failures = sum(1 for r in network_requests if r.failed or r.status_code
>= 400); update the same logic at the other occurrence that mirrors lines
5425-5428 so both readiness checks use these computed counts and not fixed
zeros, and ensure the code refers to the actual event collections used elsewhere
in this module (console_events, network_requests, or their local equivalents).
---
Outside diff comments:
In `@docs/AGENT_GRADE_APP_TESTABILITY.md`:
- Around line 87-117: The documentation currently lists readiness.status as
"ready|degraded" but the behavioral rules mention a third status "blocked";
update the contract to be consistent by adding "blocked" to the readiness.status
enum (e.g., change the literal set for readiness.status to
"ready|degraded|blocked") and, if needed, add a brief line in the readiness
section describing when readiness.status should be set to blocked to match the
prose in the behavioral rules (reference readiness.status and the behavioral
rules mentioning blocked).
---
Nitpick comments:
In `@qa/app_failure_buckets.py`:
- Around line 247-251: The current fallback coerces a successful browser probe
(classify_browser_probe returning None) into a failure bucket by using
bucket_pair("no_provider", "browser probe passed"); change this so a successful
probe returns None (or an explicit success sentinel) instead of a failure pair:
remove the "or bucket_pair(...)" fallback and let result be the direct return
value of classify_browser_probe (or return a documented "success" sentinel),
updating callers to handle None/sentinel accordingly; references:
classify_browser_probe and bucket_pair.
In `@qa/ui_playtest_app.sh`:
- Around line 95-97: APP_FAILURE_BUCKETS_JSON and the bucket_pair() function are
unused in qa/ui_playtest_app.sh; either remove both to avoid dead code or
document them if they're part of an external sourcing contract: if removing,
delete the APP_FAILURE_BUCKETS_JSON declaration and the bucket_pair() function;
if keeping for backward compatibility, add a clear comment above
APP_FAILURE_BUCKETS_JSON explaining its purpose and that
qa/app_failure_buckets.py now handles classification, and mark bucket_pair() as
deprecated with a short usage note referencing qa/app_failure_buckets.py.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 7af3cce7-f89c-4e0a-9b90-006189b66c79
📒 Files selected for processing (17)
docs/AGENT_GRADE_APP_TESTABILITY.mdqa/app_failure_buckets.pyqa/app_smoke_scripted.pyqa/export_app_evidence.pyqa/release_readiness.pyqa/test_app_failure_buckets.pyqa/test_app_smoke_scripted.pyqa/test_export_app_evidence.pyqa/test_macos_app_static.pyqa/test_release_readiness.pyqa/test_ui_playtest_app_buckets.pyqa/ui_playtest_app.shscripts/play_scripted_dm.shviewer/index.htmlviewer/openworlds/screen-table.jsxviewer/server.pyviewer/tests/test_openworlds_static.py
Member
Author
|
Manual in-app browser smoke on top of #497:
Evidence bundle: UX note for follow-up triage, not blocking this harness PR: the Chronicle can show the player echo after the scripted DM response for the palette action, and the opening narration appeared duplicated in the live log. The harness still correctly proves same-port status + accepted moves + narration advancement. |
Member
Author
|
Follow-up after CodeRabbit:
Validation after the fix:
|
This was referenced Jun 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
qa/ui_playtest_app.shthrough it while keeping the public ten-bucket contract stable/app-statuswith read-only readiness/health fields and retires stale root/legacy viewer routes to/openworlds//app-status, screenshots/a11y/status/surface/move logs, and scripted-provider summary/traceqa/export_app_evidence.py --run-dirand preserves live--app-status-urlexportCloses #483.
Closes #485.
Closes #486.
Refs #480, #481, #484, #479, #466.
Evidence
bash -n qa/ui_playtest_app.sh scripts/play_scripted_dm.shpython3 -m py_compile qa/app_smoke_scripted.py qa/export_app_evidence.py qa/app_failure_buckets.pypython3 -m pytest viewer/tests/test_openworlds_static.py qa/test_macos_app_static.py qa/test_export_app_evidence.py qa/test_release_readiness.py qa/test_app_failure_buckets.py qa/test_app_smoke_scripted.py qa/test_ui_playtest_app_buckets.py -q-> 77 passed, 6 subtests passed/Volumes/LEXAR/Codex/worldos-agent-grade-app-testability/scripted-smoke-v2-20260531T233335Z-080497e/http://127.0.0.1:8899/app-status,ready_for_smoke=true,ready_for_play=true, private art probe ok, actor Abby seated, 5 enabled actions, 11 chat lines, 7 PNG screenshots, zero evidence gaps/Volumes/LEXAR/Codex/worldos-agent-grade-app-testability/scripted-smoke-v2-20260531T233335Z-080497e-export/manifest.jsoncopied 36 files with zero gapsNotes
origin/mainat080497e; the obsolete pre-[codex] Add stable OpenWorlds agent UI hooks #495 UI-hook commit is not included.Summary by CodeRabbit
New Features
Refactor
Chores