From 41ff48868a50e90c967a2875ee21c4587cd471da Mon Sep 17 00:00:00 2001 From: lume Date: Sun, 31 May 2026 05:45:53 +0700 Subject: [PATCH] chore(qa): RRI scorer + GUI runbook/workbook + RRI gates (operating goal & scorecard) Release Readiness Index (qa/release_readiness.py): 11 hard gates -> 0-10, reads disk artifacts (robust to tool-channel fabrication). Adds image-render-rate + palette-live gates (the two owner-visible defects the old gate could pass while broken). WorldOS-GUI-RUNBOOK.md: two-surface look-and-wire loop (iterate 8799-from-canonical, gate on built .app). qa/GUI_WORKBOOK.md: living punch-list. OPERATING-GOAL section 4 + SCORECARD extended with RRI. Docs/tooling only; engine + wire contracts untouched. --- WorldOS-GUI-RUNBOOK.md | 89 +++++++++++++++++ WorldOS-OPERATING-GOAL.md | 149 +++++++++++++++++++++++++++ qa/GUI_WORKBOOK.md | 38 +++++++ qa/SCORECARD.md | 11 ++ qa/release_readiness.py | 205 ++++++++++++++++++++++++++++++++++++++ 5 files changed, 492 insertions(+) create mode 100644 WorldOS-GUI-RUNBOOK.md create mode 100644 WorldOS-OPERATING-GOAL.md create mode 100644 qa/GUI_WORKBOOK.md create mode 100644 qa/release_readiness.py diff --git a/WorldOS-GUI-RUNBOOK.md b/WorldOS-GUI-RUNBOOK.md new file mode 100644 index 00000000..1a2a6193 --- /dev/null +++ b/WorldOS-GUI-RUNBOOK.md @@ -0,0 +1,89 @@ +# WorldOS GUI Runbook — the look-and-wire release loop + +> How to test→fix→LOOK the WorldOS GUI on the REAL surface and drive it to a 10/10 release. +> Born from the 2026-05-31 reorientation: the prior loop scored a HEADLESS PROXY served from +> WORKTREES WITH NO ART, so every visible defect (no palette, no images, no map, unformatted +> chronicle, phantom companion) sailed past. This runbook makes that impossible to repeat. +> Companions: `WorldOS-OPERATING-GOAL.md` (the gate), `qa/GUI_WORKBOOK.md` (the live punch-list), +> `qa/release_readiness.py` (the RRI scorer), `qa/SCORECARD.md` (the ledger). + +## The two surfaces (never confuse them again) +- **ITERATE — visible, playable, fast:** the OpenWorlds viewer served **from the canonical repo** + `/Users/lume/ClawDnD-val` (which HAS the 2.9 GB `content/worlds/_private` art) as a LIVE PLAYABLE + session on **fixed port 8799**. This is where you fix one thing at a time and LOOK. +- **GATE — truth:** the built `dist/WorldOS.app` via `qa/ui_playtest_app.sh` (part A native #356 + + part B persona loop). Release is judged here. Same viewer code; adds the native shell. +- **Why both:** identical viewer. 8799-from-canonical skips the build + guarantees art is present, so + it's the honest fast loop. The `.app` is the shipped artifact. NEVER iterate on a worktree-served + viewer (no `_private` → 100% image 404) or a stale build. + +## Stand up the iteration surface (8799, playable, from canonical) +``` +cd /Users/lume/ClawDnD-val +git checkout main && git pull --ff-only origin main # current main (has merged fixes) +pkill -f 'viewer/server.py'; pkill -f 'scripts/play.sh'; pkill -f 'play_party.sh' # NOT node:18789 (Eva gateway) +CLAWDND_PLAY_PORT=8799 nohup bash scripts/play.sh baldurs-gate preview-$(git rev-parse --short HEAD) 8799 > /tmp/wos-8799.log 2>&1 & +# play.sh sets CLAWDND_PLAYER_MOVES → can_act:true (the move sink = the palette is live) +``` +Open `http://127.0.0.1:8799/openworlds/`. The DM cold-open takes ~30–90s; **wait for a SEATED PC** +(party non-empty), not just `can_act:true` — `can_act` can flip true before the PC is seated. + +## LOOK (verify by curl + screenshot — NEVER a single Read; the channel fabricates) +The tool channel intermittently returns fabricated/empty/doubled reads (this session it invented a +`kind=pc` palette-disabled bug and a scene-404 that were both false). **Ground every load-bearing +claim in ≥2 clean reads + a checksum/HTTP code.** +``` +curl -s http://127.0.0.1:8799/session-surface | python3 -c 'import json,sys;d=json.load(sys.stdin); \ + print("party",[ (p["name"],p.get("kind")) for p in d.get("party",[])]); \ + print("palette",[a["id"] for a in d.get("availableActions",[]) if a.get("available")]); \ + print("can_act",d.get("can_act"))' +# images: curl -s -o /dev/null -w '%{http_code}' "http://127.0.0.1:8799/image?scope=location:loc-lower-city" +``` +Per-fix visual checklist: palette buttons present + enabled in the MAIN column · a click resolves a +turn · portraits/scene/map images 200 · a multi-paragraph DM beat renders as paragraphs · prose +streams mid-turn (`/events` count climbs during the turn) · a SOLO session has the PC alone. + +## Fix one thing → PR → merge → rebuild → LOOK (the loop) +1. Confirm the symptom on 8799 with ≥2 clean reads. If it doesn't reproduce, it's a stale/corrupt + read — do NOT fix it (log to GUI_WORKBOOK "evaporated"). +2. Builder agent in a **worktree off origin/main** (never branch-op canonical): + `git -C /Users/lume/ClawDnD-val worktree add -B fix/ /tmp/wos- origin/main` +3. PR → CI green (incl. `viewer-tests`) → admin-squash-merge → delete branch → prune worktree. + **Builder PRs sometimes fail to push silently** (happened twice this session) — always + `gh pr view ` / `git ls-remote origin ` to confirm the branch+PR EXIST before relying + on them; if lost, redo the (usually small) change yourself in a clean worktree. +4. `git pull --ff-only` canonical → restart 8799 → LOOK → tick GUI_WORKBOOK with the proof. + +## The gate sweep (Phase 3 — judged on the built .app) +``` +# build + 5 personas, SEQUENTIAL (clean host for honest latency), each its own run dir: +for p in newbie veteran adversarial narrative optimizer; do + WOS_APP_PART=AB qa/ui_playtest_app.sh sweep-$p baldurs-gate $p 40 12.00 +done +# 3-lens story/mech on a duo transcript: qa/score.sh rubric_tolkien.md score_schema_tolkien.json out.json ; same w/ rubric_angry_dm.md +# behavioral: python3 qa/assert_behavioral.py (exit 0=GREEN) +# GUI health: qa/ui_audit_health.sh --port 8799 --quick --axe --ui-gate +# palette-live: the curl check above (≥6 enabled actions on a can_act surface) +# roll it up: +python3 qa/release_readiness.py --runs sweep-newbie,sweep-veteran,sweep-adversarial,sweep-narrative,sweep-optimizer \ + --story story.json --mech mech.json --behavioral GREEN|RED --ui-audit PASS|FAIL --palette-live true|false \ + --build-sha $(git rev-parse --short HEAD) --scorecard-row +``` +RRI 10/10 = all 11 gates hold on ONE build. Append the `--scorecard-row` line to `qa/SCORECARD.md`. + +## Release (when RRI = 10/10 on a fresh .app build) +Bump `.claude-plugin/plugin.json` → 1.0.4, tag `v1.0.4`, GitHub release + CHANGELOG. Then MAINTAIN: +every PR touching `viewer/ | macos/ | skills/ | servers/engine/` → rebuild + RRI sweep + SCORECARD row; +any regression (a critical bug, a sub-7 persona, sub-threshold score, image <95%, dead palette) +reverts the goal to "fix" and outranks new work. + +## Hard rules (carried from CLAUDE.md + this session's lessons) +- Engine (`servers/engine`) = SOLE writer of campaign state. Don't touch wire contracts + (`clawdnd-*`/`CLAWDND_*` MCP ids, `dev.clawdnd.app`); you MAY read `WORLDOS_REPO_ROOT`. +- `_private/` (the 2.9 GB art) is **never committed**. Building/serving from canonical is how the + art is present. +- 16 GB host: tests on **GitHub CI**, never heavy local suites. Parallel agents are fine. +- **Verify, don't trust:** ≥2 clean reads for any claim; the RRI scorer reads disk, not the live + channel; confirm builder PRs actually pushed. +- The product is the **launchable, played .app**. A green score on any other surface is a + measurement bug, not progress. diff --git a/WorldOS-OPERATING-GOAL.md b/WorldOS-OPERATING-GOAL.md new file mode 100644 index 00000000..71b94d69 --- /dev/null +++ b/WorldOS-OPERATING-GOAL.md @@ -0,0 +1,149 @@ +# WorldOS — OPERATING GOAL (v1 — re-authored from first principles 2026-05-31) + +> Read order on resume: **this file** → `WorldOS-NORTH-STAR.md` → `WorldOS-RUNBOOK.md` → +> `qa/SCORECARD.md` → `/Volumes/LEXAR/Codex/session-notes/2026-05-30/worldos-release-loop/implementation-notes.html`. + +--- + +## 0. FIRST PRINCIPLES — what are we actually trying to do? + +WorldOS is **a game** — a post-BG3 D&D 5e living world shipped as a native macOS app (`dist/WorldOS.app`). +A game has exactly **one real test: can a real person launch it and play a satisfying session without +hitting something that feels broken?** Engine depth, art, mechanics, scorecards are all **proxies**. +The product is the *launchable, playable, felt session*. **We optimize the product, never a proxy.** + +> *Why this doc exists:* for ~8h on 2026-05-30 the loop reported GREEN on a fitness function that +> measured the WRONG surface — the harness booted its OWN playable viewer while the shipped `.app` +> launched read-only. A green score on a surface the user can't reach is a **measurement bug, not +> progress.** This doc makes that impossible to repeat. + +--- + +## 1. THE OPERATING GOAL + +**Drive `dist/WorldOS.app` to release-ready for a fresh player — and keep it there as the codebase moves.** + +**Release-ready =** a user with **NO prior WorldOS knowledge** launches the freshly-rebuilt `.app`, +picks a playable canon NPC, and plays a complete **8-beat Baldur's Gate session** — the five phases each +**genuinely firing** — **without ever saying "this feels broken."** + +--- + +## 2. WHAT "COMPLETION" LOOKS LIKE — the 8-beat session, made concrete + +Done = all five phases actually happen on the built app — not narrated, not faked: + +1. **PARLEY** — the player talks with NPCs/companions; choices land; dialogue is tracked. +2. **COMBAT** — a real **engine-resolved** fight: the DM calls `start_combat` + `spawn_monster`; attack + rolls and rounds resolve **through the engine tools**, NOT prose and NOT a skill-check substitute. + *(arc4 lesson: 11 turns with ZERO combat calls = NOT complete.)* +3. **TRAVEL** — the party moves to a new location (the scene/clock actually changes). +4. **REST** — camp **restores resources + advances the clock**. +5. **TRAVEL** — the party sets out again toward the next beat. + +Verified from the engine tool-trace **and** the player-facing chronicle, **on the built `.app`.** + +--- + +## 3. P0 — THE NON-NEGOTIABLE: test the BUILT `.app`, never a proxy + +The fitness function is `qa/ui_playtest_app.sh` running the **native part-A+B §8.2 harness** against +`dist/WorldOS.app`: +- **Part A** = the native CGEvent-click gate (proves read-only→playable routing #356 on the real app window). +- **Part B** = the persona play loop on the byte-identical `play_party.sh` backend + the app's OpenWorlds viewer. + +NEVER a self-booted playable preview, a dev port, or the stale `:8765`. **Unit tests + part-B-only runs +MISS native-surface criticals** — the #405 overlay-wedge (no escape; `dismiss()` unwired; green only +because a test stubbed `useEffect`) proved this. The built, played app is the only evidence that counts. + +--- + +## 4. THE PASS GATE — Release Readiness Index (RRI 10/10, all 11 gates on the SAME fresh build) + +Computed by `qa/release_readiness.py` (hard-gate floor — a missed gate caps the score, never hidden +by an average). **RRI 10/10 = every gate holds on one fresh build:** + +1. **Native #356 gate** PASS (`ui_playtest_app.sh` part A on the built `.app`). +2. **Arc completes** — ≥1 persona finishes all 5 phases (§2): combat **through the engine**, travel + moves the party, rest restores + advances the clock. `completed_intro_flow = true`. +3. **Cross-persona satisfaction ≥ 7/10** averaged (newbie/veteran/adversarial/narrative/optimizer). + *(LATENCY lives here: if impatient personas quit on slow turns, this fails → context-leaning lever.)* +4. **No persona gives up** (`gave_up = false`). +5. **Zero critical bugs** across all 5 personas — incl. no wedge / no-escape / dead-end. +6. **Story-craft ≥ 4.3** (Tolkien lens, `score.sh` + `rubric_tolkien.md`). +7. **Mechanical ≥ 4.5** (Angry-DM lens, `score.sh` + `rubric_angry_dm.md`). +8. **Behavioral GREEN** (`qa/assert_behavioral.py`: clock advanced, ≥2 locations, combat fired, no role-bleed). +9. **`qa/ui_audit_health.sh --quick --axe --ui-gate` → 0 regressions** (a11y + per-screen render). +10. **Image-render rate ≥ 95%** (NEW — from `network.ndjson`; "no images" was THE owner-visible defect). +11. **Palette-live** (NEW — ≥6 enabled action buttons on a `can_act:true` surface; the "tools missing" defect). + +> Gates 1, 10, 11 were added 2026-05-31 after the GUI reorientation — the prior gate could pass while the +> player saw no images and no clickable tools. RRI makes the *visible* product part of the gate. +> The release DECISION is RRI = 10/10 (all gates); RRI < 10 lists exactly which gates failed. + +--- + +## 5. THE ITERATE LOOP (while the gate fails) + +`git pull` → `rm -rf dist/WorldOS.app` → build from a **worktree off origin/main** (never branch-op the +shared checkout) → `qa/ui_playtest_app.sh` × 5 personas against the built `.app` → **score** (5-persona +satisfaction + 3 lenses + behavioral + axe) → **file GitHub issues** tied to `{build_sha, version_tag}` +with `file:line` + acceptance criteria → **delegate code to builder subagents** (worktree → PR → +CI-green incl. `viewer-tests` → squash-merge) → **rebuild → re-playtest.** +**An issue closes ONLY when the NEXT build's playtest no longer reproduces it.** + +--- + +## 6. THE MAINTAIN LOOP (once the gate holds) + +Baseline `{build_sha, version_tag, scores}`. Every PR touching `viewer/`, `macos/`, `skills/`, or +`servers/engine/` → rebuild + 5-persona playtest + a SCORECARD row + `ui_audit_health`. Any regression +(a critical bug, a sub-7 persona, an axe regression, a sub-threshold score) **reverts the goal to "fix"** +and outranks new work. Cut a version bump (v1.0.x) each time the gate holds on a fresh build. + +--- + +## 7. MY ROLE — release-verifier orchestrator + +I playtest the built app, file issues, **delegate product-code fixes to builder subagents**, merge +CI-green PRs, plan sprints toward version bumps, and **verify every fix on a subsequent build before +closing.** I orchestrate + verify; I do not hand-write product code by default. Owner = orienter + final +verifier; can revert the goal to "fix" anytime. + +--- + +## 8. DISCIPLINE (load-bearing invariants) + +- The **BUILT, PLAYED `.app` is ground truth.** The harness serves the app, never the reverse. +- **Never** claim "100% confidence" / "audit complete." A merged PR is a hypothesis; a non-reproducing + NEXT build is the evidence. Close issues only on next-build non-reproduction. **Honest scores only.** +- Engine (`servers/engine`) = **SOLE writer** of campaign state. Don't touch wire contracts + (`clawdnd-*` / `CLAWDND_*` / `dev.clawdnd.app`). Build from **worktrees off origin/main** (the shared + `/Users/lume/ClawDnD-val` checkout is diverged + held by a sibling session). 16GB host → **GitHub CI** + for tests, never local heavy workers. `_private/` never committed. + +--- + +## 9. CURRENT STATUS (2026-05-31 — context, NOT part of the gate condition) + +- Fitness-function flaw (proxy surface) **fixed** — the §8.2 `.app` harness is the surface now. +- Arc-completion driven **0 → 11 turns**. Latency root-caused (turns-2+ are ~98% LLM **prefill** of the + fat resumed context; streaming was inadvertently neutered by #395, restored by #401) and **ruled OUT + as the arc-completion blocker** (11 turns at ~140s, no give-up) — but still gated by **G3**. +- **12 PRs merged this session.** The two arc4 blockers — DM never engaged combat (**#404**) and + narration duplication (**#407**) — plus the external reviewer's **critical overlay-wedge (#405 → #409**, + which also exposed a latent never-auto-dismiss bug) are fixed + merged. Viewer tests now run in CI (#403). +- **IMMEDIATE NEXT:** `arc5` (part-B) verifies combat fires + dedup + overlay handoff → build the `.app` + → native **part-A+B 5-persona sweep + 3-lens duo** = the first full PASS-GATE attempt → **v1.0.4** if it + holds. +- **Context-leaning lever** (lean the cold-open `start_world`/`load_canon_character` tool returns to cut + prefill) is **STAGED** (`/tmp/decision-context-leaning-prep.md`) — trigger ONLY if **G3** fails on latency + (first-principles decision before implementing; it's a load-bearing tool-contract change). +- Reviewer-filed open items still tracked: **#393** (latency — keep open until a real playtest confirms + no persona gives up), #406 residuals (folded into #409). + +--- + +*The mechanics are the floor. The launchable, playable, felt prestige session in the built app is the +product. Build toward the North-Star ceiling; use THIS loop to keep the floor under the user's feet +always real.* diff --git a/qa/GUI_WORKBOOK.md b/qa/GUI_WORKBOOK.md new file mode 100644 index 00000000..ac2c59f0 --- /dev/null +++ b/qa/GUI_WORKBOOK.md @@ -0,0 +1,38 @@ +# WorldOS GUI Workbook — the living punch-list (verified on the REAL surface) + +> Single source of truth for the GUI test→fix→look loop. Each row: defect, VERIFIED root cause +> (file:line), fix, status, proof. "Verified" = observed on the live playable surface +> (8799 from CANONICAL — has _private art) or read from canonical source. NOT a proxy guess. +> Iteration surface: `http://127.0.0.1:8799/openworlds/` (`scripts/play.sh` from canonical). +> Gate surface: built `dist/WorldOS.app` via `qa/ui_playtest_app.sh`. See `WorldOS-GUI-RUNBOOK.md`. + +## ★ HEADLINE (verified 2026-05-31, 3 clean reads on live canonical 8799) +**Most of what the owner saw broken was a STALE-BUILD / WORKTREE-WITHOUT-ART artifact, not broken canonical code.** Served correctly from canonical: +- `can_act:true`; palette **5 exploration actions ENABLED** (continue/say/do/check/save); attack/bonus/reaction correctly disabled ("not in combat"). +- **ALL images 200**: `location:loc-lower-city`, `portrait-rolan`, `portrait-minsc`, `map_lower-city`, `region:baldurs-gate`, `scene_baldurs-gate`. +- DM cold-open narration = 4842 chars with **27 paragraph breaks** (the prose IS well-formed). +⇒ The fix for "no images / no map / no palette" is **serve/build WITH `_private` art present** (infra), not 3 code PRs. The real *code* bugs are layout prominence + render formatting + the silent companion. + +## Phase 0 — infra (DONE) +- ✅ `launch.json` repointed off the deprecated LEXAR copy → canonical/8799/`/openworlds/`/live state. +- ✅ 8799 playable from canonical: `can_act:true`, PC Rolan + Minsc, images + map render. + +## REAL bugs (verified; the actual punch-list) +| # | Defect | VERIFIED root cause | Fix | Status | Proof | +|---|---|---|---|---|---| +| G3 | Palette buried in 280px right-rail; `slice(0,6)` drops bonus-action+reaction; no center palette | screen-table.jsx:751 | Drop slice; promote palette to main action column near Declare | Builder B in flight | live: palette enabled but only in right rail | +| G4 | Chronicle renders as ONE block despite well-formed prose | `LogEntry` (screen-table.jsx:870) renders `{text}` with default `white-space` → collapses the `\n\n` the DM DOES emit (27 breaks verified) | `white-space: pre-line` or split to `

` | Builder B in flight | chat.jsonl dm msg: 27 double-newlines; render collapses | +| G6 | Companion (Minsc/Alfira, kind='companion') silently in solo party at cold-open, no narrated meeting | solo play.sh + questgen prelude seat a companion into `party` pre-narration | Solo: no silent companion OR gate add_to_party behind a narrated meeting | Builder C in flight | live solo party=['Rolan','Minsc and Boo'] | +| G5 | "No streaming visible" — VERIFY (may be infra: owner watched a stale/wrong surface) | /events+useLiveSession+log_event exist; DM emits paragraph-rich prose | Confirm mid-turn log_event on live 8799; fix skill only if batched | Builder C verifying | DM prose well-formed; streaming path exists | +| G7 | Worktree/.app builds 404 images (no _private) | `_ingested_images_root()` (server.py:203) hardcoded to server.py's repo → worktree has no _private | Honor WORLDOS_REPO_ROOT so .app/worktree builds point at canonical art | TODO (small, infra-grade) | du _private: 2.9G canonical, 0 in worktree | + +## EVAPORATED on clean re-verification (were CORRUPTED READS — do NOT chase) +- **G1 "palette all-disabled / PC kind=pc / no active character"** — FALSE. Live PC Rolan is `kind='player'`; palette enabled. (A corrupted snapshot read invented `kind='pc'`/`npc-alfira`. Builder A killed — pushed no PR.) +- **G2 "scene plate 404"** — FALSE. `location:loc-lower-city` → 200 from canonical. +- Earlier "zero images / no map" — environment (served from worktree/stale build), not code. + +## NOT bugs (proxy artifacts; confirmed absent in canonical) +- Doubled labels; dead-hero roster (fixed engine-side 4a1d6e8b); "Alfira→Rolan" (play_party pre-seed). + +## Loop discipline (corruption-hardened) +The tool channel intermittently FABRICATES file/snapshot reads (it invented G1/G2). **Before tasking a fix, verify the symptom with ≥2 clean reads against the LIVE surface (curl /session-surface, /image HTTP codes) — never a single Read.** Per fix: builder → PR → CI(viewer-tests) → merge → rebuild 8799 → LOOK → tick + proof. Gate = RRI on rebuilt `.app`. diff --git a/qa/SCORECARD.md b/qa/SCORECARD.md index a3875b81..5463e059 100644 --- a/qa/SCORECARD.md +++ b/qa/SCORECARD.md @@ -69,3 +69,14 @@ - **v1.0.3 — 2026-05-29 (WorldOS rename).** Cut from the rename milestone (main after #300). Project renamed ClawDnD → WorldOS over four CI-green PRs: #296 (plugin/app identity + branding), #299 (code+doc refs — 0 ClawDnD left in `servers/engine`), #300 (non-breaking `WORLDOS_*` env layer with `CLAWDND_*` fallback). MCP ids + `dev.clawdnd.app` bundle deliberately preserved as wire contracts. Engine **1511** + viewer 146 + voice 17 + rules 16 + axe **0/18** + license-check green; SwiftPM clean. Also carries the audit fixes #260/#268/#271/#282 + the canon-NPC reverse-picker. **Quality read (standing, honest):** the latest GREEN PC=Dal-Lightspark canon duos remain **story 4.3 (ow-fixC) / mech 4.0 (ow-fixD)** — this release added no gameplay logic, only the rename + the carried UI fixes, so those stand. **Caelar-invention is killed by CODE** (`load_canon_character(kind="player")` in play.sh/run_duo, commit 02ae6ae) + the **#162 player-in-party invariant**, and verified in the live GUI (owshot+Read shows Dal with portrait) — NOT by a fresh duo score. **NEXT loop iteration (owed):** a post-#162 clean GREEN Dal revalidation duo (the last logged Dal duo, ow-rv1, was RED on the player_in_party half-fix that #162 has since closed) + the ~30 remaining Wave-2-4 audit issues (#244–#290, per Eva Loop-6 #294). - **v1.0.2 — 2026-05-29 (graphics-release burndown).** Cut from `583b8a5` (tag `v1.0.2`, GitHub release live). Drove the OpenWorlds UI audit (epic #242 Phase 5) into the product: **axe-core 11→0** across 16 screens (#291/#292 + scroll-region focus); the `window.slug` item-art bug fixed (3 screens rendered placeholders despite art existing, #270); title-bar overlap on every screen (#260); Bestiary/Forge demo-leaks (#262/#263/#264); Create art (#265); engine player-in-party invariant + `update_character` skills alias + combat auto-hit doctrine; 4 new QA gates (`no_rejected_tool_calls`/`xp_awarded_on_progression` FATAL, `caster_has_spellbook`/`quest_objectives_progress` WARN). Engine 1435 + viewer 86 + axe 0 + license green; native app builds clean + codesign valid. Combat-sprint 3.0→3.7. **Next (toward v1.0.3): Wave-0 features** — ST-02 save/load+Export (in flight), #263 Bestiary intel-tier (designing), #266 seed write-lane, #265 portrait gateway-gen. Story/mech lens re-measure still blocked by the 16GB host's memory pressure (duos die mid-run; Codex code-index leak — restart Codex to reclaim ~2GB). - **v1.0.1 — 2026-05-28** · **v1.0.0 — 2026-05-27** (see CHANGELOG.md). + +--- + +## Release Sweeps (RRI — `qa/release_readiness.py`, 11 gates, 10/10 = release-ready) + +> One row per full gate sweep on a BUILT `.app`. RRI = gates_passed/11 × 10. Release at 10/10. +> See `WorldOS-OPERATING-GOAL.md` §4 for the gates, `WorldOS-GUI-RUNBOOK.md` for the sweep procedure. + +| Sweep | Date | build_sha | RRI | Native | Arc | Sat (avg/no-giveup) | Crit | Story | Mech | Behav | Axe | Img% | Palette | Failed gates / notes | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| (first sweep pending — Phase 3 on b23884c+) | | | | | | | | | | | | | | GUI fixes #410/#411/#412 merged + verified on 8799; sweep next | diff --git a/qa/release_readiness.py b/qa/release_readiness.py new file mode 100644 index 00000000..3ddac74b --- /dev/null +++ b/qa/release_readiness.py @@ -0,0 +1,205 @@ +#!/usr/bin/env python3 +"""WorldOS Release Readiness Index (RRI 0-10) — roll up existing QA artifacts into ONE +release number with a HARD-GATE FLOOR (a failed gate caps the score; it is NOT a soft +average that hides a zero — mirrors the SCORECARD RED-cap discipline). + +Pure reader of on-disk artifacts (never the live HTTP channel, which can corrupt): + - /run.json (ui_playtest_app.sh) -> part_a native #356 gate + - /score.json (ui_playtest_score.py) -> intro flow, satisfaction, bugs, image_404s + - /network.ndjson (palette/playwright) -> image-render rate (img 200 vs 404) + - / (score.sh + rubrics) -> story-craft / mechanical 1-5 + - --behavioral GREEN|RED (assert_behavioral.py exit) + - --ui-audit PASS|FAIL (ui_audit_health.sh exit) + - --palette-live true|false (a clean /session-surface read done by the CALLER, not here) + +The two NEW signals the plan calls for — image-render-rate and palette-live — are computed +here (image rate from score.json/network.ndjson) and passed in (palette-live), so this stays +a pure disk reader. + +Usage: + release_readiness.py --runs [,...] \ + [--story story.json] [--mech mech.json] \ + [--behavioral GREEN|RED] [--ui-audit PASS|FAIL] [--palette-live true|false] \ + [--build-sha SHA] [--out qa/RRI.json] [--scorecard-row] + +Targets for 10/10 (each dimension is a gate; all must hold on ONE build): + native gate PASS · arc completed · cross-persona satisfaction >=7 & no give-up · + 0 critical bugs · story >=4.3 · mech >=4.5 · behavioral GREEN · ui-audit PASS · + image-render >=95% · palette-live true +""" +from __future__ import annotations + +import argparse +import json +import sys +from pathlib import Path + + +def read_json(path: Path) -> dict: + if not path or not path.exists(): + return {} + try: + return json.loads(path.read_text(encoding="utf-8")) + except (json.JSONDecodeError, OSError): + return {} + + +def read_ndjson(path: Path) -> list[dict]: + out: list[dict] = [] + if not path or not path.exists(): + return out + for line in path.read_text(encoding="utf-8").splitlines(): + line = line.strip() + if not line: + continue + try: + out.append(json.loads(line)) + except json.JSONDecodeError: + continue + return out + + +def image_render_rate(run: Path, score: dict) -> tuple[float, int, int]: + """Fraction of image requests that returned bytes (not 404). Prefers network.ndjson + (200 vs 404 image responses); falls back to score.json's image_404s with an unknown + denominator (then rate is reported as 1.0 only if 0 404s, else conservative).""" + net = read_ndjson(run / "network.ndjson") + img = [n for n in net if "/image" in str(n.get("url", ""))] + if img: + ok = sum(1 for n in img if int(n.get("status", 0) or 0) and int(n.get("status")) < 400) + total = len(img) + return (ok / total if total else 1.0), ok, total + # fallback: score.json carries image_404s but not the success count + f404 = int(score.get("image_404s", 0) or 0) + if f404 == 0: + return 1.0, 0, 0 + # unknown denominator → report the 404 count, conservative rate + return 0.0, 0, f404 + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("--runs", required=True, help="comma-separated persona run dirs") + ap.add_argument("--story", default="") + ap.add_argument("--mech", default="") + ap.add_argument("--behavioral", default="", choices=["", "GREEN", "RED"]) + ap.add_argument("--ui-audit", dest="ui_audit", default="", choices=["", "PASS", "FAIL"]) + ap.add_argument("--palette-live", dest="palette_live", default="", choices=["", "true", "false"]) + ap.add_argument("--build-sha", dest="build_sha", default="") + ap.add_argument("--out", default="qa/RRI.json") + ap.add_argument("--scorecard-row", action="store_true") + args = ap.parse_args() + + run_dirs = [Path(p.strip()) for p in args.runs.split(",") if p.strip()] + persona_scores = [] + for rd in run_dirs: + sc = read_json(rd / "score.json") + if not sc: + continue + rate, ok, total = image_render_rate(rd, sc) + persona_scores.append({ + "run": sc.get("run") or rd.name, + "persona": sc.get("persona"), + "completed_intro_flow": bool(sc.get("completed_intro_flow")), + "satisfaction": sc.get("persona_satisfaction"), + "gave_up": bool(sc.get("gave_up")), + "critical": int(sc.get("bug_reports_critical", 0) or 0), + "image_rate": rate, + "image_ok": ok, + "image_total": total, + }) + + n = len(persona_scores) or 1 + sats = [p["satisfaction"] for p in persona_scores if isinstance(p["satisfaction"], (int, float))] + avg_sat = sum(sats) / len(sats) if sats else 0.0 + any_gave_up = any(p["gave_up"] for p in persona_scores) + any_completed = any(p["completed_intro_flow"] for p in persona_scores) + total_critical = sum(p["critical"] for p in persona_scores) + # weighted image rate across personas that recorded image traffic + img_runs = [p for p in persona_scores if p["image_total"] > 0] + img_rate = (sum(p["image_ok"] for p in img_runs) / sum(p["image_total"] for p in img_runs)) if img_runs else ( + 1.0 if persona_scores and all(p["image_rate"] >= 0.95 for p in persona_scores) else 0.0) + + story = read_json(Path(args.story)) if args.story else {} + mech = read_json(Path(args.mech)) if args.mech else {} + story_overall = float(story.get("overall", 0) or 0) + mech_overall = float(mech.get("overall", 0) or 0) + + # native gate: read part_a from any run.json present + native = "" + for rd in run_dirs: + rj = read_json(rd / "run.json") + pa = (rj.get("part_a") or {}).get("result") + if pa: + native = pa + break + + # ---- the 11 gates (each contributes to RRI; all must hold for 10/10) ---- + gates = { + "native_gate": (native == "PASS", f"part_a={native or 'n/a'}"), + "arc_completed": (any_completed, f"completed_intro_flow on >=1 persona"), + "cross_persona_sat": (avg_sat >= 7.0, f"avg={avg_sat:.1f}/10 over {len(sats)}"), + "no_give_up": (not any_gave_up, f"any_gave_up={any_gave_up}"), + "zero_critical": (total_critical == 0, f"critical={total_critical}"), + "story_craft": (story_overall >= 4.3, f"story={story_overall or 'n/a'}"), + "mechanical": (mech_overall >= 4.5, f"mech={mech_overall or 'n/a'}"), + "behavioral": (args.behavioral == "GREEN", f"behavioral={args.behavioral or 'n/a'}"), + "ui_audit": (args.ui_audit == "PASS", f"ui_audit={args.ui_audit or 'n/a'}"), + "image_render": (img_rate >= 0.95, f"rate={img_rate:.2%}"), + "palette_live": (args.palette_live == "true", f"palette_live={args.palette_live or 'n/a'}"), + } + passed = sum(1 for ok, _ in gates.values() if ok) + total_gates = len(gates) + + # RRI: each gate worth 10/total; HARD FLOOR — a missed gate can't be hidden by others. + # (Equal weight keeps it honest: "10/10" literally means every gate held.) + rri = round(10.0 * passed / total_gates, 1) + release_ready = passed == total_gates + + failed = [name for name, (ok, _) in gates.items() if not ok] + + result = { + "rri": rri, + "release_ready": release_ready, + "gates_passed": passed, + "gates_total": total_gates, + "failed_gates": failed, + "build_sha": args.build_sha, + "signals": { + "native_gate": native, + "arc_completed": any_completed, + "cross_persona_satisfaction": round(avg_sat, 1), + "any_gave_up": any_gave_up, + "total_critical_bugs": total_critical, + "story_overall": story_overall, + "mech_overall": mech_overall, + "behavioral": args.behavioral, + "ui_audit": args.ui_audit, + "image_render_rate": round(img_rate, 4), + "palette_live": args.palette_live, + }, + "gate_detail": {name: detail for name, (ok, detail) in gates.items()}, + "personas": persona_scores, + } + + out = Path(args.out) + out.write_text(json.dumps(result, indent=2), encoding="utf-8") + + # human line + print(f"RRI {rri}/10 ({passed}/{total_gates} gates) release_ready={release_ready}") + if failed: + print(" FAILED: " + ", ".join(f"{f} [{gates[f][1]}]" for f in failed)) + + if args.scorecard_row: + sha = (args.build_sha or "?")[:7] + row = (f"| RRI-{sha} | (date) | baldurs-gate | 5-persona | sonnet | gate | " + f"{'**GREEN**' if release_ready else 'RED'} | {story_overall or '—'} | " + f"{mech_overall or '—'} | — | **{rri}** | " + f"RRI {passed}/{total_gates}; failed: {', '.join(failed) or 'none'} |") + print(row) + + return 0 if release_ready else 1 + + +if __name__ == "__main__": + sys.exit(main())