feat(qa): ui_playtest_app.sh — playtest the BUILT .app surface (§8.2, P0)

[100yenadmin]

What

The §8.2 infra (P0) from WorldOS-OPERATING-GOAL.md: a permanent, .app-faithful playtest harness, qa/ui_playtest_app.sh, that tests the BUILT dist/WorldOS.app surface (and the byte-identical backend the app shells) — never the dev viewer or a harness-wired port. This closes the failure that birthed the operating goal (§0): the old qa/ui_playtest.sh booted its own viewer with CLAWDND_PLAYER_MOVES set → can_act:true → a playable surface the harness wired for itself, while the shipped .app launches the viewer without that env → read-only. Every "all green" run there tested a surface the user can never reach.

Usage: qa/ui_playtest_app.sh <run> <world> <persona> <beats> <budget>

(A) Native-transition gate — re-verifies release-blocker #356

pkill WorldOSApp + this checkout's stale viewers → rm -rf dist/WorldOS.app → fresh script/build_and_run.sh (off HEAD, which must include #356) → launch the .app → discover the launcher viewer's actual port (the app's PortFinder walks up from preferredPort 8765, which is frequently already taken on a dev host — we never assume 8765) → raise the app frontmost via AppKit activateWithOptions (an app-activation API, not System-Events UI scripting, so no TCC dialog) → CGEvent-click the launcher's RESUME → PLAY CTA (computed from the window geometry; re-clicks on a focus-race miss) → assert the bridge minted a provider session: a new play-state/<run> dir appears and a new viewer (a different port) whose /session-surface reports can_act:true. PASS/FAIL is the session-surface ground truth, not a screenshot.

(B) Persona loop — the playable surface the .app reaches once #356 mints

Launch scripts/play_party.sh <world> <run> <port> (the exact command startProviderSession shells — ProviderAdapters.swift:62; solo → play.sh with a live viewer + DM cold-open + resolver loop), wait until player-ready (can_act + a seated PC + opening narration, with a grace fallback so an empty-narration bug is still surfaced rather than hanging the run), then drive the real Playwright palette (palette_server.js, CLAWDND_UIPT_URL) as <persona> for <beats> moves. Reuses ui_playtest_score.py + the existing persona briefs.

Artifacts land under qa/ui_playtest_runs/<run>/ tagged {build_sha, version}: native/transition.json, bugs.ndjson, score.json, summary.md, run.json. open() is shimmed so no browser pops on the owner's screen; the palette runs headless.

First-run results (this PR's verification)

#356 RE-VERIFY = GREEN. On a freshly-built .app off 3f946e7 (which includes #356), clicking the RESUME → PLAY banner minted run play-20260530104419; its provider viewer reported can_act:true, is_live_view:true, live:true, enabledActions=[continue,say,do,check,save], and the WebView transitioned from the read-only launcher to a playable table (free-text + Declare enabled). The harness's part-A gate reproduces this autonomously (transition.json: result=PASS). Caveat captured in the header: the CGEvent click is reliable on a pristine, frontmost launcher (a fresh build+launch, exactly what the gate does) but is flaky to re-issue against an already-clicked app on a busy multi-app desktop.

Newbie smoke (6 beats): reached the play screen, took 2 in-story turns, 0 dead clicks, 0 console errors. #357 (empty/missing opening narration) STILL REPRODUCES on 3f946e7: the DM cold-open seats the PC and builds the world, but writes no player-facing narration to /chat — the table sits locked on "Setting the opening scene…" and the newbie gave up waiting (1 major bug). (A fix/357-empty-narration branch already exists; this confirms it is not yet fixed on main.)

Notes

• Refs WorldOS-OPERATING-GOAL.md §8.2 / P0 / first-consequence Epic 1: Dice engine #2.
• Screenshots are best-effort: the harness process tree typically lacks Screen-Recording TCC, so screencapture falls back to dropping a request marker — the gate never depends on the image.
• Do not merge yet (per task) — opening for review.

Summary by CodeRabbit

• Chores
  • Added automated UI playtesting harness for comprehensive application verification and testing. Includes app launch surface validation, backend compatibility checks, automated persona-driven backend interactions, player session state verification, dynamic port discovery, resource consumption budgeting, screenshot capture for analysis, test scoring and evaluation, and detailed run metadata tracking with combined results reporting.

[coderabbitai]

Caution

Review failed

Pull request was closed or merged during review

📝 Walkthrough

Walkthrough

New Bash script qa/ui_playtest_app.sh implements a comprehensive UI playtesting harness with two parts: Part A validates native app launch by clicking the RESUME → PLAY button and confirming session creation; Part B executes persona-driven player agents against test backends using Playwright and Claude, with budget enforcement, port discovery, process cleanup, and result aggregation.

Changes

UI Playtesting Harness

Layer / File(s)	Summary
Harness initialization and setup qa/ui_playtest_app.sh (lines 1–102)	Parses CLI arguments and environment variables (run ID, world, persona, beats, budget, part), establishes run directories under qa/ui_playtest_runs/, computes build metadata (SHA, version), configures logging, disables native browser popup via a PATH shim, and defines the screenshot() helper and directory markers.
Part A: Native app launch validation qa/ui_playtest_app.sh (lines 117–327)	Elevates WorldOS app to foreground, discovers launcher viewer's dynamic port via /openworlds/ polling, captures before/after screenshots, synthesizes CGEvent clicks on the RESUME → PLAY button, polls for newly minted play-state and a viewer reporting can_act:true, captures final session-surface snapshot, writes native/transition.json with result, and tears down minted backend processes.
Part B: Player agent and backend orchestration qa/ui_playtest_app.sh (lines 328–472)	Validates persona brief and Playwright availability, enforces budget guard from DM spend, starts a separate faithful backend via scripts/play_party.sh on a free port, waits for player-ready signals (viewer reachable, can_act:true, seated player, chat narration), generates Playwright MCP config, executes claude -p player agent streaming to player.jsonl, invokes qa/ui_playtest_score.py scorer, and writes meta.json and scoring artifacts.
Supporting utilities and execution driver qa/ui_playtest_app.sh (lines 103–116, 474–530, 531–567)	dm_spend() aggregates DM cost from play-state directories; pick_free_port() probes available ports via /dev/tcp; repo_viewer_ports() enumerates running viewers via process matching; launcher_port_of() selects lowest port; _seated_player_count() parses claudnd snapshot. Main execution driver via case "$PART" runs Part A, B, or both; computes final spend totals; writes run.json combining results and metadata; and logs completion with conditional messaging.

Sequence Diagrams

sequenceDiagram
  participant App as WorldOS App
  participant Launcher as Launcher Viewer
  participant Backend as Minted Backend
  participant State as play-state dir
  
  Harness->>App: Elevate to foreground
  Harness->>Launcher: Poll /openworlds/ for port
  Harness->>Harness: Capture screenshot (before)
  Harness->>App: Synthesize RESUME → PLAY click
  App->>Backend: Trigger session creation
  Backend->>State: Mint play-state/{session}
  Harness->>Backend: Poll for new viewer + can_act:true
  Harness->>Harness: Capture session-surface snapshot
  Harness->>Backend: Tear down minted processes
  Harness->>State: Write native/transition.json

Loading

sequenceDiagram
  participant Harness
  participant Backend as Faithful Backend
  participant Viewer as Viewer / Snapshot
  participant Player as Claude Player Agent
  participant Scorer as Scorer
  
  Harness->>Harness: Enforce budget guard (DM spend)
  Harness->>Backend: Start scripts/play_party.sh on free port
  Backend->>Viewer: Initialize viewer & state
  Harness->>Viewer: Poll for can_act:true + seated player
  Viewer->>Viewer: Report player-ready signal
  Harness->>Player: Generate MCP config & run claude -p
  Player->>Backend: Interact via Playwright palette
  Backend->>Viewer: Update state & consume budget
  Player->>Harness: Stream output to player.jsonl
  Harness->>Scorer: Invoke qa/ui_playtest_score.py
  Scorer->>Harness: Produce summary.md & scoring

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related issues

• electricsheephq/WorldOS#324: This PR implements the complete AI-driven UI playtester harness (two-part design, Playwright/claude player, port discovery, budget enforcement, screenshots, scoring integration, and run directories) directly addressing the feature described in the issue.

Possibly related PRs

• electricsheephq/WorldOS#359: This PR's Part A specifically validates the native "RESUME → PLAY" CTA behavior and checks for can_act:true session state, directly testing the resumable play flow fixes introduced in #359.

Poem

🐰 A harness sprung forth from the warren,
To test the app's play button's bargain,
With screenshots keen and agents bold,
The persona's tale shall soon be told,
Run, validate, score—the quest grows strong! ✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately summarizes the main change: adding a new UI playtest harness script for testing the built .app surface, with a reference to the operating goal section.
Description check	✅ Passed	The PR description is comprehensive and addresses the template sections, though the CLA checklist is not explicitly checked off.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}