[KEYSTONE] Live Composition — stream the DM's scene as it is written (tool-arg delta streaming)

[100yenadmin]

The problem (and why this is the single highest-leverage feature in WorldOS)

DM beats are generation-bound: ~100–126s routine, 280–400s cold-open (worldos-latency-forensics, measured via duration_api_ms; all wall-clock speed levers measured/REFUTED). Today the player stares at a spinner for the entire beat, then the full scene appears at once. Every sat-killer in rc1/rc2 traces here: perceived hangs, give-ups, "the game is frozen." We cannot remove the wait — we must make the wait the experience: the scene should unfold live, word by word, as the DM composes it.

Why every previous agent failed (read this before re-deriving)

Every prior attempt looked at what the wrappers parse today — the final result event of claude -p --output-format stream-json — and concluded "the CLI only yields text when the beat completes." Two facts were missed:

1. --include-partial-messages exists in the installed CLI (verified claude --help, v2.1.x). With it, the stream-json output carries stream_event rows wrapping the raw API stream — content_block_start, content_block_delta, etc. — written incrementally to the same stdout/file as the model generates.
2. The "which text is player-safe?" problem (a beat is MULTI-turn: tool calls, internal planning, then the scene — naively streaming assistant text leaks scaffolding, the [security][major] XSS partial sanitization — inner text leaks into chronicle as a player action #732 class) dissolves when you stream the TOOL ARGUMENT instead of assistant text. content_block_start announces the tool name before its args stream; the args then arrive as input_json_delta.partial_json chunks. The DM already writes its player-facing scene through log_event(kind="narration") — so the prose is deterministically player-safe by construction, identified before the first word arrives.

Probe evidence (2026-06-10, both verified on this machine)

# PROBE A — prose deltas stream live:
claude -p "<prompt>" --output-format stream-json --include-partial-messages --verbose
→ {"type":"stream_event",...{"type":"text_delta","text":" weathered lantern flickers above the tavern door as dusk bleeds purple..."}}

# PROBE B — TOOL-ARG deltas stream live (the keystone):
claude -p "Use the Write tool to ..." --allowedTools Write --output-format stream-json --include-partial-messages
→ {"content_block_start","index":1,"content_block":{"type":"tool_use","id":"toolu_...","name":"Write","input":{}}}   ← name FIRST
→ {"type":"input_json_delta","partial_json":"led streets into silver rivers that race toward the harbor. Lantern light sm"}  ← scene text streaming inside the tool arg

Architecture (5 phases, each independently shippable)

Phase 1 — wrapper flag. Add --include-partial-messages to the DM claude -p invocations in scripts/play.sh dm_turn (~L273 _dm_invoke), scripts/play_party.sh, qa/run_duo.sh — behind WORLDOS_STREAM_BEATS (default ON for GUI lanes). The per-attempt $out jsonl files ($DM_LOG.<ts>.jsonl) become live streams. Zero behavior change to anything that reads only the final result event. Cost: none (same generation; more stdout JSON).

Phase 2 — the stream tailer (scripts/stream_tailer.py, python stdlib, launched per-beat by the wrapper next to _dm_invoke, killed on beat end). State machine over the tailing $out:

• On content_block_start with content_block.type=="tool_use" and name in {"log_event","mcp__clawdnd-engine__log_event"} → enter CANDIDATE mode for that block index.
• Accumulate input_json_delta.partial_json for that index into a buffer. Incrementally scan for the "text" key (and the "kind" key — stream ONLY once kind is known ∈ {narration, dialogue}; if kind appears after text in key order, buffer until known — JSON key order varies, the tailer must handle both orders).
• Incrementally JSON-unescape the text value as it grows (a tiny unescape state machine: track backslash/u-escape state across chunk boundaries; emit only fully-decoded characters).
• Append decoded prose chunks to $STATE_DIR/stream/current.jsonl rows: {"seq":N,"text":"...","beat":B,"ts":...} + truncate/reset on each new beat or retry attempt (the retry mints a new $out; the tailer follows the newest attempt file — reuse the wrapper's knowledge of $out).
• On content_block_stop → mark the stream segment complete. Multiple log_event calls per beat stream sequentially (each its own segment).
• NEVER stream assistant text_delta in v1 (scaffolding risk); a WORLDOS_STREAM_REPLY_TEXT=1 experimental mode may add it later gated by a sentinel marker.

Phase 3 — viewer surface. New read-only /stream endpoint in viewer/server.py (polls the stream file — the viewer stays a pure reader; the stream file is wrapper-owned sidecar state, NOT campaign state, so the engine-sole-writer invariant is untouched). In viewer/openworlds/ the chronicle renders a "composing" block: typewriter-render of the growing segments, visually marked live (subtle cursor). On beat resolution the canonical /events narration row replaces the composing block — wire into the EXISTING dedup family (engine_logged + eventsStreamedThisTurnRef + the #763 ingest machinery): the composing block is keyed by beat + segment seq, and the canonical row clears it. The #740 sanitizer applies to streamed text too. Wrapper heartbeat lines (#743/#763 WRAPPER_PROGRESS_LINES) are filtered from the stream as they are everywhere else.

Phase 4 — DM contract. Tighten the dungeon-master skill + wrapper prompts: "compose the final player-facing scene by calling log_event(kind="narration", text=<the full scene>), THEN end your turn with a brief reply." This is already the dominant behavior (#357's empty-reply fallback exists precisely because DMs end on log_event); the contract makes it reliable. Non-compliance degrades gracefully: no stream → today's spinner+heartbeat behavior (zero regression).

Phase 5 — telemetry + scoring. Stamp time_to_first_prose_s per beat (first streamed chunk ts − beat submit ts) into the run telemetry; add the column to qa/scores_db.py (additive migration); the #753 latency budget gates on it (e.g. p95 ≤ 60s routine / ≤ 150s cold-open). The persona player agents (qa/playwright palette) will SEE the composing block — expect cross_persona_sat lift; measure it.

What this moves

• cross_persona_sat (→7): the Epic 0: Scaffold + OSS-reuse spike + SRD ingest #1 felt-quality lever — the wait becomes the show. - no_give_up: a player watching prose unfold does not conclude "frozen." - Latency scoring: time_to_first_prose becomes the budgetable metric ([ux][latency] Define a per-beat latency BUDGET for the GUI loop (not just telemetry) #753). - Story immersion: the typewriter IS dramatic pacing.

Edge cases & risks (write tests for each)

1. JSON-escape across chunk boundaries (\" split between deltas; \uXXXX split mid-escape) → the unescape state machine; property-test with adversarial chunkings of a fixed scene.
2. kind key after text key → buffer-until-kind; test both key orders.
3. Retry mid-beat → stream file resets; the composing block clears (test: simulated retry).
4. log_event with non-narration kind (system/roll) → never streamed (test).
5. The DM writes the scene in MULTIPLE log_event calls → sequential segments render in order (test).
6. Tailer crash → wrapper unaffected (it's a sidecar); stream file simply stops; the beat resolves normally (test: kill the tailer mid-beat).
7. tool name namespacing: in-process MCP names may be mcp__clawdnd-engine__log_event — match on suffix (verify against a real GUI-lane transcript before hardcoding).
8. Player-agent (palette) compatibility: the a11y tree must not flood with per-token updates — batch render at ~250ms cadence (also kinder to the [a11y][major] Chronicle floods the accessibility tree by beat 2 — Do/Declare unreachable for AT users #752 a11y-overflow work).

Acceptance (the gate for closing this issue)

On a built-app GUI run @ the implementing SHA: (a) prose visibly streams during a routine beat with time_to_first_prose ≤ 60s p50; (b) an adversarial-persona transcript shows ZERO internal scaffolding/tool-JSON leaked to the chronicle; (c) the canonical narration row replaces the composing block with no duplication (extend the #720/#727 dedup tests); (d) a beat with streaming disabled (WORLDOS_STREAM_BEATS=0) behaves byte-identically to today; (e) time_to_first_prose recorded in the run telemetry + ledger row.

Phase-0 for the implementing agent

Re-run both probes above (they cost <$0.25 total) and one REAL play.sh beat with the flag on, captured to a file — confirm the engine's log_event tool-arg streams through the full MCP path before building the tailer. (The probes above used the built-in Write tool; the MCP-tool event shape must be confirmed identical — high confidence, but verify first.)