feat(ui): run the turn on a worker thread, marshal the UI to the loop

docs/DESIGN.md

@@ -2825,6 +2825,16 @@ A one-line status bar is pinned at the input as `prompt_toolkit`'s `bottom_toolb
 
 The interactive REPL is migrating from a per-turn `prompt_toolkit` `prompt` to a persistent full-screen `Application` on the alternate screen (`cli/app.py`, `build_app(...)`), so the status bar and input dock stay pinned and never scroll away mid-turn. The layout is a scrolling chat pane on top, a custom rounded multi-line **input dock**, and the persistent status bar (§31.11) pinned at the bottom. The dock border is **hand-drawn** (`horizontal_border(width, box, top=...)` — rounded `╭─╮`/`╰─╯`, ASCII `+`/`-` fallback) rather than `prompt_toolkit`'s `Frame`, which only draws square corners and reserves completion-menu height inside the box (filling the terminal height instead of hugging the input); this is the one deliberate exception to the §31.9 "borders from rich primitives" rule, since the dock lives in a `prompt_toolkit` layout. The dock border width, the pane wrap width, and the terminal width are one shared value read at render time, so a resize re-derives the border with nothing cached. Enter submits (a literal newline is Alt+Enter); the dock grows to a 10-row cap then scrolls; PageUp/PageDown scroll the (unfocused) pane while the wheel routes to the window under the cursor. The pane now renders Rich content via `AppUI` (`cli/app_ui.py`), which implements the `RuntimeUI` protocol. `AppUI` holds a `list[RenderableType]` as the transcript source of truth, with an optional in-progress open-response accumulator (accumulated token text rendered as `rich.Markdown`). `AppUI._render_ansi()` renders the renderables to a single ANSI string at the shared terminal width via a `Console(force_terminal=True, width=W)`; the result is cached by width so a resize re-derives it automatically (cache miss) when the pane's `FormattedTextControl(lambda: ANSI(ui._render_ansi()))` renders. `wrap_lines=False` on the pane window lets Rich own all wrapping. Secret redaction (`redact_structure`) and display-integrity path resolution (`workspace_display`) are applied in `show_tool_call` before the summary reaches the pane. Approval methods raise `NotImplementedError` until branch 7 wires the focus-swap; no silent default is acceptable there. `AppUI` never calls `get_app()` or `invalidate()` — those are branch-4 concerns — so it is fully testable without a running app. The non-TTY `PlainInput` path (§31.2) is unchanged.
 
+### 31.13 Worker-thread turn + loop-thread marshaling (v2, in progress)
+
+`ConversationRuntime.run_turn` drives the UI synchronously on the calling thread, calling `self._ui.<method>` for every event. In the full-screen app that thread is the prompt_toolkit event loop, so a long model turn would freeze the whole UI. Branch 4 (`cli/app_turn.py`) runs the one synchronous turn on a **worker thread** and marshals every UI callback back onto the loop thread, so `AppUI` state and the pane repaint are only ever touched on the loop thread.
+
+`ThreadedUI` is a `RuntimeUI` wrapping the real `AppUI` (`inner`) and an injected `schedule: Callable[[Callable[[], None]], None]`. Every fire-and-forget content method (`stream_token`, `stream_thinking`, `begin_response`/`end_response`, `turn_finished`, `show_status`, `show_error`, `show_tool_call`/`show_tool_result`, `show_command_output`, `show_plan_progress`) enqueues the inner call via `schedule(functools.partial(inner.method, *args))` — `functools.partial`, never a closure over a loop variable, so args bind at call time and cannot be rebound before the queued call runs. The blocking approval methods return a value, so they cannot be fire-and-forget: `ask_approval`/`ask_plan_approval` delegate straight to the inner UI and run on the worker thread; the inner `AppUI` still raises `NotImplementedError` (the focus-swap `Future` handshake is branch 7), and the worker's `try/except` surfaces that as a pane error — no approval ever silently defaults.
+
+`TurnRunner` owns the single worker thread for one turn and a `busy` flag. The core safety property: `busy` is SET in `start` (the dock-submit handler, which runs on the loop thread) and CLEARED in `_mark_done` (scheduled back onto the loop thread from the worker's `finally`), so it is only ever read or written on the loop thread — no lock needed. `start` ignores a submit while `busy` (single model, single conversation, single worker; the one-message queue is branch 9). The worker body `_run` touches nothing on the loop thread directly — it only calls `conversation.run_turn` (whose UI is the marshaling `ThreadedUI`, so every UI call is already marshaled) and routes its own error/completion through `schedule`; a `run_turn` exception is rendered as a pane error instead of silently killing the daemon thread, and the `finally` clears `busy` so a crashed turn never wedges the app (Ctrl-C turn cancellation is branch 6). `schedule` is injected so CI drives it synchronously; the real app uses `TurnRunner.schedule`, which reads `app.loop` lazily and schedules (via `loop.call_soon_threadsafe`) a callback that runs `fn()` then `app.invalidate()` (fails closed when the app is not running). prompt_toolkit coalesces the per-call `invalidate()` into one render tick, so per-token marshal+invalidate is correct without throttling.
+
+The opt-in runnable entry is `run_app` (`cli/app_main.py`), reached ONLY via `SHELLPILOT_UI=app` from `run_interactive` so a default session (TTY and non-TTY) stays byte-identical. The construction cycle — `schedule` needs `app`, `app` needs `runner.start`, `runner` needs the conversation, the conversation needs the `ThreadedUI`/`schedule` — is broken by build order and deferred attribute assignment: `AppUI` is built first (its `width_fn` reads `get_app()` lazily), then `TurnRunner` (its `schedule` reads `runner.app` lazily), then `ThreadedUI` over them; the conversation is constructed **with that `ThreadedUI` as its UI** (so its plan tools capture the marshaling bound methods — not repointed after the fact); then `build_app(on_submit=runner.start, ui=app_ui, …)`; then `runner.app`/`runner.conversation` are set after the app exists and before `app.run()`, by which time they are only read once a turn runs. The UI choice is made in `run_interactive` where the conversation is built (a `ThreadedUI` for app mode, the same `TerminalUI` as before otherwise); the default branch is unchanged. As an opt-in dev/live-test entry it has two known gaps to close when the app is promoted to the shipping default: `run_app` does not write the `session_end` audit event the default REPL emits, and a turn still in flight when `/exit` clears `app.loop` drops its final `busy`-clear in the already-dead app (harmless while the process is exiting).
+
 ## 32. Model Selection And Preload
 
 ### 32.1 Boot Model Picker

shellpilot/cli/app.py

@@ -139,12 +139,15 @@ def build_app(
     input: Input | None = None,
     output: Output | None = None,
     ui: AppUI | None = None,
+    on_submit: Callable[[str], None] | None = None,
 ) -> Application[None]:
-    """Build the inert full-screen app shell.
+    """Build the full-screen app shell.
 
     ``input``/``output`` default to the real terminal; tests inject a pipe input
-    and ``DummyOutput`` to drive the shell headlessly. The shell is standalone —
-    branch 4 wires it into the REPL; the non-TTY ``PlainInput`` path is untouched.
+    and ``DummyOutput`` to drive the shell headlessly. ``on_submit`` receives the
+    dock text on submit (branch 4 passes ``TurnRunner.start``); when it is None
+    the shell falls back to the inert branch-3 echo so the standalone shell and
+    its headless tests stay working. The non-TTY ``PlainInput`` path is untouched.
     """
     # One unicode/ascii decision drives the border, the branch glyph, and the
     # bar — recovered from the resolved glyph set the caller already probed
@@ -256,7 +259,10 @@ def _submit(event: KeyPressEvent) -> None:
             event.app.exit()
             return
         if text.strip():
-            _ui.show_status(text)
+            if on_submit is not None:
+                on_submit(text)
+            else:
+                _ui.show_status(text)
         dock_buffer.reset()
 
     @kb.add("escape", "enter", filter=dock_focused)

shellpilot/cli/app_main.py

@@ -0,0 +1,77 @@
+"""Opt-in runnable entry for the full-screen app (design section 31.13).
+
+This is the LIVE glue that constructs the full-screen ``Application``, drives a
+:class:`~shellpilot.runtime.conversation.ConversationRuntime` through a
+worker-thread :class:`~shellpilot.cli.app_turn.TurnRunner`, and ``app.run()``s
+it. It is reached ONLY via an explicit opt-in (``SHELLPILOT_UI=app``) from
+``run_interactive`` so the default REPL stays byte-identical; nothing here runs
+on a default ``shellpilot`` session.
+
+The construction cycle is broken by deferred attribute assignment — see the
+ordering note in :func:`run_app`.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Sequence
+from pathlib import Path
+from typing import TYPE_CHECKING
+
+from shellpilot.cli.app import build_app
+from shellpilot.cli.app_turn import TurnRunner
+
+if TYPE_CHECKING:
+    from shellpilot.cli.app_ui import AppUI
+    from shellpilot.cli.theme import Glyphs
+    from shellpilot.runtime.conversation import ConversationRuntime
+
+
+def run_app(
+    runtime: ConversationRuntime,
+    runner: TurnRunner,
+    app_ui: AppUI,
+    *,
+    workspace: Path,
+    model: str,
+    profile: str,
+    glyphs: Glyphs,
+    commands: Sequence[str],
+    is_cloud: bool = False,
+    ctx_pct: int = 0,
+) -> int:
+    """Build the full-screen app around an already-wired conversation and run it.
+
+    The caller has already chosen the conversation's UI as the marshaling
+    :class:`ThreadedUI` whose inner is ``app_ui`` and whose schedule is
+    ``runner.schedule`` (so the runtime's plan tools captured the marshaling
+    bound methods at construction). This function only completes the wiring:
+
+    1. ``app_ui`` was built first (its ``width_fn`` reads ``get_app()`` lazily,
+       so it needs no running app yet); ``runner`` was built next (its
+       ``schedule`` reads ``runner.app`` lazily); the conversation was built
+       with the ``ThreadedUI`` over ``app_ui``.
+    2. Build the ``Application`` from ``runner.start`` (the dock-submit handler)
+       and ``app_ui`` (the pane source of truth).
+    3. Set ``runner.app`` and ``runner.conversation`` AFTER the app exists and
+       BEFORE ``app.run()`` — both are read only once a turn runs, by which time
+       ``app.run()`` has set ``app.loop``.
+
+    NOTE: this opt-in entry does not write the ``session_end`` audit event that
+    the default REPL writes after its loop; it is a dev/live-test entry, not the
+    shipping path. Returns 0 so ``run_interactive`` can ``return run_app(...)``.
+    """
+    app = build_app(
+        workspace=workspace,
+        model=model,
+        profile=profile,
+        glyphs=glyphs,
+        commands=commands,
+        is_cloud=is_cloud,
+        ctx_pct=ctx_pct,
+        ui=app_ui,
+        on_submit=runner.start,
+    )
+    runner.app = app
+    runner.conversation = runtime
+    app.run()
+    return 0