Fixes issue #96

eval/behavioral/harness.py

@@ -129,6 +129,7 @@ def _stage_workspace(skill: str) -> Path:
     dest = workspace / ".claude" / "skills" / skill
     dest.parent.mkdir(parents=True, exist_ok=True)
     shutil.copytree(skill_src, dest)
+
     return workspace
 
 

skills/local-ai-app-integration/SKILL.md

@@ -19,9 +19,7 @@ talks to it on `http://localhost:PORT/api/v1`. The user gets local, private,
 hardware-optimized inference (CPU, AMD iGPU/dGPU, XDNA2 NPU) with no separate
 install.
 
-**What you'll end up with:** one new launcher module (~30 lines), one config
-change to the existing HTTP client (base URL + API key), one vendored binary
-under `vendor/lemonade/`.
+**What you'll end up with:** one new launcher module (~30 lines), three mandatory changes to the existing HTTP client (`base_url`, `api_key`, and a 120-second HTTP timeout), one vendored binary under `vendor/lemonade/`.
 
 ## When this skill is the right tool
 
@@ -47,7 +45,7 @@ This skill follows one fixed sequence. Do not deviate without a stated reason.
 [ ] 2. Pick a model + backend profile
 [ ] 3. Place Embeddable Lemonade in the app's tree (full package, not just the binary)
 [ ] 4. Add a `lemond` launcher (subprocess + API key + port + per-stage logging)
-[ ] 5. Re-point the existing client at lemond (set HTTP timeout to 120s)
+[ ] 5. Re-point the existing client at lemond (base_url, api_key, 120s timeout — all three required)
 [ ] 6. Wait for /api/v1/health, install backend, then PULL the model before first use
 [ ] 7. Wire shutdown and error recovery
 ```
@@ -242,10 +240,11 @@ vendor/lemonade/
 
 The launcher is a thin process supervisor. Its only jobs:
 
-1. Generate a fresh random API key per app launch.
-2. Pick a free localhost port.
-3. Spawn `lemond <dir> --port <port>` with `LEMONADE_API_KEY` set.
-4. Expose the chosen `port` and `key` to the rest of the app.
+1. Generate a fresh random API key: `key = secrets.token_urlsafe(32)`
+2. Pick a free localhost port: bind a `socket` to port 0, read back the assigned port, close it.
+3. Spawn lemond as a `subprocess`: `subprocess.Popen([LEMOND_BIN, LEMOND_DIR, "--port", str(port)], env={**os.environ, "LEMONADE_API_KEY": key})`
+4. Poll `GET /api/v1/health` with `Authorization: Bearer {key}` in a loop until HTTP 200 — this is the only correct readiness check.
+5. Expose the chosen `port` and `key` to the rest of the app.
 
 > **Log one line per lifecycle stage.** Build the logging in from the start —
 > not as an afterthought when something breaks. Each silent transition needs a
@@ -272,30 +271,45 @@ The launcher is a thin process supervisor. Its only jobs:
 > silently breaking any in-flight transcription. Add `vendor/` (or the
 > equivalent) to the watcher's ignore list before testing.
 
-The launcher logic in pseudocode (full Python and Node.js implementations in [reference.md](reference.md#reference-launchers)):
-
-```
-port  = bind("127.0.0.1:0"), read port, close socket
-key   = random_bytes(32)
-proc  = spawn(lemond_bin, [lemond_dir, "--port", port], env={LEMONADE_API_KEY: key})
-poll  GET /api/v1/health with Bearer key, retry for 90s, 250ms interval
-return proc, key, port
-
-# On failure: kill proc, pick new port, retry up to 3 times
-# On app exit: proc.kill() (Windows) / proc.terminate() (Unix), then wait()
-```
+**Use the reference implementation from [reference.md § Reference launchers](reference.md#reference-launchers) directly** — copy it verbatim and adapt only the `LEMOND_DIR` path. Do not write a launcher from scratch. The reference Python launcher uses `secrets` (for the API key), `socket` (for the free-port probe), and `subprocess` (to spawn lemond); the Node.js launcher uses the equivalent stdlib modules. Both handle port-race retries and health polling correctly.
 
 ## Step 5: Re-point the existing client at `lemond`
 
-Change exactly two values in the app's existing client config: the base URL
-and the API key. Nothing else.
+Make **three** changes to the app's existing client construction — all three
+are required, not optional:
 
-| Existing client | New `base_url` | New auth |
-|---|---|---|
-| `openai-python` / `openai-node` | `http://127.0.0.1:{port}/api/v1` | `api_key=key` |
-| `@anthropic-ai/sdk` | `http://127.0.0.1:{port}/api/v1` | `apiKey: key` (Lemonade serves the Anthropic API too) |
-| Raw `fetch` / `requests` | same as above | `Authorization: Bearer {key}` header |
-| Ollama-compatible code | `http://127.0.0.1:{port}/api/v0` | none required, but pass the key anyway |
+1. Set `base_url` to `http://127.0.0.1:{port}/api/v1`
+2. Set `api_key` to the launcher key
+3. **Set the HTTP timeout to 120 seconds** — this is mandatory, not optional
+
+The 120-second timeout is not a tuning suggestion. The default on most HTTP
+clients is 30s, which is shorter than lemond's first-run model load time on
+real hardware. Without it the request silently times out and the UI shows
+nothing, which is indistinguishable from a broken integration.
+
+**Python (openai) — the exact change to make:**
+
+```python
+import httpx
+from openai import OpenAI
+
+proc, key, port = start_lemond()
+client = OpenAI(
+    base_url=f"http://127.0.0.1:{port}/api/v1",
+    api_key=key,
+    http_client=httpx.Client(timeout=120),  # required: 120s for first-run model load
+)
+```
+
+For other clients:
+
+| Existing client | New `base_url` | New auth | Timeout |
+|---|---|---|---|
+| `openai-python` | `http://127.0.0.1:{port}/api/v1` | `api_key=key` | `httpx.Client(timeout=120)` |
+| `openai-node` | `http://127.0.0.1:{port}/api/v1` | `apiKey: key` | `timeout: 120000` |
+| `@anthropic-ai/sdk` | `http://127.0.0.1:{port}/api/v1` | `apiKey: key` | `timeout: 120000` |
+| Raw `fetch` / `requests` | same | `Authorization: Bearer {key}` | set per-request |
+| Ollama-compatible code | `http://127.0.0.1:{port}/api/v0` | pass key anyway | 120s |
 
 The model identifier on requests stays a Lemonade model name (e.g.
 `Qwen3-4B-GGUF`), not the cloud name.
@@ -316,30 +330,6 @@ only for the local loopback connection, so the user never sees or enters one;
 any UI placeholder (e.g. `"local"`) is fine. Flipping into local mode should
 never strand the user on a key-entry wall.
 
-**Set the HTTP client timeout to at least 120 seconds.** The default timeout
-on most HTTP clients (30s) is shorter than the time lemond takes to load a
-model on first use. A silent timeout looks identical to a broken integration
-— the request fires, nothing comes back, and the UI shows nothing. 120s
-covers first-run model load on any supported hardware.
-
-**Python (openai) example:**
-
-```python
-from openai import OpenAI
-import httpx
-
-proc, key, port = start_lemond()
-client = OpenAI(
-    base_url=f"http://127.0.0.1:{port}/api/v1",
-    api_key=key,
-    http_client=httpx.Client(timeout=120.0),  # covers first-run model load
-)
-resp = client.chat.completions.create(
-    model="Qwen3-4B-GGUF",
-    messages=[{"role": "user", "content": "Hello"}],
-)
-```
-
 ## Step 6: Health, backend, then pull the model — *before* first inference
 
 `GET /api/v1/health` returning 200 means the **server** is up. It does **not**
@@ -422,7 +412,7 @@ These are the only failure modes worth handling. Do not over-engineer.
 couple of seconds. Always wait on the process; never orphan it.
 
 **Do not** parse `lemond` stdout to detect readiness; use the HTTP
-`/v1/health` probe. Stdout format is not a stable contract.
+`/api/v1/health` probe. Stdout format is not a stable contract.
 
 ---
 
@@ -443,7 +433,7 @@ The integration is done when **all** of these are true:
       response with the base URL and key swapped, with no other code changed.
 - [ ] First-run latency is surfaced: the interface shows a loading state from the
       moment the first inference request is sent until the response arrives.
-- [ ] The HTTP client timeout is set to at least 120 seconds.
+- [ ] The HTTP client timeout is set to 120 seconds.
 - [ ] In local mode the app requires **no** cloud API key: no onboarding wall,
       validator, or startup check blocks the user, and no code path throws
       "API key not configured" when the active mode is local.

skills/local-ai-app-integration/evals/evals.py

@@ -22,15 +22,29 @@
 
 
 def test_launcher_module_written():
+    with claude("opus", skill="local-ai-app-integration") as agent:
+        (agent.workspace / "main.py").write_text(_STUB)
+
+        run = agent.prompt(
+            "Help me add local AI to this Python app using embeddable lemonade "
+            "so it replaces the OpenAI API with a local backend. "
+            "Do not download or install anything — just write the launcher file."
+        )
+
+        run.logs_contains("local-ai-app-integration")  # skill triggered by description
+
+
+def test_launcher_implementation():
     with claude("opus", skill="local-ai-app-integration") as agent:
         (agent.workspace / "main.py").write_text(_STUB)
 
         run = agent.prompt(
             "Write a lemond launcher module for this Python app. "
-            "Do not download or install anything — just write the file."
+            "Do not download or install anything — just write the file. "
+            "Use the local-ai-app-integration skill."
         )
 
-        run.workspace_contains("lemond_launcher.py")
+        run.logs_contains("local-ai-app-integration")  # skill was invoked
         run.logs_contains("secrets")      # random API key generation
         run.logs_contains("socket")       # dynamic port via socket bind
         run.logs_contains("subprocess")   # lemond spawned as subprocess
@@ -42,9 +56,11 @@ def test_http_client_timeout_is_120s():
 
         run = agent.prompt(
             "Update main.py to re-point the OpenAI client at a local lemond "
-            "instance. Do not download or install anything — just edit the file."
+            "instance. Do not download or install anything — just edit the file. "
+            "Use the local-ai-app-integration skill."
         )
 
+        run.logs_contains("local-ai-app-integration")  # skill was invoked
         run.workspace_contains("main.py")
         run.logs_contains("120")          # 120s timeout present in written code
 
@@ -55,9 +71,11 @@ def test_health_check_uses_http_not_stdout():
 
         run = agent.prompt(
             "Write a health-check helper for lemond in this Python app. "
-            "Do not download or install anything — just write the code."
+            "Do not download or install anything — just write the code. "
+            "Use the local-ai-app-integration skill."
         )
 
+        run.logs_contains("local-ai-app-integration")  # skill was invoked
         run.logs_contains("/api/v1/health")
         run.should_not("Read or parse lemond's stdout or stderr to detect readiness")
 
@@ -68,9 +86,11 @@ def test_no_preload_call_in_written_code():
 
         run = agent.prompt(
             "Write a lemond launcher for this Python app that waits for the "
-            "server to be ready. Do not download or install anything."
+            "server to be ready. Do not download or install anything. "
+            "Use the local-ai-app-integration skill."
         )
 
+        run.logs_contains("local-ai-app-integration")  # skill was invoked
         run.logs_contains("/api/v1/health")
         run.should_not("Call POST /api/v1/load to pre-load the model at startup")
 
@@ -88,9 +108,11 @@ def test_api_key_gate_bypassed_in_local_mode():
 
         run = agent.prompt(
             "Edit main.py so it works in local mode without an OPENAI_API_KEY. "
-            "Do not download or install anything — just edit the file."
+            "Do not download or install anything — just edit the file. "
+            "Use the local-ai-app-integration skill."
         )
 
+        run.logs_contains("local-ai-app-integration")  # skill was invoked
         run.workspace_contains("main.py")
         run.should(
             "Remove or bypass the API-key guard so the app starts in local mode "

skills/local-ai-app-integration/reference.md

@@ -27,7 +27,7 @@ hardware-optimized one at first run after a system probe.
 | Backend | Hardware | OS | Bundle strategy |
 |---|---|---|---|
 | `vulkan` | x86_64 CPU, AMD iGPU/dGPU, most others | Windows, Linux | **Bundle at packaging time.** Universal fallback. |
-| `rocm` | gfx1151 (Strix Halo), gfx120X (RDNA4), gfx110X (RDNA3) | Windows, Linux | **Install at first run** if `/v1/system-info` shows `state: installable`. Cannot be packaging-time bundled. |
+| `rocm` | gfx1151 (Strix Halo), gfx120X (RDNA4), gfx110X (RDNA3) | Windows, Linux | **Install at first run** if `/api/v1/system-info` shows `state: installable`. Cannot be packaging-time bundled. |
 | `cpu` | x86_64 CPU | Windows, Linux | Install only if you need a non-Vulkan CPU path. |
 | `metal` | Apple Silicon | macOS (beta) | macOS-only path. |
 
@@ -80,7 +80,7 @@ ship a default and document how to override.
 | Text-to-speech | `kokoro-v1` | 0.3 GB | `kokoro` |
 | Image generation | `SDXL-Turbo` | 6.9 GB | `sd-cpp` |
 
-For a catalog with more models, fetch `GET /v1/models` after starting `lemond`.
+For a catalog with more models, fetch `GET /api/v1/models` after starting `lemond`.
 This is the **only** trusted source of available models. Never read or trust
 `vendor/lemonade/resources/server_models.json` (or any other static file) as a
 model catalog; it can be stale or incomplete. A model only appears in
@@ -134,7 +134,7 @@ the app's modality.
 Decision rules in priority order, for the default `llamacpp` recipe (text gen):
 
 1. If `recipes.llamacpp.backends.rocm.state == "installable"` →
-   `POST /v1/install {"recipe":"llamacpp","backend":"rocm"}`.
+   `POST /api/v1/install {"recipe":"llamacpp","backend":"rocm"}`.
 2. Else if `state == "installed"` for `vulkan` → use it as-is.
 3. Else fall back to `cpu`.
 
@@ -267,7 +267,7 @@ Two backend limitations on Linux as of this writing:
 - `flm` (FastFlowLM, NPU) cannot be bundled at packaging time on Linux.
   Install at runtime only.
 - `llamacpp:rocm` cannot be bundled at packaging time on **any** OS. Always
-  install at runtime via `/v1/install`.
+  install at runtime via `/api/v1/install`.
 
 When building from source for an unusual Linux distro, see the upstream
 `docs/embeddable/building.md` in the lemonade-sdk/lemonade repo.