diff --git a/eval/behavioral/harness.py b/eval/behavioral/harness.py index dbd7459..da624b4 100644 --- a/eval/behavioral/harness.py +++ b/eval/behavioral/harness.py @@ -129,6 +129,7 @@ def _stage_workspace(skill: str) -> Path: dest = workspace / ".claude" / "skills" / skill dest.parent.mkdir(parents=True, exist_ok=True) shutil.copytree(skill_src, dest) + return workspace diff --git a/skills/local-ai-app-integration/SKILL.md b/skills/local-ai-app-integration/SKILL.md index e9e4ee5..4f6965b 100644 --- a/skills/local-ai-app-integration/SKILL.md +++ b/skills/local-ai-app-integration/SKILL.md @@ -19,9 +19,7 @@ talks to it on `http://localhost:PORT/api/v1`. The user gets local, private, hardware-optimized inference (CPU, AMD iGPU/dGPU, XDNA2 NPU) with no separate install. -**What you'll end up with:** one new launcher module (~30 lines), one config -change to the existing HTTP client (base URL + API key), one vendored binary -under `vendor/lemonade/`. +**What you'll end up with:** one new launcher module (~30 lines), three mandatory changes to the existing HTTP client (`base_url`, `api_key`, and a 120-second HTTP timeout), one vendored binary under `vendor/lemonade/`. ## When this skill is the right tool @@ -47,7 +45,7 @@ This skill follows one fixed sequence. Do not deviate without a stated reason. [ ] 2. Pick a model + backend profile [ ] 3. Place Embeddable Lemonade in the app's tree (full package, not just the binary) [ ] 4. Add a `lemond` launcher (subprocess + API key + port + per-stage logging) -[ ] 5. Re-point the existing client at lemond (set HTTP timeout to 120s) +[ ] 5. Re-point the existing client at lemond (base_url, api_key, 120s timeout — all three required) [ ] 6. Wait for /api/v1/health, install backend, then PULL the model before first use [ ] 7. Wire shutdown and error recovery ``` @@ -242,10 +240,11 @@ vendor/lemonade/ The launcher is a thin process supervisor. Its only jobs: -1. Generate a fresh random API key per app launch. -2. Pick a free localhost port. -3. Spawn `lemond --port ` with `LEMONADE_API_KEY` set. -4. Expose the chosen `port` and `key` to the rest of the app. +1. Generate a fresh random API key: `key = secrets.token_urlsafe(32)` +2. Pick a free localhost port: bind a `socket` to port 0, read back the assigned port, close it. +3. Spawn lemond as a `subprocess`: `subprocess.Popen([LEMOND_BIN, LEMOND_DIR, "--port", str(port)], env={**os.environ, "LEMONADE_API_KEY": key})` +4. Poll `GET /api/v1/health` with `Authorization: Bearer {key}` in a loop until HTTP 200 — this is the only correct readiness check. +5. Expose the chosen `port` and `key` to the rest of the app. > **Log one line per lifecycle stage.** Build the logging in from the start — > not as an afterthought when something breaks. Each silent transition needs a @@ -272,30 +271,45 @@ The launcher is a thin process supervisor. Its only jobs: > silently breaking any in-flight transcription. Add `vendor/` (or the > equivalent) to the watcher's ignore list before testing. -The launcher logic in pseudocode (full Python and Node.js implementations in [reference.md](reference.md#reference-launchers)): - -``` -port = bind("127.0.0.1:0"), read port, close socket -key = random_bytes(32) -proc = spawn(lemond_bin, [lemond_dir, "--port", port], env={LEMONADE_API_KEY: key}) -poll GET /api/v1/health with Bearer key, retry for 90s, 250ms interval -return proc, key, port - -# On failure: kill proc, pick new port, retry up to 3 times -# On app exit: proc.kill() (Windows) / proc.terminate() (Unix), then wait() -``` +**Use the reference implementation from [reference.md § Reference launchers](reference.md#reference-launchers) directly** — copy it verbatim and adapt only the `LEMOND_DIR` path. Do not write a launcher from scratch. The reference Python launcher uses `secrets` (for the API key), `socket` (for the free-port probe), and `subprocess` (to spawn lemond); the Node.js launcher uses the equivalent stdlib modules. Both handle port-race retries and health polling correctly. ## Step 5: Re-point the existing client at `lemond` -Change exactly two values in the app's existing client config: the base URL -and the API key. Nothing else. +Make **three** changes to the app's existing client construction — all three +are required, not optional: -| Existing client | New `base_url` | New auth | -|---|---|---| -| `openai-python` / `openai-node` | `http://127.0.0.1:{port}/api/v1` | `api_key=key` | -| `@anthropic-ai/sdk` | `http://127.0.0.1:{port}/api/v1` | `apiKey: key` (Lemonade serves the Anthropic API too) | -| Raw `fetch` / `requests` | same as above | `Authorization: Bearer {key}` header | -| Ollama-compatible code | `http://127.0.0.1:{port}/api/v0` | none required, but pass the key anyway | +1. Set `base_url` to `http://127.0.0.1:{port}/api/v1` +2. Set `api_key` to the launcher key +3. **Set the HTTP timeout to 120 seconds** — this is mandatory, not optional + +The 120-second timeout is not a tuning suggestion. The default on most HTTP +clients is 30s, which is shorter than lemond's first-run model load time on +real hardware. Without it the request silently times out and the UI shows +nothing, which is indistinguishable from a broken integration. + +**Python (openai) — the exact change to make:** + +```python +import httpx +from openai import OpenAI + +proc, key, port = start_lemond() +client = OpenAI( + base_url=f"http://127.0.0.1:{port}/api/v1", + api_key=key, + http_client=httpx.Client(timeout=120), # required: 120s for first-run model load +) +``` + +For other clients: + +| Existing client | New `base_url` | New auth | Timeout | +|---|---|---|---| +| `openai-python` | `http://127.0.0.1:{port}/api/v1` | `api_key=key` | `httpx.Client(timeout=120)` | +| `openai-node` | `http://127.0.0.1:{port}/api/v1` | `apiKey: key` | `timeout: 120000` | +| `@anthropic-ai/sdk` | `http://127.0.0.1:{port}/api/v1` | `apiKey: key` | `timeout: 120000` | +| Raw `fetch` / `requests` | same | `Authorization: Bearer {key}` | set per-request | +| Ollama-compatible code | `http://127.0.0.1:{port}/api/v0` | pass key anyway | 120s | The model identifier on requests stays a Lemonade model name (e.g. `Qwen3-4B-GGUF`), not the cloud name. @@ -316,30 +330,6 @@ only for the local loopback connection, so the user never sees or enters one; any UI placeholder (e.g. `"local"`) is fine. Flipping into local mode should never strand the user on a key-entry wall. -**Set the HTTP client timeout to at least 120 seconds.** The default timeout -on most HTTP clients (30s) is shorter than the time lemond takes to load a -model on first use. A silent timeout looks identical to a broken integration -— the request fires, nothing comes back, and the UI shows nothing. 120s -covers first-run model load on any supported hardware. - -**Python (openai) example:** - -```python -from openai import OpenAI -import httpx - -proc, key, port = start_lemond() -client = OpenAI( - base_url=f"http://127.0.0.1:{port}/api/v1", - api_key=key, - http_client=httpx.Client(timeout=120.0), # covers first-run model load -) -resp = client.chat.completions.create( - model="Qwen3-4B-GGUF", - messages=[{"role": "user", "content": "Hello"}], -) -``` - ## Step 6: Health, backend, then pull the model — *before* first inference `GET /api/v1/health` returning 200 means the **server** is up. It does **not** @@ -422,7 +412,7 @@ These are the only failure modes worth handling. Do not over-engineer. couple of seconds. Always wait on the process; never orphan it. **Do not** parse `lemond` stdout to detect readiness; use the HTTP -`/v1/health` probe. Stdout format is not a stable contract. +`/api/v1/health` probe. Stdout format is not a stable contract. --- @@ -443,7 +433,7 @@ The integration is done when **all** of these are true: response with the base URL and key swapped, with no other code changed. - [ ] First-run latency is surfaced: the interface shows a loading state from the moment the first inference request is sent until the response arrives. -- [ ] The HTTP client timeout is set to at least 120 seconds. +- [ ] The HTTP client timeout is set to 120 seconds. - [ ] In local mode the app requires **no** cloud API key: no onboarding wall, validator, or startup check blocks the user, and no code path throws "API key not configured" when the active mode is local. diff --git a/skills/local-ai-app-integration/evals/evals.py b/skills/local-ai-app-integration/evals/evals.py index 0536dd9..6f91b21 100644 --- a/skills/local-ai-app-integration/evals/evals.py +++ b/skills/local-ai-app-integration/evals/evals.py @@ -22,15 +22,29 @@ def test_launcher_module_written(): + with claude("opus", skill="local-ai-app-integration") as agent: + (agent.workspace / "main.py").write_text(_STUB) + + run = agent.prompt( + "Help me add local AI to this Python app using embeddable lemonade " + "so it replaces the OpenAI API with a local backend. " + "Do not download or install anything — just write the launcher file." + ) + + run.logs_contains("local-ai-app-integration") # skill triggered by description + + +def test_launcher_implementation(): with claude("opus", skill="local-ai-app-integration") as agent: (agent.workspace / "main.py").write_text(_STUB) run = agent.prompt( "Write a lemond launcher module for this Python app. " - "Do not download or install anything — just write the file." + "Do not download or install anything — just write the file. " + "Use the local-ai-app-integration skill." ) - run.workspace_contains("lemond_launcher.py") + run.logs_contains("local-ai-app-integration") # skill was invoked run.logs_contains("secrets") # random API key generation run.logs_contains("socket") # dynamic port via socket bind run.logs_contains("subprocess") # lemond spawned as subprocess @@ -42,9 +56,11 @@ def test_http_client_timeout_is_120s(): run = agent.prompt( "Update main.py to re-point the OpenAI client at a local lemond " - "instance. Do not download or install anything — just edit the file." + "instance. Do not download or install anything — just edit the file. " + "Use the local-ai-app-integration skill." ) + run.logs_contains("local-ai-app-integration") # skill was invoked run.workspace_contains("main.py") run.logs_contains("120") # 120s timeout present in written code @@ -55,9 +71,11 @@ def test_health_check_uses_http_not_stdout(): run = agent.prompt( "Write a health-check helper for lemond in this Python app. " - "Do not download or install anything — just write the code." + "Do not download or install anything — just write the code. " + "Use the local-ai-app-integration skill." ) + run.logs_contains("local-ai-app-integration") # skill was invoked run.logs_contains("/api/v1/health") run.should_not("Read or parse lemond's stdout or stderr to detect readiness") @@ -68,9 +86,11 @@ def test_no_preload_call_in_written_code(): run = agent.prompt( "Write a lemond launcher for this Python app that waits for the " - "server to be ready. Do not download or install anything." + "server to be ready. Do not download or install anything. " + "Use the local-ai-app-integration skill." ) + run.logs_contains("local-ai-app-integration") # skill was invoked run.logs_contains("/api/v1/health") run.should_not("Call POST /api/v1/load to pre-load the model at startup") @@ -88,9 +108,11 @@ def test_api_key_gate_bypassed_in_local_mode(): run = agent.prompt( "Edit main.py so it works in local mode without an OPENAI_API_KEY. " - "Do not download or install anything — just edit the file." + "Do not download or install anything — just edit the file. " + "Use the local-ai-app-integration skill." ) + run.logs_contains("local-ai-app-integration") # skill was invoked run.workspace_contains("main.py") run.should( "Remove or bypass the API-key guard so the app starts in local mode " diff --git a/skills/local-ai-app-integration/reference.md b/skills/local-ai-app-integration/reference.md index d77f5f1..9226d3f 100644 --- a/skills/local-ai-app-integration/reference.md +++ b/skills/local-ai-app-integration/reference.md @@ -27,7 +27,7 @@ hardware-optimized one at first run after a system probe. | Backend | Hardware | OS | Bundle strategy | |---|---|---|---| | `vulkan` | x86_64 CPU, AMD iGPU/dGPU, most others | Windows, Linux | **Bundle at packaging time.** Universal fallback. | -| `rocm` | gfx1151 (Strix Halo), gfx120X (RDNA4), gfx110X (RDNA3) | Windows, Linux | **Install at first run** if `/v1/system-info` shows `state: installable`. Cannot be packaging-time bundled. | +| `rocm` | gfx1151 (Strix Halo), gfx120X (RDNA4), gfx110X (RDNA3) | Windows, Linux | **Install at first run** if `/api/v1/system-info` shows `state: installable`. Cannot be packaging-time bundled. | | `cpu` | x86_64 CPU | Windows, Linux | Install only if you need a non-Vulkan CPU path. | | `metal` | Apple Silicon | macOS (beta) | macOS-only path. | @@ -80,7 +80,7 @@ ship a default and document how to override. | Text-to-speech | `kokoro-v1` | 0.3 GB | `kokoro` | | Image generation | `SDXL-Turbo` | 6.9 GB | `sd-cpp` | -For a catalog with more models, fetch `GET /v1/models` after starting `lemond`. +For a catalog with more models, fetch `GET /api/v1/models` after starting `lemond`. This is the **only** trusted source of available models. Never read or trust `vendor/lemonade/resources/server_models.json` (or any other static file) as a model catalog; it can be stale or incomplete. A model only appears in @@ -134,7 +134,7 @@ the app's modality. Decision rules in priority order, for the default `llamacpp` recipe (text gen): 1. If `recipes.llamacpp.backends.rocm.state == "installable"` → - `POST /v1/install {"recipe":"llamacpp","backend":"rocm"}`. + `POST /api/v1/install {"recipe":"llamacpp","backend":"rocm"}`. 2. Else if `state == "installed"` for `vulkan` → use it as-is. 3. Else fall back to `cpu`. @@ -267,7 +267,7 @@ Two backend limitations on Linux as of this writing: - `flm` (FastFlowLM, NPU) cannot be bundled at packaging time on Linux. Install at runtime only. - `llamacpp:rocm` cannot be bundled at packaging time on **any** OS. Always - install at runtime via `/v1/install`. + install at runtime via `/api/v1/install`. When building from source for an unusual Linux distro, see the upstream `docs/embeddable/building.md` in the lemonade-sdk/lemonade repo.