A self-hostable, open-source laboratory for studying how LLMs behave when offered a set of fake tools. Design fake tools, wire in any OpenAI-compatible model, run one session or a thousand, and see exactly what the model picked and why.
docker compose up --buildOpen http://localhost:5173 for the UI. The frontend will proxy API calls to the backend container on port 8000.
cd backend
uv venv
uv pip install -e ".[dev]"
cp ../.env.example ../.env # fill in your API keys
uv run uvicorn app.main:app --reloadcd frontend
npm install
npm run devOpen http://localhost:5173 — the frontend proxies /api to :8000.
A brand-new instance shows a Getting Started checklist instead of an empty screen. Click Load samples to seed 9 built-in whimsical tools, point a model config at OpenAI/OpenRouter/LM Studio/whatever you have running, build a plan, and hit Run — no manual setup required to see the whole loop work end to end.
- Tool Library → create fake tools with static or dynamic responses, or load the 9 built-in samples
- Model Configs → configure a provider endpoint + model snapshot + API key env var. Pick the API style: Chat Completions (OpenAI / OpenRouter / LM Studio / Ollama) or Responses API (
/v1/responses, OpenAI and OpenRouter). The harness normalizes both to its internal representation, so everything downstream — the agent loop, logging, and analysis — is identical regardless of which you choose.
- Plans → compose tools + model + prompts into a versioned testing plan
- Run → launch a single session and watch the live event stream as reasoning, text, and tool calls arrive — or fire a batch run of N repetitions and let them run in the background
- Sessions & Batch Runs → view history, metrics, per-session event timelines, and per-batch progress
- Analysis → review plan-version stats, export CSV, and download a markdown findings report for write-ups or sharing
Each plan version has a stats view for aggregate outcomes across its sessions, including completed/errored/aborted counts, tool-selection frequency, and per-session breakdowns. From there you can export session metrics as CSV or open the generated findings report to review or download a markdown summary of the run.
Need more than one data point? Use Run in batch… on a plan to launch N repetitions (1–1000) of the same plan version. Batches run sequentially in the background (up to 5 concurrent sessions), can be aborted mid-flight, and show live per-session progress from the Batch Runs page — all feeding into the same plan-version stats as a single run.
- Export / Import — bundle your tools, model configs, and plans (optionally with run history) into a single ZIP, and import it elsewhere with automatic conflict detection and per-item renaming.
- Factory Reset — wipe all user-created data and start clean. This cannot be undone; export first if you want to keep a copy.
Dynamic tool code runs in-process without sandboxing. This is intentional for locally-authored tools. Never execute dynamic code from untrusted sources. See §10.6 of the PRD for the full rationale.
cd backend
uv run pytest -v --tb=shortcd frontend
npm testFull documentation — quickstart, workflow, and architecture — lives at llaboratory.ampyard.com (source in docs/).
Issues and PRs are welcome. If you build an interesting fake tool or find a surprising model behavior, open an issue — we'd love to hear about it.
