feat(engine): claude_agent_sdk adapter so OAuth users can use sonnet/opus

[jonasbrami]

Summary

• Root cause (see docs/superpowers/specs/2026-05-22-claude-agent-sdk-engine-design.md): direct calls to api.anthropic.com/v1/messages using a Claude Code OAuth accessToken are gated by Anthropic to Haiku only. Sonnet/Opus instantly return a sparse 429 rate_limit_error with body {"message": "Error"} and no anthropic-ratelimit-* headers — it's a policy gate, not a usage limit. Header-spoofing (x-app: cli, User-Agent: claude-cli/…, anthropic-client-platform: claude_code_cli, X-Claude-Code-Session-Id) does not unlock it. The official claude binary works because it performs a session-binding handshake we can't replicate from outside.
• Fix: add a second LLMClient adapter, ClaudeAgentLLM, that routes inference through claude_agent_sdk (which subprocesses the claude binary). Each method registers a one-shot in-process MCP tool to capture structured output. cli/take._make_llm branches on ANTHROPIC_API_KEY: present → direct AnthropicLLM (unchanged), absent → ClaudeAgentLLM (new). The existing LLMClient Protocol, FakeLLM, generate.py, grade.py, server/app.py, and all existing tests are untouched.
• Verified end-to-end against PR chore: dummy PR for quizz rate-limit repro (DO NOT MERGE) #5: Sonnet outline call 56s, Opus outline call 33s, both returning well-formed QuizOutline objects. The same calls previously hit the 429 gate in <1s.

What changed

File	Change
src/quizz/engine/llm_claude_agent.py	NEW adapter
src/quizz/cli/take.py	_make_llm branches on ANTHROPIC_API_KEY; _generate_and_post except chain gains a RuntimeError clause for the agent-SDK error path
pyproject.toml	adds claude-agent-sdk>=0.1.44
tests/engine/test_llm_claude_agent.py	NEW, 11 tests
tests/cli/test_take_select.py	NEW, 2 tests for the adapter-selection branch
tests/cli/test_take.py	1 test added for the RuntimeError exit-1 path
docs/superpowers/specs/…design.md	spec
docs/superpowers/plans/…engine.md	plan

Notable design choices

• Protocol stays sync. The new adapter exposes the same three sync methods as AnthropicLLM and hides asyncio.run internally. No ripple through generate.py, server, or CLI tests.
• Structured output via MCP @tool. Each call builds a fresh in-process MCP server with one tool matching the existing schema (submit_quiz_outline, submit_mermaid_set, submit_grade). The handler stuffs the args into a closure-shared list; the adapter returns them as a Pydantic model. The MCP tool is allowlisted with allowed_tools=["mcp__quizz__<name>"] so the agent can't call anything else.
• max_turns=8. Empirically the agent needs ~3 turns (a ToolSearch discovery call → the actual tool call → a confirmation text block). 8 gives headroom for thinking blocks without letting a stuck agent run away.
• Error mapping. CLINotFoundError → RuntimeError("claude binary not found; install Claude Code or set ANTHROPIC_API_KEY"). CLIConnectionError/ProcessError/ClaudeSDKError → RuntimeError(f"claude agent SDK call failed: {e}"). The SDK also raises bare Exception for protocol errors like "max turns reached", so I wrap that too. take.py:_generate_and_post gets a single RuntimeError clause that exits 1 with the message.
• _drain_agent test seam. Production's _drain_agent(prompt, options, handler) ignores the handler param — the SDK fires it via MCP. Tests override _drain_agent and call handler directly to inject canned args without spawning a subprocess. Slight smell, smaller than the alternatives.

Out of scope (deliberate)

• README update to clarify the auth/model matrix. Easy follow-up, but I didn't want to mix doc churn into this PR.
• Persistent ClaudeSDKClient across a quiz (one subprocess instead of N). The per-call ~1–2s overhead is acceptable next to the ~5–15s LLM latency per call.
• Async refactor of the engine. Considered, rejected: too much ripple for a small UX win.

Test plan

• uv run pytest — 87 passed, 2 skipped
• uv run mypy src — clean (strict)
• uv run ruff check src tests + ruff format --check — clean
• Direct smoke: generate_quiz_outline against a 5-line synthetic diff with claude-sonnet-4-6 — returned 2 questions in 56s
• Same with claude-opus-4-7 — returned 3 questions (incl. an OpenQuestion) in 33s
• Full quizz take end-to-end against a real PR (deferred — the full flow with 3-4 mermaid artisans serially can exceed 5min, but each per-call code path is exercised by the smoke tests above)

🤖 Generated with Claude Code

[jonasbrami]

Quiz on your PR

Take it: quizz take or scroll down.

Question 1 — mcq

In llm_claude_agent.py, _drain_agent immediately executes del handler. If production never uses that parameter, why does _drain_agent accept a handler argument at all?

• So tests can override _drain_agent with a fake that calls handler(args) directly, injecting canned tool args without spawning a subprocess.
• So the production code can invoke handler if the agent's tool-call result arrives out-of-band, after the async stream has already closed.
• So _invoke_tool can pass the handler into ClaudeAgentOptions for the SDK to invoke during its internal permission checks.
• It is a forward-compatibility hook for a planned retry path that will call handler a second time with a tighter prompt.

Question 2 — tf

If the agent stream completes without calling the MCP tool, _invoke_tool retries the request once with a tighter prompt before giving up and returning None.

• true / false

Question 3 — mcq

_invoke_tool catches CLINotFoundError, CLIConnectionError, ProcessError, and ClaudeSDKError by name, then has a bare except Exception catch-all. What real failure does that last clause catch that the earlier ones miss?

• The SDK raises a bare Exception (not a SDK subclass) for protocol-level errors such as 'max turns reached'.
• It catches asyncio.CancelledError when the event loop is shut down before the async stream finishes.
• It catches pydantic.ValidationError raised by ClaudeAgentOptions if the model name is unrecognised.
• It catches KeyboardInterrupt forwarded from the claude subprocess when the user hits Ctrl-C.

Question 4 — tf

In take.py, setting the environment variable ANTHROPIC_API_KEY to an empty string ("") causes _make_llm to return an AnthropicLLM instance.

• true / false

Question 5 — open

The adapter sets allowed_tools=["mcp__quizz__<tool_name>"] with exactly one entry. Explain what problem this solves and why it works as a forcing mechanism for tool use.

Question 6 — mermaid

Which sequence diagram correctly shows how _invoke_tool orchestrates a production tool call, from the initial invocation through to capturing the structured tool arguments?

Option A

sequenceDiagram
    participant Caller
    participant IT as _invoke_tool
    participant DA as _drain_agent
    participant Q as query()
    participant MH as MCP handler
    Caller->>IT: _invoke_tool()
    IT->>IT: build handler, create options
    IT->>DA: _drain_agent(prompt, options, handler)
    DA->>DA: delete handler
    DA->>Q: asyncio.run(_drain())
    Q-->>DA: drain complete
    DA->>MH: handler(args)
    MH-->>DA: ok response
    DA-->>IT: return
    IT-->>Caller: captured[0]

Loading

Option B

sequenceDiagram
    participant Caller
    participant IT as _invoke_tool
    participant DA as _drain_agent
    participant Q as query()
    participant MH as MCP handler
    Caller->>IT: _invoke_tool()
    IT->>IT: build handler, create options
    IT->>DA: _drain_agent(prompt, options, handler)
    DA->>DA: delete handler
    DA->>Q: asyncio.run(_drain())
    Q-->>DA: drain complete
    DA-->>IT: return
    IT->>MH: handler(args)
    MH-->>IT: ok response
    IT-->>Caller: captured[0]

Loading

Option C

sequenceDiagram
    participant Caller
    participant IT as _invoke_tool
    participant DA as _drain_agent
    participant Q as query()
    Caller->>IT: _invoke_tool()
    IT->>IT: build handler, create options
    IT->>DA: _drain_agent(prompt, options, handler)
    DA->>DA: delete handler
    DA->>Q: asyncio.run(_drain())
    Q->>Q: process tool call
    Q-->>DA: tool args
    DA->>DA: append to captured
    DA-->>IT: return
    IT-->>Caller: captured[0]

Loading

Option D

sequenceDiagram
    participant Caller
    participant IT as _invoke_tool
    participant DA as _drain_agent
    participant Q as query()
    participant MH as MCP handler
    Caller->>IT: _invoke_tool()
    IT->>IT: build handler, create options
    IT->>DA: _drain_agent(prompt, options, handler)
    DA->>DA: delete handler
    DA->>Q: asyncio.run(_drain())
    Q->>MH: handler(args)
    MH-->>Q: ok response
    Q-->>DA: drain complete
    DA-->>IT: return
    IT-->>Caller: captured[0]

Loading

---

Quiz state (used by the CLI)

{
  "version": "1",
  "pr_number": 7,
  "questions": [
    {
      "type": "mcq",
      "id": "q1",
      "prompt": "In `llm_claude_agent.py`, `_drain_agent` immediately executes `del handler`. If production never uses that parameter, why does `_drain_agent` accept a `handler` argument at all?",
      "options": [
        "So tests can override `_drain_agent` with a fake that calls `handler(args)` directly, injecting canned tool args without spawning a subprocess.",
        "So the production code can invoke `handler` if the agent's tool-call result arrives out-of-band, after the async stream has already closed.",
        "So `_invoke_tool` can pass the handler into `ClaudeAgentOptions` for the SDK to invoke during its internal permission checks.",
        "It is a forward-compatibility hook for a planned retry path that will call `handler` a second time with a tighter prompt."
      ],
      "answer": "So tests can override `_drain_agent` with a fake that calls `handler(args)` directly, injecting canned tool args without spawning a subprocess."
    },
    {
      "type": "tf",
      "id": "q2",
      "prompt": "If the agent stream completes without calling the MCP tool, `_invoke_tool` retries the request once with a tighter prompt before giving up and returning `None`.",
      "answer": false
    },
    {
      "type": "mcq",
      "id": "q3",
      "prompt": "`_invoke_tool` catches `CLINotFoundError`, `CLIConnectionError`, `ProcessError`, and `ClaudeSDKError` by name, then has a bare `except Exception` catch-all. What real failure does that last clause catch that the earlier ones miss?",
      "options": [
        "The SDK raises a bare `Exception` (not a SDK subclass) for protocol-level errors such as 'max turns reached'.",
        "It catches `asyncio.CancelledError` when the event loop is shut down before the async stream finishes.",
        "It catches `pydantic.ValidationError` raised by `ClaudeAgentOptions` if the model name is unrecognised.",
        "It catches `KeyboardInterrupt` forwarded from the claude subprocess when the user hits Ctrl-C."
      ],
      "answer": "The SDK raises a bare `Exception` (not a SDK subclass) for protocol-level errors such as 'max turns reached'."
    },
    {
      "type": "tf",
      "id": "q4",
      "prompt": "In `take.py`, setting the environment variable `ANTHROPIC_API_KEY` to an empty string (`\"\"`) causes `_make_llm` to return an `AnthropicLLM` instance.",
      "answer": false
    },
    {
      "type": "open",
      "id": "q5",
      "prompt": "The adapter sets `allowed_tools=[\"mcp__quizz__<tool_name>\"]` with exactly one entry. Explain what problem this solves and why it works as a forcing mechanism for tool use.",
      "rubric": "A complete answer must make all three of these claims: (1) `claude_agent_sdk` exposes no `tool_choice='required'` parameter, so there is no direct way to mandate that the agent must call a specific tool; (2) restricting `allowed_tools` to a single entry means the model's only permitted action besides finishing is to call that one tool, making tool invocation the only reachable path and preventing plain-text responses; (3) this matters because `_invoke_tool` returns only the captured tool-call args — a plain-text response leaves `captured` empty, causing `_invoke_tool` to return `None`, and callers immediately raise `RuntimeError`."
    },
    {
      "type": "mermaid",
      "id": "q6",
      "prompt": "Which sequence diagram correctly shows how `_invoke_tool` orchestrates a production tool call, from the initial invocation through to capturing the structured tool arguments?",
      "options": {
        "A": "sequenceDiagram\n    participant Caller\n    participant IT as _invoke_tool\n    participant DA as _drain_agent\n    participant Q as query()\n    participant MH as MCP handler\n    Caller->>IT: _invoke_tool()\n    IT->>IT: build handler, create options\n    IT->>DA: _drain_agent(prompt, options, handler)\n    DA->>DA: delete handler\n    DA->>Q: asyncio.run(_drain())\n    Q-->>DA: drain complete\n    DA->>MH: handler(args)\n    MH-->>DA: ok response\n    DA-->>IT: return\n    IT-->>Caller: captured[0]",
        "B": "sequenceDiagram\n    participant Caller\n    participant IT as _invoke_tool\n    participant DA as _drain_agent\n    participant Q as query()\n    participant MH as MCP handler\n    Caller->>IT: _invoke_tool()\n    IT->>IT: build handler, create options\n    IT->>DA: _drain_agent(prompt, options, handler)\n    DA->>DA: delete handler\n    DA->>Q: asyncio.run(_drain())\n    Q-->>DA: drain complete\n    DA-->>IT: return\n    IT->>MH: handler(args)\n    MH-->>IT: ok response\n    IT-->>Caller: captured[0]",
        "C": "sequenceDiagram\n    participant Caller\n    participant IT as _invoke_tool\n    participant DA as _drain_agent\n    participant Q as query()\n    Caller->>IT: _invoke_tool()\n    IT->>IT: build handler, create options\n    IT->>DA: _drain_agent(prompt, options, handler)\n    DA->>DA: delete handler\n    DA->>Q: asyncio.run(_drain())\n    Q->>Q: process tool call\n    Q-->>DA: tool args\n    DA->>DA: append to captured\n    DA-->>IT: return\n    IT-->>Caller: captured[0]",
        "D": "sequenceDiagram\n    participant Caller\n    participant IT as _invoke_tool\n    participant DA as _drain_agent\n    participant Q as query()\n    participant MH as MCP handler\n    Caller->>IT: _invoke_tool()\n    IT->>IT: build handler, create options\n    IT->>DA: _drain_agent(prompt, options, handler)\n    DA->>DA: delete handler\n    DA->>Q: asyncio.run(_drain())\n    Q->>MH: handler(args)\n    MH-->>Q: ok response\n    Q-->>DA: drain complete\n    DA-->>IT: return\n    IT-->>Caller: captured[0]"
      },
      "answer": "D"
    }
  ]
}

[jonasbrami]

Quiz results

Total: 50%

• ✅ q1 — 100%
• ✅ q2 — 100%
• ❌ q3 — 0%
• ✅ q4 — 100%
• ❌ q5 — 0%

  The answer "test" is entirely non-responsive — it addresses none of the three required rubric items: the absence of a tool_choice='required' parameter in the SDK, the forcing mechanism created by restricting allowed_tools to a single entry, or the None/RuntimeError consequence of a plain-text response leaving captured empty.

• ❌ q6 — 0%

---

Results state (used by the CLI)

{
  "version": "1",
  "pr_number": 7,
  "total_score": 50,
  "per_question": [
    {
      "question_id": "q1",
      "correct": true,
      "score": 100,
      "feedback": ""
    },
    {
      "question_id": "q2",
      "correct": true,
      "score": 100,
      "feedback": ""
    },
    {
      "question_id": "q3",
      "correct": false,
      "score": 0,
      "feedback": ""
    },
    {
      "question_id": "q4",
      "correct": true,
      "score": 100,
      "feedback": ""
    },
    {
      "question_id": "q5",
      "correct": false,
      "score": 0,
      "feedback": "The answer \"test\" is entirely non-responsive — it addresses none of the three required rubric items: the absence of a `tool_choice='required'` parameter in the SDK, the forcing mechanism created by restricting `allowed_tools` to a single entry, or the `None`/`RuntimeError` consequence of a plain-text response leaving `captured` empty."
    },
    {
      "question_id": "q6",
      "correct": false,
      "score": 0,
      "feedback": ""
    }
  ]
}