chore: dummy PR for quizz rate-limit repro (DO NOT MERGE) - #5
Open
jonasbrami wants to merge 1 commit into
Open
Conversation
Throwaway — drives a dummy PR that forces `quizz take` to generate a fresh quiz so the LLM call path can be observed. Not imported anywhere.
6 tasks
jonasbrami
added a commit
that referenced
this pull request
May 23, 2026
Smoke test on PR #5 revealed two issues: - max_turns=2 was insufficient — the agent needs ~3 turns minimum (ToolSearch lookup → actual tool call → confirmation text). Bumping to 8 leaves headroom for thinking blocks. - claude_agent_sdk raises a bare `Exception` (not ClaudeSDKError) for protocol-level errors like "Reached maximum number of turns". Wrap it as RuntimeError so take.py's existing catch clause handles it uniformly with the other error types.
jonasbrami
added a commit
that referenced
this pull request
May 23, 2026
…opus (#7) * docs(spec): design for claude_agent_sdk engine adapter Adds a spec for routing quizz's inference through claude_agent_sdk (and the official `claude` binary) when ANTHROPIC_API_KEY is not set, so OAuth-only users on the Claude Code Max plan can use sonnet/opus instead of being gated to haiku by the api.anthropic.com OAuth-third-party-client policy. * docs(plan): implementation plan for claude_agent_sdk engine * build: add claude-agent-sdk runtime dep * feat(engine): scaffold ClaudeAgentLLM adapter skeleton * feat(engine): implement _invoke_tool via in-process MCP tool * feat(engine): ClaudeAgentLLM.generate_quiz_outline * feat(engine): ClaudeAgentLLM.generate_mermaid_set * feat(engine): ClaudeAgentLLM.grade_open * feat(cli): select ClaudeAgentLLM when ANTHROPIC_API_KEY absent * feat(cli): surface ClaudeAgentLLM RuntimeError as exit 1 + message * chore: ruff format + drop unused AsyncIterator import * fix(engine): bump max_turns to 8, catch bare Exception from SDK Smoke test on PR #5 revealed two issues: - max_turns=2 was insufficient — the agent needs ~3 turns minimum (ToolSearch lookup → actual tool call → confirmation text). Bumping to 8 leaves headroom for thinking blocks. - claude_agent_sdk raises a bare `Exception` (not ClaudeSDKError) for protocol-level errors like "Reached maximum number of turns". Wrap it as RuntimeError so take.py's existing catch clause handles it uniformly with the other error types. * fix(server): wrap grade() in asyncio.to_thread to unblock OpenQuestion submits ClaudeAgentLLM._invoke_tool calls asyncio.run() inside _drain_agent. Called from cli/take.py (sync) that works fine, but /submit is an `async def` route running on uvicorn's event loop — Python forbids asyncio.run() from within a running loop, so every quiz with an OpenQuestion would 500 when the user submitted answers. Wrapping grade() in asyncio.to_thread moves the sync work off the event-loop thread and incidentally also keeps AnthropicLLM's blocking HTTP calls from pinning uvicorn. Includes a regression test that posts to /submit with a ClaudeAgentLLM whose _drain_agent uses the real asyncio.run pattern — failed before this commit, passes after.
jonasbrami
added a commit
that referenced
this pull request
May 29, 2026
Adds an optional `anchor: {path, start_line, end_line}` to every question type
so the browser can show the exact diff hunk inline next to the question — the
reader no longer has to hold the diff in their head or switch to GitHub.
- models: new `Anchor` model + optional `anchor` field on all four question
types. Optional (defaults to None) → old cached quizzes load unchanged; the
range is validated (start ≤ end, lines ≥ 1) but a path need not be in the diff
(a question may anchor surrounding context).
- web: `GET /diff?path=` serves one file's unified-diff section; `_DiffProvider`
is hoisted into `main()` and shared by the MCP tools and the web app (one
fetch/cache). 503 when no diff source is wired.
- frontend: collapsible per-question code panel, fetched on first expand and
rendered DOM-built (textContent only, never innerHTML); +/- lines and `@@`
headers are styled via CSS classes. Present on both the answering and the
results-review cards; absent when a question has no anchor.
- generation prompt: instructs the author to emit a tight `anchor` whenever a
question probes specific lines; never affects grading.
Foundation for the diff coverage map (#9) and drill-this (#5).
Tested: models round-trip + backward-compat (no anchor) unit tests; /diff
endpoint tests (hit + miss + 503); full mcp/engine suite green (ruff, mypy
strict). Anchor UI verified in a real browser (panel renders, expands, loads
the hunk, color-codes lines; absent on anchorless questions).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
jonasbrami
added a commit
that referenced
this pull request
May 29, 2026
Adds an optional `anchor: {path, start_line, end_line}` to every question type
so the browser can show the exact diff hunk inline next to the question — the
reader no longer has to hold the diff in their head or switch to GitHub.
- models: new `Anchor` model + optional `anchor` field on all four question
types. Optional (defaults to None) → old cached quizzes load unchanged; the
range is validated (start ≤ end, lines ≥ 1) but a path need not be in the diff
(a question may anchor surrounding context).
- web: `GET /diff?path=` serves one file's unified-diff section; `_DiffProvider`
is hoisted into `main()` and shared by the MCP tools and the web app (one
fetch/cache). 503 when no diff source is wired.
- frontend: collapsible per-question code panel, fetched on first expand and
rendered DOM-built (textContent only, never innerHTML); +/- lines and `@@`
headers are styled via CSS classes. Present on both the answering and the
results-review cards; absent when a question has no anchor.
- generation prompt: instructs the author to emit a tight `anchor` whenever a
question probes specific lines; never affects grading.
Foundation for the diff coverage map (#9) and drill-this (#5).
Tested: models round-trip + backward-compat (no anchor) unit tests; /diff
endpoint tests (hit + miss + 503); full mcp/engine suite green (ruff, mypy
strict). Anchor UI verified in a real browser (panel renders, expands, loads
the hunk, color-codes lines; absent on anchorless questions).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
jonasbrami
added a commit
that referenced
this pull request
May 29, 2026
* feat(quiz): inline code context per question (anchors)
Adds an optional `anchor: {path, start_line, end_line}` to every question type
so the browser can show the exact diff hunk inline next to the question — the
reader no longer has to hold the diff in their head or switch to GitHub.
- models: new `Anchor` model + optional `anchor` field on all four question
types. Optional (defaults to None) → old cached quizzes load unchanged; the
range is validated (start ≤ end, lines ≥ 1) but a path need not be in the diff
(a question may anchor surrounding context).
- web: `GET /diff?path=` serves one file's unified-diff section; `_DiffProvider`
is hoisted into `main()` and shared by the MCP tools and the web app (one
fetch/cache). 503 when no diff source is wired.
- frontend: collapsible per-question code panel, fetched on first expand and
rendered DOM-built (textContent only, never innerHTML); +/- lines and `@@`
headers are styled via CSS classes. Present on both the answering and the
results-review cards; absent when a question has no anchor.
- generation prompt: instructs the author to emit a tight `anchor` whenever a
question probes specific lines; never affects grading.
Foundation for the diff coverage map (#9) and drill-this (#5).
Tested: models round-trip + backward-compat (no anchor) unit tests; /diff
endpoint tests (hit + miss + 503); full mcp/engine suite green (ruff, mypy
strict). Anchor UI verified in a real browser (panel renders, expands, loads
the hunk, color-codes lines; absent on anchorless questions).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* address review: scope inline hunk to the anchor + handle not-found
Review feedback on the anchor UI:
- The panel showed the whole file's diff; the anchor's line range was only a
cosmetic label. Now the diff is parsed into hunks and scoped to the hunk(s)
overlapping the anchor's new-side range (falls back to all hunks if none
overlap), the anchored new-side lines are highlighted, and the file-level
header lines (diff --git/index/---/+++) are dropped as noise.
- A `/diff` miss (renamed file, or a binary/minified file filtered out of the
diff) returned the "No changed file matches…" sentence with 200, which the UI
rendered as a fake one-line diff. Now the UI shows a clean "Not part of the
PR diff." note instead.
- loadHunk now logs the error on a fetch failure (was silently swallowed).
- Added a guard test that a question's anchor survives set_quiz + snapshot.
Verified in a real browser: a question anchored to the 2nd hunk shows only that
hunk with its added lines highlighted; a question anchored to a non-diff path
shows the note.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Throwaway PR used to reproduce rate-limiting behavior in
quizz takeagainstthe sonnet/opus models. Will be closed once the investigation is done — do not
merge.
Adds a single dummy script under
scripts/. The code is intentionally simple andself-contained so that the quiz generator has a coherent surface to ask about.