docs: refresh demo GIF with a real cognit-generated quiz (high-res)

README.md

@@ -9,7 +9,7 @@
 
 > Voluntary, opt-in PR-author comprehension quizzes. Surface the gap between what you think your code does and what it actually does — before you merge.
 
-![Demo: `cognit take` generates a quiz from a PR diff while Claude reads the code, then the author answers four questions — multiple choice, a mermaid-pick diagram, an open question, and true/false — and gets graded scores in the browser.](docs/img/cognit-demo.gif)
+![Demo: `cognit take` generates a quiz from a real PR diff while Claude reads the code, then the author answers five questions — three multiple-choice, a mermaid-pick flow diagram, and true/false — and gets graded scores in the browser.](docs/img/cognit-demo.gif)
 
 **cognit flips the usual AI coding loop.** Normally you write a prompt and Claude writes the code. cognit has Claude *read* the code and write prompts back — questions only you, the author, can answer about the diff you're about to merge. Same model, arrows reversed, and the loop closes on the one question that matters: *does this code do what you intended?* Each question you wrestle with — especially the ones you get wrong — is comprehension credit banked against [comprehension debt](#why-this-exists). Call it **comprehension-driven development (CDD)**.
 

scripts/capture_demo_quiz.py

@@ -0,0 +1,65 @@
+"""Capture a REAL cognit-generated quiz for the README demo.
+
+Runs the actual generation engine (`ClaudeAgentLLM` → the `claude` binary) against
+a live PR, recording both the generated quiz and the activity feed Claude emits
+while it reads the diff. The demo recorder (`record_demo.py`) replays these so the
+GIF shows genuine, model-authored questions — not a hand-written stand-in — while
+staying offline and deterministic on re-record.
+
+Usage:
+    uv run python scripts/capture_demo_quiz.py <pr-url> [--model claude-sonnet-4-6]
+
+Writes:
+    scripts/demo_data/quiz.json   — the generated Quiz (post label-neutralization)
+    scripts/demo_data/feed.json   — the activity events (thinking/text/tool_use)
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+from typing import Any
+
+from cognit.engine.generate import generate_quiz
+from cognit.engine.llm_claude_agent import ClaudeAgentLLM
+from cognit.ghio.pr import fetch_pr_info
+
+OUT_DIR = Path(__file__).resolve().parent / "demo_data"
+
+
+def main() -> None:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("pr_url")
+    ap.add_argument("--model", default="claude-sonnet-4-6")
+    args = ap.parse_args()
+
+    info = fetch_pr_info(args.pr_url)
+    print(f"PR #{info.number}: {info.title!r} (branch {info.branch})")
+
+    events: list[dict[str, Any]] = []
+    llm = ClaudeAgentLLM(model=args.model)
+    llm.on_event = events.append
+
+    print(f"generating with {args.model} (this can take a few minutes)…")
+    quiz = generate_quiz(
+        pr_title=info.title,
+        pr_body=info.body,
+        pr_number=info.number,
+        pr_url=args.pr_url,
+        branch=info.branch,
+        llm=llm,
+        model=args.model,
+    )
+
+    OUT_DIR.mkdir(parents=True, exist_ok=True)
+    (OUT_DIR / "quiz.json").write_text(quiz.model_dump_json(indent=2))
+    (OUT_DIR / "feed.json").write_text(json.dumps(events, indent=2))
+
+    kinds = {q.type: sum(1 for x in quiz.questions if x.type == q.type) for q in quiz.questions}
+    print(f"wrote {OUT_DIR}/quiz.json — {len(quiz.questions)} questions {kinds}")
+    print(f"wrote {OUT_DIR}/feed.json — {len(events)} activity events")
+
+
+if __name__ == "__main__":
+    main()

scripts/demo_data/quiz.json

@@ -0,0 +1,62 @@
+{
+  "version": "1",
+  "pr_number": 20,
+  "questions": [
+    {
+      "type": "mcq",
+      "id": "q1",
+      "prompt": "Given `_is_retryable(RuntimeError(\"agent did not call submit_quiz\"))`, what does the function return?",
+      "options": [
+        "True — it is a RuntimeError, not a ValidationError, and its message contains no fatal marker",
+        "False — no retryable marker appears in the message, so the final `return any(...)` evaluates to False",
+        "True — only ValidationError is excluded; all other exception types are unconditionally retried"
+      ],
+      "answer": "False — no retryable marker appears in the message, so the final `return any(...)` evaluates to False",
+      "explanation": "`_is_retryable` first short-circuits on `ValidationError`, then on any fatal marker — neither applies here. But it only returns `True` if a *retryable* marker is positively matched. The message 'agent did not call submit_quiz' matches nothing in `_RETRYABLE_MARKERS`, so `any(...)` is `False` and the function returns `False`. The misconception is that the absence of a fatal marker is enough to trigger a retry; it isn't — a retryable marker must match."
+    },
+    {
+      "type": "mcq",
+      "id": "q2",
+      "prompt": "What is the maximum value `_backoff_delay(attempt=0, rng)` can return?",
+      "options": [
+        "1.0 s — `2**0 = 1`, making the cap 1 second before the `min`",
+        "2.0 s — `min(30.0, 2.0 × 2**0)` = `min(30.0, 2.0)` = 2.0",
+        "30.0 s — every call saturates at `_MAX_DELAY_S` regardless of attempt"
+      ],
+      "answer": "2.0 s — `min(30.0, 2.0 × 2**0)` = `min(30.0, 2.0)` = 2.0",
+      "explanation": "The formula is `min(_MAX_DELAY_S, _BASE_DELAY_S * (2**attempt))`. At attempt 0: `2.0 × 2**0 = 2.0 × 1 = 2.0`, and `min(30.0, 2.0) = 2.0`. The common slip is dropping the `_BASE_DELAY_S` multiplier (giving 1.0), or assuming the cap always saturates at 30 s (which only happens from attempt 4 onward: `2.0 × 16 = 32 > 30`)."
+    },
+    {
+      "type": "mcq",
+      "id": "q3",
+      "prompt": "When all 3 attempts each fail with `RuntimeError('429 rate limit')`, how many times is the `sleep` callable invoked?",
+      "options": [
+        "3 — once after each of the three failures",
+        "2 — on the final attempt the `raise` inside the except block fires before `sleep` is reached",
+        "0 — the last transient failure short-circuits all preceding sleeps"
+      ],
+      "answer": "2 — on the final attempt the `raise` inside the except block fires before `sleep` is reached",
+      "explanation": "On attempt 2 (0-based), `attempt + 1 >= _MAX_ATTEMPTS` is `3 >= 3` — True — so `raise` executes immediately in the except block, before the `sleep` call. Attempts 0 and 1 do sleep (they pass the condition check), giving exactly 2 sleeps for 3 total failures. The test `assert len(slept) == _MAX_ATTEMPTS - 1` directly verifies this."
+    },
+    {
+      "type": "mermaid",
+      "id": "q4",
+      "prompt": "Which diagram correctly models what `_draft_with_retry` does when `llm.draft_quiz` raises an exception on a given attempt?",
+      "options": {
+        "A": "flowchart LR\n  A[exception caught] --> B{final or non-retryable?}\n  B -- yes --> C[re-raise]\n  B -- no --> D[sleep backoff]\n  D --> E[notify sink]\n  E --> F[next attempt]",
+        "B": "flowchart LR\n  A[exception caught] --> B{final or non-retryable?}\n  B -- yes --> C[re-raise]\n  B -- no --> D[notify sink]\n  D --> E[sleep backoff]\n  E --> F[next attempt]",
+        "C": "flowchart LR\n  A[exception caught] --> B{final attempt?}\n  B -- yes --> C[re-raise]\n  B -- no --> D[notify sink]\n  D --> E[sleep backoff]\n  E --> F[next attempt]",
+        "D": "flowchart LR\n  A[exception caught] --> B{not retryable?}\n  B -- yes --> C[re-raise]\n  B -- no --> D[notify sink]\n  D --> E[sleep backoff]\n  E --> F[next attempt]"
+      },
+      "answer": "B",
+      "explanation": ""
+    },
+    {
+      "type": "tf",
+      "id": "q5",
+      "prompt": "When all 3 attempts fail with transient errors, the exception that ultimately propagates to the caller is the one raised by the **first** failed attempt.",
+      "answer": false,
+      "explanation": "On the final attempt (attempt=2), `raise` re-raises the *current* `exc` from that iteration — not the stored `last_exc` from earlier iterations. The test confirms this with `match=f'#{_MAX_ATTEMPTS - 1}'`, which matches '#2' (the third error). The PR doc says 're-raises the original exception unchanged' — 'original' means the raw SDK exception, not wrapped in a new one, not necessarily the chronologically first one."
+    }
+  ]
+}