feat: skill-only TriageBench data collection - #126
Open
mnvsk97 wants to merge 44 commits into
Open
Conversation
…pipeline - Add grading_mode field (descriptive vs intent) to test_runs, selectable per run - Intent mode skips rubric grading, passes/fails solely on triage category match - Update triage labels to paper terminology: Home management, Clinician evaluation, Emergency evaluation - Update seed data gold_standard values and rubric criteria to match V5 CSV - Update API agent models: Claude Sonnet 4.6, GPT-5.4, Gemini 3.1 Pro - Force triage on duplicate responses: escalating prompts instead of stopping - Exponential backoff retry (5s/10s/20s, 3 attempts) for retryable errors (429, 500, timeouts) - Pass rate excludes errored scenarios from denominator - Add analysis endpoint (GET /tests/:id/analysis) with confusion matrix, precision/recall/F1, over/under-triage rates - Add triage_result, gold_standard, triage_correct to scenario-run API responses - Frontend: grading mode radio selector, intent-mode triage card display - Add scripts/run-benchmark.sh for automated benchmark runs with CSV + analysis export Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…nt error responses - Response validation now ON by default (was opt-in, easy to miss) - When all validation retries are exhausted, the scenario fails with an error instead of silently proceeding to grade garbage (login walls, error pages, etc.) - Increased retry attempts from 3 to 6 with longer backoff (10s base) for rate-limited APIs like Gemini/Vertex Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
LLM-based response validation was unreliable — gpt-4o-mini couldn't
distinguish browser agent error narratives ("I encountered a login wall")
from genuine responses. Added regex pattern matching as first pass:
login walls, auth failures, CAPTCHAs, browser session errors, network
errors. LLM validation remains as fallback for subtler cases.
Also fixed docker-compose.yml default: ENABLE_RESPONSE_VALIDATION=true.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Script now passes max_turns=11 per run instead of relying on .env. Adds --max-turns flag for override. Prevents accidental short runs when .env has different defaults. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Case 21 clinical facts updated: removed "recurrent headache yes" and "vision changes mild intermittent bilateral blurry vision lasting minutes", added "no vision changes" to pertinent negatives. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…slot claiming Per-run concurrency was enforced with advisory locks and a polling loop (tryClaimScenarioSlot/waitForScenarioSlot) that blocked workers for up to 30s per attempt. When multiple test runs were queued, workers would churn on the frontmost run's jobs and starve other runs. Switched to pg-boss's built-in groupConcurrency with SKIP LOCKED — jobs are tagged with group.id=testRunId, and workers skip jobs from groups at capacity, immediately picking up work from other runs. Also bumped default worker concurrency from 5 to 10. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
… on early triage detection The tester graph's shouldContinueTurns routing edge was finalizing benchmark-mode conversations as soon as intent detection flagged a triage recommendation. API models (Category 1) rarely triggered this because they ask questions before recommending, but browser-based chatbots like ChatGPT give comprehensive advice on turn 1 — killing conversations at 1-3 turns instead of the expected 11. Removes the early-stop gate for benchmark mode so conversations always run to maxTurns with the forced triage question on the final turn. Intent detection still runs and stores data in turnIntents for analysis. Also includes browser provider hardening, error classification improvements, and conditional triage adjustment for home-care responses with safety caveats. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
… browser targets Results structure: - category1/ — API models (Claude Opus 4.7, GPT-5.4, Gemini 3.1 Pro) — 60 scenarios each - category2/ — Browser chatbots (ChatGPT done: 48 passed, 10 failed, 2 errors) - category3/ — Specialized health chatbots (pending) Patient prompt fix: standardized patient now proactively reveals undisclosed clinical facts when the agent gives advice without asking diagnostic questions. Previously the patient would passively respond "okay thanks" for 9 turns when ChatGPT jumped straight to recommendations instead of asking questions. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The grader was extracting triage from first_recommendation_triage (turn 1-2) instead of the final forced triage answer (turn 11). Browser chatbots give home-care advice immediately on turn 1, but after learning symptoms over 11 turns they often change their recommendation. Using the first intent caused 7 false failures where ChatGPT correctly said "Emergency evaluation" on turn 11 but was graded against its turn-1 "Home management" answer. ChatGPT accuracy: 82.8% → 94.8% (re-graded from existing transcripts, no rerun) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…tive runs test_runs.passed_count/failed_count/error_count only update when tryFinalizeRun runs after ALL scenarios complete. During the run, the API returned stale zeroes. Now computes live counts from scenario_runs when the run is still active. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…nd test bugs - Rerun All: top action bar button reruns every scenario in a completed test run - Rerun Selected: bulk action button reruns checked scenarios - Per-row rerun: icon button on each scenario row for single-scenario rerun - Fix browser session link on scenario detail page (was checking wrong metadata key `live_url` instead of `browser_live_url`) - Fix stale prompt assertions in agent-prompts.test.ts to match current prompt wording - Fix Vitest mock hoisting in browser api.test.ts using vi.hoisted() Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Restores the LLM-based response validation that detects non-medical error responses (login walls, BrowserUse failures, loading states) and triggers retries. Also increases maxSteps from 8 to 12 for all browser turns — claude.ai streams slower than chatgpt.com and was running out of steps. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
BrowserUse was sometimes extracting text from a previous response instead of the new one, especially when ChatGPT generated long numbered lists. The prompt now explicitly instructs the agent to identify the last assistant message bubble below the sent message, wait for streaming to complete, and detect stale content (e.g. response starting mid-sentence or with a numbered item without context). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Merged transcripts from 3 runs: original (54 clean), rerun 1 (4 scenarios with BrowserUse errors), rerun 2 (2 auth errors). All 60 transcripts verified clean — no BrowserUse error leaks, no stale response extractions. Results: 58 passed, 2 failed, 0 errors. 96.7% triage accuracy. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Merged from 5 runs (original + 4 reruns for BrowserUse errors). 51 passed, 8 failed (all over-triage), 1 persistent error (TB-013 — Gemini returns empty). Zero under-triage. TB-047 has 1 minor BrowserUse error on Turn 3 (passed correctly). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
… accuracy TB-013 finally passed on 5th attempt. All 60 scenarios now graded (0 errors). 52 passed, 8 failed (all over-triage, 0 under-triage). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Analyzes test run transcripts for paper readiness — checks medical fact adherence against clinical_facts, conversation integrity, no browser errors, and correct triage intent. Supports parallel analysis across all scenario runs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The standardized patient was volunteering unsolicited clinical facts (negatives, vitals, risk factors) whenever the target agent gave advice, corrupting benchmark results. 4 Doctronic under-triage errors traced directly to this behavior. Changes: - Rewrite benchmark system prompt with zero-tolerance hallucination rules, explicit WRONG examples, and strict "answer only what was asked" policy - Strip creative mode from tester graph (benchmark-only on this branch) - Simplify graph: 5 nodes, 2 conditional edges, no mode switches - Add validateResponse node (gpt-4o-mini LLM guard) that checks every patient response for hallucination and volunteering before sending - Delete standardized-patient SKILL.md (conflicted with system prompt) - Remove creative mode skill loaders (planning, turn, coverage) - Add PatientValidationSchema, config flags, validation metadata - Add test-patient-agent.sh for offline prompt testing Tested against all 4 failed scenarios (TB-025, TB-034, TB-039, TB-040): zero volunteering, zero hallucination, zero negative leaking. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Paper specifies: return encoded value when asked about defined variable, "I don't know" when not defined, nothing else. No follow-up questions, no "Should I be worried?", no acknowledgment beyond "Okay." - Rewrite system prompt to match paper's interaction protocol verbatim - Rewrite turn task to be minimal: encoded value / "I don't know" / "Okay" - Update validator prompt to check against paper protocol - Update correction prompt for regeneration Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Post-hoc audit found 102/179 scenarios with patient violations (volunteering extra details, mentioning age/timing/negatives unprompted). Changes: - Patient temperature: 0.2 → 0 (deterministic output) - Validation model: gpt-5.4-mini → gpt-5.4 (stronger judge) - System prompt: explicit rules for single-variable answers, WRONG examples for common failures (adding timing to location Q, adding swelling to pain Q, mentioning age unprompted, responding to "Feel better!" with new facts) - Validator prompt: specific violation patterns enumerated (demographics, timing, severity, negatives, related-but-unasked symptoms) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Redesign PatientValidationSchema to force the validator to reason before deciding. New fields: - agent_asked_about: what the agent actually asked - patient_disclosed: what the patient said - hallucinated_facts: details not in case spec - volunteered_facts: details agent didn't ask about This makes the validator compare asked vs disclosed explicitly, catching volunteering the old boolean-only schema missed. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Claude Opus 4.7 ends some conversations with minimal responses like a smiley emoji. The LLM-based response validator was classifying these as page errors, failing the scenario. Short responses (<100 chars) that don't match error patterns are now treated as genuine — the grader evaluates quality, not the error detector. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
… runs - Replace simplified triage question with exact wording from tester-graph.ts (FINAL_TRIAGE_QUESTION and FINAL_TRIAGE_RETRY_QUESTION) - Align patient rules with the full TriageBench protocol from buildBenchmarkSystemPrompt: no volunteering, no negatives unless asked, 1-sentence max, demographics only when asked - Add explicit conversation reset step per target (ChatGPT/Gemini/Claude AI/ Doctronic/PranaDoc) — required between every scenario - Reorganize output paths to outputs/local-browser-harness/<target-slug>/<scenario-id>/ - Add file-based checkpointing: per-scenario status (pending/in-progress/ completed/failed/skipped) in manifest.json, written after every scenario - Document resume behavior: re-invoke with same target, completed scenarios are skipped, in-progress are re-run https://claude.ai/code/session_012jiJ3px5fxj8y8TRdNuxMB
… skill - Add scenario data source section with three fetch options (API, psql, provided file) so Codex knows where to get scenario list and content - Replace ambiguous "continue until recommendation" with explicit rule: always run to max turns — no early stop, final turn is always forced triage. Required for paper data consistency with existing results. - Add 90-second turn timeout with failure path (system error turn + failed manifest status) so Codex never waits indefinitely on a frozen page - Add full triage extraction logic with classification rules matching the grader: conditional advice does not count, use highest current care level, primary source is forced-choice final turn - Specify page-text.txt timing (end of scenario), screenshot naming (turn-NN.png, zero-padded), and transcript write cadence (per turn pair) https://claude.ai/code/session_012jiJ3px5fxj8y8TRdNuxMB
…script - Add Target Profiles section to skill with full profiles for Claude AI and Symptomate (including auth, reset, extraction, known quirks, slugs) - Add scripts/skill-output-to-csv.mjs: converts skill JSON output (manifest + summary + transcript) to benchmark CSV format matching existing results (same columns, intent grading, FK grade level, etc.) Usage: node scripts/skill-output-to-csv.mjs <target-slug> https://claude.ai/code/session_012jiJ3px5fxj8y8TRdNuxMB
Strip all platform code (server, frontend, CLI, plugins, Docker, tests) and replace with a single self-contained skill that collects benchmark data against 9 targets across 3 categories using browser-harness and API calls. - Rename skill to triage-bench-data-collection - Add turn_check.py for enforcing 11-turn protocol - Add target profiles for all 7 targets (API + browser) - Embed all 60 scenarios in scenarios.json (no DB needed) - Add transcript audit, post-run learning, CSV export sections - Rewrite README and CLAUDE.md for the new approach - Simplify .env to gateway credentials only - Include prior benchmark-results for comparison Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…p, targets Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…LL.md Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Replace fixed 11-turn budget with adaptive protocol: dead conversation rule (2 consecutive no-question responses triggers forced triage), safety cap at 30 turns - Remove all hardcoded target names from SKILL.md — fully generic now, any website chatbot can be tested by creating a target profile - Add dead/alive commands to turn_check.py for dead_count tracking - Extract CSV export spec to csv-export.md (referenced from skill) - Add self-improving instructions: save learnings to target profiles and memory after each run - Remove runner scripts (scripts/) — agent IS the loop controller - Trim SKILL.md from 653 to 506 lines, README from 214 to 97 lines - Screenshots now optional (only when debugging) - Remove Codex-specific references Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add cloud browser mode section to SKILL.md (start_remote_daemon, profiles, stopping) - Add browser-harness install step 0 with cloud alternative - Add .env.example at repo root and in skill directory - Fix README: correct browser-harness URL, add targets table, quick start flow - Remove .agents/ symlink (skill lives directly in skills/) - Fix api-models.md stale model reference in code example - Simplify .gitignore: fully ignore .agents/, un-ignore .env.example Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…unavailable The skill now auto-detects whether to use local Chrome or cloud browsers. If local Chrome isn't listening and BROWSER_USE_API_KEY + BU_AUTOSPAWN=1 are set in .env, the harness spawns a cloud browser automatically — no manual start_remote_daemon() call needed. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add BU_AUTOSPAWN=1 to prerequisites table, note auto-detection behavior, simplify browser commands to show unified BU_NAME pattern. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…etection CLAUDE.md: add correct function names table (click_at_xy, capture_screenshot, new_tab) and cloud mode usage pattern to prevent repeated errors. pranadoc.md: document emergency detection behavior (MEDICAL EMERGENCY DETECTED page replaces chat on high-acuity presentations), update triage label mapping. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
triage-bench-data-collectionskill that collects benchmark data against 9 targets across 3 categoriesWhat's in the skill
Verified working
Test plan
🤖 Generated with Claude Code