Dashboard: detailed extraction view with per-record reasoning chain

[TKasperczyk]

Problem

The extraction detail view shows a summary of extracted records (type, snippet, ID) but provides no insight into WHY the LLM extracted each record. When reviewing extraction quality, you need to understand the reasoning: what transcript evidence prompted each record, why certain fields were chosen, whether the LLM correctly identified the final state vs intermediate attempts.

Proposal

Enhance the extraction detail view with richer information:

• Per-record reasoning: Store and display the LLM's reasoning for each extracted record. This could be done by adding a chain-of-thought step before the tool call, or by requesting a reasoning field per record in the extraction schema.
• Source excerpt highlighting: Show the sourceExcerpt for each record with a link/expand to view it in context of the surrounding transcript.
• Transcript viewer: Embed a scrollable transcript view in the extraction detail page, with extracted records annotated inline (highlighted regions showing where each record came from).
• Dedup decisions: Show which records were deduped (matched existing), which were inserted fresh, and what the similarity score was for dedup matches.
• Token breakdown: Per-extraction-call token usage (not just aggregate), including cache hit rates.

The goal is to make extraction review a first-class workflow where you can quickly assess quality and spot issues.

[TKasperczyk]

Cross-checked this against the code; two corrections to the issue text, then a proposal to split it into four ordered sub-issues.

Corrections:

• sourceExcerpt IS already displayed per record in the detail view (220-char truncation with fallback), so the "source excerpt highlighting" bullet is partially done -- what's missing is the expander/in-context view.
• The "chain-of-thought step before the tool call" option is non-viable: extraction forces tool_choice to the extraction tool, so no preamble text is emitted, and relaxing that reintroduces the missing-tool-block failure mode the code explicitly defends against. A per-record reasoning field in the tool schema is the only viable path. (Note: it adds output tokens to every automatic extraction, not just reviewed ones.)

Proposed split, ordered:

1. Persist per-record extraction outcomes (small-medium, highest value). For each extracted record: outcome (inserted/updated/skipped-dedup/failed), dedup similarity -- computed today and discarded (findSimilar returns similarity, the caller uses only .record) -- and the extracted-id -> existing-id mapping. That mapping fixes a latent defect: run.extractedRecords summaries are keyed by extraction-time UUIDs, but dedup-updated records are stored under the EXISTING record's id, so the summaries join to nothing (the field currently has zero readers). Persist skipped/failed counts too -- today they exist only in the audit line.
   Two hazards: (a) dedup-matched existing-record IDs must stay OUT of the extractedRecordIds/updatedRecordIds union -- run deletion and re-extract delete that union, and per the documented deletion design decision that would destroy pre-existing records the run never created; (b) every new ExtractionRun field must be threaded through coerceExtractionRun (rebuilds from an explicit field list on every read, silently dropping unknowns) and the re-extract rebuild. The DELETE and re-extract routes gained their first tests in 9c7f8dc; extend those.

2. Token split (small): extraction vs usefulness-rating tokenUsage per run, plus cache hit rate (keeping input+output-only totals per the documented design decision). Must also fix backfillFromExtractionRuns, which re-emits merged run usage as a single extraction event, and tolerate old merged-only runs.

3. Per-record reasoning field (medium): optional field in the extraction tool schema, stored run-scoped and keyed via the mapping from (1) -- never spread onto MemoryRecord (pool/content-length pollution).

4. Transcript viewer (large, deferrable): excerpt anchoring is unreliable by design -- the prompt permits descriptive non-verbatim anchors for some record types, truncation markers are stripped, fallback excerpts are reconstructions, transcripts are tail-truncated before extraction, and the files are owned by Claude Code and may be deleted. Needs fuzzy anchoring plus a missing-file fallback, and the incremental start index isn't persisted (re-extract also leaves a stale isIncremental flag). An interim full-sourceExcerpt expander may deliver most of the value.

Also relevant scoping context: since 9c7f8dc the detail view renders run-level errors/status, and the existing extraction-review feature (per-record evidence + accuracy score) already covers part of the quality-assessment goal -- worth defining this issue's remainder against both.

[TKasperczyk]

Sub-issue 1 (per-record outcome persistence) landed in dc1ae47.

What's now persisted per run:

• extractedRecords rows carry outcome (inserted | updated | skipped | failed), storedRecordId (the extraction-time-id -> stored/existing-record-id mapping -- fixes the phantom-UUID join defect), dedupSimilarity (3 decimals, previously computed and discarded), and a sanitized storeError.
• skippedRecordCount / failedRecordCount on ExtractionRun: explicit 0 from new producers, undefined for legacy runs so consumers can distinguish "no data" from "zero".
• Outcome semantics: one row per record reflecting on-disk truth. Post-store supersedes failures keep inserted/updated with storeError attached; failedRecordCount counts only non-persisted records. Bonus fix: updateRecord returning false (row vanished mid-run) was silently counted as a successful update -- now it's a failed outcome.

The delete-union hazard flagged earlier is invariant-tested: dedup-updated record ids remain in the extractedRecordIds/updatedRecordIds union (per the documented "deletion includes updated records" decision), while skipped matches' ids exist only as storedRecordId and never enter the union -- run deletion cannot destroy records the run never modified.

Rendering (minimal, per the split): ExtractionDetail gained skipped/failed metric tiles (dash for legacy runs) and outcome/similarity badges on record rows with the id -> stored-id mapping; debug-cli detail shows the same. Legacy runs render unchanged.

Remaining for this issue: sub-issue 2 (token split extraction vs usefulness-rating), sub-issue 3 (per-record reasoning field, now keyed via the mapping above), sub-issue 4 (transcript viewer).