Skip to content

bench(lab): mine paid runs for where Sense came up short - #276

Merged
luuuc merged 1 commit into
lab/05-04-statusfrom
lab/06-01-the-transcript-miner
Aug 17, 2026
Merged

bench(lab): mine paid runs for where Sense came up short#276
luuuc merged 1 commit into
lab/05-04-statusfrom
lab/06-01-the-transcript-miner

Conversation

@luuuc

@luuuc luuuc commented Aug 16, 2026

Copy link
Copy Markdown
Owner

Problem

A margin says Sense helped. It does not say what to fix, and a bench that produces numbers with no route back to a product surface is a bench that banks wins and teaches the product nothing.

The route runs through the runs that already happened. Every paid run leaves a transcript and, for the sense arm, a complete record of what Sense was asked and what it returned.

Summary

lab/internal/mine reads recorded runs and reports where a Sense surface came up short, with a separate coverage report saying which surfaces the runs actually exercised.

Changes

  • Three detectors, each with a real recorded example. A gold row the answer cited that a resolver was asked for and never returned; a symbol a resolver answered with different file counts across runs; a resolver call that returned nothing. Four more are described in the retired tree — abandoned-on-empty, ignored-hint, contract-misled, wrong-tool-shape — and are deliberately absent, named in the package doc with the rule that admits them: a detector with no real example does not ship.
  • Detection is post-run, and the type enforces it. Every detector takes a Completed, whose only constructor demands the terminal outcome the run recorded. A run that never reached one has no outcome to hand over, because reading its directory refuses. The miner therefore cannot be consulted before spending and cannot become a screen. That rule is not caution: a pre-run oracle reported 16 of 16 gold dependents resolved and zero failures on the exact repository and anchor where the miner found six cited-not-returned rows, 33 minutes apart.
  • Every finding names a surface, and a finding with no surface is unrepresentable: a call to a tool that is not a Sense surface yields no surface and therefore produces nothing.
  • Discriminator misses are marked apart from the rest, because a miss on the group that carries the margin is worth more than a miss elsewhere.
  • No score and no rank. A severity number invites treating the top of a list as the work queue, and the interesting finding is often the rare one. Each finding carries its numerator and its denominator: three runs out of five and three out of twenty are different findings.
  • The miner reports, it does not interpret. "Blast was called and did not return this file" is a fact; "blast has a resolver bug" is a hypothesis, and hypotheses belong to a human with the transcript open.
  • The coverage report shares only its input with the detectors. One derived from what the detectors happened to inspect would answer "what did we look at" while appearing to answer "what was exercised". Every surface appears including the zeros, because a missing row reads as a row that is fine.
  • The capture parser pairs each reply with its own request by id, reads both file and ref as locations at any depth, and skips a truncated frame rather than refusing to mine a paid run over it.

Recorded during the build

The names reproduce; the denominators are a different run set. The hand-found block reports its misses at n/7. The fixture corpus here is six recorded runs across five models on the same repository and scenario, so the miner reports the same rows against a denominator of six. Borrowing the 7 would be reporting a number this corpus did not produce.

Nondeterminism counts distinct files, not rows — that is what an answer can use. Empty returns group on the arguments, not the symbol, because an empty return that only happens with one option set is a different finding.

The miner is handed its cited rows rather than deriving them. Matching an answer against gold is the scorer's job; a miner that re-implemented it would report misses against a second, differently wrong matcher.

Test Plan

  • make ci green: build, tests, per-file coverage floor with no new exception, zero complexity suppressions, lint clean.
  • 98% line and 100% function coverage on the new package.
  • The fixtures are six real recorded captures from the campaign whose misses were found by hand, five models between them, with the cited rows taken from the scorer's own output. Nothing is synthetic.
  • Run over them, the miner names the three misses the hand-found block reports.
  • Negative cases carry the weight: a run where no resolver was called is not a miss, a file the resolver did return is not a miss, a non-Sense tool produces no finding, and a contract-group miss is present and unmarked so "marked" is not "marked everything".
  • Replies deliberately out of order, to prove a reply lands on its own call rather than on the previous one.
  • A truncated frame, an unpaired reply and unreadable JSON each leave the rest of the run minable.
  • A run with no terminal state and a zero-value Completed both produce nothing.
  • Coverage asserted unchanged on a run that produces zero findings.

A margin says Sense helped. It does not say what to fix, and a bench that
produces numbers with no route back to a product surface banks wins and
teaches the product nothing.

Three detectors over the runs that already happened: a gold row the
answer cited that a resolver was asked for and never returned, a symbol a
resolver answered with different file counts across runs, and a resolver
call that returned nothing. Each has a real recorded example, and four
more described in the retired tree are deliberately absent because a
detector with no real example does not ship.

Detection is post-run and the type says so. Every detector takes a
Completed, whose only constructor demands the terminal outcome the run
recorded, and a run that never reached one has no outcome to hand over.
So the miner cannot be consulted before spending and cannot become a
screen. That rule is not caution: a pre-run oracle reported 16 of 16 gold
dependents resolved and zero failures on the exact repository and anchor
where the miner found six cited-not-returned rows, 33 minutes apart.

The coverage report shares its input with the detectors and nothing else,
because one derived from what the detectors inspected would answer "what
did we look at" while appearing to answer "what was exercised". Every
surface appears, including the zeros: a missing row reads as a row that
is fine.

Fixtures are six real captures from the campaign whose misses were found
by hand, five models between them. Run over them, the miner names the
three misses that block reports.
@luuuc
luuuc merged commit 7f0d1cf into lab/05-04-status Aug 17, 2026
@luuuc
luuuc deleted the lab/06-01-the-transcript-miner branch August 17, 2026 07:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant