bench(lab): mine paid runs for where Sense came up short - #276
Merged
Conversation
A margin says Sense helped. It does not say what to fix, and a bench that produces numbers with no route back to a product surface banks wins and teaches the product nothing. Three detectors over the runs that already happened: a gold row the answer cited that a resolver was asked for and never returned, a symbol a resolver answered with different file counts across runs, and a resolver call that returned nothing. Each has a real recorded example, and four more described in the retired tree are deliberately absent because a detector with no real example does not ship. Detection is post-run and the type says so. Every detector takes a Completed, whose only constructor demands the terminal outcome the run recorded, and a run that never reached one has no outcome to hand over. So the miner cannot be consulted before spending and cannot become a screen. That rule is not caution: a pre-run oracle reported 16 of 16 gold dependents resolved and zero failures on the exact repository and anchor where the miner found six cited-not-returned rows, 33 minutes apart. The coverage report shares its input with the detectors and nothing else, because one derived from what the detectors inspected would answer "what did we look at" while appearing to answer "what was exercised". Every surface appears, including the zeros: a missing row reads as a row that is fine. Fixtures are six real captures from the campaign whose misses were found by hand, five models between them. Run over them, the miner names the three misses that block reports.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A margin says Sense helped. It does not say what to fix, and a bench that produces numbers with no route back to a product surface is a bench that banks wins and teaches the product nothing.
The route runs through the runs that already happened. Every paid run leaves a transcript and, for the sense arm, a complete record of what Sense was asked and what it returned.
Summary
lab/internal/minereads recorded runs and reports where a Sense surface came up short, with a separate coverage report saying which surfaces the runs actually exercised.Changes
Completed, whose only constructor demands the terminal outcome the run recorded. A run that never reached one has no outcome to hand over, because reading its directory refuses. The miner therefore cannot be consulted before spending and cannot become a screen. That rule is not caution: a pre-run oracle reported 16 of 16 gold dependents resolved and zero failures on the exact repository and anchor where the miner found six cited-not-returned rows, 33 minutes apart.fileandrefas locations at any depth, and skips a truncated frame rather than refusing to mine a paid run over it.Recorded during the build
The names reproduce; the denominators are a different run set. The hand-found block reports its misses at n/7. The fixture corpus here is six recorded runs across five models on the same repository and scenario, so the miner reports the same rows against a denominator of six. Borrowing the 7 would be reporting a number this corpus did not produce.
Nondeterminism counts distinct files, not rows — that is what an answer can use. Empty returns group on the arguments, not the symbol, because an empty return that only happens with one option set is a different finding.
The miner is handed its cited rows rather than deriving them. Matching an answer against gold is the scorer's job; a miner that re-implemented it would report misses against a second, differently wrong matcher.
Test Plan
make cigreen: build, tests, per-file coverage floor with no new exception, zero complexity suppressions, lint clean.Completedboth produce nothing.