bench(lab): the authoring loop, the miner and the candidate chain - #280
Merged
Conversation
Authoring is a cycle, not a line, and what happens on the second attempt is where the method lives. Three verdicts send work back, and each is an opportunity to lose the thing that made the previous attempt worth improving. The instinct when a draft fails is to clear it and start clean, and the cost of yielding to it is invisible: six unrelated attempts look exactly like six iterations from the outside. lab/internal/loop holds a repository's position and refuses a re-entry that carries no table. Nothing is deleted: the attempt is filed with its anchor, its draft and the rejection, and the next attempt answers every rejection so far rather than only the last. Re-anchoring is its own decision with its own verdict; a re-question never touches the anchor, and a NO-ANCHOR records that one is owed without discarding the one that failed. Both ceilings refuse rather than warn. The authoring ceiling parks after the sixth cycle and is re-entered only by a deliberate resume, which hands a fresh cycle count and keeps the history. lab/internal/budget reads spend from the run tree every time instead of holding a counter, so a restart cannot spend the budget twice: the total is the size of a set of run directories, which makes double-counting unrepresentable rather than merely guarded against. A campaign runs unattended for hours, so no phase may ask a human anything. The lab boundary test now walks every production file for a read of os.Stdin and allows it only where a stream is handed to a command.
This is the method. Everything else in nine cycles is machinery that runs it. The binary holds no prompt logic by design, so how a scenario is crafted, how a mini-bench is read, how a session is expanded, how a pay call is made and how a win is confirmed all live as prose a human wrote. The risk is specific: a rewrite ships a cleaner engine on day one and quietly loses the discipline, because the scoreboard stays green either way. These files and LAWS.md are the only defence. Eleven plans, one per phase in the graph, each declaring the phase it belongs to, the artifact it reads, the artifact it writes and its verdict enum. lab/internal/plans checks all four against the graph in both directions: a plan naming a verdict the graph does not know cannot be routed, and a plan silent on a verdict its phase can emit is a case with no instruction, which is how a phase writes nothing and stalls a campaign unattended. Porting found a hole. The old expand plan emits REQUESTION when the discriminator step cannot survive verbatim, and the graph gave expansion AUTO alone. REQUESTION joins its enum, the re-entry table and the lever list, because a lever a recorded campaign used and the table does not carry is a hole in the test set. The five killers move to lab/KILLERS.md, tracked beside LAWS.md, since an instructions file outside the repository is one a fresh session does not have.
A campaign runs unattended for hours and is picked up later, often by a session with no memory of it. The retired instrument answered "where does this stand" with a page assembled from a results tree, two state files, a banked log and a ledger, plus a hand-maintained resume file that goes stale by construction. A stale resume line is worse than none, because it looks current. sense-lab status derives everything from the run tree. Position is the furthest phase of the latest cycle whose artifact is on disk and the first whose artifact is not, so nothing hand-maintained can disagree with what is there. The page carries the banner saying so. The uncomfortable rows lead rather than being summarised away: incomplete cells with their burned and unusable arms named, orphaned run directories that spawned and recorded no terminal state, parked repositories, and spend against the ceiling. A status that shows only progress is a status nobody trusts twice. Read returns a struct and Render takes one; the only consumer of that string is a printer, and nothing else in the tree imports the package. Parsing your own report is how a display format silently becomes a data contract. Deriving position found that indexing is per repository rather than per cycle: looking for the index under every cycle would report every re-entered repository as waiting on a scan that already happened.
A margin says Sense helped. It does not say what to fix, and a bench that produces numbers with no route back to a product surface banks wins and teaches the product nothing. Three detectors over the runs that already happened: a gold row the answer cited that a resolver was asked for and never returned, a symbol a resolver answered with different file counts across runs, and a resolver call that returned nothing. Each has a real recorded example, and four more described in the retired tree are deliberately absent because a detector with no real example does not ship. Detection is post-run and the type says so. Every detector takes a Completed, whose only constructor demands the terminal outcome the run recorded, and a run that never reached one has no outcome to hand over. So the miner cannot be consulted before spending and cannot become a screen. That rule is not caution: a pre-run oracle reported 16 of 16 gold dependents resolved and zero failures on the exact repository and anchor where the miner found six cited-not-returned rows, 33 minutes apart. The coverage report shares its input with the detectors and nothing else, because one derived from what the detectors inspected would answer "what did we look at" while appearing to answer "what was exercised". Every surface appears, including the zeros: a missing row reads as a row that is fine. Fixtures are six real captures from the campaign whose misses were found by hand, five models between them. Run over them, the miner names the three misses that block reports.
The miner produces observations. Between one and a product change there are three steps that otherwise happen in someone's head and in prose, and the retired tree shows both failure modes: one campaign's miner output sits in a log file with an empty findings directory beside it, and the findings that were written up live as three loose markdown files under a different campaign, connected to nothing. "Why was this change made to Sense" is answerable only by someone who remembers. Three records, each pointing at the last. A finding cannot be recorded without a surface or without evidence. A candidate cannot be recorded without a finding that is actually on disk, because a change with no observed problem behind it is taste. A validation cannot be recorded before its candidate wrote its hypothesis down, because a hypothesis written afterwards is a description. Evidence is a path into the run tree, never a copy: a finding that embedded a transcript excerpt would drift from the transcript the moment either was touched. And a finding inherits the state of its evidence. Invalidating a run marks every finding that cites it, nothing is removed from the evidence list, and a finding whose support is entirely withdrawn is marked rather than deleted, because it was once observed. Ids are derived from what a record is about, so re-mining the same corpus updates rather than duplicates. It is a record and not a tracker: no states, no assignment, no triage, no priority.
A candidate change to Sense has to earn its way in, and "earn" has a definition: it moves the cells its hypothesis named, it does not regress the ones it did not, and the movement is not confined to one model. Three checks, and every result carries what qualifies it. The corpus size travels with the regression result, so a pass reads as "four of four held" rather than as "no regressions": four cells is thin enough that a change can pass and still be wrong. A candidate whose movement was seen on one model is recorded as single-model, never as confirmed. And the blind spot is measured rather than assumed: when the corpus exercises none of the surface a change touches, the pass says nothing about it and the result says so. A cell holds if it lands inside its own recorded spread, because a suite demanding an exact number would fail on the instrument's own noise. A target that did not move at all has not moved as predicted: a hypothesis that survives "nothing happened" predicts nothing. And a corpus made only of the cells the hypothesis named is refused, because that is a change measured against its own intent. Nothing here ships anything. The numbers decide the verdict and a human decides to ship, and a rejection is recorded with its numbers rather than discarded, so the next person with the same idea finds the measurement instead of repeating it.
Everything measured so far has been measured against recordings. That proves the pure layer reads the past correctly and proves nothing about whether the instrument can produce a result: between a recorded transcript and a live cell sit isolation, supervision, the tee, the session, the judge and the verdict layer, and every one of them is new. This is what surrounds the live run. The model is pinned, because a banked number is a number for a model: one recorded cell measured +0.625 on one model and 0.00 on its successor, because the newer baseline closed the axis the scenario tested, and replayed on the wrong model it looks like a broken instrument while being a working one. An unreachable model picks a different CELL, never a different model, and the cell that can no longer be replayed is reported rather than dropped. The pick takes the cell whose worst run is furthest clear of the bar, not the highest margin. Against the real corpus that is mastodon at 0.775 with a spread of 0.050; discourse banked 0.708 with a spread of 0.417, so its worst run lands at 0.291 and a disagreement there could not be told from noise. Agreement is the cell's own recorded spread in both directions, and landing outside it is kept separate from still clearing the bar. A disagreement walks instrument, then environment, then a second pair, then the world: one live pair is n=1 against a spread reaching 0.250, and without that stopping point the ordering is a licence to debug forever. And the pinned inputs are diffed, because "nothing was adjusted to make it land" is a claim and a replay tuned until it agrees has proven that tuning works. The live paid run itself is outstanding and is a human trigger.
bench(lab): decide whether a candidate earned its way in
bench(lab): choose and judge the replay of a banked cell
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why this PR exists in this shape
Cycles 05 and 06 were built as eight stacked branches, one per pitch, each with its own PR. Merging that stack deleted each base branch as it went, which closed three of the PRs pointing at it and left the rest merged into intermediate branches. Only #272 reached
main.Every commit survived on this branch. This PR carries the remaining seven pitches onto
mainin one merge. The per-pitch review record stays readable on the original PRs: #273, #274, #275, #276, #277, #278, #279.Nothing was rebuilt or squashed. The commits are the ones those PRs described, in order.
What lands
Cycle 05, the authoring loop
lab/KILLERS.md.sense-lab status, derived entirely from the run tree. The uncomfortable rows lead: incomplete cells, burned arms, orphaned run directories, parked repositories, spend against the ceiling. No hand-maintained resume file.Cycle 06, miner, candidates and replay
Building.Test Plan
make cigreen on this assembled branch: build, tests, per-file coverage floor with no new exception, zero complexity suppressions, lint clean, total coverage 96.0%.