docs(motoko): iteration 8 — the fmt −74% is one benchmark and one void pair - #742
Merged
Merged
Conversation
…mark and one void pair
Queue item 6 (motoko mission iteration 8): design HOW we re-prove the -74% tokens-to-pass
result on the new tree, and decide whether `motoko_ext_fmt` survives.
The controller's Gate-2 measurements reframed the item before the designer ran. The -74%
(AC5, 2026-07-31) is real and correctly attributed, but it is not a target an instrument
can be built to reproduce:
- its own author rates it "direction, not proof" (n=1/pair, sign test p~0.11);
- 74.7% of the token saving is ONE benchmark (log_file_analyzer): all six pairs give
-74.2%, dropping it gives -47.1%;
- that benchmark is 3/30 lifetime and 0/10 over the last five nights in the rag_on
opencode rotation lane, with open issue #649 — tokens-to-pass is undefined when
nothing passes;
- one of the six pairs was quarantined by the harness itself
(`emit_exact_bytes_varied` ON: zero fmt-hook events, validity treatment_unproven)
and summed into the published total anyway;
- the run was order-confounded: all six ON rows completed before the first OFF row
started, so temporal drift is fully aliased with the treatment.
On the four pairs with both proven treatment and a currently-passable benchmark the
headline is -5.7%, not -74.2% (ON cheaper 3/4).
The doc replaces "re-prove -74%" with a paired censored win-rate that is defined when
arms fail, a counterbalanced execution schedule with an order-integrity gate, honest
rig pricing, and a decision rule pre-registered before any data.
Also surfaced and confirmed first-party: the Wednesday fmt A/B lane has banked nothing
since AC5 because `internal/executor/motoko/healthcheck.go:64` refuses unconditionally
on OPENROUTER_API_KEY, with no lane condition, while both fmt arms declare
provider: ollama / env_var: "" — its own error text ("motoko routes ALL models via
OpenRouter") is false for these entries.
PARKED needs-human-review on ONE bounded question (D-MOTOKO-FMT-1): is tracing motoko's
runtime provider resolution a precondition of D1, or does D1 need a redesign? Two quorum
rounds, both reviewers present both times, absent_reviewers empty; 3 of 4 objections
answered by controller measurement and carried into the text. No reviewer disputed the
instrument's direction in either round. Metered $0.1424.
Co-Authored-By: Claude Fable 5 (designer)
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…and one void pair
Gate 4 record for motoko iteration 8 (queue head item 6, fmt re-measurement instrument):
charter STATUS + rotation, log entry 8, dashboard overwrite, queue tags, decision ledger.
STATUS rotation asserted before writing: charter 549 -> 549 lines
(after == before + 2 - 2*moved), exactly 3 stamps remain, and the moved ITERATION 4 stamp
was confirmed PRESENT in the archive afterwards with ITERATION 2 as a firing control --
the charter-side arithmetic alone passes identically for a move and for a deletion.
Queue: item 6 -> DESIGNED + PARKED needs-human-review on D-MOTOKO-FMT-1 alone; new row 6b
carries the weekly external-issue sweep's 15 orphans of 75 enumerated open issues (per-issue
table, controls firing, list length asserted). Decision ledger gains D-MOTOKO-FMT-1 (OPEN);
scripts/mission_decisions.sh --check passes at 3 rows, --open generates exactly that one.
Gate-5 skill fix (one edit, two frictions, same zsh-array class):
1. iteration 140 -- FILES=$(...) plus unquoted $FILES does not word-split in zsh, which
produced a VACUOUS mutation test (already recorded in rule 3a(i-c));
2. this iteration -- the array that rule prescribes as the remedy is 1-INDEXED in zsh, so
${counts[0]} is empty. Gate 0's weekly-sweep table rendered all 8 per-issue counts under
the wrong file's header and dropped mission-dashboard.md from every row, while looking
exactly like a correct table. The orphan total survived only because the accumulator
summed the loop variable rather than the display array -- a broken table beside a correct
total, where the total certifies the table.
The gap is that rule 3a(i-c)'s own remedy carries the next footgun one line away: iterating
"${arr[@]}" is safe, indexing is not. Recorded at the remedy itself, with the general form --
a remedy is an instrument too, so a construct this skill prescribes brings its footguns with it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motoko mission iteration 8 — queue head item 6 (fmt re-measurement instrument)
The pick's own premise did not survive Gate 2, and that is the finding.
The −74% is one benchmark and one void pair
The −74% tokens-to-pass headline (AC5,
m-fmt-dialect-alignment.md, 2026-07-31) is real, correctlybanked, and correctly attributed to the fmt extension. It is also not a target an instrument can be
built to reproduce. Five things, all re-derived first-party this session:
log_file_analyzer→ −47.1%; that pair alone is 3,125,933 of 4,182,882 saved tokensrag_onopencode rotation lane (open #649) — tokens-to-pass is undefined when nothing passesab2_fmt_on/emit_exact_bytes_variedbanked zero fmt-hook events withvalidity={valid:False, reason:treatment_unproven}and was summed into the headline anyway; the other 5 ON rows carry exactly onestatus=formattedevent, all 6 OFF rows cleanOn the four pairs with both proven treatment and a currently-passable benchmark: −5.7% (ON
cheaper 3/4), not −74.2%.
What ships
design_docs/planned/m-motoko-fmt-remeasurement-instrument.md— replaces "re-prove −74%" with apaired censored win-rate (a non-pass is right-censored at the 4M cap; null
P(ON wins | non-tied) = 0.5, defined even when nothing passes), a counterbalanced per-benchmark
schedule with an order-integrity gate that VOIDs a slot banked otherwise, ELO-banded selection,
~9.8 rig-hours priced off a measured 4.91 min/row anchor, and a pre-registered KEEP/RETIRE rule.
Confirmed live defect, found on the way
The Wednesday fmt A/B lane has banked nothing since AC5. Both the 2026-08-05 and 08-12 fires
died at
internal/executor/motoko/healthcheck.go:64— an unconditionalOPENROUTER_API_KEYrefusal with no lane/model condition — whose own error text "motoko routes ALL models via
OpenRouter" is false for both fmt arms, which declare
provider: "ollama",env_var: ""("No API key — local inference") and
agent_model_name: "ollama/qwen3.6:35b-a3b-mxfp8"(
models.yml:1854,:1880).Quorum
Two rounds, both external reviewers present both times,
absent_reviewersempty in bothartifacts (neither verdict is an N−1 degrade). R1 BLOCKED → one revision → R2 BLOCKED. Metered
$0.1424. Every objection was classified premise-vs-design and every premise was measured
rather than forwarded; three are answered and carried into the text:
gpt5-6-sol(arm ordering) — upheld, and its "if" is fact: finding (e).gemini-3-1-pro(thecp-to-LaunchAgents claim) — upheld.PlistBuddyon theinstalled plist prints
{/bin/bash, …/dev/sunholo-data/ailang/tools/launchd/nightly-eval.sh}:the script executes in place from V1's checkout. Replaced with the real constraint (a merged
fix reaches the rig only when V1's clone pulls — open launchd drivers execute from the stale main checkout — #556's qwen3.5 retirement never reached the rig #558).
gemini-3-1-pro([nightly-eval] Nightly sustained failure: log_file_analyzer (2026-08-11) #649 unverified) — procedurally right; claim measured TRUE(OPEN, created 2026-08-11; control fix(hooks): SessionStart handed its stdout to a background child, refusing 1 in 4 mission fires #721 → MERGED).
gpt5-6-sol(D1's routing premise) — the park, see below.PARKED
needs-human-reviewon D-MOTOKO-FMT-1 alone. The narrow-refinement carve-out wasdeliberately not taken: it requires every remaining objection to need no controller judgment, and
O4's remedy is an investigation (the
mk-astresolution path and/or a live motoko run underrig.lock). No reviewer disputed the instrument's direction in either round.Also in this PR
boundary): 15 orphans of 75 enumerated open issues, per-issue table, controls firing, list
length asserted. Batched into new queue row 6b; a sweep never outranks a pick.
prescribes an array, and zsh arrays are 1-indexed —
${counts[0]}is empty, so the sweep'sfirst table shifted every column and dropped the 8th file while looking exactly like a correct
table. Recorded at the remedy itself.
in the archive with a firing control), log entry 8, dashboard overwrite, decision ledger.
🤖 Generated with Claude Code