harness: measure a runner's MLLP ingress rate instead of pass/failing one point - #118
Merged
Conversation
… one point test_load_runner red the required windows-2025 leg twice on main (9b03057, 56f7d24) with byte-identical counters -- 90 sent, 44 acked, 46 stranded, 52 read -- on runs that lost nothing. #115 widened the reconcile's stranding budget so a saturated-but-lossless run stops failing. That fixed the symptom and said nothing about the cause, and a pass/fail test at ONE fixed offered rate structurally cannot: the question is whether windows-2025 is slow or windows is, and that is a rate, not a verdict. Adds `harness.load.ingress_probe` (sweeps offered rate, prints one machine- parseable RESULT line per run) and a workflow_dispatch-only workflow that runs it across ubuntu-latest / windows-2022 / windows-2025 and writes a side-by-side table to the step summary. ubuntu is the CONTROL: it establishes what "not saturated" looks like on hosted hardware so a slow Windows row can be read against it. REPEATS ARE THE POINT, and this is a correction to my own earlier claim. One 600/s run on a developer box produced 456 stranded of 900 (50.7%) -- a near-exact match for the windows-2025 CI signature -- and I wrote it up as a clean reproduction. Four repeats of the same command on the same box then produced 0 stranded, every time; the outlier was taken while an unrelated test suite was running. So stranding here is a CONTENTION artifact, not a clean function of offered rate, and n=1 is not a measurement. Hence --repeat, defaulted to 3, with the correction recorded in the module docstring so the next reader does not redo it. What survives that correction is the weaker, still-useful claim: an unloaded box strands ZERO at up to 10x the CI profile's rate, while windows-2025 stranded ~51% at the profile's own 60/s, twice, byte-identically. Deliberately NOT a required context and structurally unfit to become one: no pull_request trigger, and the probe exits 0 even when the reconcile fails, because a machine too slow to keep up is the finding rather than an error. Recorded in .github/required-contexts.txt with the other advisory workflows. Prints no derived per-second figure either -- engine_read/wall looks like a service rate and is not one (wall includes stop grace, drain and settle, so it lands at ~25/s whether the run offered 60/s or 600/s). Scope: both legs are hosted VMs, so a slow-2025/fast-2022 result narrows the cause to the 2025 IMAGE. It cannot clear or convict Windows Server 2025 as a deployment target -- only the self-hosted WS2025 rig can, and it was unregistered (actions/runners -> total_count 0) when this was written. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
wshallwshall
enabled auto-merge (squash)
August 1, 2026 13:27
auto-merge was automatically disabled
August 1, 2026 13:31
Pull Request is not mergeable
wshallwshall
added a commit
that referenced
this pull request
Aug 1, 2026
test_backlog_status_check::test_the_real_backlog_satisfies_the_invariant fails on origin/main at ea05525 itself. #320 (filed in #117) opened with a `📋 Filed …` banner, which is not one of the five the invariant accepts (SHIPPED / DECLINED / RETIRED / Re-scored / Status). An invented emoji instead of the defined vocabulary. It fails in 0.6s inside every required test leg, so it red every PR that compiles the suite -- already blocking auto-merge on #119 and would have blocked #118. Corrected to `🚧 Status: OPEN INVESTIGATION …`, which is what the item is. The banner now also carries the two facts a reader needs: measurement tooling landed in #118, and the decisive experiment is blocked on an unregistered self-hosted WS2025 runner. Coverage hole recorded, not fixed here: #117 was docs-only, and ci.yml's `changes` job short-circuits that case (code == 'false' skips install/lint/type/test), so the guard that polices BACKLOG.md does not run on a PR that only changes BACKLOG.md. The one class of change the invariant exists to catch is the class that skips it. Follow-up: either count docs/BACKLOG.md as `code` for the short-circuit, or run the ledger/backlog guards in a cheap always-on leg.
wshallwshall
enabled auto-merge (squash)
August 1, 2026 14:16
wshallwshall
added a commit
that referenced
this pull request
Aug 4, 2026
…ones (#163) * docs(backlog): close BACKLOG #226 — the estate Hybrid-layout sweep is done, off-repo The per-feed Hybrid split (connections.toml / <INBOUND>_router.py / <INBOUND>_handler.py / _<feed>_transforms.py) landed across the ported estate in the maintainer-internal migration repository. Owner-attested; nothing in this repository changes, which is also why leaving the item open could never have closed it. Both "Also" clauses are recorded as NOT delivered, with the reason each is not a residual of this item: - "align the IDE Corepoint-import / scaffold path to emit the Hybrid layout" — there is no Corepoint-import path in ide/ to align. That tooling is #105, still open, so the clause is a constraint on #105's design rather than work #226 can perform. The scaffold half is misaddressed too: Insert Element (#48) drops per-file idioms into the current buffer (ide/src/insertElement.ts:1-5) and emits no multi-file feed layout. - "consider a recursive-glob / folder-per-feed loader enhancement" — filed as a consider, and not taken: load_config still globs *.py non-recursively (config/wiring.py:4162), the flat-merge behaviour the Hybrid layout is built around. Follows the #227 precedent: close the primary, state the off-repo/misaddressed residuals explicitly so the item is not re-opened for them. backlog_status_check.py: OK — 277 items, each declaring exactly one status. * fix(ledger): teach the number-space gates to span an archive, and fix two holes found proving it Prerequisite for moving the 185 closed BACKLOG items into docs/archive/backlog/. No item has moved yet; this only makes the guards able to see one when it does. The item namespace will span two paths, so every guard now reads their UNION: - backlog_status_check.py: scan() takes (label, text) pairs and parses them as ONE namespace. A number re-used across BACKLOG.md and the archive was structurally undetectable before -- `seen` was per-parse -- which is the erratum's own shape. - ledger_check.py: triggers on any backlog-bearing path, not the one literal, and builds head/base as the union. Reading the union on both sides also removes a false positive: the move relocates 185 items, so head-union == base-union and `head - base` stays empty, where a per-file view would report 185 vanished numbers with a remedy that renumbers cited items. - alloc.ps1: sweeps both paths in the all-refs term and the working-tree term. - backlog-hygiene.yml: accepts a banner updated in either location. Two pre-existing defects surfaced only because the gates were made to fail on purpose first, neither of which is about the archive: 1. alloc.ps1's working-tree term has NEVER worked. `[regex]'^...'` anchors at the start of the STRING; the term feeds it `Get-Content -Raw`, one string starting "# Backlog". Measured: 0 of 277 headings matched without Multiline, 277 with. The all-refs term hid it by covering every number committed somewhere -- i.e. every case except the uncommitted one this term exists for. 2. backlog-hygiene.yml diffed BASE_SHA..HEAD_SHA (two-dot), which credits a PR for main-side changes to paths it never touched. One main-side edit to BACKLOG.md -- the move being a large one -- would let every PR with an older base pass the "must update BACKLOG.md" required check while enforcing nothing. Now three-dot, matching ci.yml's form for the same question. Anti-narrowing, because a green gate over a shrunken corpus is the failure mode: - `--min-items N` fails when fewer items are found than required, and CI pins 277. Without it, 277 -> 92 fails nothing. - The scanned files are always printed with the count; a bare integer cannot distinguish "items closed" from "a file stopped being read". - A liveness receipt in the test suite asserts the same floor. - An explicitly-named --backlog path that does not exist is an error, not a skip. alloc.ps1 gains `-ShowFloor`: print the floor and the swept paths, allocate nothing. Allocation is a one-way door, so before this the only way to ask what the floor could see was to spend a number on the question -- which is how it ran a whole release reading two refs while its header promised all of them. Get-Floor takes -Peek so the inspection cannot advance the high-water ratchet; the first -ShowFloor run against a planted number moved this clone's watermark 316 -> 990 before that was fixed. Proofs run, each observed failing BEFORE the fix: - archive-only unallocated #1007 staged: old gate rc=0, new gate BLOCKED. - #990 planted in the archive: old sweep floor 353 (blind), new sweep 990. - cross-file duplicate #118: detected, naming the other file. - banner violations inside the archive only: detected. - --min-items over a narrowed corpus: rc=1 with the scanned-file list. - -ShowFloor twice against a plant: watermark unchanged at 316. ruff + mypy --strict clean; 43 gate tests pass. * docs(backlog): move the 185 closed items into docs/archive/backlog/BACKLOG-CLOSED.md docs/BACKLOG.md becomes the ~92 items someone can act on: 8,742 -> 3,648 lines. The closed items are not deleted, summarised, or rewritten -- they are relocated verbatim, so the file that gets opened, grepped and edited daily is the open set. MOVED, NOT REWRITTEN. Every relocated block is byte-identical to the one that left BACKLOG.md, headings included. Verified mechanically against a pre-move copy: - 277 items before = 92 after + 185 archived, no overlap, union identical - every OPEN block byte-identical to its source - every ARCHIVED block byte-identical to its source - all non-item prose in BACKLOG.md preserved verbatim Byte-identical headings are load-bearing, not tidiness: GitHub derives anchor slugs from heading text, so all 64 archived->archived cross-references keep resolving with no edit at all. That is the whole argument for one archive file rather than a split by status, year, or cluster -- #52 alone receives 99 of the 110 in-file anchors, and its citers span #65 to #184, so no cut isolates them. Cutting item blocks at the next '## ' heading of EITHER kind, not the next numbered item: 4 blocks in this file are followed by a section header, which a naive cut would have dragged into the archive along with the prose beneath it. Anchors, all 127 re-resolved against real headings after the edit: - 44 rewritten in BACKLOG.md -> archive/backlog/BACKLOG-CLOSED.md#<same-slug> - 1 rewritten in the archive -> ../../BACKLOG.md#<same-slug> (#226 -> #105) - 3 cross-file links repointed: AOAG-DEPLOYMENT.md (#100, #101), ADR 0026 (#30) - 64 archived->archived untouched, by design 13 anchors still do not resolve, and ALL 13 WERE ALREADY DEAD BEFORE THIS COMMIT -- confirmed by running the same check over the pre-move file, which returns the identical multiset (11 bare-number self-anchors: #40 x4, #323 x3, #28, #29, #329, #333; plus 2 links to #13 in COUNSEL-ENGAGEMENT-BRIEF.md, a number this sequence never had). They are left dead and documented in the archive header rather than repointed at a plausible neighbour: a citation resolving to the WRONG item is the erratum's failure mode, and unlike a dead link it looks like success. The archive carries its retirement banner inline rather than in a sibling README -- docs/archive/throughput/ needs a README because it indexes five documents; one file does not, and two documents that must agree is a drift surface. It states the rules that keep the namespace honest: never renumber, re-open by moving the block back (never by copying, which creates the cross-file duplicate the status check now fails), and add any future archive file to alloc.ps1's $backlogPaths AND backlog_status_check.py's DEFAULT_SOURCES in the same commit -- a file named in neither is policed by nothing. Gates verified post-move: - backlog_status_check.py --min-items 277: OK, 277 items, and it now PRINTS "scanned: docs/BACKLOG.md (92), docs/archive/backlog/BACKLOG-CLOSED.md (185)" - ledger_check.py on the staged move: rc=0 (relocation adds no numbers, because head-union == base-union -- the exact false positive the union view removes) - alloc.ps1 -ShowFloor: floor 353 across both paths, next 1000 - 43 gate tests pass Note the floor is unchanged at 353 because the highest item (#353) is open and stays in BACKLOG.md. The archive-sweep fix is therefore PROSPECTIVE, not a save: it starts mattering the first time a top-of-range item closes and moves. * docs(backlog): re-score all 92 open items on the ten-level scale (2026-08-03) Every open item now carries a current value x difficulty score. Before this, 23 had none at all and the other 69 were from the frozen 2026-07-10 pass, which predates the 2026-07-28 reconcile that closed 31 items -- and a stale score reads exactly like a fresh one. Method, unchanged from the pass it supersedes: scored from each item's own Scope / Why / Trigger / Nearest-existing-mechanism text rather than rescaled from the old number, then adversarially verified against the code -- a second reader per batch attacking build state first, then verdict/tier, then value and difficulty. 26 of 92 scores were overturned by that pass and carry the refuter's number. The banner is the live record and the table is a view of it; both are written here and a mechanical check confirms 92 banners and 92 rows agree on every triple. THE RATIONALE IS REPLACED, NOT JUST THE NUMBERS. Carrying an old justification under a new score is how a banner comes to argue against itself: - #114's surviving "clean workaround via the on-demand test probe" is a claim PR #162 explicitly retracted -- both destinations' test_connection CREATE the target dir, so the probe cannot answer the question the toggle asks. That is what lifts it off the parity-with-a-workaround band to 6/3. Its replacement rationale was ALSO stale (it described the silent-ignore #162 had just fixed) and is hand-corrected. - #105's "large greenfield 71-action mapper needing its own ADR" describes an importer that has since shipped under ADR 0086. Scheduling barely moved, which is the reassuring result: only TWO tiers changed -- #64 DEMAND-GATE -> P3 (an index over levers that live in #62/#63/#47/#34, so it ships nothing runnable of its own) and #105 P3 -> DEMAND-GATE. Neither contradicts an explicit demand-gate/on-trigger ruling in its own body; that was checked for all 51 items carrying a prior tier. Distribution is RECOMPUTED with the table rather than carried forward, and all four lines sum to 92. The superseded table keeps its own frozen lines and now says so. Tiers: P1 4, P2 19, P3 17, DEMAND-GATE 52 Quadrants: quick win 22, big bet 5, fill-in 56, money pit 9 The four P1s: #341 (9/3, a handler returning a tuple/set of Sends delivers nothing silently -- an accept-and-drop CLAUDE.md §12 forbids), #324 (7/2), #325 (6/2), #327 (6/2). NOT in this commit: 24 items were found to misdescribe their own build state -- prose asserting a gap that has since shipped, or citing messagefoundry/console/, a package retired with #103. Those are banner corrections and land separately; the scores here already price the remainder rather than the original scope. Two mechanical faults were caught by reading the output rather than trusting the run: the quadrant regex omitted the hyphen in "fill-in", so 57 of 69 items took the fallback branch and got a SECOND score inserted beside the first; and the synthesizer's own distribution lines did not follow from its own table (11 quadrant mismatches, 8 ordering violations, difficulty summing to 95 of 92). The script now refuses to write when any line carries two score spans or the scored count is not 92. backlog_status_check.py --min-items 277: OK, 277 items across both files. * docs(backlog): correct 10 items whose own prose misdescribed build state The 2026-08-03 re-score flagged 24 open items as misdescribing what the code does. Re-verified each against the tree as it stands -- after the archive move and after PR #162, both of which post-date the findings -- and 10 survived. The other 14 did not, and are recorded here rather than silently dropped: #84 #95 #99 #105 #114 #124 #125 #127 #133 #137 #167 #169 #214 #228 Most of those already carry an amendment that covers the stale sentence (#95, #99, #105, #114, #124, #125, #127, #133, #228), and stacking a second ruling saying the same thing is noise. The rest did not survive verification: the finding was itself wrong or overstated, and a wrong correction in a ledger is worse than a stale one. CORRECTIONS ARE ADDED AS DATED AMENDMENTS, NOT PROSE REWRITES. This file's convention is to leave the original claim standing and rule against it, so the record shows what was believed and what replaced it. Silently editing the stale sentence would destroy the evidence that makes the correction checkable. Applied to #62 #64 #131 #166 #179 #182 #237 #321 #329 #336. Representative: - #329 "Five MEFOR_ALLOW_INSECURE_TLS cells": the census is FOUR. #323 landed and routed transports/direct.py through the clamp; it now holds no call to the raw predicate at all (:63, :197, :215). - #321 "no test asserts the detectors can see a site code": false -- tests/test_scan_forbidden.py has per-class hit tests for at least the site code (:126), a customer name (:83), a case-sensitive code (:91) and a routable IP (:107). The detector-coverage half of its Proposed 2 is already in the tree. - #62 plans a dual-read over "existing mfenc:v1 rows", but cell-bound mfenc:v2 is the default writer (settings.py:383 -> base.py:1841; crypto.py:36), and v2 folds (table, column, pk) into the GCM tag -- so a body landing under a different column must be RE-ENCRYPTED, not merely re-encoded. That tightens the catch. - #64's ordered plan still reads live ("Nothing builds before it"), but the measure-first phase completed 2026-07-12 (ADR 0051) and its step-2 lever is refused outright (ADR 0055 withdrawn; ADR 0107 "Do not build F2 or F3"). The refuters removed two overclaims before they landed: #62's draft asserted a live store holds both mfenc markers (a fresh store under the shipped default holds only v2 -- the defensible claim is that a MIGRATION must expect both), and #64's asserted the multi-DB log split still remains, which could not be verified against ADR 0098 and would have been a fresh false claim. No item closes here: in every case the correction narrows the remainder rather than discharging it, and the 2026-08-03 scores already price the remainder. backlog_status_check.py --min-items 277: OK, 277 items, one status banner each. * docs(backlog): file BACKLOG #1000 — prove each required merge context can fail Escalated by the coordinator on the ground that it outlives the PR that fixed it. Deliberately NOT filed as "fix the two-dot diff": that instance already landed in 39b62bf, and filing shipped work is the rot the hygiene gate exists to prevent. The item is the CLASS. `.github/required-contexts.txt` names 13 contexts that block merge, and not one of them is proven able to go red. The deliverable is a negative control per context -- a fixture carrying the exact violation that context exists to catch -- plus a CI job that fails when a required context has none, so the coverage cannot silently decay as contexts are added. Scoped narrower than "test the gates" on purpose: it does not re-test what each gate checks, since the gates' own suites do that. It asserts one property per context -- this gate is capable of failing. The argument is that the class has now fired at least four times here, each found by hand and none by CI: #334 semgrep, required and blocking, scans a two-directory allow-list #327 six .gitignore rules are the sole control over maintainer-internal docs, and nothing asserts they still match anything #321 the forbidden-content gate exited 0 on a real site code and partner product #325 the same gate's home-path detector misses 1 of 4 spellings of a Windows path Each is correctly filed as its own defect. None of them establishes the property that would have caught all four before they shipped, and that property is a different artifact from any of the individual fixes. Value 7 / Difficulty 3, quick win, P1 -- not demand-gated; the trigger fired four times. Ranked table and all four distribution lines recomputed to 93 open items; a mechanical check confirms 93 banners and 93 rows agree on every triple. Number allocated atomically via scripts/coord/alloc.ps1 (#1000 -- the first in the post-partition public sequence, clamped to >= PUBLIC_BACKLOG_FLOOR), never grepped. backlog_status_check.py --min-items 277: OK, 278 items across both files. The floor is a floor, so growth passes it; it is there to catch shrinkage.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Diagnostic tooling for item #320. Two new files plus a three-line note; nothing existing changes behaviour.
Why
test_load_runnerred the required windows-2025 leg twice onmain(9b03057f,56f7d240) with byte-identical counters — 90 sent, 44 acked, 46 stranded, 52 read — on runs that lost nothing. #115 widened the reconcile's stranding budget so a saturated-but-lossless run stops failing. That fixed the symptom and says nothing about the cause.The open question is whether the windows-2025 image is slow or Windows generally is, and a pass/fail test at one fixed offered rate structurally cannot answer it. That's a rate, not a verdict.
What this adds
harness/load/ingress_probe.py— sweeps offered rate, prints one machine-parseableRESULTline per run..github/workflows/ingress-rate-probe.yml—workflow_dispatchonly, runs it across ubuntu-latest / windows-2022 / windows-2025 and writes a side-by-side table to the step summary.ubuntuis the control, not a deployment target: it establishes what "not saturated" looks like on hosted hardware so a slow Windows row can be read against something.The correction this PR is built on
I have to flag this, because I got it wrong first: I ran 600/s once on a developer box, got 456 stranded of 900 (50.7%) — a near-exact match for the windows-2025 signature — and wrote it up as a clean reproduction.
Four repeats of the same command on the same box then produced 0 stranded, every time. The outlier was taken while an unrelated test suite was running.
So stranding here is a contention artifact, not a clean function of offered rate, and n=1 is not a measurement. Hence
--repeat, defaulted to 3, with the correction written into the module docstring so the next reader doesn't redo it.What survives: an unloaded box strands zero at up to 10× the CI profile's rate, while windows-2025 stranded ~51% at the profile's own 60/s, twice, byte-identically. That's weaker than what I first claimed and still worth measuring properly.
Deliberately not a gate
No
pull_requesttrigger, so it can never report on a PR or become the required-but-absent trap. And it is structurally unfit to gate anything: the probe exits 0 even when the reconcile fails, because a machine too slow to keep up is the finding, not an error. Recorded in.github/required-contexts.txtalongside the other advisory workflows.It also prints no derived per-second figure.
engine_read / walllooks like a service rate and isn't one —wallincludes the stop grace, drain and settle-poll, so it lands at ~25/s whether the run offered 60/s or 600/s. I had that in a draft and removed it; reporting offered/ingested/stranded and letting the reader compare rows is honest, a fabricated rate is not.What it still cannot answer
Both legs are hosted VMs. A slow-2025 / fast-2022 result narrows the cause to the 2025 image — it cannot clear or convict Windows Server 2025 as a deployment target, because a hosted image can be slow for reasons a deployed server isn't (Defender policy, VM disk backing, noisy neighbours).
Only the self-hosted WS2025 rig settles that, and it is unregistered —
actions/runnersreturnstotal_count: 0andselfhosted-win2025-sql.ymlhas never run. That remains the decisive experiment, and it needs the VM brought up.Verification
ruff check+ruff formatclean; pre-commit passed including the Actions linter.pull_requesttrigger.--repeat.