Skip to content

test(redaction): calibrate the perf growth sizes so the assertions cannot fail on a fast box - #22

Merged
andrei-hasna merged 1 commit into
mainfrom
fix-runlength-calibration
Aug 1, 2026
Merged

test(redaction): calibrate the perf growth sizes so the assertions cannot fail on a fast box#22
andrei-hasna merged 1 commit into
mainfrom
fix-runlength-calibration

Conversation

@andrei-hasna

@andrei-hasna andrei-hasna commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

CI on the release PR (#21) caught a defect I shipped in #20, and it is the same defect class that PR was written to fix, pointing the other way.

the cookie rule stays linear as a single unbroken RUN grows failed on the GitHub runner while the implementation is perfectly linear. Fixed at 8/16/32 KiB, a fast quiet runner finishes those sizes in a fraction of a millisecond, and at that scale timer resolution and JIT noise dominate the ratio.

#20 replaced a perf test that could not FAIL with one that could not reliably PASS. A test that fails when nothing is wrong is exactly as useless as one that passes when something is — and worse in one respect: it trains people to re-run CI until it goes green, which is how a real failure gets waved through.

The fix

Calibrate the smallest size to the machine rather than fixing it, then measure across three doublings from there. The two cases separate themselves with no change to the 2.8 threshold, which is the part that must not move:

  • a linear implementation is fast, so calibration keeps doubling and ends up measuring where the numbers are solid;
  • a quadratic one is already far above the floor at the smallest size, so calibration stops immediately and the assertion fails at small sizes and therefore fast — a test that can only fail by timing out reports its budget rather than a duration.

Applied to growthPerDoubling as well, because it is the same family and the same flake: the pre-existing Digest control failed 1 run in 8 here at loadavg 25, and an adversarial reviewer measured it at 2 in 10 before the estimator changed.

Verified in both directions

fix     10 of 10 consecutive full-suite runs green, loadavg 24.5
mutant  ratio 4.012 — FAILS — in 30.1s

The mutant is redactCookiePairs replaced by the naive /([^\s;,=]+)=([^\s;,]*)/g that the rule was restructured to avoid. typecheck rc=0.

Test-only change — src/ is untouched, so no redaction behaviour moves.

Why this is blocking something

@hasna/tai on the registry is 0.1.4, which emits every session cookie it is asked to redact. Main carries the fixes for that (#19, #20) but is unpublished; the release PR #21 is blocked on this flake. Landing this unblocks the release.

Refs: todos 6200c4e4


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

…nnot fail on a fast box

CI caught a defect I shipped in #20: `the cookie rule stays linear as a single
unbroken RUN grows` FAILED on the GitHub runner while the implementation was
perfectly linear. Fixed at 8/16/32 KiB, a fast quiet runner finishes those sizes
in a fraction of a millisecond, and at that scale timer resolution and JIT noise
dominate the ratio.

THAT IS THE SAME DEFECT CLASS THE ASSERTION WAS WRITTEN TO FIX, POINTING THE
OTHER WAY. #20 replaced a perf test that could not FAIL with one that could not
reliably PASS. A test that fails when nothing is wrong is exactly as useless as
one that passes when something is, and it is worse in one respect: it trains
people to re-run CI until it goes green, which is how a real failure gets waved
through.

The fix is to calibrate the smallest size to the machine rather than fix it, then
measure across three doublings from there. The two cases separate themselves with
no change to the 2.8 threshold, which is the part that must not move:

  - a LINEAR implementation is fast, so calibration keeps doubling and ends up
    measuring where the numbers are solid;
  - a QUADRATIC one is already far above the floor at the smallest size, so
    calibration stops immediately and the assertion fails at small sizes and
    therefore FAST.

Applied to `growthPerDoubling` as well, because it is the same family and the
same flake. The pre-existing Digest control failed 1 run in 8 here at loadavg 25,
and an adversarial reviewer measured it at 2 in 10 before the estimator changed.

VERIFIED IN BOTH DIRECTIONS, which is the whole point of the change:

  fix     10 of 10 consecutive full-suite runs green at loadavg 24.5
  mutant  ratio 4.012, FAILS, in 30.1s

The mutant is `redactCookiePairs` replaced by the naive
`/([^\s;,=]+)=([^\s;,]*)/g` that this rule was restructured to avoid.

typecheck rc=0. Test-only change; `src/` is untouched.

Refs: todos 6200c4e4

Agent: aemilius
@andrei-hasna
andrei-hasna merged commit 2405fd9 into main Aug 1, 2026
2 checks passed
@andrei-hasna
andrei-hasna deleted the fix-runlength-calibration branch August 1, 2026 06:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant