Skip to content

fix(ci+test): the DAST closure guard matched inside INCOMPLETE, and the doc-guard list omitted it - #326

Merged
wshallwshall merged 4 commits into
mainfrom
w3-fix-dast-guard
Aug 11, 2026
Merged

fix(ci+test): the DAST closure guard matched inside INCOMPLETE, and the doc-guard list omitted it#326
wshallwshall merged 4 commits into
mainfrom
w3-fix-dast-guard

Conversation

@wshallwshall

@wshallwshall wshallwshall commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator

main is currently red. This is the fix, plus the reason it will not recur. Three defects, each with
its own fix
-- none alone is the whole answer.

1. The guard read a negation as an affirmation

The closure-claim pattern had no word boundary before the verb group:

(pen ?test|penetration test|independen\w*).{0,60}(complete|satisfied|closed|covered|discharg)

complete therefore matches inside INCOMPLETE. A ledger sentence asserting the exact opposite
of a closure tripped a guard whose purpose is to catch closure claims. The same hole accepts
uncovered, unsatisfied and undisclosed -- each a word whose meaning is the negation of the verb
being hunted. Adding \b loses no true positive, because a real closure claim spells the verb as its
own word.

A regression test pins the negation mode using the real failing sentence verbatim -- a fixture
invented to match the fix is not evidence the fix covers the fault.

2. The prose collided with a term of art

Separate from the regex: in a document where independent verification is a term of art, "independent
of the gate" is ambiguous to a human reader, not merely to a matcher. Reworded to "separately from
the gate question".

Doing only the reword leaves a guard that fires on the next person who writes "independent ...
incomplete". Doing only the guard fix leaves a sentence that reads wrong to a careful reader.

The reword is belt-and-braces, not load-bearing, and that is measured so nobody later concludes the
guard fix was unnecessary: the fixed guard does not match the original sentence, still does
match a real closure claim, and a full run with the original wording gives 29 passed.

3. How it reached main -- the list, not the gating

The first diagnosis of this was wrong, and it would have produced the wrong fix. It was recorded as a
missed local step. It was not.

The pre-merge doc-guard step already existed at ci.yml:188, already ungated on docs-only PRs, with
its own minimal [dev] install. It simply did not name tests/test_dast_claims.py.

That is also the entire explanation for an asymmetry that otherwise looks like luck: test_link_resolution
is in the list, so PR #324's link failure was caught pre-merge; the DAST guard was not, so it
reached main. Same blind mode, two outcomes, decided purely by list membership.

The blind mode is worse than a plain gap because of who pays: the failure surfaces on the next PR
that touches code, so it is misattributed to whoever opens it. That happened -- #325 inherited a red its
author had no part in.

A curated allowlist silently omits, and nothing in a green run says "a doc guard exists that I did not
run."
The step now states the rule in place: when adding a doc-scanning module anywhere in tests/,
add it here in the same commit, and confirm it runs on [dev] with no extras.

Two supporting changes, both about the step being able to lie:

  • The list was written twice -- once for the printf, once for pytest -- so the step's own "print
    what you scanned" defence could drift and print a module it did not run, or run one it did not print.
    A defence implemented by duplication defeats itself. One DOC_GUARDS variable now feeds both.
  • Every named module must exist. A path typo would otherwise error on an unknown file, or -- under a
    future -k/--ignore form -- silently scan nothing and read as a pass.

The list is not claimed complete. A grep for doc-ish test modules over-counts (it flags
test_push_guard and test_required_contexts, which are not doc guards), and separating the rest needs
reading. An honest partial plus a stated rule beats a number nobody trusts.

Scope and verification

Touches tests/test_dast_claims.py, .github/workflows/ci.yml, and one docs/BACKLOG.md sentence. No
product code.

The proof is unplanted. The new list was run against pristine main, which still carries the real
defect: 1 failed, 209 passed. The guard caught the live fault rather than an invented fixture.

Mutation control run twice. The first attempt armed the mutation through a heredoc, which collapsed the
backslash in the boundary token; the replace matched nothing and the tests "passed" over unmutated code
-- indistinguishable from a surviving mutation. An occurrence assertion refused the no-op rather than
reporting a pass. Re-armed from a written file: boundary removed, the regression test fails; boundary
restored, 18 passed. test_dast_claims and test_link_resolution both pass locally: 29 passed. YAML
parses and the doc-guard step is intact with the list appearing exactly once.

…nd reddened main

origin/main is RED on
test_dast_claims.py::test_no_file_claims_dast_closes_the_independent_gap.
The trigger is prose I wrote in the #1008 amendment:

  docs/BACKLOG.md:3739
  'independent of the gate:** the original WIP was **INCOMPLETE'

That sentence asserts the OPPOSITE of a closure. The guard read it as a
claim that the independent-verification gap is closed.

MECHANISM, measured rather than inferred. The pattern was

  (pen ?test|penetration test|independen\w*).{0,60}(complete|satisfied|closed|covered|discharg)

with NO word boundary before the verb, so 'complete' matches INSIDE
'INCOMPLETE'. The same hole accepts 'uncovered', 'unsatisfied' and
'undisclosed' -- every one a word whose meaning is the NEGATION of the
verb being hunted. Adding \b loses no true positive, because a real
closure claim spells the verb as its own word. Verified both directions
against the live sentence before the change landed.

I FIXED THE GUARD RATHER THAN THE PROSE, DELIBERATELY. Rewording my
sentence would have turned main green while leaving a guard that fires on
the next person who writes 'independent ... incomplete' -- and it would
have destroyed the only live specimen of the fault. The sentence is
correct English and correct on the facts; the pattern was wrong.

THE PRE-EXISTING NEGATIVE CONTROL COULD NOT HAVE CAUGHT THIS. The file
already asserted that 'the independent engagement has not been performed'
does not match -- but that passes because 'performed' is not in the verb
list at all, not because the pattern understands negation. A guard needs
a negative control PER FAILURE MODE, not one per test. The new test pins
the negation mode with the REAL failing sentence kept verbatim, because a
fixture invented to match the fix is not evidence that the fix covers the
fault.

VERIFIED, and the control had to be run twice because the first attempt
was vacuous. Arming it through a bash heredoc collapsed the backslash in
the word-boundary token, so the mutation never applied and the tests
'passed' over unmutated code -- indistinguishable from a surviving
mutation. My occurrence assert refused the no-op rather than reporting a
pass, which is exactly why that assert exists. Re-armed from a written
file:

  boundary REMOVED  -> test_the_closure_pattern_does_not_read_a_negation_as_a_claim FAILS
  boundary RESTORED -> 18 passed

This branch carries main's docs/BACKLOG.md unchanged, so 18/18 green here
IS the proof that main goes green. Test-only change; no product code
touched.
…ext to a term of art

Belt-and-braces, NOT the load-bearing fix. The guard fix in the previous
commit already turns main green on its own: 29 passed across
test_dast_claims.py and test_link_resolution.py with this wording still
in place, and the fixed pattern does not match the original sentence
while still matching a real closure claim. Both measured just now.

The reword stands on a DIFFERENT reason, and it is the coordinator's and
it is fair: in a document where 'independent verification' is a TERM OF
ART, writing 'independent of the gate' to mean 'separately from the gate
question' is ambiguous to a human reader, not merely to a regex. A
reader who knows the term will parse it the wrong way for a beat.

So two independent defects, two independent fixes:
  - the guard read a NEGATION as an AFFIRMATION      -> fixed in the pattern
  - the prose collided with a term of art in context -> fixed here

Doing only the reword would have left a guard that fires on the next
person who writes 'independent ... incomplete'. Doing only the guard fix
would have left a sentence that reads wrong to a careful human. Neither
alone is the whole answer, which is why the earlier framing -- 'fix the
guard, not the prose' -- was half right rather than right.

Verified locally on this branch BEFORE handover, which is the practice
this incident exists to establish: test_dast_claims.py and
test_link_resolution.py both green, ledger gates clean, 484 items each
declaring exactly one status.
…itten twice

The docs-only blind mode recurred on 2026-08-11. THE STEP WAS ALREADY
THERE; its list was simply incomplete.

tests/test_dast_claims.py scans documents for prose reading as though the
independent-verification gap were closed. It was never in the list, so a
docs-only PR (#322) merged green and RED MAIN on the push afterwards.

THE PROOF IS UNPLANTED, WHICH IS THE BEST KIND. This branch is cut from
pristine origin/main, which still carries the real defect. Running the
new list here reproduces it:

  FAILED tests/test_dast_claims.py::test_no_file_claims_dast_closes_the_independent_gap
  1 failed, 209 passed, 89 skipped

So had this module been listed, #322 would have failed PRE-MERGE instead
of reddening main. I did not have to invent a fixture; the fault was
sitting on main and the step catches it. (The 89 skips are the documented
structural vault-absence class, not a gap.)

SECOND DEFECT, found while fixing the first: THE LIST WAS WRITTEN TWICE
-- once for the printf and once for pytest -- so the step's own 'print
what you scanned' defence COULD LIE. The two copies could drift and then
it prints a module it does not run, or runs one it does not print. A
defence implemented by duplication defeats itself. Now one variable used
twice, plus an existence check that fails loudly on a path typo and a
printed module count.

WHY THE BLIND MODE IS WORSE THAN A PLAIN GAP, recorded in the step: the
failure surfaces on the next PR that touches code, so it is
MISATTRIBUTED to whoever opens it. That happened -- the next code PR
inherited a red its author had no part in.

AND THE LESSON IS ABOUT THE LIST, NOT THE GATING. A curated allowlist
silently omits, and nothing in a green run says 'a doc guard exists that
I did not run.' The step now says: when you add a doc-scanning module
anywhere in tests/, add it HERE in the same commit, and confirm it needs
no extras -- a module that cannot run on [dev] alone would red every
docs-only PR, the failure the minimal-install note exists to avoid.

I checked test_dast_claims against that rule before adding it: its only
imports are re, pathlib, pytest and scripts.security.dast_auth_sweep,
whose own top-level imports are httpx and messagefoundry.auth -- all base
install, no extras.

SCOPE STATED HONESTLY: a loose scan suggests other doc-reading modules
may also be missing, but that scan matches any test mentioning 'docs' and
over-counts badly (it flags test_push_guard and test_required_contexts,
which are not doc guards). I am NOT claiming the list is now complete --
only that the one module with a demonstrated live failure is in it, and
that the next addition has a stated rule to follow. Triaging the rest
needs reading, not a grep.

Carries the guard fix from w3-fix-dast-guard so this branch is green in
either landing order. Verified: 29 passed across test_dast_claims.py and
test_link_resolution.py; ci.yml parses and the doc-guard list now appears
exactly once.
U+21D2 RIGHTWARDS DOUBLE ARROW raises UnicodeEncodeError on a stock
Windows cp1252 console, which is the stated reason in CLAUDE.md section
11. It is a CI comment people read from terminals. Replaced with '=>'.

MEASURED BEFORE CHANGING ANYTHING ELSE, because the report also named the
em dashes and the section sign and those are a different case:

  U+2014 EM DASH      encodes FINE in cp1252
  U+00A7 SECTION SIGN encodes FINE in cp1252
  U+21D2 ARROW        raises UnicodeEncodeError   <- the only real one

And on origin/main this file ALREADY carries 134 non-ASCII characters,
127 of them em dashes. Stripping 7 more from my added lines while leaving
127 in place would change nothing a terminal can observe and would make
the file internally inconsistent. Section 11's subject is GLYPHS AND
EMOJI -- the banner alphabet and pictographs -- and its own text uses
'section' markers throughout; an em dash is punctuation, not a glyph.

So: the arrow goes, the em dashes stay, and the reason for each is the
same single test rather than a blanket rule applied by eye.

Added lines now carry ZERO cp1252-unsafe characters, verified by
encoding every non-ASCII character in the diff's added lines rather than
by reading them.

SEPARATELY, AND NOT MINE TO FIX HERE: origin/main's ci.yml already
carries THREE U+2192 arrows, which are cp1252-unsafe by the same test
that condemned mine. Pre-existing, unrelated to this change, and reported
rather than silently swept in -- a drive-by edit to unrelated lines would
make this diff harder to review for no gain.

Verified after the change: yaml parses, 29 passed across
test_dast_claims.py and test_link_resolution.py.
@wshallwshall wshallwshall changed the title fix(test): the DAST closure guard matched inside INCOMPLETE and reddened main fix(ci+test): the DAST closure guard matched inside INCOMPLETE, and the doc-guard list omitted it Aug 11, 2026
@wshallwshall
wshallwshall enabled auto-merge August 11, 2026 15:40
@wshallwshall
wshallwshall merged commit 48f8712 into main Aug 11, 2026
38 checks passed
@wshallwshall
wshallwshall deleted the w3-fix-dast-guard branch August 11, 2026 16:22
wshallwshall added a commit that referenced this pull request Aug 11, 2026
…l merge

Session A's resolve and Session A's measurements. Committed by Session B only
because of an ownership constraint, stated here so the next reader does not
wonder why the sole writer did not write it: BACKLOG numbers 1202, 1204 and
1205 are allocated to THIS worktree
(C:/Users/Scott/Code/MessageFoundry-asvs-ci-hygiene), and ledger_check's
owns() compares the claim's `worktree` field casefolded against the repo path
and never consults the branch -- so the pre-commit gate refuses this commit
from any other path, including from A's own worktree. Nobody used --no-verify.

WHY THIS IS NOT A LINE-LEVEL MERGE, which is the thing a future reader most
needs so they do not "simplify" it back. Measured on this branch against
origin/main 48f8712:

                            live   archive   namespace   in BOTH files
  origin/main                248     236        484           --
  this worktree (42050cd)   281     195        476            0
  MECHANICAL keep-both-sides 289     236        525           38   <-- the trap
  this commit                251     236        487            0

41 items differ between the branch and main. THIRTY-EIGHT OF THEM ARE ITEMS
MAIN HAS SINCE ARCHIVED. A keep-both-sides merge resurrects every one into the
live ledger while leaving it in the archive: duplicated across both files, the
namespace inflated 484 -> 525, and each of those 38 reading OPEN again after
being closed. GIT RAISES NO CONFLICT MARKER, because both sides added lines
near the tail. Only THREE of the 41 are genuinely new: 1202, 1204, 1205.

So both files are taken from main WHOLESALE and exactly those three are
spliced back, before #1208 (item headings ascend even though the file as a
whole does not). A's script asserts its own result: +3 headings exactly, none
lost, no duplicates.

THE ARCHIVE HALF IS DELIBERATE AND EXPLICIT, not incidental. The branch had
touched BACKLOG-CLOSED.md zero times (main-only 41, branch-only 0), so taking
main's copy loses nothing and is what a merge would do anyway -- but doing it
as a stated step is what makes the branch self-consistent BEFORE it lands.
Without it the tree reads 251/195/446 and a pre-merge namespace check fails
against a resolve that is actually correct: a false red aimed at the right
answer, which is the kind that gets a correct fix reverted.

Re-derived from origin/main at 48f8712 (post-#326) rather than replaying a
patch cut against ea75c37, because main moved underneath it.

One number corrected in passing: the mechanical figure was first circulated as
286/522. It is 289/525 -- the smaller version credited the mechanical merge
with telling the 3 good items from the 38 bad ones, which is precisely the one
thing "mechanical" means it cannot do, and it erred in the direction that
understates the damage. Verified here independently rather than quoted.

Expect test_link_resolution to red branch-locally: main's ledger cites a test
file main has and this older branch does not. Not in the spliced items; A
simulated the merge and the file is in the merged tree, so it resolves on
landing. Taking a newer ledger onto a stale tree necessarily imports
references the tree cannot satisfy yet.

Commit only -- deliberately not pushed. This worktree's HEAD is two commits
ahead of PR #302's head and those two are on no remote, so a push from here
would put them on the PR. That is the Coordinator's call, not a side effect of
this commit. They are anchored at refs/rescue/unpushed/asvs-ci-hygiene-42050cd4
and 8391699's content is already on origin/main.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant