ci(asvs): wire --prove-absences to something, and report what it scanned - #304
Open
wshallwshall wants to merge 9 commits into
Open
ci(asvs): wire --prove-absences to something, and report what it scanned#304wshallwshall wants to merge 9 commits into
wshallwshall wants to merge 9 commits into
Conversation
…ort what it scanned
`scripts/asvs/scorecard.py --prove-absences` shipped on 2026-08-07 and ran in NO
workflow in either repo. Measured 2026-08-09 against vault origin/main 1a59e4a1 and
engine main:
absence claims on the record : 276
carrying `observable` : 0
carrying `mutation_path` : 0
--prove-absences invoked in CI : 0 references under .github/, in EITHER repo
A mode nothing invokes cannot go red whatever is inside it, so those 276 green absence
claims were exactly as strong the day after that merge as the day before. This is the
invocation, and it is deliberately the first half of a two-step: wire it while adoption
is zero, harden it after. Follow-up named in BACKLOG #1006's own closing banner.
WHAT LANDS
* .github/workflows/asvs-prove-absences.yml -- a scheduled advisory job plus a
`selftest` job that gates it. Not a required context, no write scopes, both knobs
(`--strict`, `--detail`) default off.
* scripts/asvs/prove_report.py -- census the claim population, run the prover,
reconcile the two, report. `scorecard.py` is untouched (it belongs to another
session this run).
* tests/test_asvs_prove_absences_wiring.py -- 14 tests pinning the advisory
guarantees and running the selftest harness in the suite.
THE INPUT PROBLEM IS STATED, NOT ENGINEERED AROUND
The scorecard lives in a PRIVATE repo and this one is PUBLIC. The vault reads the
engine for free ("public: no token needed"); the reverse has no free version. The
credential path is implemented and OFF -- neither `vars.ASVS_VAULT_REPO` nor
`secrets.ASVS_VAULT_READ_TOKEN` exists, and this commit does not create them, because
the vault exists so that a public-repo compromise does not yield the security corpus.
The recommended alternative is in the workflow header: run the prover in the vault as a
NEW workflow beside asvs-scorecard.yml, which needs no new credential in either
direction. The same two scripts run there unchanged.
The output points the same way. The prover's problem lines name the cell and the
control that would not prove -- a ranked list of the weakest controls on the record --
so they are suppressed by default and only counts are printed.
WHY A NO-INPUT RUN FAILS RATHER THAN SKIPS
Advisory applies to FINDINGS, never to the INSTRUMENT. A claim that will not prove is
reported and does not fail the job. A run that could not obtain a scorecard scanned zero
claims and is not evidence about any of them, so it exits 2 -- the rule scorecard.py
already states for its own loader. Until an input is configured the scheduled job is red
daily, which is the finding rendered as a signal. Disable the workflow if that is
unwanted; do not make the no-input path green.
PROVED BY MAKING IT FAIL
`prove_report.py selftest` drives nine limbs: a biting claim proves (L1), a non-biting
claim is reported but not fatal in advisory mode (L2), the SAME claim fails under
--strict (L3, so advisory is a choice rather than the only behaviour), a missing
scorecard is an instrument failure (L4), the census sees the claim (L5), detail
suppression is attacked from both sides (L6), an unparseable prover summary is an
instrument failure rather than a report of zeros (L7), and a prover that stopped
iterating is caught by reconciliation (L8 -- its own summary line looks identical
whether it walked 276 claims or 2).
Each limb was then confirmed to go RED by injecting the matching defect, checking the
defect landed on disk first, and restoring: reconcile made a no-op (L8 red), suppression
removed (L6 red), the missing-scorecard branch returning 0 (L4 red), --strict ignored
(L3 red), an unparseable summary reported as zeros (L7 red). A first attempt at the
last one rewrote the regex and made it STRICTER, not looser -- the text landed and the
behaviour did not move, which is the "confirm the injected defect actually landed" trap
in miniature. The harness reported MISSED, correctly, and the sabotage was rewritten.
Verified: ruff format + check, mypy --strict on the new script, actionlint (pinned,
-shellcheck=), zizmor 1.29.0 (clean, and the whole workflows tree still clean), the leak
gate, and 110 tests across test_asvs_scorecard.py, this file, test_required_contexts.py
and test_quality_advisory_invariants.py.
…g leaves open `--prove-absences` now runs daily, but its input lives in the private vault while the job runs in the public engine repo. Three options, each with what it actually costs: the engine holds a vault-read credential (implemented, OFF -- a security-boundary cost); the prover runs in the vault as a new workflow beside asvs-scorecard.yml (no new credential in either direction -- the recommendation); or a self-hosted runner. The output points the same way as the input: the prover's problem lines rank the weakest controls on the record, so they are suppressed by default in a public log. The public repo is the wrong holder of the input and the wrong host for the output, and only option 2's cost is one that moving the environment can pay. Records that the no-input path must not be made green -- advisory applies to findings, never to the instrument -- and that #1006's "~81 existing absence claims" is stale: the record carries 276, measured against vault origin/main 1a59e4a1. Number allocated with scripts/coord/alloc.ps1, never grepped.
Committed at 29da4f4 as this stream's work order. It is not on main and should not land there: it names other streams, their file ownership and their sequencing, none of which is true of the repository once the work is merged. It remains readable in this branch's history at that commit. The substance that outlives the session is already in the tree -- the input-credential finding and its three options are in the header of .github/workflows/asvs-prove-absences.yml and in BACKLOG #1203.
… red A contributor who sees a red `prove absence claims (advisory)` on the Actions tab needs to know, without reading the workflow, that it is not their PR's problem and cannot block a merge. The daily red is the finding it carries: the scorecard is in a private repo, no input is configured, and a run that scanned nothing must not report success. BACKLOG #1203 is the decision that clears it. The table is a curated list of the notable workflows (8 rows against 23 files) and makes no completeness claim, so this is an addition on merit rather than an enumeration being kept whole.
… to that line `_run_prove_absences` prints proved / static-screened / skipped / problems and no TOTAL, and those four do not close to the claim count -- a static-screened claim can also raise a SUSPECT problem, and four other outcomes raise a problem while incrementing no counter. So a prover that stopped iterating prints the same shape as one that walked every claim. The right fix is in `scorecard.py` (a `saw N absence claim(s);` term), which belongs to another session this run. Prepending it there would have broken this parser and turned a helpful change into a red gate for a reason unrelated to it. `.*?` after the prefix absorbs that insertion. Tolerance kept narrow on purpose, and measured rather than asserted: the line as it prints today parses, the line with a `saw N` prefix parses, and both `prove-absences ran, trust me` and a truncated near-miss still return None. Selftest limb L7 pins the last of those -- an unparseable summary must be an instrument failure, never a report of zeros. All five sabotage injections re-run and still caught.
…g reason The proving loop counts `mutated == 1` as "the control bit". That is sound only if the observable went red because of the SEMANTIC change the claim describes. Application is append-based, so a reintroduction bites by redefinition shadowing -- and two mutations that shadow nothing semantic redden the observable at exit 1, indistinguishable from a surgical proof to every check that existed before this change. Both holes were MEASURED with the screen neutered, not reasoned about. Both reported `proved=1, problems=0` -- a clean green false proof: * WRONG ARITY. `def scan(p)` reintroduced as `def scan()` raises TypeError at every call site. The claim proves that calling a function with the wrong number of arguments breaks it, which is true of every function in the repository. Not hypothetical: the mutation is authored in a TOML file in a DIFFERENT REPOSITORY from the signature it copies, with nothing keeping the two in step. * A MUTATION THAT DOES NOT PARSE. Appending invalid Python breaks import of the target. A module-scope import in the observable surfaces as a collection error (exit 2) and the shipped fail-closed branch already caught it; an import inside the test body surfaces as an ordinary failure at exit 1 and was counted as a proof. That difference is why the screen is static rather than left to exit codes. `_signature` compares parameter NAMES, not only counts -- a rename breaks every keyword call, the same failure and invisible to an arity-only check. Defaults compare by COUNT and never by value: a mutation legitimately changes what a default IS. NOT IMPLEMENTED, DELIBERATELY: a blanket refusal of `raise` in a mutation. That rule belongs to a schema of typed mutation kinds where an "ablate" limb weakens a control and must never throw. No such schema exists in this repository -- there are no mutation kinds, every mutation is a REINTRODUCTION, and a reintroduction that raises is an anticipated shape: the static backstop exists precisely to flag one landing in a swallowing handler, with two tests pinning it. Banning `raise` would delete the case the backstop was written for. EFFICIENCY, same change One pristine copy for the whole pass with save/apply/run/restore per claim, replacing a whole-tree copytree per claim (~1.2s each). Baselines cached by observable node id -- the baseline is a property of the pristine tree and the node, so re-running it per claim re-measured a constant at one pytest subprocess each. Reusing one tree is only sound while it stays pristine, so the reuse is CHECKED: the restore is verified byte-wise, and a stat-only inventory sweep after each claim rebuilds the tree and drops cached baselines if anything else moved. Residue from claim N would otherwise be attributed to claim N+1's mutation, which is the protection the per-claim copy was buying. Honest residual: the inventory cannot see a same-length in-place rewrite. `saw N absence claim(s);` joins the summary line. The other four counters do not close to the total -- a static-screened claim can also raise a SUSPECT problem, and five outcomes raise a problem while incrementing no counter -- so a pass that stopped iterating after two claims printed the same SHAPE as one that walked all 276. The total is computed from `cells`, independent of anything the loop did. PROVED BY MAKING EACH TEST FAIL Seven injections, each confirmed on disk before the run, each caught by the named test and no other: screen disabled (the two refusal tests red, the honest-control test HELD green), screen refusing everything (the honest control red), signature reduced to counts, per-claim copy restored, baseline cache dropped, restore deleted with the two layers behind it held constant so the test pins the restore rather than defence in depth, and the saw-N term dropped. 67 tests in test_asvs_scorecard.py (was 60), 142 across the ASVS suite, ruff, mypy --strict on both scripts. prove_report.py parses the new summary line unchanged -- its `.*?` tolerance was written for exactly this insertion.
…ose ratchet Two repo-wide guards reddened #304. Both are real defects in my additions, and both live in a class of check that the scoped test selection I reported on cannot reach -- which is the actual finding here and is recorded at the bottom. 1. CRYPTO DISCOVERY (ASVS 11.1.3), three symptoms one cause `scripts/asvs/prove_report.py` calls `hashlib.sha256` and was in none of the three inventories, so `tests/test_security_static.py::test_crypto_roots_carry_no_unrecorded_call_site`, `::test_crypto_inventory_gate_clean_on_real_tree` and the standalone gate all failed. Registered, not suppressed. The call is SHA-256 over the SCORECARD FILE, printed truncated to 16 hex characters so a run states WHICH revision of the record it read -- two runs reporting different counts are otherwise indistinguishable from one run whose input moved underneath it. No secret, no key, no message authentication, nothing user- or PHI-derived. The precedent sits one line away: `scripts/asvs/scorecard.py` is registered for the corpus pin under ADR 0156. This differs only in what it covers (the record rather than a build input) and in that nothing is gated on the value. All three places updated, per the gate's own instruction to add it to "INVENTORY + ASVS section 4": - scripts/security/crypto_inventory_check.py (the machine-readable inventory) - tests/test_security_static.py (the outside-the-package pin) - docs/ASVS-L2-PHASE0-CHANGES.md section 4 (the human-readable companion) Note the gate is BIDIRECTIONAL by design: registering a file is a standing commitment, and dropping the import later fails the other way. 2. PROSE RATCHET -- the text moved the number, so the text is what changes `tests/test_cutover_slug_rot.py` counts present-tense mirror/private-repo prose and may only ever fall. Measured with the guard's OWN regexes and file selection: before 58 hits across 1534 files (ceiling 54) mine 4 pre-existing elsewhere 54 <- exactly the ceiling after 54 hits <- baseline restored, not lowered The ceiling is not wrong and was not touched. Every excess hit was mine, in four lines that say "a private repo" where they mean THE VAULT. That phrase is loaded in this codebase -- it is what pre-cutover prose used for the retired private source repo -- so naming the vault is more precise for a human reader as well as correct for the guard. No pre-existing hit was altered: fixing someone else's triage while fixing my own would have made the count fall for two reasons and told nobody which. 3. THE ROOT CAUSE, which is neither of the above I reported "67 tests in test_asvs_scorecard.py, 142 across the ASVS suite" -- a SCOPED selection. Both failures live in repo-wide guards that no scoped selection can reach by construction: a crypto-call-site walk over five roots, and a prose count over 1,534 tracked files. Neither has anything to do with the module under test; both are about the tree. This is not two unlucky outliers. Measured: 74 of 614 test modules (12%) walk the repo root or `git ls-files` rather than their own fixtures. A scoped run silently skips that entire class, and a scoped run reporting a count reads exactly like a full one. Standing rule taken from this: report "clean" only after the full suite, or state which subset ran AND that repo-wide guards were not exercised. A count without its scope is one word short of a claim.
wshallwshall
force-pushed
the
asvs-assurance
branch
from
August 9, 2026 12:16
42e74e4 to
3fa2960
Compare
…cron here
Location decision, 2026-08-09: option 2. The scorecard lives in the vault, so the
scheduled pass runs there. Two reasons, and the second is the one that is easy to miss:
a vault-read token in the public repo's secret store collapses the separation the vault
exists to provide, and the prover's problem lines are a ranked list of the weakest
controls on the record -- the disclosure that got the verdict-attributed anchor manifest
rejected in the 2026-08-08 tracking-rework diagnosis. Public repo is the wrong holder of
the input AND the wrong host for the output; only the environment cost is one that
moving the job can pay.
WHAT MOVES AND WHAT STAYS
prove_report.py stays here and is mirrored into the vault, same ADR 0156 section 7
footing as scorecard.py: one tool, developed in the repo whose code
it constrains, run in the repo that holds the data.
the selftest job stays here. No credential, runs on any PR touching the wiring, and
it is what stops the tool rotting in the repo that develops it.
the schedule goes. The `prove` job is now `workflow_dispatch` only.
The vault-side workflow is NOT built from this branch -- different repo, sequenced
separately.
WHY DROP THE CRON RATHER THAN LEAVE IT RED
Two questions that a `schedule:` here would have conflated. The job still exits 2 when
it obtains no scorecard, and that is unchanged and must stay: advisory applies to
findings, never to the instrument, and a run that scanned zero claims is not evidence
about any of them. But a job that fails closed on no input must not be SCHEDULED to
obtain no input. With the credential path off, a nightly run here would be red every day
by construction -- not reporting a finding, just re-announcing a decision already
recorded in the file. A gate whose first act is to fail is one somebody switches off, and
a disabled workflow reads the same as a passing one at a glance.
The selftest job takes no cron either, and that is a measurement rather than an
oversight: its harness also runs as
tests/test_asvs_prove_absences_wiring.py::test_selftest_all_limbs_pass in the unfiltered
ci.yml suite on every code PR and push, so a nightly re-run here would re-measure
something already measured.
PROVED BY MAKING IT FAIL
The decision is now a property of TWO places that can drift apart -- the trigger and the
job's `if:` -- so each has its own assertion and each was injected separately:
re-adding a cron reddens only test_the_workflow_has_no_schedule_trigger; widening the
job gate back to `schedule` reddens only test_the_prove_job_is_dispatch_only. Each
injection confirmed on disk first, and in each case the other test HELD green, which is
what shows the two are independently pinned rather than one assertion counted twice.
The `if:` keeps an explicit event test rather than being deleted, so re-adding a cron
cannot silently start running the job: someone has to change that line too, and that
line sits next to the reason not to.
docs/CI.md and BACKLOG #1203 updated to match. #1203 stays OPEN -- the decision is made,
the vault-side build is not.
Full suite: 1 failed, 11080 passed, 851 skipped (26:10). The one failure is
test_gate_installed_parity, a local-machine test that skips on CI, pre-existing, not
caused by this branch, and already with the owner.
wshallwshall
marked this pull request as ready for review
August 9, 2026 14:01
wshallwshall
enabled auto-merge (squash)
August 9, 2026 14:01
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft, and it needs an owner decision before it is useful. See "The open question" below.
The problem
prove_absencesshipped 2026-08-07. It copies the tree to a scratch dir, asserts a named observable is green, applies a stated mutation, and requires the observable to go red. That is real mutation testing of a control.It is wired to nothing. Measured against vault
origin/main1a59e4a1:So those 276 green absence claims are today exactly as strong as they were before that merge landed. Wire it before hardening it -- making it fail on zero adoption is correct and completely inert until something runs it.
What this adds
.github/workflows/asvs-prove-absences.yml--selftest(runs on any PR touching the wiring, no credential) andprove(needs: selftest, scheduled + dispatch only).permissions: {}at file level,contents: readper job, actions SHA-pinned, no${{ }}interpolation in any run body.scripts/asvs/prove_report.py-- censuses the claims from rawtomllib, deliberately not viaload_scorecard, because a total derived from the prover's own loader agrees with it by construction. Then reconciles the prover's counters against that independent census.tests/test_asvs_prove_absences_wiring.py-- 14 tests pinning no-write-scopes, not-a-required-context, the sparse checkout, and the advisory default.docs/CI.mdwarns contributors the scheduled job is red and cannot block them.Real run today:
276 seen / 0 provable / 276 skipped / reconciliation OK / adoption 0.0%, exit 0 in 0.23s.The open question, and it is the owner's
The scorecard lives in a private repo; this one is public. The vault reads this repo for free; the reverse has no free version.
asvs-scorecard.yml, leaving that job's stdlib-only 5-minute constraint untouched. No new credential in either direction. Cost is one install against a lockfile the vault does not own. Recommended.The output points the same way, and it is easy to miss. The prover's problem lines name the cell and the control that would not prove -- a ranked list of the weakest controls, which is the disclosure that got the anchor manifest rejected on 2026-08-08. They are suppressed by default here; only counts print. So the public repo is the wrong holder of the input and the wrong host for the output, and only option 2 pays its cost by moving the environment rather than by moving a secret.
Consequence, stated deliberately: the scheduled job is RED daily until this is decided. Advisory applies to findings, never to the instrument -- a run that obtained no input scanned zero claims and exits 2. Disable the workflow if the noise is unwanted; do not make that path green.
What was proven by making it fail
Nine selftest limbs, then five defect injections, each confirmed present on disk before the run and restored after:
reconcilemade a no-op--strictignoredThe first sabotage attempt rewrote a regex and made it stricter rather than looser: the text landed, the behaviour did not move, and the harness correctly reported MISSED. That is the confirm-the-defect-landed trap in miniature, and it is recorded in the commit rather than quietly fixed.
Not in scope here
scripts/asvs/scorecard.pyis untouched -- another stream owns it. One patch is proposed for that owner rather than taken:_run_prove_absencesprints no total, and its four counters do not close to the claim count (static_screenedcan co-occur with a SUSPECT problem, and four other outcomes raise a problem with no counter). Adding asaw N absence claim(s);term would close it; the parser here already tolerates that exact insertion.Separately, BACKLOG #1006's closing banner says "~81 existing absence claims". The record carries 276. Recorded in #1203 so the stale figure is not quoted onward.