diff --git a/.github/workflows/asvs-tally-lint.yml b/.github/workflows/asvs-tally-lint.yml new file mode 100644 index 00000000..8019ee66 --- /dev/null +++ b/.github/workflows/asvs-tally-lint.yml @@ -0,0 +1,74 @@ +name: ASVS tally lint + +# Refuses a NEW hard-coded ASVS verdict tally in a document (ADR 0156, BACKLOG #1203). +# +# WHY THIS IS A SEPARATE WORKFLOW RATHER THAN A pytest TEST, which is where a guard would normally +# live in this repo. The defect this catches is written into DOCUMENTS, so the pull request that +# introduces it is usually docs-only -- and on a docs-only pull request ci.yml's `changes` job sets +# `code=false` and the whole pytest suite is skipped. A guard reachable only through pytest would +# therefore not run on the exact shape it exists to police. ci.yml already carries two guards that +# were ungated for precisely this reason (the BACKLOG number-space gate and the status invariant), +# and the note above them records what the second one cost before it was. +# +# The natural home is that same ungated "Doc guards" step in ci.yml. It is not used here because +# ci.yml's `changes` job is under an open pull request (MEFORORG#299) and a second edit to those +# lines would conflict. FOLLOW-UP: once #299 lands, fold this into the doc-guards list and delete +# this file, so there is one place that decides which guards run on a docs-only pull request rather +# than two. +# +# BOTH PATHS ARE LIVE MEANWHILE, deliberately and not as belt-and-braces theatre: this workflow +# covers docs-only pull requests, and tests/test_asvs_tally_lint.py runs the same lint inside the +# required `test` legs for code pull requests. They cannot disagree -- both invoke the same module +# against the same frozen baseline. +# +# NOT A REQUIRED CHECK, and it must not become one without the owner's decision: it is paths- +# filtered, so it does not report on a pull request that touches no document, and a required check +# that never reports blocks that pull request forever (the required-but-absent trap, docs/CI.md). +# .github/required-contexts.txt is the record of what is required; it is not listed there. +on: + pull_request: + # The filter MUST include this workflow, the lint, and the baseline. A gate excluded from its + # own trigger cannot observe changes to itself -- a broken lint or a quietly widened baseline + # would merge green and be discovered later by an unrelated documentation edit. + paths: &tally_paths + - 'docs/**' + - 'scripts/docs/asvs_tally_lint.py' + - 'scripts/docs/asvs_tally_baseline.txt' + - '.github/workflows/asvs-tally-lint.yml' + push: + branches: [main] + paths: *tally_paths + workflow_dispatch: + +permissions: + contents: read + +concurrency: + group: asvs-tally-lint-${{ github.ref }} + cancel-in-progress: true + +jobs: + tally: + name: no new hard-coded ASVS tally + runs-on: ubuntu-latest + timeout-minutes: 5 + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false # read-only lint job; don't persist the token (zizmor: artipacked) + + - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: '3.14' + + # No install step and no dependency: the lint is stdlib-only, which is also what lets the + # assessment repo mirror and run it on a bare interpreter. tests/test_asvs_verifier_vault_contract.py + # holds that property for every mirrored tool, so it cannot be lost quietly here. + # + # The lint prints the file count, the idiom breakdown and the per-file counts before its + # verdict, so a run that scanned nothing cannot be read as a run that found nothing. + - name: Refuse a new hard-coded ASVS tally + run: | + python scripts/docs/asvs_tally_lint.py docs \ + --baseline scripts/docs/asvs_tally_baseline.txt \ + --allow docs/security/ASVS-CURRENT.md diff --git a/docs/BACKLOG.md b/docs/BACKLOG.md index 053630c1..fbba25d5 100644 --- a/docs/BACKLOG.md +++ b/docs/BACKLOG.md @@ -8824,3 +8824,163 @@ larger change than this one. **Source:** found 2026-08-09 while probing for a second instance of the `#1106` class before building a generalised check, on the reasoning that a meta-check built from one instance is shaped like that instance. Two domains were probed; this one leaked. + +## 1202. the vault ASVS gate runs a verifier this repo owns, on a bare interpreter, and nothing here checked it would run + +> 🔢 **Filed 2026-08-09 - FIXED in the same change. Proved by injecting four contract violations into `scripts/asvs/scorecard.py` and confirming each landed before believing the red.** Value **6/10** · Difficulty **2/10**. The ASVS gate lives in the vault and measures THIS tree. Two of its three claims about this tree need no private data -- that `scripts/asvs/scorecard.py` exists at that literal path, and that it runs as a bare script on stdlib alone -- and neither repo checked either of them. + +**Cluster:** CI correctness / gate blindness. **Priority:** P2. **Verdict:** build (done). +**Severity:** no product effect and no PHI effect. The cost is that the ASVS record would go on being +verified by a stale instrument while every check stayed green. + +**The shape.** The vault executes the verifier twice, and neither job installs anything: +`asvs-scorecard.yml` runs `python scripts/asvs/scorecard.py` after a bare `actions/setup-python`, and +`asvs-verifier-drift.yml`'s `preflight` job runs the INCOMING engine copy as +`python ../engine/scripts/asvs/scorecard.py` from the vault checkout -- so `sys.path[0]` is +`scripts/asvs`, not either repo root. `asvs-scorecard.yml` states the invariant in as many words: +"No install step and no dependency: the verifier is stdlib-only (tomllib, json, re) precisely so this +job cannot rot on a lockfile it does not own." **Asserted in a comment, enforced nowhere** -- which is +the compensating-control-on-a-false-premise defect Secure_Development_Standards SDS-3.7 names. + +**Why this repo's own tests could not see it.** `tests/test_asvs_scorecard.py` imports the module -- +`from scripts.asvs.scorecard import ...` -- inside a pytest session with the repo root on `sys.path` +and the project's full extras installed. That is a different import context from the one the vault +uses, so the invocation that actually matters was never exercised here. + +**Measured.** With a USED `import httpx` appended to `scripts/asvs/scorecard.py` (httpx is a real +engine dependency, so this is not dead code a linter rejects anyway), every pre-existing gate returned +a verdict IDENTICAL to its unmutated verdict: `ruff check .` PASS -> PASS, `ruff format --check .` +unchanged, `mypy` unchanged (it types `messagefoundry` and `messagefoundry_webconsole`, not +`scripts/`), and `tests/test_asvs_scorecard.py` 54 tests PASS -> PASS. `python -I -S -c "import httpx"` +returns 1, confirming the mutation models the vault runner and not a lint opinion. + +**What breaking it would cost, stated exactly.** Not a red gate. The mirror job opens its pull request +as a DRAFT when the incoming verifier does not run, and a draft is never merged -- so the vault keeps +verifying the record with its PREVIOUS copy while reporting drift as a warning. That is the recurring +condition (six hand-made mirror commits, the last found 326 lines behind) that splitting +`asvs-verifier-drift.yml` out was written to end. + +**The guard.** `tests/test_asvs_verifier_vault_contract.py`: the file exists at the hardcoded path; a +stdlib-only `ast` scan over EVERY import including deferred ones; and a `python -I -S