From 198951b686819615201811f55fb630e6952a71ef Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Wed, 5 Aug 2026 00:03:06 -0500 Subject: [PATCH] ci: run the doc guards on docs-only PRs, and raise an anti-narrowing floor that had 23 items of slack MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit TEN doc-drift modules exist to police documents and every one is reachable ONLY through pytest, which is gated on `code == 'true'`. So on a docs-only PR -- the exact shape they exist to check -- none of them runs. That is the same defect the "Backlog status invariant" step above was added to fix, at ten times the surface, and this change extends that step's own argument rather than inventing a design: "gating it would skip it on exactly the pull requests it exists to police". NOT HYPOTHETICAL. On 2026-08-04 four docs-only PRs merged (#197, #198, #200, #201) and the doc guards ran on none of them. Two carried citation errors found only by reading: an ADR named ONE inbound citation of docs/releases/ where there are 23, and a docs/SECURITY.md route-table row asserted a refusal DELETE /me/mfa does not make. Neither of those two would have been caught by these particular modules -- but they are the evidence that docs-only PRs land unchecked against the guards built to check them. NO NEW REQUIRED CONTEXT, and that is deliberate. The steps ride the existing required `test (ubuntu-latest, py3.14)` job, which already RUNS on every PR -- only its steps are gated. So this needs no branch-protection change and no owner action, unlike a new job. .github/required-contexts.txt is explicit that it is "not the enforcement -- the server is", and adding a required context is all-or-nothing against a 422. `[dev]` rather than a bare `pip install pytest`, and this one would have bitten: pyproject sets asyncio_mode = "auto" (needs pytest-asyncio) and addopts "--timeout=60 --timeout-method=thread" (needs pytest-timeout), so a bare pytest ERRORS on an unknown option before collecting anything -- a step that could never pass, reding every docs-only PR. Still far lighter than the gated install, which adds console + fhir + dicom + x12 + xml that no doc scan touches. THE FLOOR HAD 23 ITEMS OF SLACK. `--min-items 277` against a corpus that has reached 300, in a guard whose entire purpose is to notice the corpus shrinking. Raised to 300 in BOTH places -- and the second place is the finding: the floor also lives as `_MIN_TOTAL_ITEMS` in tests/test_backlog_status_check.py and NOTHING compares them, so the lower copy silently becomes the only floor that binds. That is BACKLOG #1018's duplicated-constant shape. Both sites now name the other; it is not mechanised here because a test parsing workflow YAML to compare an integer is a new coupling, and the honest fix belongs with #1018's helper. 89 of the 241 tests SKIP and that is structural, recorded in the step: they assert against docs/security/THREAT-MODEL.md, which is vault-only and absent here. ADR 0156 records the class and ASVS 15.1.3 is open on it. The step prints the module list BEFORE running and uses -rs so no skip reads as a pass. ⚠️ THIS PR CANNOT EXERCISE ITS OWN NEW STEPS. It changes ci.yml and a test, so `code == 'true'` here and the new steps skip. Evidence is a local run of the exact command (152 passed, 89 skipped, 5.12s) plus confirming [dev] supplies both plugins, that "Set up Python" is ungated, and that constraints.lock is tracked. The first docs-only PR is the real test, and it fails CLOSED -- a broken step reds that PR rather than passing silently. Verified: 57 workflow-structure tests pass (incl. test_workflow_shell_syntax and test_ci_venv_pinning), 27 backlog/lint-scope tests pass, ruff check and format clean, and the YAML parses with both floors reading 300. --- .github/workflows/ci.yml | 62 +++++++++++++++++++++++++++++- tests/test_backlog_status_check.py | 10 ++++- 2 files changed, 70 insertions(+), 2 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 85c9116f..21c63f38 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -139,9 +139,69 @@ jobs: # contradictions, no duplicates — is satisfied just as easily by scanning a remnant of the corpus # as the whole of it. Without a floor, a change that stopped the archive being read would go green. # Raise the number when the total legitimately grows; it must never be lowered to make CI pass. + # ⚠️ THE FLOOR LIVES IN TWO PLACES AND NOTHING COMPARES THEM: here, and `_MIN_TOTAL_ITEMS` in + # tests/test_backlog_status_check.py. Raise BOTH, or the lower one becomes the only floor that + # binds. Found on 2026-08-05 at 277 against a corpus of 300 — 23 items of accumulated slack, in a + # guard whose entire purpose is to notice the corpus shrinking. - name: Backlog status invariant (ungated — see above) if: runner.os == 'Linux' - run: python scripts/docs/backlog_status_check.py --min-items 277 + run: python scripts/docs/backlog_status_check.py --min-items 300 + + # THE SAME ARGUMENT, ONE STEP FURTHER — and the evidence is from 2026-08-04. + # + # Ten `*_doc_drift` / `docs_*` modules exist to police documents, and every one of them is + # reachable ONLY through pytest, which is gated on `code == 'true'` two steps below. So on a + # docs-only PR — the exact shape they exist to check — none of them runs. That is the same defect + # the status invariant above was added to fix, at ten times the surface. + # + # It is not hypothetical. On 2026-08-04 four docs-only PRs merged (#197, #198, #200, #201) and the + # doc guards ran on none of them. Two carried citation errors found only by reading: an ADR named + # ONE inbound citation of `docs/releases/` where there are 23 (12 via `docs/releases/...`, 15 via + # relative `(releases/...`, and neither grep form alone finds them all), and a `docs/SECURITY.md` + # route-table row asserted a refusal `DELETE /me/mfa` does not make. + # + # WHY A SEPARATE MINIMAL INSTALL instead of ungating the install below. Unlike the status + # invariant, these modules import the package (`config.settings`, `config.models`, + # `parsing.binary`, `auth.service`) and one imports `fastapi.routing`, so they cannot run on + # stdlib alone. But they need NO extras — fhir/dicom/x12/xml are irrelevant to a doc scan — so a + # docs-only PR pays a base editable install and ~5s of tests rather than the full extras install. + # The gated steps below are deliberately UNTOUCHED, so a code PR is byte-identical to before. + # + # ⚠️ 89 of these tests SKIP here and that is structural, not a gap to fix in this step: + # `tests/test_threat_model_doc_drift.py` asserts against `docs/security/THREAT-MODEL.md`, which is + # vault-only and absent from this tree. ADR 0156 records that class (six `*_doc_drift` modules + # assert against documents `git ls-files docs/security/` shows are not present) and ASVS 15.1.3 is + # open on it. 152 assertions DO run, which is the point; do not read the skips as coverage. + # `[dev]` and not a bare `pip install pytest`, and this is NOT belt-and-braces. pyproject sets + # `asyncio_mode = "auto"` (needs pytest-asyncio) and `addopts = "--timeout=60 + # --timeout-method=thread"` (needs pytest-timeout), so a bare pytest ERRORS on an unknown option + # before collecting anything — a step that could never pass, reding every docs-only PR. `[dev]` is + # the canonical "can run this suite" extra and pins both plugins; it is still far lighter than the + # gated install below, which adds console + fhir + dicom + x12 + xml that no doc scan touches. + # `--constraint constraints.lock` for the same DEP-1 reason every other install here carries it. + - name: Install (minimal — for the doc guards on a docs-only PR) + if: runner.os == 'Linux' && needs.changes.outputs.code != 'true' && github.event_name == 'pull_request' + run: | + python -m pip install --upgrade pip + pip install -e ".[dev]" --constraint constraints.lock + + - name: Doc guards (ungated — the docs-only blind spot; see above) + if: runner.os == 'Linux' && needs.changes.outputs.code != 'true' && github.event_name == 'pull_request' + run: | + # PRINT WHAT IS SCANNED BEFORE RUNNING IT. A list that silently shrinks is how this class of + # guard goes quiet, and `-rs` names every skip rather than letting one read as a pass. + echo "doc guards, docs-only PR — scanning these modules:" + printf ' %s\n' tests/test_asvs_file_surface_doc_drift.py tests/test_cloud_phi_hipaa_doc_drift.py \ + tests/test_crit2_inline_doc_drift.py tests/test_doc_ref_handle.py tests/test_docs_db_grants.py \ + tests/test_docs_runbooks.py tests/test_docs_security_pathways.py tests/test_security_doc_drift.py \ + tests/test_security_doc_rate_limits.py tests/test_threat_model_doc_drift.py \ + tests/test_backlog_status_check.py + pytest -q -rs \ + tests/test_asvs_file_surface_doc_drift.py tests/test_cloud_phi_hipaa_doc_drift.py \ + tests/test_crit2_inline_doc_drift.py tests/test_doc_ref_handle.py tests/test_docs_db_grants.py \ + tests/test_docs_runbooks.py tests/test_docs_security_pathways.py tests/test_security_doc_drift.py \ + tests/test_security_doc_rate_limits.py tests/test_threat_model_doc_drift.py \ + tests/test_backlog_status_check.py # PySide6's offscreen platform plugin needs a few system libraries even # headless. Linux-only; Windows runners need no equivalent. diff --git a/tests/test_backlog_status_check.py b/tests/test_backlog_status_check.py index 4ae96dac..40efac84 100644 --- a/tests/test_backlog_status_check.py +++ b/tests/test_backlog_status_check.py @@ -170,7 +170,15 @@ def test_duplicate_across_two_sources_is_an_error() -> None: # below) but never deleted. So a total below the count at the time this floor was set means either an # item was dropped or — the failure this exists for — a file holding items stopped being scanned. # Both are bugs; neither is visible from `errors == []`, which passes happily over a smaller corpus. -_MIN_TOTAL_ITEMS = 277 +# +# ⚠️ THIS FLOOR EXISTS IN TWO PLACES AND THEY ARE NOT CROSS-CHECKED. The other is the +# `--min-items` argument in `.github/workflows/ci.yml`'s "Backlog status invariant" step. Raise BOTH or +# the lower one silently becomes the only floor that binds — on 2026-08-05 this copy was found at 277 +# while the corpus had reached 300, so 23 items of slack had accumulated unnoticed. That is the same +# duplicated-constant shape as BACKLOG #1018 (three copies of one scan, nothing comparing them); it is +# noted rather than mechanised here because a test parsing a workflow YAML to compare an integer is a +# new coupling, and the honest fix belongs with #1018's helper rather than beside it. +_MIN_TOTAL_ITEMS = 300 @pytest.mark.skipif(