Retraction material: state the verification failure plainly, warn against pipeline use, and stop republishing withdrawn conformance figures - #423
Merged
Conversation
The change-scope classifier routes by path, so a branch that mixes documentation with an unrelated tooling change still runs the full Rust matrix, the live conformance clone, and the mutation shards. Add an explicit per-PR opt-out: a docs-only label makes the classifier emit every scope as false. A label rather than a path rule keeps the decision visible on the PR and attributable to whoever made it, and leaves every unlabelled PR untouched.
The homepage e2e asserts that .hero__subheadline discloses that the conformance percentage is temporarily unknown and that no replacement figure ships before robustness and mutation testing. Earlier hero copy dropped those sentences while the assertions stayed, so the Website Build job failed on main and on this branch. Fixed by restoring the disclosure to the English and Chinese heroes, not by relaxing the test — the assertion guards a documentation-honesty claim that should be on the page.
The measurement can no longer run. python/typing's TYPE_CHECKERS tuple no longer registers a Basilisk checker after our withdrawal, and upstream matches --only-run by name against that tuple, so '--only-run basilisk' matches nothing: main.py grades no checker and writes no results, and run_harness() raises 'the real harness wrote no results ... it did not run'. That unconditional failure blocked the release job, the PyPI wheels, and every publish job behind them, and failed make test. Type checking is not conformance, and a gate that can only fail must not stop releases shipping. Commented out, not reimplemented — vendoring a scorer or injecting an adapter is a BUILD FAILURE under [CHKARCH-CONFORMANCE]: - release.yml: the conformance and conformance-wheel jobs, with release now needing only build, pypi-wheels needing nothing, and publish-pypi needing pypi-wheels and release. - test-rust.sh: both conformance passes. The fixture sync still runs; it mirrors files without invoking the harness. Also corrects the coverage-thresholds.json note, which claimed the failure was an argparse 'invalid choice' error and exit 2. It is not; --only-run takes any string and silently matches no checker.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TLDR
Rewrites the public conformance-retraction material to state plainly that the withdrawn result was a verification failure rather than an attempt to game the suite, replaces the original announcement post's body with a full retraction, adds a "don't put the checker in your pipeline" warning to every README and the site hero, and stops the site republishing withdrawn conformance figures out of historical GitHub release notes.
What Was Added?
redactWithdrawnClaims()runs over each GitHub release body before markdown rendering. A line is redacted only when it pairs a conformance subject (conformance|conformant) with a figure (percentage,n/m, or "N false positives"); the line's list marker is preserved and the text is replaced with a visible*[withdrawn conformance claim redacted — see the correction](/docs/conformance/)*. Nothing is silently dropped and each release heading still links to the unmodified release on GitHub.docs/readme/*.src.md, vscode-extension,basilisk.nvim,basilisk-zed):christianfindlay.com/blog/basilisk-conformance-apology) from the site hero (en + zh), both conformance docs, both blog posts, CONTRIBUTING, the integrity audit, and every README.What Was Changed or Deleted?
eval(special case, withint("3")accepted silently), the intent statement, what has been done, what is true today, and a request that anyone citing the post update or remove the citation. Title, description, excerpt, keywords, and the FAQ block are rewritten to match — the FAQ no longer answers "which checker has the highest conformance score" and instead answers whether Basilisk is listed (no), what its percentage is (none published), whether it was gaming the suite (no — verification failure), and whether to use it in CI (not yet).pep conformanceandconformancemarketplace keywords — the extension listing no longer markets on a withdrawn claim.conformance._docnote is prefixed with DEAD AS OF 2026-08-08 — the withdrawal removedBasiliskTypeCheckerfrom the upstream suite'sconformance/src/type_checker.py, so the measurement can no longer run. (See the addendum below for the exact mechanism, which the first draft of this note got wrong.) The note records that this fails the release workflow's conformance job, every job depending on it, andmake test, and that restoring it by vendoring a scorer or injecting an adapter is a BUILD FAILURE under[CHKARCH-CONFORMANCE]. Whether the block is deleted or replaced by a disclosed non-official harness is left as the user's decision. Thethreshold: 100andmax_false_positives: 0values are unchanged — nothing was loosened.How Do The Automated Tests Prove It Works?
No new automated tests. The only executable change is
redactWithdrawnClaims()in the 11ty data layer, exercised by the website build job (website=truefires onwebsite/*), which renders/docs/releases/from the live GitHub releases API — a throw or a template break in that function fails the build. The predicate is conjunctive by design (subject and figure on the same line), so a release note mentioning conformance without a number, or a number without conformance, renders untouched.Local
make ciwas skipped at the user's request for this PR. Note that the full Rust matrix cannot currently go green on any PR: the conformance gate it runs is dead upstream, as documented in thecoverage-thresholds.jsonchange above.Spec / Doc Changes
Everything in this PR is docs, site content, or metadata:
README.md,README.zh.md,README-pypi.md,docs/readme/README.src.md,docs/readme/README.zh.src.md,vscode-extension/README.{md,zh.md},basilisk.nvim/README.{md,zh.md},basilisk-zed/README.{md,zh.md}CONTRIBUTING.md,docs/CONFORMANCE-INTEGRITY-AUDIT.mdwebsite/src/docs/conformance.md,website/src/zh/docs/conformance.md,website/src/docs/releases.njk,website/src/index.njk,website/src/zh/index.njk, both blog postscoverage-thresholds.json(_docprose only),vscode-extension/package.json(keywords),website/src/_data/releases.jsNo spec IDs added or renamed. No checker rule, crate, or diagnostic behaviour is touched.
Breaking Changes
Refs #379
Addendum — CI and release-pipeline fixes added after review
Homepage disclosure restored (
69350ee9)Website Buildwas failing, and it was already failing onmain: homepage.spec.ts:37 asserts.hero__subheadlinecontains "current conformance percentage is temporarily unknown" and "robustness and mutation testing", and the zh test asserts "当前符合性百分比暂时未知". Earlier hero copy dropped those sentences while the assertions stayed. This PR does not touchwebsite/tests/.Fixed by putting the disclosure back on both heroes — not by relaxing the test. The assertion guards a documentation-honesty claim that belongs on the page. Verified locally:
npx playwright testover homepage/navigation/benchmarks/errors/screenshots → 45 passed, 5 skipped, desktop and mobile projects.Dead conformance gates commented out (
17736ebf)The conformance measurement cannot run, so every gate built on it could only fail. Verified against upstream:
python/typing@main'sTYPE_CHECKERStuple registers only mypy, pyright, zuban, pyrefly, pycroscope, and ty.--only-runis matched by name against that tuple (conformance/src/main.py:if options.only_run and options.only_run != type_checker.name: continue), so--only-run basiliskis not an argparse error — it matches nothing, grades no checker, writes noresults/basilisk/*.toml, andrun_harness()raises"the real harness wrote no results ... it did not run".That blocked the entire publish path. Commented out — not reimplemented, since vendoring a scorer or injecting an adapter is a BUILD FAILURE under
[CHKARCH-CONFORMANCE]:conformanceandconformance-wheeljobs. Rewired:releaseneeds[build],pypi-wheelsneeds nothing,publish-pypineeds[pypi-wheels, release]. Job graph validated — 13 jobs, no danglingneeds, no conformance job remaining. This is what lets the GitHub Release, the per-platform VSIX, Homebrew, Scoop, nvim, Zed, and the PyPI wheels ship again.invalid choice: 'basilisk'error with exit 2. That claim is wrong and is corrected to the mechanism verified above.threshold: 100andmax_false_positives: 0are still untouched.Removing these gates is a deliberate, disclosed decision made by the repo owner, recorded here rather than done silently. No conformance figure is published, quoted, or restored anywhere.
CI scope
Carries the
docs-onlylabel, the mechanism added in55eead98on this branch: this PR changes no checker behaviour, so the code matrix is skipped rather than gating a documentation change on a Rust matrix that is unrelated to it.