Skip to content

Retraction material: state the verification failure plainly, warn against pipeline use, and stop republishing withdrawn conformance figures - #423

Merged
MelbourneDeveloper merged 10 commits into
mainfrom
docs
Aug 8, 2026
Merged

Retraction material: state the verification failure plainly, warn against pipeline use, and stop republishing withdrawn conformance figures#423
MelbourneDeveloper merged 10 commits into
mainfrom
docs

Conversation

@MelbourneDeveloper

@MelbourneDeveloper MelbourneDeveloper commented Aug 8, 2026

Copy link
Copy Markdown
Collaborator

TLDR

Rewrites the public conformance-retraction material to state plainly that the withdrawn result was a verification failure rather than an attempt to game the suite, replaces the original announcement post's body with a full retraction, adds a "don't put the checker in your pipeline" warning to every README and the site hero, and stops the site republishing withdrawn conformance figures out of historical GitHub release notes.

What Was Added?

  • Withdrawn-claim redaction for release noteswebsite/src/_data/releases.js: redactWithdrawnClaims() runs over each GitHub release body before markdown rendering. A line is redacted only when it pairs a conformance subject (conformance|conformant) with a figure (percentage, n/m, or "N false positives"); the line's list marker is preserved and the text is replaced with a visible *[withdrawn conformance claim redacted — see the correction](/docs/conformance/)*. Nothing is silently dropped and each release heading still links to the unmodified release on GitHub.
  • Pipeline warning block in every README (root, zh, pypi, docs/readme/*.src.md, vscode-extension, basilisk.nvim, basilisk-zed): ⚠️ Do not use Basilisk's type checker in your pipeline — rules that decide from spelling can be wrong in both directions (false error on correct code, silence on a genuine bug); don't gate CI, don't read a clean run as a clean codebase; the LSP/refactoring/formatting/debugging/profiling side is unaffected.
  • "This was a mistake, not an attempt to game the suite" sections in docs/CONFORMANCE-INTEGRITY-AUDIT.md (On intent.), website/src/docs/conformance.md, website/src/zh/docs/conformance.md, the READMEs, and CONTRIBUTING.md — stating that the submission ran the suite's own unmodified harness with default configuration and all specification rules enabled, nothing was concealed, and the failure was that no semantics-preserving mutation check existed.
  • Links to the author's personal account and apology (christianfindlay.com/blog/basilisk-conformance-apology) from the site hero (en + zh), both conformance docs, both blog posts, CONTRIBUTING, the integrity audit, and every README.
  • Statement of direction: where a rule can't be made reliable straightforwardly, Basilisk will depend on an established external type checker rather than ship its own unreliable version — replacing the previous "we haven't decided yet" wording.

What Was Changed or Deleted?

  • website/src/blog/basilisk-100-percent-python-typing-conformance.md and its zh counterpart: the original announcement body is deleted, not banner-annotated. The leaderboard table, the ranking against pyright/pyrefly/ty/mypy/zuban/pycroscope, the pass counts, the "what 100% means" argument, and the measurement write-up are all gone. What remains is the retraction: what the original claimed, why the claim was wrong (the type-alias rule's raw-text eval( special case, with int("3") accepted silently), the intent statement, what has been done, what is true today, and a request that anyone citing the post update or remove the citation. Title, description, excerpt, keywords, and the FAQ block are rewritten to match — the FAQ no longer answers "which checker has the highest conformance score" and instead answers whether Basilisk is listed (no), what its percentage is (none published), whether it was gaming the suite (no — verification failure), and whether to use it in CI (not yet).
  • website/src/docs/releases.njk: the caveat no longer says historical claims are "preserved verbatim"; it now explains that claim-bearing lines are redacted and that each heading links to the unmodified GitHub release.
  • vscode-extension/package.json: dropped the pep conformance and conformance marketplace keywords — the extension listing no longer markets on a withdrawn claim.
  • coverage-thresholds.json: the conformance._doc note is prefixed with DEAD AS OF 2026-08-08 — the withdrawal removed BasiliskTypeChecker from the upstream suite's conformance/src/type_checker.py, so the measurement can no longer run. (See the addendum below for the exact mechanism, which the first draft of this note got wrong.) The note records that this fails the release workflow's conformance job, every job depending on it, and make test, and that restoring it by vendoring a scorer or injecting an adapter is a BUILD FAILURE under [CHKARCH-CONFORMANCE]. Whether the block is deleted or replaced by a disclosed non-official harness is left as the user's decision. The threshold: 100 and max_false_positives: 0 values are unchanged — nothing was loosened.
  • Site hero (en + zh): leads with "Don't put the type checker in your pipeline yet", and the proof CTA now reads "This was a mistake and a failure to verify, not an attempt to game the suite."
  • Wording throughout: "100% conformance claim" → "former conformance claim", so the correction material stops restating the figure it retracts.

How Do The Automated Tests Prove It Works?

No new automated tests. The only executable change is redactWithdrawnClaims() in the 11ty data layer, exercised by the website build job (website=true fires on website/*), which renders /docs/releases/ from the live GitHub releases API — a throw or a template break in that function fails the build. The predicate is conjunctive by design (subject and figure on the same line), so a release note mentioning conformance without a number, or a number without conformance, renders untouched.

Local make ci was skipped at the user's request for this PR. Note that the full Rust matrix cannot currently go green on any PR: the conformance gate it runs is dead upstream, as documented in the coverage-thresholds.json change above.

Spec / Doc Changes

Everything in this PR is docs, site content, or metadata:

  • README.md, README.zh.md, README-pypi.md, docs/readme/README.src.md, docs/readme/README.zh.src.md, vscode-extension/README.{md,zh.md}, basilisk.nvim/README.{md,zh.md}, basilisk-zed/README.{md,zh.md}
  • CONTRIBUTING.md, docs/CONFORMANCE-INTEGRITY-AUDIT.md
  • website/src/docs/conformance.md, website/src/zh/docs/conformance.md, website/src/docs/releases.njk, website/src/index.njk, website/src/zh/index.njk, both blog posts
  • coverage-thresholds.json (_doc prose only), vscode-extension/package.json (keywords), website/src/_data/releases.js

No spec IDs added or renamed. No checker rule, crate, or diagnostic behaviour is touched.

Breaking Changes

  • None

Refs #379


Addendum — CI and release-pipeline fixes added after review

Homepage disclosure restored (69350ee9)

Website Build was failing, and it was already failing on main: homepage.spec.ts:37 asserts .hero__subheadline contains "current conformance percentage is temporarily unknown" and "robustness and mutation testing", and the zh test asserts "当前符合性百分比暂时未知". Earlier hero copy dropped those sentences while the assertions stayed. This PR does not touch website/tests/.

Fixed by putting the disclosure back on both heroes — not by relaxing the test. The assertion guards a documentation-honesty claim that belongs on the page. Verified locally: npx playwright test over homepage/navigation/benchmarks/errors/screenshots → 45 passed, 5 skipped, desktop and mobile projects.

Dead conformance gates commented out (17736ebf)

The conformance measurement cannot run, so every gate built on it could only fail. Verified against upstream: python/typing@main's TYPE_CHECKERS tuple registers only mypy, pyright, zuban, pyrefly, pycroscope, and ty. --only-run is matched by name against that tuple (conformance/src/main.py: if options.only_run and options.only_run != type_checker.name: continue), so --only-run basilisk is not an argparse error — it matches nothing, grades no checker, writes no results/basilisk/*.toml, and run_harness() raises "the real harness wrote no results ... it did not run".

That blocked the entire publish path. Commented out — not reimplemented, since vendoring a scorer or injecting an adapter is a BUILD FAILURE under [CHKARCH-CONFORMANCE]:

  • .github/workflows/release.yml — the conformance and conformance-wheel jobs. Rewired: release needs [build], pypi-wheels needs nothing, publish-pypi needs [pypi-wheels, release]. Job graph validated — 13 jobs, no dangling needs, no conformance job remaining. This is what lets the GitHub Release, the per-platform VSIX, Homebrew, Scoop, nvim, Zed, and the PyPI wheels ship again.
  • scripts/test-rust.sh — both conformance passes (the instrumented coverage pass and the gate pass). The fixture sync above them still runs; it mirrors files without invoking the harness.
  • coverage-thresholds.json — the note added earlier in this PR claimed the failure was an argparse invalid choice: 'basilisk' error with exit 2. That claim is wrong and is corrected to the mechanism verified above. threshold: 100 and max_false_positives: 0 are still untouched.

Removing these gates is a deliberate, disclosed decision made by the repo owner, recorded here rather than done silently. No conformance figure is published, quoted, or restored anywhere.

CI scope

Carries the docs-only label, the mechanism added in 55eead98 on this branch: this PR changes no checker behaviour, so the code matrix is skipped rather than gating a documentation change on a Rust matrix that is unrelated to it.

The change-scope classifier routes by path, so a branch that mixes
documentation with an unrelated tooling change still runs the full Rust
matrix, the live conformance clone, and the mutation shards.

Add an explicit per-PR opt-out: a docs-only label makes the classifier
emit every scope as false. A label rather than a path rule keeps the
decision visible on the PR and attributable to whoever made it, and
leaves every unlabelled PR untouched.
The homepage e2e asserts that .hero__subheadline discloses that the
conformance percentage is temporarily unknown and that no replacement
figure ships before robustness and mutation testing. Earlier hero copy
dropped those sentences while the assertions stayed, so the Website Build
job failed on main and on this branch.

Fixed by restoring the disclosure to the English and Chinese heroes, not
by relaxing the test — the assertion guards a documentation-honesty
claim that should be on the page.
@MelbourneDeveloper MelbourneDeveloper added the docs-only CI skips the code matrix for this PR — documentation only label Aug 8, 2026
The measurement can no longer run. python/typing's TYPE_CHECKERS tuple no
longer registers a Basilisk checker after our withdrawal, and upstream
matches --only-run by name against that tuple, so '--only-run basilisk'
matches nothing: main.py grades no checker and writes no results, and
run_harness() raises 'the real harness wrote no results ... it did not run'.

That unconditional failure blocked the release job, the PyPI wheels, and
every publish job behind them, and failed make test. Type checking is not
conformance, and a gate that can only fail must not stop releases shipping.

Commented out, not reimplemented — vendoring a scorer or injecting an
adapter is a BUILD FAILURE under [CHKARCH-CONFORMANCE]:
- release.yml: the conformance and conformance-wheel jobs, with release
  now needing only build, pypi-wheels needing nothing, and publish-pypi
  needing pypi-wheels and release.
- test-rust.sh: both conformance passes. The fixture sync still runs; it
  mirrors files without invoking the harness.

Also corrects the coverage-thresholds.json note, which claimed the failure
was an argparse 'invalid choice' error and exit 2. It is not; --only-run
takes any string and silently matches no checker.
@MelbourneDeveloper
MelbourneDeveloper merged commit 2008a04 into main Aug 8, 2026
24 checks passed
@MelbourneDeveloper
MelbourneDeveloper deleted the docs branch August 8, 2026 02:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs-only CI skips the code matrix for this PR — documentation only

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant