Skip to content

Withdraw conformance and benchmark claims sitewide; decouple site build from wasm - #420

Merged
MelbourneDeveloper merged 1 commit into
mainfrom
website
Aug 6, 2026
Merged

Withdraw conformance and benchmark claims sitewide; decouple site build from wasm#420
MelbourneDeveloper merged 1 commit into
mainfrom
website

Conversation

@MelbourneDeveloper

Copy link
Copy Markdown
Collaborator

TLDR

Withdraws Basilisk's 100% conformance claim and its published benchmark figures across the whole public site (English and Chinese), replaces them with a linked correction and the integrity audit that explains why, and decouples the Eleventy build from the Rust wasm build so a non-compiling checker can no longer take the entire site down.

What Was Added?

  • docs/CONFORMANCE-INTEGRITY-AUDIT.md (285 lines) — the record of what was found: checker logic fitted to the exact contents of conformance fixtures rather than implementing the typing specification generally, including a type-alias validator matching prefixes and substrings of raw source text with a special case for eval( because that spelling appeared in one test file.
  • Explicit Build playground engine (wasm) step in both ci.yml's website job and deploy-pages.yml, replacing the implicit one that npm run build used to chain.
  • .bench-caveat notice on /docs/benchmarks/, stating the figures are withdrawn pending an integrity review and must not be used to compare Basilisk with other tools.
  • New e2e assertions that the old claims are absent: expect(body).not.toContain("scores 100%") and expect(body).not.toContain("fastest we've benchmarked").

What Was Changed or Deleted?

  • Homepage (EN + ZH) — headline drops "the only Python type checker that scores 100% … and the fastest we've benchmarked" for "An open-source Python type checker and language server, built in Rust." The two hero proof cards now read Current typing conformance and Published benchmark figures, both pointing at the withdrawal.
  • /docs/conformance/ — retitled "Basilisk Conformance Results Are Withdrawn"; the leaderboard table, the "How it's measured" section and the reproduce-it-yourself instructions are removed. States plainly that Basilisk was removed from the python/typing results table at our request and that its current percentage is temporarily unknown.
  • _data/conformanceOfficial.js — no longer exports live claim data (basiliskRank, perfectCount, basiliskIsSolePerfect). It now exports withdrawn: true, publicationStatus: "historical-withdrawn", and nests the frozen snapshot under historical.* with basiliskWasSolePerfectAtSnapshot. resultsDir is repointed from main to the pinned commit so the snapshot cannot silently re-render as current.
  • website/package.json"build" changes from npm run build:wasm && eleventy to eleventy. Every page but the playground renders from committed data, so the site build no longer has a Rust dependency.
  • Site/social metadataeleventy.config.js description, package.json description, site.json, og-image, banner SVG/PNGs and the og:title all updated; page title drops "Fast".
  • Blog posts (EN + ZH) — the 100%-conformance announcement and five other posts updated so no retracted claim survives in an older article.
  • /docs/comparison.md, /docs/index.md, quick-start, migration, install-vscode, releases.njk, benchmark-section.njk, conformance-chart.njk — comparative claims removed or re-scoped.

How Do The Automated Tests Prove It Works?

npm run test:e2e52 passed, 6 skipped, 0 failed on both the desktop (Desktop Chrome) and mobile (iPhone SE, 375×667) projects, run with CI=1 so reuseExistingServer is off exactly as in CI.

The suite proves the correction is actually rendered, not just committed:

  • homepage.spec.tsputs the integrity correction beside the product introduction — asserts the subheadline contains "withdrawn both our former conformance claim and our published benchmark figures", "removed from the official python/typing results at our request", "current conformance percentage is temporarily unknown", and "robustness and mutation testing".
  • homepage.spec.tsshows both withdrawn result notices and their detail links — asserts exactly 2 stat cards, "removed from the official results table", "Both sets of figures are withdrawn", and that the body no longer contains "scores 100%" or "fastest we've benchmarked". This is the assertion that fails if a retracted claim is reintroduced anywhere on the page.
  • homepage.spec.tsChinese homepage › renders the same correction and withdrawn-result notices — the same gates in zh, so one locale cannot quietly keep a claim the other withdrew, plus keeps its disclosure structure aligned with the English page.
  • benchmarks.spec.ts — asserts .bench-caveat contains "benchmark figures are withdrawn pending an integrity review" and "Do not use the values below to compare Basilisk with other tools".
  • playground.spec.ts (4 tests × 2 viewports) — drives the real wasm engine, proving the npm run build decoupling did not break the playground.

Also run locally and green: python3 scripts/gen_rules_reference.py --data + diff (rules.json in sync, 169 codes), python3 scripts/gen_conformance_reference.py --check (stamped references and generated READMEs in sync), npm run build:wasm under CI's RUSTFLAGS="-D warnings -C link-arg=-zstack-size=16777216", npm run build (302 files), cargo fmt --all --check (0 diffs) and cargo check --workspace --all-targets.

make ci (lint test build) was not run in full: this branch changes no Rust, and CI path-filters those jobs off for it (code=false). cargo check --workspace --all-targets and cargo fmt --all --check both pass.

Spec / Doc Changes

  • docs/CONFORMANCE-INTEGRITY-AUDIT.md — new; linked from the site's conformance correction.
  • docs/INDEX.md — registers the audit under a new Conformance integrity section.
  • docs/specs/WASM-SPEC.md — records that the playground engine is built by an explicit build:wasm step, separate from the Eleventy build.
  • docs/specs/WEBSITE-E2E-SPEC.md[WEBSITE-E2E-SMOKE] rewritten: the homepage gates now assert the integrity disclosures and the absence of the withdrawn claims, in both locales.

Breaking Changes

  • None to any shipped binary or API. _data/conformanceOfficial.js changes shape (live fields replaced by withdrawn + historical.*), which is internal to the site build and consumed only by templates updated in this PR.

Refs #379, #408

@MelbourneDeveloper
MelbourneDeveloper merged commit 4f3e62c into main Aug 6, 2026
24 checks passed
@MelbourneDeveloper
MelbourneDeveloper deleted the website branch August 6, 2026 11:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant