Withdraw conformance and benchmark claims sitewide; decouple site build from wasm - #420
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TLDR
Withdraws Basilisk's 100% conformance claim and its published benchmark figures across the whole public site (English and Chinese), replaces them with a linked correction and the integrity audit that explains why, and decouples the Eleventy build from the Rust wasm build so a non-compiling checker can no longer take the entire site down.
What Was Added?
docs/CONFORMANCE-INTEGRITY-AUDIT.md(285 lines) — the record of what was found: checker logic fitted to the exact contents of conformance fixtures rather than implementing the typing specification generally, including a type-alias validator matching prefixes and substrings of raw source text with a special case foreval(because that spelling appeared in one test file.Build playground engine (wasm)step in bothci.yml's website job anddeploy-pages.yml, replacing the implicit one thatnpm run buildused to chain..bench-caveatnotice on/docs/benchmarks/, stating the figures are withdrawn pending an integrity review and must not be used to compare Basilisk with other tools.expect(body).not.toContain("scores 100%")andexpect(body).not.toContain("fastest we've benchmarked").What Was Changed or Deleted?
/docs/conformance/— retitled "Basilisk Conformance Results Are Withdrawn"; the leaderboard table, the "How it's measured" section and the reproduce-it-yourself instructions are removed. States plainly that Basilisk was removed from thepython/typingresults table at our request and that its current percentage is temporarily unknown._data/conformanceOfficial.js— no longer exports live claim data (basiliskRank,perfectCount,basiliskIsSolePerfect). It now exportswithdrawn: true,publicationStatus: "historical-withdrawn", and nests the frozen snapshot underhistorical.*withbasiliskWasSolePerfectAtSnapshot.resultsDiris repointed frommainto the pinned commit so the snapshot cannot silently re-render as current.website/package.json—"build"changes fromnpm run build:wasm && eleventytoeleventy. Every page but the playground renders from committed data, so the site build no longer has a Rust dependency.eleventy.config.jsdescription,package.jsondescription,site.json, og-image, banner SVG/PNGs and theog:titleall updated; page title drops "Fast"./docs/comparison.md,/docs/index.md,quick-start,migration,install-vscode,releases.njk,benchmark-section.njk,conformance-chart.njk— comparative claims removed or re-scoped.How Do The Automated Tests Prove It Works?
npm run test:e2e— 52 passed, 6 skipped, 0 failed on both thedesktop(Desktop Chrome) andmobile(iPhone SE, 375×667) projects, run withCI=1soreuseExistingServeris off exactly as in CI.The suite proves the correction is actually rendered, not just committed:
homepage.spec.ts› puts the integrity correction beside the product introduction — asserts the subheadline contains "withdrawn both our former conformance claim and our published benchmark figures", "removed from the official python/typing results at our request", "current conformance percentage is temporarily unknown", and "robustness and mutation testing".homepage.spec.ts› shows both withdrawn result notices and their detail links — asserts exactly 2 stat cards, "removed from the official results table", "Both sets of figures are withdrawn", and that the body no longer contains "scores 100%" or "fastest we've benchmarked". This is the assertion that fails if a retracted claim is reintroduced anywhere on the page.homepage.spec.ts› Chinese homepage › renders the same correction and withdrawn-result notices — the same gates inzh, so one locale cannot quietly keep a claim the other withdrew, plus keeps its disclosure structure aligned with the English page.benchmarks.spec.ts— asserts.bench-caveatcontains "benchmark figures are withdrawn pending an integrity review" and "Do not use the values below to compare Basilisk with other tools".playground.spec.ts(4 tests × 2 viewports) — drives the real wasm engine, proving thenpm run builddecoupling did not break the playground.Also run locally and green:
python3 scripts/gen_rules_reference.py --data+diff(rules.json in sync, 169 codes),python3 scripts/gen_conformance_reference.py --check(stamped references and generated READMEs in sync),npm run build:wasmunder CI'sRUSTFLAGS="-D warnings -C link-arg=-zstack-size=16777216",npm run build(302 files),cargo fmt --all --check(0 diffs) andcargo check --workspace --all-targets.make ci(lint test build) was not run in full: this branch changes no Rust, and CI path-filters those jobs off for it (code=false).cargo check --workspace --all-targetsandcargo fmt --all --checkboth pass.Spec / Doc Changes
docs/CONFORMANCE-INTEGRITY-AUDIT.md— new; linked from the site's conformance correction.docs/INDEX.md— registers the audit under a new Conformance integrity section.docs/specs/WASM-SPEC.md— records that the playground engine is built by an explicitbuild:wasmstep, separate from the Eleventy build.docs/specs/WEBSITE-E2E-SPEC.md—[WEBSITE-E2E-SMOKE]rewritten: the homepage gates now assert the integrity disclosures and the absence of the withdrawn claims, in both locales.Breaking Changes
_data/conformanceOfficial.jschanges shape (live fields replaced bywithdrawn+historical.*), which is internal to the site build and consumed only by templates updated in this PR.Refs #379, #408