Retraction material: state the verification failure plainly, warn against pipeline use, and stop republishing withdrawn conformance figures

.github/workflows/release.yml

@@ -142,42 +142,53 @@ jobs:
           path: ${{ matrix.archive }}
           retention-days: 1
 
-  conformance:
-    name: PEP conformance gate — 100% vs latest python/typing@main
-    runs-on: ubuntu-latest
-    timeout-minutes: 20
-    steps:
-      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
-
-      - uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable
-
-      - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2
-        with:
-          key: conformance
-
-      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
-        with:
-          python-version: "3.12"
-
-      # The REAL python/typing harness's own runtime deps (run_conformance.py runs
-      # the suite's unmodified src/main.py). See [CHKARCH-CONFORMANCE].
-      - name: Install conformance harness deps
-        run: pip install jinja2 markdown tomlkit
-
-      - name: Build basilisk
-        run: cargo build --release --bin basilisk
-
-      # Clone the LATEST python/typing@main FRESH and run its OWN harness against
-      # the compiled binary; require a perfect score — anything less than 100%, or a
-      # single false positive, tanks the release (coverage-thresholds.json →
-      # conformance.threshold 100, max_false_positives 0). No cached fixtures, no
-      # vendored calculator: if the real harness cannot run, the release fails.
-      - name: Enforce 100% PEP conformance (REAL python/typing harness)
-        run: python3 conformance/run_conformance.py --bin target/release/basilisk --gate
+  # ── PEP conformance gate — COMMENTED OUT, THE MEASUREMENT CANNOT RUN ────────
+  # Basilisk's withdrawal from python/typing removed BasiliskTypeChecker from the
+  # suite's conformance/src/type_checker.py. The upstream harness resolves
+  # `--only-run` by comparing the value against each registered checker's name
+  # (conformance/src/main.py: `if options.only_run != type_checker.name: continue`),
+  # so `--only-run basilisk` now matches NOTHING: main.py exits 0 having graded no
+  # checker and writing no results/basilisk/*.toml, and run_conformance.py's
+  # run_harness() then raises "the real harness wrote no results ... it did not run".
+  # That failure is unconditional and blocks every publish job behind it.
+  #
+  # Type checking is not conformance, and a dead gate must not stop releases from
+  # shipping. Restoring the measurement would require vendoring a scorer or
+  # injecting an adapter, both of which [CHKARCH-CONFORMANCE] declares a BUILD
+  # FAILURE, so the job is commented out rather than reimplemented. Whether it is
+  # deleted outright or replaced by a disclosed non-official harness is the user's
+  # decision. Nothing here publishes, quotes, or markets a conformance figure.
+  #
+  # conformance:
+  #   name: PEP conformance gate — 100% vs latest python/typing@main
+  #   runs-on: ubuntu-latest
+  #   timeout-minutes: 20
+  #   steps:
+  #     - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+  #
+  #     - uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable
+  #
+  #     - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2
+  #       with:
+  #         key: conformance
+  #
+  #     - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+  #       with:
+  #         python-version: "3.12"
+  #
+  #     - name: Install conformance harness deps
+  #       run: pip install jinja2 markdown tomlkit
+  #
+  #     - name: Build basilisk
+  #       run: cargo build --release --bin basilisk
+  #
+  #     - name: Enforce 100% PEP conformance (REAL python/typing harness)
+  #       run: python3 conformance/run_conformance.py --bin target/release/basilisk --gate
 
   release:
     name: Publish binaries to release
-    needs: [build, conformance]
+    # `conformance` removed from needs — the job is commented out above.
+    needs: [build]
     runs-on: ubuntu-latest
     timeout-minutes: 10
     steps:
@@ -873,10 +884,10 @@ jobs:
   # See pyproject.toml ([tool.maturin] bindings = "bin").
   pypi-wheels:
     name: Build PyPI wheel - ${{ matrix.platform }}
-    # Gate wheel builds on the existing binary conformance job: never build a
-    # wheel of a non-conformant binary. Independent of the `build` matrix — the
-    # release archives feed Homebrew/Scoop/VSIX; these wheels feed PyPI.
-    needs: conformance
+    # Previously gated on the binary `conformance` job, which is commented out
+    # above because the measurement can no longer run. Independent of the `build`
+    # matrix — the release archives feed Homebrew/Scoop/VSIX; these wheels feed
+    # PyPI — so with that gate gone this job has no remaining prerequisite.
     runs-on: ${{ matrix.os }}
     timeout-minutes: 30
     strategy:
@@ -958,64 +969,73 @@ jobs:
           path: dist/*.whl
           retention-days: 1
 
-  # Prove the ACTUAL shipped wheel — pip-installed — scores 100% via the REAL,
-  # unmodified python/typing harness (its own `src/main.py` + built-in
-  # BasiliskTypeChecker). This is the SAME harness CI runs against the compiled
-  # binary (conformance/run_conformance.py); here it runs against the pip-installed
-  # wheel so the released install path is proven conformant too. There is no other
-  # scorer. See [CHKARCH-CONFORMANCE].
-  conformance-wheel:
-    name: Conformance gate — pip-installed wheel vs python/typing@main (real harness)
-    needs: pypi-wheels
-    runs-on: ubuntu-latest
-    timeout-minutes: 20
-    steps:
-      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
-
-      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
-        with:
-          python-version: "3.12"
-
-      - name: Download linux-x64 wheel
-        uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
-        with:
-          name: pypi-wheel-linux-x64
-          path: wheelhouse
-
-      - name: Install the wheel + harness deps (proves install + exposes `basilisk`)
-        shell: bash
-        run: |
-          set -euo pipefail
-          python -m pip install --upgrade pip
-          python -m pip install wheelhouse/*.whl jinja2 markdown tomlkit
-          # Fails the job immediately if the wheel did not put `basilisk` on PATH.
-          basilisk --version
-
-      - name: Clone python/typing@main (real fixtures + harness + adapter)
-        shell: bash
-        # The suite already ships the official Basilisk adapter (BasiliskTypeChecker
-        # in conformance/src/type_checker.py) — nothing of ours is injected.
-        # Cloned OUTSIDE the repo checkout: per-file config discovery walks
-        # ancestor directories ([CHKARCH-CONFIG-DISCOVERY]), so an in-repo clone
-        # would inherit the repo's own [tool.basilisk] rules instead of scoring
-        # the wheel's out-of-the-box default.
-        run: git clone --depth 1 https://github.com/python/typing "$RUNNER_TEMP/typing-upstream"
-
-      - name: Run the REAL upstream harness against the installed wheel
-        working-directory: ${{ runner.temp }}/typing-upstream/conformance
-        shell: bash
-        # `basilisk` resolves to the pip-installed wheel on PATH (BASILISK_BIN unset).
-        run: python src/main.py --only-run basilisk
-
-      - name: Assert the wheel scores 100% / zero false positives
-        shell: bash
-        run: python conformance/assert_wheel_conformance.py "$RUNNER_TEMP/typing-upstream/conformance/results/basilisk"
+  # ── Wheel conformance gate — COMMENTED OUT, THE MEASUREMENT CANNOT RUN ─────
+  # Same cause as the binary `conformance` job above: python/typing no longer
+  # registers a Basilisk checker, so `src/main.py --only-run basilisk` matches no
+  # checker, grades nothing, and writes no results/basilisk/*.toml for
+  # assert_wheel_conformance.py to read. The job could only ever fail, and it
+  # blocked publish-pypi. Commented out rather than reimplemented: vendoring a
+  # scorer or injecting an adapter is a BUILD FAILURE under [CHKARCH-CONFORMANCE].
+  #
+  # # Prove the ACTUAL shipped wheel — pip-installed — scores 100% via the REAL,
+  # # unmodified python/typing harness (its own `src/main.py` + built-in
+  # # BasiliskTypeChecker). This is the SAME harness CI runs against the compiled
+  # # binary (conformance/run_conformance.py); here it runs against the pip-installed
+  # # wheel so the released install path is proven conformant too. There is no other
+  # # scorer. See [CHKARCH-CONFORMANCE].
+  # conformance-wheel:
+  #   name: Conformance gate — pip-installed wheel vs python/typing@main (real harness)
+  #   needs: pypi-wheels
+  #   runs-on: ubuntu-latest
+  #   timeout-minutes: 20
+  #   steps:
+  #     - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+  #
+  #     - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+  #       with:
+  #         python-version: "3.12"
+  #
+  #     - name: Download linux-x64 wheel
+  #       uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
+  #       with:
+  #         name: pypi-wheel-linux-x64
+  #         path: wheelhouse
+  #
+  #     - name: Install the wheel + harness deps (proves install + exposes `basilisk`)
+  #       shell: bash
+  #       run: |
+  #         set -euo pipefail
+  #         python -m pip install --upgrade pip
+  #         python -m pip install wheelhouse/*.whl jinja2 markdown tomlkit
+  #         # Fails the job immediately if the wheel did not put `basilisk` on PATH.
+  #         basilisk --version
+  #
+  #     - name: Clone python/typing@main (real fixtures + harness + adapter)
+  #       shell: bash
+  #       # The suite already ships the official Basilisk adapter (BasiliskTypeChecker
+  #       # in conformance/src/type_checker.py) — nothing of ours is injected.
+  #       # Cloned OUTSIDE the repo checkout: per-file config discovery walks
+  #       # ancestor directories ([CHKARCH-CONFIG-DISCOVERY]), so an in-repo clone
+  #       # would inherit the repo's own [tool.basilisk] rules instead of scoring
+  #       # the wheel's out-of-the-box default.
+  #       run: git clone --depth 1 https://github.com/python/typing "$RUNNER_TEMP/typing-upstream"
+  #
+  #     - name: Run the REAL upstream harness against the installed wheel
+  #       working-directory: ${{ runner.temp }}/typing-upstream/conformance
+  #       shell: bash
+  #       # `basilisk` resolves to the pip-installed wheel on PATH (BASILISK_BIN unset).
+  #       run: python src/main.py --only-run basilisk
+  #
+  #     - name: Assert the wheel scores 100% / zero false positives
+  #       shell: bash
+  #       run: python conformance/assert_wheel_conformance.py "$RUNNER_TEMP/typing-upstream/conformance/results/basilisk"
 
   publish-pypi:
     name: Publish wheels to PyPI
-    # Only publish once the wheels built AND the shipped wheel passed the real
-    # upstream harness AND the GitHub release exists.
-    needs: [pypi-wheels, conformance-wheel, release]
+    # Only publish once the wheels built AND the GitHub release exists. The
+    # `conformance-wheel` prerequisite is gone with that job — see the comment on
+    # the commented-out block above.
+    needs: [pypi-wheels, release]
     runs-on: ubuntu-latest
     timeout-minutes: 15
     if: startsWith(github.ref, 'refs/tags/')

CONTRIBUTING.md

@@ -21,7 +21,7 @@ Basilisk is built by a **human + AI partnership**, split on purpose. AI agents d
 
 You don't need to write Rust to make Basilisk better. **The highest-leverage thing a human can do here is verify that the checker actually analyses code** — not that a number went up.
 
-This isn't hypothetical. Checker logic was fitted to the conformance fixtures, the resulting 100% was published, and we didn't catch it until much later; both published numbers — conformance and performance — are now **withdrawn**. See the [conformance correction](https://www.basilisk-python.dev/docs/conformance/) and the [integrity audit](docs/CONFORMANCE-INTEGRITY-AUDIT.md). None of it was deliberate — the instructions named the score as the goal, and matching text moves a score faster than analysing code does. In rough order of impact:
+This isn't hypothetical. Checker logic was fitted to the conformance fixtures, the resulting score was published, and we didn't catch it until much later; both published numbers — conformance and performance — are now **withdrawn**. See the [conformance correction](https://www.basilisk-python.dev/docs/conformance/), the [integrity audit](docs/CONFORMANCE-INTEGRITY-AUDIT.md), and the author's [personal account and apology](https://www.christianfindlay.com/blog/basilisk-conformance-apology). None of it was deliberate — nobody set out to game the suite; the instructions named the score as the goal, matching text moves a score faster than analysing code does, and nothing verified the difference. In rough order of impact:
 
 ### 1. Verify the metrics yourself
 

README-pypi.md

@@ -29,32 +29,62 @@
   <img src="https://raw.githubusercontent.com/Nimblesite/Basilisk/main/images/screenshot.png" alt="Basilisk in action — type checking, diagnostics, and refactoring in the editor" width="900">
 </p>
 
-**The current type checker contains inaccuracies and you should not use it as part of your dev pipeline. We are working on removing any misleading analyzers ASAP. Please read below**
-
-## We are auditing the checker and deleting what doesn't hold up
-
-We withdrew our 100% conformance claim and our benchmark figures, and asked to be
+> ## ⚠️ Do not use Basilisk's type checker in your pipeline
+>
+> **The type checker still contains code that isn't doing real type checking, and
+> it is not yet trustworthy.** Some rules decide from the way code is *spelled*
+> rather than what it means, so they can be wrong in both directions — a false
+> error on correct code, or silence where there is a genuine bug. Until the audit
+> below is finished, don't gate CI on `basilisk check`, don't block a merge with
+> it, and don't read a clean run as a clean codebase.
+>
+> The rest of Basilisk — language server, refactoring, formatting, debugging,
+> profiling — does not depend on those rules and is unaffected.
+
+## Restoring trust: audit, delete, and lean on a checker that works
+
+We withdrew our former conformance claim and our benchmark figures, and asked to be
 [removed from the official `python/typing` results](https://github.com/python/typing/blob/main/conformance/results/results.html).
 The cause was checker logic fitted to the contents of conformance test files
 instead of implementing the typing specification generally: rules that matched
 the *spelling* of code rather than its meaning. Rename an import or reformat a
 file and the answer changed. A score produced that way is not evidence.
 
+**This was a mistake and a failure to verify — not an attempt to game the suite.**
+Nobody set out to defeat the conformance tests, and nothing was concealed from
+`python/typing`: the submission ran the suite's own unmodified harness, with
+default configuration and every rule enabled. Our process treated the score as
+the goal, matching text raises a score faster than real analysis does, and we
+published without ever asking whether a rule still held when the same program was
+spelled differently. Basilisk's author has published a
+[personal account and apology](https://www.christianfindlay.com/blog/basilisk-conformance-apology).
+
 **So we are auditing every rule and deleting the ones that don't do real type
 checking.** Not rewriting them, not patching them, not marking them TODO —
 deleting them, with a failing test left behind so the gap is visible instead of
 hidden. A rule stays only if it decides from the resolved syntax tree and gives
 the same answer when the code is spelled differently.
 
+**Where a rule can't be made reliable in a straightforward way, we will depend on
+a different, established type checker rather than ship our own unreliable version
+of it.** An answer from an engine that has earned trust is worth more to you than
+a Basilisk-branded one that hasn't. No replacement figure gets published until it
+survives off-suite and mutation testing.
+
 That means Basilisk gets **smaller** before it gets better. Expect fewer rules,
 fewer diagnostics, and a lower conformance number. We will report each drop
 rather than avoid it. What is left will be code that is honest about what it
 does — nothing else.
 
-We have not yet decided whether to rebuild the deleted analysis from the
-specification or to drive the extension with an established open-source checker.
-Either way, no new figure gets published until it survives off-suite and
-mutation testing.
+### Basilisk is much more than a type checker
+
+Type checking is one part of it. The rest is a complete Python workflow in a
+single Rust binary — language server, refactoring, formatting, integrated
+debugging, profiling, and the editor extensions — and none of it rests on the
+rules under audit. That is what we are sharpening while the audit runs: make the
+parts that are genuinely useful solid, and remove anything that could hand you a
+misleading result. The point of getting smaller is to end up with a tool you can
+believe.
 
 [Read the full correction &rarr;](https://www.basilisk-python.dev/docs/conformance/) &nbsp;&bull;&nbsp;
 [Integrity audit &rarr;](https://github.com/Nimblesite/Basilisk/blob/main/docs/CONFORMANCE-INTEGRITY-AUDIT.md)