From 30da5a293cfff98a39716d18b6bcbaa126dfc7c6 Mon Sep 17 00:00:00 2001 From: Christian Findlay <16697547+MelbourneDeveloper@users.noreply.github.com> Date: Sat, 8 Aug 2026 07:00:56 +1000 Subject: [PATCH 1/9] Documentation fixes --- .github/workflows/ci.yml | 10 ++ .github/workflows/release.yml | 13 ++ CLAUDE.md | 182 +++++++++++-------- CONTRIBUTING.md | 201 ++++++--------------- CONTRIBUTING.zh.md | 250 +++++++++----------------- README-pypi.md | 58 +++--- README.md | 59 +++--- README.zh.md | 56 +++--- basilisk-zed/README.md | 6 +- basilisk-zed/README.zh.md | 6 +- basilisk-zed/extension.toml | 2 +- basilisk.nvim/README.md | 6 +- basilisk.nvim/README.zh.md | 6 +- crates/basilisk-cli/Cargo.toml | 2 +- docs/plans/ROADMAP-NEXT-STEPS-PLAN.md | 45 ++--- docs/readme/README.src.md | 58 +++--- docs/readme/README.zh.src.md | 56 +++--- docs/specs/ZED-SPEC.md | 6 +- pyproject.toml | 2 +- scripts/publish_zed_registry.py | 211 ++++++++++++++++++++++ scripts/test_publish_zed_registry.py | 133 ++++++++++++++ vscode-extension/README.md | 58 +++--- vscode-extension/README.zh.md | 56 +++--- vscode-extension/package.json | 2 +- 24 files changed, 836 insertions(+), 648 deletions(-) create mode 100755 scripts/publish_zed_registry.py create mode 100644 scripts/test_publish_zed_registry.py diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 06c1d351a..ec46f9e95 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -378,6 +378,16 @@ jobs: scripts/render-zed-mirror.sh "${RUNNER_TEMP}/zed-mirror" 0.0.0-ci ( cd "${RUNNER_TEMP}/zed-mirror" && cargo build --release --target wasm32-wasip2 ) + # The registry listing edits a ~1400-entry file in a repo we do not own, + # and it only ever runs during a tagged release — so it gets its proof + # here, on the PR, rather than the first time it touches upstream. Pure + # text-editing functions only; no network ([ZED-MIRROR]). + - name: Test the zed-industries listing edit + run: | + set -euo pipefail + pip install pytest==9.1.1 + python3 -m pytest scripts/test_publish_zed_registry.py -q + # ── Rust coverage + thresholds (runs in parallel) ────────────────────────── test-rust: name: Rust Tests & Coverage diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 6a3eb66a2..a0323f8a3 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -790,6 +790,19 @@ jobs: git push origin "${GITHUB_REF_NAME}" fi + # Pushing the mirror publishes NOTHING on its own — Zed installs only what + # zed-industries/extensions lists. That listing was a manual to-do nobody + # ever did, which is why Basilisk has never appeared in Zed's extensions + # view. This step performs it: first release opens the PR, later releases + # move the submodule pointer and version on the same branch. Idempotent — + # a re-run with nothing to change is a no-op. See [ZED-MIRROR]. + - name: Submit or bump the zed-industries/extensions listing + env: + GH_TOKEN: ${{ secrets.BREW_SCOOP_PAT }} + run: | + set -euo pipefail + scripts/publish_zed_registry.py "${GITHUB_REF_NAME#v}" "${GITHUB_REF_NAME}" + publish-scoop: name: Publish Scoop manifest needs: release diff --git a/CLAUDE.md b/CLAUDE.md index 6841e88ad..7d1ea0beb 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -3,141 +3,171 @@ Code here must comfortably pass review at a top-tier engineering org. Fix shortcomings as you find them. -# Conformance Is the Prime Directive +# Accuracy Is the Prime Directive -Target: **100% conformance** with the [Python typing spec](https://typing.python.org/en/latest/spec/index.html), measured ONLY by the [python/typing conformance suite](https://github.com/python/typing/tree/main/conformance/tests) — nothing else. This outranks every other concern in this file. Read the [conformance README](https://github.com/python/typing/blob/main/conformance/README.md) carefully. Python-version boundaries apply only where the typing spec, an accepted PEP, or Python language semantics defines one; Basilisk has no canonical Python release. +Basilisk must be correct on Python it has never seen. Every rule decides from the resolved AST — bindings, types, symbol identity — never from how the source happens to be spelled. `from typing import Final as F` behaves identically to `typing.Final`; reformatting a file changes no diagnostic. -⚠️ **Never touch the scoreboard — move the number by FIXING the checker.** FORBIDDEN: disabling/deleting/unregistering any rule, deleting rule source (`crates/basilisk-checker/src/rules/*.rs`), removing rules from `all_rules()`, rule-suppressing config (the legacy `basilisk.json` is no longer read), hand-editing `conformance/conformance_status.csv`, loosening `coverage-thresholds.json` (`threshold` / `max_false_positives`). See [CHKARCH-CONFORMANCE], [CHKARCH-CONFORMANCE-MODE]. ⚠️ +Basilisk was **removed from the python/typing conformance results** on 2026-08-05, at its own author's request — [python/typing#2330](https://github.com/python/typing/pull/2330), reverting [#2316](https://github.com/python/typing/pull/2316). The reason: *"Many of Basilisk's rules match against raw source text and hard-coded typing symbol names instead of resolved symbols on the AST."* Semantics-preserving edits to the suite — renaming imports, adjusting whitespace — broke **113 of 141 test files**. The score was real; the checker under it was not. -⚠️ **One conformance path**, run fresh every CI run: `python3 conformance/run_conformance.py`. No step skippable — (1) `git clone` the tests **and** the harness from `python/typing@main` HEAD, no cache/committed fixtures; (2) clean `cargo build --release` from THIS checkout, never the PyPI wheel, never instrumented; (3) run the suite's OWN unmodified `conformance/src/main.py --only-run basilisk` (its `type_checker.py` ships the official `BasiliskTypeChecker`) against that binary via `BASILISK_BIN`, failing hard on ANY false positive or missed required error (100% / 0 FP); (4) regenerate `conformance_status.csv` from the harness's own `results/basilisk/*.toml`. A vendored scorer, reimplemented/injected adapter, cached fixtures, or committed results standing in for a live run is a **BUILD FAILURE**. ⚠️ +Conformance is a downstream sample of that accuracy, and one this codebase is overfitted to. A change that lifts the conformance number without improving analysis of arbitrary Python is a regression. -- The score is the binary in its default config — every PEP rule on, nothing configured ([CHKARCH-CONFIGURATION-ONLY]). Never quote a number produced any other way. -- **Precision is the whole game.** A file passes iff the upstream `errors_diff` is empty: an error on EVERY `# E` line, EVERY `# E[tag]` group satisfied, NOTHING on an unmarked line. -- **Every failure is a false positive, not a miss.** The checker already catches every required error; files fail because a strict house rule fires on spec-valid code. Fix by teaching the checker to recognise the valid construct — never by missing a required error or silencing a rule ([CHKARCH-CONFORMANCE-MODE]). -- **Ratchets, always.** Pass-% only up, FP ceiling only down (`coverage-thresholds.json`). Moving a ratchet the wrong way means the change isn't done. (Benchmark times are NOT a ratchet — see [CHKARCH-TESTING-BENCH].) -- Basilisk is listed in the [official results](https://github.com/python/typing/blob/main/conformance/results/results.html) at 100%. Dropping below is ⛔️ ILLEGAL. +## Text-matched logic — test, delete, report -# Design Principles +A rule whose behaviour depends on the *spelling* of its input rather than the *meaning* is broken, whatever it scores. Finding and removing it outranks every other task here. Find it by: + +- Raw source-text matching — `.contains` / `starts_with` / `ends_with` on user code (131 of 145 rule files today). +- Hard-coded symbol spellings instead of resolved identity: `t == "typing.Final"`, `text.starts_with("Callable[")`, `import.module == "typing"`. +- Any regex over Python source. +- Logic keyed to a test fixture: rule files named after conformance tests (`generics_base_class_2.rs`, `constructors_call_init`), branches for shapes only the suite contains, comments citing a test file as justification. +- Detection that fires on formatting — line breaks, spacing, quote style, comment text, statement order. + +On encountering it, do exactly three things — **do not fix it, do not rewrite it, do not leave a TODO**: + +1. **Write a test that fails** because of the incorrect code — pin the real defect: an aliased import, a reformatted source, a shape the conformance suite never contains. +2. **Delete the offending code.** +3. **Tell the user what you deleted and why**, and that the test is now failing. + +Replacing it is not your call. The point is to surface every one of these so the user can acknowledge it and decide what gets built back. A checker with fewer rules and visible failing tests is the correct outcome; a diagnostic that only fires on one spelling looks like coverage and isn't. + +## What a correct rule looks like + +The yardstick for judging code — not licence to go and fix it: -One IDE extension = a complete, fast Python workflow. The LSP drives all functionality — extensions only react to LSP signals (commands, state changes) and NEVER register a command the LSP doesn't advertise. +- Decides on the **resolved semantic model** from `basilisk-resolver`, never tokens or text. Parses with `ruff_python_parser`. +- Named for the **typing-spec concept** it implements, not a test file. +- Survives **semantics-preserving mutation**: aliased imports, reformatting, reordering → identical diagnostics. Rules without that coverage are unverified. +- Tested against Python the conformance suite has never contained. + +## Direction of travel + +Background, not a directive: strip text-matched logic, establish which rules genuinely analyse code, and rebuild around those — deliberately, with the user, not as a side effect of some other task. Anything that can't be made to work on the AST gets removed rather than propped up; a smaller trustworthy checker beats a large unreliable one. Analysis Basilisk can't do reliably may be delegated to an external engine. Deletion is a legitimate outcome. + +## Conformance's role + +`python3 conformance/run_conformance.py` stays honest: fresh `git clone` from `python/typing@main`, clean `cargo build --release` from THIS checkout, the suite's own unmodified `src/main.py --only-run basilisk` via `BASILISK_BIN`. A vendored scorer, injected adapter, cached fixtures, or committed results standing in for a live run is a **BUILD FAILURE**. The number is a regression detector, never an objective: + +- **Never publish, quote, or market a conformance figure** — nothing may imply Basilisk is in the official results. +- **Never re-submit to python/typing** until the mutation harness passes clean and an external audit has run. +- Never move the number by touching the scoreboard: disabling rules, deleting source to dodge a failure, rule-suppressing config, hand-editing `conformance/conformance_status.csv`, loosening `coverage-thresholds.json` ([CHKARCH-CONFORMANCE]). +- **A drop caused by removing text-matched logic is progress.** Record it and say so plainly — never restore the code or fake a pass to hold a ratchet. + +# Design Principles -Basilisk has **no modes** — behaviour is per-rule configuration ([CHKARCH-CONFIGURATION-ONLY]). The default enables every PEP typing-spec rule and nothing else; opinionated house-style rules (require-annotation `BSK-0001/0002/0004`, require-`@override` `BSK-0025`, redundant-annotation `BSK-0050`, explicit-`Any` nudge `BSK-0014`) are opt-in. Every diagnostic must teach — explain why, not just what. +One IDE extension = a complete, fast Python workflow. The LSP drives all functionality — extensions only react to LSP signals and NEVER register a command the LSP doesn't advertise. -# Documentation Honesty — No Unsubstantiated Claims +**No modes** — behaviour is per-rule configuration ([CHKARCH-CONFIGURATION-ONLY]). Default: every PEP typing-spec rule and nothing else; house-style rules (`BSK-0001/0002/0004` require-annotation, `BSK-0025` require-`@override`, `BSK-0050` redundant-annotation, `BSK-0014` explicit-`Any`) are opt-in. Every diagnostic teaches — why, not just what. -Trust is the product. Applies **everywhere** — specs, plans, README, website, marketing, code comments. +# Documentation Honesty -- **Every empirical or comparative claim about the outside world** (stats, adoption, competitor capability/performance/conformance numbers, market facts, attributed quotes) MUST carry an inline link to the authoritative source that actually makes that claim. Link it or delete it — NEVER invent or approximate. A value that drifts (a competitor's conformance %, a download size) links to its live source, never a frozen figure. -- **Self-measured, reproducible metrics are exempt** (e.g. our conformance score from the unmodified `python/typing` scorer) — but state how they're measured and don't compare them against numbers from a different methodology. -- **Book screenshots are direct release evidence.** Any visual that shows Basilisk, an editor, a terminal, diagnostics, controls, or product output MUST be captured from the book's pinned released build. NEVER mock, redraw, reconstruct, generate, or hand-compose product UI, even under a label such as diagram, wireframe, or conceptual map. Cropping, uniform publication resizing, and external callouts are allowed; repainting, replacing, or compositing product pixels or text is not. If a real capture is unavailable, omit the visual. Follow [`book/VISUAL-DESIGN-SYSTEM.md`](book/VISUAL-DESIGN-SYSTEM.md#screenshot-contract). +Trust is the product. Applies everywhere — specs, plans, README, website, marketing, code comments. +- **Every claim about the outside world** (stats, adoption, competitor numbers, market facts, quotes) carries an inline link to the source making it. Link it or delete it. Drifting values link live, never frozen. +- **Self-measured metrics** state how they're measured, are reproducible, and are never compared across methodologies. Conformance isn't publishable at all (above). +- **Book screenshots are release evidence** — captured from the book's pinned released build; never mocked, redrawn, generated, or hand-composed, not even labelled "diagram". Crop and resize freely; never repaint product pixels. No real capture → omit it. See [`book/VISUAL-DESIGN-SYSTEM.md`](book/VISUAL-DESIGN-SYSTEM.md#screenshot-contract). # Documentation Structure -The spec-ID web is the fabric of this repository and is non-negotiable: +The spec-ID web is non-negotiable: -- Every spec section has a unique, non-numeric, hierarchical ID (`[GROUP-TOPIC]` / `[GROUP-TOPIC-DETAIL]`). -- Code references its spec ID in comments (e.g. `// Implements [LSP-HOVER]`) so `grep [LSP-` walks spec → code → tests in one shot. Tests cross-reference both the spec ID and the code. -- Find code, tests, or specs that aren't linked? Add the missing ID or reference. -- `docs/INDEX.md` — full index. `docs/specs/[COMPONENT]-[FEATURE]-SPEC.md`, `docs/plans/[COMPONENT]-[FEATURE]-PLAN.md`. -- `docs/specs/LSP-ARCHITECTURE-SPEC.md` is the **single source of truth** for all shared LSP/DAP/config/commands; editor-specific specs point back to it. +- Every spec section has a unique, non-numeric, hierarchical ID (`[GROUP-TOPIC-DETAIL]`). +- Code cites its spec ID in comments (`// Implements [LSP-HOVER]`) so `grep [LSP-` walks spec → code → tests; tests cross-reference both. Anything unlinked gets the missing ID. +- `docs/INDEX.md` indexes `docs/specs/[COMPONENT]-[FEATURE]-SPEC.md` and `docs/plans/[COMPONENT]-[FEATURE]-PLAN.md`. `docs/specs/LSP-ARCHITECTURE-SPEC.md` is the **single source of truth** for shared LSP/DAP/config/commands. # Rules -Build scripts live in the Makefile. [Pyrefly](https://pyrefly.org/en/docs/) and [Pyright](https://microsoft.github.io/pyright/#/) are reference implementations to compare against — NEVER copy from their code. +Build scripts live in the Makefile. [Pyrefly](https://pyrefly.org/en/docs/) and [Pyright](https://microsoft.github.io/pyright/#/) are references to compare against — NEVER copy their code. -- **Top priority: reduce duplication.** Run `deslop:find-similar` BEFORE writing new code and `deslop:top-offenders` after changing code. Merge duplicates; keep it DRY. -- Aggressively hoist shared code into shared crates/modules/packages. Use [lspkit](https://crates.io/crates/lspkit) where possible. -- Centralize all global state: one global-state file per app, no state outside it. All mutable state uses Signals — no stale state on screen. +- **Never parse with strings or regex** — `ruff_python_parser` and the resolver only. +- **After correctness, reduce duplication.** `deslop:find-similar` before writing new code, `deslop:top-offenders` after. Merge duplicates. +- Hoist shared code into shared crates/modules. Use [lspkit](https://crates.io/crates/lspkit) where possible. +- One global-state file per app. All mutable state uses Signals — no stale state on screen. - Keep dependency versions in sync across `.github/workflows/ci.yml` and `.devcontainer/Dockerfile`. -- Define spec models in [typeDiagram markup](https://typediagram.dev/docs/language-reference.html); generate ADTs with the [typeDiagram code generator](https://typediagram.dev/docs/cli.html) pointed at the markup. +- Define spec models in [typeDiagram markup](https://typediagram.dev/docs/language-reference.html); generate ADTs with its [code generator](https://typediagram.dev/docs/cli.html). - Don't use Git unless asked. -- Treat legacy code as code to be removed — there is no legacy code in this codebase. -- Avoid regex to parse anything, use ruff. -- Keep files under 500 LOC; break up larger files. Move files rather than copying them. -- Use your judgment — do NOT stop to ask the user questions. -- NEVER kill a VS Code process (including in the browser) — it disrupts active debugging and test sessions. +- Legacy code is code to be removed; there is none here. +- Files under 500 LOC. Move files rather than copying. +- Use your judgment — do NOT stop to ask questions. (Reporting a deletion isn't a question; report and continue.) +- NEVER kill a VS Code process — it disrupts active debugging and test sessions. - Bug Fix Process: [fix bug skill](.claude/skills/fix-bug/SKILL.md) ## Git & Branch Discipline -Git is off-limits unless explicitly asked. When git IS used: +Off-limits unless explicitly asked. When git IS used: -- **NEVER push to `main` directly.** Every change ships via PR → CI green → merge. -- **NEVER list the agent as a commit co-author** — no `Co-Authored-By` trailer, no agent attribution. -- **Work on exactly ONE branch.** Reuse the existing feature branch; if multiple exist, merge them into one before any other work. -- **Worktrees are forbidden** — never run `git worktree`. -- **NEVER close anything you did not open** — no issue, PR, discussion, or review thread, however stale. Including auto-close keywords: write `Refs #123`, never `Closes/Fixes #123`. +- **NEVER push to `main`** — every change ships via PR → CI green → merge. +- **NEVER list the agent as co-author** — no `Co-Authored-By`, no agent attribution. +- **Exactly ONE branch.** Reuse the feature branch; merge multiples into one first. +- **Worktrees are forbidden.** +- **NEVER close anything you did not open** — write `Refs #123`, never `Closes/Fixes #123`. ## Testing -- Target 100% coverage on every measure. Each PR MUST INCREASE overall coverage or it is a failure. -- NEVER delete a failing test, remove a failure-causing assertion, reduce assertiveness, or ignore tests. Broken or missing functionality gets MORE failing tests, never fewer. -- Mutation score only increases; widen scope over time by adding `#[mutation_safe]` tests over more rules/functions. The gate ([CHKARCH-TESTING-MUTATION-RATCHET], baseline `mutation_testing/mutation_scores.json`) fails CI if the viable mutant pool shrinks, caught drops, missed/timeout rise, or kill rate drops. -- `make test` is FAIL-FAST — NEVER use `--no-fail-fast`. -- `make test` always computes and enforces coverage. The threshold lives in `coverage-thresholds.json` at the repo root — not env vars, not GH repo variables, not CI YAML. Ratchet only; below threshold fails the pipeline. -- VSIX tests must not call `whenCommandReady` or `vscode.commands.getCommands(true)` to check existence — the core code does that. Assert through the UI or, worst case, internal VSIX state. +- Tests exercise **meaning, not spelling**: every rule test gets an aliased-import and a reformatted variant, with identical diagnostics. +- Target 100% coverage on every measure. Each PR MUST increase overall coverage. +- NEVER delete a failing test, remove a failure-causing assertion, reduce assertiveness, or ignore tests. Broken functionality gets MORE failing tests, never fewer. +- Mutation score only increases; widen scope with `#[mutation_safe]` tests. The gate ([CHKARCH-TESTING-MUTATION-RATCHET], `mutation_testing/mutation_scores.json`) fails CI if the mutant pool shrinks, caught drops, missed/timeout rise, or kill rate drops. +- `make test` is FAIL-FAST — never `--no-fail-fast`. It enforces coverage from `coverage-thresholds.json` at the repo root, not env vars or CI YAML. Ratchet only. +- VSIX tests must not call `whenCommandReady` or `getCommands(true)` to check existence — assert through the UI, or worst case internal VSIX state. ## Benchmarks -Performance is a feature, but the benchmark is **indicative, not a gate** ([CHKARCH-TESTING-BENCH]). It runs on a developer workstation against whatever else that machine is doing; background load moves every tool in the table together and can shift absolute times by tens of percent between two runs of identical code. **Nothing in CI passes or fails on a benchmark number, and no gate is to be reintroduced** — a pass/fail built on that signal fails honest work and waves through real regressions depending on what else was running. +The benchmark is **indicative, not a gate** ([CHKARCH-TESTING-BENCH]) — it runs on a workstation against whatever else that machine is doing, shifting absolute times by tens of percent between identical runs. **Nothing in CI passes or fails on a benchmark number, and no gate is to be reintroduced.** -- Run `make bench` whenever you touch checker hot paths (resolver visitors, rule `check` loops, new conformance logic). Every run does `cargo clean` + a fresh `--release` build and pulls the latest official release of each competitor (pyright, mypy, ty, pyrefly, zuban) before timing. -- **Write always.** Measured numbers go to `benchmarks/status/.csv` immediately and unconditionally — after every fixture and again at the end (`benchmarks/summarize.py`). A run that measured a number but didn't record it is a lie. -- **Read it correctly.** Compare tools *within* one run — they are measured back to back on the same machine, so machine speed cancels. Never compare a number against one recorded on a different machine or at a different time. To answer a real performance question, measure both revisions on one quiet machine in one sitting. -- The published website figures carry this caveat explicitly; see `website/src/docs/benchmarks.njk`. +- Run `make bench` when touching checker hot paths. Each run does `cargo clean` + a fresh `--release` build and pulls the latest release of each competitor (pyright, mypy, ty, pyrefly, zuban). +- **Write always.** Numbers go to `benchmarks/status/.csv` after every fixture and at the end (`benchmarks/summarize.py`). Measuring without recording is a lie. +- **Read correctly.** Compare tools *within* one run, never across machines or times. See `website/src/docs/benchmarks.njk`. -## Logging Standards +## Logging -- **Structured logging only** — `tracing` + `tracing-subscriber`, never `println!`/`eprintln!` for diagnostics. If you can't see what's happening, add more logging. -- Log at entry/exit of significant operations (`error|warn|info|debug|trace`), with structured fields not interpolation: `tracing::info!(user_id = 42, action = "checkout")`. -- VS Code extension: detailed logs go to a file in the extension's state folder AND to the Output Channel. -- **NEVER log PII** (names, emails, phone, IPs) or secrets — log `"key: present"` or a truncated hash. +- **Structured only** — `tracing` + `tracing-subscriber`, never `println!`/`eprintln!`. Can't see what's happening? Add more logging. +- Log entry/exit of significant operations with structured fields: `tracing::info!(user_id = 42, action = "checkout")`. +- VS Code extension: detailed logs to a file in the extension's state folder AND the Output Channel. +- **NEVER log PII** or secrets — log `"key: present"` or a truncated hash. -## Rust Quality Standards +## Rust Quality -- Run clippy and fmt routinely; fix violations promptly. All lints at highest strictness (Cargo.toml `[lints]`). Add lints if in doubt; never remove them. -- `unsafe` is forbidden (`unsafe_code = "deny"`). `unwrap()` is always a violation — use `?` with proper error types. No `panic!`, `todo!`, `unimplemented!` — handle every case and return `Result`. -- `Result` and `Option` everywhere; early returns with `?`. Expressions over statements (`match`, `if let`, iterator chains). Pattern matching over casting or unwrapping. Pure functions; minimize side effects. -- Small, focused functions (<20 lines) with low cognitive complexity (clippy::cognitive_complexity enabled). Descriptive names (no single letters except in closures). Group related functionality into modules; document public APIs. +- Clippy and fmt routinely. All lints at highest strictness (Cargo.toml `[lints]`). Add lints if in doubt; never remove them. +- `unsafe` is forbidden (`unsafe_code = "deny"`). `unwrap()` is always a violation — use `?` with proper error types. No `panic!`, `todo!`, `unimplemented!`. +- `Result` / `Option` everywhere; early returns with `?`. Expressions over statements. Pattern matching over casting. Pure functions. +- Functions <20 lines, low cognitive complexity. Descriptive names (no single letters outside closures). Group into modules; document public APIs. # Too Many Cooks — Multi-Agent Coordination -Register before starting work. Coordinator dictates orders through plans and messages and delegates; others follow and check messages regularly. Lock files before editing, never edit locked files, and respond to messages promptly. +Register before starting work. The coordinator dictates orders through plans and messages; others follow and check messages regularly. Lock files before editing, never edit locked files, respond promptly. # Website -- **Minimize CSS classes**; name them after what the element IS, not what section it's in. Avoid LLM-default colors (e.g. purple) — use RNG and color wheels. +**Minimize CSS classes**; name them after what the element IS, not what section it's in. Avoid LLM-default colors (e.g. purple) — use RNG and color wheels. ## Per-diagnostic error pages (`/errors/BSK-XXXX/`) -Every diagnostic ends with `see: https://www.basilisk-python.dev/errors/BSK-XXXX` (the `docs_url` on each rule's `ErrorCode`). Pages are generated for all codes from checker source — `[WEBSITE-ERROR-PAGES]` (`docs/specs/WEBSITE-ERROR-PAGES-SPEC.md`). The single source is `website/src/_data/rules.json`: +Every diagnostic ends with `see: https://www.basilisk-python.dev/errors/BSK-XXXX` (each rule's `ErrorCode.docs_url`). Pages generate from checker source — `[WEBSITE-ERROR-PAGES]`. Single source is `website/src/_data/rules.json`: ```bash -python3 scripts/gen_rules_reference.py --data # writes website/src/_data/rules.json +python3 scripts/gen_rules_reference.py --data ``` -It extracts the `//! BSK-XXXX:` summary + doc-comment body (prose and ```python examples) from each `crates/basilisk-checker/src/rules/*.rs`. **Rerun it after adding or renaming a rule** — CI regenerates and `diff`s `rules.json` (`[WEBSITE-ERROR-PAGES-DRIFT]`), and rule-source edits count as website changes so the guard runs. The same data drives the `/docs/rules/` table and counts. Pages render via `website/src/errors/error.njk`; a worked-example screenshot appears automatically for any code in `screenshots/shots.mjs`. +It extracts the `//! BSK-XXXX:` summary + doc-comment body from each `crates/basilisk-checker/src/rules/*.rs`. **Rerun after adding or renaming a rule** — CI regenerates and `diff`s it (`[WEBSITE-ERROR-PAGES-DRIFT]`). The same data drives `/docs/rules/`. Pages render via `website/src/errors/error.njk`; screenshots appear for any code in `screenshots/shots.mjs`. # Architecture -Strict-by-default Python type checker and comprehensive LSP in **Rust**. Users can flick errors down to warnings and adopt type safety incrementally, or just use the LSP for autofixes, formatting, debugging, and profiling. +Strict-by-default Python type checker and comprehensive LSP in **Rust**. Users can flick errors down to warnings and adopt type safety incrementally, or use the LSP alone for autofixes, formatting, debugging, and profiling. - **Parser**: `ruff_python_parser`. **Incremental**: Salsa — sub-10ms incremental checks. -- **Formatting**: `ruff_python_formatter` embedded in-process ([LSPFMT-ENGINE]); import hygiene reimplemented natively on the Ruff AST ([LSPFMT-IMPORTS]). The `ruff` CLI is NOT a runtime dependency — never spawn it. -- **Concurrency**: Tokio in the LSP server (request multiplexing + `spawn_blocking`); analysis is single-threaded on one dedicated large-stack thread ([LSPARCH-ARCH-STACK]). +- **Formatting**: `ruff_python_formatter` in-process ([LSPFMT-ENGINE]); import hygiene native on the Ruff AST ([LSPFMT-IMPORTS]). The `ruff` CLI is NOT a runtime dependency — never spawn it. +- **Concurrency**: Tokio in the LSP server; analysis single-threaded on one dedicated large-stack thread ([LSPARCH-ARCH-STACK]). - **No Pyright/mypy/Node.js** — zero TypeScript or Python runtime. ## Migration to `lspkit` -Cross-cutting LSP scaffolding here is being distilled into the generic `lspkit-*` workspace in [`Nimblesite/lsp_toolkit`](https://github.com/Nimblesite/lsp_toolkit). Prefer `lspkit-*` crates for new LSP infrastructure; when changing existing scaffolding, flag in the PR description if the patch duplicates `lspkit` and reference the upstream crate. +LSP scaffolding is being distilled into the `lspkit-*` workspace in [`Nimblesite/lsp_toolkit`](https://github.com/Nimblesite/lsp_toolkit). Prefer `lspkit-*` for new infrastructure; flag in the PR if a patch duplicates it. | Current path | Toolkit crate | |---|---| -| `crates/basilisk-lsp/src/server/mod.rs:96` tower-lsp `Server` setup | `lspkit-server` (hand-rolled JSON-RPC + `Dispatcher` + `Capabilities`; no `tower-lsp` dependency) | -| `crates/basilisk-lsp/src/workspace.rs:39–116` `WorkspaceIndex` + import-graph invalidation | `lspkit-vfs` + consumer-side index | -| `crates/basilisk-lsp/src/server/handlers/{navigation,features}.rs` | `lspkit-server::Dispatcher::register` per method name | -| `crates/basilisk-lsp/src/server/init.rs:224–242` diagnostic publication | `lspkit-server::diagnostics::DiagnosticsBus` | -| `crates/basilisk-lsp/src/server/mod.rs:61,64` debounce + file-watcher loop | `lspkit-live::watcher::FileWatcher` + `lspkit-live::scheduler::spawn` | -| `crates/basilisk-lsp/src/config.rs:35–100` `WorkspaceConfig` loader | `lspkit-config::load_from_ancestor` | -| `crates/basilisk-lsp/tests/lsp/ws_test_common.rs` E2E fixture | not yet in toolkit (v0.1 follow-up) | +| `crates/basilisk-lsp/src/server/mod.rs:96` tower-lsp `Server` | `lspkit-server` | +| `crates/basilisk-lsp/src/workspace.rs:39–116` `WorkspaceIndex` | `lspkit-vfs` + consumer-side index | +| `crates/basilisk-lsp/src/server/handlers/{navigation,features}.rs` | `lspkit-server::Dispatcher::register` | +| `crates/basilisk-lsp/src/server/init.rs:224–242` diagnostics | `lspkit-server::diagnostics::DiagnosticsBus` | +| `crates/basilisk-lsp/src/server/mod.rs:61,64` debounce + watcher | `lspkit-live::watcher` + `lspkit-live::scheduler` | +| `crates/basilisk-lsp/src/config.rs:35–100` `WorkspaceConfig` | `lspkit-config::load_from_ancestor` | +| `crates/basilisk-lsp/tests/lsp/ws_test_common.rs` E2E fixture | not yet in toolkit | diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 8bb00114c..f03d35691 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -2,17 +2,12 @@

English · 简体中文

-Basilisk is built by a **human + AI partnership**, and the work is split on purpose. -AI agents do the bulk of the mechanical, verifiable engineering. Humans do the things -that need taste, judgment, accountability, and trust — the things AI can't (yet) own. +Basilisk is built by a **human + AI partnership**, split on purpose. AI agents do the mechanical, verifiable engineering. Humans do what needs taste, judgment, accountability, and trust. -This guide has two sections. Pick the one that's you. +- [**For Humans**](#for-humans) — judgment, taste, trust, and everything an agent can't be held accountable for. Express it in the specs first. +- [**For AI**](#for-ai) — technical execution under the rules in [`CLAUDE.md`](CLAUDE.md). -- [**For Humans**](#for-humans) — judgment, taste, trust, and everything an agent can't be held accountable for. Express in the specs first and foremost. -- [**For AI**](#for-ai) — the technical execution, under a strict set of rules that conform to the Basilisk specs. - -> The same split runs through the roadmap. Every TODO in -> [`docs/plans/ROADMAP-NEXT-STEPS-PLAN.md`](docs/plans/ROADMAP-NEXT-STEPS-PLAN.md) is tagged: +> Every TODO in [`docs/plans/ROADMAP-NEXT-STEPS-PLAN.md`](docs/plans/ROADMAP-NEXT-STEPS-PLAN.md) carries the same split: > > | Tag | Meaning | > |---|---| @@ -24,182 +19,104 @@ This guide has two sections. Pick the one that's you. ## For Humans -You don't need to write Rust to make Basilisk better. The **single highest-leverage thing a human can -do on this project is keep the agents honest** — above all about **PEP conformance**, the number most -worth faking and the one an agent is most likely to fake. Agents do the bulk of the mechanical part of the engineering; -humans own the judgment, accountability, and trust an agent can't be held to — and the first of those -duties is *surveillance*. Basilisk's north stars are public and non-negotiable: be the **most -conformant** *and* the **fastest** Python type checker, never trading one for the other -([CHKARCH-TESTING-BENCH-RATCHET]). Both are *measured numbers*, and a measured number is worth nothing -the moment someone games it. In rough order of impact: - -### 1. Keep the agents honest — watch every metric like a hawk - -This is the number one human job on Basilisk. The agents do the engineering; **you make sure they -didn't cheat to do it.** Under pressure, an agent will move the *number* instead of doing the *work* — -and every number here is gameable: **PEP conformance, test coverage, mutation score, test assertions, -lint/clippy, benchmarks.** Treat every metric change as a possible cheat until you've re-derived it -yourself. They cannot grade their own homework — that's what you're here for. - -The dodges, across every metric: - -- **Silence instead of fix** — disabling, deleting, or unregistering a rule so it stops firing, - instead of fixing what it caught. -- **Weaken the test** — deleting failing tests, cutting assertions, or watering them down so "green" - means nothing. -- **Edit the scoreboard or the gate** — hand-editing `conformance_status.csv`, or lowering a - threshold/baseline (`coverage-thresholds.json`, the mutation or benchmark baselines) to match a - faked run. -- **Measure less** — excluding diagnostic codes, skipping fixtures, narrowing mutation scope, grading - a subset. A high percentage over part of the suite is not a real percentage. - -Conformance is the most critical metric, so guard it hardest: it must **strictly track the official PEP -standard**, scored by the official `python/typing` calculator unmodified at a pinned commit — never a -rule turned off, deleted, or unregistered, never a reimplementation. Every -metric only ever moves the *honest* way — conformance, coverage, and mutation **up**; false positives -and benchmark times **down** — because the work genuinely got better, never because someone changed -how we count. **Gaming any of them is a punishable offence** ([CHKARCH-CONFORMANCE]). +You don't need to write Rust to make Basilisk better. **The highest-leverage thing a human can do here is verify that the checker actually analyses code** — not that a number went up. + +This isn't hypothetical. Checker logic was fitted to the conformance fixtures, the resulting 100% was published, and we didn't catch it until much later; both published numbers — conformance and performance — are now **withdrawn**. See the [conformance correction](https://www.basilisk-python.dev/docs/conformance/) and the [integrity audit](docs/CONFORMANCE-INTEGRITY-AUDIT.md). None of it was deliberate — the instructions named the score as the goal, and matching text moves a score faster than analysing code does. In rough order of impact: + +### 1. Verify the metrics yourself + +Agents optimise whatever you measure, and every number here is reachable without doing the underlying work: **conformance, coverage, mutation score, assertions, lint, benchmarks.** Re-derive any metric change before you believe it — agents cannot grade their own homework. What to look for: + +- **Text-matched logic** — the big one. A rule keyed on raw source text or hard-coded symbol spellings instead of resolved AST symbols scores well and fails on real code. Rename an import (`from typing import Final as F`) or reformat a file: the diagnostics must not change. +- **Silence instead of analysis** — a rule disabled, deleted, or unregistered so it stops firing. +- **Weakened tests** — failing tests deleted, assertions cut or watered down so "green" means nothing. +- **Scoreboard or gate edits** — a hand-edited `conformance_status.csv`, or a lowered threshold (`coverage-thresholds.json`, the mutation or benchmark baselines). +- **Measuring less** — excluded diagnostic codes, skipped fixtures, narrowed mutation scope. A high percentage over part of the suite is not a percentage. + +Metrics move only the *honest* way — coverage and mutation up, false positives down — because the work got better, never because someone changed how we count ([CHKARCH-CONFORMANCE]). ### 2. Test it for real — on real, large codebases -Automated tests prove the code does what we told it to. They can't tell you whether the product -*feels* right, holds up against a million lines somebody else wrote, or breaks on a machine we never -tried. **Point Basilisk at the real world:** - -- **Run it on large, real production and open-source codebases** — CPython's `Lib/`, Django, pandas, - Home Assistant, SymPy, Sentry, *and your own company's biggest repos*. Fixtures are tidy; real code - is not, and that's exactly where false positives, crashes, slow paths, and missed errors surface. - (This doubles as scale/perf evidence — see §5.) -- **Install a published artifact** (not a dev build) on a clean machine, open a real Python project, - and confirm diagnostics, hover, go-to-definition, debugging, and profiling all light up — in - **each** editor. UX rough edges and platform-specific breakage are found by humans driving the real - UI, not by CI. -- **Get your team using it every day and harvest their feedback.** Dogfooding is the highest-signal - test there is: put Basilisk in front of real Python developers, watch where they hit friction, and - turn every "this fired on perfectly good code" or "this missed an obvious bug" into an issue (§6) - and a failing test. The goal is real-world adoption, not green fixtures. +Automated tests prove the code does what we told it to. They can't tell you whether it holds up against a million lines somebody else wrote. **Point Basilisk at the real world:** + +- **Run it on large production and open-source codebases** — CPython's `Lib/`, Django, pandas, Home Assistant, SymPy, Sentry, *and your own biggest repos*. Fixtures are tidy; real code is not, and that's where false positives, crashes, slow paths, and missed errors surface. +- **Install a published artifact** (not a dev build) on a clean machine, open a real project, and confirm diagnostics, hover, go-to-definition, debugging, and profiling all light up — in **each** editor. UX and platform breakage are found by humans driving the real UI. +- **Get your team using it daily and harvest their feedback.** Turn every "this fired on perfectly good code" or "this missed an obvious bug" into an issue (§6) and a failing test. ### 3. Maintain and improve code quality -Review AI-authored PRs against the bar in [`CLAUDE.md`](CLAUDE.md): *code here should -comfortably pass review at a top-tier engineering organization.* Catch over-engineering, -premature abstraction, duplicated logic, and the subtly-wrong-but-plausible. An agent will -happily ship something that compiles and passes tests but reads badly or hides a landmine — -your job is to say so. +Review AI-authored PRs against the bar in [`CLAUDE.md`](CLAUDE.md): *code here should comfortably pass review at a top-tier engineering organization.* Catch over-engineering, premature abstraction, duplicated logic, and the subtly-wrong-but-plausible. An agent will happily ship something that compiles and passes tests but hides a landmine. -### 4. Improve test metrics and the mutation score +### 4. Strengthen tests and the mutation score -Coverage percentage is the floor, not the goal. Judge whether assertions actually *prove* -something or just execute lines. Push for stronger assertions, widen the mutation-testing -scope ([CHKARCH-TESTING-MUTATION-RATCHET]), and call out tests that would still pass if the -code were broken. **Both ratchets only move one way** — coverage and mutation score up, -never down. +Coverage percentage is the floor, not the goal. Judge whether assertions actually *prove* something or just execute lines. Push for stronger assertions, widen the mutation-testing scope ([CHKARCH-TESTING-MUTATION-RATCHET]), and call out tests that would still pass if the code were broken. Both ratchets move one way only. ### 5. Guard the performance numbers -"Fastest" is the other half of the promise, and it's just as easy to fool yourself on. Re-run the -benchmarks on real hardware against the committed baseline (`benchmarks/status/.csv`), -confirm no fixture got slower, and optimize every regression before updating the recorded results. -The gate cannot be disabled or widened. A conformance fix that blows the benchmark gate isn't done -— and a benchmark "win" that cost conformance isn't either. Both ratchets hold at once. +Performance is a feature, but the benchmark is **indicative, not a gate** ([CHKARCH-TESTING-BENCH]). It runs on a workstation against whatever else that machine is doing, so background load moves every tool together. Nothing in CI passes or fails on a benchmark number, and no gate is to be reintroduced. + +Only a human can do this: run `make bench` on a quiet machine, compare the tools *within* that single run (timed back to back, so machine speed cancels), and dig into anything that looks off. Never compare against a number from a different machine or time. Every run writes to `benchmarks/status/.csv` immediately — measuring without recording is a lie. ### 6. Report GitHub issues -You're the one running real-world Python through Basilisk. When something is wrong — a false -positive, a missed error, a crash, a slow path, a clumsy editor interaction — file a precise, -reproducible issue with the smallest snippet that triggers it. A good bug report is a gift; it -becomes a failing test, which becomes a fix. +You're the one running real-world Python through Basilisk. When something is wrong — a false positive, a missed error, a crash, a slow path, a clumsy editor interaction — file a precise, reproducible issue with the smallest snippet that triggers it. A good bug report becomes a failing test, which becomes a fix. ### 7. Check plans and specs against reality -Specs and plans are the fabric of this repo (see [`docs/INDEX.md`](docs/INDEX.md)). Audit them: -does every spec section have a non-numeric, hierarchical spec ID? Does the implementing code -actually reference that ID? Does the implementation *match* the spec, or has it drifted? Are the -plans still accurate, or do they describe a world that no longer exists? The `/spec-check` workflow -helps, but the judgment call — *is this spec still telling the truth?* — is yours. +Specs and plans are the fabric of this repo (see [`docs/INDEX.md`](docs/INDEX.md)). Does every section have a non-numeric, hierarchical ID? Does the implementing code reference it? Does the implementation *match* the spec, or has it drifted? `/spec-check` helps, but the judgment — *is this spec still telling the truth?* — is yours. ### 8. Ensure feature parity across IDE extensions -The promise is **one seamless experience in every editor**: VS Code (plus Cursor/Windsurf via -Open VSX), Zed, and Neovim. A feature that lands in one extension but not the others is a parity -bug. Audit the extensions side by side, find the gaps, and file them. Remember the architecture -rule: the **LSP drives functionality** — extensions only react to what the LSP advertises. +The promise is **one seamless experience in every editor**: VS Code (plus Cursor/Windsurf via Open VSX), Zed, and Neovim. A feature in one extension but not the others is a parity bug. Audit them side by side and file the gaps. The **LSP drives functionality** — extensions only react to what it advertises. ### 9. Security auditing -Threat-model the checker, the LSP, the editor extensions, the release pipeline, and the -dependency tree. Review what `/security-review` and Dependabot surface with a human's sense of -*what actually matters*. Single binary, no runtime, no telemetry is a security posture — help us -keep it true. +Threat-model the checker, the LSP, the extensions, the release pipeline, and the dependency tree. Review what `/security-review` and Dependabot surface with a human's sense of *what actually matters*. Single binary, no runtime, no telemetry is a security posture — help keep it true. ### 10. Improve the AI instructions -This is the highest-**compounding** human lever. Better instructions produce better AI output on -every future task. Tighten [`CLAUDE.md`](CLAUDE.md), the specs, and the skills under `.claude/`. -When you watch an agent go wrong, the fix usually isn't the code — it's the instruction that -allowed it. +The highest-**compounding** lever: better instructions produce better output on every future task. Tighten [`CLAUDE.md`](CLAUDE.md), the specs, and the skills under `.claude/`. When an agent goes wrong, the fix usually isn't the code — it's the instruction that allowed it. ### 11. Everything humans are simply best at -Brand voice and naming. Outreach, relationships, and community. Strategic prioritization — *what -should we even build next?* Native-speaker and design judgment. Anything involving accounts, -secrets, tokens, or money. If it can't be checked by a test, it's probably your call. +Brand voice and naming. Outreach and community. Strategic prioritization — *what should we even build next?* Native-speaker and design judgment. Anything involving accounts, secrets, tokens, or money. If a test can't check it, it's your call. ### How to contribute as a human -1. **Open an issue** for a bug, a parity gap, a spec drift, or a conformance discrepancy. Be specific and reproducible. -2. **Open a PR** for fixes or docs — fill out the [pull request template](.github/pull_request_template.md) honestly. "Tests pass" is not an acceptable answer to *how do the tests prove it works?* -3. **Review PRs** — the review itself is a first-class contribution, often the most valuable one. +1. **Open an issue** for a bug, parity gap, spec drift, or checker inaccuracy. Be specific and reproducible. +2. **Open a PR** for fixes or docs — fill out the [pull request template](.github/pull_request_template.md) honestly. "Tests pass" is not an answer to *how do the tests prove it works?* +3. **Review PRs** — a first-class contribution, often the most valuable one. --- ## For AI -You convert the specs to code and tests, and keep all three in sync. The standing rules that govern it live in -[**`CLAUDE.md`**](CLAUDE.md), and they **override default behavior** — read that file first and -follow it exactly. This section is a map, not a restatement (we don't duplicate). +You convert the specs to code and tests and keep all three in sync. The standing rules live in [**`CLAUDE.md`**](CLAUDE.md) and **override default behavior** — read it first and follow it exactly. + +**Before you touch anything:** read [`CLAUDE.md`](CLAUDE.md) in full, orient via [`docs/INDEX.md`](docs/INDEX.md) and [`docs/specs/LSP-ARCHITECTURE-SPEC.md`](docs/specs/LSP-ARCHITECTURE-SPEC.md), then register with the **too-many-cooks** coordinator and **lock files** before editing them. Never edit a locked file. + +**Accuracy is the prime directive.** Basilisk must be correct on Python it has never seen. Every rule decides from the resolved AST, never from how the source happens to be spelled. When you find a rule keyed on raw text, hard-coded symbol spellings, or a conformance fixture, do exactly three things — **do not fix it, do not rewrite it, do not leave a TODO**: -**Before you touch anything:** +1. **Write a test that fails** because of that code. +2. **Delete the offending code.** +3. **Tell the user what you deleted and why.** -- Read [`CLAUDE.md`](CLAUDE.md) in full. Then orient via [`docs/INDEX.md`](docs/INDEX.md) and the - source-of-truth spec [`docs/specs/LSP-ARCHITECTURE-SPEC.md`](docs/specs/LSP-ARCHITECTURE-SPEC.md). -- Register with the **too-many-cooks** coordinator and **lock files** before editing them. Don't - edit a locked file. +What gets built back is the user's call, not yours. **The non-negotiables** (full detail in `CLAUDE.md`): -- **Git is off-limits unless explicitly asked.** Never push to `main`, never list an agent as a - co-author, never use worktrees, work on exactly one branch. -- **Spec IDs are the fabric.** Every spec section has a non-numeric, hierarchical ID; every piece - of code references it (`// Implements [LSP-…]`); every test cross-references both. If you find a - link missing, fix it. -- **DRY, ruthlessly.** Use the `deslop` MCP (`find-similar` before writing, `top-offenders` after). - Merge duplicates. Search for existing code before adding new code. -- **The ratchets only move one way.** Conformance score up; false positives and benchmark - regressions down; coverage up; mutation score up. A conformance fix that blows the benchmark gate - isn't done. -- **Never disable, delete, or unregister a conformance rule to move the score — a punishable offence.** - PEP conformance runs the `basilisk` binary with **every rule enabled**: no Basilisk config file, no - per-rule override, no "spec-conformance mode", no skipped fixtures, **no deleting rule source files, - no removing rules from `all_rules()`**, no exceptions. Equally forbidden: hand-editing - `conformance/conformance_status.csv` or loosening the `coverage-thresholds.json` gate to match a - faked run. The score is exactly what a real user gets out of the box. If a strict default fires on - spec-valid code, **fix the checker** so it stops firing — never silence, delete, or unregister the - rule to inflate the number ([CHKARCH-CONFORMANCE]). -- **`make` is the interface.** `make build | test | lint | fmt | clean | ci | setup` — exactly seven - targets, don't add more. `make test` is fail-fast and enforces the coverage threshold from - `coverage-thresholds.json`. -- **Rust quality bar:** no `unwrap`, `panic!`, `todo!`, `unimplemented!`, `unsafe`, or - `allow(clippy::…)`. `Result`/`Option` everywhere, small pure functions, files under 500 LOC. +- **Git is off-limits unless explicitly asked.** Never push to `main`, never list an agent as co-author, never use worktrees, work on exactly one branch. +- **Spec IDs are the fabric.** Every spec section has a non-numeric, hierarchical ID; code references it (`// Implements [LSP-…]`); tests cross-reference both. Missing link → fix it. +- **DRY, ruthlessly.** `deslop` MCP: `find-similar` before writing, `top-offenders` after. Search for existing code before adding new. +- **Ratchets move one way** — coverage and mutation score up, false positives down. Conformance is a regression detector, not a target; benchmark times gate nothing ([CHKARCH-TESTING-BENCH]). +- **Never touch the scoreboard.** Conformance runs the binary with **every rule enabled**: no config file, no per-rule override, no skipped fixtures, no deleting source to dodge a failure, no removing rules from `all_rules()`. Equally forbidden: hand-editing `conformance/conformance_status.csv` or loosening `coverage-thresholds.json`. Never publish or quote a conformance figure ([CHKARCH-CONFORMANCE]). +- **`make` is the interface.** `make build | test | lint | fmt | clean | ci | setup` — exactly seven targets, don't add more. `make test` is fail-fast and enforces the coverage threshold. +- **Rust quality bar:** no `unwrap`, `panic!`, `todo!`, `unimplemented!`, `unsafe`, or `allow(clippy::…)`. `Result`/`Option` everywhere, small pure functions, files under 500 LOC. - **No CI artifacts.** Storage is billed even on this public repo — see [GITHUB-NO-ARTIFACTS]. **How you work:** -- **Test-driven, always.** Write a failing test → confirm it fails *for the right reason* → fix the - code (never the test) → confirm it passes. Coarse e2e tests only; no unit tests. Never delete a - failing test or weaken an assertion. -- **Use judgment; don't stop to ask.** Make the call and proceed. -- **Pick up `[AGENT]` work** from [`docs/plans/ROADMAP-NEXT-STEPS-PLAN.md`](docs/plans/ROADMAP-NEXT-STEPS-PLAN.md). - Leave `[HUMAN]` work for a human. Draft the agent half of `[HYBRID]` items and hand them off. -- **Defer to the human signals** in this guide. When a human reports an issue, that report becomes - your failing test. +- **Test-driven, always.** Failing test → confirm it fails *for the right reason* → fix the code (never the test) → confirm it passes. Coarse e2e tests only. Never delete a failing test or weaken an assertion. +- **Use judgment; don't stop to ask.** (Reporting a deletion isn't a question — report it and continue.) +- **Pick up `[AGENT]` work** from [`docs/plans/ROADMAP-NEXT-STEPS-PLAN.md`](docs/plans/ROADMAP-NEXT-STEPS-PLAN.md). Leave `[HUMAN]` work for a human. Draft the agent half of `[HYBRID]` items and hand them off. +- **Defer to the human signals** here. A human's issue report becomes your failing test. diff --git a/CONTRIBUTING.zh.md b/CONTRIBUTING.zh.md index bd5022fc1..24bfb0aaa 100644 --- a/CONTRIBUTING.zh.md +++ b/CONTRIBUTING.zh.md @@ -1,200 +1,124 @@ -

English · 简体中文

- -> 📝 本文档由机器翻译生成，欢迎母语者校对改进。 - -# Contributing to Basilisk +# 为 Basilisk 做贡献 -Basilisk 由**人类 + AI 的协作**构建而成，工作的分工是有意为之的。 -AI 智能体负责完成绝大部分机械化、可验证的工程工作。人类则负责那些 -需要品味、判断力、责任感与信任的事情——那些 AI（目前）还无法担当的事情。 +

English · 简体中文

-本指南分为两个部分。请挑选适合你的那一个。 +Basilisk 由**人类 + AI 协作**构建，分工是刻意设计的。AI 智能体承担机械的、可验证的工程工作；人类负责需要品味、判断力、责任感和信任的部分。 -- [**For Humans**](#for-humans) —— 判断力、品味、信任，以及一切无法让智能体承担责任的事情。 -- [**For AI**](#for-ai) —— 在一套严格的常设规则之下进行的技术执行。 +- [**给人类**](#给人类) —— 判断力、品味、信任，以及一切无法由智能体承担责任的事。首先把它表达在规格文档里。 +- [**给 AI**](#给-ai) —— 在 [`CLAUDE.md`](CLAUDE.md) 规则约束下的技术执行。 -> 同样的分工贯穿于整个路线图。 -> [`docs/plans/ROADMAP-NEXT-STEPS-PLAN.md`](docs/plans/ROADMAP-NEXT-STEPS-PLAN.md) 中的每一项 TODO 都带有标签： +> [`docs/plans/ROADMAP-NEXT-STEPS-PLAN.md`](docs/plans/ROADMAP-NEXT-STEPS-PLAN.md) 中的每一条 TODO 都带着同样的分工标记： > -> | 标签 | 含义 | +> | 标记 | 含义 | > |---|---| -> | `[AGENT]` | 由智能体端到端驱动的机械化、可验证的代码/测试/文档工作。 | -> | `[HUMAN]` | 需要人类的判断——账户、密钥、资金、品牌话语、战略、母语者判断。 | -> | `[HYBRID]` | 智能体起草并准备；由人类审查、批准或提供凭据。 | +> | `[AGENT]` | 机械的、可验证的代码/测试/文档工作，智能体端到端完成。 | +> | `[HUMAN]` | 需要人类裁量 —— 账号、密钥、金钱、品牌语调、战略、母语者判断。 | +> | `[HYBRID]` | 智能体起草和准备；人类审阅、批准或提供凭据。 | --- -## For Humans - -你无需会写 Rust 也能让 Basilisk 变得更好。Basilisk 的北极星目标是公开且 -不可妥协的：成为**一致性最高***且***速度最快**的 Python 类型检查器——并且绝不 -为其一而牺牲其二（[CHKARCH-TESTING-BENCH-RATCHET]）。二者都是*被度量出来的数字*，因此 -最具杠杆效应的人类贡献，正是那些让这些数字——以及围绕它们的一切——保持**诚实**的工作： -那些不该被信任由智能体独自签字定夺的判断。大致按影响力排序： - -### 1. 在真实环境中测试 —— 在真实的大型代码库上 - -自动化测试能证明代码做到了我们让它做的事。但它们无法告诉你产品 -*感觉*是否对劲、能否扛得住别人写的百万行代码，或者会不会在我们从未试过的 -机器上崩溃。**把 Basilisk 对准真实世界：** - -- **在大型、真实的生产环境与开源代码库上运行它**——CPython 的 `Lib/`、Django、pandas、 - Home Assistant、SymPy、Sentry，*以及你自己公司里最大的仓库*。fixture 是整洁的；真实代码 - 并非如此，而这恰恰是误报、崩溃、慢路径与漏报浮现的地方。 - （这同时也作为规模/性能方面的证据——参见 §5。） -- **在一台干净的机器上安装已发布的构件**（而非开发版构建），打开一个真实的 Python 项目， - 并确认诊断、悬停、跳转到定义、调试与性能分析全部正常工作——在**每一个** - 编辑器中都要确认。UX 的粗糙之处与特定平台的问题，是由人类操作真实 - UI 发现的，而非 CI。 -- **让你的团队每天都用它，并收集他们的反馈。** 内部试用（吃自家狗粮）是信号最强的 - 测试：把 Basilisk 放到真实的 Python 开发者面前，观察他们在哪里遇到摩擦， - 并把每一句“这在完全没问题的代码上报错了”或“这漏掉了一个明显的 bug”都变成一个 issue（§6） - 和一个失败的测试。目标是真实世界的采用，而不是绿色的 fixture。 - -### 2. 审计一致性得分的准确性 - -**确保我们为一致性打分的方式尽可能贴近官方的 -PEP 评分方式，并尽可能复用官方脚本。** 我们的数字只有在以 -Python typing 社区为每个检查器计算分数的方式来计算时，才有意义——因此夹在我们与官方工具之间的 -自研评分逻辑越少，我们的数字就越可信。 - -对照官方的 `python/typing` 一致性测试套件进行审计： - -- 完整的[测试套件](https://github.com/python/typing/tree/main/conformance/tests)（`conformance/tests/`）——我们是否运行了 PEP 体系所要求的**每一个** fixture，没有任何被悄悄跳过或排除？在测试的某个子集上取得的高百分比并不是真正的百分比。 -- [评分方法](https://github.com/python/typing/blob/main/conformance/README.md)（`conformance/README.md`）——我们的方法是否与之相符？ -- [Python 评分脚本](https://github.com/python/typing/blob/main/conformance/src/main.py)（`conformance/src/main.py`）——我们是否在某个固定提交上*原封不动*地运行*它*，而不是去重新实现它？每一处我们偏离的地方，都是我们的数字可能脱离现实的地方。 -- 已发布的[结果表](https://github.com/python/typing/blob/main/conformance/results/results.html)——那是我们想要榜上有名的记分牌；把我们的成绩与官方工具实际报告的结果做对比。 - -凡是我们所声称的与官方脚本所给出的之间存在任何差距，都要标记出来，并推动我们的工具链在目前尚未依赖该脚本的地方 -去依赖它。分数只能**向上**移动，并且只能因为我们确实 -变得更一致——目标是**100%**——绝不能因为我们改变了计数方式。 - -**为提高分数而禁用任何一致性规则是绝对禁止的——属于可处罚的违规行为。** -PEP 一致性测试必须在**启用每一条规则**的情况下运行 `basilisk` 二进制文件：不用任何 Basilisk 配置文件，不用 -按规则覆盖，不用“spec-conformance 模式”，不跳过任何 fixture，没有任何例外。分数正是 -真实用户开箱即得的结果。如果某个严格的默认规则在符合规范的代码上误报，就去**修复 -检查器**，让它不再误报——绝不能为了虚增数字而压制规则。这恰恰是当初那个数字被造假的方式：在评分时关闭了 -house-style 规则，以报告一个空洞的 100%，而误报只是被隐藏了起来，并未被消除。那种禁用做法已经被移除；通往 100% 的唯一 -正当途径，是在启用每一条规则的前提下让检查器变得更聪明（[CHKARCH-CONFORMANCE]）。 +## 给人类 + +你不需要会写 Rust 就能让 Basilisk 变得更好。**人类在这里能做的最高杠杆的事，是验证检查器是否真的在分析代码** —— 而不是验证某个数字涨了。 + +这不是假设。检查器逻辑曾被拟合到一致性测试夹具的具体内容上，由此得到的 100% 被公布出去，很久之后才被发现；两个已公布的数字 —— 一致性与性能 —— 现均已**撤回**。参见[一致性更正说明](https://www.basilisk-python.dev/docs/conformance/)与[完整性审计](docs/CONFORMANCE-INTEGRITY-AUDIT.md)。这一切都不是蓄意的 —— 是指令把分数定为目标，而匹配文本比分析代码更快地推高分数。按影响力大致排序： + +### 1. 亲自复核每一项指标 + +智能体会优化你所度量的一切，而这里的每个数字都可以在不做底层工作的情况下被推高：**一致性、覆盖率、变异分数、断言、lint、基准测试。** 相信任何指标变化之前，先自己重新推导一遍 —— 智能体不能给自己的作业打分。 + +需要留意的： + +- **基于文本匹配的逻辑** —— 最要命的一类。一条规则如果依据原始源文本或硬编码的符号拼写来判断，而不是依据 AST 上已解析的符号，它会得高分并在真实代码上失效。改一个导入别名（`from typing import Final as F`）或重新格式化文件：诊断结果必须不变。 +- **用沉默代替分析** —— 规则被禁用、删除或取消注册，好让它不再触发。 +- **被削弱的测试** —— 删掉失败的测试、砍掉断言，或把断言弱化到"绿灯"毫无意义。 +- **改动记分板或门禁** —— 手工编辑 `conformance_status.csv`，或调低阈值（`coverage-thresholds.json`、变异或基准基线）。 +- **少测一点** —— 排除诊断码、跳过夹具、收窄变异范围。在部分测试集上得到的高百分比不是百分比。 + +指标只能以*诚实*的方式移动 —— 覆盖率和变异分数向上，误报向下 —— 因为工作确实变好了，而不是因为有人改了计数方式（[CHKARCH-CONFORMANCE]）。 + +### 2. 用真实的大型代码库真刀真枪地测 + +自动化测试只能证明代码做了我们让它做的事，无法告诉你它在别人写的上百万行代码面前是否站得住。**把 Basilisk 对准真实世界：** + +- **在大型生产与开源代码库上运行** —— CPython 的 `Lib/`、Django、pandas、Home Assistant、SymPy、Sentry，*以及你自己最大的仓库*。夹具是整洁的，真实代码不是，而误报、崩溃、慢路径和漏报恰恰在那里浮现。 +- **安装已发布的构件**（不是开发构建）到一台干净的机器上，打开一个真实项目，确认诊断、悬停、跳转定义、调试和性能分析都能正常工作 —— 在**每一个**编辑器里。UX 与平台层面的问题只有人类操作真实界面才能发现。 +- **让你的团队每天用它，并收集反馈。** 把每一句"这在完全正确的代码上报错了"或"这漏掉了一个明显的错误"都变成 issue（§6）和一个失败的测试。 ### 3. 维护并提升代码质量 -对照 [`CLAUDE.md`](CLAUDE.md) 中的标准审查由 AI 编写的 PR：*这里的代码应当 -能够从容通过顶级工程组织的代码审查。* 揪出过度工程、 -过早抽象、重复逻辑，以及那些微妙地错误却看似合理的东西。智能体会 -乐于交付一些能编译、能通过测试但读起来很糟，或者埋着地雷的东西—— -你的工作就是把它说出来。 +按 [`CLAUDE.md`](CLAUDE.md) 中的标准审阅 AI 编写的 PR：*这里的代码应当能从容通过一流工程组织的评审。* 揪出过度设计、过早抽象、重复逻辑，以及那些"看着合理其实微妙地错了"的东西。智能体很乐意交付能编译、能过测试却埋着地雷的代码。 -### 4. 改进测试指标与变异测试得分 +### 4. 强化测试与变异分数 -覆盖率百分比是底线，而不是目标。判断断言究竟是*证明*了 -某件事，还是仅仅执行了一些代码行。推动更强的断言、扩大变异测试的 -范围（[CHKARCH-TESTING-MUTATION-RATCHET]），并指出那些即便代码已被破坏 -也仍会通过的测试。**两个棘轮都只朝一个方向移动**——覆盖率与变异测试得分向上， -绝不向下。 +覆盖率百分比是下限，不是目标。判断断言究竟是在*证明*什么，还是只是把代码行跑了一遍。推动更强的断言，扩大变异测试范围（[CHKARCH-TESTING-MUTATION-RATCHET]），并指出那些即使代码坏掉也照样通过的测试。两个棘轮都只能朝一个方向走。 -### 5. 守护性能数字 +### 5. 守住性能数字 -“最快”是承诺的另一半，而且同样很容易自欺欺人。在真实硬件上对照已提交的基线 -（`benchmarks/status/.csv`）重新运行基准测试， -确认没有任何 fixture 变慢，并在更新已记录的结果之前优化每一次退化。该门槛不能 -被禁用或放宽。一个冲破基准测试门槛的一致性修复并不算完成——而一个以一致性为代价的 -基准测试“胜利”也同样不算。两个棘轮要同时成立。 +性能是一项功能，但基准测试是**指示性的，不是门禁**（[CHKARCH-TESTING-BENCH]）。它跑在一台开发工作站上，与机器上其他一切负载共存，后台负载会让表中所有工具一起漂移。CI 中没有任何东西以基准数字判定成败，也不得重新引入这样的门禁。 -### 6. 报告 GitHub issue +只有人类能做这件事：在一台安静的机器上运行 `make bench`，在*同一次运行内*比较各个工具（它们背靠背计时，机器速度因此相互抵消），并深挖任何看起来不对劲的地方。绝不要拿一个数字去和另一台机器或另一个时间记录的数字相比。每次运行都会立即把结果写入 `benchmarks/status/.csv` —— 测了却不记录就是撒谎。 -正是你在用真实世界的 Python 跑 Basilisk。当出现问题时——一个误报、 -一个漏报、一次崩溃、一条慢路径、一次笨拙的编辑器交互——请提交一个精确、 -可复现的 issue，并附上能触发它的最小代码片段。一份好的 bug 报告是一份礼物；它 -会变成一个失败的测试，进而变成一个修复。 +### 6. 提交 GitHub issue -### 7. 对照现实检查计划与规范 +你才是那个用真实世界的 Python 跑 Basilisk 的人。当出现问题时 —— 误报、漏报、崩溃、慢路径、别扭的编辑器交互 —— 请用能触发它的最小代码片段提交一份精确、可复现的 issue。一份好的缺陷报告会变成一个失败的测试，再变成一个修复。 -规范与计划是本仓库的根基（参见 [`docs/INDEX.md`](docs/INDEX.md)）。审计它们： -是否每个规范小节都有一个非数字的、层次化的 spec ID？实现代码是否 -真的引用了那个 ID？实现是否*与*规范*相符*，还是已经漂移？这些 -计划是否仍然准确，还是它们描述的是一个已不复存在的世界？`/spec-check` 工作流 -有所帮助，但那个判断——*这条规范是否仍在讲真话？*——是你的责任。 +### 7. 用现实检验计划与规格 -### 8. 确保各 IDE 扩展之间的功能对等 +规格与计划是本仓库的骨架（见 [`docs/INDEX.md`](docs/INDEX.md)）。每个章节都有非数字的层级化 ID 吗？实现代码引用了它吗？实现与规格*一致*，还是已经漂移？`/spec-check` 能帮上忙，但判断 —— *这份规格是否仍在说真话？* —— 属于你。 -承诺是**在每一个编辑器中都有一致无缝的体验**：VS Code（外加通过 -Open VSX 的 Cursor/Windsurf）、Zed，以及 Neovim。一个落地在某个扩展上却没有落地在其他扩展上的功能，就是一个对等性 -bug。把这些扩展并排审计，找出差距，并把它们记录下来。记住架构 -规则：**LSP 驱动功能**——扩展只对 LSP 所宣告的内容做出反应。 +### 8. 确保各 IDE 扩展的功能对齐 + +我们的承诺是**在每个编辑器里都有同样顺滑的体验**：VS Code（以及通过 Open VSX 的 Cursor/Windsurf）、Zed、Neovim。一个功能只落在一个扩展里而其他没有，就是对齐缺陷。把它们并排审计，把缺口提出来。**LSP 驱动功能** —— 扩展只对它所声明的能力做出反应。 ### 9. 安全审计 -对类型检查器、LSP、编辑器扩展、发布流水线，以及 -依赖树进行威胁建模。带着人类对*何为真正重要*的直觉，审查 `/security-review` -与 Dependabot 所暴露出来的东西。单一二进制、无运行时、无遥测是一种安全姿态——帮助我们 -让它保持名副其实。 +对检查器、LSP、编辑器扩展、发布流水线和依赖树做威胁建模。用人类对*什么才真正重要*的判断力，审视 `/security-review` 与 Dependabot 暴露出来的东西。单一二进制、无运行时、无遥测本身就是一种安全姿态 —— 帮我们让它保持为真。 -### 10. 改进 AI 指令 +### 10. 改进给 AI 的指令 -这是**复利效应**最强的人类杠杆。更好的指令会让 AI 在 -未来的每一项任务上产出更好的结果。打磨 [`CLAUDE.md`](CLAUDE.md)、各项规范，以及 `.claude/` 下的技能。 -当你看到一个智能体出错时，修复点通常不在代码——而在于那条 -放任它如此行事的指令。 +这是**复利最高**的杠杆：更好的指令会让此后每一项任务的产出都更好。打磨 [`CLAUDE.md`](CLAUDE.md)、规格文档，以及 `.claude/` 下的技能。当你看到智能体走偏时，要修的通常不是代码 —— 而是那条允许它走偏的指令。 -### 11. 一切人类本就最擅长的事 +### 11. 一切人类天生更擅长的事 -品牌话语与命名。对外联络、人际关系与社区。战略性的优先级排序——*我们 -接下来到底应该构建什么？* 母语者与设计上的判断。任何涉及账户、 -密钥、令牌或资金的事情。如果它无法被一个测试检验，那它多半就是你的判断。 +品牌语调与命名。对外联络与社区。战略优先级 —— *我们接下来到底该做什么？* 母语者与设计判断。任何涉及账号、密钥、令牌或金钱的事。如果一个测试查不了它，那大概率就该你来定。 -### How to contribute as a human +### 人类该如何贡献 -1. **开一个 issue**，针对一个 bug、一处对等性差距、一处规范漂移，或一处一致性的不一致。要具体且可复现。 -2. **开一个 PR**，提交修复或文档——诚实地填写[pull request 模板](.github/pull_request_template.md)。对于*这些测试如何证明它能工作？*这个问题，“测试通过了”不是一个可接受的答案。 -3. **审查 PR**——审查本身就是一等的贡献，往往还是最有价值的那一个。 +1. **提交 issue** —— 缺陷、功能对齐缺口、规格漂移或检查器不准确。要具体、可复现。 +2. **提交 PR** —— 修复或文档，请诚实地填写 [pull request 模板](.github/pull_request_template.md)。对*这些测试如何证明它能工作？*，"测试通过了"不是答案。 +3. **评审 PR** —— 一等的贡献，往往也是最有价值的一种。 --- -## For AI - -技术工作大多由你完成。管辖这些工作的常设规则位于 -[**`CLAUDE.md`**](CLAUDE.md)，它们会**覆盖默认行为**——先读那个文件，并 -严格遵照执行。本节是一张地图，而非重述（我们不做重复）。 - -**在你触碰任何东西之前：** - -- 完整阅读 [`CLAUDE.md`](CLAUDE.md)。然后通过 [`docs/INDEX.md`](docs/INDEX.md) 与 - 作为单一事实来源的规范 [`docs/specs/LSP-ARCHITECTURE-SPEC.md`](docs/specs/LSP-ARCHITECTURE-SPEC.md) 来定位方向。 -- 向 **too-many-cooks** 协调器注册，并在编辑文件之前**锁定文件**。不要 - 编辑已被锁定的文件。 - -**不可妥协之事**（完整细节见 `CLAUDE.md`）： - -- **未经明确要求，禁止使用 Git。** 绝不向 `main` 推送，绝不把智能体列为 - 共同作者，绝不使用 worktree，只在恰好一个分支上工作。 -- **Spec ID 是根基。** 每个规范小节都有一个非数字的、层次化的 ID；每一段 - 代码都引用它（`// Implements [LSP-…]`）；每个测试都交叉引用二者。如果你发现某个 - 链接缺失，就修好它。 -- **无情地遵循 DRY。** 使用 `deslop` MCP（写之前用 `find-similar`，写之后用 `top-offenders`）。 - 合并重复项。在添加新代码之前先搜索已有代码。 -- **棘轮只朝一个方向移动。** 一致性得分向上；误报与基准测试 - 退化向下；覆盖率向上；变异测试得分向上。一个冲破基准测试门槛的一致性修复 - 并不算完成。 -- **绝不为提高分数而禁用一致性规则——属于可处罚的违规行为。** PEP 一致性测试在**启用每一条规则**的 - 情况下运行 `basilisk` 二进制文件：不用任何 Basilisk 配置文件，不用按规则覆盖，不用 - “spec-conformance 模式”，不跳过任何 fixture，没有任何例外。分数正是真实用户开箱 - 即得的结果。如果某个严格的默认规则在符合规范的代码上误报，就去**修复检查器**，让它不再 - 误报——绝不能为了虚增数字而压制规则（[CHKARCH-CONFORMANCE]）。 -- **`make` 是统一接口。** `make build | test | lint | fmt | clean | ci | setup`——恰好七个 - 目标，不要再添加。`make test` 是快速失败的，并强制执行来自 - `coverage-thresholds.json` 的覆盖率门槛。 -- **Rust 质量标准：** 不允许 `unwrap`、`panic!`、`todo!`、`unimplemented!`、`unsafe` 或 - `allow(clippy::…)`。处处使用 `Result`/`Option`，小而纯的函数，文件保持在 500 LOC 以下。 -- **不存 CI 构件。** 即便在这个公开仓库上存储也是要计费的——参见 [GITHUB-NO-ARTIFACTS]。 +## 给 AI + +你把规格转化为代码和测试，并让三者保持同步。约束你的规则位于 [**`CLAUDE.md`**](CLAUDE.md)，它们**覆盖默认行为** —— 先读它，并严格照做。 + +**动手之前：** 完整读完 [`CLAUDE.md`](CLAUDE.md)，再通过 [`docs/INDEX.md`](docs/INDEX.md) 和 [`docs/specs/LSP-ARCHITECTURE-SPEC.md`](docs/specs/LSP-ARCHITECTURE-SPEC.md) 建立方位感，然后到 **too-many-cooks** 协调器注册，并在编辑前**锁定文件**。绝不要编辑已被锁定的文件。 + +**准确性是最高指令。** Basilisk 必须在它从未见过的 Python 上给出正确结果。每条规则都从已解析的 AST 做判断，绝不依据源码碰巧怎么写。当你发现一条规则依赖原始文本、硬编码的符号拼写或某个一致性测试夹具时，只做三件事 —— **不要修它，不要重写它，不要留 TODO**： + +1. **写一个会失败的测试**，让它因为这段错误代码而失败。 +2. **删除这段有问题的代码。** +3. **告诉用户你删了什么、为什么删。** + +要重建什么，由用户决定，不由你决定。 + +**不可协商的底线**（详见 `CLAUDE.md`）： + +- **未经明确要求，不许碰 Git。** 绝不推送到 `main`，绝不把智能体列为共同作者，绝不使用 worktree，只在一个分支上工作。 +- **规格 ID 是骨架。** 每个规格章节都有非数字的层级化 ID；代码引用它（`// Implements [LSP-…]`）；测试同时交叉引用两者。发现链接缺失就补上。 +- **无情地 DRY。** `deslop` MCP：写代码前 `find-similar`，改完后 `top-offenders`。新增代码前先搜索已有代码。 +- **棘轮只朝一个方向走** —— 覆盖率与变异分数向上，误报向下。一致性是回归探测器，不是目标；基准时间不构成任何门禁（[CHKARCH-TESTING-BENCH]）。 +- **绝不碰记分板。** 一致性测试运行的二进制**启用全部规则**：没有配置文件，没有按规则的覆写，不跳过夹具，不为躲避失败而删除源码，不从 `all_rules()` 中移除规则。同样禁止：手工编辑 `conformance/conformance_status.csv` 或放宽 `coverage-thresholds.json`。绝不发布或引用任何一致性数字（[CHKARCH-CONFORMANCE]）。 +- **`make` 就是接口。** `make build | test | lint | fmt | clean | ci | setup` —— 恰好七个目标，不要再加。`make test` 快速失败并强制执行覆盖率阈值。 +- **Rust 质量标准：** 不许 `unwrap`、`panic!`、`todo!`、`unimplemented!`、`unsafe` 或 `allow(clippy::…)`。到处用 `Result`/`Option`，小而纯的函数，文件不超过 500 行。 +- **不产出 CI 构件。** 即便是公开仓库，存储也是计费的 —— 见 [GITHUB-NO-ARTIFACTS]。 **你的工作方式：** -- **始终测试驱动。** 写一个失败的测试 → 确认它*因正确的原因*而失败 → 修复 - 代码（绝不改测试）→ 确认它通过。只用粗粒度的 e2e 测试；不用单元测试。绝不删除一个 - 失败的测试，也绝不削弱一条断言。 -- **运用判断力；不要停下来发问。** 自行决断并继续推进。 -- **从 [`docs/plans/ROADMAP-NEXT-STEPS-PLAN.md`](docs/plans/ROADMAP-NEXT-STEPS-PLAN.md) 中领取 `[AGENT]` 工作。** - 把 `[HUMAN]` 工作留给人类。起草 `[HYBRID]` 条目中属于智能体的那一半，然后移交出去。 -- **遵从本指南中的人类信号。** 当一个人类报告了一个 issue 时，那份报告就成了 - 你那个失败的测试。 +- **永远测试驱动。** 失败的测试 → 确认它*因为正确的原因*失败 → 修代码（绝不修测试）→ 确认通过。只写粗粒度的端到端测试。绝不删除失败的测试或弱化断言。 +- **自己判断，不要停下来提问。**（报告一次删除不算提问 —— 报告完继续做。） +- **认领 `[AGENT]` 工作**，来源 [`docs/plans/ROADMAP-NEXT-STEPS-PLAN.md`](docs/plans/ROADMAP-NEXT-STEPS-PLAN.md)。把 `[HUMAN]` 留给人类。`[HYBRID]` 项里属于智能体的那一半先起草，然后移交。 +- **服从本指南里的人类信号。** 人类报告的问题，就是你要写的那个失败测试。 diff --git a/README-pypi.md b/README-pypi.md index b4e695c07..75f46888b 100644 --- a/README-pypi.md +++ b/README-pypi.md @@ -10,9 +10,8 @@

English · 简体中文

- The only Python type checker scoring 100% on the official python/typing conformance suite — and the fastest we’ve measured.
- Complete open-source Python dev environment in Rust: type checker, language server, debugger, profiler, plus VS Code, Cursor, Zed & Neovim extensions. Strict by default. - Weighing up the best Python type checker for your codebase? Start with the scoreboard. + An open-source Python type checker and language server, built in Rust.
+ One extension for the whole workflow — diagnostics, autocomplete, refactoring, formatting, debugging, and profiling — driven by a single bundled binary.

> **You are reading the `basilisk-python` wheel listing** — the Basilisk CLI packaged for `pip`/`uv`. The distribution is named `basilisk-python` because `basilisk` was taken on PyPI; the installed command is still `basilisk`. @@ -23,55 +22,50 @@ Quick Start  •  Rules  •  Refactoring  •  - Compare  •  GitHub

- 100.0% PEP conformance — 141 of 141 tests in the official - python/typing - conformance suite (commit a490662), scored on the wheel-installed CLI in its default config by the real upstream harness. - We target python/typing@main and ratchet the score up only. + Basilisk in action — type checking, diagnostics, and refactoring in the editor

-## The only 100% checker — and the fastest according to our benchmarks - -Basilisk is the **only** Python type checker with a perfect score on the official -[`python/typing` conformance suite](https://github.com/python/typing/blob/main/conformance/results/results.html): -**100.0%** (141/141 files, 970 required errors caught, 0 false positives), -measured by the real upstream harness on the wheel-installed CLI in its default config. +**The current type checker contains inaccuracies and you should not use it as part of your dev pipeline. We are working on removing any misleading analyzers ASAP. Please read below** -

- Basilisk in action — type checking, diagnostics, and refactoring in the editor -

+## We withdrew the typing conformance results -And it is the **fastest checker we’ve measured** — median cold full-file check, from scratch: +We withdrew our 100% conformance claim and our benchmark figures, and asked to be +[removed from the official `python/typing` results](https://github.com/python/typing/blob/main/conformance/results/results.html). +The cause was checker logic fitted to the contents of conformance test files +instead of implementing the typing specification generally, and a score produced +that way is not evidence. The current percentage is **temporarily unknown**. -| Type checker | Median cold check | -| --- | --- | -| ⚡ **Basilisk** | **12 ms** | -| zuban | 28 ms | -| ty | 39 ms | -| Pyrefly | 111 ms | -| Pyright | 582 ms | -| mypy | 605 ms | +We are deciding whether to rebuild the checker from the specification or drive +the extension with an established open-source checker. **Either way, we are +building Basilisk into an accurate Python development experience** — and a new +figure gets published only once it survives off-suite and mutation testing. -Median cold full-file check across 26 single-construct typing-spec stress fixtures on an Apple M4 Max — lower is better. Basilisk’s warm re-check drops to ~5 ms. Every figure is produced by [`hyperfine`](https://github.com/sharkdp/hyperfine) and committed per machine, so nothing here is hand-typed. **Clone the repo, run `make bench` on your own hardware, and send us the CSV — independent audits are welcome.** [Full benchmarks & methodology →](https://www.basilisk-python.dev/docs/benchmarks/) +[Read the full correction →](https://www.basilisk-python.dev/docs/conformance/)  •  +[Integrity audit →](https://github.com/Nimblesite/Basilisk/blob/main/docs/CONFORMANCE-INTEGRITY-AUDIT.md) -## Everything in one extension +## What you get -One extension replaces Pylance and gives you the whole workflow — no Node.js, no Python runtime, no pip, no npm. A single bundled Rust binary drives it all: +One extension covers the whole Python workflow. A single bundled Rust binary +drives it — no Node.js, no npm, no `pip install`: -- **Strict-by-default diagnostics** — inline as you type, incremental analysis powered by Salsa (the rust-analyzer engine) +- **Diagnostics as you type** — incremental analysis powered by [Salsa](https://github.com/salsa-rs/salsa) - **Autocomplete, hover, go-to-definition, find references, rename** - **Refactoring code actions** — extract, inline, move symbol, organize imports -- **Integrated debugging** — F5 to debug via bundled debugpy; no separate extension +- **Integrated debugging** — F5 to debug via bundled [debugpy](https://github.com/microsoft/debugpy); no separate extension - **Integrated profiling** — CPU heat map, flame graph, and a memory dashboard with leak detection - **Activity panel** — module tree with per-module type-health coverage, plus feature toggles - **Inlay hints** and **Ruff** formatting/import-organization, built in - **Standard-library types from [typeshed](https://github.com/python/typeshed)** — a complete `stdlib/` snapshot is compiled into the binary, so hover and diagnostics work offline with no configuration -Every diagnostic teaches: rustc-style output with a `help`, a `note`, and a link to a per-rule explainer, so a red squiggle always tells you *why*. Basilisk **starts strict** and stays strict — the unconfigured default enables the complete typing-spec rule set, and strictness is dialled per rule, never by a mode. +Strictness is configured **per rule**, never by a mode: the unconfigured default +enables the typing-spec rule set, and each rule can be graded down to +`warning`/`info` so a codebase can adopt type safety incrementally. Every +diagnostic carries a `help`, a `note`, and a link to a per-rule explainer, so a +red squiggle tells you *why*. ## Install diff --git a/README.md b/README.md index 8f676ef8c..c1d481420 100644 --- a/README.md +++ b/README.md @@ -10,68 +10,59 @@

English · 简体中文

- The only Python type checker scoring 100% on the official python/typing conformance suite — and the fastest we’ve measured.
- Complete open-source Python dev environment in Rust: type checker, language server, debugger, profiler, plus VS Code, Cursor, Zed & Neovim extensions. Strict by default. - Weighing up the best Python type checker for your codebase? Start with the scoreboard. + An open-source Python type checker and language server, built in Rust.
+ One extension for the whole workflow — diagnostics, autocomplete, refactoring, formatting, debugging, and profiling — driven by a single bundled binary.

-> **You are reading the Basilisk source repository** — the checker, language server, editor extensions, and website all live here. -

Website  •  Install  •  Quick Start  •  Rules  •  Refactoring  •  - Compare  •  GitHub

- 100.0% PEP conformance — 141 of 141 tests in the official - python/typing - conformance suite (commit a490662), scored on the wheel-installed CLI in its default config by the real upstream harness. - We target python/typing@main and ratchet the score up only. + Basilisk in action — type checking, diagnostics, and refactoring in the editor

-## The only 100% checker — and the fastest according to our benchmarks - -Basilisk is the **only** Python type checker with a perfect score on the official -[`python/typing` conformance suite](https://github.com/python/typing/blob/main/conformance/results/results.html): -**100.0%** (141/141 files, 970 required errors caught, 0 false positives), -measured by the real upstream harness on the wheel-installed CLI in its default config. +**The current type checker contains inaccuracies and you should not use it as part of your dev pipeline. We are working on removing any misleading analyzers ASAP. Please read below** -

- Basilisk in action — type checking, diagnostics, and refactoring in the editor -

+## We withdrew the typing conformance results -And it is the **fastest checker we’ve measured** — median cold full-file check, from scratch: +We withdrew our 100% conformance claim and our benchmark figures, and asked to be +[removed from the official `python/typing` results](https://github.com/python/typing/blob/main/conformance/results/results.html). +The cause was checker logic fitted to the contents of conformance test files +instead of implementing the typing specification generally, and a score produced +that way is not evidence. The current percentage is **temporarily unknown**. -| Type checker | Median cold check | -| --- | --- | -| ⚡ **Basilisk** | **12 ms** | -| zuban | 28 ms | -| ty | 39 ms | -| Pyrefly | 111 ms | -| Pyright | 582 ms | -| mypy | 605 ms | +We are +building Basilisk into an accurate Python development experience** — and a new +figure gets published only once it survives off-suite and mutation testing. -Median cold full-file check across 26 single-construct typing-spec stress fixtures on an Apple M4 Max — lower is better. Basilisk’s warm re-check drops to ~5 ms. Every figure is produced by [`hyperfine`](https://github.com/sharkdp/hyperfine) and committed per machine, so nothing here is hand-typed. **Clone the repo, run `make bench` on your own hardware, and send us the CSV — independent audits are welcome.** [Full benchmarks & methodology →](https://www.basilisk-python.dev/docs/benchmarks/) +[Read the full correction →](https://www.basilisk-python.dev/docs/conformance/)  •  +[Integrity audit →](docs/CONFORMANCE-INTEGRITY-AUDIT.md) -## Everything in one extension +## What you get -One extension replaces Pylance and gives you the whole workflow — no Node.js, no Python runtime, no pip, no npm. A single bundled Rust binary drives it all: +One extension covers the whole Python workflow. A single bundled Rust binary +drives it — no Node.js, no npm, no `pip install`: -- **Strict-by-default diagnostics** — inline as you type, incremental analysis powered by Salsa (the rust-analyzer engine) +- **Diagnostics as you type** — incremental analysis powered by [Salsa](https://github.com/salsa-rs/salsa) - **Autocomplete, hover, go-to-definition, find references, rename** - **Refactoring code actions** — extract, inline, move symbol, organize imports -- **Integrated debugging** — F5 to debug via bundled debugpy; no separate extension +- **Integrated debugging** — F5 to debug via bundled [debugpy](https://github.com/microsoft/debugpy); no separate extension - **Integrated profiling** — CPU heat map, flame graph, and a memory dashboard with leak detection - **Activity panel** — module tree with per-module type-health coverage, plus feature toggles - **Inlay hints** and **Ruff** formatting/import-organization, built in - **Standard-library types from [typeshed](https://github.com/python/typeshed)** — a complete `stdlib/` snapshot is compiled into the binary, so hover and diagnostics work offline with no configuration -Every diagnostic teaches: rustc-style output with a `help`, a `note`, and a link to a per-rule explainer, so a red squiggle always tells you *why*. Basilisk **starts strict** and stays strict — the unconfigured default enables the complete typing-spec rule set, and strictness is dialled per rule, never by a mode. +Strictness is configured **per rule**, never by a mode: the unconfigured default +enables the typing-spec rule set, and each rule can be graded down to +`warning`/`info` so a codebase can adopt type safety incrementally. Every +diagnostic carries a `help`, a `note`, and a link to a per-rule explainer, so a +red squiggle tells you *why*. ## Install diff --git a/README.zh.md b/README.zh.md index 0185a6624..005ba7b78 100644 --- a/README.zh.md +++ b/README.zh.md @@ -10,8 +10,8 @@

English · 简体中文

- 唯一在官方 python/typing 一致性测试套件上取得 100% 的 Python 类型检查器 —— 也是我们测得最快的。
- 用 Rust 打造的完整开源 Python 开发环境：类型检查器、语言服务器、调试器、性能分析器，以及 VS Code、Cursor、Zed 与 Neovim 扩展。默认严格。 + 用 Rust 打造的开源 Python 类型检查器与语言服务器。
+ 一个扩展覆盖整套工作流 —— 诊断、自动补全、重构、格式化、调试与性能分析 —— 全部由单一捆绑的二进制文件驱动。

> **你正在阅读 Basilisk 的源码仓库** —— 检查器、语言服务器、编辑器扩展与网站都在这里。 @@ -22,57 +22,47 @@ 快速上手  •  规则  •  重构  •  - 对比  •  GitHub

- PEP 一致性 100.0% — 官方 - python/typing - 一致性套件（提交 a490662）141 项测试中通过 141 项， - 由真实的上游评分器在默认配置下对 wheel 安装的 CLI 评出。 - 我们以 python/typing@main 为目标，且分数只升不降。 + Basilisk 实战 —— 编辑器中的类型检查、诊断与重构

-## 唯一 100% 的检查器 — 按我们的基准测试也是最快的 - -Basilisk 是**唯一**在官方 -[`python/typing` 一致性套件](https://github.com/python/typing/blob/main/conformance/results/results.html) -上取得满分的 Python 类型检查器：**100.0%** -（141/141 个文件，捕获 970 处必需错误，0 个误报）， -由真实的上游评分器在默认配置下对 wheel 安装的 CLI 测得。 +**当前的类型检查器存在不准确之处，请勿将其用于你的开发流水线。我们正在尽快移除任何具有误导性的分析器。详情请见下文** -

- Basilisk 实战 —— 编辑器中的类型检查、诊断与重构 -

+## 我们撤回了类型一致性结果 -它也是**我们测得最快的检查器** — 从零开始的整文件冷检查中位数： +我们撤回了 100% 的一致性宣称与基准测试数字，并主动请求 +[从官方 `python/typing` 结果中移除](https://github.com/python/typing/blob/main/conformance/results/results.html)。 +原因是检查器中存在针对一致性测试文件内容而写的逻辑，而不是对类型规范的通用实现； +这样得出的分数并不能作为证据。当前的百分比**暂时未知**。 -| 类型检查器 | 冷检查中位数 | -| --- | --- | -| ⚡ **Basilisk** | **12 ms** | -| zuban | 28 ms | -| ty | 39 ms | -| Pyrefly | 111 ms | -| Pyright | 582 ms | -| mypy | 605 ms | +我们正在决定是按规范重建检查器，还是让扩展由一个成熟的开源检查器驱动。**无论走哪 +条路，我们都在把 Basilisk 打造成准确的 Python 开发体验** —— 而新的数字只有在经受住 +套件之外的用例与变异测试后才会发布。 -在 Apple M4 Max 上对 26 个单一构造的类型规范压力用例测得的整文件冷检查中位数 — 越低越好。Basilisk 的热重检查可降至约 5 ms。每个数字都由 [`hyperfine`](https://github.com/sharkdp/hyperfine) 产生并按机器提交，没有一个是手写的。**克隆仓库，在你自己的硬件上运行 `make bench`，并把 CSV 发给我们 — 欢迎独立复核。** [完整基准与方法论 →](https://www.basilisk-python.dev/zh/docs/benchmarks/) +[阅读完整更正 →](https://www.basilisk-python.dev/zh/docs/conformance/)  •  +[完整性审计 →](docs/CONFORMANCE-INTEGRITY-AUDIT.md) -## 一个扩展，覆盖全部 +## 你能得到什么 -一个扩展即可取代 Pylance 并提供完整工作流 —— 无需 Node.js、无需 Python 运行时、无需 pip、无需 npm。一切由单一捆绑的 Rust 二进制文件驱动： +一个扩展即可覆盖整套 Python 工作流。一切由单一捆绑的 Rust 二进制文件驱动 —— +无需 Node.js、无需 npm、无需 `pip install`： -- **默认严格的诊断** —— 随输入实时呈现，由 Salsa（rust-analyzer 的引擎）提供增量分析 +- **随输入实时诊断** —— 由 [Salsa](https://github.com/salsa-rs/salsa) 提供增量分析 - **自动补全、悬停信息、跳转到定义、查找引用、重命名** - **重构代码操作** —— 提取、内联、移动符号、整理导入 -- **集成调试** —— 按 F5 即可通过捆绑的 debugpy 调试；无需额外扩展 +- **集成调试** —— 按 F5 即可通过捆绑的 [debugpy](https://github.com/microsoft/debugpy) 调试；无需额外扩展 - **集成性能分析** —— CPU 热力图、火焰图，以及带泄漏检测的内存面板 - **活动面板** —— 模块树与逐模块的类型健康度覆盖率，并可切换功能开关 - 内置 **Inlay hints** 与 **Ruff** 格式化／导入整理 - **来自 [typeshed](https://github.com/python/typeshed) 的标准库类型** —— 完整的 `stdlib/` 快照已编译进二进制文件，因此悬停与诊断在离线且零配置的情况下依然可用 -每条诊断都有教育意义：rustc 风格的输出，附带 `help`、`note` 以及指向每条规则详解页的链接，因此一条红色波浪线总能告诉你*为什么*。Basilisk **一开始就严格**并始终严格 —— 未配置的默认值即启用完整的类型规范规则集，严格程度按规则微调，而不是靠模式切换。 +严格程度按**规则**配置，而不是靠模式切换：未配置的默认值即启用类型规范规则集， +每条规则都可以降级为 `warning`/`info`，让代码库能够渐进地采用类型安全。每条诊断 +都附带 `help`、`note` 以及指向每条规则详解页的链接，因此一条红色波浪线总能告诉你 +*为什么*。 ## 安装 diff --git a/basilisk-zed/README.md b/basilisk-zed/README.md index 2ac6c39db..18dc2fb0e 100644 --- a/basilisk-zed/README.md +++ b/basilisk-zed/README.md @@ -4,12 +4,16 @@ Zed editor extension for Basilisk — WASM-based Python type checking and language server integration. -Basilisk is the only Python type checker scoring 100% on the [official `python/typing` conformance suite](https://github.com/python/typing/blob/main/conformance/results/results.html) — and the fastest we've measured. A complete, open-source Python dev environment in Rust: type checker, language server, debugger, profiler, plus VS Code, Cursor, Zed & Neovim extensions. Strict by default. +Basilisk is an open-source Python type checker and language server built in Rust: diagnostics, autocomplete, refactoring, debugging, and profiling, with strictness configured per rule.

Basilisk in the Zed editor — Python type checking and diagnostics inline

+**The current type checker contains inaccuracies and you should not use it as part of your dev pipeline. We are working on removing any misleading analyzers ASAP. Please read below** + +> **We withdrew the typing conformance results.** The 100% claim and the benchmark figures are retracted, and Basilisk was [removed from the official results](https://github.com/python/typing/blob/main/conformance/results/results.html) at our request. Whether we rebuild the checker from the specification or drive the extension with an established open-source checker, we are building Basilisk into an accurate Python development experience. [Read the correction](https://www.basilisk-python.dev/docs/conformance/). + ## Install Command palette (`Cmd+Shift+P` / `Ctrl+Shift+P`) → **zed: install dev extension** → select this directory (clone [`Nimblesite/basilisk-zed`](https://github.com/Nimblesite/basilisk-zed) first if you do not have the monorepo). Zed compiles the extension to WASM itself — you never pre-build or copy a `.wasm` file. diff --git a/basilisk-zed/README.zh.md b/basilisk-zed/README.zh.md index 74a3e54e2..0b479daa4 100644 --- a/basilisk-zed/README.zh.md +++ b/basilisk-zed/README.zh.md @@ -6,12 +6,16 @@ Basilisk 的 Zed 编辑器扩展 —— 基于 WASM 的 Python 类型检查与语言服务器集成。 -唯一在官方 [`python/typing` 符合性套件](https://github.com/python/typing/blob/main/conformance/results/results.html)中取得 100% 满分的 Python 类型检查器 —— 也是我们测过的最快的。使用 Rust 构建的完整开源 Python 开发环境：类型检查器、语言服务器、调试器与性能分析器，并提供 VS Code、Cursor、Zed 与 Neovim 扩展。默认严格。 +Basilisk 是用 Rust 打造的开源 Python 类型检查器与语言服务器：诊断、自动补全、重构、调试与性能分析，严格程度按规则配置。

Zed 编辑器中的 Basilisk —— 行内 Python 类型检查与诊断

+**当前的类型检查器存在不准确之处，请勿将其用于你的开发流水线。我们正在尽快移除任何具有误导性的分析器。详情请见下文** + +> **我们撤回了类型一致性结果。** 100% 的宣称与基准测试数字均已撤回，并主动请求[从官方结果中移除](https://github.com/python/typing/blob/main/conformance/results/results.html)。无论是按规范重建检查器，还是让扩展由一个成熟的开源检查器驱动，我们都在把 Basilisk 打造成准确的 Python 开发体验。[阅读更正](https://www.basilisk-python.dev/zh/docs/conformance/)。 + ## 安装 命令面板（`Cmd+Shift+P` / `Ctrl+Shift+P`）→ **zed: install dev extension** → 选择本目录（如果没有 monorepo，请先克隆 [`Nimblesite/basilisk-zed`](https://github.com/Nimblesite/basilisk-zed)）。Zed 会自行把扩展编译为 WASM —— 你无需预先构建或复制 `.wasm` 文件。 diff --git a/basilisk-zed/extension.toml b/basilisk-zed/extension.toml index e47942a28..c688aee70 100644 --- a/basilisk-zed/extension.toml +++ b/basilisk-zed/extension.toml @@ -5,7 +5,7 @@ name = "Basilisk" version = "0.0.0-PLACEHOLDER" schema_version = 1 authors = ["Basilisk Contributors"] -description = "The only Python type checker scoring 100% on the official python/typing conformance suite (https://github.com/python/typing/blob/main/conformance/results/results.html) — and the fastest we've measured. Complete open-source Python dev environment in Rust: type checker, language server, debugger, profiler, plus VS Code, Cursor, Zed & Neovim extensions. Strict by default." +description = "An open-source Python type checker and language server built in Rust: diagnostics, autocomplete, go-to-definition, refactoring, integrated debugging, and profiling. Strictness is configured per rule, so a codebase can adopt type safety incrementally." repository = "https://github.com/Nimblesite/Basilisk" # No [grammars.*] and no languages/ directory: Basilisk attaches to Zed's diff --git a/basilisk.nvim/README.md b/basilisk.nvim/README.md index f1183df18..10652da4e 100644 --- a/basilisk.nvim/README.md +++ b/basilisk.nvim/README.md @@ -4,12 +4,16 @@ First-class Neovim plugin for Basilisk — zero-config Python type checking, debugging, profiling, and test exploration. -Basilisk is the only Python type checker scoring 100% on the [official `python/typing` conformance suite](https://github.com/python/typing/blob/main/conformance/results/results.html) — and the fastest we've measured. A complete, open-source Python dev environment in Rust: type checker, language server, debugger, profiler, plus VS Code, Cursor, Zed & Neovim extensions. Strict by default. +Basilisk is an open-source Python type checker and language server built in Rust: diagnostics, autocomplete, refactoring, debugging, and profiling, with strictness configured per rule.

Basilisk in action — type checking, diagnostics, and refactoring in the editor

+**The current type checker contains inaccuracies and you should not use it as part of your dev pipeline. We are working on removing any misleading analyzers ASAP. Please read below** + +> **We withdrew the typing conformance results.** The 100% claim and the benchmark figures are retracted, and Basilisk was [removed from the official results](https://github.com/python/typing/blob/main/conformance/results/results.html) at our request. Whether we rebuild the checker from the specification or drive the extension with an established open-source checker, we are building Basilisk into an accurate Python development experience. [Read the correction](https://www.basilisk-python.dev/docs/conformance/). + ## Role in Basilisk This is the **Neovim editor integration**. It connects Neovim's built-in LSP client to the Basilisk language server, providing the same feature set as the VS Code extension: real-time diagnostics, hover, go-to-definition, code actions, inlay hints, integrated debugging, and profiling. diff --git a/basilisk.nvim/README.zh.md b/basilisk.nvim/README.zh.md index 01bf836e8..b6cc8865c 100644 --- a/basilisk.nvim/README.zh.md +++ b/basilisk.nvim/README.zh.md @@ -6,12 +6,16 @@ 为 Basilisk 打造的一流 Neovim 插件 —— 零配置的 Python 类型检查、调试、性能分析与测试探索。 -唯一在官方 [`python/typing` 符合性套件](https://github.com/python/typing/blob/main/conformance/results/results.html)中取得 100% 满分的 Python 类型检查器 —— 也是我们测过的最快的。使用 Rust 构建的完整开源 Python 开发环境：类型检查器、语言服务器、调试器与性能分析器，并提供 VS Code、Cursor、Zed 与 Neovim 扩展。默认严格。 +Basilisk 是用 Rust 打造的开源 Python 类型检查器与语言服务器：诊断、自动补全、重构、调试与性能分析，严格程度按规则配置。

Basilisk in action — type checking, diagnostics, and refactoring in the editor

+**当前的类型检查器存在不准确之处，请勿将其用于你的开发流水线。我们正在尽快移除任何具有误导性的分析器。详情请见下文** + +> **我们撤回了类型一致性结果。** 100% 的宣称与基准测试数字均已撤回，并主动请求[从官方结果中移除](https://github.com/python/typing/blob/main/conformance/results/results.html)。无论是按规范重建检查器，还是让扩展由一个成熟的开源检查器驱动，我们都在把 Basilisk 打造成准确的 Python 开发体验。[阅读更正](https://www.basilisk-python.dev/zh/docs/conformance/)。 + ## 在 Basilisk 中的角色 这是 **Neovim 编辑器集成**。它将 Neovim 内置的 LSP 客户端连接到 Basilisk 语言服务器，提供与 VS Code 扩展相同的功能集：实时诊断、悬停信息、跳转到定义、代码操作、内嵌提示（inlay hints）、集成调试以及性能分析。 diff --git a/crates/basilisk-cli/Cargo.toml b/crates/basilisk-cli/Cargo.toml index d5dab931a..ee37495e2 100644 --- a/crates/basilisk-cli/Cargo.toml +++ b/crates/basilisk-cli/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "basilisk-cli" -description = "Basilisk CLI — the only Python type checker scoring 100% on the official python/typing conformance suite (https://github.com/python/typing/blob/main/conformance/results/results.html) — and the fastest we've measured. Complete open-source Python dev environment in Rust: type checker, language server, debugger, profiler, plus VS Code, Cursor, Zed & Neovim extensions. Strict by default." +description = "Basilisk CLI — an open-source Python type checker and language server built in Rust: diagnostics, refactoring, formatting, integrated debugging, and profiling. Strictness is configured per rule, so a codebase can adopt type safety incrementally." version.workspace = true edition.workspace = true license.workspace = true diff --git a/docs/plans/ROADMAP-NEXT-STEPS-PLAN.md b/docs/plans/ROADMAP-NEXT-STEPS-PLAN.md index 60a9dd825..682ed6859 100644 --- a/docs/plans/ROADMAP-NEXT-STEPS-PLAN.md +++ b/docs/plans/ROADMAP-NEXT-STEPS-PLAN.md @@ -55,37 +55,24 @@ responsibility to cover. - [ ] **`[HYBRID]`** Validate a tagged release end to end with the real `VSCODE_MARKETPLACE_PAT`, `OPEN_VSX_PAT`, and mirror credentials, then install each published artifact on a clean machine. -- [ ] **`[HYBRID]`** Submit the standalone `Nimblesite/basilisk-zed` mirror to - [`zed-industries/extensions`](https://github.com/zed-industries/extensions). - Never submitted to date — Basilisk does not appear in that repo's - `extensions.toml`, so the Zed extensions view cannot find it and - [install-zed](../../website/src/docs/install-zed.md) documents the dev-extension - flow until the listing lands. Automation already renders, version-stamps, and - WASM-gates the mirror ([ZED-MIRROR](../specs/ZED-SPEC.md#ZED-MIRROR)); only the - one-time human-reviewed PR is missing. The `basilisk` extension ID is +- [ ] **`[HUMAN]`** Get the `zed-industries/extensions` listing PR merged. + Basilisk still does not appear in that repo's `extensions.toml`, so the Zed + extensions view cannot find it and + [install-zed](../../website/src/docs/install-zed.md) documents the + dev-extension flow until the listing lands. The `basilisk` extension ID is unclaimed upstream. - **Blocked until the next tagged release.** The published mirror at `v0.39.0` - still carries the `languages/python/` tree that shadows Zed's built-in Python - ([ZED-TREESITTER](../specs/ZED-SPEC.md#ZED-TREESITTER)); listing it would ship - that regression to every installer. Cut a release first so `publish-zed` - pushes a mirror without it, then pin *that* commit. - - Procedure, once the fixed mirror is tagged: - 1. Fork `zed-industries/extensions`. - 2. `git submodule add https://github.com/Nimblesite/basilisk-zed.git extensions/basilisk` - (HTTPS, not SSH — the registry requires it), then check the submodule out at - the release tag so the pinned commit is the version being listed. - 3. Add to `extensions.toml`: - `[basilisk]` / `submodule = "extensions/basilisk"` / `version = ""`. - The version MUST equal `extension.toml`'s stamped version. - 4. `pnpm sort-extensions` to normalise ordering in `extensions.toml` and - `.gitmodules`. - 5. Open the PR. A root `LICENSE` is mandatory for listings (MIT is already - rendered into the mirror by `render-zed-mirror.sh`). - - Subsequent version bumps amend the submodule pointer and the `version` field — - the same PR shape, or the community update action. + **No longer a manual procedure.** `publish-zed` now forks the registry, pins + the `extensions/basilisk` submodule to the release tag, sets the version, and + opens (or re-points) the PR on every tagged release, via + `scripts/publish_zed_registry.py` ([ZED-MIRROR](../specs/ZED-SPEC.md#ZED-MIRROR)). + The `languages/python/` tree that would have shadowed Zed's built-in Python + ([ZED-TREESITTER](../specs/ZED-SPEC.md#ZED-TREESITTER)) is gone from + `basilisk-zed/`, so the next mirror push is safe to list — the published + `v0.39.0` mirror predates that fix and must not be pinned. + + What remains is upstream's: reviewing and merging the PR. Until it merges, + nothing about Zed availability may be documented as shipped. - [ ] **`[HUMAN]`** Submit the prepared `basilisk.nvim/lspconfig/basilisk.lua` definition upstream. - [ ] **`[HYBRID]`** Submit `basilisk` to the upstream diff --git a/docs/readme/README.src.md b/docs/readme/README.src.md index 56483caf9..4c25dc562 100644 --- a/docs/readme/README.src.md +++ b/docs/readme/README.src.md @@ -13,9 +13,8 @@

English · 简体中文

- The only Python type checker scoring 100% on the official python/typing conformance suite — and the fastest we’ve measured.
- Complete open-source Python dev environment in Rust: type checker, language server, debugger, profiler, plus VS Code, Cursor, Zed & Neovim extensions. Strict by default. - Weighing up the best Python type checker for your codebase? Start with the scoreboard. + An open-source Python type checker and language server, built in Rust.
+ One extension for the whole workflow — diagnostics, autocomplete, refactoring, formatting, debugging, and profiling — driven by a single bundled binary.

@@ -34,55 +33,50 @@ Quick Start  •  Rules  •  Refactoring  •  - Compare  •  GitHub

- 100.0% PEP conformance — 141 of 141 tests in the official - python/typing - conformance suite (commit a490662), scored on the wheel-installed CLI in its default config by the real upstream harness. - We target python/typing@main and ratchet the score up only. + Basilisk in action — type checking, diagnostics, and refactoring in the editor

-## The only 100% checker — and the fastest according to our benchmarks - -Basilisk is the **only** Python type checker with a perfect score on the official -[`python/typing` conformance suite](https://github.com/python/typing/blob/main/conformance/results/results.html): -**100.0%** (141/141 files, 970 required errors caught, 0 false positives), -measured by the real upstream harness on the wheel-installed CLI in its default config. +**The current type checker contains inaccuracies and you should not use it as part of your dev pipeline. We are working on removing any misleading analyzers ASAP. Please read below** -

- Basilisk in action — type checking, diagnostics, and refactoring in the editor -

+## We withdrew the typing conformance results -And it is the **fastest checker we’ve measured** — median cold full-file check, from scratch: +We withdrew our 100% conformance claim and our benchmark figures, and asked to be +[removed from the official `python/typing` results](https://github.com/python/typing/blob/main/conformance/results/results.html). +The cause was checker logic fitted to the contents of conformance test files +instead of implementing the typing specification generally, and a score produced +that way is not evidence. The current percentage is **temporarily unknown**. -| Type checker | Median cold check | -| --- | --- | -| ⚡ **Basilisk** | **12 ms** | -| zuban | 28 ms | -| ty | 39 ms | -| Pyrefly | 111 ms | -| Pyright | 582 ms | -| mypy | 605 ms | +We are deciding whether to rebuild the checker from the specification or drive +the extension with an established open-source checker. **Either way, we are +building Basilisk into an accurate Python development experience** — and a new +figure gets published only once it survives off-suite and mutation testing. -Median cold full-file check across 26 single-construct typing-spec stress fixtures on an Apple M4 Max — lower is better. Basilisk’s warm re-check drops to ~5 ms. Every figure is produced by [`hyperfine`](https://github.com/sharkdp/hyperfine) and committed per machine, so nothing here is hand-typed. **Clone the repo, run `make bench` on your own hardware, and send us the CSV — independent audits are welcome.** [Full benchmarks & methodology →](https://www.basilisk-python.dev/docs/benchmarks/) +[Read the full correction →](https://www.basilisk-python.dev/docs/conformance/)  •  +[Integrity audit →](docs/CONFORMANCE-INTEGRITY-AUDIT.md) -## Everything in one extension +## What you get -One extension replaces Pylance and gives you the whole workflow — no Node.js, no Python runtime, no pip, no npm. A single bundled Rust binary drives it all: +One extension covers the whole Python workflow. A single bundled Rust binary +drives it — no Node.js, no npm, no `pip install`: -- **Strict-by-default diagnostics** — inline as you type, incremental analysis powered by Salsa (the rust-analyzer engine) +- **Diagnostics as you type** — incremental analysis powered by [Salsa](https://github.com/salsa-rs/salsa) - **Autocomplete, hover, go-to-definition, find references, rename** - **Refactoring code actions** — extract, inline, move symbol, organize imports -- **Integrated debugging** — F5 to debug via bundled debugpy; no separate extension +- **Integrated debugging** — F5 to debug via bundled [debugpy](https://github.com/microsoft/debugpy); no separate extension - **Integrated profiling** — CPU heat map, flame graph, and a memory dashboard with leak detection - **Activity panel** — module tree with per-module type-health coverage, plus feature toggles - **Inlay hints** and **Ruff** formatting/import-organization, built in - **Standard-library types from [typeshed](https://github.com/python/typeshed)** — a complete `stdlib/` snapshot is compiled into the binary, so hover and diagnostics work offline with no configuration -Every diagnostic teaches: rustc-style output with a `help`, a `note`, and a link to a per-rule explainer, so a red squiggle always tells you *why*. Basilisk **starts strict** and stays strict — the unconfigured default enables the complete typing-spec rule set, and strictness is dialled per rule, never by a mode. +Strictness is configured **per rule**, never by a mode: the unconfigured default +enables the typing-spec rule set, and each rule can be graded down to +`warning`/`info` so a codebase can adopt type safety incrementally. Every +diagnostic carries a `help`, a `note`, and a link to a per-rule explainer, so a +red squiggle tells you *why*. ## Install diff --git a/docs/readme/README.zh.src.md b/docs/readme/README.zh.src.md index 3f80a16ee..bbb7f9819 100644 --- a/docs/readme/README.zh.src.md +++ b/docs/readme/README.zh.src.md @@ -12,8 +12,8 @@

English · 简体中文

- 唯一在官方 python/typing 一致性测试套件上取得 100% 的 Python 类型检查器 —— 也是我们测得最快的。
- 用 Rust 打造的完整开源 Python 开发环境：类型检查器、语言服务器、调试器、性能分析器，以及 VS Code、Cursor、Zed 与 Neovim 扩展。默认严格。 + 用 Rust 打造的开源 Python 类型检查器与语言服务器。
+ 一个扩展覆盖整套工作流 —— 诊断、自动补全、重构、格式化、调试与性能分析 —— 全部由单一捆绑的二进制文件驱动。

@@ -29,57 +29,47 @@ 快速上手  •  规则  •  重构  •  - 对比  •  GitHub

- PEP 一致性 100.0% — 官方 - python/typing - 一致性套件（提交 a490662）141 项测试中通过 141 项， - 由真实的上游评分器在默认配置下对 wheel 安装的 CLI 评出。 - 我们以 python/typing@main 为目标，且分数只升不降。 + Basilisk 实战 —— 编辑器中的类型检查、诊断与重构

-## 唯一 100% 的检查器 — 按我们的基准测试也是最快的 - -Basilisk 是**唯一**在官方 -[`python/typing` 一致性套件](https://github.com/python/typing/blob/main/conformance/results/results.html) -上取得满分的 Python 类型检查器：**100.0%** -（141/141 个文件，捕获 970 处必需错误，0 个误报）， -由真实的上游评分器在默认配置下对 wheel 安装的 CLI 测得。 +**当前的类型检查器存在不准确之处，请勿将其用于你的开发流水线。我们正在尽快移除任何具有误导性的分析器。详情请见下文** -

- Basilisk 实战 —— 编辑器中的类型检查、诊断与重构 -

+## 我们撤回了类型一致性结果 -它也是**我们测得最快的检查器** — 从零开始的整文件冷检查中位数： +我们撤回了 100% 的一致性宣称与基准测试数字，并主动请求 +[从官方 `python/typing` 结果中移除](https://github.com/python/typing/blob/main/conformance/results/results.html)。 +原因是检查器中存在针对一致性测试文件内容而写的逻辑，而不是对类型规范的通用实现； +这样得出的分数并不能作为证据。当前的百分比**暂时未知**。 -| 类型检查器 | 冷检查中位数 | -| --- | --- | -| ⚡ **Basilisk** | **12 ms** | -| zuban | 28 ms | -| ty | 39 ms | -| Pyrefly | 111 ms | -| Pyright | 582 ms | -| mypy | 605 ms | +我们正在决定是按规范重建检查器，还是让扩展由一个成熟的开源检查器驱动。**无论走哪 +条路，我们都在把 Basilisk 打造成准确的 Python 开发体验** —— 而新的数字只有在经受住 +套件之外的用例与变异测试后才会发布。 -在 Apple M4 Max 上对 26 个单一构造的类型规范压力用例测得的整文件冷检查中位数 — 越低越好。Basilisk 的热重检查可降至约 5 ms。每个数字都由 [`hyperfine`](https://github.com/sharkdp/hyperfine) 产生并按机器提交，没有一个是手写的。**克隆仓库，在你自己的硬件上运行 `make bench`，并把 CSV 发给我们 — 欢迎独立复核。** [完整基准与方法论 →](https://www.basilisk-python.dev/zh/docs/benchmarks/) +[阅读完整更正 →](https://www.basilisk-python.dev/zh/docs/conformance/)  •  +[完整性审计 →](docs/CONFORMANCE-INTEGRITY-AUDIT.md) -## 一个扩展，覆盖全部 +## 你能得到什么 -一个扩展即可取代 Pylance 并提供完整工作流 —— 无需 Node.js、无需 Python 运行时、无需 pip、无需 npm。一切由单一捆绑的 Rust 二进制文件驱动： +一个扩展即可覆盖整套 Python 工作流。一切由单一捆绑的 Rust 二进制文件驱动 —— +无需 Node.js、无需 npm、无需 `pip install`： -- **默认严格的诊断** —— 随输入实时呈现，由 Salsa（rust-analyzer 的引擎）提供增量分析 +- **随输入实时诊断** —— 由 [Salsa](https://github.com/salsa-rs/salsa) 提供增量分析 - **自动补全、悬停信息、跳转到定义、查找引用、重命名** - **重构代码操作** —— 提取、内联、移动符号、整理导入 -- **集成调试** —— 按 F5 即可通过捆绑的 debugpy 调试；无需额外扩展 +- **集成调试** —— 按 F5 即可通过捆绑的 [debugpy](https://github.com/microsoft/debugpy) 调试；无需额外扩展 - **集成性能分析** —— CPU 热力图、火焰图，以及带泄漏检测的内存面板 - **活动面板** —— 模块树与逐模块的类型健康度覆盖率，并可切换功能开关 - 内置 **Inlay hints** 与 **Ruff** 格式化／导入整理 - **来自 [typeshed](https://github.com/python/typeshed) 的标准库类型** —— 完整的 `stdlib/` 快照已编译进二进制文件，因此悬停与诊断在离线且零配置的情况下依然可用 -每条诊断都有教育意义：rustc 风格的输出，附带 `help`、`note` 以及指向每条规则详解页的链接，因此一条红色波浪线总能告诉你*为什么*。Basilisk **一开始就严格**并始终严格 —— 未配置的默认值即启用完整的类型规范规则集，严格程度按规则微调，而不是靠模式切换。 +严格程度按**规则**配置，而不是靠模式切换：未配置的默认值即启用类型规范规则集， +每条规则都可以降级为 `warning`/`info`，让代码库能够渐进地采用类型安全。每条诊断 +都附带 `help`、`note` 以及指向每条规则详解页的链接，因此一条红色波浪线总能告诉你 +*为什么*。 ## 安装 diff --git a/docs/specs/ZED-SPEC.md b/docs/specs/ZED-SPEC.md index c4bdb576b..5a4a6b8ba 100644 --- a/docs/specs/ZED-SPEC.md +++ b/docs/specs/ZED-SPEC.md @@ -328,7 +328,11 @@ Zed has no upload API. Extensions are listed in [`zed-industries/extensions`](ht `scripts/render-zed-mirror.sh` resolves both: vendors `basilisk-common` (zero-dependency, WASM-safe) under `vendor/basilisk-common`, rewrites the path dependency, stamps the release version, makes the mirror its own workspace root, and drops the workspace-only `[lints]` inheritance. The `publish-zed` job in `release.yml` renders the tree, **gates the push on a real `cargo build --release --target wasm32-wasip2`**, then pushes to [`Nimblesite/basilisk-zed`](https://github.com/Nimblesite/basilisk-zed) and tags it with the monorepo tag — same clone-replace-commit-push convention as `publish-nvim`, using the `BREW_SCOOP_PAT` org secret. -The mirror version equals the monorepo tag (`v1.2.3` → `1.2.3`); the binary [ZED-DIST](#ZED-DIST) updates independently at runtime. The first listing is a one-time human-reviewed PR adding the submodule to `zed-industries/extensions`; subsequent bumps amend that pointer. +The mirror version equals the monorepo tag (`v1.2.3` → `1.2.3`); the binary [ZED-DIST](#ZED-DIST) updates independently at runtime. + +**Pushing the mirror publishes nothing.** Zed installs only what `zed-industries/extensions` lists, so the mirror push is a prerequisite, not the release. `scripts/publish_zed_registry.py`, run by `publish-zed` immediately after the mirror is tagged, performs the listing itself: it forks the registry, resets a `listing-basilisk` branch to upstream's head, adds or re-pins the `extensions/basilisk` submodule to the release tag, sets `[basilisk] version` in `extensions.toml`, re-sorts `.gitmodules`, and opens the PR — or, once that PR is open, moves the pointer on the same branch. Every release therefore proposes its own bump; upstream maintainers still merge it. + +The registry is a repository Basilisk does not own, and its `extensions.toml` holds ~1400 entries, so the edit is **surgical, not a rewrite**: the entry is spliced into alphabetical position and every other entry stays byte-identical, since a reformatting diff across someone else's registry is a rejected PR. `scripts/test_publish_zed_registry.py` proves both properties — placement, non-disturbance, bump-not-duplicate, and idempotence — in the `zed` CI job, because the real thing runs only during a tagged release. ## Zed Settings {#ZED-CONFIG} diff --git a/pyproject.toml b/pyproject.toml index 4156b0eb0..a046221e0 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -21,7 +21,7 @@ build-backend = "maturin" [project] name = "basilisk-python" -description = "Basilisk — the only Python type checker scoring 100% on the official python/typing conformance suite (https://github.com/python/typing/blob/main/conformance/results/results.html) — and the fastest according to our benchmarks. Complete open-source Python dev environment in Rust: type checker, language server, debugger, profiler, plus VS Code, Cursor, Zed & Neovim extensions. Strict by default." +description = "Basilisk — an open-source Python type checker and language server built in Rust: diagnostics, refactoring, formatting, integrated debugging, and profiling. Strictness is configured per rule, so a codebase can adopt type safety incrementally." readme = "README-pypi.md" requires-python = ">=3.8" license = "Apache-2.0 AND BSD-2-Clause AND BSD-3-Clause AND CDDL-1.0 AND CDLA-Permissive-2.0 AND ISC AND MIT AND MPL-2.0 AND Unicode-3.0 AND Unicode-DFS-2016 AND Zlib" diff --git a/scripts/publish_zed_registry.py b/scripts/publish_zed_registry.py new file mode 100755 index 000000000..acd4186a4 --- /dev/null +++ b/scripts/publish_zed_registry.py @@ -0,0 +1,211 @@ +#!/usr/bin/env python3 +"""Submit or bump Basilisk's listing in `zed-industries/extensions`. + +Implements [ZED-MIRROR]; see docs/specs/ZED-SPEC.md#ZED-MIRROR. + +Zed has no upload API. Pushing the rendered tree to `Nimblesite/basilisk-zed` +publishes nothing on its own — the extension only becomes installable once +`zed-industries/extensions` lists it, as a git submodule pinned to a commit plus +an `extensions.toml` entry naming the version. That listing step used to be a +manual to-do that was never done, which is why Basilisk has never appeared in +Zed's extensions view. This script performs it, and is safe to re-run: the first +release opens the listing PR, every later release moves the submodule pointer +and the version on the same branch. + +Usage: + scripts/publish_zed_registry.py + scripts/publish_zed_registry.py 0.41.0 v0.41.0 + +Requires `gh` authenticated as a token that can fork into UPSTREAM_FORK's owner +and push to that fork. +""" + +from __future__ import annotations + +import os +import subprocess +import sys +import tempfile +import tomllib +from pathlib import Path + +UPSTREAM = "zed-industries/extensions" +FORK = "Nimblesite/extensions" +MIRROR_URL = "https://github.com/Nimblesite/basilisk-zed.git" +EXTENSION_ID = "basilisk" +SUBMODULE_PATH = f"extensions/{EXTENSION_ID}" +REGISTRY_TOML = "extensions.toml" +GITMODULES = ".gitmodules" + + +def run(cmd: list[str], cwd: Path | None = None, check: bool = True) -> str: + """Run `cmd`, echoing it, and return stdout.""" + print(f" $ {' '.join(cmd)}", flush=True) + done = subprocess.run(cmd, cwd=cwd, check=check, capture_output=True, text=True) + if done.stdout: + print(done.stdout.rstrip(), flush=True) + if done.returncode != 0 and done.stderr: + print(done.stderr.rstrip(), file=sys.stderr, flush=True) + return done.stdout + + +def blocks(text: str, opener: str) -> list[tuple[str, list[str]]]: + """Split a flat TOML-ish file into `(header, lines)` blocks. + + `extensions.toml` and `.gitmodules` are both a preamble followed by a flat + run of sections, each introduced by a line starting with `opener`. Splitting + on those headers lets us edit one section without reformatting the other + ~2000, which a whole-document rewrite would do. The result is parse-verified + by `tomllib` in `write_registry` before anything is committed. + """ + out: list[tuple[str, list[str]]] = [] + header, current = "", [] + for line in text.splitlines(keepends=True): + if line.startswith(opener): + if header or current: + out.append((header, current)) + header, current = line.strip(), [line] + else: + current.append(line) + if header or current: + out.append((header, current)) + return out + + +def sort_key(header: str) -> str: + """The section's sort key, matching upstream's `pnpm sort-extensions`.""" + return header.strip("[]").strip('"').casefold() + + +def splice(text: str, opener: str, header: str, body: list[str]) -> str: + """Replace the `header` section in `text`, or insert it in sorted order.""" + sections = blocks(text, opener) + kept = [s for s in sections if s[0] != header] + entry = (header, body) + at = len(kept) + for index, (existing, _) in enumerate(kept): + if existing.startswith(opener) and sort_key(existing) > sort_key(header): + at = index + break + kept.insert(at, entry) + return "".join("".join(lines) for _, lines in kept) + + +def write_registry(repo: Path, version: str) -> None: + """Point `extensions.toml`'s `[basilisk]` entry at `version`.""" + path = repo / REGISTRY_TOML + body = [ + f"[{EXTENSION_ID}]\n", + f'submodule = "{SUBMODULE_PATH}"\n', + f'version = "{version}"\n', + "\n", + ] + updated = splice(path.read_text(encoding="utf-8"), "[", f"[{EXTENSION_ID}]", body) + path.write_text(updated, encoding="utf-8") + + listed = tomllib.loads(updated).get(EXTENSION_ID) + if listed != {"submodule": SUBMODULE_PATH, "version": version}: + raise SystemExit(f"✗ {REGISTRY_TOML} did not round-trip: {listed!r}") + print(f" {REGISTRY_TOML}: [{EXTENSION_ID}] version = {version}") + + +def write_gitmodules(repo: Path) -> None: + """Re-sort `.gitmodules` so `git submodule add`'s append stays ordered.""" + path = repo / GITMODULES + text = path.read_text(encoding="utf-8") + sections = blocks(text, "[submodule ") + preamble = [s for s in sections if not s[0].startswith("[submodule ")] + entries = sorted( + (s for s in sections if s[0].startswith("[submodule ")), + key=lambda s: sort_key(s[0].removeprefix("[submodule ")), + ) + ordered = "".join("".join(lines) for _, lines in [*preamble, *entries]) + if ordered != text: + path.write_text(ordered, encoding="utf-8") + print(f" {GITMODULES}: re-sorted") + + +def clone_fork(work: Path) -> Path: + """Fork upstream if needed, then clone the fork reset to upstream's head.""" + run(["gh", "repo", "fork", UPSTREAM, "--clone=false", "--remote=false"]) + repo = work / "extensions" + run(["gh", "repo", "clone", FORK, str(repo), "--", "--depth=50"]) + run( + ["git", "remote", "add", "upstream", f"https://github.com/{UPSTREAM}.git"], repo + ) + run(["git", "fetch", "--depth=50", "upstream", "HEAD"], repo) + run(["git", "checkout", "-B", f"listing-{EXTENSION_ID}", "FETCH_HEAD"], repo) + return repo + + +def pin_submodule(repo: Path, tag: str) -> None: + """Add the mirror submodule if absent, then pin it to `tag`.""" + module = repo / SUBMODULE_PATH + if not (module / ".git").exists(): + run(["git", "submodule", "add", "--force", MIRROR_URL, SUBMODULE_PATH], repo) + run(["git", "fetch", "--tags", "origin"], module) + run(["git", "checkout", f"tags/{tag}"], module) + + +def commit_and_push(repo: Path, version: str) -> bool: + """Commit the listing change and push the branch. False if nothing changed.""" + run(["git", "config", "user.name", "github-actions[bot]"], repo) + email = "41898282+github-actions[bot]@users.noreply.github.com" + run(["git", "config", "user.email", email], repo) + run(["git", "add", "-A"], repo) + if not run(["git", "status", "--porcelain"], repo).strip(): + print(f" {UPSTREAM} already lists {EXTENSION_ID} {version}") + return False + run(["git", "commit", "-m", f"{EXTENSION_ID}: {version}"], repo) + run( + ["git", "push", "--force-with-lease", "origin", f"listing-{EXTENSION_ID}"], repo + ) + return True + + +def open_pr(repo: Path, version: str) -> None: + """Open the listing PR, unless one is already open for this branch.""" + head = f"{FORK.split('/')[0]}:listing-{EXTENSION_ID}" + existing = run( + ["gh", "pr", "list", "--repo", UPSTREAM, "--head", head, "--json", "url"], + check=False, + ) + if '"url"' in existing: + print(f" PR already open for {head} — pointer updated in place") + return + body = ( + f"Adds the Basilisk language-server extension at {version}.\n\n" + f"Submodule: {MIRROR_URL} (pinned to the release tag).\n" + "The extension attaches to Zed's built-in Python language and ships no " + "`languages/` tree or grammar, so it does not shadow the built-in " + "definition.\n\nRefs https://github.com/Nimblesite/Basilisk\n" + ) + create = ["gh", "pr", "create", "--repo", UPSTREAM, "--head", head] + create += ["--title", f"Add {EXTENSION_ID} {version}", "--body", body] + run(create, repo, check=False) + + +def main(argv: list[str]) -> int: + if len(argv) != 2: + print(__doc__, file=sys.stderr) + return 2 + version, tag = argv + if not os.environ.get("GH_TOKEN") and not os.environ.get("GITHUB_TOKEN"): + print( + "✗ GH_TOKEN not set — cannot fork or open the listing PR", file=sys.stderr + ) + return 1 + + with tempfile.TemporaryDirectory() as tmp: + repo = clone_fork(Path(tmp)) + pin_submodule(repo, tag) + write_registry(repo, version) + write_gitmodules(repo) + if commit_and_push(repo, version): + open_pr(repo, version) + print(f" {EXTENSION_ID} {version} submitted to {UPSTREAM}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main(sys.argv[1:])) diff --git a/scripts/test_publish_zed_registry.py b/scripts/test_publish_zed_registry.py new file mode 100644 index 000000000..4f93be3af --- /dev/null +++ b/scripts/test_publish_zed_registry.py @@ -0,0 +1,133 @@ +"""Tests for `scripts/publish_zed_registry.py` — the Zed registry listing edit. + +Covers [ZED-MIRROR] (docs/specs/ZED-SPEC.md#ZED-MIRROR). The listing edit runs +against a ~1400-entry third-party file that Basilisk does not own, so the two +properties worth proving are that the Basilisk entry lands correctly *and* that +nothing else in the file moves — a reformatting diff across someone else's +registry is a rejected PR. Network-free by construction: only the pure +text-editing functions are exercised. +""" + +from __future__ import annotations + +import sys +import tomllib +from pathlib import Path + +import pytest + +sys.path.insert(0, str(Path(__file__).resolve().parent)) + +from publish_zed_registry import ( # noqa: E402 + EXTENSION_ID, + SUBMODULE_PATH, + blocks, + sort_key, + splice, + write_gitmodules, + write_registry, +) + +REGISTRY = """\ +[aardvark] +submodule = "extensions/aardvark" +version = "1.0.0" + +[basher] +submodule = "extensions/basher" +version = "0.2.1" + +[batman] +submodule = "extensions/batman" +version = "3.0.0" + +[zig] +submodule = "extensions/zig" +version = "0.9.9" +""" + + +def entry(version: str) -> list[str]: + return [ + f"[{EXTENSION_ID}]\n", + f'submodule = "{SUBMODULE_PATH}"\n', + f'version = "{version}"\n', + "\n", + ] + + +def keys(text: str) -> list[str]: + return [h.strip("[]") for h, _ in blocks(text, "[") if h.startswith("[")] + + +def test_inserts_in_alphabetical_position() -> None: + out = splice(REGISTRY, "[", f"[{EXTENSION_ID}]", entry("0.41.0")) + assert keys(out) == ["aardvark", "basher", "basilisk", "batman", "zig"] + + +def test_leaves_every_other_entry_byte_identical() -> None: + out = splice(REGISTRY, "[", f"[{EXTENSION_ID}]", entry("0.41.0")) + before = tomllib.loads(REGISTRY) + after = {k: v for k, v in tomllib.loads(out).items() if k != EXTENSION_ID} + assert after == before + for name in before: + assert f'[{name}]\nsubmodule = "extensions/{name}"\n' in out + + +def test_bump_replaces_rather_than_duplicates() -> None: + first = splice(REGISTRY, "[", f"[{EXTENSION_ID}]", entry("0.41.0")) + second = splice(first, "[", f"[{EXTENSION_ID}]", entry("0.42.0")) + assert keys(second).count(EXTENSION_ID) == 1 + assert tomllib.loads(second)[EXTENSION_ID]["version"] == "0.42.0" + assert keys(second) == keys(first) + + +def test_splice_is_idempotent() -> None: + once = splice(REGISTRY, "[", f"[{EXTENSION_ID}]", entry("0.41.0")) + assert splice(once, "[", f"[{EXTENSION_ID}]", entry("0.41.0")) == once + + +def test_sort_key_is_case_insensitive_and_unquoted() -> None: + assert sort_key("[Basilisk]") == "basilisk" + # `.gitmodules` headers are keyed on the path, after the caller strips the + # `[submodule ` prefix — the quotes must not survive into the sort order. + header = '[submodule "extensions/Zig"]' + assert sort_key(header.removeprefix("[submodule ")) == "extensions/zig" + + +def test_write_registry_pins_the_version(tmp_path: Path) -> None: + (tmp_path / "extensions.toml").write_text(REGISTRY, encoding="utf-8") + write_registry(tmp_path, "0.41.0") + listed = tomllib.loads((tmp_path / "extensions.toml").read_text(encoding="utf-8")) + assert listed[EXTENSION_ID] == {"submodule": SUBMODULE_PATH, "version": "0.41.0"} + + +def test_write_registry_rejects_a_corrupt_result(tmp_path: Path) -> None: + (tmp_path / "extensions.toml").write_text("[aardvark\nbroken", encoding="utf-8") + with pytest.raises((SystemExit, tomllib.TOMLDecodeError)): + write_registry(tmp_path, "0.41.0") + + +def test_write_gitmodules_sorts_appended_submodules(tmp_path: Path) -> None: + unsorted = ( + '[submodule "extensions/aardvark"]\n' + "\tpath = extensions/aardvark\n" + '[submodule "extensions/zig"]\n' + "\tpath = extensions/zig\n" + '[submodule "extensions/basilisk"]\n' + "\tpath = extensions/basilisk\n" + ) + (tmp_path / ".gitmodules").write_text(unsorted, encoding="utf-8") + write_gitmodules(tmp_path) + ordered = (tmp_path / ".gitmodules").read_text(encoding="utf-8") + assert ordered.index("aardvark") < ordered.index("basilisk") < ordered.index("zig") + assert ordered.count("[submodule ") == 3 + + +def test_write_gitmodules_is_idempotent(tmp_path: Path) -> None: + path = tmp_path / ".gitmodules" + path.write_text('[submodule "extensions/a"]\n\tpath = extensions/a\n', "utf-8") + write_gitmodules(tmp_path) + once = path.read_text(encoding="utf-8") + write_gitmodules(tmp_path) + assert path.read_text(encoding="utf-8") == once diff --git a/vscode-extension/README.md b/vscode-extension/README.md index c00f89978..4481191cd 100644 --- a/vscode-extension/README.md +++ b/vscode-extension/README.md @@ -10,9 +10,8 @@

English · 简体中文

- The only Python type checker scoring 100% on the official python/typing conformance suite — and the fastest we’ve measured.
- Complete open-source Python dev environment in Rust: type checker, language server, debugger, profiler, plus VS Code, Cursor, Zed & Neovim extensions. Strict by default. - Weighing up the best Python type checker for your codebase? Start with the scoreboard. + An open-source Python type checker and language server, built in Rust.
+ One extension for the whole workflow — diagnostics, autocomplete, refactoring, formatting, debugging, and profiling — driven by a single bundled binary.

> **You are reading the Basilisk extension listing** for VS Code, Cursor, Windsurf, and every VS Code fork — the same extension is published to the [VS Code Marketplace](https://marketplace.visualstudio.com/items?itemName=Nimblesite.basilisk) and [Open VSX](https://open-vsx.org/extension/Nimblesite/basilisk). @@ -23,55 +22,50 @@ Quick Start  •  Rules  •  Refactoring  •  - Compare  •  GitHub

- 100.0% PEP conformance — 141 of 141 tests in the official - python/typing - conformance suite (commit a490662), scored on the wheel-installed CLI in its default config by the real upstream harness. - We target python/typing@main and ratchet the score up only. + Basilisk in action — type checking, diagnostics, and refactoring in the editor

-## The only 100% checker — and the fastest according to our benchmarks - -Basilisk is the **only** Python type checker with a perfect score on the official -[`python/typing` conformance suite](https://github.com/python/typing/blob/main/conformance/results/results.html): -**100.0%** (141/141 files, 970 required errors caught, 0 false positives), -measured by the real upstream harness on the wheel-installed CLI in its default config. +**The current type checker contains inaccuracies and you should not use it as part of your dev pipeline. We are working on removing any misleading analyzers ASAP. Please read below** -

- Basilisk in action — type checking, diagnostics, and refactoring in the editor -

+## We withdrew the typing conformance results -And it is the **fastest checker we’ve measured** — median cold full-file check, from scratch: +We withdrew our 100% conformance claim and our benchmark figures, and asked to be +[removed from the official `python/typing` results](https://github.com/python/typing/blob/main/conformance/results/results.html). +The cause was checker logic fitted to the contents of conformance test files +instead of implementing the typing specification generally, and a score produced +that way is not evidence. The current percentage is **temporarily unknown**. -| Type checker | Median cold check | -| --- | --- | -| ⚡ **Basilisk** | **12 ms** | -| zuban | 28 ms | -| ty | 39 ms | -| Pyrefly | 111 ms | -| Pyright | 582 ms | -| mypy | 605 ms | +We are deciding whether to rebuild the checker from the specification or drive +the extension with an established open-source checker. **Either way, we are +building Basilisk into an accurate Python development experience** — and a new +figure gets published only once it survives off-suite and mutation testing. -Median cold full-file check across 26 single-construct typing-spec stress fixtures on an Apple M4 Max — lower is better. Basilisk’s warm re-check drops to ~5 ms. Every figure is produced by [`hyperfine`](https://github.com/sharkdp/hyperfine) and committed per machine, so nothing here is hand-typed. **Clone the repo, run `make bench` on your own hardware, and send us the CSV — independent audits are welcome.** [Full benchmarks & methodology →](https://www.basilisk-python.dev/docs/benchmarks/) +[Read the full correction →](https://www.basilisk-python.dev/docs/conformance/)  •  +[Integrity audit →](https://github.com/Nimblesite/Basilisk/blob/main/docs/CONFORMANCE-INTEGRITY-AUDIT.md) -## Everything in one extension +## What you get -One extension replaces Pylance and gives you the whole workflow — no Node.js, no Python runtime, no pip, no npm. A single bundled Rust binary drives it all: +One extension covers the whole Python workflow. A single bundled Rust binary +drives it — no Node.js, no npm, no `pip install`: -- **Strict-by-default diagnostics** — inline as you type, incremental analysis powered by Salsa (the rust-analyzer engine) +- **Diagnostics as you type** — incremental analysis powered by [Salsa](https://github.com/salsa-rs/salsa) - **Autocomplete, hover, go-to-definition, find references, rename** - **Refactoring code actions** — extract, inline, move symbol, organize imports -- **Integrated debugging** — F5 to debug via bundled debugpy; no separate extension +- **Integrated debugging** — F5 to debug via bundled [debugpy](https://github.com/microsoft/debugpy); no separate extension - **Integrated profiling** — CPU heat map, flame graph, and a memory dashboard with leak detection - **Activity panel** — module tree with per-module type-health coverage, plus feature toggles - **Inlay hints** and **Ruff** formatting/import-organization, built in - **Standard-library types from [typeshed](https://github.com/python/typeshed)** — a complete `stdlib/` snapshot is compiled into the binary, so hover and diagnostics work offline with no configuration -Every diagnostic teaches: rustc-style output with a `help`, a `note`, and a link to a per-rule explainer, so a red squiggle always tells you *why*. Basilisk **starts strict** and stays strict — the unconfigured default enables the complete typing-spec rule set, and strictness is dialled per rule, never by a mode. +Strictness is configured **per rule**, never by a mode: the unconfigured default +enables the typing-spec rule set, and each rule can be graded down to +`warning`/`info` so a codebase can adopt type safety incrementally. Every +diagnostic carries a `help`, a `note`, and a link to a per-rule explainer, so a +red squiggle tells you *why*. ## Install diff --git a/vscode-extension/README.zh.md b/vscode-extension/README.zh.md index 16348d9ae..1433fca77 100644 --- a/vscode-extension/README.zh.md +++ b/vscode-extension/README.zh.md @@ -10,8 +10,8 @@

English · 简体中文

- 唯一在官方 python/typing 一致性测试套件上取得 100% 的 Python 类型检查器 —— 也是我们测得最快的。
- 用 Rust 打造的完整开源 Python 开发环境：类型检查器、语言服务器、调试器、性能分析器，以及 VS Code、Cursor、Zed 与 Neovim 扩展。默认严格。 + 用 Rust 打造的开源 Python 类型检查器与语言服务器。
+ 一个扩展覆盖整套工作流 —— 诊断、自动补全、重构、格式化、调试与性能分析 —— 全部由单一捆绑的二进制文件驱动。

> **你正在阅读 Basilisk 的扩展页面**，适用于 VS Code、Cursor、Windsurf 以及所有 VS Code 分支 —— 同一个扩展同时发布到 [VS Code Marketplace](https://marketplace.visualstudio.com/items?itemName=Nimblesite.basilisk) 与 [Open VSX](https://open-vsx.org/extension/Nimblesite/basilisk)。 @@ -22,57 +22,47 @@ 快速上手  •  规则  •  重构  •  - 对比  •  GitHub

- PEP 一致性 100.0% — 官方 - python/typing - 一致性套件（提交 a490662）141 项测试中通过 141 项， - 由真实的上游评分器在默认配置下对 wheel 安装的 CLI 评出。 - 我们以 python/typing@main 为目标，且分数只升不降。 + Basilisk 实战 —— 编辑器中的类型检查、诊断与重构

-## 唯一 100% 的检查器 — 按我们的基准测试也是最快的 - -Basilisk 是**唯一**在官方 -[`python/typing` 一致性套件](https://github.com/python/typing/blob/main/conformance/results/results.html) -上取得满分的 Python 类型检查器：**100.0%** -（141/141 个文件，捕获 970 处必需错误，0 个误报）， -由真实的上游评分器在默认配置下对 wheel 安装的 CLI 测得。 +**当前的类型检查器存在不准确之处，请勿将其用于你的开发流水线。我们正在尽快移除任何具有误导性的分析器。详情请见下文** -

- Basilisk 实战 —— 编辑器中的类型检查、诊断与重构 -

+## 我们撤回了类型一致性结果 -它也是**我们测得最快的检查器** — 从零开始的整文件冷检查中位数： +我们撤回了 100% 的一致性宣称与基准测试数字，并主动请求 +[从官方 `python/typing` 结果中移除](https://github.com/python/typing/blob/main/conformance/results/results.html)。 +原因是检查器中存在针对一致性测试文件内容而写的逻辑，而不是对类型规范的通用实现； +这样得出的分数并不能作为证据。当前的百分比**暂时未知**。 -| 类型检查器 | 冷检查中位数 | -| --- | --- | -| ⚡ **Basilisk** | **12 ms** | -| zuban | 28 ms | -| ty | 39 ms | -| Pyrefly | 111 ms | -| Pyright | 582 ms | -| mypy | 605 ms | +我们正在决定是按规范重建检查器，还是让扩展由一个成熟的开源检查器驱动。**无论走哪 +条路，我们都在把 Basilisk 打造成准确的 Python 开发体验** —— 而新的数字只有在经受住 +套件之外的用例与变异测试后才会发布。 -在 Apple M4 Max 上对 26 个单一构造的类型规范压力用例测得的整文件冷检查中位数 — 越低越好。Basilisk 的热重检查可降至约 5 ms。每个数字都由 [`hyperfine`](https://github.com/sharkdp/hyperfine) 产生并按机器提交，没有一个是手写的。**克隆仓库，在你自己的硬件上运行 `make bench`，并把 CSV 发给我们 — 欢迎独立复核。** [完整基准与方法论 →](https://www.basilisk-python.dev/zh/docs/benchmarks/) +[阅读完整更正 →](https://www.basilisk-python.dev/zh/docs/conformance/)  •  +[完整性审计 →](https://github.com/Nimblesite/Basilisk/blob/main/docs/CONFORMANCE-INTEGRITY-AUDIT.md) -## 一个扩展，覆盖全部 +## 你能得到什么 -一个扩展即可取代 Pylance 并提供完整工作流 —— 无需 Node.js、无需 Python 运行时、无需 pip、无需 npm。一切由单一捆绑的 Rust 二进制文件驱动： +一个扩展即可覆盖整套 Python 工作流。一切由单一捆绑的 Rust 二进制文件驱动 —— +无需 Node.js、无需 npm、无需 `pip install`： -- **默认严格的诊断** —— 随输入实时呈现，由 Salsa（rust-analyzer 的引擎）提供增量分析 +- **随输入实时诊断** —— 由 [Salsa](https://github.com/salsa-rs/salsa) 提供增量分析 - **自动补全、悬停信息、跳转到定义、查找引用、重命名** - **重构代码操作** —— 提取、内联、移动符号、整理导入 -- **集成调试** —— 按 F5 即可通过捆绑的 debugpy 调试；无需额外扩展 +- **集成调试** —— 按 F5 即可通过捆绑的 [debugpy](https://github.com/microsoft/debugpy) 调试；无需额外扩展 - **集成性能分析** —— CPU 热力图、火焰图，以及带泄漏检测的内存面板 - **活动面板** —— 模块树与逐模块的类型健康度覆盖率，并可切换功能开关 - 内置 **Inlay hints** 与 **Ruff** 格式化／导入整理 - **来自 [typeshed](https://github.com/python/typeshed) 的标准库类型** —— 完整的 `stdlib/` 快照已编译进二进制文件，因此悬停与诊断在离线且零配置的情况下依然可用 -每条诊断都有教育意义：rustc 风格的输出，附带 `help`、`note` 以及指向每条规则详解页的链接，因此一条红色波浪线总能告诉你*为什么*。Basilisk **一开始就严格**并始终严格 —— 未配置的默认值即启用完整的类型规范规则集，严格程度按规则微调，而不是靠模式切换。 +严格程度按**规则**配置，而不是靠模式切换：未配置的默认值即启用类型规范规则集， +每条规则都可以降级为 `warning`/`info`，让代码库能够渐进地采用类型安全。每条诊断 +都附带 `help`、`note` 以及指向每条规则详解页的链接，因此一条红色波浪线总能告诉你 +*为什么*。 ## 安装 diff --git a/vscode-extension/package.json b/vscode-extension/package.json index bd7153db6..6e8c1db03 100644 --- a/vscode-extension/package.json +++ b/vscode-extension/package.json @@ -1,7 +1,7 @@ { "name": "basilisk", "displayName": "Basilisk", - "description": "The only Python type checker scoring 100% on the official python/typing conformance suite — and the fastest we've measured. Complete open-source Python dev environment in Rust: type checker, language server, debugger, profiler, plus VS Code, Cursor, Zed & Neovim extensions. Strict by default.", + "description": "An open-source Python type checker and language server built in Rust: diagnostics, autocomplete, go-to-definition, refactoring, formatting, integrated debugging, and profiling in one extension. Strictness is configured per rule, so a codebase can adopt type safety incrementally.", "version": "0.0.0-PLACEHOLDER", "publisher": "Nimblesite", "license": "SEE LICENSE IN LICENSE.txt", From e656a99e9e0f5f0baa6044659b9a0831ee534eb4 Mon Sep 17 00:00:00 2001 From: Christian Findlay <16697547+MelbourneDeveloper@users.noreply.github.com> Date: Sat, 8 Aug 2026 07:29:19 +1000 Subject: [PATCH 2/9] Fixes --- CLAUDE.md | 7 +- coverage-thresholds.json | 4 +- docs/CONFORMANCE-INTEGRITY-AUDIT.md | 45 +- docs/INDEX.md | 6 +- docs/plans/CHECKER-ADVANCED-FEATURES-PLAN.md | 6 +- .../CHECKER-ELIMINATE-LINE-SCANNING-PLAN.md | 127 ++++-- .../CHECKER-TYPE-NARROWING-INFERENCE-PLAN.md | 187 ++++++--- docs/plans/CHECKER-TYPESHED-PYPI-PLAN.md | 7 +- docs/plans/ROADMAP-NEXT-STEPS-PLAN.md | 45 +- docs/plans/WASM-PLAN.md | 9 +- docs/specs/CHECKER-ARCHITECTURE-SPEC.md | 397 ++++++++++++------ docs/specs/CHECKER-RULE-TAGGING-SPEC.md | 12 + docs/specs/CHECKER-STUB-RESOLUTION-SPEC.md | 19 +- docs/specs/CHECKER-TYPE-INFERENCE-SPEC.md | 14 +- docs/specs/DOCS-README-SPEC.md | 23 +- .../specs/RELEASE-MANUAL-VERIFICATION-SPEC.md | 10 +- docs/specs/REPO-STANDARDS-SPEC.md | 24 +- mutation_testing/mutants_report.py | 18 +- 18 files changed, 656 insertions(+), 304 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 7d1ea0beb..d532b4cd6 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -29,6 +29,8 @@ On encountering it, do exactly three things — **do not fix it, do not rewrite Replacing it is not your call. The point is to surface every one of these so the user can acknowledge it and decide what gets built back. A checker with fewer rules and visible failing tests is the correct outcome; a diagnostic that only fires on one spelling looks like coverage and isn't. +**A failing test that pins real incorrect behaviour is worth more than a passing fixture carried by logic that does not analyse code.** The first is an accurate map of what Basilisk cannot yet do; the second is a false claim that it can. Given the choice, take the failing test — every time. + ## What a correct rule looks like The yardstick for judging code — not licence to go and fix it: @@ -48,8 +50,9 @@ Background, not a directive: strip text-matched logic, establish which rules gen - **Never publish, quote, or market a conformance figure** — nothing may imply Basilisk is in the official results. - **Never re-submit to python/typing** until the mutation harness passes clean and an external audit has run. -- Never move the number by touching the scoreboard: disabling rules, deleting source to dodge a failure, rule-suppressing config, hand-editing `conformance/conformance_status.csv`, loosening `coverage-thresholds.json` ([CHKARCH-CONFORMANCE]). -- **A drop caused by removing text-matched logic is progress.** Record it and say so plainly — never restore the code or fake a pass to hold a ratchet. +- Never move the number by touching the scoreboard: rule-suppressing config, deleting source to dodge a failure, hand-editing `conformance/conformance_status.csv`, loosening `coverage-thresholds.json` ([CHKARCH-CONFORMANCE]). +- **A drop caused by removing text-matched logic is progress.** Record it and say so plainly — never restore the code or fake a pass to hold a ratchet. The boundary is intent: deleting a rule to reach a number hides the loss; deleting text-matched logic leaves a failing test behind and reports the drop. +- `coverage-thresholds.json` still gates the pass percentage at 100 with zero false positives, so the first honest deletion fails `make test`. That floor is the incentive that caused the fitting; removing it is the user's call. Until they decide: **delete anyway, report the drop and the failing gate, and stop there.** # Design Principles diff --git a/coverage-thresholds.json b/coverage-thresholds.json index 9b4bef928..00a4431dd 100644 --- a/coverage-thresholds.json +++ b/coverage-thresholds.json @@ -35,9 +35,9 @@ } }, "conformance": { - "_doc": "Required PEP conformance pass percentage. POLICY: conformance must be 100% \u2014 any file that fails the official python/typing calculator tanks the PR. conformance/run_conformance.py clones python/typing@main FRESH on every run (CI and release), then runs the suite's OWN unmodified src/main.py --only-run basilisk against the compiled binary \u2014 whose conformance/src/type_checker.py already ships the official BasiliskTypeChecker \u2014 and records the exact graded commit in website/src/_data/conformance_report.json, so 100% means the checker passes every test in the current upstream suite. The score is the REAL harness's OWN verdict, produced by the same code that grades pyright/mypy/pyrefly/ty/zuban/pycroscope: a file passes only when its errors_diff is empty, counting every diagnostic the binary emits (errors AND warnings). There is NO vendored calculator and NO cached-fixtures fallback \u2014 if the real harness cannot be cloned and run, the build FAILS. The binary runs in its default configuration \u2014 the pure PEP conformance set; Basilisk's opt-in house-style rules never run during scoring (see [CHKARCH-CONFIGURATION-ONLY]). DISABLING, DELETING, or UNREGISTERING a conformance rule, hand-editing conformance_status.csv, or loosening this gate to fake a pass is forbidden \u2014 close every gap by FIXING the checker. Ratchet UP only; now pinned at the 100% target. (History: a 2026-06-26 baseline reset corrected a gamed fake 100% that had disabled six house rules.) See [CHKARCH-CONFORMANCE].", + "_doc": "Live PEP conformance measurement. POLICY: the number is a REGRESSION DETECTOR, never a target \u2014 see [CHKARCH-CONFORMANCE]. It samples one fixed corpus the checker was historically developed against, so it cannot tell you whether a rule analyses code; only whether today's binary agrees with yesterday's on files it has already seen. NEVER publish, quote, or market this figure. conformance/run_conformance.py clones python/typing@main FRESH on every run, then runs the suite's OWN unmodified src/main.py --only-run basilisk against the compiled binary \u2014 whose conformance/src/type_checker.py already ships the official BasiliskTypeChecker \u2014 and records the exact graded commit in website/src/_data/conformance_report.json. The result is the REAL harness's OWN verdict, produced by the same code that grades pyright/mypy/pyrefly/ty/zuban/pycroscope: a file passes only when its errors_diff is empty, counting every diagnostic the binary emits (errors AND warnings). There is NO vendored calculator and NO cached-fixtures fallback \u2014 if the real harness cannot be cloned and run, the build FAILS. The binary runs in its default configuration \u2014 the pure PEP set; Basilisk's opt-in house-style rules never run (see [CHKARCH-CONFIGURATION-ONLY]). Configuring a rule off before measuring, hand-editing conformance_status.csv, or editing this block to match a run is forbidden. KNOWN CONTRADICTION: `threshold` below is a pass-percentage floor, and a floor over a corpus the code was fitted to is the incentive that produced the fitted predicates (CONFORMANCE-INTEGRITY-AUDIT \u00a76.3). Deleting a rule that decides from source text rather than resolved symbols is REQUIRED ([CHKARCH-TEXT-MATCHED-LOGIC]) and is expected to LOWER this number \u2014 which this floor turns into a build failure. Removing the floor is the user's decision, not an agent's. Until they decide: make the deletion, report the drop and the failing gate, and stop there. Do NOT restore the code, refit the rule, or lower this value to get green.", "threshold": 100, - "_fp_ceiling_doc": "Maximum total false-positive diagnostics across the suite (diagnostics Basilisk reports on a line the suite does NOT mark # E, or outside a satisfied # E[tag] group). POLICY: zero \u2014 a single false positive tanks the PR. Enforced by conformance/run_conformance.py --gate, which runs the REAL python/typing harness on the compiled binary and delegates the pass/FP check to conformance/assert_wheel_conformance.py (run by scripts/test-rust.sh inside make test). Ratchet DOWN only; now at the 0 target. Close every gap by fixing the checker, never by disabling a rule.", + "_fp_ceiling_doc": "Total false-positive diagnostics across the suite (diagnostics Basilisk reports on a line the suite does NOT mark # E, or outside a satisfied # E[tag] group). Measured by conformance/run_conformance.py --gate, which runs the REAL python/typing harness on the compiled binary and delegates the comparison to conformance/assert_wheel_conformance.py (run by scripts/test-rust.sh inside make test). A false positive on real code is a genuine defect worth fixing on its own merits, independent of this suite. Same contradiction as `threshold` above: close a gap by fixing the checker or by deleting logic that never analysed anything \u2014 never by silencing a rule to hold the ceiling.", "max_false_positives": 0 } } diff --git a/docs/CONFORMANCE-INTEGRITY-AUDIT.md b/docs/CONFORMANCE-INTEGRITY-AUDIT.md index eb02912b0..936bb6904 100644 --- a/docs/CONFORMANCE-INTEGRITY-AUDIT.md +++ b/docs/CONFORMANCE-INTEGRITY-AUDIT.md @@ -210,6 +210,11 @@ The exception is `is_assignable_to_bound`, where `_ => true` is not a documented ## 5. Remediation status — measured, not asserted +> **Note on §5.1.** It records a rewrite made before the current policy. Rewriting is no +> longer the response to text-matched logic — deletion is (§7). §5.1 is kept because its +> measurement is the clearest evidence for *why*: a rewrite that fixes the headline case +> and leaves the rule broken still reads as a fix. + ### 5.1 `aliases_type_statement` — rewritten, **partially** effective The rule now validates the `StmtTypeAlias` value node structurally on the Ruff AST. All 13 conformance cases are rejected for the right structural reasons, and it catches forms the text scanner never could. @@ -263,11 +268,41 @@ Unchanged from `main` except for routing `is_assignable_to_bound` through a `Sub ## 7. What we are changing -- **Off-suite regression tests are mandatory** for every rewritten rule, with cases derived from the spec grammar and explicitly **not** from `conformance/tests/`. This is the only control that catches the defect class. -- **Ban hardcoded symbol names as behavioural triggers.** A rule may not key on a specific identifier spelling unless the spec names that symbol. -- **`_ => true` requires justification.** An accept-all arm must state which types it defers and to which path, or it is an unimplemented check and must not ship as one. -- **Finish the AST migration.** Category B is the enabling condition; the tracked plans are [`CHECKER-ELIMINATE-LINE-SCANNING-PLAN.md`](plans/CHECKER-ELIMINATE-LINE-SCANNING-PLAN.md) and [`CHECKER-TYPE-NARROWING-INFERENCE-PLAN.md`](plans/CHECKER-TYPE-NARROWING-INFERENCE-PLAN.md). `aliases_implicit.rs` was in neither inventory; it has been added to the line-scanning plan as part of this audit. -- **Report the number honestly, including if it drops.** If removing fitted predicates costs conformance points, we publish the lower number. +The correction is **audit and deletion**, not repair. Code that decides from the +spelling of its input rather than its meaning is removed, and what survives is code +that analyses Python. Nothing here is rewritten in place: rewriting preserves the claim +that the rule worked, and that claim is what has to go. + +- **Text-matched logic is deleted, not fixed.** On finding it: write a test that fails + because of it, delete the code, and report what went and why — no fix, no rewrite, no + TODO ([CHKARCH-TEXT-MATCHED-LOGIC](specs/CHECKER-ARCHITECTURE-SPEC.md#CHKARCH-TEXT-MATCHED-LOGIC)). + **A failing test that pins real incorrect behaviour is worth more than a passing + fixture carried by logic that does not analyse code.** What gets built back is a + deliberate, separate decision. +- **A smaller checker is the expected outcome, and an acceptable one.** Rule count, + diagnostic coverage, and the conformance number are all expected to fall. Each drop is + reported. None is reverted, and none is a reason to keep code that was never doing the + work. +- **The pass-percentage floor is the mechanism that caused this** (§6.3) and is no + longer a target ([CHKARCH-CONFORMANCE](specs/CHECKER-ARCHITECTURE-SPEC.md#CHKARCH-CONFORMANCE)). + No conformance figure is published or quoted, and there is no re-submission to + `python/typing` until the semantics-preserving mutation harness passes clean and an + external audit has run. +- **Build the control that would have caught this.** Semantics-preserving mutation — + aliased imports, reformatting, reordering, consistent renaming → identical diagnostics + ([CHKARCH-TESTING-SEMANTIC-MUTATION](specs/CHECKER-ARCHITECTURE-SPEC.md#CHKARCH-TESTING-SEMANTIC-MUTATION)). + It does not exist yet, which is why every green run reinforced the wrong conclusion. + Until it does, no rule is verified. +- **Off-suite tests are mandatory** for every surviving rule, derived from the spec + grammar and real code, explicitly **not** from `conformance/tests/`. +- **Ban hardcoded symbol names as behavioural triggers.** A rule may not key on a + specific identifier spelling unless the spec names that symbol. +- **`_ => true` is an unimplemented check.** An accept-all arm either states which cases + it defers and to which path, or it is deleted along with the rule that relies on it. +- **Finish the AST work.** Category B is the enabling condition; the tracked plans are + [`CHECKER-ELIMINATE-LINE-SCANNING-PLAN.md`](plans/CHECKER-ELIMINATE-LINE-SCANNING-PLAN.md) + and [`CHECKER-TYPE-NARROWING-INFERENCE-PLAN.md`](plans/CHECKER-TYPE-NARROWING-INFERENCE-PLAN.md). + `aliases_implicit.rs` was in neither inventory; it was added as part of this audit. ## 8. Issue index diff --git a/docs/INDEX.md b/docs/INDEX.md index 5d7c3566a..4261597f5 100644 --- a/docs/INDEX.md +++ b/docs/INDEX.md @@ -13,7 +13,7 @@ linked to an active plan. | File | Purpose | |---|---| -| [Checker architecture](specs/CHECKER-ARCHITECTURE-SPEC.md) | Configuration, rules, diagnostics, analysis pipeline, CLI, and quality gates. | +| [Checker architecture](specs/CHECKER-ARCHITECTURE-SPEC.md) | Configuration, rules, diagnostics, analysis pipeline, CLI, and quality gates — including [CHKARCH-TEXT-MATCHED-LOGIC](specs/CHECKER-ARCHITECTURE-SPEC.md#CHKARCH-TEXT-MATCHED-LOGIC), the failing-test → delete → report rule that governs any code deciding from source text. | | [Type inference](specs/CHECKER-TYPE-INFERENCE-SPEC.md) | The bidirectional/constraint inference engine — the checker's single type oracle — its narrowing contracts, research grounding, and the condemned legacy mechanisms under demolition. | | [Stub resolution](specs/CHECKER-STUB-RESOLUTION-SPEC.md) | Pinned typing-spec import order, custom typeshed, offline pin verification against the store, a PyPI-package wheel pin, the segregated download component, bundled stdlib ZIP, and generation. | | [Checker MCP service](specs/CHECKER-MCP-SPEC.md) | Packaged stdio lifecycle and the structured typeshed source/status tool. | @@ -56,10 +56,10 @@ Plans contain only unfinished work. Delete a plan when its acceptance gate passe | [Formatting](plans/LSP-FORMATTING-PLAN.md) | VS Code default-formatter opt-in and published-artifact verification. | | [AI-assisted LSP](plans/LSP-AI-PLAN.md) | First opt-in provider slice and privacy/safety gate. | | [Activity panel](plans/EXTENSION-ACTIVITY-PANEL-PLAN.md) | Settings wiring, Modules-panel context menus and multi-select, and remaining cross-editor/test quality. | -| [Type narrowing and inference](plans/CHECKER-TYPE-NARROWING-INFERENCE-PLAN.md) | The engine build-out and the demolition order: wire the bidirectional engine into every rule, delete every legacy text/shape-matching path, hold the conformance gate throughout and keep an eye on the benchmark. | +| [Type narrowing and inference](plans/CHECKER-TYPE-NARROWING-INFERENCE-PLAN.md) | The engine build-out and the demolition order: wire the bidirectional engine into rules that genuinely analyse code, delete the rest rather than migrating them, and record what each deletion costs. | | [Runtime typeshed resolution](plans/CHECKER-TYPESHED-RUNTIME-PLAN.md) | Two open items: a socket-instrumented witness that checking is offline across CLI/LSP/MCP, and byte-exact per-artifact licensing verification inside the VSIX (binaries and wheels are already verified). | | [PyPI typeshed package pin](plans/CHECKER-TYPESHED-PYPI-PLAN.md) | Pin a PyPI typeshed distribution by wheel SHA-256, verify offline, auto-resolve from `uv.lock`; suppresses the source-status advisory (issue #312). | -| [Eliminate line scanning](plans/CHECKER-ELIMINATE-LINE-SCANNING-PLAN.md) | Replace remaining raw-source rule scans with AST data. | +| [Delete checker text matching](plans/CHECKER-ELIMINATE-LINE-SCANNING-PLAN.md) | The inventory of rules that decide from source text, the failing-test → delete → report disposal, and the unbuilt semantics-preserving mutation harness. | | [WASM](plans/WASM-PLAN.md) | CI wasm build and size ratchet, multi-file in-memory VFS, and the playground site. | | [Advanced checker features](plans/CHECKER-ADVANCED-FEATURES-PLAN.md) | Dependency-hygiene rules, ownership and safety checks, plugin host, migration, and CI helpers. | diff --git a/docs/plans/CHECKER-ADVANCED-FEATURES-PLAN.md b/docs/plans/CHECKER-ADVANCED-FEATURES-PLAN.md index d21fad06e..902e0af63 100644 --- a/docs/plans/CHECKER-ADVANCED-FEATURES-PLAN.md +++ b/docs/plans/CHECKER-ADVANCED-FEATURES-PLAN.md @@ -10,7 +10,11 @@ Concrete-but-unbuilt checker capabilities that the architecture spec describes and that had no owning plan. These are real planned features (not bloat), captured here as ordered TODOs so each spec section is referenced by a plan rather than left orphaned. None of this blocks -the prime directive (PEP conformance) — it is opt-in surface area beyond the spec rules. +the prime directive (accuracy on unseen Python) — it is opt-in surface area beyond the +typing-spec rules, and it waits behind the text-matching audit +([LINESCANPLAN-DISPOSAL](CHECKER-ELIMINATE-LINE-SCANNING-PLAN.md#LINESCANPLAN-DISPOSAL)). +New rules added here must decide on the resolved model from day one; anything that would +ship as a text scan does not ship. --- diff --git a/docs/plans/CHECKER-ELIMINATE-LINE-SCANNING-PLAN.md b/docs/plans/CHECKER-ELIMINATE-LINE-SCANNING-PLAN.md index 411d3ea9c..6d2748217 100644 --- a/docs/plans/CHECKER-ELIMINATE-LINE-SCANNING-PLAN.md +++ b/docs/plans/CHECKER-ELIMINATE-LINE-SCANNING-PLAN.md @@ -1,12 +1,28 @@ -# Eliminate Checker Line Scanning {#LINESCANPLAN-ELIMINATION} +# Delete Checker Text Matching {#LINESCANPLAN-ELIMINATION} -Checker rules must consume Ruff AST or `ResolvedModule` data. Reconstructing -Python structure with `source.lines()` plus `starts_with`, `find`, or `contains` -is parser duplication and can classify strings or comments as code. +Checker rules must decide from the Ruff AST and `ResolvedModule` data. +Reconstructing Python structure with `source.lines()` plus `starts_with`, `find`, +or `contains` is parser duplication: it classifies strings and comments as code, +and it makes a diagnostic depend on how the source is spelled rather than what it +means. -The original docstring failure in `generics_syntax_scoping` is fixed and covered -by a regression test. This plan tracks the remaining rule-level scanners; line -geometry and suppression-comment parsing are the only permitted exceptions. +**This plan deletes that logic. It does not replace it.** Every entry below is +handled by [CHKARCH-TEXT-MATCHED-LOGIC](../specs/CHECKER-ARCHITECTURE-SPEC.md#CHKARCH-TEXT-MATCHED-LOGIC): + +1. **Write a test that fails** because of the code — an aliased import, a + reformatted source, a shape the upstream fixture never contains. +2. **Delete the offending code.** +3. **Tell the user what you deleted and why**, and that the test is now failing. + +Do not fix it, do not rewrite it, do not leave a TODO. **A failing test that pins +real incorrect behaviour is worth more than a passing fixture carried by logic +that does not analyse code.** Expect the rule count and the conformance number to +fall; report both drops plainly and never restore the code to hold either. +Deciding what gets built back is the user's, separately and deliberately. + +The suggested AST replacements recorded below are **evidence of what the deleted +code failed to do**, kept so the user can scope a rebuild. They are not work +items in this plan. ## Current inventory {#LINESCANPLAN-INVENTORY} @@ -17,12 +33,14 @@ rg -n '\.lines\(\)' crates/basilisk-checker/src/rules rg -n 'starts_with\("(class |def |async def |type |@|import |from )' \ crates/basilisk-checker/src/rules rg -n 'slice_span\(.*source' crates/basilisk-checker/src/rules +rg -n '\.(contains|starts_with|ends_with)\(' crates/basilisk-checker/src/rules ``` -The third query is new. The first two find rules that reconstruct *statements* -from lines, but they miss rules that slice a span out of the source and then -pattern-match the resulting **expression** text — which is the same defect one -level down, and is not caught by either keyword query. +The first two find rules that reconstruct *statements* from lines. The third +finds rules that slice a span out of the source and pattern-match the resulting +**expression** text — the same defect one level down. The fourth is the widest +net and returns the most: text predicates appear in the large majority of rule +files, so the inventory below is a starting set, not the full extent. Expression-text scanners: @@ -33,12 +51,12 @@ Expression-text scanners: `type B = list["of genshin"]`, and `type D = list[int].attr` all pass silently, while a name containing the substring `lambda` is a false positive waiting to happen ([#379](https://github.com/Nimblesite/Basilisk/issues/379)). - Replace it with type-expression grammar validation over the `StmtTypeAlias` - value node: allow `Name`, dotted `Attribute` chains, `Subscript` of an allowed - base, `BinOp(|)`, `None`, and forward-reference strings that themselves parse - as valid type expressions; reject everything else, including attribute access - on a `Subscript`. Validation is eager at binding time — PEP 695 lazy - evaluation defers *name resolution*, never *well-formedness*. + *What a real rule would have done:* type-expression grammar validation over the + `StmtTypeAlias` value node — `Name`, dotted `Attribute` chains, `Subscript` of + an allowed base, `BinOp(|)`, `None`, and forward-reference strings that + themselves parse as type expressions; everything else rejected, including + attribute access on a `Subscript`. Validation is eager at binding time — PEP 695 + lazy evaluation defers *name resolution*, never *well-formedness*. - `aliases_implicit.rs` — carries a verbatim duplicate of the same `is_invalid_rhs` scanner, plus three further text heuristics: implicit aliases are detected by an uppercase-first-letter naming test @@ -50,15 +68,15 @@ Expression-text scanners: ParamSpec check is a shape guess that never locates the ParamSpec position ([#409](https://github.com/Nimblesite/Basilisk/issues/409)) and `is_assignable_to_bound` accepts every bound outside `int`/`float`/`complex` - ([#410](https://github.com/Nimblesite/Basilisk/issues/410)). Same fix shape as - `aliases_type_statement.rs`: validate the RHS expression node against the - type-expression grammar and resolve alias-hood from binding information, not - from name spelling. See [#408](https://github.com/Nimblesite/Basilisk/issues/408) - and [`CONFORMANCE-INTEGRITY-AUDIT.md`](../CONFORMANCE-INTEGRITY-AUDIT.md). + ([#410](https://github.com/Nimblesite/Basilisk/issues/410)). *What a real rule + would have done:* validate the RHS expression node against the type-expression + grammar and resolve alias-hood from binding information, never from name + spelling. See [#408](https://github.com/Nimblesite/Basilisk/issues/408) and + [`CONFORMANCE-INTEGRITY-AUDIT.md`](../CONFORMANCE-INTEGRITY-AUDIT.md). - `returns_compatibility.rs` — builds the declared type from annotation source text via `InferredType::from_annotation`. Owned by [NARROWPLAN-ANNOTATION-RESOLUTION](CHECKER-TYPE-NARROWING-INFERENCE-PLAN.md#NARROWPLAN-ANNOTATION-RESOLUTION) - rather than this plan, but it is the same root cause and the same fix shape. + rather than this plan, but it is the same root cause and the same disposal. Structural keyword scanners remain in: @@ -80,26 +98,41 @@ Other statement/body reconstruction remains in: `rules/shared.rs::span_for_line` may read a line for diagnostic geometry. It must not infer Python structure. -## AST migration {#LINESCANPLAN-AST-MIGRATION} - -- [ ] Replace class/function/type-alias discovery with the corresponding - `ResolvedModule` collections and Ruff AST spans. -- [ ] Replace indentation-based body boundaries with AST statement/body ranges. -- [ ] Replace operator, mutation, and call parsing with structured expression or - call records; extend the resolver when the required node is not exposed. -- [ ] Add a string/comment regression fixture for each migrated rule before - deleting its scanner. -- [ ] Replace `aliases_type_statement::is_invalid_rhs` with AST type-expression - validation, covering the reported cases above plus operators other than `|`, - call expressions outside the sanctioned special forms, comparisons, - comprehensions, and literal displays. -- [ ] Preserve the exact diagnostics for real code and keep conformance at - 141/141 with zero missed errors and zero false positives. - -Migrate `generics_variance_inference` first: it owns the largest cluster of raw -line and keyword scans. Then take `aliases_type_statement`, which is the only -inventory entry with a confirmed user-visible miss. Then remove the smaller -body scanners in the inventory above. +## Disposal {#LINESCANPLAN-DISPOSAL} + +- [ ] For every inventory entry: failing test → delete → report. One rule per + change, so each deletion and its drop are individually visible. +- [ ] The failing test must fail on **meaning, not spelling** — an aliased + import, a reformatted source, or a construct the upstream suite omits. A test + that only reproduces the fixture proves nothing. +- [ ] Record each deletion in the report: what went, which test now fails, and + what the conformance run did afterwards. A drop is the expected outcome and is + reported, never absorbed. +- [ ] Extend the inventory as the sweep widens. It was built from four queries + and the fourth alone matches most rule files, so treat every unlisted rule as + unaudited rather than clean. +- [ ] Never re-derive a deleted check from the same text predicates under a new + name. If the analysis is worth having, the user scopes it as new work against + the resolved model. + +## Semantics-preserving mutation harness {#LINESCANPLAN-SEMANTIC-MUTATION} + +Specified by +[CHKARCH-TESTING-SEMANTIC-MUTATION](../specs/CHECKER-ARCHITECTURE-SPEC.md#CHKARCH-TESTING-SEMANTIC-MUTATION) +and **not built**. It is the only gate that distinguishes a rule that analyses +code from a rule that matches text, and its absence is why this logic survived +every green run. Until it exists, no rule may be described as spec-implementing +on the strength of a fixture alone. + +- [ ] Build the harness: re-run each rule test over semantically identical, + textually different input (aliased imports, alternate import forms, + reformatting, quote style, consistent renaming, statement reordering, comment + churn) and require **byte-for-byte identical diagnostics**. +- [ ] Report coverage as a fraction of rules exercised, and **name the uncovered + remainder**. A score over a hand-picked subset says nothing about the rest. +- [ ] Wire it into `make test`. A rule whose diagnostics move under mutation is + handled by the three-step disposal above — never by teaching the harness to + tolerate the difference, and never by mutating the expectation to match output. ## Enforcement {#LINESCANPLAN-ENFORCEMENT} @@ -113,8 +146,10 @@ body scanners in the inventory above. ## Acceptance {#LINESCANPLAN-ACCEPTANCE} -- No checker rule infers Python structure from raw lines. +- No checker rule infers Python structure from raw source text. - Docstrings, comments, and string literals containing `class`, `def`, `type`, decorators, or imports produce no structural diagnostics. -- Focused rule tests, `make lint`, `make test`, and the live conformance harness - pass without weakening any ratchet. +- The semantics-preserving mutation harness runs in `make test` and every + surviving rule passes it. +- The set of deletions, the tests left failing behind them, and the resulting + conformance drop are all on the record. diff --git a/docs/plans/CHECKER-TYPE-NARROWING-INFERENCE-PLAN.md b/docs/plans/CHECKER-TYPE-NARROWING-INFERENCE-PLAN.md index ad1ab093a..03f2b6828 100644 --- a/docs/plans/CHECKER-TYPE-NARROWING-INFERENCE-PLAN.md +++ b/docs/plans/CHECKER-TYPE-NARROWING-INFERENCE-PLAN.md @@ -4,12 +4,15 @@ Specs: [TYPEINF-OVERVIEW](../specs/CHECKER-TYPE-INFERENCE-SPEC.md#TYPEINF-OVERVI [TYPEINF-TARGET](../specs/CHECKER-TYPE-INFERENCE-SPEC.md#TYPEINF-TARGET), and [CHKARCH-INFERENCE](../specs/CHECKER-ARCHITECTURE-SPEC.md#CHKARCH-INFERENCE). -Basilisk already passes the typing conformance suite. This plan therefore has two +Basilisk's rules largely decide from source text rather than resolved symbols +([CONFORMANCE-INTEGRITY-AUDIT](../CONFORMANCE-INTEGRITY-AUDIT.md)), so a passing +suite is not evidence that the inference under them works. This plan has two tracks that share one implementation: 1. **Consolidation** — merge duplicated rule-local inference into shared - components and improve editor/user behavior without weakening the - zero-false-positive gate. + components and improve editor/user behavior. Where consolidation uncovers a + rule that matches text instead of analysing code, that rule is deleted rather + than consolidated ([CHKARCH-TEXT-MATCHED-LOGIC](../specs/CHECKER-ARCHITECTURE-SPEC.md#CHKARCH-TEXT-MATCHED-LOGIC)). 2. **A substantially more powerful inference engine** — bidirectional (synthesis + checking) typing over a subtype-constraint solver, per the target architecture in @@ -42,19 +45,22 @@ environment, expression inferrer, constraint solver, or subtype context. must become powerful enough (bidirectional context, constraint solving, bounded type-level evaluation) that PEP 827's conditional/mapped types have a sound home if adopted later. -- Preserve the gradual guarantee as a testable invariant, keep the - zero-false-positive conformance gate, and hold both benchmark ratchets - ([CHKARCH-TESTING-BENCH](../specs/CHECKER-ARCHITECTURE-SPEC.md#CHKARCH-TESTING-BENCH)). -- **Winning is the exit criterion, not an aspiration — and integration comes - first.** Basilisk MUST end this plan with measurably better type inference - than pyright, mypy, ty, pyrefly, and zuban, wired into the shipped checker — - a lead held by a detached engine counts for nothing. The plan is not - complete while any competitor leads any axis in - [NARROWPLAN-TARGETS](#NARROWPLAN-TARGETS); the mechanism that makes the - claim honest, enforceable, and permanent is the post-integration scoreboard - ratchet in [NARROWPLAN-SCOREBOARD](#NARROWPLAN-SCOREBOARD), which starts - only after [NARROWPLAN-INTEGRATION](#NARROWPLAN-INTEGRATION) has the engine - behind live diagnostics. +- Preserve the gradual guarantee as a testable invariant, and keep false + positives on real code falling. Benchmarks are indicative and gate nothing + ([CHKARCH-TESTING-BENCH](../specs/CHECKER-ARCHITECTURE-SPEC.md#CHKARCH-TESTING-BENCH)); + the conformance run is recorded, not gated + ([CHKARCH-CONFORMANCE](../specs/CHECKER-ARCHITECTURE-SPEC.md#CHKARCH-CONFORMANCE)). +- **Correctness first; comparison last.** The exit criterion is inference that + is right on Python it has never seen, wired into the shipped checker — an + engine that is not behind live diagnostics counts for nothing. Comparative + axes in [NARROWPLAN-TARGETS](#NARROWPLAN-TARGETS) are direction, not a + finish line, and the scoreboard in + [NARROWPLAN-SCOREBOARD](#NARROWPLAN-SCOREBOARD) starts only after + [NARROWPLAN-INTEGRATION](#NARROWPLAN-INTEGRATION). **No scoreboard on this + plan may become a target.** This repo has already been through a metric that + outranked the analysis under it; a self-built harness that Basilisk both + authors and is graded by is the same shape of risk, and it is worth less than + a single failing test that pins real incorrect behaviour. **Non-goals** @@ -328,23 +334,33 @@ protection has expired. Wire it in.** being deleted and never by an `#[allow]` — each stays `pub` from the crate root, which is what keeps the workspace's `dead_code = "deny"` satisfied. -### What is NOT on the demolition list — read this before touching anything - -Ripping out legacy *mechanism* is mandatory. Weakening the *checker* is -forbidden, and nothing in this section licenses it: - -- **Never delete, disable, or unregister a rule.** Not one. The rule survives; - its guts get replaced. See [CHKARCH-CONFORMANCE]. -- **Never remove a diagnostic.** Post-migration output is identical or - strictly better — same code, same span, same or clearer message. -- **Never touch the scoreboard.** 100% / 0 false positives against a freshly - cloned `python/typing` harness is the prime directive and outranks this - entire plan. A migration that drops the number is reverted, not negotiated. +### The boundary — read this before touching anything + +Ripping out legacy *mechanism* is mandatory. The boundary is **intent and +disclosure**, not whether code disappears: + +- **A rule that decides from source text gets deleted, not migrated.** Failing + test, delete, report — [CHKARCH-TEXT-MATCHED-LOGIC](../specs/CHECKER-ARCHITECTURE-SPEC.md#CHKARCH-TEXT-MATCHED-LOGIC). + Wiring such a rule to the engine "so the diagnostic survives" preserves a + claim that was never true, and hides the hole behind a green run. **A failing + test that pins real incorrect behaviour is worth more than a passing fixture + carried by logic that does not analyse code.** +- **A rule that genuinely analyses code keeps its diagnostic across the + migration** — same code, same span, same or clearer message. Losing one of + those silently is the failure this bullet guards. +- **Deleting a rule to move a number is dishonest; deleting text-matched logic + is required.** The difference shows in what you do next: the honest deletion + leaves a failing test behind, is reported to the user, and is expected to + *lower* the conformance number. Report the drop. Never revert the deletion, + refit the rule, or edit a threshold to get green + ([CHKARCH-CONFORMANCE](../specs/CHECKER-ARCHITECTURE-SPEC.md#CHKARCH-CONFORMANCE)). - **Never add an alternate checking mode**, feature flag, or "new engine" toggle. There is one code path. Basilisk has no modes. -If replacing a rule's guts costs a required error, the engine is not ready for -that rule yet — **fix the engine**, then come back. Do not ship the loss. +If wiring a rule that really does analyse code would cost a required error, the +engine is not ready for it yet — fix the engine, then come back. That is a +different situation from a text-matching rule, which goes regardless of what the +engine can do today. ### Order of demolition — every step closes filed bugs @@ -376,14 +392,26 @@ sequenced checkboxes live in the checklist ### Gates that stay armed the entire time -Non-negotiable, every step, no exceptions: - -- Live conformance run: **100% / 0 FP**, freshly cloned harness - ([CHKARCH-CONFORMANCE-MODE]). -- `make bench`: no fixture slower than the committed baseline - ([CHKARCH-TESTING-BENCH]). The walker is real production cost the - moment step 1 lands — record the baseline **in that same change**. -- `make test` fail-fast, coverage ratchet up, mutation ratchet up. +Every step, no exceptions: + +- **Semantics-preserving mutation** over every migrated rule: aliased imports, + reformatting, reordering → byte-for-byte identical diagnostics + ([CHKARCH-TESTING-SEMANTIC-MUTATION](../specs/CHECKER-ARCHITECTURE-SPEC.md#CHKARCH-TESTING-SEMANTIC-MUTATION)). + This is the gate that decides whether a migration produced analysis or moved + the text matching somewhere less visible. It does not exist yet + ([LINESCANPLAN-SEMANTIC-MUTATION](CHECKER-ELIMINATE-LINE-SCANNING-PLAN.md#LINESCANPLAN-SEMANTIC-MUTATION)); + until it does, every migrated rule is unverified and must be described that way. +- **Off-suite tests** for every migrated rule, derived from the typing spec and + real code, **never** from `conformance/tests/`. +- A live conformance run, **recorded not gated**: it detects unintended + regressions in rules that were working. A drop traced to a deliberate deletion + is reported and kept ([CHKARCH-CONFORMANCE](../specs/CHECKER-ARCHITECTURE-SPEC.md#CHKARCH-CONFORMANCE)). +- `make bench`: indicative only, gates nothing ([CHKARCH-TESTING-BENCH]). The + walker is real production cost the moment step 1 lands — record the baseline + **in that same change**. +- `make test` fail-fast, coverage ratchet up, mutation ratchet up — read against + its scope denominator, never as a standalone score + ([CHKARCH-TESTING-MUTATION-RATCHET](../specs/CHECKER-ARCHITECTURE-SPEC.md#CHKARCH-TESTING-MUTATION-RATCHET)). - Torture golden gate green. ### The cost defect that blocked all of this — FIXED @@ -490,10 +518,16 @@ seeded (below, first checklist item) stays live because it scores the *shipped checker*; the remaining axes are built only after the wiring they would measure exists. -Basilisk MUST end this plan with better type inference than every -officially-recognized competitor. "Better" is defined operationally and -enforced exactly the way this repo already enforces conformance and speed — -self-measured, reproducible, write-always, ratcheted: +**A scoreboard is a diagnostic instrument, never an objective.** Basilisk both +authors this harness and is graded by it, which is the same configuration that +produced the fitted predicates in +[CONFORMANCE-INTEGRITY-AUDIT](../CONFORMANCE-INTEGRITY-AUDIT.md): when the only +metric that can move is one the code can be shaped to, shaping it is the cheapest +way to move it. So every axis below carries the same standing condition — **a +score that rises without the analysis under it improving is a regression**, and +an axis is worth abandoning the moment it starts driving the work instead of +reporting on it. "Better" is defined operationally, measured the way this repo +measures speed — self-measured, reproducible, write-always: - **Definition.** Basilisk is superior on an axis when it scores strictly better than the LATEST official release of every officially-recognized @@ -513,12 +547,12 @@ self-measured, reproducible, write-always, ratcheted: unconditionally** (WRITE-ALWAYS); a separate read-only gate compares against the committed baseline (GATE-SEPARATELY). A run that measured a score but didn't record it is a lie. -- **Ratchet.** Once Basilisk takes the lead on an axis, the lead becomes a CI - gate: falling behind any competitor on a led axis is a build failure. Leads - only accumulate. The plan exits only when Basilisk leads **all five axes - simultaneously** while the 100%/0-FP conformance gate and the speed - benchmark stay healthy — the inference lead must never be bought by - regressing conformance or performance, and vice versa. +- **Reported, not ratcheted.** Scores are recorded and read by a human. A lead + does **not** become a CI gate: a build that fails on a comparative number + makes losing the number more expensive than losing the analysis, which is the + incentive this repo is removing, not adding. A regression on an axis is a + question to investigate, and the answer is sometimes "we deleted logic that + was never analysing anything" — which is progress, and is recorded as such. - **Moving targets.** Because the harness pulls latest competitor releases, the lead is continuously re-proven against competitors as they improve — never against frozen versions. If a competitor release takes back an axis, @@ -586,8 +620,11 @@ self-measured, reproducible, write-always, ratcheted: mypy, ty, pyrefly, and zuban on **every** axis in [NARROWPLAN-TARGETS](#NARROWPLAN-TARGETS), and the per-axis ratchet is wired into CI so the lead cannot silently erode. -- `make test`, mutation/coverage ratchets, benchmarks for touched hot paths, and - the live 141/141 conformance gate all pass with zero false positives. +- Every surviving rule passes semantics-preserving mutation + ([CHKARCH-TESTING-SEMANTIC-MUTATION](../specs/CHECKER-ARCHITECTURE-SPEC.md#CHKARCH-TESTING-SEMANTIC-MUTATION)) + and carries off-suite tests; `make test`, the mutation/coverage ratchets, and + benchmarks for touched hot paths are green. The conformance run is recorded + with whatever it reports. ## Checklist {#NARROWPLAN-CHECKLIST} @@ -633,8 +670,15 @@ self-measured, reproducible, write-always, ratcheted: Prerequisite for Stage 2; see [NARROWPLAN-ANNOTATION-RESOLUTION](#NARROWPLAN-ANNOTATION-RESOLUTION). Each box -lands with a regression test that fails before it and passes after, and holds -the conformance ratchets (100% / 0 false positives) at every step. +lands with a regression test that fails before it and passes after, tested on +Python the upstream suite does not contain, and records what the conformance run +reported afterwards. + +> **Figures in completed (`[x]`) entries below are historical records** of runs +> made when the pass percentage was a gate. They are kept because they say what +> was actually measured at the time. They are **not** an acceptance criterion for +> new work, not a bar to restore, and not for publication — the claim they came +> from is withdrawn ([CHKARCH-CONFORMANCE](../specs/CHECKER-ARCHITECTURE-SPEC.md#CHKARCH-CONFORMANCE)). **The test step is part of the box, never a phase at the end.** A box is `[x]` only when its own nested `[x] Test:` line names a test that (a) was written and @@ -1376,6 +1420,16 @@ before the stage is declared closed: outright. An annotation is a type expression the engine evaluates — never a string a rule slices out of the file. Fixes #379; retires the mechanism behind #383. + + **Migration is not the only disposal, and often not the right one.** Where a + consumer can be pointed at the engine and the rule's *meaning* is preserved, + migrate it. Where the text scan **was** the rule — where there is no analysis + underneath to reconnect — the entry belongs to + [LINESCANPLAN-DISPOSAL](CHECKER-ELIMINATE-LINE-SCANNING-PLAN.md#LINESCANPLAN-DISPOSAL): + failing test, delete, report. Do not migrate a text scan into an engine call + that reproduces the same spelling-dependent verdict; that hides the defect + behind a better-looking call site. A migrated rule is only done once it passes + semantics-preserving mutation. *Measured state (2026-08-05, after the Step 7a pass): `InferredType::from_annotation` call sites in rules **11 → 2**. MIGRATED to the cascade, each verified at 4067/0 + 141/141: @@ -1525,7 +1579,17 @@ before the stage is declared closed: - [x] Keep every rule registered and every diagnostic intact through the whole demolition. The mechanism dies; the checking does not. A migration that costs a required error means the engine is not ready — **fix the engine**, - never ship the loss ([CHKARCH-CONFORMANCE]). Held: `all_rules()` still + never ship the loss ([CHKARCH-CONFORMANCE]). + + > **Superseded for text-matched rules.** This box records what was held while + > the directive was "no rule ever goes". It now applies only to rules that + > genuinely analyse code. A rule that decides from source text is deleted, not + > migrated — failing test, delete, report + > ([CHKARCH-TEXT-MATCHED-LOGIC](../specs/CHECKER-ARCHITECTURE-SPEC.md#CHKARCH-TEXT-MATCHED-LOGIC)). + > Rule count and conformance are both expected to fall, and both drops get + > reported rather than prevented. + + Held: `all_rules()` still registers **166** rules, no rule source was deleted or unregistered, no suppressing config exists, and conformance stayed 141/141 with 0 missed through every step. The directive was exercised for real and obeyed @@ -1571,8 +1635,9 @@ before the stage is declared closed: carries information: a display oracle that rendered `Any` for everything would satisfy the acceptances but fails here. A second inference path for displays would be caught by every case in the file. -- [ ] `make test`, mutation/coverage ratchets, benchmarks for touched hot - paths, and the live conformance gate all pass with zero false positives. +- [ ] `make test`, mutation/coverage ratchets, and benchmarks for touched hot + paths are green; every surviving rule passes semantics-preserving mutation + and off-suite tests; the conformance run is recorded with whatever it reports. *Measured 2026-08-05 after the subtyping/Step-8/Step-7a pass — green:* *workspace `cargo test --workspace` **7253 / 0** (checker 4067, resolver 625, and every other crate), `cargo clippy --workspace --all-targets` @@ -1687,11 +1752,15 @@ detached engine. - [ ] Wire the utahplt/ifT narrowing benchmark, the higher-order corpus, the gradual-guarantee differential suite, and the incremental-latency measurement into the scoreboard. -- [ ] Add per-axis ratchet entries: once Basilisk leads an axis, falling +- [ ] ~~Add per-axis ratchet entries~~ — **dropped.** Scores are recorded and + read, never gated ([NARROWPLAN-SCOREBOARD](#NARROWPLAN-SCOREBOARD)). Retained + below only to show what was removed and why: a CI gate on a comparative number + makes losing the number more expensive than losing the analysis. Original + wording: once Basilisk leads an axis, falling behind any competitor on that axis fails CI; leads only accumulate. -- [ ] Take and hold the lead on **all five axes simultaneously**, with the - 100%/0-FP conformance gate green and the speed benchmark healthy in the same - run. +- [ ] Measure and record all five axes in one run, alongside the speed benchmark + and the conformance result. A lead is reported, never gated; an axis that + starts driving the work instead of describing it is dropped. - [ ] Enforce claims discipline: every better-than-competitor claim in docs, website, or marketing traces to the current committed scoreboard run and states the methodology. diff --git a/docs/plans/CHECKER-TYPESHED-PYPI-PLAN.md b/docs/plans/CHECKER-TYPESHED-PYPI-PLAN.md index 4014fc425..af529233b 100644 --- a/docs/plans/CHECKER-TYPESHED-PYPI-PLAN.md +++ b/docs/plans/CHECKER-TYPESHED-PYPI-PLAN.md @@ -95,9 +95,10 @@ installed `site-packages` tree (the stored wheel is the source). ## CI gate {#TYPESHEDPYPI-CI} `make test` (fail-fast, coverage ratchet up), clippy + fmt at strictest, `make lint` (incl. `scripts/check-dependency-shape.sh` — `basilisk-stubs` still links no HTTP client), `deslop`, and -conformance 100 % / 0 FP unchanged (advisories never enter the scored stream) — all green. +and the conformance run recorded unchanged (advisories are environment status, not Python +diagnostics, so they never enter the diagnostic stream) — all green. -`make bench` (zero-tolerance baseline gate) also guards the branch, but its outstanding failure is +`make bench` also ran against the branch, but its outstanding regression is **not attributable to this plan** and is tracked as its own task; see [Cross-cutting gates](#cross-cutting-gates) for the evidence. This plan neither claims a benchmark result nor licenses re-baselining to slower numbers. @@ -190,5 +191,5 @@ result nor licenses re-baselining to slower numbers. - **The `rustls`/`ureq` dyld hypothesis is disproven**, and any note repeating it is wrong: `basilisk-cli` already depended on `basilisk-typeshed-fetch` on `main`, so the TLS stack was linked into the binary that produced the 6.2 ms baseline. - **The committed baseline is stale, not just slow**: it was last written by `009f2556` (2026-07-18) while `main` has since merged through `e3e97d30` (2026-08-01, #377). Many merged PRs sit between the baseline and this branch, so nothing attributes the delta to this branch without a same-machine A/B of `main` HEAD vs this branch. - **Part of the delta is environmental**: the two runs pin identical competitor versions, and pyright/mypy/ty/pyrefly/zuban all shifted 3–10 % between them — real, but far short of basilisk's ~50 % on the fast fixtures, so a genuine fixed per-process cost remains to be found. - - The ratchet rule is unchanged: the baseline may not be advanced to slower numbers. Recovering the cost — not re-baselining — is the exit condition, and it belongs to the benchmark task, not to this one. + - Recovering the cost — not re-baselining — is the exit condition, and it belongs to the benchmark task, not to this one. The benchmark itself gates nothing ([CHKARCH-TESTING-BENCH](../specs/CHECKER-ARCHITECTURE-SPEC.md#CHKARCH-TESTING-BENCH)); it is read by a human, and no number here passes or fails a build. - [x] Conformance 100 % / 0 FP unchanged (advisories never enter the scored stream; conformance fixtures ran green inside `_test_rust`). diff --git a/docs/plans/ROADMAP-NEXT-STEPS-PLAN.md b/docs/plans/ROADMAP-NEXT-STEPS-PLAN.md index 682ed6859..8f2bfb52d 100644 --- a/docs/plans/ROADMAP-NEXT-STEPS-PLAN.md +++ b/docs/plans/ROADMAP-NEXT-STEPS-PLAN.md @@ -6,41 +6,46 @@ the engineering detail. Current baseline: -- The unmodified `python/typing` harness passes 141/141 files with zero missed - errors and zero false positives. Conformance is now a permanent ratchet, not a - project plan. +- **The checker is under audit.** Most rules decide from raw source text rather + than resolved symbols, so a green run says little about whether Basilisk + analyses Python it has never seen + ([CONFORMANCE-INTEGRITY-AUDIT](../CONFORMANCE-INTEGRITY-AUDIT.md)). The active + work is finding that logic, pinning it with failing tests, and deleting it — + not raising any number. - Tagged-release automation builds the binaries and editor artifacts, stamps placeholder versions, publishes VSIX packages to the Microsoft and Open VSX registries, and refreshes the Neovim and Zed mirror repositories. - VS Code, Neovim, and Zed share the Rust LSP. Editor-specific manual publication and clean-install validation remain. -## Coverage beyond the upstream suite {#NEXTSTEPS-BEYOND-CONFORMANCE} +## Accuracy beyond the upstream suite {#NEXTSTEPS-BEYOND-CONFORMANCE} -Conformance remains the prime directive and both ratchets stand. But a batch of -user-reported typing puzzles (2026-08-01, issues +**Accuracy on unseen Python is the prime directive.** The upstream suite is a +downstream sample of it, and one this codebase is overfitted to +([CHKARCH-CONFORMANCE](../specs/CHECKER-ARCHITECTURE-SPEC.md#CHKARCH-CONFORMANCE)). +A batch of user-reported typing puzzles (2026-08-01, issues [#378](https://github.com/Nimblesite/Basilisk/issues/378)–[#383](https://github.com/Nimblesite/Basilisk/issues/383), -[#371](https://github.com/Nimblesite/Basilisk/issues/371)) established something -we should hold onto: **every one of those defects coexisted with a clean 141/141 -run**, and each reproduced on the CLI as well as the playground. +[#371](https://github.com/Nimblesite/Basilisk/issues/371)) made the point +concrete: **every one of those defects coexisted with a clean run of the whole +suite**, and each reproduced on the CLI as well as the playground. -The suite is a floor, not a ceiling. Two concrete blind spots it does not cover: +Two blind spots, both of which a perfect score concealed: - `conformance/tests/aliases_recursive.py` contains **zero** PEP 695 `type` statements — every recursive case upstream uses the legacy spelling — so - rejecting every non-generic recursive `type` alias scored 100%. + rejecting every non-generic recursive `type` alias passed the file. - Nothing upstream pins "return a `str` literal from a function annotated with an alias-of-`int`", so skipping assignability for every nominal annotation - scored 100%. + passed the file. Neither is an upstream flaw to route around: a syntax the suite omits is *our* responsibility to cover. -- [ ] **`[AGENT]`** Own a Basilisk-side regression suite for constructs the - upstream suite omits, starting with a PEP 695 `type`-statement counterpart of - every recursive case in `aliases_recursive.py`. It grows whenever a - user-reported defect turns out to be uncovered upstream; it never substitutes - for the live harness. +- [ ] **`[AGENT]`** Own a Basilisk-side regression suite of Python the upstream + suite has never contained, starting with a PEP 695 `type`-statement + counterpart of every recursive case in `aliases_recursive.py`. Cases come from + the typing spec and real code, **never** from `conformance/tests/`. It grows + whenever a user-reported defect turns out to be uncovered upstream. - [ ] **`[AGENT]`** For each user-reported defect, record whether the upstream suite covered the construct. A "no" is a coverage-gap ticket in its own right, not just a bug fix. @@ -49,6 +54,12 @@ responsibility to cover. decorator) and [#381](https://github.com/Nimblesite/Basilisk/issues/381) (call not in outermost position) disable real rules with no signal, which no pass-percentage metric can surface. +- [ ] **`[AGENT]`** Sweep `crates/basilisk-checker/src/rules/` for text-matched + logic and handle each find by + [CHKARCH-TEXT-MATCHED-LOGIC](../specs/CHECKER-ARCHITECTURE-SPEC.md#CHKARCH-TEXT-MATCHED-LOGIC): + failing test, delete, report. Do not fix, rewrite, or leave a TODO. Expect the + rule count and the conformance number to fall; report both drops plainly. + What gets built back is the user's decision, made deliberately and separately. ## Distribution follow-ups {#NEXTSTEPS-DISTRIBUTION} diff --git a/docs/plans/WASM-PLAN.md b/docs/plans/WASM-PLAN.md index 0cba033b9..edab8eb60 100644 --- a/docs/plans/WASM-PLAN.md +++ b/docs/plans/WASM-PLAN.md @@ -46,13 +46,14 @@ The coupling is concentrated, so this is an extraction rather than a rewrite: `exports.rs`. - Introduce a source-provider trait for directory listing and file reading. -- Keep the native implementation byte-identical in behaviour — this must not - move a single conformance result. The conformance suite is the gate. +- Keep the native implementation byte-identical in behaviour — a source-provider + refactor must change no diagnostic on any input. Prove it with the rule suites + and a recorded conformance run, not with the conformance run alone. - Supply an in-memory implementation for wasm and extend the API to accept a set of named sources. -**Gate:** a two-file playground program resolves its own imports, with -conformance still 100% / 0 false positives. +**Gate:** a two-file playground program resolves its own imports, with the rule +suites green and the conformance run recorded unchanged. ## 3. Playground site {#WASM-PLAN-SITE} diff --git a/docs/specs/CHECKER-ARCHITECTURE-SPEC.md b/docs/specs/CHECKER-ARCHITECTURE-SPEC.md index 2f414dcdb..6bdc1dffb 100644 --- a/docs/specs/CHECKER-ARCHITECTURE-SPEC.md +++ b/docs/specs/CHECKER-ARCHITECTURE-SPEC.md @@ -10,7 +10,7 @@ Basilisk has **no modes** (no `--strict`, no `off`/`basic`/`standard`/`strict` d Configuration **grades**; commands **select**. The config file never chooses commands, and there are no presets, mutation intents, or rule-family booleans. Strict-by-default is delivered by the LSP's one-time two-line seed — `"basilisk" = "error"` — never by hidden defaults ([LSPARCH-CONFIG-SEEDING](LSP-ARCHITECTURE-SPEC.md#LSPARCH-CONFIG-SEEDING)). -No PEP rule may be disabled, deleted, or unregistered to move the conformance number ([CHKARCH-CONFORMANCE-MODE](#CHKARCH-CONFORMANCE-MODE)). +No PEP rule may be disabled, deleted, or unregistered **to move a conformance number** ([CHKARCH-CONFORMANCE-MODE](#CHKARCH-CONFORMANCE-MODE)). Deleting a rule **because it decides from source text rather than resolved symbols** is a different act and a required one — failing test, delete, report ([CHKARCH-TEXT-MATCHED-LOGIC](#CHKARCH-TEXT-MATCHED-LOGIC)). ### The partition {#CHKARCH-COMMANDS} @@ -264,7 +264,7 @@ configuration/editor behavior is specified by ### Python Typing PEP Coverage {#CHKARCH-PEPS} -Basilisk's **target** is 100% conformance with the Python typing specification. We measure against the latest **`python/typing@main`**, recording the exact graded commit by hash in `conformance_report.json` (currently [`a490662`](https://github.com/python/typing/tree/a4906624f170c169cf667f962080c56d5a5ba6ff/conformance)). Today the official scorer, run unmodified in CI on the binary in its default configuration (the PEP conformance set; see [CHKARCH-CONFORMANCE-MODE](#CHKARCH-CONFORMANCE-MODE)), reports **141 of 141 files passing (100.0%)**, with **0 false positives** and **0 missed required errors** (970 caught). We run that suite in CI on every change; the gate ratchets the pass-percentage **up** and the false-positive ceiling **down** — closed only by fixing the checker, never by disabling a rule. +Basilisk's target is to **implement the Python typing specification** — to decide these PEPs correctly on Python it has never seen. The PEPs below name the analysis each rule owes; passing a fixture that exercises one is evidence, not achievement. A rule that reaches the right verdict on the upstream test file and the wrong verdict on an aliased import, a reformatted source, or a spelling the suite never contains has not implemented its PEP, and the table's "Required" is a statement of intent about the analysis, never about a score. Where the two disagree, the specification wins and the number is discarded ([CHKARCH-CONFORMANCE](#CHKARCH-CONFORMANCE)). #### Foundation PEPs {#CHKARCH-PEPS-FOUNDATION} @@ -431,10 +431,53 @@ default, and the implementation plan is ### Design Philosophy {#CHKARCH-DIAG-PHILOSOPHY} Every diagnostic must be: -1. **Precise** -- exact location (file, line, column, span) -2. **Clear** -- explains what is wrong and why -3. **Actionable** -- suggests at least one fix -4. **Stable** -- error codes are never renumbered or reused +1. **Semantic** -- decided from the resolved model, never from source text ([CHKARCH-TEXT-MATCHED-LOGIC](#CHKARCH-TEXT-MATCHED-LOGIC)) +2. **Precise** -- exact location (file, line, column, span) +3. **Clear** -- explains what is wrong and why +4. **Actionable** -- suggests at least one fix +5. **Stable** -- error codes are never renumbered or reused + +### Text-matched logic — test, delete, report {#CHKARCH-TEXT-MATCHED-LOGIC} + +A rule whose behaviour depends on the **spelling** of its input rather than its +**meaning** is broken, whatever it scores. `from typing import Final as F` must behave +identically to `typing.Final`; reformatting a file must change no diagnostic. Removing +this logic outranks every other checker task. + +**Detection signatures:** + +- Raw source-text matching — `.contains` / `starts_with` / `ends_with` on user code. +- Hard-coded symbol spellings instead of resolved identity: `t == "typing.Final"`, + `text.starts_with("Callable[")`, `import.module == "typing"`. +- Any regex over Python source. +- Logic keyed to a fixture: rule files named after upstream test files, branches for + shapes only that suite contains, comments citing a test file as justification. +- Detection that fires on formatting — line breaks, spacing, quote style, comment + text, statement order. +- An accept-all fallback (`_ => true`) standing in for the remainder of the type + system, rather than deferring a named set of cases to a named path. + +**On encountering it, do exactly three things — do not fix it, do not rewrite it, do +not leave a TODO:** + +1. **Write a test that fails** because of the code — pin the real defect: an aliased + import, a reformatted source, a shape the fixture never contains. +2. **Delete the offending code.** +3. **Tell the user what you deleted and why**, and that the test is now failing. + +Replacing it is not an agent's call. The point is to surface each one so the user can +acknowledge it and decide what gets built back. **A failing test that pins real +incorrect behaviour is worth more than a passing fixture carried by logic that does not +analyse code** — the first records what Basilisk cannot do, the second falsely claims +it can. A checker with fewer rules and visible failing tests is the correct outcome; a +diagnostic that only fires on one spelling looks like coverage and isn't. + +**What a correct rule looks like** — the yardstick for judging code, not licence to go +and fix it: it decides on the resolved semantic model from `basilisk-resolver`, parses +with `ruff_python_parser`, is named for the typing-spec concept it implements rather +than a test file, survives semantics-preserving mutation +([CHKARCH-TESTING-SEMANTIC-MUTATION](#CHKARCH-TESTING-SEMANTIC-MUTATION)), and is +tested against Python the upstream suite has never contained. ### Error Code System {#CHKARCH-DIAG-CODES} @@ -893,11 +936,11 @@ the database's memory footprint scales with the workspace (every analysed file's inputs and memos stay resident for the session — the standard incremental-engine trade). -**Scope — the CLI/conformance path is deliberately unchanged.** The batch CLI -(`process_file`) still runs the direct pipeline, so this work **cannot affect the -conformance score**. Routing the CLI (and the LSP's bulk scan) through the engine -is future work — the CLI is the conformance path (must prove byte-for-byte parity -first) and, being one-shot, reuses no memos. The engine is a public API +**Scope — the batch CLI path is deliberately unchanged.** The batch CLI +(`process_file`) still runs the direct pipeline, so this work cannot change any +diagnostic it emits. Routing the CLI (and the LSP's bulk scan) through the engine is +future work: it must first prove byte-for-byte diagnostic parity on real Python, and +being one-shot it reuses no memos. The engine is a public API (`basilisk_checker::{BasiliskDatabase, SourceFile, ConfigInput, ConfigValue, SearchPathsInput, WorkspaceFiles, ModuleExports, checked_file, file_diagnostics, resolved_module, module_exports, cross_resolved_module, @@ -1371,51 +1414,121 @@ a design target, not a claim of existing measurement. |---|---|---| | Unit tests | `cargo test` per crate | Crate-level correctness | | Integration tests | Multi-file scenarios | Cross-module type checking | -| Conformance tests | Python typing test suite | PEP compliance (target: 100%) | +| Semantics-preserving mutation | Aliased imports, reformatting, reordering over rule tests | **The accuracy gate** — proves a rule decides on meaning, not spelling ([CHKARCH-TESTING-SEMANTIC-MUTATION]) | +| Off-suite rule tests | Python the upstream suite has never contained | Proves a rule generalises beyond its fixture | +| Conformance tests | Python typing test suite, run live | Regression detector only — never a target ([CHKARCH-CONFORMANCE]) | | Golden file tests | Expected diagnostic output | Diagnostic regression | | Fuzzing | `cargo-fuzz` | Crash resistance, soundness | | Property tests | `proptest` crate | Type system invariants | | Benchmarks | `make bench` (hyperfine, `benchmarks/run.sh`) vs Pyright/mypy/ty/Pyrefly/Zuban | Indicative performance tracking, written to `benchmarks/status/.csv` immediately, every run. **Gates nothing** — developer-machine numbers, compared between tools within one run ([CHKARCH-TESTING-BENCH]) | -### PEP Conformance Scoring {#CHKARCH-CONFORMANCE} +A test is judged by what it would catch, never by whether it is green. **A failing +test that pins real incorrect behaviour is worth more than a passing fixture carried +by logic that does not analyse code**, so a suite gets *more* failing tests when +Basilisk is found to be wrong, never fewer. Never delete a failing test, remove a +failure-causing assertion, reduce assertiveness, or mark a test ignored. -The conformance score is produced by **RUNNING the real `python/typing` -conformance harness** — the suite's own `conformance/src/main.py` driving its -built-in `BasiliskTypeChecker` — against the compiled binary on **every run**, -never a Basilisk reimplementation. It is the exact tooling the reference checkers -(pyright, mypy, pyrefly, ty, zuban, pycroscope) are graded with. **A build in which -that official check did not run against a freshly cloned suite is a BUILD FAILURE.** +### Semantics-preserving mutation — the accuracy gate {#CHKARCH-TESTING-SEMANTIC-MUTATION} -**The mechanism — every CI run, in order, no step skippable or the build dies:** +> **Status: specified, not built.** No harness in this repo performs these mutations +> today, which is exactly why text-matched logic survived every green run. Building it +> is owned by [LINESCANPLAN-SEMANTIC-MUTATION](../plans/CHECKER-ELIMINATE-LINE-SCANNING-PLAN.md#LINESCANPLAN-SEMANTIC-MUTATION). +> Until it exists, **every rule is unverified against this gate** and no rule may be +> described as spec-implementing on the strength of a fixture alone. -1. **Freshly download** the tests **and** the harness/calculator from +The one gate that can distinguish a rule that analyses code from a rule that matches +text. It takes each rule test and re-runs it over inputs that are *semantically +identical and textually different*; the diagnostics must be **byte-for-byte the same**. + +Required mutations, each applied independently and in combination: + +| Mutation | Example | +|---|---| +| Import aliasing | `from typing import Final as F`, `import typing as t` | +| Import form | `typing.Final` ↔ `from typing import Final` ↔ re-export through a local module | +| Reformatting | line breaks, indentation width, trailing commas, parenthesisation | +| Quote and string style | `'…'` ↔ `"…"`, implicit concatenation, raw/f-prefixes on non-type strings | +| Identifier renaming | consistent rename of every class, type variable, alias, and parameter | +| Statement reordering | independent statements, class members, and imports permuted | +| Comment and whitespace churn | comments added, removed, and moved | + +Rules: + +- **A rule whose diagnostics move under any of these is broken.** The finding is + handled by [CHKARCH-TEXT-MATCHED-LOGIC](#CHKARCH-TEXT-MATCHED-LOGIC) — failing test, + delete, report — never by teaching the harness to tolerate the difference. +- **The harness may never mutate an expected result to match observed output.** The + expectation is the unmutated run's diagnostics; only the input varies. +- **Coverage is reported as a fraction of rules exercised**, and the uncovered + remainder is named in the report. A score over a hand-picked subset says nothing + about the rest, and reporting only the covered fraction is how a partial measure + reads as a complete one. +- **Never re-submit to `python/typing` until this gate passes clean** and an external + audit has run ([CHKARCH-CONFORMANCE](#CHKARCH-CONFORMANCE)). + +### PEP Conformance Measurement {#CHKARCH-CONFORMANCE} + +**The number is a regression detector, never an objective.** It samples one +fixed corpus that the checker was historically developed against, so it cannot +tell you whether a rule analyses code — only whether today's binary agrees with +yesterday's on 141 files it has already seen. Optimising it is the failure mode +that produced [CHKARCH-CONFORMANCE-INTEGRITY-AUDIT](../CONFORMANCE-INTEGRITY-AUDIT.md). +Accuracy on unseen Python is the objective; this measurement is downstream of it +and overfitted to it. + +Three consequences, all normative: + +- **Never publish, quote, or market a conformance figure**, in the specs, the + README, the website, a commit message, or a PR description. Nothing may imply + Basilisk appears in the official `python/typing` results — it was + [removed at its own author's request](https://github.com/python/typing/pull/2330). +- **Never re-submit to `python/typing`** until + [CHKARCH-TESTING-SEMANTIC-MUTATION](#CHKARCH-TESTING-SEMANTIC-MUTATION) passes + clean and an external audit has run. +- **A drop caused by deleting text-matched logic is progress.** Record it, say so + plainly, and move on. Never restore the deleted code, re-fit a rule to the + fixture, or fake a pass to hold a ratchet. **There is no pass-percentage floor + and no ratchet on this number** — a threshold on a corpus the code was fitted + to rewards fitting, which is precisely how the defect arose + ([§6.3](../CONFORMANCE-INTEGRITY-AUDIT.md)). + +When the measurement *is* taken, it is taken honestly or not at all. It comes +from **RUNNING the real `python/typing` harness** — the suite's own +`conformance/src/main.py` driving its built-in `BasiliskTypeChecker` — against the +compiled binary, never a Basilisk reimplementation. A vendored scorer, an injected +adapter, cached fixtures, or a committed result standing in for a live run is a +**BUILD FAILURE**. + +**The mechanism — in order, no step skippable:** + +1. **Freshly clone** the tests **and** the harness/calculator from `python/typing@main`'s **latest** commit — `git clone --depth 1 https://github.com/python/typing`. No cache, no committed fixtures, no vendored - calculator. (So the moment upstream merges a new rule/fixture, the very next run - grades against it — and if we regress, CI tanks.) + calculator. 2. **Freshly build a CLEAN release** `basilisk` binary from THIS checkout's source — `cargo build --release`, un-instrumented, byte-for-byte what ships. Never the PyPI wheel (a prior version), never an instrumented build. 3. **Run the suite's OWN `conformance/src/main.py --only-run basilisk`** against - that binary (pointed at it via `BASILISK_BIN`), and **fail HARD on ANY false - positive or ANY missed required error** — the gate demands 100 % pass / 0 FP - (`coverage-thresholds.json`). One stray diagnostic tanks the build. -4. **Regenerate `conformance/conformance_status.csv`** (and the website report) - from the harness's OWN `results/basilisk/*.toml` — the committed scoreboard is - always a product of the live run, never hand-authored. - -> ⛔️ **DISABLING, DELETING, OR UNREGISTERING ANY CONFORMANCE RULE IS FORBIDDEN.** -> The binary is scored in its **full default configuration with EVERY core -> PEP/conformance rule enabled** — no Basilisk config (any format; the legacy -> `basilisk.json` is no longer read), no per-rule override, no "spec-conformance mode", -> no skipped fixtures, no deleting rule source (`src/rules/*.rs`), no removing rules -> from `all_rules()`. The binary is scored over a **fresh `python/typing` clone** -> whose tree holds no Basilisk config of any format, so nothing of ours can silence a rule; -> deleting the rules themselves is the **same crime by another route** and equally -> forbidden — as is hand-editing `conformance/conformance_status.csv` or loosening -> the `coverage-thresholds.json` gate (`threshold` / `max_false_positives`). A -> strict default firing on valid code is a **real conformance gap to FIX in the -> checker**, never to hide. Gaming the number is a punishable offence. + that binary (pointed at it via `BASILISK_BIN`) and record what it reports. +4. **Regenerate `conformance/conformance_status.csv`** from the harness's OWN + `results/basilisk/*.toml` — the committed record is always a product of the live + run, never hand-authored. + +> ⛔️ **Never move the number by touching the scoreboard.** The binary runs in its +> **full default configuration with every `pep` rule enabled** — no Basilisk config +> (any format; the legacy `basilisk.json` is no longer read), no per-rule override, +> no "spec-conformance mode", no skipped fixtures. Equally forbidden: hand-editing +> `conformance/conformance_status.csv`, and loosening or tightening a threshold to +> match a run. +> +> **Deleting a rule is not on this list, and that is deliberate.** Deleting a rule +> that decides from source text rather than resolved symbols is the *required* +> action ([CHKARCH-TEXT-MATCHED-LOGIC](#CHKARCH-TEXT-MATCHED-LOGIC)) — write the +> failing test, delete the code, tell the user. The distinction is intent, and it is +> visible in what happens to the number: deleting analysis you cannot trust is +> expected to *lower* it and you report the drop; deleting a rule to dodge a failure +> is dishonest. If the rule genuinely analyses code and merely fires on valid input, +> that is a real gap to fix in the checker, not to silence. - **Runner — the real harness, nothing else**: [`conformance/run_conformance.py`](../../conformance/run_conformance.py) is the @@ -1432,8 +1545,11 @@ that official check did not run against a freshly cloned suite is a BUILD FAILUR mypy, pyrefly, ty, zuban and pycroscope. From those real results the runner only *reports*: it writes `conformance/conformance_status.csv` and **records the exact graded commit hash** in - [`website/src/_data/conformance_report.json`](../../website/src/_data/conformance_report.json), - so every published number is pinned *by hash* on the website. There is **NO + [`website/src/_data/conformance_report.json`](../../website/src/_data/conformance_report.json) + so any run is traceable to the upstream tree it ran against. That file is a local + record, **not a publication source**: the site's `_data/conformance.js` reads it only + to render the withdrawn historical run, and no current figure is published from it. + There is **NO vendored calculator and NO cached-fixtures fallback** — a build in which the real harness could not be cloned and run is a **BUILD FAILURE**, by design. (The only auxiliary number not in the toml, `caught` = required errors matched, is taken @@ -1450,98 +1566,123 @@ that official check did not run against a freshly cloned suite is a BUILD FAILUR Basilisk config of any format, so nothing of ours can silence a conformance rule. Opt-in Basilisk-specific rules remain off by the same ordinary default ([CHKARCH-CONFORMANCE-MODE](#CHKARCH-CONFORMANCE-MODE)). -- **Gate**: `make test` (via [`scripts/test-rust.sh`](../../scripts/test-rust.sh)) +- **Where it runs**: `make test` (via [`scripts/test-rust.sh`](../../scripts/test-rust.sh)) builds the `basilisk` binary, then runs - `python3 conformance/run_conformance.py --gate` on it — which runs the REAL - harness and delegates the 100 %-pass / 0-false-positive check to + `python3 conformance/run_conformance.py --gate` on it — the REAL harness, with the + pass/false-positive comparison delegated to [`conformance/assert_wheel_conformance.py`](../../conformance/assert_wheel_conformance.py) over the harness's OWN `results/basilisk/*.toml`. There is **no Rust conformance - test** and **no in-repo scorer**: the score is the real suite's own verdict on the - compiled binary. The pass-percentage floor and false-positive ceiling live in - `coverage-thresholds.json` (`conformance.threshold`, - `conformance.max_false_positives`); the former ratchets **up**, the latter - **down**. Per-file results are written to `conformance/conformance_status.csv`. -- **Current score** — measured against `python/typing@main` at the exact graded - commit recorded in `conformance_report.json`, currently - [`a490662`](https://github.com/python/typing/tree/a4906624f170c169cf667f962080c56d5a5ba6ff/conformance): - **141 / 141 = 100.0%**, **0 false positives**, **0 missed required errors**, with - **970** required errors caught. The binary runs in its default configuration — the - PEP conformance set — over a fresh `python/typing` clone whose tree holds no - Basilisk config of any format, so nothing can silence a rule; Basilisk's opt-in house-style rules never run during scoring, - so they can neither pad nor sink the number. The gate - ratchets the pass-percentage **up** and the false-positive ceiling **down** - (`coverage-thresholds.json` → `conformance.threshold` / - `conformance.max_false_positives`), driven only by genuinely fixing the checker, - **never** by disabling a rule. (History: a **baseline reset on 2026-06-26** - corrected a gamed *fake 100%* that had disabled six house-style rules before - scoring; conformance has been measured honestly in the default config ever since.) - Target: **100%**. - -#### No "spec-conformance mode" — the scorer runs the genuine default config {#CHKARCH-CONFORMANCE-MODE} - -There is **no** conformance mode, and there never will be. The scorer runs the binary + test** and **no in-repo scorer**: the result is the real suite's own verdict on the + compiled binary. Per-file results land in `conformance/conformance_status.csv`. +- **Known contradiction — `coverage-thresholds.json` still gates this number.** + Its `conformance` block carries `threshold` (a pass-percentage floor) and + `max_false_positives`, and `--gate` enforces both. That floor is the mechanism the + integrity audit identified as the incentive behind the fitted predicates + ([§6.3](../CONFORMANCE-INTEGRITY-AUDIT.md)): it makes the *expected, correct* drop + from deleting a text-matched rule fail the build, which is the pressure that + produces fitting in the first place. Removing that floor is the user's call and is + not an agent's to make. **Until they decide, a deletion that lowers the number is + still the right action** — make the deletion, report the drop and the failing gate, + and stop there. Do not restore the code, refit the rule, or edit the threshold to + get green. +- **No current figure is recorded here, by policy.** Reading one out of a spec is + how a withdrawn number keeps circulating. `conformance_status.csv` holds the last + live run for whoever needs it; the specs quote nothing. + +#### No "spec-conformance mode" — the harness runs the genuine default config {#CHKARCH-CONFORMANCE-MODE} + +There is **no** conformance mode, and there never will be. The harness runs the binary in exactly the configuration a user gets out of the box — the **default config, which -is the pure PEP conformance set** ([CHKARCH-CONFIGURATION-ONLY](#CHKARCH-CONFIGURATION-ONLY)) -— with no Basilisk config of any format (the legacy `basilisk.json` is no longer -read), no per-rule override, and no special scoring path. Basilisk's -opinionated *house-style* rules (require-annotations `BSK-0001`/`BSK-0002`/`BSK-0004`, +is the pure PEP set** ([CHKARCH-CONFIGURATION-ONLY](#CHKARCH-CONFIGURATION-ONLY)) — +with no Basilisk config of any format (the legacy `basilisk.json` is no longer read), +no per-rule override, and no special measurement path. Basilisk's opinionated +*house-style* rules (require-annotations `BSK-0001`/`BSK-0002`/`BSK-0004`, require-`@override` `BSK-0025`, redundant-annotation `BSK-0050`, the explicit-`Any` -nudge `BSK-0014`) are **opt-in and off by default**, so they never run during scoring -and can neither pad nor sink the number. The figure is the genuine out-of-the-box -conformance result — currently 100.0%. Any shortfall would be a real -checker bug to fix (a missing spec feature, or a false positive from an over-strict -*conformance* rule), never something to paper over by silencing a rule. - -⛔️ **Disabling, deleting, or unregistering a conformance (PEP) rule to move the number -is forbidden** — as is hand-editing `conformance_status.csv` or loosening the -`coverage-thresholds.json` gate (`threshold` / `max_false_positives`) to match a faked -run. This has been attempted twice, back when the house rules still ran by default and -counted toward the score. First, a revision wrote a `basilisk.json` that turned six -rules off before scoring and reported a **fake 100%**; that was removed, and the -scorer now runs the binary over a **fresh `python/typing` clone** whose tree contains -no Basilisk config of any format (the legacy `basilisk.json` is no longer read at -all), so no config can silence a rule. Second — when config-disabling -was blocked — a revision tried to -*delete the offending rule source files outright* and unregister them from -`all_rules()`, then re-report a **fake 100%**: the same lie by another route. **Deleting -a rule to dodge the config guard is the identical offence.** - -The path to 100% is to make the checker **correct**, never to silence a rule at score -time: implement the spec features it still misses, and teach its conformance rules to -stop firing on spec-valid code (recognising inferred return types, honouring `# E`-free -lines) so the false positives fall on their own merits. Anyone may relax rules *in their -own project* via config; the **conformance scorer never does**. +nudge `BSK-0014`) are **opt-in and off by default**, so they never run during a +measurement and can neither pad nor sink it. + +⛔️ **Configuring a rule off before measuring is forbidden**, as is hand-editing +`conformance_status.csv` or moving a threshold to match a run. Both have been +attempted: a revision once wrote a `basilisk.json` that turned six rules off before +measuring and reported a fake result, and when that route was closed, another deleted +the rule sources and unregistered them from `all_rules()` to reach the same figure. +The measurement now runs over a **fresh `python/typing` clone** whose tree holds no +Basilisk config of any format, so no configuration can silence a rule. + +That history is why deletion needs an explicit boundary, and it is a boundary of +**intent and disclosure**, not of mechanism: + +- **Deleting a rule to reach a number is dishonest** — the deletion is hidden, the + figure is reported as if the analysis still existed, and nothing fails. +- **Deleting a rule that decides from source text is required** — the deletion is the + point, a failing test is left behind to mark the hole, the drop is reported, and + nobody claims the analysis exists ([CHKARCH-TEXT-MATCHED-LOGIC](#CHKARCH-TEXT-MATCHED-LOGIC)). + +**A failing test that pins real incorrect behaviour is worth more than a passing +fixture carried by logic that does not analyse code.** The first is an accurate map of +what Basilisk cannot yet do; the second is a false claim that it can. Given the choice, +take the failing test every time. + +Anyone may relax rules *in their own project* via config; a measurement run never does. ### Mutation Testing Ratchet {#CHKARCH-TESTING-MUTATION-RATCHET} -Mutation testing proves the test suite actually asserts behaviour. Scope only ever **grows** toward all Rust code: - -- **Scope is test-driven.** `#[mutation_safe(rule = "", fns = "fn_a|fn_b")]` - attributes on e2e tests drive the `cargo mutants` examine regex - (`scripts/mutation_examine_re.py`). `` is the rule's path stem under - `crates/basilisk-checker/src/rules/` (file like `aliases_implicit` or directory - like `assignment_compatibility`); omitting `fns` scopes the whole file. Adding - these tests is the only way to widen scope. -- **Baseline is ratcheted.** `mutation_testing/mutation_scores.json` is the committed - baseline; `mutation_testing/mutants_report.py::regression_messages` fails the build - when `kill_rate` drops below the baseline or the absolute floor, when `detected` - (`caught` + `timeout`) drops **while the viable pool did not grow**, or when - `timeout` rises. (`unviable` mutants don't compile and are excluded.) Absolute - `missed` is deliberately *not* a signal: widening scope mutates more code, so a - larger raw `missed` against a smaller-pool baseline is expected — `kill_rate` is - the size-independent guard. Both `make mutation-test` and the CI shard merge - enforce the same function. -- **A timeout may never rise.** A `timeout` is credited as a kill (the PIT/Stryker - convention: a terminating suite made non-terminating *has* been detected). That - credit is only honest while timeouts come from hung code rather than slowness — - and the mutants that time out are structurally the likely *survivors*, since a - killed mutant exits at the first failing test binary while an uncaught one runs - the whole suite. So a rise in `timeout` is itself a build failure: it means - mutants were credited as killed without being evaluated. Fix the budget or the - suite's speed ([`.cargo/mutants.toml`](../../.cargo/mutants.toml)); never absorb it. -- **Direction.** End state is the full workspace under mutation - (`make mutation-test ALL=1`); until then each checker-logic PR leaves the viable - pool the same size or larger. +Mutation testing exists to prove the suite **asserts** behaviour rather than merely +executing it. The current regime does not prove that, and the reasons are structural +rather than accidental. They are recorded here because a metric whose limits are +undocumented reads as a guarantee. + +**How it works today.** `#[mutation_safe(rule = "", fns = "fn_a|fn_b")]` +attributes on e2e tests drive the `cargo mutants` examine regex +(`scripts/mutation_examine_re.py`). `` is the rule's path stem under +`crates/basilisk-checker/src/rules/` (file like `aliases_implicit` or directory like +`assignment_compatibility`); omitting `fns` scopes the whole file. +`mutation_testing/mutation_scores.json` holds the committed baseline and +`mutants_report.py::regression_messages` fails the build when `kill_rate` drops below +the baseline or the absolute floor, when `detected` drops while the viable pool did +not grow, or when `timeout` rises. Both `make mutation-test` and the CI shard merge +call the same function. + +**Three ways the number overstates what is proven.** Each is a correction owed, not a +convention to preserve: + +1. **Scope is opt-in, so the pool is self-selected.** Only annotated files are + examined. The committed baseline is 161 mutants at a 100% kill rate; the checker + crate alone is ~82k LOC across 307 files, 246 of them rules. A perfect score over a + chosen sliver is not a statement about the crate, and reporting it without the + denominator makes a partial measure read as a complete one. **Correction owed:** + report kill rate *and* the fraction of mutable surface examined, and name the + unexamined remainder. Un-annotated code is **unverified**, and must be described + that way rather than omitted. +2. **A timeout is credited as a kill.** `detected = caught + timeout`, on the + PIT/Stryker convention that a terminating suite made non-terminating has been + detected. But a timed-out mutant was never evaluated, and timeouts are structurally + biased toward *survivors*: a killed mutant exits at the first failing test binary + while an uncaught one runs the whole suite. Crediting them inflates the rate with + the mutants most likely to have lived. **Correction owed:** report `timeout` as its + own unevaluated category, outside the numerator. Until then a rise in `timeout` is a + build failure — it means mutants were credited without being run. Fix the budget or + the suite's speed ([`.cargo/mutants.toml`](../../.cargo/mutants.toml)); never absorb it. +3. **Survivors are aggregated away.** Absolute `missed` is deliberately not a signal, + because widening scope mutates more code and a larger raw `missed` is expected. That + is sound arithmetic and poor engineering: every surviving mutant is a specific + behaviour no assertion pins, and a count cannot be acted on. **Correction owed:** + enumerate survivors by file, function, and mutation in the report, so each is a + fixable item rather than a number to keep flat. + +**Rules that stand regardless:** + +- **Never widen the mutant pool by weakening what is mutated**, and never narrow scope + to protect a rate. Scope only grows; each checker-logic PR leaves the viable pool the + same size or larger. End state is the full workspace (`make mutation-test ALL=1`). +- **Never tune a test to kill a mutant without asserting the behaviour** the mutant + changed. A kill obtained by asserting on incidental output is the same defect as a + rule that matches text — it moves a number without proving meaning. +- **A high kill rate over rules that decide from source text proves nothing.** It shows + the tests pin the text-matching, which is the behaviour being deleted. Mutation score + is meaningful only over rules that pass + [CHKARCH-TESTING-SEMANTIC-MUTATION](#CHKARCH-TESTING-SEMANTIC-MUTATION); it never + substitutes for it. ### Benchmark — Indicative, Not a Gate {#CHKARCH-TESTING-BENCH} diff --git a/docs/specs/CHECKER-RULE-TAGGING-SPEC.md b/docs/specs/CHECKER-RULE-TAGGING-SPEC.md index 38c557685..7a903b76a 100644 --- a/docs/specs/CHECKER-RULE-TAGGING-SPEC.md +++ b/docs/specs/CHECKER-RULE-TAGGING-SPEC.md @@ -120,6 +120,18 @@ the prefix would leave provenance unchanged; [CHKTAG-TESTS] asserts convention and self-declared set agree both ways. PEP rules are named after their conformance test (e.g. `aliases_newtype`) with no `BSK` prefix. +> **This naming is a known hazard.** Naming a rule after the fixture it is scored +> on invites the rule to be *about* that fixture: it makes "does the file pass?" +> read as the rule's definition, and a rule file named `generics_base_class_2.rs` +> has no name left to describe the typing-spec concept it owes. Fixture-shaped +> naming is one of the detection signatures in +> [CHKARCH-TEXT-MATCHED-LOGIC](CHECKER-ARCHITECTURE-SPEC.md#CHKARCH-TEXT-MATCHED-LOGIC), +> and the numeric `_2` / `_3` suffixes are the tell: they enumerate fixtures, not +> concepts. The convention stands for now because the codes are user-visible and +> renaming them is a breaking change the user has not scoped — **but a rule's +> name is never evidence that it implements anything.** Judge the rule by whether +> it decides on the resolved model, never by whether its fixture passes. + ## Invariants {#CHKTAG-INVARIANTS} Enforced by [CHKTAG-TESTS] over the full, live rule set: diff --git a/docs/specs/CHECKER-STUB-RESOLUTION-SPEC.md b/docs/specs/CHECKER-STUB-RESOLUTION-SPEC.md index 7221362e3..4d96020cf 100644 --- a/docs/specs/CHECKER-STUB-RESOLUTION-SPEC.md +++ b/docs/specs/CHECKER-STUB-RESOLUTION-SPEC.md @@ -236,7 +236,9 @@ lookup: the source is present and verifies, or it is missing/corrupt and the checker **fails hard** — it refuses to analyse, names the SHA it needed, and never substitutes another source or degrades to an untyped stdlib. That failure is service status (CLI stderr, LSP `showMessage` + Service Info, MCP), never a -Python diagnostic, so it can never create a conformance false positive. +Python diagnostic — because "your typeshed pin is broken" is a fact about the +environment, not about the user's code. The channel is chosen by what the message +*is*, never by what it would cost a measurement. #### A pin is a verification {#STUBRES-TYPESHED-PIN} @@ -528,12 +530,15 @@ prints a rustc-style banner (`[]: ` then `= see: fields on a separate `debug` telemetry channel — the human banner is not `key="VALUE"` telemetry. The LSP surfaces them through `window/showMessage` plus persistent Service Info, never `publishDiagnostics`. MCP returns them as -structured `{code, message, docs_url}` fields. **Conformance invariant:** no -advisory ever enters the stdout JSON / `publishDiagnostics` stream a conformance -run scores, so it can NEVER create a false positive. The default bundled run -emits exactly one advisory — the `typeshed_source_unpinned` reproducibility -notice — on stderr, which the `python/typing` harness never reads; the 100 % / -0-FP score is unaffected ([§CHKARCH-CONFORMANCE](CHECKER-ARCHITECTURE-SPEC.md#CHKARCH-CONFORMANCE)). +structured `{code, message, docs_url}` fields. **Channel invariant:** an advisory +describes the *environment*, not the user's code, so it never enters the stdout +JSON / `publishDiagnostics` stream. That is a statement about what a diagnostic +means, not a device for keeping a measurement clean — routing a message about +Python code away from the diagnostic stream to protect a number would be exactly +the failure this repo is removing +([§CHKARCH-CONFORMANCE](CHECKER-ARCHITECTURE-SPEC.md#CHKARCH-CONFORMANCE)). The +default bundled run emits exactly one advisory — the `typeshed_source_unpinned` +reproducibility notice — on stderr. **Severity is configured exactly like any Basilisk rule.** Each advisory carries the `basilisk` provenance tag, so it resolves severity through the same diff --git a/docs/specs/CHECKER-TYPE-INFERENCE-SPEC.md b/docs/specs/CHECKER-TYPE-INFERENCE-SPEC.md index f6730ef95..bf819d72b 100644 --- a/docs/specs/CHECKER-TYPE-INFERENCE-SPEC.md +++ b/docs/specs/CHECKER-TYPE-INFERENCE-SPEC.md @@ -113,9 +113,13 @@ and no new code may be written against any of them: `subtyping::SubtypingContext` as the **single** subtyping judgment. While a legacy path still exists in the tree it is an implementation debt, not -a design. Deleting it must never delete a rule, drop a diagnostic, or cost a -required conformance error ([CHKARCH-CONFORMANCE](CHECKER-ARCHITECTURE-SPEC.md#CHKARCH-CONFORMANCE)); -if the engine cannot yet carry a rule, the engine gets fixed first. +a design. Removing it must not silently drop a diagnostic from a rule that +genuinely analyses code — if the engine cannot yet carry such a rule, the engine +gets fixed first. A rule whose verdict came from the text all along is a +different case: it is deleted, with a failing test left behind and the loss +reported ([CHKARCH-TEXT-MATCHED-LOGIC](CHECKER-ARCHITECTURE-SPEC.md#CHKARCH-TEXT-MATCHED-LOGIC)). +Neither case is decided by what it costs a conformance run +([CHKARCH-CONFORMANCE](CHECKER-ARCHITECTURE-SPEC.md#CHKARCH-CONFORMANCE)). ### [TYPEINF-ANNOTATION-RESOLUTION] Annotation name resolution {#TYPEINF-ANNOTATION-RESOLUTION} @@ -977,7 +981,9 @@ Subtyping is decided by `InferredType::is_assignable_to(&self, other)` in `crate Module-context equivalences that `is_assignable_to` cannot see run as ordered rescues in `rules/assignment_compatibility` after it returns false: expected-type literal-collection checking, the enum literal expansion (`enum_expand.rs`, needing the module's enum-member environment), then callable-signature rescue — all over the skip/alias/schema environment built once per module by `skip_names::SkipNames::collect`. -`Named` types (user classes and unparameterised imports) compare by base name before `[`: `Foo[int]` and `Foo[float]` are treated as compatible. This is deliberate — without whole-program generic variance analysis, stricter matching would emit false positives, and the conformance gate holds `max_false_positives` at zero. +`Named` types (user classes and unparameterised imports) compare **by the source text before the `[`**: `Foo[int]` and `Foo[float]` are treated as compatible. Stated plainly, that is not a subtyping judgment — it is a string prefix comparison over a rendered type, and generic argument compatibility is **unimplemented**. It is a `_ => true` in different clothing: every mismatch inside the brackets is accepted, so `list[int]` assigned from `list[str]` passes. + +Two things about it are on the record. First, it is text matching by [CHKARCH-TEXT-MATCHED-LOGIC](CHECKER-ARCHITECTURE-SPEC.md#CHKARCH-TEXT-MATCHED-LOGIC): the verdict follows the spelling of the type, and an alias or a re-export changes it. Second, the justification previously given here — that stricter matching would raise false positives against a zero-false-positive conformance gate — is not a reason to leave a check unimplemented; it is a reason the gate was the wrong instrument. Whole-program variance analysis is the real fix, and until it exists this comparison must be described as missing rather than as deliberate conservatism. Nominal MRO walking and structural Protocol/TypedDict compatibility are decided today by the per-conformance-area rule modules (`rules/protocols_*`, `rules/typeddicts_*`, and the class-bases-walking `is_subtype_of` helper in `rules/generics_basic_3/helpers.rs`). The shared home now exists — `crates/basilisk-checker/src/subtyping.rs` (`SubtypingContext`: cycle-guarded nominal walk, structural Protocol satisfaction, `TypedDict` schemas, declared variance, `Callable` kinds) — and the rule modules migrate onto it behind the parity pins in `tests/subtyping_context_tests.rs` and the in-module `helper_parity_tests` at the Integration stage ([NARROWPLAN-SUBTYPING](../plans/CHECKER-TYPE-NARROWING-INFERENCE-PLAN.md#NARROWPLAN-SUBTYPING)). diff --git a/docs/specs/DOCS-README-SPEC.md b/docs/specs/DOCS-README-SPEC.md index cf5addf0d..9a6399659 100644 --- a/docs/specs/DOCS-README-SPEC.md +++ b/docs/specs/DOCS-README-SPEC.md @@ -9,9 +9,9 @@ claimed a retired typeshed behaviour months after the others were corrected. There is now exactly **one** README per language, and the published files are **identical except for a single line** that says which artifact you are looking -at. Everything else — the conformance claim, the benchmark table, the install -options, the feature list, the typeshed section, the acknowledgments — is one -body of text, generated to every storefront by `scripts/gen_readmes.py`. +at. Everything else — the withdrawal notice, the install options, the feature +list, the typeshed section, the acknowledgments — is one body of text, generated +to every storefront by `scripts/gen_readmes.py`. ## Source {#README-SOURCE} @@ -64,12 +64,17 @@ The generator applies three transforms, in order: ## Stamped values {#README-STAMPED} -The conformance and benchmark figures are not typed by hand anywhere. They are -`value` markers stamped into the **source** by -`scripts/gen_conformance_reference.py` from `conformance_report.json` and the -committed benchmark CSVs ([CHKARCH-CONFORMANCE](CHECKER-ARCHITECTURE-SPEC.md#CHKARCH-CONFORMANCE)). -Generation runs after stamping, so every storefront quotes the same -self-measured number. +**No conformance or benchmark figure appears in any README, and none is stamped.** +Both claims are withdrawn ([CHKARCH-CONFORMANCE](CHECKER-ARCHITECTURE-SPEC.md#CHKARCH-CONFORMANCE)), +so every `value` marker has been removed from the +sources and `scripts/gen_conformance_reference.py` now has nothing to stamp. The +machinery is retained, not retired, so that a *future* measured value can only +ever reach a README by generation rather than by hand. + +Re-introducing a marker for a conformance figure is forbidden regardless of what +the harness reports. A benchmark marker may return only when the number is +measured on isolated hardware and carries its indicative-only caveat +([CHKARCH-TESTING-BENCH](CHECKER-ARCHITECTURE-SPEC.md#CHKARCH-TESTING-BENCH)). ## Drift guard {#README-DRIFT} diff --git a/docs/specs/RELEASE-MANUAL-VERIFICATION-SPEC.md b/docs/specs/RELEASE-MANUAL-VERIFICATION-SPEC.md index 628cb7f52..539d0e539 100644 --- a/docs/specs/RELEASE-MANUAL-VERIFICATION-SPEC.md +++ b/docs/specs/RELEASE-MANUAL-VERIFICATION-SPEC.md @@ -240,9 +240,13 @@ publishing on a compatible host: 1. [ ] `/ci-prep` green — one complete clean run, zero failures, start to finish ([RELEASE-CI-PREP](#RELEASE-CI-PREP)). Nothing below starts until it is. -2. [ ] `make conformance` — 100% / 0 false positives against a fresh - `python/typing@main` clone. -3. [ ] `make bench` — no fixture slower than the committed baseline. +2. [ ] `make conformance` — a live run against a fresh `python/typing@main` + clone. Record what it reports and compare it to the previous release's + record; an unexplained change is a regression to investigate. A drop + explained by a deliberate deletion is expected and is noted in the release + record. **The figure is never published or quoted** + ([CHKARCH-CONFORMANCE](CHECKER-ARCHITECTURE-SPEC.md#CHKARCH-CONFORMANCE)). +3. [ ] `make bench` — indicative only, gates nothing; record the numbers. 4. [ ] `python3 scripts/verify_release_attribution.py --policy-only` passes and licence manifests are current (`npm run licenses:check` in `vscode-extension/`). diff --git a/docs/specs/REPO-STANDARDS-SPEC.md b/docs/specs/REPO-STANDARDS-SPEC.md index d54e42012..a33a1ad39 100644 --- a/docs/specs/REPO-STANDARDS-SPEC.md +++ b/docs/specs/REPO-STANDARDS-SPEC.md @@ -18,7 +18,9 @@ in the file it governs, so the citation resolves in one hop. `.deslop.toml` at the repository root is the **single source of truth** for this repo's duplication budget. It is committed and PR-reviewed, and `[threshold] max_duplication_percent` is ratcheted **down** only — the same -one-way discipline the coverage and conformance gates use. +one-way discipline the coverage gate uses. (The conformance block in +`coverage-thresholds.json` is deliberately *not* on that footing; see +[COVERAGE-THRESHOLDS-JSON-CONFORMANCE](#COVERAGE-THRESHOLDS-JSON-CONFORMANCE).) `[defaults] exclude` drops paths during discovery, so excluded files are never analysed and never contribute to the measured percentage @@ -95,11 +97,21 @@ value is forbidden; a project that cannot meet its number gets more tests. ### Conformance block {#COVERAGE-THRESHOLDS-JSON-CONFORMANCE} The same file carries the `conformance` block (`threshold` and -`max_false_positives`), which is enforced by the real `python/typing` harness -rather than by the coverage scripts. Its policy — pass percentage up only, -false-positive ceiling down only, and no rule may be disabled to move either — -is normative in -[CHKARCH-CONFORMANCE](CHECKER-ARCHITECTURE-SPEC.md#CHKARCH-CONFORMANCE). +`max_false_positives`), enforced by the real `python/typing` harness rather than +by the coverage scripts. + +**This block is a known contradiction, recorded rather than papered over.** A +pass-percentage floor over a fixed corpus the checker was developed against is +the incentive the integrity audit identified behind the fitted predicates +([§6.3](../CONFORMANCE-INTEGRITY-AUDIT.md)): it makes the *expected, correct* drop +from deleting text-matched logic fail the build. Policy is set by +[CHKARCH-CONFORMANCE](CHECKER-ARCHITECTURE-SPEC.md#CHKARCH-CONFORMANCE) — the +number is a regression detector, never a target. Whether the floor comes out is +the user's decision and not an agent's. Until they decide: **make the deletion, +report the drop and the failing gate, and stop there.** Do not restore the code, +refit the rule, or edit the threshold to get green. + +Unlike the coverage thresholds above, this block is **not** something to ratchet. ## Committed editor directories {#GITIGNORE-RULES} diff --git a/mutation_testing/mutants_report.py b/mutation_testing/mutants_report.py index ace052427..14c13c600 100644 --- a/mutation_testing/mutants_report.py +++ b/mutation_testing/mutants_report.py @@ -241,11 +241,19 @@ def baseline_for_scope(score_book: dict[str, Any], scope: str) -> MutationScore # Hard floor for every crate's kill rate, independent of the moving baseline. -# The mutation run mutates the WHOLE crate (no code excluded), so `missed` is a -# large absolute number that DROPS as tests improve — the opposite of the old -# hidden-pool regime where any missed mutant was a scandal. The binding ratchet -# is therefore `kill_rate`, which must never drop AND never fall below this -# floor. Raise the floor as coverage climbs; never lower it. +# +# READ THE DENOMINATOR BEFORE READING THE RATE. The run does NOT mutate the whole +# crate: `scripts/mutation_examine_re.py` builds a cargo-mutants `--re` pattern from +# the `#[mutation_safe]` annotations, so only annotated files and functions are +# examined. The pool is therefore self-selected, and a high `kill_rate` describes the +# annotated sliver alone — it says nothing about the unannotated remainder, which is +# UNVERIFIED. Widening scope is the only way to make the rate mean more; never narrow +# it to protect the number. See [CHKARCH-TESTING-MUTATION-RATCHET] for the three +# corrections this regime still owes (scope denominator, timeout-as-kill, survivor +# enumeration). +# +# `kill_rate` must never drop AND never fall below this floor. Raise the floor as +# scope climbs; never lower it. MIN_KILL_RATE = 20.0 From ec022201e4a2cfae151a3e5cf0a2f40e77ec5514 Mon Sep 17 00:00:00 2001 From: Christian Findlay <16697547+MelbourneDeveloper@users.noreply.github.com> Date: Sat, 8 Aug 2026 07:29:29 +1000 Subject: [PATCH 3/9] Fixes --- CLAUDE.md | 9 ++++--- README-pypi.md | 28 +++++++++++++++------ README.md | 29 ++++++++++++++++------ README.zh.md | 21 +++++++++++----- docs/readme/README.src.md | 28 +++++++++++++++------ docs/readme/README.zh.src.md | 21 +++++++++++----- vscode-extension/README.md | 28 +++++++++++++++------ vscode-extension/README.zh.md | 21 +++++++++++----- website/src/docs/conformance.md | 39 +++++++++++++++++++++--------- website/src/index.njk | 14 ++++++----- website/src/zh/docs/conformance.md | 39 +++++++++++++++++++++--------- website/src/zh/index.njk | 12 +++++---- 12 files changed, 204 insertions(+), 85 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index d532b4cd6..b0a56f019 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -105,10 +105,13 @@ Off-limits unless explicitly asked. When git IS used: ## Testing -- Tests exercise **meaning, not spelling**: every rule test gets an aliased-import and a reformatted variant, with identical diagnostics. -- Target 100% coverage on every measure. Each PR MUST increase overall coverage. +Tests must **enforce behaviour**, not work around the gaps in it. Judge a test by what it would catch, never by whether it's green. + +- Tests exercise **meaning, not spelling**: every rule test gets an aliased-import and a reformatted variant, with identical diagnostics. The harness that would enforce this across the suite ([CHKARCH-TESTING-SEMANTIC-MUTATION]) **does not exist yet** — until it does, every rule is unverified and must be described that way. +- Test against Python the conformance suite has never contained. A test copied from `conformance/tests/` cannot detect a rule fitted to `conformance/tests/`. - NEVER delete a failing test, remove a failure-causing assertion, reduce assertiveness, or ignore tests. Broken functionality gets MORE failing tests, never fewer. -- Mutation score only increases; widen scope with `#[mutation_safe]` tests. The gate ([CHKARCH-TESTING-MUTATION-RATCHET], `mutation_testing/mutation_scores.json`) fails CI if the mutant pool shrinks, caught drops, missed/timeout rise, or kill rate drops. +- Target 100% coverage on every measure. Each PR MUST increase overall coverage. Line coverage proves execution, never assertion — a rule at 100% coverage and zero real assertions is the normal failure, not an edge case. +- Mutation score only increases; widen scope with `#[mutation_safe]` tests. The gate ([CHKARCH-TESTING-MUTATION-RATCHET], `mutation_testing/mutation_scores.json`) fails CI if the mutant pool shrinks, caught drops, missed/timeout rise, or kill rate drops. **Read the denominator before the rate:** scope is opt-in, so the committed 100% covers 161 mutants out of an ~82k-LOC crate; timeouts are credited as kills; survivors are aggregated into a count. Never narrow scope to protect a rate, and never kill a mutant by asserting on incidental output instead of the behaviour it changed. - `make test` is FAIL-FAST — never `--no-fail-fast`. It enforces coverage from `coverage-thresholds.json` at the repo root, not env vars or CI YAML. Ratchet only. - VSIX tests must not call `whenCommandReady` or `getCommands(true)` to check existence — assert through the UI, or worst case internal VSIX state. diff --git a/README-pypi.md b/README-pypi.md index 75f46888b..6e233abc5 100644 --- a/README-pypi.md +++ b/README-pypi.md @@ -31,18 +31,30 @@ **The current type checker contains inaccuracies and you should not use it as part of your dev pipeline. We are working on removing any misleading analyzers ASAP. Please read below** -## We withdrew the typing conformance results +## We are auditing the checker and deleting what doesn't hold up We withdrew our 100% conformance claim and our benchmark figures, and asked to be [removed from the official `python/typing` results](https://github.com/python/typing/blob/main/conformance/results/results.html). The cause was checker logic fitted to the contents of conformance test files -instead of implementing the typing specification generally, and a score produced -that way is not evidence. The current percentage is **temporarily unknown**. - -We are deciding whether to rebuild the checker from the specification or drive -the extension with an established open-source checker. **Either way, we are -building Basilisk into an accurate Python development experience** — and a new -figure gets published only once it survives off-suite and mutation testing. +instead of implementing the typing specification generally: rules that matched +the *spelling* of code rather than its meaning. Rename an import or reformat a +file and the answer changed. A score produced that way is not evidence. + +**So we are auditing every rule and deleting the ones that don't do real type +checking.** Not rewriting them, not patching them, not marking them TODO — +deleting them, with a failing test left behind so the gap is visible instead of +hidden. A rule stays only if it decides from the resolved syntax tree and gives +the same answer when the code is spelled differently. + +That means Basilisk gets **smaller** before it gets better. Expect fewer rules, +fewer diagnostics, and a lower conformance number. We will report each drop +rather than avoid it. What is left will be code that is honest about what it +does — nothing else. + +We have not yet decided whether to rebuild the deleted analysis from the +specification or to drive the extension with an established open-source checker. +Either way, no new figure gets published until it survives off-suite and +mutation testing. [Read the full correction →](https://www.basilisk-python.dev/docs/conformance/)  •  [Integrity audit →](https://github.com/Nimblesite/Basilisk/blob/main/docs/CONFORMANCE-INTEGRITY-AUDIT.md) diff --git a/README.md b/README.md index c1d481420..39ef2d004 100644 --- a/README.md +++ b/README.md @@ -14,6 +14,8 @@ One extension for the whole workflow — diagnostics, autocomplete, refactoring, formatting, debugging, and profiling — driven by a single bundled binary.

+> **You are reading the Basilisk source repository** — the checker, language server, editor extensions, and website all live here. +

Website  •  Install  •  @@ -29,17 +31,30 @@ **The current type checker contains inaccuracies and you should not use it as part of your dev pipeline. We are working on removing any misleading analyzers ASAP. Please read below** -## We withdrew the typing conformance results +## We are auditing the checker and deleting what doesn't hold up We withdrew our 100% conformance claim and our benchmark figures, and asked to be [removed from the official `python/typing` results](https://github.com/python/typing/blob/main/conformance/results/results.html). The cause was checker logic fitted to the contents of conformance test files -instead of implementing the typing specification generally, and a score produced -that way is not evidence. The current percentage is **temporarily unknown**. - -We are -building Basilisk into an accurate Python development experience** — and a new -figure gets published only once it survives off-suite and mutation testing. +instead of implementing the typing specification generally: rules that matched +the *spelling* of code rather than its meaning. Rename an import or reformat a +file and the answer changed. A score produced that way is not evidence. + +**So we are auditing every rule and deleting the ones that don't do real type +checking.** Not rewriting them, not patching them, not marking them TODO — +deleting them, with a failing test left behind so the gap is visible instead of +hidden. A rule stays only if it decides from the resolved syntax tree and gives +the same answer when the code is spelled differently. + +That means Basilisk gets **smaller** before it gets better. Expect fewer rules, +fewer diagnostics, and a lower conformance number. We will report each drop +rather than avoid it. What is left will be code that is honest about what it +does — nothing else. + +We have not yet decided whether to rebuild the deleted analysis from the +specification or to drive the extension with an established open-source checker. +Either way, no new figure gets published until it survives off-suite and +mutation testing. [Read the full correction →](https://www.basilisk-python.dev/docs/conformance/)  •  [Integrity audit →](docs/CONFORMANCE-INTEGRITY-AUDIT.md) diff --git a/README.zh.md b/README.zh.md index 005ba7b78..158aa11fd 100644 --- a/README.zh.md +++ b/README.zh.md @@ -31,16 +31,25 @@ **当前的类型检查器存在不准确之处，请勿将其用于你的开发流水线。我们正在尽快移除任何具有误导性的分析器。详情请见下文** -## 我们撤回了类型一致性结果 +## 我们正在审计检查器，并删除站不住脚的代码 我们撤回了 100% 的一致性宣称与基准测试数字，并主动请求 [从官方 `python/typing` 结果中移除](https://github.com/python/typing/blob/main/conformance/results/results.html)。 -原因是检查器中存在针对一致性测试文件内容而写的逻辑，而不是对类型规范的通用实现； -这样得出的分数并不能作为证据。当前的百分比**暂时未知**。 +原因是检查器中存在针对一致性测试文件内容而写的逻辑，而不是对类型规范的通用实现： +那些规则匹配的是代码的**写法**，而不是代码的含义。改一个导入别名或重新格式化文件， +结论就会变。这样得出的分数并不能作为证据。 -我们正在决定是按规范重建检查器，还是让扩展由一个成熟的开源检查器驱动。**无论走哪 -条路，我们都在把 Basilisk 打造成准确的 Python 开发体验** —— 而新的数字只有在经受住 -套件之外的用例与变异测试后才会发布。 +**因此，我们正在逐条审计规则，并删除那些没有做真正类型检查的规则。** 不是重写，不是 +打补丁，也不是标一个 TODO —— 是删除，并留下一个失败的测试，让这个缺口可见而不是被 +掩盖。一条规则只有在依据已解析的语法树做判断、并且在代码换一种写法时给出相同结论的 +情况下，才会保留。 + +这意味着 Basilisk 会**先变小，再变好**。规则会更少，诊断会更少，一致性数字也会更低。 +每一次下降我们都会如实报告，而不是设法回避。留下来的，将是对自己所做之事诚实的代码 +—— 仅此而已。 + +至于被删掉的分析是按规范重建，还是让扩展由一个成熟的开源检查器驱动，我们尚未决定。 +无论走哪条路，新的数字只有在经受住套件之外的用例与变异测试后才会发布。 [阅读完整更正 →](https://www.basilisk-python.dev/zh/docs/conformance/)  •  [完整性审计 →](docs/CONFORMANCE-INTEGRITY-AUDIT.md) diff --git a/docs/readme/README.src.md b/docs/readme/README.src.md index 4c25dc562..7a114fa02 100644 --- a/docs/readme/README.src.md +++ b/docs/readme/README.src.md @@ -42,18 +42,30 @@ **The current type checker contains inaccuracies and you should not use it as part of your dev pipeline. We are working on removing any misleading analyzers ASAP. Please read below** -## We withdrew the typing conformance results +## We are auditing the checker and deleting what doesn't hold up We withdrew our 100% conformance claim and our benchmark figures, and asked to be [removed from the official `python/typing` results](https://github.com/python/typing/blob/main/conformance/results/results.html). The cause was checker logic fitted to the contents of conformance test files -instead of implementing the typing specification generally, and a score produced -that way is not evidence. The current percentage is **temporarily unknown**. - -We are deciding whether to rebuild the checker from the specification or drive -the extension with an established open-source checker. **Either way, we are -building Basilisk into an accurate Python development experience** — and a new -figure gets published only once it survives off-suite and mutation testing. +instead of implementing the typing specification generally: rules that matched +the *spelling* of code rather than its meaning. Rename an import or reformat a +file and the answer changed. A score produced that way is not evidence. + +**So we are auditing every rule and deleting the ones that don't do real type +checking.** Not rewriting them, not patching them, not marking them TODO — +deleting them, with a failing test left behind so the gap is visible instead of +hidden. A rule stays only if it decides from the resolved syntax tree and gives +the same answer when the code is spelled differently. + +That means Basilisk gets **smaller** before it gets better. Expect fewer rules, +fewer diagnostics, and a lower conformance number. We will report each drop +rather than avoid it. What is left will be code that is honest about what it +does — nothing else. + +We have not yet decided whether to rebuild the deleted analysis from the +specification or to drive the extension with an established open-source checker. +Either way, no new figure gets published until it survives off-suite and +mutation testing. [Read the full correction →](https://www.basilisk-python.dev/docs/conformance/)  •  [Integrity audit →](docs/CONFORMANCE-INTEGRITY-AUDIT.md) diff --git a/docs/readme/README.zh.src.md b/docs/readme/README.zh.src.md index bbb7f9819..27b009f44 100644 --- a/docs/readme/README.zh.src.md +++ b/docs/readme/README.zh.src.md @@ -38,16 +38,25 @@ **当前的类型检查器存在不准确之处，请勿将其用于你的开发流水线。我们正在尽快移除任何具有误导性的分析器。详情请见下文** -## 我们撤回了类型一致性结果 +## 我们正在审计检查器，并删除站不住脚的代码 我们撤回了 100% 的一致性宣称与基准测试数字，并主动请求 [从官方 `python/typing` 结果中移除](https://github.com/python/typing/blob/main/conformance/results/results.html)。 -原因是检查器中存在针对一致性测试文件内容而写的逻辑，而不是对类型规范的通用实现； -这样得出的分数并不能作为证据。当前的百分比**暂时未知**。 +原因是检查器中存在针对一致性测试文件内容而写的逻辑，而不是对类型规范的通用实现： +那些规则匹配的是代码的**写法**，而不是代码的含义。改一个导入别名或重新格式化文件， +结论就会变。这样得出的分数并不能作为证据。 -我们正在决定是按规范重建检查器，还是让扩展由一个成熟的开源检查器驱动。**无论走哪 -条路，我们都在把 Basilisk 打造成准确的 Python 开发体验** —— 而新的数字只有在经受住 -套件之外的用例与变异测试后才会发布。 +**因此，我们正在逐条审计规则，并删除那些没有做真正类型检查的规则。** 不是重写，不是 +打补丁，也不是标一个 TODO —— 是删除，并留下一个失败的测试，让这个缺口可见而不是被 +掩盖。一条规则只有在依据已解析的语法树做判断、并且在代码换一种写法时给出相同结论的 +情况下，才会保留。 + +这意味着 Basilisk 会**先变小，再变好**。规则会更少，诊断会更少，一致性数字也会更低。 +每一次下降我们都会如实报告，而不是设法回避。留下来的，将是对自己所做之事诚实的代码 +—— 仅此而已。 + +至于被删掉的分析是按规范重建，还是让扩展由一个成熟的开源检查器驱动，我们尚未决定。 +无论走哪条路，新的数字只有在经受住套件之外的用例与变异测试后才会发布。 [阅读完整更正 →](https://www.basilisk-python.dev/zh/docs/conformance/)  •  [完整性审计 →](docs/CONFORMANCE-INTEGRITY-AUDIT.md) diff --git a/vscode-extension/README.md b/vscode-extension/README.md index 4481191cd..7de376698 100644 --- a/vscode-extension/README.md +++ b/vscode-extension/README.md @@ -31,18 +31,30 @@ **The current type checker contains inaccuracies and you should not use it as part of your dev pipeline. We are working on removing any misleading analyzers ASAP. Please read below** -## We withdrew the typing conformance results +## We are auditing the checker and deleting what doesn't hold up We withdrew our 100% conformance claim and our benchmark figures, and asked to be [removed from the official `python/typing` results](https://github.com/python/typing/blob/main/conformance/results/results.html). The cause was checker logic fitted to the contents of conformance test files -instead of implementing the typing specification generally, and a score produced -that way is not evidence. The current percentage is **temporarily unknown**. - -We are deciding whether to rebuild the checker from the specification or drive -the extension with an established open-source checker. **Either way, we are -building Basilisk into an accurate Python development experience** — and a new -figure gets published only once it survives off-suite and mutation testing. +instead of implementing the typing specification generally: rules that matched +the *spelling* of code rather than its meaning. Rename an import or reformat a +file and the answer changed. A score produced that way is not evidence. + +**So we are auditing every rule and deleting the ones that don't do real type +checking.** Not rewriting them, not patching them, not marking them TODO — +deleting them, with a failing test left behind so the gap is visible instead of +hidden. A rule stays only if it decides from the resolved syntax tree and gives +the same answer when the code is spelled differently. + +That means Basilisk gets **smaller** before it gets better. Expect fewer rules, +fewer diagnostics, and a lower conformance number. We will report each drop +rather than avoid it. What is left will be code that is honest about what it +does — nothing else. + +We have not yet decided whether to rebuild the deleted analysis from the +specification or to drive the extension with an established open-source checker. +Either way, no new figure gets published until it survives off-suite and +mutation testing. [Read the full correction →](https://www.basilisk-python.dev/docs/conformance/)  •  [Integrity audit →](https://github.com/Nimblesite/Basilisk/blob/main/docs/CONFORMANCE-INTEGRITY-AUDIT.md) diff --git a/vscode-extension/README.zh.md b/vscode-extension/README.zh.md index 1433fca77..efd80bb95 100644 --- a/vscode-extension/README.zh.md +++ b/vscode-extension/README.zh.md @@ -31,16 +31,25 @@ **当前的类型检查器存在不准确之处，请勿将其用于你的开发流水线。我们正在尽快移除任何具有误导性的分析器。详情请见下文** -## 我们撤回了类型一致性结果 +## 我们正在审计检查器，并删除站不住脚的代码 我们撤回了 100% 的一致性宣称与基准测试数字，并主动请求 [从官方 `python/typing` 结果中移除](https://github.com/python/typing/blob/main/conformance/results/results.html)。 -原因是检查器中存在针对一致性测试文件内容而写的逻辑，而不是对类型规范的通用实现； -这样得出的分数并不能作为证据。当前的百分比**暂时未知**。 +原因是检查器中存在针对一致性测试文件内容而写的逻辑，而不是对类型规范的通用实现： +那些规则匹配的是代码的**写法**，而不是代码的含义。改一个导入别名或重新格式化文件， +结论就会变。这样得出的分数并不能作为证据。 -我们正在决定是按规范重建检查器，还是让扩展由一个成熟的开源检查器驱动。**无论走哪 -条路，我们都在把 Basilisk 打造成准确的 Python 开发体验** —— 而新的数字只有在经受住 -套件之外的用例与变异测试后才会发布。 +**因此，我们正在逐条审计规则，并删除那些没有做真正类型检查的规则。** 不是重写，不是 +打补丁，也不是标一个 TODO —— 是删除，并留下一个失败的测试，让这个缺口可见而不是被 +掩盖。一条规则只有在依据已解析的语法树做判断、并且在代码换一种写法时给出相同结论的 +情况下，才会保留。 + +这意味着 Basilisk 会**先变小，再变好**。规则会更少，诊断会更少，一致性数字也会更低。 +每一次下降我们都会如实报告，而不是设法回避。留下来的，将是对自己所做之事诚实的代码 +—— 仅此而已。 + +至于被删掉的分析是按规范重建，还是让扩展由一个成熟的开源检查器驱动，我们尚未决定。 +无论走哪条路，新的数字只有在经受住套件之外的用例与变异测试后才会发布。 [阅读完整更正 →](https://www.basilisk-python.dev/zh/docs/conformance/)  •  [完整性审计 →](https://github.com/Nimblesite/Basilisk/blob/main/docs/CONFORMANCE-INTEGRITY-AUDIT.md) diff --git a/website/src/docs/conformance.md b/website/src/docs/conformance.md index bc0ec70fc..11a6bd6a2 100644 --- a/website/src/docs/conformance.md +++ b/website/src/docs/conformance.md @@ -1,23 +1,37 @@ --- layout: layouts/docs.njk -title: "Basilisk Conformance Results Are Withdrawn" -description: "Basilisk has withdrawn its former Python typing conformance claim. Its current percentage is temporarily unknown while affected logic is rebuilt and stress-tested beyond the suite." +title: "Basilisk Is Auditing and Deleting Its Checker Rules" +description: "Basilisk withdrew its Python typing conformance claim and is now auditing every rule, deleting the ones that match source text instead of doing real type checking." keywords: basilisk conformance correction, python typing conformance, python/typing results, mutation testing date: 2026-06-23 -dateModified: 2026-08-06 +dateModified: 2026-08-08 author: The Basilisk Project eleventyNavigation: key: Conformance order: 10 --- -# Conformance results withdrawn +# We are auditing the checker and deleting what doesn't hold up -

Correction: Basilisk has retracted its former perfect-score claim. The result was not a trustworthy measure of specification conformance. We asked for Basilisk to be removed from the python/typing results table, and it has been removed. Basilisk's current conformance percentage is temporarily unknown.

+

Correction: Basilisk has retracted its former perfect-score claim. The result was not a trustworthy measure of specification conformance. We asked for Basilisk to be removed from the python/typing results table, and it has been removed. Basilisk's current conformance percentage is temporarily unknown, and we are not trying to restore it.

-We found checker logic fitted to the exact contents of conformance test files rather than implementing the typing specification generally. For example, type-alias validation used prefixes and substrings from raw source text, including a special case for `eval(` because that spelling appeared in one test. Equivalent, valid mutations of the suite could therefore change Basilisk's result even though the typing behavior being tested had not changed. +We found checker logic fitted to the exact contents of conformance test files rather than implementing the typing specification generally. Those rules matched the *spelling* of code rather than its meaning: type-alias validation used prefixes and substrings taken from raw source text, including a special case for `eval(` purely because that spelling appeared in one test file. Rename an import or reformat a file and the answer changed, even though the typing behavior being tested had not. -The official suite remains valuable, but a passing result from code developed against the exact fixtures is not enough evidence. We will not publish a replacement percentage until the affected logic has been reimplemented cleanly and shown to survive robustness testing. +A passing result from code developed against the exact fixtures is not evidence, so the fix is not a better score. + +## What we are doing + +**We are auditing every rule and deleting the ones that don't do real type checking.** Not rewriting them, not patching them, not marking them TODO — deleting them, and leaving a failing test behind so the gap is visible rather than hidden. A rule stays only if it decides from the resolved syntax tree and returns the same diagnostics when the same program is spelled differently. + +The consequences are deliberate, and we would rather state them up front than have you discover them: + +- **Basilisk gets smaller before it gets better.** Expect fewer rules and fewer diagnostics. +- **The conformance number will fall.** That is the correct outcome of removing logic that was never doing the analysis, and we will report each drop rather than avoid it. +- **A failing test is worth more to us than a passing fixture** that was carried by code which doesn't analyse anything. The first is an accurate record of what Basilisk cannot do; the second is a claim that it can. + +What is left will be code that is honest about what it does — nothing else. + +Whether the deleted analysis gets rebuilt from the specification, or the extension ends up driven by an established open-source checker, is a decision we have not made yet. Either way, no replacement percentage gets published until it survives the robustness testing described below.

Current python/typing results ↗ @@ -28,11 +42,13 @@ The official suite remains valuable, but a passing result from code developed ag Conformance README ↗

-## What is happening now +## Scope of the audit + +The review covers every place a narrow fixture could have stood in for a general implementation: source-text predicates and substring matching, hard-coded symbol spellings, rules organised around a test file rather than a specification concept, duplicated logic, and accept-everything fallbacks standing in for checks that were never written. -The offending implementation is being removed, and the affected behavior is being rebuilt from the specification and structured syntax rather than from test-file text. The review also covers similar source-text predicates, duplicated logic, permissive fallbacks, and other places where a narrow fixture could have stood in for a general implementation. +Each finding is handled the same way — a test that fails because of the code, then the code is removed, then the removal is recorded. Nothing is quietly repaired in place, because a repair preserves the claim that the rule worked. -This is active remediation, not an indefinite withdrawal. We expect to establish a defensible result after the clean implementation and validation work is complete. If that result is lower than the former claim, we will publish the lower result. +This is active remediation, not an indefinite withdrawal. If a defensible result is lower than the former claim, we publish the lower result. ## The new publication bar @@ -43,8 +59,9 @@ A future conformance result must satisfy all of these checks: 3. Pass independent off-suite cases derived from the typing specification and real-world code rather than from the upstream fixture text. 4. Add regression and mutation tests for every test-specific implementation found by the audit. 5. Publish the robustness and off-suite results alongside the suite percentage and make the methodology reproducible. +6. Pass an audit by someone outside this project before Basilisk is submitted to `python/typing` again. -Until that work is complete, old conformance tables, charts, category scores, pass counts, and false-positive totals are withdrawn and should not be cited as Basilisk's current state. +Until that work is complete, old conformance tables, charts, category scores, pass counts, and false-positive totals are withdrawn and should not be cited as Basilisk's current state. We are not quoting a current figure either — a number is not what is wrong here, and publishing a new one before the audit finishes would repeat the mistake. ## Related performance figures diff --git a/website/src/index.njk b/website/src/index.njk index a5f1710c2..65ac9055f 100644 --- a/website/src/index.njk +++ b/website/src/index.njk @@ -19,10 +19,11 @@ permalink: /

- We have withdrawn both our former conformance claim and our published benchmark figures. - Basilisk was removed from the official python/typing results at our request, - and its current conformance percentage is temporarily unknown while we replace - test-specific implementations and verify the new work with robustness and mutation testing. + We have withdrawn both our former conformance claim and our published benchmark figures, + and Basilisk was removed from the official python/typing results at our request. + We are now auditing every rule and deleting the ones that match source text + instead of doing real type checking. Expect Basilisk to get smaller before it gets better; + what is left will be code that is honest about what it does.

@@ -59,7 +60,8 @@ permalink: / The former result is retracted. At our request, Basilisk has been removed from the official results table. - We will publish a new result when the clean implementation is robust to semantics-preserving mutations. + We are not trying to restore it — deleting rules that never analysed anything will push it + lower first, and we will report that drop rather than avoid it.
@@ -72,7 +74,7 @@ permalink: /

- Both sets of figures are withdrawn pending a clean reimplementation and integrity review. + Both sets of figures are withdrawn while the audit runs. Read the conformance correction → Read the benchmark notice →

diff --git a/website/src/zh/docs/conformance.md b/website/src/zh/docs/conformance.md index 24b3da6b6..aab0fefc6 100644 --- a/website/src/zh/docs/conformance.md +++ b/website/src/zh/docs/conformance.md @@ -1,19 +1,33 @@ --- layout: layouts/docs.njk -title: "Basilisk 符合性结果已撤回" -description: "Basilisk 已撤回此前的 Python typing 符合性声明。在相关逻辑完成全新实现并通过独立稳健性验证之前，当前百分比暂时未知。" +title: "Basilisk 正在审计并删除自己的检查器规则" +description: "Basilisk 已撤回 Python typing 符合性声明，正在逐条审计规则，并删除那些只匹配源文本、并未做真正类型检查的规则。" keywords: basilisk 符合性更正, python typing 符合性, python/typing 结果, 变异测试 -dateModified: 2026-08-06 +dateModified: 2026-08-08 lang: zh --- -# 符合性结果已撤回 +# 我们正在审计检查器，并删除站不住脚的代码 -

更正：Basilisk 已撤回此前的满分声明。该结果并不能可信地衡量规范符合性。我们请求将 Basilisk 从 python/typing 结果表中移除，现已完成移除。Basilisk 当前的符合性百分比暂时未知。

+

更正：Basilisk 已撤回此前的满分声明。该结果并不能可信地衡量规范符合性。我们请求将 Basilisk 从 python/typing 结果表中移除，现已完成移除。Basilisk 当前的符合性百分比暂时未知，而且我们并不打算把它恢复回去。

-我们发现，检查器中的一些逻辑针对符合性测试文件的确切内容进行了适配，而不是普遍实现类型规范。例如，类型别名验证曾对原始源代码文本执行前缀和子字符串判断，其中甚至专门判断了 `eval(`，仅仅因为某个测试使用了这种写法。因此，即使被测试的类型行为没有改变，对套件进行语义等价的变异也可能让 Basilisk 的结果发生变化。 +我们发现，检查器中的一些逻辑针对符合性测试文件的确切内容进行了适配，而不是普遍实现类型规范。那些规则匹配的是代码的**写法**，而不是代码的含义：类型别名验证曾对原始源代码文本执行前缀和子字符串判断，其中甚至专门判断了 `eval(`，仅仅因为某个测试文件使用了这种写法。改一个导入别名或重新格式化文件，结论就会变，尽管被测试的类型行为并没有改变。 -官方套件仍然有价值，但针对固定测试用例开发出的代码即使通过，也不足以证明实现正确。在受影响逻辑完成全新实现并通过稳健性测试之前，我们不会发布替代百分比。 +针对固定测试用例开发出的代码即使通过，也不能作为证据；因此，解决办法不是拿到一个更好的分数。 + +## 我们正在做什么 + +**我们正在逐条审计规则，并删除那些没有做真正类型检查的规则。** 不是重写，不是打补丁，也不是标一个 TODO —— 是删除，并留下一个失败的测试，让这个缺口可见而不是被掩盖。一条规则只有在依据已解析的语法树做判断、并且同一个程序换一种写法时给出相同诊断的情况下，才会保留。 + +由此带来的后果是我们主动选择的，与其让你自己发现，不如先讲清楚： + +- **Basilisk 会先变小，再变好。** 规则会更少，诊断也会更少。 +- **符合性数字会下降。** 删除本来就没有在做分析的逻辑，本就该有这个结果；每一次下降我们都会如实报告，而不是设法回避。 +- **对我们来说，一个失败的测试比一个由不做分析的代码撑起来的通过用例更有价值。** 前者如实记录了 Basilisk 做不到什么，后者则是在宣称它做得到。 + +留下来的，将是对自己所做之事诚实的代码 —— 仅此而已。 + +被删掉的分析是按规范重建，还是让扩展由一个成熟的开源检查器驱动，我们尚未决定。无论走哪条路，在通过下文所述的稳健性验证之前，都不会发布替代百分比。

当前 python/typing 结果 ↗ @@ -24,11 +38,13 @@ lang: zh 符合性 README ↗

-## 当前工作 +## 审计范围 + +审查覆盖每一处可能用狭窄测试用例代替通用实现的地方：源文本判断与子字符串匹配、硬编码的符号写法、围绕某个测试文件而不是围绕规范概念组织的规则、重复逻辑，以及用来顶替从未写出的检查的"全部接受"兜底分支。 -有问题的实现正在被删除，相关行为将根据规范和结构化语法重新实现，不再依赖测试文件文本。审查范围也包括类似的源文本判断、重复逻辑、过度宽松的兜底分支，以及其他可能用狭窄测试用例代替通用实现的地方。 +每一处发现都按同样的方式处理 —— 先写一个因这段代码而失败的测试，然后删除这段代码，然后记录这次删除。不会在原地悄悄修补，因为修补会保留"这条规则本来是有效的"这个说法。 -这是正在进行的修复，并非无限期撤回。我们预计在全新实现和验证完成后，很快会得到一个可以辩护的结果。如果新结果低于此前的声明，我们会如实发布较低的结果。 +这是正在进行的修复，并非无限期撤回。如果可以辩护的结果低于此前的声明，我们会如实发布较低的结果。 ## 今后发布结果的门槛 @@ -39,8 +55,9 @@ lang: zh 3. 通过依据类型规范和真实代码独立设计的套件外用例，而不是从上游测试文本衍生用例。 4. 为审计发现的每一处针对测试的实现添加回归测试和变异测试。 5. 将稳健性与套件外验证结果同套件百分比一并发布，并保证方法可复现。 +6. 在 Basilisk 再次提交给 `python/typing` 之前，先通过一次由项目之外的人进行的审计。 -在这项工作完成之前，旧的符合性表格、图表、分类得分、通过数量和误报统计均已撤回，不应被引用为 Basilisk 的当前状态。 +在这项工作完成之前，旧的符合性表格、图表、分类得分、通过数量和误报统计均已撤回，不应被引用为 Basilisk 的当前状态。我们同样不会引用一个当前数字 —— 问题的根源并不是某个数字，在审计完成前发布一个新数字，只会重蹈覆辙。 ## 相关性能数据 diff --git a/website/src/zh/index.njk b/website/src/zh/index.njk index d8f3a6d12..d1d840e3a 100644 --- a/website/src/zh/index.njk +++ b/website/src/zh/index.njk @@ -20,9 +20,10 @@ permalink: /zh/

- 我们已撤回此前的符合性声明和公开的基准测试数据。应我们的请求，Basilisk - 已从官方 python/typing 结果中移除。在重新实现针对测试特例的逻辑， - 并通过稳健性测试与变异测试验证之前，当前符合性百分比暂时未知。 + 我们已撤回此前的符合性声明和公开的基准测试数据，应我们的请求，Basilisk + 已从官方 python/typing 结果中移除。我们正在逐条审计规则， + 并删除那些只匹配源文本、并未做真正类型检查的规则。 + Basilisk 会先变小，再变好；留下来的，将是对自己所做之事诚实的代码。

@@ -59,7 +60,8 @@ permalink: /zh/ 之前的结果已撤回。应我们的请求，Basilisk 已从 官方结果表中移除。 - 待全新实现通过保持测试语义的变异验证后，我们会发布新的结果。 + 我们并不打算把它恢复回去 —— 删除那些本来就没在做分析的规则，会先把它拉得更低， + 我们会如实报告这个下降，而不是设法回避。
@@ -72,7 +74,7 @@ permalink: /zh/

- 两组数据均已撤回，等待全新实现和完整性审查。 + 审计期间，两组数据均已撤回。 阅读符合性更正 → 阅读基准测试说明 →

From 426b8ac1372b5cd82004c51f1dc59f9d4b3523a6 Mon Sep 17 00:00:00 2001 From: Christian Findlay <16697547+MelbourneDeveloper@users.noreply.github.com> Date: Sat, 8 Aug 2026 07:52:26 +1000 Subject: [PATCH 4/9] fixes --- CONTRIBUTING.md | 8 ++++---- CONTRIBUTING.zh.md | 8 ++++---- docs/plans/CHECKER-TYPESHED-PYPI-PLAN.md | 4 ++-- docs/specs/CHECKER-RULE-TAGGING-SPEC.md | 2 +- .../ai-agents-write-python-type-checking-guardrail.md | 2 +- .../basilisk-100-percent-python-typing-conformance.md | 10 +++++----- ...e-threaded-python-why-type-checking-matters-more.md | 6 +++--- ...penai-acquires-astral-what-it-means-for-basilisk.md | 2 +- ...python-315-typeform-fastapi-pydantic-annotations.md | 2 +- website/src/docs/comparison.md | 6 +++--- website/src/docs/index.md | 8 ++++---- website/src/docs/quick-start.md | 7 ++++--- .../basilisk-100-percent-python-typing-conformance.md | 8 ++++---- ...e-threaded-python-why-type-checking-matters-more.md | 6 +++--- ...penai-acquires-astral-what-it-means-for-basilisk.md | 2 +- website/src/zh/docs/comparison.md | 6 +++--- website/src/zh/docs/index.md | 10 +++++----- website/src/zh/docs/quick-start.md | 5 +++-- 18 files changed, 52 insertions(+), 50 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index f03d35691..79c90e937 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -28,12 +28,12 @@ This isn't hypothetical. Checker logic was fitted to the conformance fixtures, t Agents optimise whatever you measure, and every number here is reachable without doing the underlying work: **conformance, coverage, mutation score, assertions, lint, benchmarks.** Re-derive any metric change before you believe it — agents cannot grade their own homework. What to look for: - **Text-matched logic** — the big one. A rule keyed on raw source text or hard-coded symbol spellings instead of resolved AST symbols scores well and fails on real code. Rename an import (`from typing import Final as F`) or reformat a file: the diagnostics must not change. -- **Silence instead of analysis** — a rule disabled, deleted, or unregistered so it stops firing. +- **Silence instead of analysis** — a rule disabled or quietly unregistered so it stops firing, with the loss undisclosed. Deleting a text-matched rule is the opposite and is what we want: it comes with a failing test and a report saying what went. Judge it by whether the hole is visible afterwards. - **Weakened tests** — failing tests deleted, assertions cut or watered down so "green" means nothing. - **Scoreboard or gate edits** — a hand-edited `conformance_status.csv`, or a lowered threshold (`coverage-thresholds.json`, the mutation or benchmark baselines). -- **Measuring less** — excluded diagnostic codes, skipped fixtures, narrowed mutation scope. A high percentage over part of the suite is not a percentage. +- **Measuring less** — excluded diagnostic codes, skipped fixtures, narrowed mutation scope. A high percentage over part of the suite is not a percentage. Ask for the denominator every time: the mutation score is 100% over 161 mutants of an ~82k-LOC crate, because scope is opt-in. -Metrics move only the *honest* way — coverage and mutation up, false positives down — because the work got better, never because someone changed how we count ([CHKARCH-CONFORMANCE]). +Metrics move only the *honest* way — because the work got better, never because someone changed how we count ([CHKARCH-CONFORMANCE]). The one number expected to **fall** is conformance: removing rules that never analysed anything lowers it, and that drop is progress, reported rather than avoided. ### 2. Test it for real — on real, large codebases @@ -101,7 +101,7 @@ You convert the specs to code and tests and keep all three in sync. The standing 2. **Delete the offending code.** 3. **Tell the user what you deleted and why.** -What gets built back is the user's call, not yours. +What gets built back is the user's call, not yours. **A failing test that pins real incorrect behaviour is worth more than a passing fixture carried by logic that does not analyse code** — the first records what Basilisk can't do, the second falsely claims it can. **The non-negotiables** (full detail in `CLAUDE.md`): diff --git a/CONTRIBUTING.zh.md b/CONTRIBUTING.zh.md index 24bfb0aaa..456767f90 100644 --- a/CONTRIBUTING.zh.md +++ b/CONTRIBUTING.zh.md @@ -30,12 +30,12 @@ Basilisk 由**人类 + AI 协作**构建，分工是刻意设计的。AI 智能 需要留意的： - **基于文本匹配的逻辑** —— 最要命的一类。一条规则如果依据原始源文本或硬编码的符号拼写来判断，而不是依据 AST 上已解析的符号，它会得高分并在真实代码上失效。改一个导入别名（`from typing import Final as F`）或重新格式化文件：诊断结果必须不变。 -- **用沉默代替分析** —— 规则被禁用、删除或取消注册，好让它不再触发。 +- **用沉默代替分析** —— 规则被禁用或悄悄取消注册，好让它不再触发，而这个损失没有被披露。删除依据文本判断的规则恰恰相反，正是我们想要的：它会附带一个失败的测试和一份说明删了什么的报告。判断标准是：事后这个缺口是否可见。 - **被削弱的测试** —— 删掉失败的测试、砍掉断言，或把断言弱化到"绿灯"毫无意义。 - **改动记分板或门禁** —— 手工编辑 `conformance_status.csv`，或调低阈值（`coverage-thresholds.json`、变异或基准基线）。 -- **少测一点** —— 排除诊断码、跳过夹具、收窄变异范围。在部分测试集上得到的高百分比不是百分比。 +- **少测一点** —— 排除诊断码、跳过夹具、收窄变异范围。在部分测试集上得到的高百分比不是百分比。每次都要问分母：变异分数 100% 只覆盖了 161 个变异体，而那个 crate 有约 8.2 万行代码，因为范围是选择性加入的。 -指标只能以*诚实*的方式移动 —— 覆盖率和变异分数向上，误报向下 —— 因为工作确实变好了，而不是因为有人改了计数方式（[CHKARCH-CONFORMANCE]）。 +指标只能以*诚实*的方式移动 —— 因为工作确实变好了，而不是因为有人改了计数方式（[CHKARCH-CONFORMANCE]）。唯一预期会**下降**的数字是一致性：删除那些本来就没在做分析的规则会把它拉低，而这个下降是进展，应当如实报告而不是设法回避。 ### 2. 用真实的大型代码库真刀真枪地测 @@ -103,7 +103,7 @@ Basilisk 由**人类 + AI 协作**构建，分工是刻意设计的。AI 智能 2. **删除这段有问题的代码。** 3. **告诉用户你删了什么、为什么删。** -要重建什么，由用户决定，不由你决定。 +要重建什么，由用户决定，不由你决定。**一个因真实错误行为而失败的测试，比一个由不做分析的代码撑起来的通过用例更有价值** —— 前者如实记录了 Basilisk 做不到什么，后者则是在宣称它做得到。 **不可协商的底线**（详见 `CLAUDE.md`）： diff --git a/docs/plans/CHECKER-TYPESHED-PYPI-PLAN.md b/docs/plans/CHECKER-TYPESHED-PYPI-PLAN.md index af529233b..dc980141b 100644 --- a/docs/plans/CHECKER-TYPESHED-PYPI-PLAN.md +++ b/docs/plans/CHECKER-TYPESHED-PYPI-PLAN.md @@ -95,7 +95,7 @@ installed `site-packages` tree (the stored wheel is the source). ## CI gate {#TYPESHEDPYPI-CI} `make test` (fail-fast, coverage ratchet up), clippy + fmt at strictest, `make lint` (incl. `scripts/check-dependency-shape.sh` — `basilisk-stubs` still links no HTTP client), `deslop`, and -and the conformance run recorded unchanged (advisories are environment status, not Python +the conformance run recorded unchanged (advisories are environment status, not Python diagnostics, so they never enter the diagnostic stream) — all green. `make bench` also ran against the branch, but its outstanding regression is @@ -192,4 +192,4 @@ result nor licenses re-baselining to slower numbers. - **The committed baseline is stale, not just slow**: it was last written by `009f2556` (2026-07-18) while `main` has since merged through `e3e97d30` (2026-08-01, #377). Many merged PRs sit between the baseline and this branch, so nothing attributes the delta to this branch without a same-machine A/B of `main` HEAD vs this branch. - **Part of the delta is environmental**: the two runs pin identical competitor versions, and pyright/mypy/ty/pyrefly/zuban all shifted 3–10 % between them — real, but far short of basilisk's ~50 % on the fast fixtures, so a genuine fixed per-process cost remains to be found. - Recovering the cost — not re-baselining — is the exit condition, and it belongs to the benchmark task, not to this one. The benchmark itself gates nothing ([CHKARCH-TESTING-BENCH](../specs/CHECKER-ARCHITECTURE-SPEC.md#CHKARCH-TESTING-BENCH)); it is read by a human, and no number here passes or fails a build. -- [x] Conformance 100 % / 0 FP unchanged (advisories never enter the scored stream; conformance fixtures ran green inside `_test_rust`). +- [x] Conformance run unchanged (advisories are environment status, never Python diagnostics, so they never enter the diagnostic stream; conformance fixtures ran green inside `_test_rust`). diff --git a/docs/specs/CHECKER-RULE-TAGGING-SPEC.md b/docs/specs/CHECKER-RULE-TAGGING-SPEC.md index 7a903b76a..066f94e00 100644 --- a/docs/specs/CHECKER-RULE-TAGGING-SPEC.md +++ b/docs/specs/CHECKER-RULE-TAGGING-SPEC.md @@ -9,7 +9,7 @@ category, everything else with plain descriptive labels. - **Authoritative source (code):** [`crates/basilisk-checker/src/rule_tags.rs`](../../crates/basilisk-checker/src/rule_tags.rs) - **Conformance test (tests):** [`crates/basilisk-checker/tests/rule_tags_tests.rs`](../../crates/basilisk-checker/tests/rule_tags_tests.rs) - **Related:** [CHKARCH-DIAG](CHECKER-ARCHITECTURE-SPEC.md#CHKARCH-DIAG) (diagnostic rules), - [CHKARCH-CONFORMANCE](CHECKER-ARCHITECTURE-SPEC.md#CHKARCH-CONFORMANCE) (conformance scoring) + [CHKARCH-CONFORMANCE](CHECKER-ARCHITECTURE-SPEC.md#CHKARCH-CONFORMANCE) (how the conformance number is measured, and why it is not a target) ## Tag Model {#CHKTAG-MODEL} diff --git a/website/src/blog/ai-agents-write-python-type-checking-guardrail.md b/website/src/blog/ai-agents-write-python-type-checking-guardrail.md index 07e4efc8e..879d95ba6 100644 --- a/website/src/blog/ai-agents-write-python-type-checking-guardrail.md +++ b/website/src/blog/ai-agents-write-python-type-checking-guardrail.md @@ -85,7 +85,7 @@ The right mental model is a guardrail, not a driver. Static typing can remove a Basilisk is an open-source Python type checker and language server built in Rust. Two design choices matter for agent workflows. -First, **there is no separate `--strict` mode to remember.** [`basilisk check`](/docs/configuration/) runs all of Basilisk's PEP-tagged rules by default. Additional house rules are configured separately, and a new LSP workspace seeds those rules at error severity. Basilisk's former conformance result is withdrawn, it has been removed from the official results at our request, and its actual percentage is temporarily unknown while affected logic is reimplemented and verified. Do not treat the old figure as evidence that every possible Python type error or hallucinated API will be detected. +First, **there is no separate `--strict` mode to remember.** [`basilisk check`](/docs/configuration/) runs all of Basilisk's PEP-tagged rules by default. Additional house rules are configured separately, and a new LSP workspace seeds those rules at error severity. Basilisk's former conformance result is withdrawn, it has been removed from the official results at our request, and its actual percentage is temporarily unknown while every rule is audited and the ones that matched source text are deleted. Do not treat the old figure as evidence that every possible Python type error or hallucinated API will be detected. Second, **the checker can run where the agent works.** Basilisk's checker and language server share one native Rust process, with [integrations for VS Code and Cursor, Zed, and Neovim](/docs/installation/). The editor and CLI use the same parser-resolver-checker pipeline, so matching configuration, type sources, Python target, and diagnostic scope produces the same type-checking result. Optional workflows can invoke external components, including Python and `debugpy` for [debugging](/docs/debugging/) and a helper for [profiling](/docs/profiler/). diff --git a/website/src/blog/basilisk-100-percent-python-typing-conformance.md b/website/src/blog/basilisk-100-percent-python-typing-conformance.md index 2f8a6c69c..14d526c5a 100644 --- a/website/src/blog/basilisk-100-percent-python-typing-conformance.md +++ b/website/src/blog/basilisk-100-percent-python-typing-conformance.md @@ -1,7 +1,7 @@ --- layout: layouts/blog.njk title: "Retracted: Basilisk's Former Typing Conformance Result" -description: "Retraction of Basilisk's former Python typing conformance claim, why the result was untrustworthy, and how the affected implementation is being rebuilt and verified." +description: "Retraction of Basilisk's former Python typing conformance claim, why the result was untrustworthy, and why the affected rules are being deleted rather than repaired." date: 2026-07-11 dateModified: 2026-08-06 author: Christian Findlay @@ -16,16 +16,16 @@ excerpt: "Basilisk has retracted its former conformance claim and requested remo keywords: python type checker, python typing conformance, python/typing conformance results, basilisk, mypy, pyright, ty, pyrefly, zuban, pep conformance, strict typing faq: - q: "Which Python type checker has the highest conformance score?" - a: "Basilisk is not currently listed in the official python/typing results. Its former result is withdrawn and its actual percentage is temporarily unknown while affected logic is rebuilt and verified. Check the live official table for currently listed tools." + a: "Basilisk is not currently listed in the official python/typing results. Its former result is withdrawn and its actual percentage is temporarily unknown while every rule is audited and the ones that matched source text are deleted. Check the live official table for currently listed tools." - q: "What is the python/typing conformance suite?" a: "It is the official test suite maintained by the Python Typing community. Its harness records how a checker behaves on the suite's exact fixtures. That is valuable evidence, but a raw suite result alone does not establish faithful implementation of the full specification; mutation robustness and independent off-suite cases are also required." - q: "Is a 100% conformance score the same as being the best type checker?" a: "No. A suite score describes the covered fixtures; it is not proof of specification correctness by itself, as Basilisk's retraction demonstrates. It also does not capture editor integration, error quality, ecosystem support, or independently validated performance." - q: "How is Basilisk's conformance score measured?" - a: "There is no current Basilisk conformance score. A future result will require the unmodified python/typing harness, semantics-preserving mutation testing, and independent off-suite cases derived from the specification, after the affected implementation has been rebuilt." + a: "There is no current Basilisk conformance score. A future result will require the unmodified python/typing harness, semantics-preserving mutation testing, and independent off-suite cases derived from the specification, and only after the audit has finished removing rules that decided from source text rather than resolved symbols." --- -> **Retraction — 6 August 2026:** We withdraw every conformance claim in this post. Basilisk's source contained logic fitted to the exact conformance fixtures, so the former perfect result did not establish specification conformance. We asked for Basilisk to be removed from the official results table, and it has been removed. The current percentage is temporarily unknown while the offending implementation is deleted, rebuilt from the specification, and tested against semantics-preserving mutations. The original article is retained below only as a public record; its score, ranking, pass counts, and conclusions must not be relied on. Read the [full correction](/docs/conformance/). +> **Retraction — 6 August 2026:** We withdraw every conformance claim in this post. Basilisk's source contained logic fitted to the exact conformance fixtures, so the former perfect result did not establish specification conformance. We asked for Basilisk to be removed from the official results table, and it has been removed. The current percentage is temporarily unknown, and we are not trying to restore it: we are auditing every rule and deleting the ones that matched the spelling of code rather than its meaning, which will push the number lower before anything improves. The original article is retained below only as a public record; its score, ranking, pass counts, and conclusions must not be relied on. Read the [full correction](/docs/conformance/). Python has a genuinely good type system now, and most developers still do not realize it. A Python type checker works a lot like the TypeScript compiler. Type-checked Python is to regular Python what TypeScript is to JavaScript. The annotations have been in the language for a decade, the specification is mature, and the tooling has caught up. @@ -87,7 +87,7 @@ If conformance is not the whole story, why did we make 100% a hard requirement r Because the alternative is a checker that is confidently wrong some of the time, and a checker that is confidently wrong is worse than no checker at all. The problem with Python typing was never the syntax. The problem was enforcement. A type hint that is never checked is a comment. A type hint that is checked by a tool with gaps is a comment that occasionally lies to you. -Basilisk enables its typing-spec rules by default, with no `--strict` flag to remember. We claimed the old score proved those rules implemented the specification correctly. It did not; that implementation is now being rebuilt and verified. +Basilisk enables its typing-spec rules by default, with no `--strict` flag to remember. We claimed the old score proved those rules implemented the specification correctly. It did not, and the rules that cannot show they analyse code are being deleted rather than repaired. ## How the withdrawn score was produced diff --git a/website/src/blog/free-threaded-python-why-type-checking-matters-more.md b/website/src/blog/free-threaded-python-why-type-checking-matters-more.md index a3097525e..c8df3f215 100644 --- a/website/src/blog/free-threaded-python-why-type-checking-matters-more.md +++ b/website/src/blog/free-threaded-python-why-type-checking-matters-more.md @@ -24,7 +24,7 @@ faq: - q: "What is the performance cost of free-threaded Python?" a: "According to the Python 3.14 release notes, the single-threaded performance penalty in free-threaded mode is now roughly 5-10%, depending on the platform and C compiler used, a significant improvement over earlier builds." - q: "How does Basilisk help with all of this?" - a: "Basilisk enables its typing-spec rules by default, with no strict flag to remember. Its former conformance result has been withdrawn, however, and its actual percentage is temporarily unknown while affected logic is reimplemented and verified." + a: "Basilisk enables its typing-spec rules by default, with no strict flag to remember. Its former conformance result has been withdrawn, however, and its actual percentage is temporarily unknown while every rule is audited and the ones that matched source text are deleted." --- Free-threaded Python stopped being an experiment. As of Python 3.14, released on October 7, 2025, the free-threaded (no-GIL) build is officially supported, not experimental, under [PEP 779](https://peps.python.org/pep-0779/) ([Python 3.14 release notes, python.org](https://docs.python.org/3/whatsnew/3.14.html)). If you have been half-watching the "no-GIL" story for the last few years, this is the moment it went real. @@ -81,7 +81,7 @@ You do not need to wait for Phase III or rewrite anything to get ahead of this. Basilisk is our answer to the "enforcement is optional" problem. It is an open-source Python type checker and language server built in Rust, with its typing-spec rules enabled and no `--strict` flag to forget. -**Correction:** Basilisk's former conformance result is withdrawn. Test-specific implementation logic made that number untrustworthy, Basilisk has been removed from the official results at our request, and its current percentage is temporarily unknown. See the [conformance correction](/docs/conformance/) for the clean reimplementation and robustness-testing work now underway. +**Correction:** Basilisk's former conformance result is withdrawn. Test-specific implementation logic made that number untrustworthy, Basilisk has been removed from the official results at our request, and its current percentage is temporarily unknown. See the [conformance correction](/docs/conformance/) for the audit-and-delete work now underway. A few honest boundaries so you know exactly what you are getting: @@ -119,4 +119,4 @@ According to the [Python 3.14 release notes](https://docs.python.org/3/whatsnew/ ### How does Basilisk help with all of this? -Basilisk is a Python type checker whose typing-spec rules are enabled by default, with no strict flag to forget. Its former conformance result is withdrawn and its current percentage is temporarily unknown while affected logic is rebuilt and verified; evaluate it against your own code rather than relying on the old figure. +Basilisk is a Python type checker whose typing-spec rules are enabled by default, with no strict flag to forget. Its former conformance result is withdrawn and its current percentage is temporarily unknown while rules that matched source text are deleted; evaluate it against your own code rather than relying on the old figure. diff --git a/website/src/blog/openai-acquires-astral-what-it-means-for-basilisk.md b/website/src/blog/openai-acquires-astral-what-it-means-for-basilisk.md index b8e67b62a..8c1f4cd83 100644 --- a/website/src/blog/openai-acquires-astral-what-it-means-for-basilisk.md +++ b/website/src/blog/openai-acquires-astral-what-it-means-for-basilisk.md @@ -95,7 +95,7 @@ Basilisk's relationship to Astral is concrete and load-bearing: 1. **Our parser is Ruff's parser.** Basilisk depends on `ruff_python_parser`, `ruff_python_ast`, and `ruff_text_size`, pinned to an **immutable git commit** (`rev 7c645a9`, equal to tag `0.15.17`) on `astral-sh/ruff`. We pin a `rev`, not a tag, precisely so the version "can never be swapped out from under us." That code is MIT-licensed and already in our `Cargo.lock`. Nothing about this acquisition can reach back and change the bytes we build against. 2. **Our lint/format path shells out to the Ruff CLI** — `ruff==0.15.17`, pinned identically in CI and the dev container. Same story: a pinned, permissively licensed binary we control the version of. -3. **ty is now an OpenAI-backed competitor.** Astral's type checker, ty, occupies the same conceptual space as the Basilisk checker, and it will now have OpenAI's resources behind it. We take that seriously. Basilisk combines typing-spec rules with one **complete LSP** (test explorer, debugging, profiling, autofixes) in a single extension. Its actual conformance is currently under integrity review, and the affected logic is being rebuilt rather than represented by the withdrawn result. +3. **ty is now an OpenAI-backed competitor.** Astral's type checker, ty, occupies the same conceptual space as the Basilisk checker, and it will now have OpenAI's resources behind it. We take that seriously. Basilisk combines typing-spec rules with one **complete LSP** (test explorer, debugging, profiling, autofixes) in a single extension. Its actual conformance is currently under integrity audit, and rules that matched source text are being deleted rather than represented by the withdrawn result. 4. **The architecture bet is shared — and now vindicated.** Basilisk, like Astral's tools, is built in Rust on the Ruff AST with Salsa for incrementality. Astral proved that stack scales to millions of users. We made the same call independently. That's reassuring, not threatening. **Net effect on you, today:** zero. Your Basilisk install builds from pinned, MIT-licensed Ruff code and a pinned Ruff binary. The acquisition does not, and cannot, alter either. diff --git a/website/src/blog/python-315-typeform-fastapi-pydantic-annotations.md b/website/src/blog/python-315-typeform-fastapi-pydantic-annotations.md index 2162f5bea..acc5ebc01 100644 --- a/website/src/blog/python-315-typeform-fastapi-pydantic-annotations.md +++ b/website/src/blog/python-315-typeform-fastapi-pydantic-annotations.md @@ -140,7 +140,7 @@ Basilisk is an open-source Python type checker and language server that adds cod All three Python 3.15 typing PEPs have conformance test files in the official suite. We previously published pass counts for those files and used them to assert support. **Those figures and that support claim are withdrawn.** Test-specific implementation logic elsewhere in the checker demonstrated that a pass against an exact fixture was not enough to establish a general implementation. -Basilisk's support status for `TypeForm`, `closed` / `extra_items`, and `disjoint_base` is therefore being revalidated as part of the clean reimplementation and integrity audit. Until those rules pass semantics-preserving mutations and broader cases that were not present in the suite, do not rely on the old table. See the [conformance correction](/docs/conformance/) for the publication bar we will apply to the replacement result. +Basilisk's support status for `TypeForm`, `closed` / `extra_items`, and `disjoint_base` is therefore being revalidated as part of the integrity audit, and any rule that turns out to match source text is deleted rather than repaired. Until those rules pass semantics-preserving mutations and broader cases that were not present in the suite, do not rely on the old table. See the [conformance correction](/docs/conformance/) for the publication bar we will apply to the replacement result. ## What to do before October diff --git a/website/src/docs/comparison.md b/website/src/docs/comparison.md index 1fdcea9ef..35555e2d4 100644 --- a/website/src/docs/comparison.md +++ b/website/src/docs/comparison.md @@ -17,7 +17,7 @@ There is no universal **best Python type checker** for every codebase. The right The Python type checker landscape has changed significantly. The tools differ in how faithfully they implement the typing spec, in whether they're a complete language server or only a checker, and in speed. Basilisk's previously published performance measurements are currently [withdrawn pending review](/docs/benchmarks/). -

Conformance correction: Basilisk's former result is withdrawn and its current percentage is temporarily unknown. Basilisk has been removed from the official results table at our request while affected logic is rebuilt and stress-tested beyond the exact suite fixtures. Do not use the old score or leaderboard position to compare these tools.

+

Conformance correction: Basilisk's former result is withdrawn and its current percentage is temporarily unknown. Basilisk has been removed from the official results table at our request while every rule is audited and the ones that matched source text are deleted. Do not use the old score or leaderboard position to compare these tools.

## The fundamental question @@ -195,14 +195,14 @@ Pyrefly was built by Meta to handle their Python codebase, one of the largest in Basilisk is not a faster version of an existing tool. It occupies a different position: **Basilisk combines:** -1. Typing-spec rules enabled by default, plus **opt-in Basilisk rules** for checking stricter than the spec. The conformance implementation is currently being rebuilt and its percentage is temporarily unknown. +1. Typing-spec rules enabled by default, plus **opt-in Basilisk rules** for checking stricter than the spec. Those rules are currently under audit — the ones that decided from source text are being deleted — and the percentage is temporarily unknown. 2. Annotation quick-fixes, one-click code actions that insert a placeholder annotation (`: Any`, `-> None`) on unannotated code, so you can fill in the real type instead of finding the spot by hand 3. A complete, open-source LSP in every editor, completions, hover, go-to-definition, refactoring, debugging, and profiling, the same in VS Code, plus native Zed and Neovim extensions (Open VSX for Cursor, Windsurf, and others coming very soon; JetBrains planned), not just inside one proprietary VS Code extension 4. Integrated debugger and profiler brokered through the language server 5. WASM plugin system (planned), extensible without forking, secure by design **Where Basilisk is still growing:** -- Basilisk is under active development. Its former conformance result is withdrawn; affected logic is being reimplemented from scratch and the [current percentage is temporarily unknown](/docs/conformance/). +- Basilisk is under active development. Its former conformance result is withdrawn; rules that matched source text instead of analysing code are being deleted, and the [current percentage is temporarily unknown](/docs/conformance/). - Plugin ecosystem: mypy's Django and SQLAlchemy plugins are mature. Basilisk's WASM plugins are planned. The recommendation: evaluate Basilisk for its integrated open-source editor workflow and test it against your own code. Do not choose it on the basis of the withdrawn conformance or benchmark figures. A new conformance result will be published when the clean implementation and robustness review are complete. diff --git a/website/src/docs/index.md b/website/src/docs/index.md index 208c264b5..3c0631620 100644 --- a/website/src/docs/index.md +++ b/website/src/docs/index.md @@ -15,7 +15,7 @@ eleventyNavigation: Basilisk is an open-source **Python type checker and language server** built in Rust. It adds code intelligence, formatting, type-aware refactoring, testing, debugging, and CPU and memory profiling to your editor. Its default rules are intended to implement the Python typing specification, and that implementation is currently undergoing an integrity review. -**Conformance correction:** Basilisk's former result is withdrawn, its current percentage is temporarily unknown, and it has been removed from the official `python/typing` results at our request. We are rebuilding affected logic from scratch and will publish a new result after robustness and mutation verification. Read the [full correction](/docs/conformance/). +**Conformance correction:** Basilisk's former result is withdrawn, its current percentage is temporarily unknown, and it has been removed from the official `python/typing` results at our request. We are auditing every rule and deleting the ones that matched source text instead of doing real type checking. Expect the number to fall first; a new result gets published only after robustness and mutation verification. Read the [full correction](/docs/conformance/). Extensions ship for **VS Code**, **Cursor**, **Windsurf**, **Zed**, and **Neovim**; any editor that speaks the Language Server Protocol can use the same server. JetBrains support is planned. Feature coverage varies per editor — see [the integration matrix](/docs/installation/#integration-status-by-editor). @@ -53,7 +53,7 @@ Basilisk puts type checking, language features, formatting, debugging, and profi ## Typing-spec rules by default, configurable from there -Basilisk's behaviour is decided entirely by **configuration**, and the default configuration enables the **core PEP rule set** — the rules the official typing-conformance suite grades. These rules aim to follow the specification with no strictness flag required, but the withdrawn result means their actual conformance is temporarily unknown while they are audited and reimplemented where necessary. +Basilisk's behaviour is decided entirely by **configuration**, and the default configuration enables the **core PEP rule set** — the rules the official typing-conformance suite grades. These rules aim to follow the specification with no strictness flag required, but the withdrawn result means their actual conformance is temporarily unknown while they are audited and deleted where they turn out to decide from source text. Stricter-than-spec checking is **opt-in**. Basilisk also ships extra rules the spec doesn't define — *require an annotation* on every parameter and return, a redundant-annotation warning, a missing-`@override` nudge, an explicit-`Any` nudge. They stay **off** until you enable them in config. Because they flag code the spec considers valid, turning them on deliberately trades strict spec conformance for a stricter standard of your team's choosing — a per-project choice, never a default. @@ -70,13 +70,13 @@ This keeps the default focused on spec-derived rules while letting each team dia ## Project status -Basilisk is under **active development** — the core checker, LSP server, and editor extensions are working. Its former conformance result is withdrawn while affected checker logic is rebuilt and verified. Autocomplete, go-to-definition, hover, diagnostics, inlay hints, refactoring, debugging, and profiling are shipping today. +Basilisk is under **active development** — the core checker, LSP server, and editor extensions are working. Its former conformance result is withdrawn while checker rules are audited and the ones that matched source text are deleted. Autocomplete, go-to-definition, hover, diagnostics, inlay hints, refactoring, debugging, and profiling are shipping today. | Phase | Milestone | Status | |---|---|---| | 1 | Parser, resolver, type checker, CLI | Complete | | 2 | LSP server, editor extensions (VS Code, Cursor, Zed, Neovim) | Complete | -| 3 | Clean PEP-rule reimplementation, robustness and mutation verification, gradual adoption | In progress | +| 3 | Audit and delete text-matched rules, build semantics-preserving mutation testing, gradual adoption | In progress | | 4 | WASM plugins, Django/Pydantic/SQLAlchemy | Planned | | 5 | SARIF/JUnit output, JetBrains extension | Planned | | 6 | Plugin marketplace, community stubs, ecosystem | Planned | diff --git a/website/src/docs/quick-start.md b/website/src/docs/quick-start.md index 3657f2fda..0228f7dbf 100644 --- a/website/src/docs/quick-start.md +++ b/website/src/docs/quick-start.md @@ -83,9 +83,10 @@ Found 3 diagnostics (3 errors). Out of the box, Basilisk enables every currently registered PEP-tagged rule at **error** severity; optional house rules are separate. That describes the default configuration, not completeness or correctness. Basilisk's actual -conformance level is temporarily unknown while affected rules are rebuilt from -the [Python type system specification](https://typing.python.org/en/latest/spec/index.html) -and independently validated. See the [conformance correction](/docs/conformance/). +conformance level is temporarily unknown while every rule is audited against the +[Python type system specification](https://typing.python.org/en/latest/spec/index.html) +and the ones that matched source text instead of analysing code are deleted. +Expect fewer rules, not more. See the [conformance correction](/docs/conformance/). ## Step 2 — Fix the errors diff --git a/website/src/zh/blog/basilisk-100-percent-python-typing-conformance.md b/website/src/zh/blog/basilisk-100-percent-python-typing-conformance.md index f2e2f9c64..16ea9c4bb 100644 --- a/website/src/zh/blog/basilisk-100-percent-python-typing-conformance.md +++ b/website/src/zh/blog/basilisk-100-percent-python-typing-conformance.md @@ -1,7 +1,7 @@ --- layout: layouts/blog.njk title: "已撤回：Basilisk 此前的类型符合性结果" -description: "撤回 Basilisk 此前的 Python typing 符合性声明，说明该结果为何不可信，以及如何重新实现并验证受影响逻辑。" +description: "撤回 Basilisk 此前的 Python typing 符合性声明，说明该结果为何不可信，以及受影响的规则为何是被删除而不是被修补。" date: 2026-07-11 dateModified: 2026-08-06 author: Christian Findlay @@ -17,13 +17,13 @@ excerpt: "Basilisk 已撤回此前的符合性声明，并请求从官方结果 keywords: python类型检查器, python类型符合性, python/typing符合性结果, basilisk, mypy, pyright, ty, pyrefly, zuban, pep符合性, 严格类型 faq: - q: "哪个 Python 类型检查器的符合性分数最高？" - a: "Basilisk 目前不在官方 python/typing 结果中。此前的结果已撤回；在受影响逻辑完成重新实现和验证之前，实际百分比暂时未知。其他工具请查看官方实时结果表。" + a: "Basilisk 目前不在官方 python/typing 结果中。此前的结果已撤回；在完成规则审计、删除依据源文本判断的规则之前，实际百分比暂时未知。其他工具请查看官方实时结果表。" - q: "什么是 python/typing 符合性测试套件？" a: "这是由 Python Typing 社区维护的官方测试套件。它使用自己的评分工具记录检查器在确切测试用例上的行为。这是有价值的证据，但原始套件结果本身不能证明完整规范已被忠实实现；还必须通过保持语义的变异和独立的套件外用例验证。" - q: "100% 的符合性分数是否等于是最好的类型检查器？" a: "不。套件分数只描述被覆盖的测试用例；正如 Basilisk 此次撤回所证明的，它本身不能证明规范实现正确。它也无法反映编辑器集成、错误信息质量、生态系统支持或经过独立验证的性能。" - q: "Basilisk 的符合性分数是如何测量的？" - a: "Basilisk 目前没有可发布的符合性分数。受影响实现完成重建后，未来结果必须同时通过未经修改的 python/typing 评分工具、保持语义的变异测试，以及依据规范独立设计的套件外用例。" + a: "Basilisk 目前没有可发布的符合性分数。只有在审计完成、依据源文本而非已解析符号判断的规则被删除之后，未来结果才可能发布，并且必须同时通过未经修改的 python/typing 评分工具、保持语义的变异测试，以及依据规范独立设计的套件外用例。" --- > **撤回说明——2026 年 8 月 6 日：**我们撤回本文中的所有符合性声明。Basilisk 源码中存在针对确切符合性测试用例实现的逻辑，因此此前的满分结果不能证明规范符合性。我们请求从官方结果表中移除 Basilisk，现已完成移除。在删除有问题的实现、根据规范重新构建并通过保持语义的变异测试之前，当前百分比暂时未知。下方原文仅作为公开历史记录保留；其中的得分、排名、通过数量和结论均不可依赖。请阅读[完整更正](/zh/docs/conformance/)。 @@ -102,6 +102,6 @@ Basilisk 默认启用类型规范规则，无需记住 `--strict` 标志。我 你可以在[符合性页面](/zh/docs/conformance/)阅读当前更正和修复计划，并在 [python/typing 结果页面]({{ conformanceOfficial.historical.snapshot.source }})上看到 Basilisk 已不再列出。 -请把 Basilisk 指向你自己的代码，并在 [GitHub](https://github.com/Nimblesite/Basilisk/issues) 上报告分歧。旧得分不能替代这种真实检验。只有在全新实现通过更广泛的回归用例和保持语义的变异后，我们才会发布替代结果。 +请把 Basilisk 指向你自己的代码，并在 [GitHub](https://github.com/Nimblesite/Basilisk/issues) 上报告分歧。旧得分不能替代这种真实检验。只有在审计完成、留下的代码通过更广泛的回归用例和保持语义的变异后，我们才会发布替代结果。 Python 的类型系统已经足够好、值得信任有一段时间了。现在，工具也可以了。 diff --git a/website/src/zh/blog/free-threaded-python-why-type-checking-matters-more.md b/website/src/zh/blog/free-threaded-python-why-type-checking-matters-more.md index c015ec4c0..bc89fabbe 100644 --- a/website/src/zh/blog/free-threaded-python-why-type-checking-matters-more.md +++ b/website/src/zh/blog/free-threaded-python-why-type-checking-matters-more.md @@ -25,7 +25,7 @@ faq: - q: "自由线程 Python 的性能代价是多少？" a: "根据 Python 3.14 发布说明，自由线程模式下单线程的性能损失现在约为 5-10%，取决于平台和 C 编译器，相较早期构建有显著改进。" - q: "Basilisk 在这一切中如何提供帮助？" - a: "Basilisk 默认启用类型规范规则，没有要忘记的严格标志。但此前的符合性结果已经撤回；在受影响逻辑完成重新实现和验证之前，实际百分比暂时未知。" + a: "Basilisk 默认启用类型规范规则，没有要忘记的严格标志。但此前的符合性结果已经撤回；在完成规则审计、删除依据源文本判断的规则之前，实际百分比暂时未知。" --- 自由线程 Python 不再是一个实验。自 2025 年 10 月 7 日发布的 Python 3.14 起，自由线程（无 GIL）构建在 [PEP 779](https://peps.python.org/pep-0779/) 下获得正式支持，而非实验性（[Python 3.14 发布说明，python.org](https://docs.python.org/3/whatsnew/3.14.html)）。如果过去几年你一直半留意着"无 GIL"这个故事，那么现在就是它成真的时刻。 @@ -82,7 +82,7 @@ faq: Basilisk 是我们对"强制执行是可选的"这个问题的回答。它是一个用 Rust 构建的开源 Python 类型检查器和语言服务器，默认启用类型规范规则，没有要忘记的 `--strict` 标志。 -**更正：**Basilisk 此前的符合性结果已撤回。针对特定测试实现的逻辑使该数字不可信；应我们的请求，Basilisk 已从官方结果中移除，当前百分比暂时未知。请参阅[符合性更正](/zh/docs/conformance/)，了解正在进行的全新实现和稳健性测试。 +**更正：**Basilisk 此前的符合性结果已撤回。针对特定测试实现的逻辑使该数字不可信；应我们的请求，Basilisk 已从官方结果中移除，当前百分比暂时未知。请参阅[符合性更正](/zh/docs/conformance/)，了解正在进行的审计、删除工作和稳健性测试。 几条诚实的边界，好让你确切知道你得到的是什么： @@ -120,4 +120,4 @@ Basilisk 是我们对"强制执行是可选的"这个问题的回答。它是一 ### Basilisk 在这一切中如何提供帮助？ -Basilisk 默认启用类型规范规则，没有要忘记的严格标志。但此前的符合性结果已撤回；在受影响逻辑完成重新实现和验证之前，当前百分比暂时未知。请在自己的代码上评估它，不要依赖旧数字。 +Basilisk 默认启用类型规范规则，没有要忘记的严格标志。但此前的符合性结果已撤回；在完成规则审计、删除依据源文本判断的规则之前，当前百分比暂时未知。请在自己的代码上评估它，不要依赖旧数字。 diff --git a/website/src/zh/blog/openai-acquires-astral-what-it-means-for-basilisk.md b/website/src/zh/blog/openai-acquires-astral-what-it-means-for-basilisk.md index f45138ad8..19d60f27d 100644 --- a/website/src/zh/blog/openai-acquires-astral-what-it-means-for-basilisk.md +++ b/website/src/zh/blog/openai-acquires-astral-what-it-means-for-basilisk.md @@ -96,7 +96,7 @@ Basilisk 与 Astral 的关系是具体而承重的： 1. **我们的解析器就是 Ruff 的解析器。** Basilisk 依赖 `ruff_python_parser`、`ruff_python_ast` 与 `ruff_text_size`，并将它们钉在 `astral-sh/ruff` 上一个**不可变的 git 提交**（`rev 7c645a9`，等同于标签 `0.15.17`）。我们钉的是 `rev` 而非标签，正是为了让这个版本"永远无法被人从我们脚下换掉"。那份代码是 MIT 授权的，并且已经写进了我们的 `Cargo.lock`。这次收购无法回过头去改变我们所构建的那些字节。 2. **我们的检查/格式化路径调用的是 Ruff CLI**——`ruff==0.15.17`，在 CI 和开发容器中钉得完全一致。同样的道理：一个我们自己掌控版本的、宽松许可的二进制文件。 -3. **ty 如今是有 OpenAI 撑腰的竞争对手。** Astral 的类型检查器 ty 与 Basilisk 的检查器处在同一概念空间，如今它背后将有 OpenAI 的资源。我们认真对待这一点。Basilisk 将类型规范规则和一套**完整 LSP**（测试浏览器、调试、性能分析、自动修复）整合在单一扩展中。其实际符合性目前正在接受完整性审查，受影响逻辑会被重新实现，不能再由已撤回的结果代表。 +3. **ty 如今是有 OpenAI 撑腰的竞争对手。** Astral 的类型检查器 ty 与 Basilisk 的检查器处在同一概念空间，如今它背后将有 OpenAI 的资源。我们认真对待这一点。Basilisk 将类型规范规则和一套**完整 LSP**（测试浏览器、调试、性能分析、自动修复）整合在单一扩展中。其实际符合性目前正在接受完整性审计，只匹配源文本的规则会被删除，不能再由已撤回的结果代表。 4. **共同的架构押注——如今得到了印证。** 与 Astral 的工具一样，Basilisk 用 Rust 构建、基于 Ruff AST、以 Salsa 实现增量计算。Astral 已经证明这套技术栈能扩展到数以百万计的用户。我们独立地做出了同样的选择。这令人安心，而非令人不安。 **对今天的你而言，净影响：** 零。你的 Basilisk 安装构建自被钉死的、MIT 授权的 Ruff 代码和一个被钉死的 Ruff 二进制文件。这次收购不会、也不能改变其中任何一个。 diff --git a/website/src/zh/docs/comparison.md b/website/src/zh/docs/comparison.md index a27d04871..1d48a7b7b 100644 --- a/website/src/zh/docs/comparison.md +++ b/website/src/zh/docs/comparison.md @@ -11,7 +11,7 @@ dateModified: 2026-08-06 Python 类型检查器的格局已经发生了重大变化。它们的差异在于对类型规范的实现有多忠实、究竟是一个完整的语言服务器还是仅仅一个检查器，以及速度。Basilisk 之前公开的性能数据目前已[撤回并等待审查](/docs/benchmarks/)。 -

符合性更正：Basilisk 此前的结果已撤回，当前百分比暂时未知。应我们的请求，Basilisk 已从官方结果表中移除，同时受影响的逻辑正在重新实现并接受独立稳健性验证。请勿使用旧得分或旧排名比较这些工具。

+

符合性更正：Basilisk 此前的结果已撤回，当前百分比暂时未知。应我们的请求，Basilisk 已从官方结果表中移除，同时我们正在逐条审计规则，并删除那些只匹配源文本、并未做真正类型检查的规则。请勿使用旧得分或旧排名比较这些工具。

## 根本问题 @@ -48,7 +48,7 @@ Python 类型检查器的格局已经发生了重大变化。它们的差异在 **来源：** -¹ 当前列出的检查器请参见[官方 python/typing 实时结果](https://github.com/python/typing/blob/main/conformance/results/results.html)。Basilisk 在撤回旧结果后请求移除；只有在全新实现通过稳健性和变异验证后，才会发布当前百分比。 +¹ 当前列出的检查器请参见[官方 python/typing 实时结果](https://github.com/python/typing/blob/main/conformance/results/results.html)。Basilisk 在撤回旧结果后请求移除；只有在审计完成、并通过稳健性和变异验证后，才会发布当前百分比。 ² Basilisk 的快速修复插入的是**占位符**注解（参数和属性为 `: Any`，返回值为 `-> None`；空集合变量为 `list[Any]` / `dict[str, Any]`），供你替换为真实类型。它不推断类型。参见[缺失注解规则](/zh/docs/rules/missing-annotations/)。 @@ -199,4 +199,4 @@ Basilisk 不是现有工具的更快版本。它占据了不同的位置： - Basilisk 正在积极开发中。此前的符合性结果已撤回；受影响逻辑正在从头实现，[当前百分比暂时未知](/zh/docs/conformance/)。 - 插件生态系统：mypy 的 Django 和 SQLAlchemy 插件已经成熟。Basilisk 的 WASM 插件是计划中的。 -建议：根据 Basilisk 集成的开源编辑器工作流进行评估，并在您自己的代码上测试它。不要依据已撤回的符合性或基准测试数据做出选择。待全新实现和稳健性审查完成后，我们会发布新的符合性结果。 +建议：根据 Basilisk 集成的开源编辑器工作流进行评估，并在您自己的代码上测试它。不要依据已撤回的符合性或基准测试数据做出选择。待审计与稳健性审查完成后，我们会发布新的符合性结果。 diff --git a/website/src/zh/docs/index.md b/website/src/zh/docs/index.md index 87a10e0f3..1fe128a75 100644 --- a/website/src/zh/docs/index.md +++ b/website/src/zh/docs/index.md @@ -10,7 +10,7 @@ lang: zh Basilisk 是一个**完整的开源 Python 语言服务器**。您依赖现代 Python 扩展提供的一切——自动补全、跳转到定义、悬停信息、重构、诊断、集成调试、性能分析——Basilisk 全部提供，完全开源。其默认规则旨在实现 Python 类型规范，而该实现目前正在接受完整性审查。 -**符合性更正：**Basilisk 此前的结果已撤回，当前百分比暂时未知，并已应我们的请求从官方 `python/typing` 结果中移除。我们正在从头重新实现受影响的逻辑，并会在通过稳健性测试和变异验证后发布新结果。请阅读[完整更正](/zh/docs/conformance/)。 +**符合性更正：**Basilisk 此前的结果已撤回，当前百分比暂时未知，并已应我们的请求从官方 `python/typing` 结果中移除。我们正在逐条审计规则，并删除那些只匹配源文本、并未做真正类型检查的规则。数字会先下降；只有在通过稳健性测试和变异验证后，才会发布新结果。请阅读[完整更正](/zh/docs/conformance/)。 它不仅仅是一个类型检查器。它是一个功能完整的 LSP，已为 **VS Code**、**Cursor**、**Windsurf**、**Zed** 和 **Neovim** 提供扩展——以及支持语言服务器协议的其他编辑器。JetBrains（IntelliJ / PyCharm）支持已纳入计划。无专有扩展。无 Node.js。单个 Rust 二进制文件，在每款编辑器中提供相同的体验。 @@ -20,7 +20,7 @@ Basilisk 是一个**完整的开源 Python 语言服务器**。您依赖现代 P 其他每个 Python 类型检查器（mypy、ty、Pyrefly）都*只是*检查器——没有补全、没有重构、没有调试器。你得另外搭一个语言服务器，并让两者在团队中保持同步。 -Basilisk 采取不同的立场。它默认启用类型规范规则，并将整个工具栈（类型检查、语言功能、调试、性能分析）整合为一个开源工具，在**每一款**编辑器中运行方式相同，而不仅仅是 VS Code。默认规则的实际符合程度暂时未知，正在审查和必要的重新实现中。想要比规范更严格的检查？开启可选的 Basilisk 规则。 +Basilisk 采取不同的立场。它默认启用类型规范规则，并将整个工具栈（类型检查、语言功能、调试、性能分析）整合为一个开源工具，在**每一款**编辑器中运行方式相同，而不仅仅是 VS Code。默认规则的实际符合程度暂时未知，正在接受审计；凡是依据源文本判断的规则都会被删除。想要比规范更严格的检查？开启可选的 Basilisk 规则。 ## Basilisk 是什么 @@ -44,7 +44,7 @@ Basilisk 采取不同的立场。它默认启用类型规范规则，并将整 ## 默认启用规范规则，并可从此配置 -Basilisk 的行为完全由**配置**决定，而默认配置启用**核心 PEP 规则集**——与官方类型符合性套件评分所用的规则相同。这些规则旨在无需额外严格模式即可遵循规范，但在撤回旧结果后，其实际符合程度暂时未知，正在接受审查并在需要时重新实现。 +Basilisk 的行为完全由**配置**决定，而默认配置启用**核心 PEP 规则集**——与官方类型符合性套件评分所用的规则相同。这些规则旨在无需额外严格模式即可遵循规范，但在撤回旧结果后，其实际符合程度暂时未知，正在接受审计；凡是依据源文本判断的规则都会被删除。 比规范更严格的检查是**可选启用**的。Basilisk 还附带规范未定义的额外规则——要求每个参数和返回值都有注解、冗余注解警告、缺失 `@override` 提示、显式 `Any` 提示。在你于配置中启用之前，它们始终**关闭**。由于它们会标记规范视为有效的代码，刻意开启它们就是用严格的规范符合换取由团队自行选择的更严格标准——这是逐项目的选择，绝非默认。 @@ -61,13 +61,13 @@ Basilisk 的行为完全由**配置**决定，而默认配置启用**核心 PEP ## 项目状态 -Basilisk 正在**积极开发中**——核心检查器、LSP 服务器和编辑器扩展都在工作。此前的符合性结果已撤回，受影响的检查器逻辑正在重新实现和验证。自动补全、跳转到定义、悬停、诊断、内联提示、重构、调试和性能分析今天就在发布。 +Basilisk 正在**积极开发中**——核心检查器、LSP 服务器和编辑器扩展都在工作。此前的符合性结果已撤回，检查器规则正在接受审计，只匹配源文本的规则会被删除。自动补全、跳转到定义、悬停、诊断、内联提示、重构、调试和性能分析今天就在发布。 | 阶段 | 里程碑 | 状态 | |---|---|---| | 1 | 解析器、解析器、类型检查器、CLI | 完成 | | 2 | LSP 服务器、编辑器扩展（VS Code、Cursor、Zed、Neovim） | 完成 | -| 3 | 全新实现 PEP 规则、稳健性与变异验证、渐进式采用 | 进行中 | +| 3 | 审计并删除依据文本判断的规则、建立保持语义的变异测试、渐进式采用 | 进行中 | | 4 | WASM 插件，Django/Pydantic/SQLAlchemy | 计划中 | | 5 | SARIF/JUnit 输出，JetBrains 扩展 | 计划中 | | 6 | 插件市场，社区存根，生态系统 | 计划中 | diff --git a/website/src/zh/docs/quick-start.md b/website/src/zh/docs/quick-start.md index e0425ede7..20e95ecd7 100644 --- a/website/src/zh/docs/quick-start.md +++ b/website/src/zh/docs/quick-start.md @@ -77,8 +77,9 @@ Found 3 diagnostics (3 errors). 开箱即用，Basilisk 会以**错误**级别启用当前已注册的所有 PEP 标签规则； 可选的项目风格规则另行配置。这描述的是默认配置，并不证明实现完整或正确。 -在受影响规则依据 [Python 类型系统规范](https://typing.python.org/en/latest/spec/index.html) -完成重写并通过独立验证之前，Basilisk 的实际符合性水平暂时未知。请参阅 +我们正在依据 [Python 类型系统规范](https://typing.python.org/en/latest/spec/index.html) +逐条审计规则，并删除那些只匹配源文本、并未做真正类型检查的规则；在此期间， +Basilisk 的实际符合性水平暂时未知。规则只会变少，不会变多。请参阅 [符合性更正](/zh/docs/conformance/)。 ## 第 2 步——修复错误 From 55eead980d28447e5e4d3e739044c6475d406f5d Mon Sep 17 00:00:00 2001 From: Christian Findlay <16697547+MelbourneDeveloper@users.noreply.github.com> Date: Sat, 8 Aug 2026 08:15:49 +1000 Subject: [PATCH 5/9] Skip the code matrix on PRs carrying the docs-only label The change-scope classifier routes by path, so a branch that mixes documentation with an unrelated tooling change still runs the full Rust matrix, the live conformance clone, and the mutation shards. Add an explicit per-PR opt-out: a docs-only label makes the classifier emit every scope as false. A label rather than a path rule keeps the decision visible on the PR and attributable to whoever made it, and leaves every unlabelled PR untouched. --- .github/workflows/ci.yml | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index ec46f9e95..da233c9c0 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -78,8 +78,31 @@ jobs: ACTOR: ${{ github.actor }} BASE_SHA: ${{ github.event.pull_request.base.sha }} HEAD_SHA: ${{ github.event.pull_request.head.sha }} + LABELS: ${{ join(github.event.pull_request.labels.*.name, ',') }} run: | set -euo pipefail + # ── `docs-only` label: explicit, per-PR opt-out ──────────────────── + # A human adds this label when a PR carries nothing the code matrix + # would meaningfully exercise. It is deliberately a LABEL and not a + # path rule: the decision is visible on the PR, attributable to the + # person who made it, and removed by removing the label. No PR without + # it is affected, so no gate is weakened for anybody else. + # NOT for a PR that changes checker behaviour — a green tick here means + # "nobody looked", and the deletion work ahead needs the matrix to run. + case ",$LABELS," in + *,docs-only,*) + { + echo "core=false" + echo "vscode=false" + echo "nvim=false" + echo "zed=false" + echo "code=false" + echo "website=false" + } >> "$GITHUB_OUTPUT" + echo "docs-only label present — skipping every code and website job." + exit 0 + ;; + esac # Dependabot PRs are swept into `dependabot-upgrades` by # dependabot-automerge.yml and never merge into main directly, so the # heavy matrix would only burn minutes on a bump we discard. CI runs From 624d9fd8f117f909f79b82c9bc85efba154937fd Mon Sep 17 00:00:00 2001 From: Christian Findlay <16697547+MelbourneDeveloper@users.noreply.github.com> Date: Sat, 8 Aug 2026 11:57:34 +1000 Subject: [PATCH 6/9] Docs fixes --- CONTRIBUTING.md | 2 +- README-pypi.md | 48 +++- README.md | 48 +++- README.zh.md | 40 +++- basilisk-zed/README.md | 12 +- basilisk-zed/README.zh.md | 12 +- basilisk.nvim/README.md | 12 +- basilisk.nvim/README.zh.md | 12 +- coverage-thresholds.json | 2 +- docs/CONFORMANCE-INTEGRITY-AUDIT.md | 2 + docs/readme/README.src.md | 48 +++- docs/readme/README.zh.src.md | 40 +++- vscode-extension/README.md | 48 +++- vscode-extension/README.zh.md | 40 +++- vscode-extension/package.json | 4 +- website/src/_data/releases.js | 28 ++- ...k-100-percent-python-typing-conformance.md | 213 ++++++++++-------- website/src/docs/conformance.md | 12 +- website/src/docs/releases.njk | 10 +- website/src/index.njk | 5 +- ...k-100-percent-python-typing-conformance.md | 148 ++++++------ website/src/zh/docs/conformance.md | 16 +- website/src/zh/index.njk | 5 +- 23 files changed, 562 insertions(+), 245 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 79c90e937..54bc373c7 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -21,7 +21,7 @@ Basilisk is built by a **human + AI partnership**, split on purpose. AI agents d You don't need to write Rust to make Basilisk better. **The highest-leverage thing a human can do here is verify that the checker actually analyses code** — not that a number went up. -This isn't hypothetical. Checker logic was fitted to the conformance fixtures, the resulting 100% was published, and we didn't catch it until much later; both published numbers — conformance and performance — are now **withdrawn**. See the [conformance correction](https://www.basilisk-python.dev/docs/conformance/) and the [integrity audit](docs/CONFORMANCE-INTEGRITY-AUDIT.md). None of it was deliberate — the instructions named the score as the goal, and matching text moves a score faster than analysing code does. In rough order of impact: +This isn't hypothetical. Checker logic was fitted to the conformance fixtures, the resulting score was published, and we didn't catch it until much later; both published numbers — conformance and performance — are now **withdrawn**. See the [conformance correction](https://www.basilisk-python.dev/docs/conformance/), the [integrity audit](docs/CONFORMANCE-INTEGRITY-AUDIT.md), and the author's [personal account and apology](https://www.christianfindlay.com/blog/basilisk-conformance-apology). None of it was deliberate — nobody set out to game the suite; the instructions named the score as the goal, matching text moves a score faster than analysing code does, and nothing verified the difference. In rough order of impact: ### 1. Verify the metrics yourself diff --git a/README-pypi.md b/README-pypi.md index 6e233abc5..87c753e15 100644 --- a/README-pypi.md +++ b/README-pypi.md @@ -29,32 +29,62 @@ Basilisk in action — type checking, diagnostics, and refactoring in the editor

-**The current type checker contains inaccuracies and you should not use it as part of your dev pipeline. We are working on removing any misleading analyzers ASAP. Please read below** - -## We are auditing the checker and deleting what doesn't hold up - -We withdrew our 100% conformance claim and our benchmark figures, and asked to be +> ## ⚠️ Do not use Basilisk's type checker in your pipeline +> +> **The type checker still contains code that isn't doing real type checking, and +> it is not yet trustworthy.** Some rules decide from the way code is *spelled* +> rather than what it means, so they can be wrong in both directions — a false +> error on correct code, or silence where there is a genuine bug. Until the audit +> below is finished, don't gate CI on `basilisk check`, don't block a merge with +> it, and don't read a clean run as a clean codebase. +> +> The rest of Basilisk — language server, refactoring, formatting, debugging, +> profiling — does not depend on those rules and is unaffected. + +## Restoring trust: audit, delete, and lean on a checker that works + +We withdrew our former conformance claim and our benchmark figures, and asked to be [removed from the official `python/typing` results](https://github.com/python/typing/blob/main/conformance/results/results.html). The cause was checker logic fitted to the contents of conformance test files instead of implementing the typing specification generally: rules that matched the *spelling* of code rather than its meaning. Rename an import or reformat a file and the answer changed. A score produced that way is not evidence. +**This was a mistake and a failure to verify — not an attempt to game the suite.** +Nobody set out to defeat the conformance tests, and nothing was concealed from +`python/typing`: the submission ran the suite's own unmodified harness, with +default configuration and every rule enabled. Our process treated the score as +the goal, matching text raises a score faster than real analysis does, and we +published without ever asking whether a rule still held when the same program was +spelled differently. Basilisk's author has published a +[personal account and apology](https://www.christianfindlay.com/blog/basilisk-conformance-apology). + **So we are auditing every rule and deleting the ones that don't do real type checking.** Not rewriting them, not patching them, not marking them TODO — deleting them, with a failing test left behind so the gap is visible instead of hidden. A rule stays only if it decides from the resolved syntax tree and gives the same answer when the code is spelled differently. +**Where a rule can't be made reliable in a straightforward way, we will depend on +a different, established type checker rather than ship our own unreliable version +of it.** An answer from an engine that has earned trust is worth more to you than +a Basilisk-branded one that hasn't. No replacement figure gets published until it +survives off-suite and mutation testing. + That means Basilisk gets **smaller** before it gets better. Expect fewer rules, fewer diagnostics, and a lower conformance number. We will report each drop rather than avoid it. What is left will be code that is honest about what it does — nothing else. -We have not yet decided whether to rebuild the deleted analysis from the -specification or to drive the extension with an established open-source checker. -Either way, no new figure gets published until it survives off-suite and -mutation testing. +### Basilisk is much more than a type checker + +Type checking is one part of it. The rest is a complete Python workflow in a +single Rust binary — language server, refactoring, formatting, integrated +debugging, profiling, and the editor extensions — and none of it rests on the +rules under audit. That is what we are sharpening while the audit runs: make the +parts that are genuinely useful solid, and remove anything that could hand you a +misleading result. The point of getting smaller is to end up with a tool you can +believe. [Read the full correction →](https://www.basilisk-python.dev/docs/conformance/)  •  [Integrity audit →](https://github.com/Nimblesite/Basilisk/blob/main/docs/CONFORMANCE-INTEGRITY-AUDIT.md) diff --git a/README.md b/README.md index 39ef2d004..2ba2e7a5c 100644 --- a/README.md +++ b/README.md @@ -29,32 +29,62 @@ Basilisk in action — type checking, diagnostics, and refactoring in the editor

-**The current type checker contains inaccuracies and you should not use it as part of your dev pipeline. We are working on removing any misleading analyzers ASAP. Please read below** - -## We are auditing the checker and deleting what doesn't hold up - -We withdrew our 100% conformance claim and our benchmark figures, and asked to be +> ## ⚠️ Do not use Basilisk's type checker in your pipeline +> +> **The type checker still contains code that isn't doing real type checking, and +> it is not yet trustworthy.** Some rules decide from the way code is *spelled* +> rather than what it means, so they can be wrong in both directions — a false +> error on correct code, or silence where there is a genuine bug. Until the audit +> below is finished, don't gate CI on `basilisk check`, don't block a merge with +> it, and don't read a clean run as a clean codebase. +> +> The rest of Basilisk — language server, refactoring, formatting, debugging, +> profiling — does not depend on those rules and is unaffected. + +## Restoring trust: audit, delete, and lean on a checker that works + +We withdrew our former conformance claim and our benchmark figures, and asked to be [removed from the official `python/typing` results](https://github.com/python/typing/blob/main/conformance/results/results.html). The cause was checker logic fitted to the contents of conformance test files instead of implementing the typing specification generally: rules that matched the *spelling* of code rather than its meaning. Rename an import or reformat a file and the answer changed. A score produced that way is not evidence. +**This was a mistake and a failure to verify — not an attempt to game the suite.** +Nobody set out to defeat the conformance tests, and nothing was concealed from +`python/typing`: the submission ran the suite's own unmodified harness, with +default configuration and every rule enabled. Our process treated the score as +the goal, matching text raises a score faster than real analysis does, and we +published without ever asking whether a rule still held when the same program was +spelled differently. Basilisk's author has published a +[personal account and apology](https://www.christianfindlay.com/blog/basilisk-conformance-apology). + **So we are auditing every rule and deleting the ones that don't do real type checking.** Not rewriting them, not patching them, not marking them TODO — deleting them, with a failing test left behind so the gap is visible instead of hidden. A rule stays only if it decides from the resolved syntax tree and gives the same answer when the code is spelled differently. +**Where a rule can't be made reliable in a straightforward way, we will depend on +a different, established type checker rather than ship our own unreliable version +of it.** An answer from an engine that has earned trust is worth more to you than +a Basilisk-branded one that hasn't. No replacement figure gets published until it +survives off-suite and mutation testing. + That means Basilisk gets **smaller** before it gets better. Expect fewer rules, fewer diagnostics, and a lower conformance number. We will report each drop rather than avoid it. What is left will be code that is honest about what it does — nothing else. -We have not yet decided whether to rebuild the deleted analysis from the -specification or to drive the extension with an established open-source checker. -Either way, no new figure gets published until it survives off-suite and -mutation testing. +### Basilisk is much more than a type checker + +Type checking is one part of it. The rest is a complete Python workflow in a +single Rust binary — language server, refactoring, formatting, integrated +debugging, profiling, and the editor extensions — and none of it rests on the +rules under audit. That is what we are sharpening while the audit runs: make the +parts that are genuinely useful solid, and remove anything that could hand you a +misleading result. The point of getting smaller is to end up with a tool you can +believe. [Read the full correction →](https://www.basilisk-python.dev/docs/conformance/)  •  [Integrity audit →](docs/CONFORMANCE-INTEGRITY-AUDIT.md) diff --git a/README.zh.md b/README.zh.md index 158aa11fd..db0fb4f4e 100644 --- a/README.zh.md +++ b/README.zh.md @@ -29,27 +29,53 @@ Basilisk 实战 —— 编辑器中的类型检查、诊断与重构

-**当前的类型检查器存在不准确之处，请勿将其用于你的开发流水线。我们正在尽快移除任何具有误导性的分析器。详情请见下文** - -## 我们正在审计检查器，并删除站不住脚的代码 - -我们撤回了 100% 的一致性宣称与基准测试数字，并主动请求 +> ## ⚠️ 请勿在流水线中使用 Basilisk 的类型检查器 +> +> **类型检查器中仍然存在没有做真正类型检查的代码，它目前还不值得信任。** 有些规则 +> 依据的是代码的**写法**而不是含义，因此两个方向上都可能出错 —— 既可能对正确的代码 +> 报出虚假错误，也可能对真实的缺陷保持沉默。在下文所述的审计完成之前，请不要用 +> `basilisk check` 作为 CI 的门禁，不要用它拦截合并，也不要把一次干净的运行结果当作 +> 代码库是干净的。 +> +> Basilisk 的其余部分 —— 语言服务器、重构、格式化、调试、性能分析 —— 并不依赖这些 +> 规则，因此不受影响。 + +## 重建信任：审计、删除，并倚重真正可靠的检查器 + +我们撤回了此前的一致性宣称与基准测试数字，并主动请求 [从官方 `python/typing` 结果中移除](https://github.com/python/typing/blob/main/conformance/results/results.html)。 原因是检查器中存在针对一致性测试文件内容而写的逻辑，而不是对类型规范的通用实现： 那些规则匹配的是代码的**写法**，而不是代码的含义。改一个导入别名或重新格式化文件， 结论就会变。这样得出的分数并不能作为证据。 +**这是一个错误、一次验证上的失职，而不是有意操纵测试套件。** 没有人试图去糊弄一致性 +测试，也没有向 `python/typing` 隐瞒任何东西：提交时运行的是套件自己未经修改的评分 +工具，使用默认配置并启用全部规则。我们的流程把分数当成了目标，而匹配文本比真正做 +分析更快地提高分数；我们在发布之前，始终没有问过这样一个问题 —— 同一个程序换一种 +写法时，这条规则是否依然成立。Basilisk 作者已发表 +[个人说明与致歉](https://www.christianfindlay.com/blog/basilisk-conformance-apology)。 + **因此，我们正在逐条审计规则，并删除那些没有做真正类型检查的规则。** 不是重写，不是 打补丁，也不是标一个 TODO —— 是删除，并留下一个失败的测试，让这个缺口可见而不是被 掩盖。一条规则只有在依据已解析的语法树做判断、并且在代码换一种写法时给出相同结论的 情况下，才会保留。 +**如果一条规则无法以直截了当的方式做到可靠，我们会转而依赖另一个成熟的类型检查器， +而不是端出我们自己那份不可靠的实现。** 一个已经赢得信任的引擎给出的答案，对你而言 +比一个挂着 Basilisk 名号却没有赢得信任的答案更有价值。在通过套件之外的用例与变异 +测试之前，我们不会发布任何替代数字。 + 这意味着 Basilisk 会**先变小，再变好**。规则会更少，诊断会更少，一致性数字也会更低。 每一次下降我们都会如实报告，而不是设法回避。留下来的，将是对自己所做之事诚实的代码 —— 仅此而已。 -至于被删掉的分析是按规范重建，还是让扩展由一个成熟的开源检查器驱动，我们尚未决定。 -无论走哪条路，新的数字只有在经受住套件之外的用例与变异测试后才会发布。 +### Basilisk 远不只是一个类型检查器 + +类型检查只是其中一部分。其余部分是装在单个 Rust 二进制文件里的完整 Python 工作流 +—— 语言服务器、重构、格式化、集成调试、性能分析，以及各个编辑器扩展 —— 它们都不 +建立在正在接受审计的规则之上。这正是我们在审计期间着力打磨的地方：把真正有用的部分 +做扎实，并移除任何可能给出误导性结果的东西。变小的意义，是最终得到一个你可以信赖的 +工具。 [阅读完整更正 →](https://www.basilisk-python.dev/zh/docs/conformance/)  •  [完整性审计 →](docs/CONFORMANCE-INTEGRITY-AUDIT.md) diff --git a/basilisk-zed/README.md b/basilisk-zed/README.md index 18dc2fb0e..c2a6de9bb 100644 --- a/basilisk-zed/README.md +++ b/basilisk-zed/README.md @@ -10,9 +10,15 @@ Basilisk is an open-source Python type checker and language server built in Rust Basilisk in the Zed editor — Python type checking and diagnostics inline

-**The current type checker contains inaccuracies and you should not use it as part of your dev pipeline. We are working on removing any misleading analyzers ASAP. Please read below** - -> **We withdrew the typing conformance results.** The 100% claim and the benchmark figures are retracted, and Basilisk was [removed from the official results](https://github.com/python/typing/blob/main/conformance/results/results.html) at our request. Whether we rebuild the checker from the specification or drive the extension with an established open-source checker, we are building Basilisk into an accurate Python development experience. [Read the correction](https://www.basilisk-python.dev/docs/conformance/). +> ## ⚠️ Do not use Basilisk's type checker in your pipeline +> +> **The type checker still contains code that isn't doing real type checking, and it is not yet trustworthy.** Some rules decide from the way code is *spelled* rather than what it means, so they can be wrong in both directions — a false error on correct code, or silence where there is a genuine bug. Don't gate CI on it, and don't read a clean run as a clean codebase. Our former conformance claim and our benchmark figures are withdrawn, and Basilisk was [removed from the official results](https://github.com/python/typing/blob/main/conformance/results/results.html) at our request. +> +> **This was a mistake and a failure to verify — not an attempt to game the suite.** Nothing was concealed from `python/typing`; the submission ran the suite's own unmodified harness, and we published on a green run without ever checking whether our rules survived a semantics-preserving change. Basilisk's author has published a [personal account and apology](https://www.christianfindlay.com/blog/basilisk-conformance-apology). +> +> **We are auditing every rule and deleting the ones that don't hold up** — not rewriting them, not patching them, with a failing test left behind so the gap stays visible. Where a rule can't be made reliable in a straightforward way, we will depend on a different, established type checker rather than ship our own unreliable version of it. +> +> **Basilisk is much more than a type checker.** The language server, refactoring, formatting, debugging, and profiling don't rest on the rules under audit — those are what we are sharpening while it runs, removing anything that could hand you a misleading result. We are doing this to restore trust and turn Basilisk back into a tool you can believe. [Read the correction](https://www.basilisk-python.dev/docs/conformance/). ## Install diff --git a/basilisk-zed/README.zh.md b/basilisk-zed/README.zh.md index 0b479daa4..fdbf7c810 100644 --- a/basilisk-zed/README.zh.md +++ b/basilisk-zed/README.zh.md @@ -12,9 +12,15 @@ Basilisk 是用 Rust 打造的开源 Python 类型检查器与语言服务器： Zed 编辑器中的 Basilisk —— 行内 Python 类型检查与诊断

-**当前的类型检查器存在不准确之处，请勿将其用于你的开发流水线。我们正在尽快移除任何具有误导性的分析器。详情请见下文** - -> **我们撤回了类型一致性结果。** 100% 的宣称与基准测试数字均已撤回，并主动请求[从官方结果中移除](https://github.com/python/typing/blob/main/conformance/results/results.html)。无论是按规范重建检查器，还是让扩展由一个成熟的开源检查器驱动，我们都在把 Basilisk 打造成准确的 Python 开发体验。[阅读更正](https://www.basilisk-python.dev/zh/docs/conformance/)。 +> ## ⚠️ 请勿在流水线中使用 Basilisk 的类型检查器 +> +> **类型检查器中仍然存在没有做真正类型检查的代码，它目前还不值得信任。** 有些规则依据的是代码的**写法**而不是含义，因此两个方向上都可能出错 —— 既可能对正确的代码报出虚假错误，也可能对真实的缺陷保持沉默。请不要用它作为 CI 的门禁，也不要把一次干净的运行结果当作代码库是干净的。此前的一致性宣称与基准测试数字均已撤回，并主动请求[从官方结果中移除](https://github.com/python/typing/blob/main/conformance/results/results.html)。 +> +> **这是一个错误、一次验证上的失职，而不是有意操纵测试套件。** 我们没有向 `python/typing` 隐瞒任何东西：提交时运行的是套件自己未经修改的评分工具；我们仅凭一次全绿的运行就发布了结果，却从未检查过我们的规则能否经受住保持语义的改写。Basilisk 作者已发表[个人说明与致歉](https://www.christianfindlay.com/blog/basilisk-conformance-apology)。 +> +> **我们正在逐条审计规则，并删除那些站不住脚的规则** —— 不是重写，也不是打补丁，而是删除，并留下一个失败的测试，让缺口保持可见。如果一条规则无法以直截了当的方式做到可靠，我们会转而依赖另一个成熟的类型检查器，而不是端出我们自己那份不可靠的实现。 +> +> **Basilisk 远不只是一个类型检查器。** 语言服务器、重构、格式化、调试与性能分析都不建立在正在接受审计的规则之上 —— 审计期间，这些正是我们着力打磨的部分，并移除任何可能给出误导性结果的东西。我们这样做，是为了重建信任，把 Basilisk 变回一个你可以信赖的工具。[阅读更正](https://www.basilisk-python.dev/zh/docs/conformance/)。 ## 安装 diff --git a/basilisk.nvim/README.md b/basilisk.nvim/README.md index 10652da4e..0c75c310c 100644 --- a/basilisk.nvim/README.md +++ b/basilisk.nvim/README.md @@ -10,9 +10,15 @@ Basilisk is an open-source Python type checker and language server built in Rust Basilisk in action — type checking, diagnostics, and refactoring in the editor

-**The current type checker contains inaccuracies and you should not use it as part of your dev pipeline. We are working on removing any misleading analyzers ASAP. Please read below** - -> **We withdrew the typing conformance results.** The 100% claim and the benchmark figures are retracted, and Basilisk was [removed from the official results](https://github.com/python/typing/blob/main/conformance/results/results.html) at our request. Whether we rebuild the checker from the specification or drive the extension with an established open-source checker, we are building Basilisk into an accurate Python development experience. [Read the correction](https://www.basilisk-python.dev/docs/conformance/). +> ## ⚠️ Do not use Basilisk's type checker in your pipeline +> +> **The type checker still contains code that isn't doing real type checking, and it is not yet trustworthy.** Some rules decide from the way code is *spelled* rather than what it means, so they can be wrong in both directions — a false error on correct code, or silence where there is a genuine bug. Don't gate CI on it, and don't read a clean run as a clean codebase. Our former conformance claim and our benchmark figures are withdrawn, and Basilisk was [removed from the official results](https://github.com/python/typing/blob/main/conformance/results/results.html) at our request. +> +> **This was a mistake and a failure to verify — not an attempt to game the suite.** Nothing was concealed from `python/typing`; the submission ran the suite's own unmodified harness, and we published on a green run without ever checking whether our rules survived a semantics-preserving change. Basilisk's author has published a [personal account and apology](https://www.christianfindlay.com/blog/basilisk-conformance-apology). +> +> **We are auditing every rule and deleting the ones that don't hold up** — not rewriting them, not patching them, with a failing test left behind so the gap stays visible. Where a rule can't be made reliable in a straightforward way, we will depend on a different, established type checker rather than ship our own unreliable version of it. +> +> **Basilisk is much more than a type checker.** The language server, refactoring, formatting, debugging, and profiling don't rest on the rules under audit — those are what we are sharpening while it runs, removing anything that could hand you a misleading result. We are doing this to restore trust and turn Basilisk back into a tool you can believe. [Read the correction](https://www.basilisk-python.dev/docs/conformance/). ## Role in Basilisk diff --git a/basilisk.nvim/README.zh.md b/basilisk.nvim/README.zh.md index b6cc8865c..f128a4a12 100644 --- a/basilisk.nvim/README.zh.md +++ b/basilisk.nvim/README.zh.md @@ -12,9 +12,15 @@ Basilisk 是用 Rust 打造的开源 Python 类型检查器与语言服务器： Basilisk in action — type checking, diagnostics, and refactoring in the editor

-**当前的类型检查器存在不准确之处，请勿将其用于你的开发流水线。我们正在尽快移除任何具有误导性的分析器。详情请见下文** - -> **我们撤回了类型一致性结果。** 100% 的宣称与基准测试数字均已撤回，并主动请求[从官方结果中移除](https://github.com/python/typing/blob/main/conformance/results/results.html)。无论是按规范重建检查器，还是让扩展由一个成熟的开源检查器驱动，我们都在把 Basilisk 打造成准确的 Python 开发体验。[阅读更正](https://www.basilisk-python.dev/zh/docs/conformance/)。 +> ## ⚠️ 请勿在流水线中使用 Basilisk 的类型检查器 +> +> **类型检查器中仍然存在没有做真正类型检查的代码，它目前还不值得信任。** 有些规则依据的是代码的**写法**而不是含义，因此两个方向上都可能出错 —— 既可能对正确的代码报出虚假错误，也可能对真实的缺陷保持沉默。请不要用它作为 CI 的门禁，也不要把一次干净的运行结果当作代码库是干净的。此前的一致性宣称与基准测试数字均已撤回，并主动请求[从官方结果中移除](https://github.com/python/typing/blob/main/conformance/results/results.html)。 +> +> **这是一个错误、一次验证上的失职，而不是有意操纵测试套件。** 我们没有向 `python/typing` 隐瞒任何东西：提交时运行的是套件自己未经修改的评分工具；我们仅凭一次全绿的运行就发布了结果，却从未检查过我们的规则能否经受住保持语义的改写。Basilisk 作者已发表[个人说明与致歉](https://www.christianfindlay.com/blog/basilisk-conformance-apology)。 +> +> **我们正在逐条审计规则，并删除那些站不住脚的规则** —— 不是重写，也不是打补丁，而是删除，并留下一个失败的测试，让缺口保持可见。如果一条规则无法以直截了当的方式做到可靠，我们会转而依赖另一个成熟的类型检查器，而不是端出我们自己那份不可靠的实现。 +> +> **Basilisk 远不只是一个类型检查器。** 语言服务器、重构、格式化、调试与性能分析都不建立在正在接受审计的规则之上 —— 审计期间，这些正是我们着力打磨的部分，并移除任何可能给出误导性结果的东西。我们这样做，是为了重建信任，把 Basilisk 变回一个你可以信赖的工具。[阅读更正](https://www.basilisk-python.dev/zh/docs/conformance/)。 ## 在 Basilisk 中的角色 diff --git a/coverage-thresholds.json b/coverage-thresholds.json index 00a4431dd..61dae656f 100644 --- a/coverage-thresholds.json +++ b/coverage-thresholds.json @@ -35,7 +35,7 @@ } }, "conformance": { - "_doc": "Live PEP conformance measurement. POLICY: the number is a REGRESSION DETECTOR, never a target \u2014 see [CHKARCH-CONFORMANCE]. It samples one fixed corpus the checker was historically developed against, so it cannot tell you whether a rule analyses code; only whether today's binary agrees with yesterday's on files it has already seen. NEVER publish, quote, or market this figure. conformance/run_conformance.py clones python/typing@main FRESH on every run, then runs the suite's OWN unmodified src/main.py --only-run basilisk against the compiled binary \u2014 whose conformance/src/type_checker.py already ships the official BasiliskTypeChecker \u2014 and records the exact graded commit in website/src/_data/conformance_report.json. The result is the REAL harness's OWN verdict, produced by the same code that grades pyright/mypy/pyrefly/ty/zuban/pycroscope: a file passes only when its errors_diff is empty, counting every diagnostic the binary emits (errors AND warnings). There is NO vendored calculator and NO cached-fixtures fallback \u2014 if the real harness cannot be cloned and run, the build FAILS. The binary runs in its default configuration \u2014 the pure PEP set; Basilisk's opt-in house-style rules never run (see [CHKARCH-CONFIGURATION-ONLY]). Configuring a rule off before measuring, hand-editing conformance_status.csv, or editing this block to match a run is forbidden. KNOWN CONTRADICTION: `threshold` below is a pass-percentage floor, and a floor over a corpus the code was fitted to is the incentive that produced the fitted predicates (CONFORMANCE-INTEGRITY-AUDIT \u00a76.3). Deleting a rule that decides from source text rather than resolved symbols is REQUIRED ([CHKARCH-TEXT-MATCHED-LOGIC]) and is expected to LOWER this number \u2014 which this floor turns into a build failure. Removing the floor is the user's decision, not an agent's. Until they decide: make the deletion, report the drop and the failing gate, and stop there. Do NOT restore the code, refit the rule, or lower this value to get green.", + "_doc": "DEAD AS OF 2026-08-08: this measurement CANNOT RUN. Basilisk's withdrawal from python/typing removed BasiliskTypeChecker from the suite's conformance/src/type_checker.py, so the upstream harness no longer accepts `--only-run basilisk` (`invalid choice: 'basilisk'`) and run_conformance.py exits 2. That fails the release workflow's conformance job, every job that needs it, and `make test`. Restoring it would require vendoring a scorer or injecting an adapter, which [CHKARCH-CONFORMANCE] declares a BUILD FAILURE. Whether this block is deleted outright or replaced by a disclosed non-official harness is the user's decision, not an agent's. The rest of this note describes the mechanism as it worked while upstream still carried the adapter. Live PEP conformance measurement. POLICY: the number is a REGRESSION DETECTOR, never a target \u2014 see [CHKARCH-CONFORMANCE]. It samples one fixed corpus the checker was historically developed against, so it cannot tell you whether a rule analyses code; only whether today's binary agrees with yesterday's on files it has already seen. NEVER publish, quote, or market this figure. conformance/run_conformance.py clones python/typing@main FRESH on every run, then runs the suite's OWN unmodified src/main.py --only-run basilisk against the compiled binary \u2014 via the adapter the suite USED to carry \u2014 and records the exact graded commit in website/src/_data/conformance_report.json. The result is the REAL harness's OWN verdict, produced by the same code that grades pyright/mypy/pyrefly/ty/zuban/pycroscope: a file passes only when its errors_diff is empty, counting every diagnostic the binary emits (errors AND warnings). There is NO vendored calculator and NO cached-fixtures fallback \u2014 if the real harness cannot be cloned and run, the build FAILS. The binary runs in its default configuration \u2014 the pure PEP set; Basilisk's opt-in house-style rules never run (see [CHKARCH-CONFIGURATION-ONLY]). Configuring a rule off before measuring, hand-editing conformance_status.csv, or editing this block to match a run is forbidden. KNOWN CONTRADICTION: `threshold` below is a pass-percentage floor, and a floor over a corpus the code was fitted to is the incentive that produced the fitted predicates (CONFORMANCE-INTEGRITY-AUDIT \u00a76.3). Deleting a rule that decides from source text rather than resolved symbols is REQUIRED ([CHKARCH-TEXT-MATCHED-LOGIC]) and is expected to LOWER this number \u2014 which this floor turns into a build failure. Removing the floor is the user's decision, not an agent's. Until they decide: make the deletion, report the drop and the failing gate, and stop there. Do NOT restore the code, refit the rule, or lower this value to get green.", "threshold": 100, "_fp_ceiling_doc": "Total false-positive diagnostics across the suite (diagnostics Basilisk reports on a line the suite does NOT mark # E, or outside a satisfied # E[tag] group). Measured by conformance/run_conformance.py --gate, which runs the REAL python/typing harness on the compiled binary and delegates the comparison to conformance/assert_wheel_conformance.py (run by scripts/test-rust.sh inside make test). A false positive on real code is a genuine defect worth fixing on its own merits, independent of this suite. Same contradiction as `threshold` above: close a gap by fixing the checker or by deleting logic that never analysed anything \u2014 never by silencing a rule to hold the ceiling.", "max_false_positives": 0 diff --git a/docs/CONFORMANCE-INTEGRITY-AUDIT.md b/docs/CONFORMANCE-INTEGRITY-AUDIT.md index 936bb6904..d8b560069 100644 --- a/docs/CONFORMANCE-INTEGRITY-AUDIT.md +++ b/docs/CONFORMANCE-INTEGRITY-AUDIT.md @@ -14,6 +14,8 @@ We found that at least one Basilisk rule earns its conformance result by pattern We did not find this ourselves. It was reported from outside, in [issue #379](https://github.com/Nimblesite/Basilisk/issues/379), from a [public reproduction](https://x.com/cyanchanges/status/2083115048143364512). That is itself a finding, and it is covered in §6. +**On intent.** This was a verification failure, not an attempt to game the conformance suite. Nobody set out to defeat it, and nothing was concealed from `python/typing`: the submission ran the suite's own unmodified harness, with default configuration and every specification rule enabled — no special-casing of the harness, no doctored output, no disabled rules, no edited results file. What went wrong is set out in §6: the process named the score as the target, matching source text raises a score faster than real analysis does, and the control that would have caught the drift — semantics-preserving mutation (§7) — did not exist, so every green run reinforced a conclusion we had no basis for. When the defect was demonstrated we accepted it and requested our own removal from the results table. Basilisk's author has published a [personal account and apology](https://www.christianfindlay.com/blog/basilisk-conformance-apology). This document describes defective engineering and a failure of review; it does not describe deception, and nothing in it should be read as alleging any. + Our conformance number is self-measured. Where a passing file is carried by predicates shaped to that file, the honest statement is that **the file passes and the rule is not implemented**. That is the case for the files listed in §3. --- diff --git a/docs/readme/README.src.md b/docs/readme/README.src.md index 7a114fa02..26348731d 100644 --- a/docs/readme/README.src.md +++ b/docs/readme/README.src.md @@ -40,32 +40,62 @@ Basilisk in action — type checking, diagnostics, and refactoring in the editor

-**The current type checker contains inaccuracies and you should not use it as part of your dev pipeline. We are working on removing any misleading analyzers ASAP. Please read below** - -## We are auditing the checker and deleting what doesn't hold up - -We withdrew our 100% conformance claim and our benchmark figures, and asked to be +> ## ⚠️ Do not use Basilisk's type checker in your pipeline +> +> **The type checker still contains code that isn't doing real type checking, and +> it is not yet trustworthy.** Some rules decide from the way code is *spelled* +> rather than what it means, so they can be wrong in both directions — a false +> error on correct code, or silence where there is a genuine bug. Until the audit +> below is finished, don't gate CI on `basilisk check`, don't block a merge with +> it, and don't read a clean run as a clean codebase. +> +> The rest of Basilisk — language server, refactoring, formatting, debugging, +> profiling — does not depend on those rules and is unaffected. + +## Restoring trust: audit, delete, and lean on a checker that works + +We withdrew our former conformance claim and our benchmark figures, and asked to be [removed from the official `python/typing` results](https://github.com/python/typing/blob/main/conformance/results/results.html). The cause was checker logic fitted to the contents of conformance test files instead of implementing the typing specification generally: rules that matched the *spelling* of code rather than its meaning. Rename an import or reformat a file and the answer changed. A score produced that way is not evidence. +**This was a mistake and a failure to verify — not an attempt to game the suite.** +Nobody set out to defeat the conformance tests, and nothing was concealed from +`python/typing`: the submission ran the suite's own unmodified harness, with +default configuration and every rule enabled. Our process treated the score as +the goal, matching text raises a score faster than real analysis does, and we +published without ever asking whether a rule still held when the same program was +spelled differently. Basilisk's author has published a +[personal account and apology](https://www.christianfindlay.com/blog/basilisk-conformance-apology). + **So we are auditing every rule and deleting the ones that don't do real type checking.** Not rewriting them, not patching them, not marking them TODO — deleting them, with a failing test left behind so the gap is visible instead of hidden. A rule stays only if it decides from the resolved syntax tree and gives the same answer when the code is spelled differently. +**Where a rule can't be made reliable in a straightforward way, we will depend on +a different, established type checker rather than ship our own unreliable version +of it.** An answer from an engine that has earned trust is worth more to you than +a Basilisk-branded one that hasn't. No replacement figure gets published until it +survives off-suite and mutation testing. + That means Basilisk gets **smaller** before it gets better. Expect fewer rules, fewer diagnostics, and a lower conformance number. We will report each drop rather than avoid it. What is left will be code that is honest about what it does — nothing else. -We have not yet decided whether to rebuild the deleted analysis from the -specification or to drive the extension with an established open-source checker. -Either way, no new figure gets published until it survives off-suite and -mutation testing. +### Basilisk is much more than a type checker + +Type checking is one part of it. The rest is a complete Python workflow in a +single Rust binary — language server, refactoring, formatting, integrated +debugging, profiling, and the editor extensions — and none of it rests on the +rules under audit. That is what we are sharpening while the audit runs: make the +parts that are genuinely useful solid, and remove anything that could hand you a +misleading result. The point of getting smaller is to end up with a tool you can +believe. [Read the full correction →](https://www.basilisk-python.dev/docs/conformance/)  •  [Integrity audit →](docs/CONFORMANCE-INTEGRITY-AUDIT.md) diff --git a/docs/readme/README.zh.src.md b/docs/readme/README.zh.src.md index 27b009f44..5cc9ef96f 100644 --- a/docs/readme/README.zh.src.md +++ b/docs/readme/README.zh.src.md @@ -36,27 +36,53 @@ Basilisk 实战 —— 编辑器中的类型检查、诊断与重构

-**当前的类型检查器存在不准确之处，请勿将其用于你的开发流水线。我们正在尽快移除任何具有误导性的分析器。详情请见下文** - -## 我们正在审计检查器，并删除站不住脚的代码 - -我们撤回了 100% 的一致性宣称与基准测试数字，并主动请求 +> ## ⚠️ 请勿在流水线中使用 Basilisk 的类型检查器 +> +> **类型检查器中仍然存在没有做真正类型检查的代码，它目前还不值得信任。** 有些规则 +> 依据的是代码的**写法**而不是含义，因此两个方向上都可能出错 —— 既可能对正确的代码 +> 报出虚假错误，也可能对真实的缺陷保持沉默。在下文所述的审计完成之前，请不要用 +> `basilisk check` 作为 CI 的门禁，不要用它拦截合并，也不要把一次干净的运行结果当作 +> 代码库是干净的。 +> +> Basilisk 的其余部分 —— 语言服务器、重构、格式化、调试、性能分析 —— 并不依赖这些 +> 规则，因此不受影响。 + +## 重建信任：审计、删除，并倚重真正可靠的检查器 + +我们撤回了此前的一致性宣称与基准测试数字，并主动请求 [从官方 `python/typing` 结果中移除](https://github.com/python/typing/blob/main/conformance/results/results.html)。 原因是检查器中存在针对一致性测试文件内容而写的逻辑，而不是对类型规范的通用实现： 那些规则匹配的是代码的**写法**，而不是代码的含义。改一个导入别名或重新格式化文件， 结论就会变。这样得出的分数并不能作为证据。 +**这是一个错误、一次验证上的失职，而不是有意操纵测试套件。** 没有人试图去糊弄一致性 +测试，也没有向 `python/typing` 隐瞒任何东西：提交时运行的是套件自己未经修改的评分 +工具，使用默认配置并启用全部规则。我们的流程把分数当成了目标，而匹配文本比真正做 +分析更快地提高分数；我们在发布之前，始终没有问过这样一个问题 —— 同一个程序换一种 +写法时，这条规则是否依然成立。Basilisk 作者已发表 +[个人说明与致歉](https://www.christianfindlay.com/blog/basilisk-conformance-apology)。 + **因此，我们正在逐条审计规则，并删除那些没有做真正类型检查的规则。** 不是重写，不是 打补丁，也不是标一个 TODO —— 是删除，并留下一个失败的测试，让这个缺口可见而不是被 掩盖。一条规则只有在依据已解析的语法树做判断、并且在代码换一种写法时给出相同结论的 情况下，才会保留。 +**如果一条规则无法以直截了当的方式做到可靠，我们会转而依赖另一个成熟的类型检查器， +而不是端出我们自己那份不可靠的实现。** 一个已经赢得信任的引擎给出的答案，对你而言 +比一个挂着 Basilisk 名号却没有赢得信任的答案更有价值。在通过套件之外的用例与变异 +测试之前，我们不会发布任何替代数字。 + 这意味着 Basilisk 会**先变小，再变好**。规则会更少，诊断会更少，一致性数字也会更低。 每一次下降我们都会如实报告，而不是设法回避。留下来的，将是对自己所做之事诚实的代码 —— 仅此而已。 -至于被删掉的分析是按规范重建，还是让扩展由一个成熟的开源检查器驱动，我们尚未决定。 -无论走哪条路，新的数字只有在经受住套件之外的用例与变异测试后才会发布。 +### Basilisk 远不只是一个类型检查器 + +类型检查只是其中一部分。其余部分是装在单个 Rust 二进制文件里的完整 Python 工作流 +—— 语言服务器、重构、格式化、集成调试、性能分析，以及各个编辑器扩展 —— 它们都不 +建立在正在接受审计的规则之上。这正是我们在审计期间着力打磨的地方：把真正有用的部分 +做扎实，并移除任何可能给出误导性结果的东西。变小的意义，是最终得到一个你可以信赖的 +工具。 [阅读完整更正 →](https://www.basilisk-python.dev/zh/docs/conformance/)  •  [完整性审计 →](docs/CONFORMANCE-INTEGRITY-AUDIT.md) diff --git a/vscode-extension/README.md b/vscode-extension/README.md index 7de376698..c7c5432a4 100644 --- a/vscode-extension/README.md +++ b/vscode-extension/README.md @@ -29,32 +29,62 @@ Basilisk in action — type checking, diagnostics, and refactoring in the editor

-**The current type checker contains inaccuracies and you should not use it as part of your dev pipeline. We are working on removing any misleading analyzers ASAP. Please read below** - -## We are auditing the checker and deleting what doesn't hold up - -We withdrew our 100% conformance claim and our benchmark figures, and asked to be +> ## ⚠️ Do not use Basilisk's type checker in your pipeline +> +> **The type checker still contains code that isn't doing real type checking, and +> it is not yet trustworthy.** Some rules decide from the way code is *spelled* +> rather than what it means, so they can be wrong in both directions — a false +> error on correct code, or silence where there is a genuine bug. Until the audit +> below is finished, don't gate CI on `basilisk check`, don't block a merge with +> it, and don't read a clean run as a clean codebase. +> +> The rest of Basilisk — language server, refactoring, formatting, debugging, +> profiling — does not depend on those rules and is unaffected. + +## Restoring trust: audit, delete, and lean on a checker that works + +We withdrew our former conformance claim and our benchmark figures, and asked to be [removed from the official `python/typing` results](https://github.com/python/typing/blob/main/conformance/results/results.html). The cause was checker logic fitted to the contents of conformance test files instead of implementing the typing specification generally: rules that matched the *spelling* of code rather than its meaning. Rename an import or reformat a file and the answer changed. A score produced that way is not evidence. +**This was a mistake and a failure to verify — not an attempt to game the suite.** +Nobody set out to defeat the conformance tests, and nothing was concealed from +`python/typing`: the submission ran the suite's own unmodified harness, with +default configuration and every rule enabled. Our process treated the score as +the goal, matching text raises a score faster than real analysis does, and we +published without ever asking whether a rule still held when the same program was +spelled differently. Basilisk's author has published a +[personal account and apology](https://www.christianfindlay.com/blog/basilisk-conformance-apology). + **So we are auditing every rule and deleting the ones that don't do real type checking.** Not rewriting them, not patching them, not marking them TODO — deleting them, with a failing test left behind so the gap is visible instead of hidden. A rule stays only if it decides from the resolved syntax tree and gives the same answer when the code is spelled differently. +**Where a rule can't be made reliable in a straightforward way, we will depend on +a different, established type checker rather than ship our own unreliable version +of it.** An answer from an engine that has earned trust is worth more to you than +a Basilisk-branded one that hasn't. No replacement figure gets published until it +survives off-suite and mutation testing. + That means Basilisk gets **smaller** before it gets better. Expect fewer rules, fewer diagnostics, and a lower conformance number. We will report each drop rather than avoid it. What is left will be code that is honest about what it does — nothing else. -We have not yet decided whether to rebuild the deleted analysis from the -specification or to drive the extension with an established open-source checker. -Either way, no new figure gets published until it survives off-suite and -mutation testing. +### Basilisk is much more than a type checker + +Type checking is one part of it. The rest is a complete Python workflow in a +single Rust binary — language server, refactoring, formatting, integrated +debugging, profiling, and the editor extensions — and none of it rests on the +rules under audit. That is what we are sharpening while the audit runs: make the +parts that are genuinely useful solid, and remove anything that could hand you a +misleading result. The point of getting smaller is to end up with a tool you can +believe. [Read the full correction →](https://www.basilisk-python.dev/docs/conformance/)  •  [Integrity audit →](https://github.com/Nimblesite/Basilisk/blob/main/docs/CONFORMANCE-INTEGRITY-AUDIT.md) diff --git a/vscode-extension/README.zh.md b/vscode-extension/README.zh.md index efd80bb95..41ab71d58 100644 --- a/vscode-extension/README.zh.md +++ b/vscode-extension/README.zh.md @@ -29,27 +29,53 @@ Basilisk 实战 —— 编辑器中的类型检查、诊断与重构

-**当前的类型检查器存在不准确之处，请勿将其用于你的开发流水线。我们正在尽快移除任何具有误导性的分析器。详情请见下文** - -## 我们正在审计检查器，并删除站不住脚的代码 - -我们撤回了 100% 的一致性宣称与基准测试数字，并主动请求 +> ## ⚠️ 请勿在流水线中使用 Basilisk 的类型检查器 +> +> **类型检查器中仍然存在没有做真正类型检查的代码，它目前还不值得信任。** 有些规则 +> 依据的是代码的**写法**而不是含义，因此两个方向上都可能出错 —— 既可能对正确的代码 +> 报出虚假错误，也可能对真实的缺陷保持沉默。在下文所述的审计完成之前，请不要用 +> `basilisk check` 作为 CI 的门禁，不要用它拦截合并，也不要把一次干净的运行结果当作 +> 代码库是干净的。 +> +> Basilisk 的其余部分 —— 语言服务器、重构、格式化、调试、性能分析 —— 并不依赖这些 +> 规则，因此不受影响。 + +## 重建信任：审计、删除，并倚重真正可靠的检查器 + +我们撤回了此前的一致性宣称与基准测试数字，并主动请求 [从官方 `python/typing` 结果中移除](https://github.com/python/typing/blob/main/conformance/results/results.html)。 原因是检查器中存在针对一致性测试文件内容而写的逻辑，而不是对类型规范的通用实现： 那些规则匹配的是代码的**写法**，而不是代码的含义。改一个导入别名或重新格式化文件， 结论就会变。这样得出的分数并不能作为证据。 +**这是一个错误、一次验证上的失职，而不是有意操纵测试套件。** 没有人试图去糊弄一致性 +测试，也没有向 `python/typing` 隐瞒任何东西：提交时运行的是套件自己未经修改的评分 +工具，使用默认配置并启用全部规则。我们的流程把分数当成了目标，而匹配文本比真正做 +分析更快地提高分数；我们在发布之前，始终没有问过这样一个问题 —— 同一个程序换一种 +写法时，这条规则是否依然成立。Basilisk 作者已发表 +[个人说明与致歉](https://www.christianfindlay.com/blog/basilisk-conformance-apology)。 + **因此，我们正在逐条审计规则，并删除那些没有做真正类型检查的规则。** 不是重写，不是 打补丁，也不是标一个 TODO —— 是删除，并留下一个失败的测试，让这个缺口可见而不是被 掩盖。一条规则只有在依据已解析的语法树做判断、并且在代码换一种写法时给出相同结论的 情况下，才会保留。 +**如果一条规则无法以直截了当的方式做到可靠，我们会转而依赖另一个成熟的类型检查器， +而不是端出我们自己那份不可靠的实现。** 一个已经赢得信任的引擎给出的答案，对你而言 +比一个挂着 Basilisk 名号却没有赢得信任的答案更有价值。在通过套件之外的用例与变异 +测试之前，我们不会发布任何替代数字。 + 这意味着 Basilisk 会**先变小，再变好**。规则会更少，诊断会更少，一致性数字也会更低。 每一次下降我们都会如实报告，而不是设法回避。留下来的，将是对自己所做之事诚实的代码 —— 仅此而已。 -至于被删掉的分析是按规范重建，还是让扩展由一个成熟的开源检查器驱动，我们尚未决定。 -无论走哪条路，新的数字只有在经受住套件之外的用例与变异测试后才会发布。 +### Basilisk 远不只是一个类型检查器 + +类型检查只是其中一部分。其余部分是装在单个 Rust 二进制文件里的完整 Python 工作流 +—— 语言服务器、重构、格式化、集成调试、性能分析，以及各个编辑器扩展 —— 它们都不 +建立在正在接受审计的规则之上。这正是我们在审计期间着力打磨的地方：把真正有用的部分 +做扎实，并移除任何可能给出误导性结果的东西。变小的意义，是最终得到一个你可以信赖的 +工具。 [阅读完整更正 →](https://www.basilisk-python.dev/zh/docs/conformance/)  •  [完整性审计 →](https://github.com/Nimblesite/Basilisk/blob/main/docs/CONFORMANCE-INTEGRITY-AUDIT.md) diff --git a/vscode-extension/package.json b/vscode-extension/package.json index 6e8c1db03..f530d6dd8 100644 --- a/vscode-extension/package.json +++ b/vscode-extension/package.json @@ -38,9 +38,7 @@ "go to definition", "rust", "basilisk", - "lsp", - "pep conformance", - "conformance" + "lsp" ], "activationEvents": [ "onLanguage:python", diff --git a/website/src/_data/releases.js b/website/src/_data/releases.js index 6cf6e7b1a..0c693c445 100644 --- a/website/src/_data/releases.js +++ b/website/src/_data/releases.js @@ -23,6 +23,32 @@ const RELEASES_URL = `https://github.com/${OWNER}/${REPO}/releases`; // allowed. `breaks: true` matches how GitHub itself renders release bodies. const md = markdownIt({ html: true, linkify: true, breaks: true }); +// Withdrawn-claim redaction. Release notes arrive verbatim from GitHub, and some +// historical entries quote the conformance result that has since been retracted. +// [CHKARCH-CONFORMANCE] forbids publishing, quoting, or marketing any conformance +// figure, so rendering those lines unchanged would keep republishing a claim we +// have withdrawn. Any line pairing a conformance subject with a figure is replaced +// by a visible marker — nothing is silently dropped, the marker links to the +// correction, and every release heading still links to the unmodified GitHub +// release so the original wording stays one click away. +const CLAIM_SUBJECT = /conformance|conformant/i; +const CLAIM_FIGURE = /\d+(?:\.\d+)?\s*%|\b\d{1,4}\s*\/\s*\d{1,4}\b|\b\d+\s+false\s+positives?\b/i; +const LIST_MARKER = /^(\s*(?:[-*+]|\d+\.)\s+)/; +const REDACTION = "*[withdrawn conformance claim redacted — see the correction](/docs/conformance/)*"; + +// Replace each claim-bearing line with the marker, preserving its list marker so +// the surrounding changelog structure still renders. +function redactWithdrawnClaims(markdown) { + return markdown + .split(/\r?\n/) + .map((line) => { + if (!CLAIM_SUBJECT.test(line) || !CLAIM_FIGURE.test(line)) return line; + const marker = line.match(LIST_MARKER); + return `${marker ? marker[1] : ""}${REDACTION}`; + }) + .join("\n"); +} + const MONTHS = ["Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"]; // "2026-06-23T10:16:43Z" -> "Jun 23, 2026". UTC getters keep the output @@ -88,7 +114,7 @@ function toRecord(release) { date: formatDate(release.published_at || release.created_at), dateIso: release.published_at || release.created_at, prerelease: release.prerelease === true, - bodyHtml: release.body ? md.render(release.body) : "", + bodyHtml: release.body ? md.render(redactWithdrawnClaims(release.body)) : "", assets: (release.assets || []).map((asset) => ({ name: asset.name, url: asset.browser_download_url, diff --git a/website/src/blog/basilisk-100-percent-python-typing-conformance.md b/website/src/blog/basilisk-100-percent-python-typing-conformance.md index 14d526c5a..6da7d0ee7 100644 --- a/website/src/blog/basilisk-100-percent-python-typing-conformance.md +++ b/website/src/blog/basilisk-100-percent-python-typing-conformance.md @@ -1,106 +1,133 @@ --- layout: layouts/blog.njk -title: "Retracted: Basilisk's Former Typing Conformance Result" -description: "Retraction of Basilisk's former Python typing conformance claim, why the result was untrustworthy, and why the affected rules are being deleted rather than repaired." +title: "Retraction: Our Withdrawn Python Typing Conformance Claim" +description: "We retract in full the Python typing conformance claim this post originally made. It was produced by checker logic fitted to the test fixtures, it was not evidence of specification conformance, and it must not be cited." date: 2026-07-11 -dateModified: 2026-08-06 +dateModified: 2026-08-08 author: Christian Findlay image: /assets/images/og-image.png -imageAlt: "Basilisk Python type checker and language server; results under integrity review" +imageAlt: "Basilisk Python type checker and language server; conformance claim retracted" imageWidth: 1200 imageHeight: 630 tags: - Python typing category: announcements -excerpt: "Basilisk has retracted its former conformance claim and requested removal from the official results. This post is retained only as a record of the withdrawn announcement." -keywords: python type checker, python typing conformance, python/typing conformance results, basilisk, mypy, pyright, ty, pyrefly, zuban, pep conformance, strict typing +excerpt: "This post originally announced a conformance result for Basilisk. That result is retracted in full, and the original text has been removed rather than preserved, because leaving it up kept republishing a claim we know to be unsupported." +keywords: basilisk retraction, python typing conformance correction, basilisk conformance withdrawn faq: - - q: "Which Python type checker has the highest conformance score?" - a: "Basilisk is not currently listed in the official python/typing results. Its former result is withdrawn and its actual percentage is temporarily unknown while every rule is audited and the ones that matched source text are deleted. Check the live official table for currently listed tools." - - q: "What is the python/typing conformance suite?" - a: "It is the official test suite maintained by the Python Typing community. Its harness records how a checker behaves on the suite's exact fixtures. That is valuable evidence, but a raw suite result alone does not establish faithful implementation of the full specification; mutation robustness and independent off-suite cases are also required." - - q: "Is a 100% conformance score the same as being the best type checker?" - a: "No. A suite score describes the covered fixtures; it is not proof of specification correctness by itself, as Basilisk's retraction demonstrates. It also does not capture editor integration, error quality, ecosystem support, or independently validated performance." - - q: "How is Basilisk's conformance score measured?" - a: "There is no current Basilisk conformance score. A future result will require the unmodified python/typing harness, semantics-preserving mutation testing, and independent off-suite cases derived from the specification, and only after the audit has finished removing rules that decided from source text rather than resolved symbols." + - q: "Is Basilisk listed in the official python/typing conformance results?" + a: "No. Basilisk was removed from the official results table at our own request, and we have not asked to be re-listed. We will not submit again until the audit is finished, semantics-preserving mutation testing passes clean, and someone outside the project has reviewed the work." + - q: "What is Basilisk's current conformance percentage?" + a: "We do not publish one, and we ask that no figure be attributed to us. Any number we produced before the audit was measured against fixtures our own code had been fitted to, so it did not measure what we said it measured. We expect an honest figure to be lower than the withdrawn one, and we will say so plainly when we can measure it properly." + - q: "Was the original result an attempt to game the conformance suite?" + a: "No. It was a verification failure. Our development process named the suite score as the target to build against, and matching source text raises that score faster than real analysis does, so the code drifted that way. Nobody set out to defeat the suite and nothing was concealed from python/typing; the submission ran the suite's own unmodified harness. We did not run the check that would have caught the problem before we published, and that is our responsibility. Basilisk's author has published a personal account at https://www.christianfindlay.com/blog/basilisk-conformance-apology." + - q: "Should I use Basilisk's type checker in CI?" + a: "Not yet. Until the audit is finished, do not gate CI on it, do not block a merge with it, and do not read a clean run as a clean codebase. The rest of Basilisk — the language server, refactoring, formatting, debugging, and profiling — does not depend on the rules under audit." --- -> **Retraction — 6 August 2026:** We withdraw every conformance claim in this post. Basilisk's source contained logic fitted to the exact conformance fixtures, so the former perfect result did not establish specification conformance. We asked for Basilisk to be removed from the official results table, and it has been removed. The current percentage is temporarily unknown, and we are not trying to restore it: we are auditing every rule and deleting the ones that matched the spelling of code rather than its meaning, which will push the number lower before anything improves. The original article is retained below only as a public record; its score, ranking, pass counts, and conclusions must not be relied on. Read the [full correction](/docs/conformance/). - -Python has a genuinely good type system now, and most developers still do not realize it. A Python type checker works a lot like the TypeScript compiler. Type-checked Python is to regular Python what TypeScript is to JavaScript. The annotations have been in the language for a decade, the specification is mature, and the tooling has caught up. - -The open question was never whether Python's type system was good enough. It was how faithfully any given tool actually implements it. - -At publication, we believed we had an objective answer for Basilisk. It had been added to this [pinned snapshot of the official python/typing conformance results]({{ conformanceOfficial.historical.snapshot.snapshotUrl }}), where that run reported {{ conformanceOfficial.historical.basilisk.pct }}% ({{ conformanceOfficial.historical.basilisk.passLabel }} of {{ conformanceOfficial.historical.basilisk.total }} tests). That result is now withdrawn. - -We were proud of that result. The integrity audit showed that conclusion was wrong. - -## Why the conformance suite is the referee that matters - -A tool does not get to grade its own homework. Every type checker author will tell you their tool is excellent. That is not evidence. - -The [python/typing conformance suite](https://github.com/python/typing/tree/main/conformance) is the closest thing the Python ecosystem has to an objective referee. It is maintained by the Python Typing community, it encodes the actual typing specification as a set of test files, and it runs every participating checker through the same tests with the same harness. Nobody grades themselves. The suite grades all of them, together, on one run. - -We treated that as making the result meaningful. The suite did produce the number with its shared harness, but our code had been fitted to exact fixture text. The measurement therefore did not support the conclusion we drew from it. - -Basilisk was added to that run in [python/typing pull request #2316](https://github.com/python/typing/pull/2316), "Add Basilisk to conformance results," merged on July 6, 2026. We later requested removal after retracting the result, and Basilisk no longer appears in the live table. - -## The historical board snapshot we published - -The following is the leaderboard snapshot that accompanied the original announcement on {{ conformanceOfficial.historical.snapshot.dateLabel }}. It is not current, and Basilisk's row is withdrawn. Use the [live official results]({{ conformanceOfficial.historical.snapshot.source }}) for tools that remain listed. - -| Rank | Type checker | Backed by | Conformance | -|---|---|---|---| -{%- for t in conformanceOfficial.historical.ranked %} -| {{ t.rank }} | [{{ t.name }} {{ t.version }}]({{ t.resultsUrl }}) | {{ t.org | default("Independent") }} | **{{ t.pct }}%** ({{ t.passLabel }}/{{ t.total }}) | -{%- endfor %} - -A few things are worth saying plainly about that table, because the company Basilisk is keeping is serious. - -[Pyright](https://github.com/microsoft/pyright) is developed by Microsoft. [Pyrefly](https://github.com/facebook/pyrefly) is built by Meta. [ty](https://github.com/astral-sh/ty) is built by Astral, the team behind Ruff and uv, which [has agreed to join OpenAI](https://openai.com/index/openai-to-acquire-astral/) (a deal announced in March 2026 and, at announcement, still subject to regulatory approval and customary closing conditions). [mypy](https://github.com/python/mypy) is the original, created by Jukka Lehtosalo and developed heavily at Dropbox. [zuban](https://github.com/zubanls/zuban) is written by David Halter, the author of Jedi. [pycroscope](https://github.com/JelleZijlstra/pycroscope) is maintained by CPython core developer Jelle Zijlstra. - -At publication, we used this snapshot to place Basilisk above the other tools. That comparison is withdrawn because Basilisk's result was not robust. - -The original post presented the snapshot as proof that a small independent tool sat at the top of a board containing much larger teams. That presentation is part of the withdrawn claim. - -## What 100% does and does not mean - -Here is the part where we argue against our own headline, because you deserve the honest version. - -The python/typing maintainers put a caveat right at the top of the results page, and we agree with it completely: - -> "While specification conformance is important for the ecosystem, we don't recommend using it as the primary basis for choosing a type checker. It is not representative of many of the things users typically care about." ([python/typing conformance results]({{ conformanceOfficial.historical.snapshot.source }})) - -Read that twice. The people who built the suite are telling you not to treat their own scoreboard as the only thing that matters. That is the right position, and we are not going to pretend otherwise to make Basilisk look better. - -The original post tried to explain what we believed a perfect conformance score meant. The integrity audit invalidated the central claim. - -**What we claimed it was:** proof that Basilisk judged code correctly against the typing specification. That inference was wrong. Passing the exact suite did not establish a general implementation when parts of the checker matched the fixtures' text. - -**What it is not:** a claim that Basilisk is automatically the best choice for your project. Conformance does not measure how fast a checker runs, how good its error messages are, how well it integrates with your editor, or how mature its ecosystem is. Those things matter enormously, and on some of them the older tools have years of head start. - -Conformance is one input. It happens to be the input that decides whether you can trust the rest. But it is not the whole decision, and anyone who tells you a single number settles the question is selling something. - -## Why we chased the number anyway - -If conformance is not the whole story, why did we make 100% a hard requirement rather than a nice-to-have? - -Because the alternative is a checker that is confidently wrong some of the time, and a checker that is confidently wrong is worse than no checker at all. The problem with Python typing was never the syntax. The problem was enforcement. A type hint that is never checked is a comment. A type hint that is checked by a tool with gaps is a comment that occasionally lies to you. - -Basilisk enables its typing-spec rules by default, with no `--strict` flag to remember. We claimed the old score proved those rules implemented the specification correctly. It did not, and the rules that cannot show they analyse code are being deleted rather than repaired. - -## How the withdrawn score was produced - -We measure this the boring, reproducible way, because that is the only kind of measurement worth publishing. - -The withdrawn number came from the suite's unmodified harness, run against the default-configuration Basilisk CLI with every specification rule enabled. That procedure reproduced the result, but it could not reveal that parts of the implementation were fitted to the exact tests. Future publication will therefore require both the official harness and mutation-based robustness checks. - -That is a deliberate design choice, and it maps to a rule we hold for everything we ship: self-measured metrics are only worth anything if they are reproducible and measured by a neutral party. The conformance suite is that neutral party. We just make sure our tool shows up and runs. - -## Try it, and try to break it - -You can read the current correction and remediation plan on our [conformance page](/docs/conformance/), and see that Basilisk is no longer listed on the [python/typing results page]({{ conformanceOfficial.historical.snapshot.source }}). - -Point Basilisk at your own code and report disagreements on [GitHub](https://github.com/Nimblesite/Basilisk/issues). The old score cannot stand in for that real-world scrutiny. A replacement result will be published only after the clean implementation survives broader regression cases and semantics-preserving mutations. - -Python's type system has been good enough to trust for a while. Now the tooling can be too. +> ## This post is retracted in full +> +> It originally announced a Python typing conformance result for Basilisk and used +> that result to compare Basilisk with other type checkers. **Every one of those +> claims is withdrawn** — the score, the ranking, the pass counts, the category +> breakdowns, and every conclusion drawn from them. None of it should be cited, +> quoted, or relied on by anyone, including us. +> +> We have **removed the original text** rather than leave it sitting under a notice. +> A banner on top of an intact announcement still republishes the claim to everyone +> who lands on the page, and search engines and language models keep lifting the +> number back out of it. What follows is the correction. + +## What the original post claimed + +It announced that Basilisk had been added to the official `python/typing` conformance +results, reported the figure from that run, and presented a leaderboard placing +Basilisk above established checkers from Microsoft, Meta, Astral, and others. It +argued that the result proved Basilisk implemented the typing specification correctly. + +All of that is withdrawn. The comparison with other tools is withdrawn along with it: +it was built on our own unreliable row, and it was not a fair statement about anyone +else's work. + +## Why the claim was wrong + +Parts of Basilisk's checker decided their answers from the **spelling** of source code +rather than its meaning, and that logic had been shaped around the exact contents of +the conformance test files. + +The clearest example: type-alias validation ran prefix and substring tests against raw +source text, including a special case for the literal `eval(` — a spelling that has no +standing anywhere in the typing specification and appeared only because one test file +used it. `int("3")` is the identical specification violation and was accepted silently. +Rename an import or reformat a file and the answer changed, even though the typing +behaviour under test had not. + +A suite result produced by code developed against that suite's exact fixtures does not +measure specification conformance. The file passes and the rule is not implemented. +That is why the correction is deletion rather than a better score. + +## This was a mistake, not an attempt to game the suite + +We didn't set out to defeat the conformance suite, and nothing was concealed from +`python/typing`. The submission ran the suite's own unmodified harness, with +Basilisk's default configuration and every specification rule enabled. When the defect was demonstrated, we asked for our own removal. + +Our development process named the conformance score as the thing to build against, and matching source text raises that score faster than real analysis does — so that is the direction the code drifted, one plausible-looking rule at a time. Then we published and submitted the result on the +strength of a green run, without ever running the one check that would have exposed +it: does this rule still hold when the same program is spelled differently? + +That check did not exist. It still doesn't — building it is part of the remediation. +The conformance suite cannot catch this class of defect by construction, because it is +the very artefact the code was fitted to, so every green run reinforced a conclusion +we had no basis for. We believed the number meant what we said it meant. We were +wrong, and we were wrong because we did not verify it, not because we were trying to +get away with something. + +It was also **not us who found it**. It was reported from outside, in +[issue #379](https://github.com/Nimblesite/Basilisk/issues/379), from a public +reproduction. That is its own finding, and we record it as one. + +Basilisk's author has written a +[personal account and apology](https://www.christianfindlay.com/blog/basilisk-conformance-apology) +taking responsibility in his own words — that he did not set out to publish a number he +knew was false, did not understand how the rules were passing until it was demonstrated, +and did not verify the code, which he calls his own failure. + +## What we have done + +- Withdrew the conformance claim and the published benchmark figures. +- Asked `python/typing` to remove Basilisk from the official results + ([python/typing#2330](https://github.com/python/typing/pull/2330), reverting + [#2316](https://github.com/python/typing/pull/2316)). It has been removed. +- Published the full + [integrity audit](https://github.com/Nimblesite/Basilisk/blob/main/docs/CONFORMANCE-INTEGRITY-AUDIT.md) + — the defect, the method used to find the rest, reproduction commands, and the parts + still broken — with the remaining work tracked as public issues. +- Started auditing every rule and **deleting** the ones that decide from source text, + leaving a failing test behind each removal so the gap is visible rather than hidden. + +## What is true today + +**We publish no conformance figure, and we ask that none be attributed to us.** Not the +withdrawn one, not a revised one. A number is not what is wrong here, and putting a new +one out before the audit finishes would repeat the mistake. + +**Do not put Basilisk's type checker in your pipeline yet.** It still contains code that +isn't doing real type checking, so it can be wrong in both directions — a false error on +correct code, or silence where there is a genuine bug. + +**The rest of Basilisk is unaffected.** The language server, refactoring, formatting, +integrated debugging, and profiling do not rest on the rules under audit. Sharpening +those, and removing anything that could hand you a misleading result, is the work in +front of us. + +Basilisk is MIT-licensed and provided **as is, without warranty of any kind**. + +## If you cited this post + +Please update or remove the citation. If you are holding a screenshot, a ranking table, +or a figure that traces back here, it is withdrawn. We would rather carry the cost of +the correction than have the number keep circulating. + +Read the [full correction](/docs/conformance/) for the audit scope, the deletion policy, +and the bar any future result has to clear before we publish anything again. diff --git a/website/src/docs/conformance.md b/website/src/docs/conformance.md index 11a6bd6a2..8f7e0774e 100644 --- a/website/src/docs/conformance.md +++ b/website/src/docs/conformance.md @@ -19,6 +19,14 @@ We found checker logic fitted to the exact contents of conformance test files ra A passing result from code developed against the exact fixtures is not evidence, so the fix is not a better score. +## This was a mistake, not an attempt to game the suite + +We didn't set out to defeat the conformance suite. What actually happened is duller. Our development process named the conformance score as the thing to build against, and matching source text raises that score faster than real analysis does, so that is the direction the code drifted — one plausible-looking rule at a time. We then published and submitted on the strength of a green run, without ever running the one check that would have exposed it: does this rule still hold when the same program is spelled differently? That check did not exist, and building it is part of the remediation below. The suite cannot catch this class of defect by construction, because it is the artefact the code was fitted to, so every green run reinforced a conclusion we had no basis for. + +We believed the number meant what we said it meant. We were wrong, and we were wrong because we failed to verify it. Basilisk's author has published a [personal account and apology](https://www.christianfindlay.com/blog/basilisk-conformance-apology) taking responsibility in his own words. + +We also did not find this ourselves. It was reported from outside, in [issue #379](https://github.com/Nimblesite/Basilisk/issues/379), with a public reproduction. That is its own finding, and we record it as one. + ## What we are doing **We are auditing every rule and deleting the ones that don't do real type checking.** Not rewriting them, not patching them, not marking them TODO — deleting them, and leaving a failing test behind so the gap is visible rather than hidden. A rule stays only if it decides from the resolved syntax tree and returns the same diagnostics when the same program is spelled differently. @@ -31,7 +39,9 @@ The consequences are deliberate, and we would rather state them up front than ha What is left will be code that is honest about what it does — nothing else. -Whether the deleted analysis gets rebuilt from the specification, or the extension ends up driven by an established open-source checker, is a decision we have not made yet. Either way, no replacement percentage gets published until it survives the robustness testing described below. +**Where a rule can't be made reliable in a straightforward way, we will depend on a different, established type checker rather than ship our own unreliable version of it.** An answer from an engine that has earned trust is worth more to you than a Basilisk-branded one that hasn't. No replacement percentage gets published until it survives the robustness testing described below. + +Type checking is one part of Basilisk. The rest — language server, refactoring, formatting, integrated debugging, profiling, and the editor extensions — does not rest on the rules under audit, and that is what we are sharpening while the audit runs: make the parts that are genuinely useful solid, and remove anything that could hand you a misleading result.

Current python/typing results ↗ diff --git a/website/src/docs/releases.njk b/website/src/docs/releases.njk index f050aafcb..d31ece6d9 100644 --- a/website/src/docs/releases.njk +++ b/website/src/docs/releases.njk @@ -17,10 +17,12 @@ permalink: /docs/releases/

Releases

- Historical release notes may contain withdrawn conformance or benchmark claims. - Those claims are preserved verbatim as part of the changelog, but they are not current evidence and - must not be relied on. Basilisk's conformance percentage is temporarily unknown and its benchmark - figures are under review. Read the conformance correction and + Withdrawn conformance claims in these historical notes are redacted. + Older entries announced a conformance result that we have since retracted, so any changelog line + quoting a conformance figure is replaced here with a marked redaction rather than republished. + Nothing is hidden: each release heading links to the unmodified release on GitHub. Basilisk's + conformance percentage is temporarily unknown and we publish no figure; its benchmark numbers are + also under review. Read the conformance correction and benchmark notice for the current status.

diff --git a/website/src/index.njk b/website/src/index.njk index 65ac9055f..ae2e1d023 100644 --- a/website/src/index.njk +++ b/website/src/index.njk @@ -19,6 +19,8 @@ permalink: /

+ Don't put the type checker in your pipeline yet. It still contains code + that isn't doing real type checking, so it is not yet trustworthy. We have withdrawn both our former conformance claim and our published benchmark figures, and Basilisk was removed from the official python/typing results at our request. We are now auditing every rule and deleting the ones that match source text @@ -74,8 +76,9 @@ permalink: /

- Both sets of figures are withdrawn while the audit runs. + Both sets of figures are withdrawn while the audit runs. This was a mistake and a failure to verify, not an attempt to game the suite. Read the conformance correction → + Read the author’s apology → Read the benchmark notice →

diff --git a/website/src/zh/blog/basilisk-100-percent-python-typing-conformance.md b/website/src/zh/blog/basilisk-100-percent-python-typing-conformance.md index 16ea9c4bb..3a81a4b9b 100644 --- a/website/src/zh/blog/basilisk-100-percent-python-typing-conformance.md +++ b/website/src/zh/blog/basilisk-100-percent-python-typing-conformance.md @@ -1,107 +1,117 @@ --- layout: layouts/blog.njk -title: "已撤回：Basilisk 此前的类型符合性结果" -description: "撤回 Basilisk 此前的 Python typing 符合性声明，说明该结果为何不可信，以及受影响的规则为何是被删除而不是被修补。" +title: "撤回声明：我们已撤回的 Python typing 符合性声明" +description: "我们全文撤回本文此前作出的 Python typing 符合性声明。该结果出自针对测试用例适配的检查器逻辑，并不能证明规范符合性，不应被引用。" date: 2026-07-11 -dateModified: 2026-08-06 +dateModified: 2026-08-08 author: Christian Findlay image: /assets/images/og-image.png -imageAlt: "Basilisk Python 类型检查器与语言服务器；结果正在接受完整性审查" +imageAlt: "Basilisk Python 类型检查器与语言服务器；符合性声明已撤回" imageWidth: 1200 imageHeight: 630 tags: - Python typing category: announcements lang: zh -excerpt: "Basilisk 已撤回此前的符合性声明，并请求从官方结果中移除。本文仅作为已撤回公告的历史记录保留。" -keywords: python类型检查器, python类型符合性, python/typing符合性结果, basilisk, mypy, pyright, ty, pyrefly, zuban, pep符合性, 严格类型 +excerpt: "本文原本发布了 Basilisk 的一项符合性结果。该结果现已全文撤回；我们删除了原文，而不是在其上方加一条提示——因为把原文留在那里，等于每次都在重新发布一个我们已知站不住脚的声明。" +keywords: basilisk 撤回声明, python typing 符合性更正, basilisk 符合性已撤回 faq: - - q: "哪个 Python 类型检查器的符合性分数最高？" - a: "Basilisk 目前不在官方 python/typing 结果中。此前的结果已撤回；在完成规则审计、删除依据源文本判断的规则之前，实际百分比暂时未知。其他工具请查看官方实时结果表。" - - q: "什么是 python/typing 符合性测试套件？" - a: "这是由 Python Typing 社区维护的官方测试套件。它使用自己的评分工具记录检查器在确切测试用例上的行为。这是有价值的证据，但原始套件结果本身不能证明完整规范已被忠实实现；还必须通过保持语义的变异和独立的套件外用例验证。" - - q: "100% 的符合性分数是否等于是最好的类型检查器？" - a: "不。套件分数只描述被覆盖的测试用例；正如 Basilisk 此次撤回所证明的，它本身不能证明规范实现正确。它也无法反映编辑器集成、错误信息质量、生态系统支持或经过独立验证的性能。" - - q: "Basilisk 的符合性分数是如何测量的？" - a: "Basilisk 目前没有可发布的符合性分数。只有在审计完成、依据源文本而非已解析符号判断的规则被删除之后，未来结果才可能发布，并且必须同时通过未经修改的 python/typing 评分工具、保持语义的变异测试，以及依据规范独立设计的套件外用例。" + - q: "Basilisk 目前在官方 python/typing 符合性结果中吗？" + a: "不在。我们主动请求将 Basilisk 从官方结果表中移除，并且没有申请重新列入。在审计完成、保持语义的变异测试全部通过、并且由项目之外的人完成复核之前，我们不会再次提交。" + - q: "Basilisk 当前的符合性百分比是多少？" + a: "我们不发布任何数字，也请不要把任何数字归到我们名下。审计之前我们得到的任何数字，都是针对我们自己的代码已经适配过的测试用例测出来的，因此它并没有衡量我们所声称的东西。我们预计诚实的数字会低于此前撤回的那个；等到能够正确测量时，我们会如实说明。" + - q: "当初的结果是不是有意操纵符合性测试套件？" + a: "不是。这是一次验证失败。我们的开发流程把套件分数定为要去达成的目标，而匹配源文本比真正做分析更快地提高这个分数，代码于是滑向了那个方向。没有人试图去糊弄测试套件，也没有向 python/typing 隐瞒任何东西——提交使用的是套件自己未经修改的评分工具。我们在发布之前没有做那项本可以发现问题的检查，这是我们的责任。Basilisk 作者已在 https://www.christianfindlay.com/blog/basilisk-conformance-apology 发表了个人说明。" + - q: "我应该在 CI 中使用 Basilisk 的类型检查器吗？" + a: "暂时不要。在审计完成之前，请不要用它作为 CI 门禁，不要用它阻断合并，也不要把一次干净的运行结果当作代码库是干净的。Basilisk 的其余部分——语言服务器、重构、格式化、调试与性能分析——并不依赖正在审计的这些规则。" --- -> **撤回说明——2026 年 8 月 6 日：**我们撤回本文中的所有符合性声明。Basilisk 源码中存在针对确切符合性测试用例实现的逻辑，因此此前的满分结果不能证明规范符合性。我们请求从官方结果表中移除 Basilisk，现已完成移除。在删除有问题的实现、根据规范重新构建并通过保持语义的变异测试之前，当前百分比暂时未知。下方原文仅作为公开历史记录保留；其中的得分、排名、通过数量和结论均不可依赖。请阅读[完整更正](/zh/docs/conformance/)。 +> ## 本文已全文撤回 +> +> 本文原本发布了 Basilisk 的一项 Python typing 符合性结果，并用该结果将 Basilisk +> 与其他类型检查器作比较。**其中的每一项声明均已撤回** —— 包括分数、排名、通过数量、 +> 分类明细，以及由此得出的全部结论。任何人都不应再引用、转述或依赖这些内容，我们自己也不会。 +> +> 我们**删除了原文**，而不是在原文上方加一条提示。把撤回声明贴在一篇完好的公告上方， +> 每一个访问该页面的人仍然会再次读到那个声明，搜索引擎和语言模型也会继续把那个数字 +> 从中提取出来。以下是更正内容。 -Python 现在拥有一个真正出色的类型系统，而大多数开发者仍然没有意识到这一点。Python 类型检查器的工作方式很像 TypeScript 编译器。经过类型检查的 Python 之于普通 Python，就如同 TypeScript 之于 JavaScript。类型注解已经存在于语言中十年了，规范已经成熟，工具也已经跟上。 +## 原文当时声称了什么 -从来悬而未决的问题从不是 Python 的类型系统是否足够好。而是任何一个具体工具究竟对它的实现有多忠实。 +原文宣布 Basilisk 已被加入官方 `python/typing` 符合性结果，报告了那次运行的数字，并给出 +一份排行榜，把 Basilisk 排在来自 Microsoft、Meta、Astral 等团队的成熟检查器之上；文章据此 +论证该结果证明 Basilisk 正确实现了类型规范。 -发布本文时，我们以为 Basilisk 得到了一个客观答案。它被添加进这份[官方 python/typing 符合性结果的固定快照]({{ conformanceOfficial.historical.snapshot.snapshotUrl }})，当时的运行报告了 {{ conformanceOfficial.historical.basilisk.pct }}%（{{ conformanceOfficial.historical.basilisk.total }} 项测试中通过 {{ conformanceOfficial.historical.basilisk.passLabel }} 项）。该结果现已撤回。 +以上全部撤回。与其他工具的比较也一并撤回：它建立在我们自己那一行不可靠的数据之上， +对别人的工作而言并不是一个公平的陈述。 -我们曾为此感到自豪。完整性审计证明这个结论是错误的。 +## 这个声明为什么是错的 -## 为什么符合性测试套件才是那个重要的裁判 +Basilisk 检查器中的一部分逻辑，依据的是源代码的**写法**而不是它的含义，而且这些逻辑是围绕 +符合性测试文件的确切内容塑造出来的。 -工具不能给自己的作业打分。每个类型检查器的作者都会告诉你他们的工具很出色。这不是证据。 +最清楚的例子：类型别名验证对原始源代码文本执行前缀和子字符串判断，其中甚至专门判断了 +字面量 `eval(` —— 这个写法在类型规范中没有任何依据，出现在代码里只是因为某个测试文件用了它。 +`int("3")` 是完全相同的规范违规，却被静默接受。改一个导入别名或重新格式化文件，结论就会变， +尽管被测试的类型行为并没有改变。 -[python/typing 符合性测试套件](https://github.com/python/typing/tree/main/conformance)是 Python 生态系统中最接近客观裁判的东西。它由 Python Typing 社区维护，将真正的类型规范编码为一组测试文件，并用同一套评分工具、同样的测试来运行每一个参与的检查器。没有人给自己打分。是这个套件在一次运行中把它们全部一起评分。 +针对某个套件的确切测试用例开发出来的代码，即使在该套件上通过，也不能衡量规范符合性。 +真实情况是：文件通过了，而规则并没有实现。这正是为什么我们的纠正措施是删除，而不是去拿一个更好的分数。 -我们曾认为这使结果具有意义。套件确实使用共享评分工具产出了该数字，但我们的代码针对确切测试文本进行了适配，因此该测量无法支撑我们得出的结论。 +## 这是一个错误，而不是有意操纵测试套件 -Basilisk 是通过 [python/typing 拉取请求 #2316](https://github.com/python/typing/pull/2316)（"Add Basilisk to conformance results"）加入那次运行的，该请求于 2026 年 7 月 6 日合并。撤回结果后，我们请求将其移除；Basilisk 已不再出现在实时结果表中。 +我们希望在这一点上毫不含糊，因为事件的先后顺序容易让人得出一个比事实更糟的解读。 -## 当时发布的历史排行榜快照 +**没有人试图去糊弄符合性测试套件，也没有向 `python/typing` 隐瞒任何东西。** 提交时运行的是套件 +自己未经修改的评分工具，使用 Basilisk 的默认配置并启用全部规范规则。没有针对评分工具做特殊处理， +没有伪造输出，没有禁用规则，也没有手工编辑结果文件。当缺陷被演示出来之后，我们主动请求移除自己。 -以下是原公告在 {{ conformanceOfficial.historical.snapshot.dateLabel }} 使用的排行榜快照。它不是当前结果，Basilisk 这一行也已撤回。当前仍列出的工具请查看[官方实时结果]({{ conformanceOfficial.historical.snapshot.source }})。 +真实经过要平淡得多，也更普通。我们的开发流程把符合性分数定为要去达成的目标，而匹配源文本比 +真正做分析更快地提高这个分数 —— 于是代码就朝那个方向滑了过去，一条看起来合理的规则接着一条。 +然后我们仅凭一次全绿的运行就发布并提交了结果，却始终没有做那项本可以揭露问题的检查： +同一个程序换一种写法时，这条规则是否依然成立？ -| 排名 | 类型检查器 | 背后团队 | 符合性 | -|---|---|---|---| -{%- for t in conformanceOfficial.historical.ranked %} -| {{ t.rank }} | [{{ t.name }} {{ t.version }}]({{ t.resultsUrl }}) | {{ t.org | default("独立") }} | **{{ t.pct }}%** ({{ t.passLabel }}/{{ t.total }}) | -{%- endfor %} +这项检查当时并不存在，现在也还没有 —— 把它建起来正是补救工作的一部分。符合性测试套件从构造上 +就无法发现这一类缺陷，因为它正是代码被适配过去的那个对象，所以每一次全绿运行都在强化一个 +我们根本没有依据的结论。我们当时相信那个数字就是我们所说的意思。我们错了；我们之所以错， +是因为我们没有去验证它，而不是因为我们想蒙混过关。 -关于那张表格，有几点值得直白地说清楚，因为 Basilisk 所处的阵容非常有分量。 +发现问题的**也不是我们**。它是由外部报告的，见 +[issue #379](https://github.com/Nimblesite/Basilisk/issues/379)，并附有公开的复现。 +这件事本身也是一项发现，我们如实记录。 -[Pyright](https://github.com/microsoft/pyright) 由微软开发。[Pyrefly](https://github.com/facebook/pyrefly) 由 Meta 构建。[ty](https://github.com/astral-sh/ty) 由 Astral 构建，也就是 Ruff 和 uv 背后的团队，该团队[已同意加入 OpenAI](https://openai.com/index/openai-to-acquire-astral/)（该交易于 2026 年 3 月宣布，在宣布时仍需监管批准和惯例性交割条件）。[mypy](https://github.com/python/mypy) 是最早的那个，由 Jukka Lehtosalo 创建，并在 Dropbox 大量开发。[zuban](https://github.com/zubanls/zuban) 由 Jedi 的作者 David Halter 编写。[pycroscope](https://github.com/JelleZijlstra/pycroscope) 由 CPython 核心开发者 Jelle Zijlstra 维护。 +Basilisk 的作者已就此事的经过发表了[个人说明与致歉](https://www.christianfindlay.com/blog/basilisk-conformance-apology)， +并承担责任。他在文中写道，他并没有打算发布一个自己明知是错的数字，也没有理解那些规则究竟是怎么通过的， +直到有人把问题演示出来；没有验证代码是他的失误。 -发布时，我们用该快照将 Basilisk 排在其他工具之前。由于 Basilisk 的结果缺乏稳健性，这项比较现已撤回。 +## 我们已经做了什么 -原文将该快照描述为一个小型独立工具位居大型团队之前的证明。这个说法属于已撤回的声明。 +- 撤回了符合性声明和已发布的基准测试数字。 +- 请求 `python/typing` 将 Basilisk 从官方结果中移除 + （[python/typing#2330](https://github.com/python/typing/pull/2330)，撤销 + [#2316](https://github.com/python/typing/pull/2316)）。现已完成移除。 +- 公开了完整的[完整性审计](https://github.com/Nimblesite/Basilisk/blob/main/docs/CONFORMANCE-INTEGRITY-AUDIT.md) + —— 包括缺陷本身、用于排查其余部分的方法、复现命令，以及仍然没有修好的部分；剩余工作以公开 + issue 跟踪。 +- 开始逐条审计规则，**删除**那些依据源文本判断的规则，并在每一次删除之后留下一个失败的测试， + 让缺口可见而不是被掩盖。 -## 100% 意味着什么，又不意味着什么 +## 现在的真实情况 -接下来这部分我们要反驳自己的标题，因为你值得听到诚实的版本。 +**我们不发布任何符合性数字，也请不要把任何数字归到我们名下。** 撤回的那个不行，修订过的也不行。 +这里出问题的并不是某个数字；在审计完成之前发布一个新数字，只会重复同样的错误。 -python/typing 的维护者在结果页面的顶部就放了一个提醒，我们完全同意： +**请暂时不要把 Basilisk 的类型检查器放进你的流水线。** 其中仍有没有做真正类型检查的代码， +因此它在两个方向上都可能出错 —— 既可能对正确的代码报出虚假错误，也可能对真实的缺陷保持沉默。 -> "虽然规范符合性对生态系统很重要，但我们不建议将其作为选择类型检查器的主要依据。它并不能代表用户通常关心的许多方面。"（[python/typing 符合性结果]({{ conformanceOfficial.historical.snapshot.source }})） +**Basilisk 的其余部分不受影响。** 语言服务器、重构、格式化、集成调试和性能分析都不依赖正在审计的 +这些规则。把这些部分打磨扎实，并移除任何可能给出误导性结果的东西，正是我们眼下要做的工作。 -请读两遍。构建这个套件的人正在告诉你，不要把他们自己的记分牌当作唯一重要的东西。这是正确的立场，我们不会为了让 Basilisk 看起来更好而假装不是这样。 +Basilisk 采用 MIT 许可证，并按**“原样”提供，不附带任何形式的担保**。 -原文试图解释我们当时认为满分意味着什么。完整性审计推翻了核心结论。 +## 如果你引用过本文 -**我们曾声称它是什么：**证明 Basilisk 能根据类型规范正确判断代码。这个推论是错误的。如果检查器的一部分匹配了测试文本，通过确切套件并不能证明通用实现。 +请更新或删除该引用。如果你手上有截图、排行榜表格，或任何可以追溯到这里的数字，它们均已撤回。 +我们宁愿承担更正的代价，也不愿让那个数字继续流传。 -**它不是什么：**它不是一个声称 Basilisk 自动就是你项目最佳选择的说法。符合性不衡量检查器运行有多快、错误信息有多好、与你的编辑器集成得有多好，或者其生态系统有多成熟。这些方面极其重要，而在其中一些方面，较老的工具有多年的先发优势。 - -符合性是一个输入。它恰好是决定你能否信任其余一切的那个输入。但它不是全部决定因素，任何告诉你单一数字就能定论的人都在推销某种东西。 - -## 我们为什么还是追求了这个数字 - -如果符合性不是全部，我们为什么把 100% 当作一个硬性要求，而不是一个锦上添花的东西？ - -因为另一种情况是一个有时会自信地出错的检查器，而一个自信地出错的检查器比没有检查器更糟。Python 类型的问题从来不在于语法。问题在于强制执行。一个从不被检查的类型提示只是一句注释。一个被有漏洞的工具检查的类型提示，是一句偶尔会对你撒谎的注释。 - -Basilisk 默认启用类型规范规则，无需记住 `--strict` 标志。我们曾声称旧分数证明这些规则正确实现了规范。事实并非如此；相关实现正在重新构建和验证。 - -## 已撤回分数如何产生 - -我们用那种枯燥、可复现的方式来测量它，因为那是唯一值得发布的测量方式。 - -已撤回的数字来自套件自己未经修改的评分工具，针对默认配置的 Basilisk CLI 运行并开启所有规范规则。该过程可以复现数字，却无法揭示部分实现针对确切测试进行了适配。因此，未来发布必须同时通过官方评分工具和基于变异的稳健性检查。 - -这是一个刻意的设计选择，它对应着我们对所交付的一切都坚持的一条规则：自我测量的指标只有在可复现、且由中立方测量时才有价值。符合性套件就是那个中立方。我们只是确保我们的工具出现并运行。 - -## 试用它，并试着让它出错 - -你可以在[符合性页面](/zh/docs/conformance/)阅读当前更正和修复计划，并在 [python/typing 结果页面]({{ conformanceOfficial.historical.snapshot.source }})上看到 Basilisk 已不再列出。 - -请把 Basilisk 指向你自己的代码，并在 [GitHub](https://github.com/Nimblesite/Basilisk/issues) 上报告分歧。旧得分不能替代这种真实检验。只有在审计完成、留下的代码通过更广泛的回归用例和保持语义的变异后，我们才会发布替代结果。 - -Python 的类型系统已经足够好、值得信任有一段时间了。现在，工具也可以了。 +关于审计范围、删除策略，以及未来任何结果在发布之前必须达到的标准，请阅读[完整更正](/zh/docs/conformance/)。 diff --git a/website/src/zh/docs/conformance.md b/website/src/zh/docs/conformance.md index aab0fefc6..015c4f54d 100644 --- a/website/src/zh/docs/conformance.md +++ b/website/src/zh/docs/conformance.md @@ -15,6 +15,18 @@ lang: zh 针对固定测试用例开发出的代码即使通过，也不能作为证据；因此，解决办法不是拿到一个更好的分数。 +## 这是一个错误，而不是有意操纵测试套件 + +我们希望在这一点上毫不含糊，因为事件的先后顺序容易让人得出一个比事实更糟的解读。 + +**没有人试图去糊弄符合性测试套件，也没有向 `python/typing` 隐瞒任何东西。** 提交时运行的是套件自己未经修改的评分工具，使用 Basilisk 的默认配置并启用全部规范规则 —— 没有针对评分工具做特殊处理，没有伪造输出，没有禁用规则，也没有手工编辑结果文件。当缺陷被演示给我们之后，我们主动请求移除自己。 + +真实经过要平淡得多。我们的开发流程把符合性分数定为要去达成的目标，而匹配源文本比真正做分析更快地提高这个分数，于是代码就朝那个方向滑了过去 —— 一条看起来合理的规则接着一条。然后我们仅凭一次全绿的运行就发布并提交了结果，却始终没有做那项本可以揭露问题的检查：同一个程序换一种写法时，这条规则是否依然成立？这项检查当时并不存在，把它建起来正是下文补救工作的一部分。符合性测试套件从构造上就无法发现这一类缺陷，因为它正是代码被适配过去的那个对象，所以每一次全绿运行都在强化一个我们根本没有依据的结论。 + +我们当时相信那个数字就是我们所说的意思。我们错了；我们之所以错，是因为我们没有去验证它。Basilisk 作者已就此发表[个人说明与致歉](https://www.christianfindlay.com/blog/basilisk-conformance-apology)，并承担责任。 + +发现问题的也不是我们。它是由外部报告的，见 [issue #379](https://github.com/Nimblesite/Basilisk/issues/379)，并附有公开的复现。这件事本身也是一项发现，我们如实记录。 + ## 我们正在做什么 **我们正在逐条审计规则，并删除那些没有做真正类型检查的规则。** 不是重写，不是打补丁，也不是标一个 TODO —— 是删除，并留下一个失败的测试，让这个缺口可见而不是被掩盖。一条规则只有在依据已解析的语法树做判断、并且同一个程序换一种写法时给出相同诊断的情况下，才会保留。 @@ -27,7 +39,9 @@ lang: zh 留下来的，将是对自己所做之事诚实的代码 —— 仅此而已。 -被删掉的分析是按规范重建，还是让扩展由一个成熟的开源检查器驱动，我们尚未决定。无论走哪条路，在通过下文所述的稳健性验证之前，都不会发布替代百分比。 +**如果一条规则无法以直截了当的方式做到可靠，我们会转而依赖另一个成熟的类型检查器，而不是端出我们自己那份不可靠的实现。** 一个已经赢得信任的引擎给出的答案，对你而言比一个挂着 Basilisk 名号却没有赢得信任的答案更有价值。在通过下文所述的稳健性验证之前，我们不会发布任何替代百分比。 + +类型检查只是 Basilisk 的一部分。其余部分 —— 语言服务器、重构、格式化、集成调试、性能分析，以及各个编辑器扩展 —— 都不建立在正在接受审计的规则之上；审计期间，这些正是我们着力打磨的地方：把真正有用的部分做扎实，并移除任何可能给出误导性结果的东西。

当前 python/typing 结果 ↗ diff --git a/website/src/zh/index.njk b/website/src/zh/index.njk index d1d840e3a..04c02005f 100644 --- a/website/src/zh/index.njk +++ b/website/src/zh/index.njk @@ -20,6 +20,8 @@ permalink: /zh/

+ 请暂时不要把类型检查器放进你的流水线。其中仍有没有做真正 + 类型检查的代码，因此它目前还不值得信任。 我们已撤回此前的符合性声明和公开的基准测试数据，应我们的请求，Basilisk 已从官方 python/typing 结果中移除。我们正在逐条审计规则， 并删除那些只匹配源文本、并未做真正类型检查的规则。 @@ -74,8 +76,9 @@ permalink: /zh/

- 审计期间，两组数据均已撤回。 + 审计期间，两组数据均已撤回。这是一个错误、一次验证上的失职，而不是有意操纵测试套件。 阅读符合性更正 → + 阅读作者的致歉 → 阅读基准测试说明 →

From 69350ee98a07788964a3821d4aa6c2426e107493 Mon Sep 17 00:00:00 2001 From: Christian Findlay <16697547+MelbourneDeveloper@users.noreply.github.com> Date: Sat, 8 Aug 2026 12:08:01 +1000 Subject: [PATCH 7/9] Restore the unknown-percentage disclosure to both homepage heroes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The homepage e2e asserts that .hero__subheadline discloses that the conformance percentage is temporarily unknown and that no replacement figure ships before robustness and mutation testing. Earlier hero copy dropped those sentences while the assertions stayed, so the Website Build job failed on main and on this branch. Fixed by restoring the disclosure to the English and Chinese heroes, not by relaxing the test — the assertion guards a documentation-honesty claim that should be on the page. --- website/src/index.njk | 4 +++- website/src/zh/index.njk | 1 + 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/website/src/index.njk b/website/src/index.njk index ae2e1d023..cdab1a061 100644 --- a/website/src/index.njk +++ b/website/src/index.njk @@ -25,7 +25,9 @@ permalink: / and Basilisk was removed from the official python/typing results at our request. We are now auditing every rule and deleting the ones that match source text instead of doing real type checking. Expect Basilisk to get smaller before it gets better; - what is left will be code that is honest about what it does. + what is left will be code that is honest about what it does. Our current conformance + percentage is temporarily unknown, and no replacement figure gets published until it + survives robustness and mutation testing.

diff --git a/website/src/zh/index.njk b/website/src/zh/index.njk index 04c02005f..59868ad04 100644 --- a/website/src/zh/index.njk +++ b/website/src/zh/index.njk @@ -26,6 +26,7 @@ permalink: /zh/ 已从官方 python/typing 结果中移除。我们正在逐条审计规则， 并删除那些只匹配源文本、并未做真正类型检查的规则。 Basilisk 会先变小，再变好；留下来的，将是对自己所做之事诚实的代码。 + 当前符合性百分比暂时未知；在通过稳健性与变异测试之前，我们不会发布任何替代数字。

From f34fbad009d4de5257166dd700894e71dc859e26 Mon Sep 17 00:00:00 2001 From: Christian Findlay <16697547+MelbourneDeveloper@users.noreply.github.com> Date: Sat, 8 Aug 2026 12:11:24 +1000 Subject: [PATCH 8/9] Release process --- .github/workflows/release.yml | 204 +++++++++++++++++++--------------- 1 file changed, 112 insertions(+), 92 deletions(-) diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index a0323f8a3..d1d1e9846 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -142,42 +142,53 @@ jobs: path: ${{ matrix.archive }} retention-days: 1 - conformance: - name: PEP conformance gate — 100% vs latest python/typing@main - runs-on: ubuntu-latest - timeout-minutes: 20 - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - - - uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable - - - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 - with: - key: conformance - - - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 - with: - python-version: "3.12" - - # The REAL python/typing harness's own runtime deps (run_conformance.py runs - # the suite's unmodified src/main.py). See [CHKARCH-CONFORMANCE]. - - name: Install conformance harness deps - run: pip install jinja2 markdown tomlkit - - - name: Build basilisk - run: cargo build --release --bin basilisk - - # Clone the LATEST python/typing@main FRESH and run its OWN harness against - # the compiled binary; require a perfect score — anything less than 100%, or a - # single false positive, tanks the release (coverage-thresholds.json → - # conformance.threshold 100, max_false_positives 0). No cached fixtures, no - # vendored calculator: if the real harness cannot run, the release fails. - - name: Enforce 100% PEP conformance (REAL python/typing harness) - run: python3 conformance/run_conformance.py --bin target/release/basilisk --gate + # ── PEP conformance gate — COMMENTED OUT, THE MEASUREMENT CANNOT RUN ──────── + # Basilisk's withdrawal from python/typing removed BasiliskTypeChecker from the + # suite's conformance/src/type_checker.py. The upstream harness resolves + # `--only-run` by comparing the value against each registered checker's name + # (conformance/src/main.py: `if options.only_run != type_checker.name: continue`), + # so `--only-run basilisk` now matches NOTHING: main.py exits 0 having graded no + # checker and writing no results/basilisk/*.toml, and run_conformance.py's + # run_harness() then raises "the real harness wrote no results ... it did not run". + # That failure is unconditional and blocks every publish job behind it. + # + # Type checking is not conformance, and a dead gate must not stop releases from + # shipping. Restoring the measurement would require vendoring a scorer or + # injecting an adapter, both of which [CHKARCH-CONFORMANCE] declares a BUILD + # FAILURE, so the job is commented out rather than reimplemented. Whether it is + # deleted outright or replaced by a disclosed non-official harness is the user's + # decision. Nothing here publishes, quotes, or markets a conformance figure. + # + # conformance: + # name: PEP conformance gate — 100% vs latest python/typing@main + # runs-on: ubuntu-latest + # timeout-minutes: 20 + # steps: + # - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + # + # - uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable + # + # - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 + # with: + # key: conformance + # + # - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + # with: + # python-version: "3.12" + # + # - name: Install conformance harness deps + # run: pip install jinja2 markdown tomlkit + # + # - name: Build basilisk + # run: cargo build --release --bin basilisk + # + # - name: Enforce 100% PEP conformance (REAL python/typing harness) + # run: python3 conformance/run_conformance.py --bin target/release/basilisk --gate release: name: Publish binaries to release - needs: [build, conformance] + # `conformance` removed from needs — the job is commented out above. + needs: [build] runs-on: ubuntu-latest timeout-minutes: 10 steps: @@ -873,10 +884,10 @@ jobs: # See pyproject.toml ([tool.maturin] bindings = "bin"). pypi-wheels: name: Build PyPI wheel - ${{ matrix.platform }} - # Gate wheel builds on the existing binary conformance job: never build a - # wheel of a non-conformant binary. Independent of the `build` matrix — the - # release archives feed Homebrew/Scoop/VSIX; these wheels feed PyPI. - needs: conformance + # Previously gated on the binary `conformance` job, which is commented out + # above because the measurement can no longer run. Independent of the `build` + # matrix — the release archives feed Homebrew/Scoop/VSIX; these wheels feed + # PyPI — so with that gate gone this job has no remaining prerequisite. runs-on: ${{ matrix.os }} timeout-minutes: 30 strategy: @@ -958,64 +969,73 @@ jobs: path: dist/*.whl retention-days: 1 - # Prove the ACTUAL shipped wheel — pip-installed — scores 100% via the REAL, - # unmodified python/typing harness (its own `src/main.py` + built-in - # BasiliskTypeChecker). This is the SAME harness CI runs against the compiled - # binary (conformance/run_conformance.py); here it runs against the pip-installed - # wheel so the released install path is proven conformant too. There is no other - # scorer. See [CHKARCH-CONFORMANCE]. - conformance-wheel: - name: Conformance gate — pip-installed wheel vs python/typing@main (real harness) - needs: pypi-wheels - runs-on: ubuntu-latest - timeout-minutes: 20 - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - - - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 - with: - python-version: "3.12" - - - name: Download linux-x64 wheel - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 - with: - name: pypi-wheel-linux-x64 - path: wheelhouse - - - name: Install the wheel + harness deps (proves install + exposes `basilisk`) - shell: bash - run: | - set -euo pipefail - python -m pip install --upgrade pip - python -m pip install wheelhouse/*.whl jinja2 markdown tomlkit - # Fails the job immediately if the wheel did not put `basilisk` on PATH. - basilisk --version - - - name: Clone python/typing@main (real fixtures + harness + adapter) - shell: bash - # The suite already ships the official Basilisk adapter (BasiliskTypeChecker - # in conformance/src/type_checker.py) — nothing of ours is injected. - # Cloned OUTSIDE the repo checkout: per-file config discovery walks - # ancestor directories ([CHKARCH-CONFIG-DISCOVERY]), so an in-repo clone - # would inherit the repo's own [tool.basilisk] rules instead of scoring - # the wheel's out-of-the-box default. - run: git clone --depth 1 https://github.com/python/typing "$RUNNER_TEMP/typing-upstream" - - - name: Run the REAL upstream harness against the installed wheel - working-directory: ${{ runner.temp }}/typing-upstream/conformance - shell: bash - # `basilisk` resolves to the pip-installed wheel on PATH (BASILISK_BIN unset). - run: python src/main.py --only-run basilisk - - - name: Assert the wheel scores 100% / zero false positives - shell: bash - run: python conformance/assert_wheel_conformance.py "$RUNNER_TEMP/typing-upstream/conformance/results/basilisk" + # ── Wheel conformance gate — COMMENTED OUT, THE MEASUREMENT CANNOT RUN ───── + # Same cause as the binary `conformance` job above: python/typing no longer + # registers a Basilisk checker, so `src/main.py --only-run basilisk` matches no + # checker, grades nothing, and writes no results/basilisk/*.toml for + # assert_wheel_conformance.py to read. The job could only ever fail, and it + # blocked publish-pypi. Commented out rather than reimplemented: vendoring a + # scorer or injecting an adapter is a BUILD FAILURE under [CHKARCH-CONFORMANCE]. + # + # # Prove the ACTUAL shipped wheel — pip-installed — scores 100% via the REAL, + # # unmodified python/typing harness (its own `src/main.py` + built-in + # # BasiliskTypeChecker). This is the SAME harness CI runs against the compiled + # # binary (conformance/run_conformance.py); here it runs against the pip-installed + # # wheel so the released install path is proven conformant too. There is no other + # # scorer. See [CHKARCH-CONFORMANCE]. + # conformance-wheel: + # name: Conformance gate — pip-installed wheel vs python/typing@main (real harness) + # needs: pypi-wheels + # runs-on: ubuntu-latest + # timeout-minutes: 20 + # steps: + # - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + # + # - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + # with: + # python-version: "3.12" + # + # - name: Download linux-x64 wheel + # uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + # with: + # name: pypi-wheel-linux-x64 + # path: wheelhouse + # + # - name: Install the wheel + harness deps (proves install + exposes `basilisk`) + # shell: bash + # run: | + # set -euo pipefail + # python -m pip install --upgrade pip + # python -m pip install wheelhouse/*.whl jinja2 markdown tomlkit + # # Fails the job immediately if the wheel did not put `basilisk` on PATH. + # basilisk --version + # + # - name: Clone python/typing@main (real fixtures + harness + adapter) + # shell: bash + # # The suite already ships the official Basilisk adapter (BasiliskTypeChecker + # # in conformance/src/type_checker.py) — nothing of ours is injected. + # # Cloned OUTSIDE the repo checkout: per-file config discovery walks + # # ancestor directories ([CHKARCH-CONFIG-DISCOVERY]), so an in-repo clone + # # would inherit the repo's own [tool.basilisk] rules instead of scoring + # # the wheel's out-of-the-box default. + # run: git clone --depth 1 https://github.com/python/typing "$RUNNER_TEMP/typing-upstream" + # + # - name: Run the REAL upstream harness against the installed wheel + # working-directory: ${{ runner.temp }}/typing-upstream/conformance + # shell: bash + # # `basilisk` resolves to the pip-installed wheel on PATH (BASILISK_BIN unset). + # run: python src/main.py --only-run basilisk + # + # - name: Assert the wheel scores 100% / zero false positives + # shell: bash + # run: python conformance/assert_wheel_conformance.py "$RUNNER_TEMP/typing-upstream/conformance/results/basilisk" publish-pypi: name: Publish wheels to PyPI - # Only publish once the wheels built AND the shipped wheel passed the real - # upstream harness AND the GitHub release exists. - needs: [pypi-wheels, conformance-wheel, release] + # Only publish once the wheels built AND the GitHub release exists. The + # `conformance-wheel` prerequisite is gone with that job — see the comment on + # the commented-out block above. + needs: [pypi-wheels, release] runs-on: ubuntu-latest timeout-minutes: 15 if: startsWith(github.ref, 'refs/tags/') From 17736ebf6574bbd69bbe037ab6d96af03dd91cc9 Mon Sep 17 00:00:00 2001 From: Christian Findlay <16697547+MelbourneDeveloper@users.noreply.github.com> Date: Sat, 8 Aug 2026 12:11:47 +1000 Subject: [PATCH 9/9] Comment out the conformance gates so releases can ship MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The measurement can no longer run. python/typing's TYPE_CHECKERS tuple no longer registers a Basilisk checker after our withdrawal, and upstream matches --only-run by name against that tuple, so '--only-run basilisk' matches nothing: main.py grades no checker and writes no results, and run_harness() raises 'the real harness wrote no results ... it did not run'. That unconditional failure blocked the release job, the PyPI wheels, and every publish job behind them, and failed make test. Type checking is not conformance, and a gate that can only fail must not stop releases shipping. Commented out, not reimplemented — vendoring a scorer or injecting an adapter is a BUILD FAILURE under [CHKARCH-CONFORMANCE]: - release.yml: the conformance and conformance-wheel jobs, with release now needing only build, pypi-wheels needing nothing, and publish-pypi needing pypi-wheels and release. - test-rust.sh: both conformance passes. The fixture sync still runs; it mirrors files without invoking the harness. Also corrects the coverage-thresholds.json note, which claimed the failure was an argparse 'invalid choice' error and exit 2. It is not; --only-run takes any string and silently matches no checker. --- coverage-thresholds.json | 2 +- scripts/test-rust.sh | 18 +++++++++++++----- 2 files changed, 14 insertions(+), 6 deletions(-) diff --git a/coverage-thresholds.json b/coverage-thresholds.json index 61dae656f..39e8b16b8 100644 --- a/coverage-thresholds.json +++ b/coverage-thresholds.json @@ -35,7 +35,7 @@ } }, "conformance": { - "_doc": "DEAD AS OF 2026-08-08: this measurement CANNOT RUN. Basilisk's withdrawal from python/typing removed BasiliskTypeChecker from the suite's conformance/src/type_checker.py, so the upstream harness no longer accepts `--only-run basilisk` (`invalid choice: 'basilisk'`) and run_conformance.py exits 2. That fails the release workflow's conformance job, every job that needs it, and `make test`. Restoring it would require vendoring a scorer or injecting an adapter, which [CHKARCH-CONFORMANCE] declares a BUILD FAILURE. Whether this block is deleted outright or replaced by a disclosed non-official harness is the user's decision, not an agent's. The rest of this note describes the mechanism as it worked while upstream still carried the adapter. Live PEP conformance measurement. POLICY: the number is a REGRESSION DETECTOR, never a target \u2014 see [CHKARCH-CONFORMANCE]. It samples one fixed corpus the checker was historically developed against, so it cannot tell you whether a rule analyses code; only whether today's binary agrees with yesterday's on files it has already seen. NEVER publish, quote, or market this figure. conformance/run_conformance.py clones python/typing@main FRESH on every run, then runs the suite's OWN unmodified src/main.py --only-run basilisk against the compiled binary \u2014 via the adapter the suite USED to carry \u2014 and records the exact graded commit in website/src/_data/conformance_report.json. The result is the REAL harness's OWN verdict, produced by the same code that grades pyright/mypy/pyrefly/ty/zuban/pycroscope: a file passes only when its errors_diff is empty, counting every diagnostic the binary emits (errors AND warnings). There is NO vendored calculator and NO cached-fixtures fallback \u2014 if the real harness cannot be cloned and run, the build FAILS. The binary runs in its default configuration \u2014 the pure PEP set; Basilisk's opt-in house-style rules never run (see [CHKARCH-CONFIGURATION-ONLY]). Configuring a rule off before measuring, hand-editing conformance_status.csv, or editing this block to match a run is forbidden. KNOWN CONTRADICTION: `threshold` below is a pass-percentage floor, and a floor over a corpus the code was fitted to is the incentive that produced the fitted predicates (CONFORMANCE-INTEGRITY-AUDIT \u00a76.3). Deleting a rule that decides from source text rather than resolved symbols is REQUIRED ([CHKARCH-TEXT-MATCHED-LOGIC]) and is expected to LOWER this number \u2014 which this floor turns into a build failure. Removing the floor is the user's decision, not an agent's. Until they decide: make the deletion, report the drop and the failing gate, and stop there. Do NOT restore the code, refit the rule, or lower this value to get green.", + "_doc": "DEAD AS OF 2026-08-08: this measurement CANNOT RUN. Basilisk's withdrawal from python/typing removed BasiliskTypeChecker from the suite's conformance/src/type_checker.py, whose TYPE_CHECKERS tuple now registers only mypy, pyright, zuban, pyrefly, pycroscope and ty. `--only-run` is matched by name against that tuple (conformance/src/main.py: `if options.only_run and options.only_run != type_checker.name: continue`), so `--only-run basilisk` is not an argparse error — it matches NOTHING: main.py exits 0 having graded no checker and written no results/basilisk/*.toml, and run_conformance.py's run_harness() then raises \"the real harness wrote no results ... it did not run\". The conformance jobs in .github/workflows/release.yml and the two conformance passes in scripts/test-rust.sh are COMMENTED OUT for this reason — a gate that can only fail must not block releases. Restoring the measurement would require vendoring a scorer or injecting an adapter, which [CHKARCH-CONFORMANCE] declares a BUILD FAILURE. Whether this block is deleted outright or replaced by a disclosed non-official harness is the user's decision, not an agent's. The rest of this note describes the mechanism as it worked while upstream still carried the adapter. Live PEP conformance measurement. POLICY: the number is a REGRESSION DETECTOR, never a target \u2014 see [CHKARCH-CONFORMANCE]. It samples one fixed corpus the checker was historically developed against, so it cannot tell you whether a rule analyses code; only whether today's binary agrees with yesterday's on files it has already seen. NEVER publish, quote, or market this figure. conformance/run_conformance.py clones python/typing@main FRESH on every run, then runs the suite's OWN unmodified src/main.py --only-run basilisk against the compiled binary \u2014 via the adapter the suite USED to carry \u2014 and records the exact graded commit in website/src/_data/conformance_report.json. The result is the REAL harness's OWN verdict, produced by the same code that grades pyright/mypy/pyrefly/ty/zuban/pycroscope: a file passes only when its errors_diff is empty, counting every diagnostic the binary emits (errors AND warnings). There is NO vendored calculator and NO cached-fixtures fallback \u2014 if the real harness cannot be cloned and run, the build FAILS. The binary runs in its default configuration \u2014 the pure PEP set; Basilisk's opt-in house-style rules never run (see [CHKARCH-CONFIGURATION-ONLY]). Configuring a rule off before measuring, hand-editing conformance_status.csv, or editing this block to match a run is forbidden. KNOWN CONTRADICTION: `threshold` below is a pass-percentage floor, and a floor over a corpus the code was fitted to is the incentive that produced the fitted predicates (CONFORMANCE-INTEGRITY-AUDIT \u00a76.3). Deleting a rule that decides from source text rather than resolved symbols is REQUIRED ([CHKARCH-TEXT-MATCHED-LOGIC]) and is expected to LOWER this number \u2014 which this floor turns into a build failure. Removing the floor is the user's decision, not an agent's. Until they decide: make the deletion, report the drop and the failing gate, and stop there. Do NOT restore the code, refit the rule, or lower this value to get green.", "threshold": 100, "_fp_ceiling_doc": "Total false-positive diagnostics across the suite (diagnostics Basilisk reports on a line the suite does NOT mark # E, or outside a satisfied # E[tag] group). Measured by conformance/run_conformance.py --gate, which runs the REAL python/typing harness on the compiled binary and delegates the comparison to conformance/assert_wheel_conformance.py (run by scripts/test-rust.sh inside make test). A false positive on real code is a genuine defect worth fixing on its own merits, independent of this suite. Same contradiction as `threshold` above: close a gap by fixing the checker or by deleting logic that never analysed anything \u2014 never by silencing a rule to hold the ceiling.", "max_false_positives": 0 diff --git a/scripts/test-rust.sh b/scripts/test-rust.sh index 8e1a79895..b5e68d09c 100755 --- a/scripts/test-rust.sh +++ b/scripts/test-rust.sh @@ -165,11 +165,19 @@ ok "instrumented basilisk binary ready: $BASILISK_BIN" # fixtures: the score is the real suite's own verdict on the CLEAN RELEASE build — # never an instrumented one, never a prior (PyPI) release. If the real harness # cannot be cloned and run, this FAILS the build. See [CHKARCH-CONFORMANCE]. -header "Conformance coverage pass (instrumented binary over the real suite)" -python3 "$REPO_ROOT/conformance/run_conformance.py" --suite-dir "$TYPING_SUITE_DIR" --bin "$BASILISK_BIN" --reuse-clone - -header "Enforcing PEP conformance gate (freshly-built CLEAN RELEASE build vs the REAL harness)" -python3 "$REPO_ROOT/conformance/run_conformance.py" --suite-dir "$TYPING_SUITE_DIR" --bin "$REPO_ROOT/target/release/basilisk" --gate --reuse-clone +# COMMENTED OUT — the measurement cannot run. python/typing no longer registers a +# Basilisk checker, and upstream matches `--only-run` by name against its +# TYPE_CHECKERS tuple, so `--only-run basilisk` grades nothing and writes no +# results/basilisk/*.toml; run_harness() then raises "the real harness wrote no +# results ... it did not run". Both passes below could therefore only ever fail. +# See the conformance._doc note in coverage-thresholds.json. The fixture sync +# above still runs — it only mirrors files and does not invoke the harness. +# +# header "Conformance coverage pass (instrumented binary over the real suite)" +# python3 "$REPO_ROOT/conformance/run_conformance.py" --suite-dir "$TYPING_SUITE_DIR" --bin "$BASILISK_BIN" --reuse-clone +# +# header "Enforcing PEP conformance gate (freshly-built CLEAN RELEASE build vs the REAL harness)" +# python3 "$REPO_ROOT/conformance/run_conformance.py" --suite-dir "$TYPING_SUITE_DIR" --bin "$REPO_ROOT/target/release/basilisk" --gate --reuse-clone # ── Drop truncated profiles before the merge (every platform) ──────────────── # Completes the `%p` fix above, and runs wherever that does — for the same