-**The current type checker contains inaccuracies and you should not use it as part of your dev pipeline. We are working on removing any misleading analyzers ASAP. Please read below**
-
-## We are auditing the checker and deleting what doesn't hold up
-
-We withdrew our 100% conformance claim and our benchmark figures, and asked to be
+> ## ⚠️ Do not use Basilisk's type checker in your pipeline
+>
+> **The type checker still contains code that isn't doing real type checking, and
+> it is not yet trustworthy.** Some rules decide from the way code is *spelled*
+> rather than what it means, so they can be wrong in both directions — a false
+> error on correct code, or silence where there is a genuine bug. Until the audit
+> below is finished, don't gate CI on `basilisk check`, don't block a merge with
+> it, and don't read a clean run as a clean codebase.
+>
+> The rest of Basilisk — language server, refactoring, formatting, debugging,
+> profiling — does not depend on those rules and is unaffected.
+
+## Restoring trust: audit, delete, and lean on a checker that works
+
+We withdrew our former conformance claim and our benchmark figures, and asked to be
[removed from the official `python/typing` results](https://github.com/python/typing/blob/main/conformance/results/results.html).
The cause was checker logic fitted to the contents of conformance test files
instead of implementing the typing specification generally: rules that matched
the *spelling* of code rather than its meaning. Rename an import or reformat a
file and the answer changed. A score produced that way is not evidence.
+**This was a mistake and a failure to verify — not an attempt to game the suite.**
+Nobody set out to defeat the conformance tests, and nothing was concealed from
+`python/typing`: the submission ran the suite's own unmodified harness, with
+default configuration and every rule enabled. Our process treated the score as
+the goal, matching text raises a score faster than real analysis does, and we
+published without ever asking whether a rule still held when the same program was
+spelled differently. Basilisk's author has published a
+[personal account and apology](https://www.christianfindlay.com/blog/basilisk-conformance-apology).
+
**So we are auditing every rule and deleting the ones that don't do real type
checking.** Not rewriting them, not patching them, not marking them TODO —
deleting them, with a failing test left behind so the gap is visible instead of
hidden. A rule stays only if it decides from the resolved syntax tree and gives
the same answer when the code is spelled differently.
+**Where a rule can't be made reliable in a straightforward way, we will depend on
+a different, established type checker rather than ship our own unreliable version
+of it.** An answer from an engine that has earned trust is worth more to you than
+a Basilisk-branded one that hasn't. No replacement figure gets published until it
+survives off-suite and mutation testing.
+
That means Basilisk gets **smaller** before it gets better. Expect fewer rules,
fewer diagnostics, and a lower conformance number. We will report each drop
rather than avoid it. What is left will be code that is honest about what it
does — nothing else.
-We have not yet decided whether to rebuild the deleted analysis from the
-specification or to drive the extension with an established open-source checker.
-Either way, no new figure gets published until it survives off-suite and
-mutation testing.
+### Basilisk is much more than a type checker
+
+Type checking is one part of it. The rest is a complete Python workflow in a
+single Rust binary — language server, refactoring, formatting, integrated
+debugging, profiling, and the editor extensions — and none of it rests on the
+rules under audit. That is what we are sharpening while the audit runs: make the
+parts that are genuinely useful solid, and remove anything that could hand you a
+misleading result. The point of getting smaller is to end up with a tool you can
+believe.
[Read the full correction →](https://www.basilisk-python.dev/docs/conformance/) •
[Integrity audit →](https://github.com/Nimblesite/Basilisk/blob/main/docs/CONFORMANCE-INTEGRITY-AUDIT.md)
diff --git a/README.md b/README.md
index 39ef2d00..2ba2e7a5 100644
--- a/README.md
+++ b/README.md
@@ -29,32 +29,62 @@
-**The current type checker contains inaccuracies and you should not use it as part of your dev pipeline. We are working on removing any misleading analyzers ASAP. Please read below**
-
-## We are auditing the checker and deleting what doesn't hold up
-
-We withdrew our 100% conformance claim and our benchmark figures, and asked to be
+> ## ⚠️ Do not use Basilisk's type checker in your pipeline
+>
+> **The type checker still contains code that isn't doing real type checking, and
+> it is not yet trustworthy.** Some rules decide from the way code is *spelled*
+> rather than what it means, so they can be wrong in both directions — a false
+> error on correct code, or silence where there is a genuine bug. Until the audit
+> below is finished, don't gate CI on `basilisk check`, don't block a merge with
+> it, and don't read a clean run as a clean codebase.
+>
+> The rest of Basilisk — language server, refactoring, formatting, debugging,
+> profiling — does not depend on those rules and is unaffected.
+
+## Restoring trust: audit, delete, and lean on a checker that works
+
+We withdrew our former conformance claim and our benchmark figures, and asked to be
[removed from the official `python/typing` results](https://github.com/python/typing/blob/main/conformance/results/results.html).
The cause was checker logic fitted to the contents of conformance test files
instead of implementing the typing specification generally: rules that matched
the *spelling* of code rather than its meaning. Rename an import or reformat a
file and the answer changed. A score produced that way is not evidence.
+**This was a mistake and a failure to verify — not an attempt to game the suite.**
+Nobody set out to defeat the conformance tests, and nothing was concealed from
+`python/typing`: the submission ran the suite's own unmodified harness, with
+default configuration and every rule enabled. Our process treated the score as
+the goal, matching text raises a score faster than real analysis does, and we
+published without ever asking whether a rule still held when the same program was
+spelled differently. Basilisk's author has published a
+[personal account and apology](https://www.christianfindlay.com/blog/basilisk-conformance-apology).
+
**So we are auditing every rule and deleting the ones that don't do real type
checking.** Not rewriting them, not patching them, not marking them TODO —
deleting them, with a failing test left behind so the gap is visible instead of
hidden. A rule stays only if it decides from the resolved syntax tree and gives
the same answer when the code is spelled differently.
+**Where a rule can't be made reliable in a straightforward way, we will depend on
+a different, established type checker rather than ship our own unreliable version
+of it.** An answer from an engine that has earned trust is worth more to you than
+a Basilisk-branded one that hasn't. No replacement figure gets published until it
+survives off-suite and mutation testing.
+
That means Basilisk gets **smaller** before it gets better. Expect fewer rules,
fewer diagnostics, and a lower conformance number. We will report each drop
rather than avoid it. What is left will be code that is honest about what it
does — nothing else.
-We have not yet decided whether to rebuild the deleted analysis from the
-specification or to drive the extension with an established open-source checker.
-Either way, no new figure gets published until it survives off-suite and
-mutation testing.
+### Basilisk is much more than a type checker
+
+Type checking is one part of it. The rest is a complete Python workflow in a
+single Rust binary — language server, refactoring, formatting, integrated
+debugging, profiling, and the editor extensions — and none of it rests on the
+rules under audit. That is what we are sharpening while the audit runs: make the
+parts that are genuinely useful solid, and remove anything that could hand you a
+misleading result. The point of getting smaller is to end up with a tool you can
+believe.
[Read the full correction →](https://www.basilisk-python.dev/docs/conformance/) •
[Integrity audit →](docs/CONFORMANCE-INTEGRITY-AUDIT.md)
diff --git a/README.zh.md b/README.zh.md
index 158aa11f..db0fb4f4 100644
--- a/README.zh.md
+++ b/README.zh.md
@@ -29,27 +29,53 @@
-**当前的类型检查器存在不准确之处，请勿将其用于你的开发流水线。我们正在尽快移除任何具有误导性的分析器。详情请见下文**
-
-## 我们正在审计检查器，并删除站不住脚的代码
-
-我们撤回了 100% 的一致性宣称与基准测试数字，并主动请求
+> ## ⚠️ 请勿在流水线中使用 Basilisk 的类型检查器
+>
+> **类型检查器中仍然存在没有做真正类型检查的代码，它目前还不值得信任。** 有些规则
+> 依据的是代码的**写法**而不是含义，因此两个方向上都可能出错 —— 既可能对正确的代码
+> 报出虚假错误，也可能对真实的缺陷保持沉默。在下文所述的审计完成之前，请不要用
+> `basilisk check` 作为 CI 的门禁，不要用它拦截合并，也不要把一次干净的运行结果当作
+> 代码库是干净的。
+>
+> Basilisk 的其余部分 —— 语言服务器、重构、格式化、调试、性能分析 —— 并不依赖这些
+> 规则，因此不受影响。
+
+## 重建信任：审计、删除，并倚重真正可靠的检查器
+
+我们撤回了此前的一致性宣称与基准测试数字，并主动请求
[从官方 `python/typing` 结果中移除](https://github.com/python/typing/blob/main/conformance/results/results.html)。
原因是检查器中存在针对一致性测试文件内容而写的逻辑，而不是对类型规范的通用实现：
那些规则匹配的是代码的**写法**，而不是代码的含义。改一个导入别名或重新格式化文件，
结论就会变。这样得出的分数并不能作为证据。
+**这是一个错误、一次验证上的失职，而不是有意操纵测试套件。** 没有人试图去糊弄一致性
+测试，也没有向 `python/typing` 隐瞒任何东西：提交时运行的是套件自己未经修改的评分
+工具，使用默认配置并启用全部规则。我们的流程把分数当成了目标，而匹配文本比真正做
+分析更快地提高分数；我们在发布之前，始终没有问过这样一个问题 —— 同一个程序换一种
+写法时，这条规则是否依然成立。Basilisk 作者已发表
+[个人说明与致歉](https://www.christianfindlay.com/blog/basilisk-conformance-apology)。
+
**因此，我们正在逐条审计规则，并删除那些没有做真正类型检查的规则。** 不是重写，不是
打补丁，也不是标一个 TODO —— 是删除，并留下一个失败的测试，让这个缺口可见而不是被
掩盖。一条规则只有在依据已解析的语法树做判断、并且在代码换一种写法时给出相同结论的
情况下，才会保留。
+**如果一条规则无法以直截了当的方式做到可靠，我们会转而依赖另一个成熟的类型检查器，
+而不是端出我们自己那份不可靠的实现。** 一个已经赢得信任的引擎给出的答案，对你而言
+比一个挂着 Basilisk 名号却没有赢得信任的答案更有价值。在通过套件之外的用例与变异
+测试之前，我们不会发布任何替代数字。
+
这意味着 Basilisk 会**先变小，再变好**。规则会更少，诊断会更少，一致性数字也会更低。
每一次下降我们都会如实报告，而不是设法回避。留下来的，将是对自己所做之事诚实的代码
—— 仅此而已。
-至于被删掉的分析是按规范重建，还是让扩展由一个成熟的开源检查器驱动，我们尚未决定。
-无论走哪条路，新的数字只有在经受住套件之外的用例与变异测试后才会发布。
+### Basilisk 远不只是一个类型检查器
+
+类型检查只是其中一部分。其余部分是装在单个 Rust 二进制文件里的完整 Python 工作流
+—— 语言服务器、重构、格式化、集成调试、性能分析，以及各个编辑器扩展 —— 它们都不
+建立在正在接受审计的规则之上。这正是我们在审计期间着力打磨的地方：把真正有用的部分
+做扎实，并移除任何可能给出误导性结果的东西。变小的意义，是最终得到一个你可以信赖的
+工具。
[阅读完整更正 →](https://www.basilisk-python.dev/zh/docs/conformance/) •
[完整性审计 →](docs/CONFORMANCE-INTEGRITY-AUDIT.md)
diff --git a/basilisk-zed/README.md b/basilisk-zed/README.md
index 18dc2fb0..c2a6de9b 100644
--- a/basilisk-zed/README.md
+++ b/basilisk-zed/README.md
@@ -10,9 +10,15 @@ Basilisk is an open-source Python type checker and language server built in Rust
-**The current type checker contains inaccuracies and you should not use it as part of your dev pipeline. We are working on removing any misleading analyzers ASAP. Please read below**
-
-> **We withdrew the typing conformance results.** The 100% claim and the benchmark figures are retracted, and Basilisk was [removed from the official results](https://github.com/python/typing/blob/main/conformance/results/results.html) at our request. Whether we rebuild the checker from the specification or drive the extension with an established open-source checker, we are building Basilisk into an accurate Python development experience. [Read the correction](https://www.basilisk-python.dev/docs/conformance/).
+> ## ⚠️ Do not use Basilisk's type checker in your pipeline
+>
+> **The type checker still contains code that isn't doing real type checking, and it is not yet trustworthy.** Some rules decide from the way code is *spelled* rather than what it means, so they can be wrong in both directions — a false error on correct code, or silence where there is a genuine bug. Don't gate CI on it, and don't read a clean run as a clean codebase. Our former conformance claim and our benchmark figures are withdrawn, and Basilisk was [removed from the official results](https://github.com/python/typing/blob/main/conformance/results/results.html) at our request.
+>
+> **This was a mistake and a failure to verify — not an attempt to game the suite.** Nothing was concealed from `python/typing`; the submission ran the suite's own unmodified harness, and we published on a green run without ever checking whether our rules survived a semantics-preserving change. Basilisk's author has published a [personal account and apology](https://www.christianfindlay.com/blog/basilisk-conformance-apology).
+>
+> **We are auditing every rule and deleting the ones that don't hold up** — not rewriting them, not patching them, with a failing test left behind so the gap stays visible. Where a rule can't be made reliable in a straightforward way, we will depend on a different, established type checker rather than ship our own unreliable version of it.
+>
+> **Basilisk is much more than a type checker.** The language server, refactoring, formatting, debugging, and profiling don't rest on the rules under audit — those are what we are sharpening while it runs, removing anything that could hand you a misleading result. We are doing this to restore trust and turn Basilisk back into a tool you can believe. [Read the correction](https://www.basilisk-python.dev/docs/conformance/).
## Install
diff --git a/basilisk-zed/README.zh.md b/basilisk-zed/README.zh.md
index 0b479daa..fdbf7c81 100644
--- a/basilisk-zed/README.zh.md
+++ b/basilisk-zed/README.zh.md
@@ -12,9 +12,15 @@ Basilisk 是用 Rust 打造的开源 Python 类型检查器与语言服务器：
-**当前的类型检查器存在不准确之处，请勿将其用于你的开发流水线。我们正在尽快移除任何具有误导性的分析器。详情请见下文**
-
-> **我们撤回了类型一致性结果。** 100% 的宣称与基准测试数字均已撤回，并主动请求[从官方结果中移除](https://github.com/python/typing/blob/main/conformance/results/results.html)。无论是按规范重建检查器，还是让扩展由一个成熟的开源检查器驱动，我们都在把 Basilisk 打造成准确的 Python 开发体验。[阅读更正](https://www.basilisk-python.dev/zh/docs/conformance/)。
+> ## ⚠️ 请勿在流水线中使用 Basilisk 的类型检查器
+>
+> **类型检查器中仍然存在没有做真正类型检查的代码，它目前还不值得信任。** 有些规则依据的是代码的**写法**而不是含义，因此两个方向上都可能出错 —— 既可能对正确的代码报出虚假错误，也可能对真实的缺陷保持沉默。请不要用它作为 CI 的门禁，也不要把一次干净的运行结果当作代码库是干净的。此前的一致性宣称与基准测试数字均已撤回，并主动请求[从官方结果中移除](https://github.com/python/typing/blob/main/conformance/results/results.html)。
+>
+> **这是一个错误、一次验证上的失职，而不是有意操纵测试套件。** 我们没有向 `python/typing` 隐瞒任何东西：提交时运行的是套件自己未经修改的评分工具；我们仅凭一次全绿的运行就发布了结果，却从未检查过我们的规则能否经受住保持语义的改写。Basilisk 作者已发表[个人说明与致歉](https://www.christianfindlay.com/blog/basilisk-conformance-apology)。
+>
+> **我们正在逐条审计规则，并删除那些站不住脚的规则** —— 不是重写，也不是打补丁，而是删除，并留下一个失败的测试，让缺口保持可见。如果一条规则无法以直截了当的方式做到可靠，我们会转而依赖另一个成熟的类型检查器，而不是端出我们自己那份不可靠的实现。
+>
+> **Basilisk 远不只是一个类型检查器。** 语言服务器、重构、格式化、调试与性能分析都不建立在正在接受审计的规则之上 —— 审计期间，这些正是我们着力打磨的部分，并移除任何可能给出误导性结果的东西。我们这样做，是为了重建信任，把 Basilisk 变回一个你可以信赖的工具。[阅读更正](https://www.basilisk-python.dev/zh/docs/conformance/)。
## 安装
diff --git a/basilisk.nvim/README.md b/basilisk.nvim/README.md
index 10652da4..0c75c310 100644
--- a/basilisk.nvim/README.md
+++ b/basilisk.nvim/README.md
@@ -10,9 +10,15 @@ Basilisk is an open-source Python type checker and language server built in Rust
-**The current type checker contains inaccuracies and you should not use it as part of your dev pipeline. We are working on removing any misleading analyzers ASAP. Please read below**
-
-> **We withdrew the typing conformance results.** The 100% claim and the benchmark figures are retracted, and Basilisk was [removed from the official results](https://github.com/python/typing/blob/main/conformance/results/results.html) at our request. Whether we rebuild the checker from the specification or drive the extension with an established open-source checker, we are building Basilisk into an accurate Python development experience. [Read the correction](https://www.basilisk-python.dev/docs/conformance/).
+> ## ⚠️ Do not use Basilisk's type checker in your pipeline
+>
+> **The type checker still contains code that isn't doing real type checking, and it is not yet trustworthy.** Some rules decide from the way code is *spelled* rather than what it means, so they can be wrong in both directions — a false error on correct code, or silence where there is a genuine bug. Don't gate CI on it, and don't read a clean run as a clean codebase. Our former conformance claim and our benchmark figures are withdrawn, and Basilisk was [removed from the official results](https://github.com/python/typing/blob/main/conformance/results/results.html) at our request.
+>
+> **This was a mistake and a failure to verify — not an attempt to game the suite.** Nothing was concealed from `python/typing`; the submission ran the suite's own unmodified harness, and we published on a green run without ever checking whether our rules survived a semantics-preserving change. Basilisk's author has published a [personal account and apology](https://www.christianfindlay.com/blog/basilisk-conformance-apology).
+>
+> **We are auditing every rule and deleting the ones that don't hold up** — not rewriting them, not patching them, with a failing test left behind so the gap stays visible. Where a rule can't be made reliable in a straightforward way, we will depend on a different, established type checker rather than ship our own unreliable version of it.
+>
+> **Basilisk is much more than a type checker.** The language server, refactoring, formatting, debugging, and profiling don't rest on the rules under audit — those are what we are sharpening while it runs, removing anything that could hand you a misleading result. We are doing this to restore trust and turn Basilisk back into a tool you can believe. [Read the correction](https://www.basilisk-python.dev/docs/conformance/).
## Role in Basilisk
diff --git a/basilisk.nvim/README.zh.md b/basilisk.nvim/README.zh.md
index b6cc8865..f128a4a1 100644
--- a/basilisk.nvim/README.zh.md
+++ b/basilisk.nvim/README.zh.md
@@ -12,9 +12,15 @@ Basilisk 是用 Rust 打造的开源 Python 类型检查器与语言服务器：
-**当前的类型检查器存在不准确之处，请勿将其用于你的开发流水线。我们正在尽快移除任何具有误导性的分析器。详情请见下文**
-
-> **我们撤回了类型一致性结果。** 100% 的宣称与基准测试数字均已撤回，并主动请求[从官方结果中移除](https://github.com/python/typing/blob/main/conformance/results/results.html)。无论是按规范重建检查器，还是让扩展由一个成熟的开源检查器驱动，我们都在把 Basilisk 打造成准确的 Python 开发体验。[阅读更正](https://www.basilisk-python.dev/zh/docs/conformance/)。
+> ## ⚠️ 请勿在流水线中使用 Basilisk 的类型检查器
+>
+> **类型检查器中仍然存在没有做真正类型检查的代码，它目前还不值得信任。** 有些规则依据的是代码的**写法**而不是含义，因此两个方向上都可能出错 —— 既可能对正确的代码报出虚假错误，也可能对真实的缺陷保持沉默。请不要用它作为 CI 的门禁，也不要把一次干净的运行结果当作代码库是干净的。此前的一致性宣称与基准测试数字均已撤回，并主动请求[从官方结果中移除](https://github.com/python/typing/blob/main/conformance/results/results.html)。
+>
+> **这是一个错误、一次验证上的失职，而不是有意操纵测试套件。** 我们没有向 `python/typing` 隐瞒任何东西：提交时运行的是套件自己未经修改的评分工具；我们仅凭一次全绿的运行就发布了结果，却从未检查过我们的规则能否经受住保持语义的改写。Basilisk 作者已发表[个人说明与致歉](https://www.christianfindlay.com/blog/basilisk-conformance-apology)。
+>
+> **我们正在逐条审计规则，并删除那些站不住脚的规则** —— 不是重写，也不是打补丁，而是删除，并留下一个失败的测试，让缺口保持可见。如果一条规则无法以直截了当的方式做到可靠，我们会转而依赖另一个成熟的类型检查器，而不是端出我们自己那份不可靠的实现。
+>
+> **Basilisk 远不只是一个类型检查器。** 语言服务器、重构、格式化、调试与性能分析都不建立在正在接受审计的规则之上 —— 审计期间，这些正是我们着力打磨的部分，并移除任何可能给出误导性结果的东西。我们这样做，是为了重建信任，把 Basilisk 变回一个你可以信赖的工具。[阅读更正](https://www.basilisk-python.dev/zh/docs/conformance/)。
## 在 Basilisk 中的角色
diff --git a/coverage-thresholds.json b/coverage-thresholds.json
index 00a4431d..39e8b16b 100644
--- a/coverage-thresholds.json
+++ b/coverage-thresholds.json
@@ -35,7 +35,7 @@
}
},
"conformance": {
- "_doc": "Live PEP conformance measurement. POLICY: the number is a REGRESSION DETECTOR, never a target \u2014 see [CHKARCH-CONFORMANCE]. It samples one fixed corpus the checker was historically developed against, so it cannot tell you whether a rule analyses code; only whether today's binary agrees with yesterday's on files it has already seen. NEVER publish, quote, or market this figure. conformance/run_conformance.py clones python/typing@main FRESH on every run, then runs the suite's OWN unmodified src/main.py --only-run basilisk against the compiled binary \u2014 whose conformance/src/type_checker.py already ships the official BasiliskTypeChecker \u2014 and records the exact graded commit in website/src/_data/conformance_report.json. The result is the REAL harness's OWN verdict, produced by the same code that grades pyright/mypy/pyrefly/ty/zuban/pycroscope: a file passes only when its errors_diff is empty, counting every diagnostic the binary emits (errors AND warnings). There is NO vendored calculator and NO cached-fixtures fallback \u2014 if the real harness cannot be cloned and run, the build FAILS. The binary runs in its default configuration \u2014 the pure PEP set; Basilisk's opt-in house-style rules never run (see [CHKARCH-CONFIGURATION-ONLY]). Configuring a rule off before measuring, hand-editing conformance_status.csv, or editing this block to match a run is forbidden. KNOWN CONTRADICTION: `threshold` below is a pass-percentage floor, and a floor over a corpus the code was fitted to is the incentive that produced the fitted predicates (CONFORMANCE-INTEGRITY-AUDIT \u00a76.3). Deleting a rule that decides from source text rather than resolved symbols is REQUIRED ([CHKARCH-TEXT-MATCHED-LOGIC]) and is expected to LOWER this number \u2014 which this floor turns into a build failure. Removing the floor is the user's decision, not an agent's. Until they decide: make the deletion, report the drop and the failing gate, and stop there. Do NOT restore the code, refit the rule, or lower this value to get green.",
+ "_doc": "DEAD AS OF 2026-08-08: this measurement CANNOT RUN. Basilisk's withdrawal from python/typing removed BasiliskTypeChecker from the suite's conformance/src/type_checker.py, whose TYPE_CHECKERS tuple now registers only mypy, pyright, zuban, pyrefly, pycroscope and ty. `--only-run` is matched by name against that tuple (conformance/src/main.py: `if options.only_run and options.only_run != type_checker.name: continue`), so `--only-run basilisk` is not an argparse error — it matches NOTHING: main.py exits 0 having graded no checker and written no results/basilisk/*.toml, and run_conformance.py's run_harness() then raises \"the real harness wrote no results ... it did not run\". The conformance jobs in .github/workflows/release.yml and the two conformance passes in scripts/test-rust.sh are COMMENTED OUT for this reason — a gate that can only fail must not block releases. Restoring the measurement would require vendoring a scorer or injecting an adapter, which [CHKARCH-CONFORMANCE] declares a BUILD FAILURE. Whether this block is deleted outright or replaced by a disclosed non-official harness is the user's decision, not an agent's. The rest of this note describes the mechanism as it worked while upstream still carried the adapter. Live PEP conformance measurement. POLICY: the number is a REGRESSION DETECTOR, never a target \u2014 see [CHKARCH-CONFORMANCE]. It samples one fixed corpus the checker was historically developed against, so it cannot tell you whether a rule analyses code; only whether today's binary agrees with yesterday's on files it has already seen. NEVER publish, quote, or market this figure. conformance/run_conformance.py clones python/typing@main FRESH on every run, then runs the suite's OWN unmodified src/main.py --only-run basilisk against the compiled binary \u2014 via the adapter the suite USED to carry \u2014 and records the exact graded commit in website/src/_data/conformance_report.json. The result is the REAL harness's OWN verdict, produced by the same code that grades pyright/mypy/pyrefly/ty/zuban/pycroscope: a file passes only when its errors_diff is empty, counting every diagnostic the binary emits (errors AND warnings). There is NO vendored calculator and NO cached-fixtures fallback \u2014 if the real harness cannot be cloned and run, the build FAILS. The binary runs in its default configuration \u2014 the pure PEP set; Basilisk's opt-in house-style rules never run (see [CHKARCH-CONFIGURATION-ONLY]). Configuring a rule off before measuring, hand-editing conformance_status.csv, or editing this block to match a run is forbidden. KNOWN CONTRADICTION: `threshold` below is a pass-percentage floor, and a floor over a corpus the code was fitted to is the incentive that produced the fitted predicates (CONFORMANCE-INTEGRITY-AUDIT \u00a76.3). Deleting a rule that decides from source text rather than resolved symbols is REQUIRED ([CHKARCH-TEXT-MATCHED-LOGIC]) and is expected to LOWER this number \u2014 which this floor turns into a build failure. Removing the floor is the user's decision, not an agent's. Until they decide: make the deletion, report the drop and the failing gate, and stop there. Do NOT restore the code, refit the rule, or lower this value to get green.",
"threshold": 100,
"_fp_ceiling_doc": "Total false-positive diagnostics across the suite (diagnostics Basilisk reports on a line the suite does NOT mark # E, or outside a satisfied # E[tag] group). Measured by conformance/run_conformance.py --gate, which runs the REAL python/typing harness on the compiled binary and delegates the comparison to conformance/assert_wheel_conformance.py (run by scripts/test-rust.sh inside make test). A false positive on real code is a genuine defect worth fixing on its own merits, independent of this suite. Same contradiction as `threshold` above: close a gap by fixing the checker or by deleting logic that never analysed anything \u2014 never by silencing a rule to hold the ceiling.",
"max_false_positives": 0
diff --git a/docs/CONFORMANCE-INTEGRITY-AUDIT.md b/docs/CONFORMANCE-INTEGRITY-AUDIT.md
index 936bb690..d8b56006 100644
--- a/docs/CONFORMANCE-INTEGRITY-AUDIT.md
+++ b/docs/CONFORMANCE-INTEGRITY-AUDIT.md
@@ -14,6 +14,8 @@ We found that at least one Basilisk rule earns its conformance result by pattern
We did not find this ourselves. It was reported from outside, in [issue #379](https://github.com/Nimblesite/Basilisk/issues/379), from a [public reproduction](https://x.com/cyanchanges/status/2083115048143364512). That is itself a finding, and it is covered in §6.
+**On intent.** This was a verification failure, not an attempt to game the conformance suite. Nobody set out to defeat it, and nothing was concealed from `python/typing`: the submission ran the suite's own unmodified harness, with default configuration and every specification rule enabled — no special-casing of the harness, no doctored output, no disabled rules, no edited results file. What went wrong is set out in §6: the process named the score as the target, matching source text raises a score faster than real analysis does, and the control that would have caught the drift — semantics-preserving mutation (§7) — did not exist, so every green run reinforced a conclusion we had no basis for. When the defect was demonstrated we accepted it and requested our own removal from the results table. Basilisk's author has published a [personal account and apology](https://www.christianfindlay.com/blog/basilisk-conformance-apology). This document describes defective engineering and a failure of review; it does not describe deception, and nothing in it should be read as alleging any.
+
Our conformance number is self-measured. Where a passing file is carried by predicates shaped to that file, the honest statement is that **the file passes and the rule is not implemented**. That is the case for the files listed in §3.
---
diff --git a/docs/readme/README.src.md b/docs/readme/README.src.md
index 7a114fa0..26348731 100644
--- a/docs/readme/README.src.md
+++ b/docs/readme/README.src.md
@@ -40,32 +40,62 @@
-**The current type checker contains inaccuracies and you should not use it as part of your dev pipeline. We are working on removing any misleading analyzers ASAP. Please read below**
-
-## We are auditing the checker and deleting what doesn't hold up
-
-We withdrew our 100% conformance claim and our benchmark figures, and asked to be
+> ## ⚠️ Do not use Basilisk's type checker in your pipeline
+>
+> **The type checker still contains code that isn't doing real type checking, and
+> it is not yet trustworthy.** Some rules decide from the way code is *spelled*
+> rather than what it means, so they can be wrong in both directions — a false
+> error on correct code, or silence where there is a genuine bug. Until the audit
+> below is finished, don't gate CI on `basilisk check`, don't block a merge with
+> it, and don't read a clean run as a clean codebase.
+>
+> The rest of Basilisk — language server, refactoring, formatting, debugging,
+> profiling — does not depend on those rules and is unaffected.
+
+## Restoring trust: audit, delete, and lean on a checker that works
+
+We withdrew our former conformance claim and our benchmark figures, and asked to be
[removed from the official `python/typing` results](https://github.com/python/typing/blob/main/conformance/results/results.html).
The cause was checker logic fitted to the contents of conformance test files
instead of implementing the typing specification generally: rules that matched
the *spelling* of code rather than its meaning. Rename an import or reformat a
file and the answer changed. A score produced that way is not evidence.
+**This was a mistake and a failure to verify — not an attempt to game the suite.**
+Nobody set out to defeat the conformance tests, and nothing was concealed from
+`python/typing`: the submission ran the suite's own unmodified harness, with
+default configuration and every rule enabled. Our process treated the score as
+the goal, matching text raises a score faster than real analysis does, and we
+published without ever asking whether a rule still held when the same program was
+spelled differently. Basilisk's author has published a
+[personal account and apology](https://www.christianfindlay.com/blog/basilisk-conformance-apology).
+
**So we are auditing every rule and deleting the ones that don't do real type
checking.** Not rewriting them, not patching them, not marking them TODO —
deleting them, with a failing test left behind so the gap is visible instead of
hidden. A rule stays only if it decides from the resolved syntax tree and gives
the same answer when the code is spelled differently.
+**Where a rule can't be made reliable in a straightforward way, we will depend on
+a different, established type checker rather than ship our own unreliable version
+of it.** An answer from an engine that has earned trust is worth more to you than
+a Basilisk-branded one that hasn't. No replacement figure gets published until it
+survives off-suite and mutation testing.
+
That means Basilisk gets **smaller** before it gets better. Expect fewer rules,
fewer diagnostics, and a lower conformance number. We will report each drop
rather than avoid it. What is left will be code that is honest about what it
does — nothing else.
-We have not yet decided whether to rebuild the deleted analysis from the
-specification or to drive the extension with an established open-source checker.
-Either way, no new figure gets published until it survives off-suite and
-mutation testing.
+### Basilisk is much more than a type checker
+
+Type checking is one part of it. The rest is a complete Python workflow in a
+single Rust binary — language server, refactoring, formatting, integrated
+debugging, profiling, and the editor extensions — and none of it rests on the
+rules under audit. That is what we are sharpening while the audit runs: make the
+parts that are genuinely useful solid, and remove anything that could hand you a
+misleading result. The point of getting smaller is to end up with a tool you can
+believe.
[Read the full correction →](https://www.basilisk-python.dev/docs/conformance/) •
[Integrity audit →](docs/CONFORMANCE-INTEGRITY-AUDIT.md)
diff --git a/docs/readme/README.zh.src.md b/docs/readme/README.zh.src.md
index 27b009f4..5cc9ef96 100644
--- a/docs/readme/README.zh.src.md
+++ b/docs/readme/README.zh.src.md
@@ -36,27 +36,53 @@
-**当前的类型检查器存在不准确之处，请勿将其用于你的开发流水线。我们正在尽快移除任何具有误导性的分析器。详情请见下文**
-
-## 我们正在审计检查器，并删除站不住脚的代码
-
-我们撤回了 100% 的一致性宣称与基准测试数字，并主动请求
+> ## ⚠️ 请勿在流水线中使用 Basilisk 的类型检查器
+>
+> **类型检查器中仍然存在没有做真正类型检查的代码，它目前还不值得信任。** 有些规则
+> 依据的是代码的**写法**而不是含义，因此两个方向上都可能出错 —— 既可能对正确的代码
+> 报出虚假错误，也可能对真实的缺陷保持沉默。在下文所述的审计完成之前，请不要用
+> `basilisk check` 作为 CI 的门禁，不要用它拦截合并，也不要把一次干净的运行结果当作
+> 代码库是干净的。
+>
+> Basilisk 的其余部分 —— 语言服务器、重构、格式化、调试、性能分析 —— 并不依赖这些
+> 规则，因此不受影响。
+
+## 重建信任：审计、删除，并倚重真正可靠的检查器
+
+我们撤回了此前的一致性宣称与基准测试数字，并主动请求
[从官方 `python/typing` 结果中移除](https://github.com/python/typing/blob/main/conformance/results/results.html)。
原因是检查器中存在针对一致性测试文件内容而写的逻辑，而不是对类型规范的通用实现：
那些规则匹配的是代码的**写法**，而不是代码的含义。改一个导入别名或重新格式化文件，
结论就会变。这样得出的分数并不能作为证据。
+**这是一个错误、一次验证上的失职，而不是有意操纵测试套件。** 没有人试图去糊弄一致性
+测试，也没有向 `python/typing` 隐瞒任何东西：提交时运行的是套件自己未经修改的评分
+工具，使用默认配置并启用全部规则。我们的流程把分数当成了目标，而匹配文本比真正做
+分析更快地提高分数；我们在发布之前，始终没有问过这样一个问题 —— 同一个程序换一种
+写法时，这条规则是否依然成立。Basilisk 作者已发表
+[个人说明与致歉](https://www.christianfindlay.com/blog/basilisk-conformance-apology)。
+
**因此，我们正在逐条审计规则，并删除那些没有做真正类型检查的规则。** 不是重写，不是
打补丁，也不是标一个 TODO —— 是删除，并留下一个失败的测试，让这个缺口可见而不是被
掩盖。一条规则只有在依据已解析的语法树做判断、并且在代码换一种写法时给出相同结论的
情况下，才会保留。
+**如果一条规则无法以直截了当的方式做到可靠，我们会转而依赖另一个成熟的类型检查器，
+而不是端出我们自己那份不可靠的实现。** 一个已经赢得信任的引擎给出的答案，对你而言
+比一个挂着 Basilisk 名号却没有赢得信任的答案更有价值。在通过套件之外的用例与变异
+测试之前，我们不会发布任何替代数字。
+
这意味着 Basilisk 会**先变小，再变好**。规则会更少，诊断会更少，一致性数字也会更低。
每一次下降我们都会如实报告，而不是设法回避。留下来的，将是对自己所做之事诚实的代码
—— 仅此而已。
-至于被删掉的分析是按规范重建，还是让扩展由一个成熟的开源检查器驱动，我们尚未决定。
-无论走哪条路，新的数字只有在经受住套件之外的用例与变异测试后才会发布。
+### Basilisk 远不只是一个类型检查器
+
+类型检查只是其中一部分。其余部分是装在单个 Rust 二进制文件里的完整 Python 工作流
+—— 语言服务器、重构、格式化、集成调试、性能分析，以及各个编辑器扩展 —— 它们都不
+建立在正在接受审计的规则之上。这正是我们在审计期间着力打磨的地方：把真正有用的部分
+做扎实，并移除任何可能给出误导性结果的东西。变小的意义，是最终得到一个你可以信赖的
+工具。
[阅读完整更正 →](https://www.basilisk-python.dev/zh/docs/conformance/) •
[完整性审计 →](docs/CONFORMANCE-INTEGRITY-AUDIT.md)
diff --git a/scripts/test-rust.sh b/scripts/test-rust.sh
index 8e1a7989..b5e68d09 100755
--- a/scripts/test-rust.sh
+++ b/scripts/test-rust.sh
@@ -165,11 +165,19 @@ ok "instrumented basilisk binary ready: $BASILISK_BIN"
# fixtures: the score is the real suite's own verdict on the CLEAN RELEASE build —
# never an instrumented one, never a prior (PyPI) release. If the real harness
# cannot be cloned and run, this FAILS the build. See [CHKARCH-CONFORMANCE].
-header "Conformance coverage pass (instrumented binary over the real suite)"
-python3 "$REPO_ROOT/conformance/run_conformance.py" --suite-dir "$TYPING_SUITE_DIR" --bin "$BASILISK_BIN" --reuse-clone
-
-header "Enforcing PEP conformance gate (freshly-built CLEAN RELEASE build vs the REAL harness)"
-python3 "$REPO_ROOT/conformance/run_conformance.py" --suite-dir "$TYPING_SUITE_DIR" --bin "$REPO_ROOT/target/release/basilisk" --gate --reuse-clone
+# COMMENTED OUT — the measurement cannot run. python/typing no longer registers a
+# Basilisk checker, and upstream matches `--only-run` by name against its
+# TYPE_CHECKERS tuple, so `--only-run basilisk` grades nothing and writes no
+# results/basilisk/*.toml; run_harness() then raises "the real harness wrote no
+# results ... it did not run". Both passes below could therefore only ever fail.
+# See the conformance._doc note in coverage-thresholds.json. The fixture sync
+# above still runs — it only mirrors files and does not invoke the harness.
+#
+# header "Conformance coverage pass (instrumented binary over the real suite)"
+# python3 "$REPO_ROOT/conformance/run_conformance.py" --suite-dir "$TYPING_SUITE_DIR" --bin "$BASILISK_BIN" --reuse-clone
+#
+# header "Enforcing PEP conformance gate (freshly-built CLEAN RELEASE build vs the REAL harness)"
+# python3 "$REPO_ROOT/conformance/run_conformance.py" --suite-dir "$TYPING_SUITE_DIR" --bin "$REPO_ROOT/target/release/basilisk" --gate --reuse-clone
# ── Drop truncated profiles before the merge (every platform) ────────────────
# Completes the `%p` fix above, and runs wherever that does — for the same
diff --git a/vscode-extension/README.md b/vscode-extension/README.md
index 7de37669..c7c5432a 100644
--- a/vscode-extension/README.md
+++ b/vscode-extension/README.md
@@ -29,32 +29,62 @@
-**The current type checker contains inaccuracies and you should not use it as part of your dev pipeline. We are working on removing any misleading analyzers ASAP. Please read below**
-
-## We are auditing the checker and deleting what doesn't hold up
-
-We withdrew our 100% conformance claim and our benchmark figures, and asked to be
+> ## ⚠️ Do not use Basilisk's type checker in your pipeline
+>
+> **The type checker still contains code that isn't doing real type checking, and
+> it is not yet trustworthy.** Some rules decide from the way code is *spelled*
+> rather than what it means, so they can be wrong in both directions — a false
+> error on correct code, or silence where there is a genuine bug. Until the audit
+> below is finished, don't gate CI on `basilisk check`, don't block a merge with
+> it, and don't read a clean run as a clean codebase.
+>
+> The rest of Basilisk — language server, refactoring, formatting, debugging,
+> profiling — does not depend on those rules and is unaffected.
+
+## Restoring trust: audit, delete, and lean on a checker that works
+
+We withdrew our former conformance claim and our benchmark figures, and asked to be
[removed from the official `python/typing` results](https://github.com/python/typing/blob/main/conformance/results/results.html).
The cause was checker logic fitted to the contents of conformance test files
instead of implementing the typing specification generally: rules that matched
the *spelling* of code rather than its meaning. Rename an import or reformat a
file and the answer changed. A score produced that way is not evidence.
+**This was a mistake and a failure to verify — not an attempt to game the suite.**
+Nobody set out to defeat the conformance tests, and nothing was concealed from
+`python/typing`: the submission ran the suite's own unmodified harness, with
+default configuration and every rule enabled. Our process treated the score as
+the goal, matching text raises a score faster than real analysis does, and we
+published without ever asking whether a rule still held when the same program was
+spelled differently. Basilisk's author has published a
+[personal account and apology](https://www.christianfindlay.com/blog/basilisk-conformance-apology).
+
**So we are auditing every rule and deleting the ones that don't do real type
checking.** Not rewriting them, not patching them, not marking them TODO —
deleting them, with a failing test left behind so the gap is visible instead of
hidden. A rule stays only if it decides from the resolved syntax tree and gives
the same answer when the code is spelled differently.
+**Where a rule can't be made reliable in a straightforward way, we will depend on
+a different, established type checker rather than ship our own unreliable version
+of it.** An answer from an engine that has earned trust is worth more to you than
+a Basilisk-branded one that hasn't. No replacement figure gets published until it
+survives off-suite and mutation testing.
+
That means Basilisk gets **smaller** before it gets better. Expect fewer rules,
fewer diagnostics, and a lower conformance number. We will report each drop
rather than avoid it. What is left will be code that is honest about what it
does — nothing else.
-We have not yet decided whether to rebuild the deleted analysis from the
-specification or to drive the extension with an established open-source checker.
-Either way, no new figure gets published until it survives off-suite and
-mutation testing.
+### Basilisk is much more than a type checker
+
+Type checking is one part of it. The rest is a complete Python workflow in a
+single Rust binary — language server, refactoring, formatting, integrated
+debugging, profiling, and the editor extensions — and none of it rests on the
+rules under audit. That is what we are sharpening while the audit runs: make the
+parts that are genuinely useful solid, and remove anything that could hand you a
+misleading result. The point of getting smaller is to end up with a tool you can
+believe.
[Read the full correction →](https://www.basilisk-python.dev/docs/conformance/) •
[Integrity audit →](https://github.com/Nimblesite/Basilisk/blob/main/docs/CONFORMANCE-INTEGRITY-AUDIT.md)
diff --git a/vscode-extension/README.zh.md b/vscode-extension/README.zh.md
index efd80bb9..41ab71d5 100644
--- a/vscode-extension/README.zh.md
+++ b/vscode-extension/README.zh.md
@@ -29,27 +29,53 @@
-**当前的类型检查器存在不准确之处，请勿将其用于你的开发流水线。我们正在尽快移除任何具有误导性的分析器。详情请见下文**
-
-## 我们正在审计检查器，并删除站不住脚的代码
-
-我们撤回了 100% 的一致性宣称与基准测试数字，并主动请求
+> ## ⚠️ 请勿在流水线中使用 Basilisk 的类型检查器
+>
+> **类型检查器中仍然存在没有做真正类型检查的代码，它目前还不值得信任。** 有些规则
+> 依据的是代码的**写法**而不是含义，因此两个方向上都可能出错 —— 既可能对正确的代码
+> 报出虚假错误，也可能对真实的缺陷保持沉默。在下文所述的审计完成之前，请不要用
+> `basilisk check` 作为 CI 的门禁，不要用它拦截合并，也不要把一次干净的运行结果当作
+> 代码库是干净的。
+>
+> Basilisk 的其余部分 —— 语言服务器、重构、格式化、调试、性能分析 —— 并不依赖这些
+> 规则，因此不受影响。
+
+## 重建信任：审计、删除，并倚重真正可靠的检查器
+
+我们撤回了此前的一致性宣称与基准测试数字，并主动请求
[从官方 `python/typing` 结果中移除](https://github.com/python/typing/blob/main/conformance/results/results.html)。
原因是检查器中存在针对一致性测试文件内容而写的逻辑，而不是对类型规范的通用实现：
那些规则匹配的是代码的**写法**，而不是代码的含义。改一个导入别名或重新格式化文件，
结论就会变。这样得出的分数并不能作为证据。
+**这是一个错误、一次验证上的失职，而不是有意操纵测试套件。** 没有人试图去糊弄一致性
+测试，也没有向 `python/typing` 隐瞒任何东西：提交时运行的是套件自己未经修改的评分
+工具，使用默认配置并启用全部规则。我们的流程把分数当成了目标，而匹配文本比真正做
+分析更快地提高分数；我们在发布之前，始终没有问过这样一个问题 —— 同一个程序换一种
+写法时，这条规则是否依然成立。Basilisk 作者已发表
+[个人说明与致歉](https://www.christianfindlay.com/blog/basilisk-conformance-apology)。
+
**因此，我们正在逐条审计规则，并删除那些没有做真正类型检查的规则。** 不是重写，不是
打补丁，也不是标一个 TODO —— 是删除，并留下一个失败的测试，让这个缺口可见而不是被
掩盖。一条规则只有在依据已解析的语法树做判断、并且在代码换一种写法时给出相同结论的
情况下，才会保留。
+**如果一条规则无法以直截了当的方式做到可靠，我们会转而依赖另一个成熟的类型检查器，
+而不是端出我们自己那份不可靠的实现。** 一个已经赢得信任的引擎给出的答案，对你而言
+比一个挂着 Basilisk 名号却没有赢得信任的答案更有价值。在通过套件之外的用例与变异
+测试之前，我们不会发布任何替代数字。
+
这意味着 Basilisk 会**先变小，再变好**。规则会更少，诊断会更少，一致性数字也会更低。
每一次下降我们都会如实报告，而不是设法回避。留下来的，将是对自己所做之事诚实的代码
—— 仅此而已。
-至于被删掉的分析是按规范重建，还是让扩展由一个成熟的开源检查器驱动，我们尚未决定。
-无论走哪条路，新的数字只有在经受住套件之外的用例与变异测试后才会发布。
+### Basilisk 远不只是一个类型检查器
+
+类型检查只是其中一部分。其余部分是装在单个 Rust 二进制文件里的完整 Python 工作流
+—— 语言服务器、重构、格式化、集成调试、性能分析，以及各个编辑器扩展 —— 它们都不
+建立在正在接受审计的规则之上。这正是我们在审计期间着力打磨的地方：把真正有用的部分
+做扎实，并移除任何可能给出误导性结果的东西。变小的意义，是最终得到一个你可以信赖的
+工具。
[阅读完整更正 →](https://www.basilisk-python.dev/zh/docs/conformance/) •
[完整性审计 →](https://github.com/Nimblesite/Basilisk/blob/main/docs/CONFORMANCE-INTEGRITY-AUDIT.md)
diff --git a/vscode-extension/package.json b/vscode-extension/package.json
index 6e8c1db0..f530d6dd 100644
--- a/vscode-extension/package.json
+++ b/vscode-extension/package.json
@@ -38,9 +38,7 @@
"go to definition",
"rust",
"basilisk",
- "lsp",
- "pep conformance",
- "conformance"
+ "lsp"
],
"activationEvents": [
"onLanguage:python",
diff --git a/website/src/_data/releases.js b/website/src/_data/releases.js
index 6cf6e7b1..0c693c44 100644
--- a/website/src/_data/releases.js
+++ b/website/src/_data/releases.js
@@ -23,6 +23,32 @@ const RELEASES_URL = `https://github.com/${OWNER}/${REPO}/releases`;
// allowed. `breaks: true` matches how GitHub itself renders release bodies.
const md = markdownIt({ html: true, linkify: true, breaks: true });
+// Withdrawn-claim redaction. Release notes arrive verbatim from GitHub, and some
+// historical entries quote the conformance result that has since been retracted.
+// [CHKARCH-CONFORMANCE] forbids publishing, quoting, or marketing any conformance
+// figure, so rendering those lines unchanged would keep republishing a claim we
+// have withdrawn. Any line pairing a conformance subject with a figure is replaced
+// by a visible marker — nothing is silently dropped, the marker links to the
+// correction, and every release heading still links to the unmodified GitHub
+// release so the original wording stays one click away.
+const CLAIM_SUBJECT = /conformance|conformant/i;
+const CLAIM_FIGURE = /\d+(?:\.\d+)?\s*%|\b\d{1,4}\s*\/\s*\d{1,4}\b|\b\d+\s+false\s+positives?\b/i;
+const LIST_MARKER = /^(\s*(?:[-*+]|\d+\.)\s+)/;
+const REDACTION = "*[withdrawn conformance claim redacted — see the correction](/docs/conformance/)*";
+
+// Replace each claim-bearing line with the marker, preserving its list marker so
+// the surrounding changelog structure still renders.
+function redactWithdrawnClaims(markdown) {
+ return markdown
+ .split(/\r?\n/)
+ .map((line) => {
+ if (!CLAIM_SUBJECT.test(line) || !CLAIM_FIGURE.test(line)) return line;
+ const marker = line.match(LIST_MARKER);
+ return `${marker ? marker[1] : ""}${REDACTION}`;
+ })
+ .join("\n");
+}
+
const MONTHS = ["Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"];
// "2026-06-23T10:16:43Z" -> "Jun 23, 2026". UTC getters keep the output
@@ -88,7 +114,7 @@ function toRecord(release) {
date: formatDate(release.published_at || release.created_at),
dateIso: release.published_at || release.created_at,
prerelease: release.prerelease === true,
- bodyHtml: release.body ? md.render(release.body) : "",
+ bodyHtml: release.body ? md.render(redactWithdrawnClaims(release.body)) : "",
assets: (release.assets || []).map((asset) => ({
name: asset.name,
url: asset.browser_download_url,
diff --git a/website/src/blog/basilisk-100-percent-python-typing-conformance.md b/website/src/blog/basilisk-100-percent-python-typing-conformance.md
index 14d526c5..6da7d0ee 100644
--- a/website/src/blog/basilisk-100-percent-python-typing-conformance.md
+++ b/website/src/blog/basilisk-100-percent-python-typing-conformance.md
@@ -1,106 +1,133 @@
---
layout: layouts/blog.njk
-title: "Retracted: Basilisk's Former Typing Conformance Result"
-description: "Retraction of Basilisk's former Python typing conformance claim, why the result was untrustworthy, and why the affected rules are being deleted rather than repaired."
+title: "Retraction: Our Withdrawn Python Typing Conformance Claim"
+description: "We retract in full the Python typing conformance claim this post originally made. It was produced by checker logic fitted to the test fixtures, it was not evidence of specification conformance, and it must not be cited."
date: 2026-07-11
-dateModified: 2026-08-06
+dateModified: 2026-08-08
author: Christian Findlay
image: /assets/images/og-image.png
-imageAlt: "Basilisk Python type checker and language server; results under integrity review"
+imageAlt: "Basilisk Python type checker and language server; conformance claim retracted"
imageWidth: 1200
imageHeight: 630
tags:
- Python typing
category: announcements
-excerpt: "Basilisk has retracted its former conformance claim and requested removal from the official results. This post is retained only as a record of the withdrawn announcement."
-keywords: python type checker, python typing conformance, python/typing conformance results, basilisk, mypy, pyright, ty, pyrefly, zuban, pep conformance, strict typing
+excerpt: "This post originally announced a conformance result for Basilisk. That result is retracted in full, and the original text has been removed rather than preserved, because leaving it up kept republishing a claim we know to be unsupported."
+keywords: basilisk retraction, python typing conformance correction, basilisk conformance withdrawn
faq:
- - q: "Which Python type checker has the highest conformance score?"
- a: "Basilisk is not currently listed in the official python/typing results. Its former result is withdrawn and its actual percentage is temporarily unknown while every rule is audited and the ones that matched source text are deleted. Check the live official table for currently listed tools."
- - q: "What is the python/typing conformance suite?"
- a: "It is the official test suite maintained by the Python Typing community. Its harness records how a checker behaves on the suite's exact fixtures. That is valuable evidence, but a raw suite result alone does not establish faithful implementation of the full specification; mutation robustness and independent off-suite cases are also required."
- - q: "Is a 100% conformance score the same as being the best type checker?"
- a: "No. A suite score describes the covered fixtures; it is not proof of specification correctness by itself, as Basilisk's retraction demonstrates. It also does not capture editor integration, error quality, ecosystem support, or independently validated performance."
- - q: "How is Basilisk's conformance score measured?"
- a: "There is no current Basilisk conformance score. A future result will require the unmodified python/typing harness, semantics-preserving mutation testing, and independent off-suite cases derived from the specification, and only after the audit has finished removing rules that decided from source text rather than resolved symbols."
+ - q: "Is Basilisk listed in the official python/typing conformance results?"
+ a: "No. Basilisk was removed from the official results table at our own request, and we have not asked to be re-listed. We will not submit again until the audit is finished, semantics-preserving mutation testing passes clean, and someone outside the project has reviewed the work."
+ - q: "What is Basilisk's current conformance percentage?"
+ a: "We do not publish one, and we ask that no figure be attributed to us. Any number we produced before the audit was measured against fixtures our own code had been fitted to, so it did not measure what we said it measured. We expect an honest figure to be lower than the withdrawn one, and we will say so plainly when we can measure it properly."
+ - q: "Was the original result an attempt to game the conformance suite?"
+ a: "No. It was a verification failure. Our development process named the suite score as the target to build against, and matching source text raises that score faster than real analysis does, so the code drifted that way. Nobody set out to defeat the suite and nothing was concealed from python/typing; the submission ran the suite's own unmodified harness. We did not run the check that would have caught the problem before we published, and that is our responsibility. Basilisk's author has published a personal account at https://www.christianfindlay.com/blog/basilisk-conformance-apology."
+ - q: "Should I use Basilisk's type checker in CI?"
+ a: "Not yet. Until the audit is finished, do not gate CI on it, do not block a merge with it, and do not read a clean run as a clean codebase. The rest of Basilisk — the language server, refactoring, formatting, debugging, and profiling — does not depend on the rules under audit."
---
-> **Retraction — 6 August 2026:** We withdraw every conformance claim in this post. Basilisk's source contained logic fitted to the exact conformance fixtures, so the former perfect result did not establish specification conformance. We asked for Basilisk to be removed from the official results table, and it has been removed. The current percentage is temporarily unknown, and we are not trying to restore it: we are auditing every rule and deleting the ones that matched the spelling of code rather than its meaning, which will push the number lower before anything improves. The original article is retained below only as a public record; its score, ranking, pass counts, and conclusions must not be relied on. Read the [full correction](/docs/conformance/).
-
-Python has a genuinely good type system now, and most developers still do not realize it. A Python type checker works a lot like the TypeScript compiler. Type-checked Python is to regular Python what TypeScript is to JavaScript. The annotations have been in the language for a decade, the specification is mature, and the tooling has caught up.
-
-The open question was never whether Python's type system was good enough. It was how faithfully any given tool actually implements it.
-
-At publication, we believed we had an objective answer for Basilisk. It had been added to this [pinned snapshot of the official python/typing conformance results]({{ conformanceOfficial.historical.snapshot.snapshotUrl }}), where that run reported {{ conformanceOfficial.historical.basilisk.pct }}% ({{ conformanceOfficial.historical.basilisk.passLabel }} of {{ conformanceOfficial.historical.basilisk.total }} tests). That result is now withdrawn.
-
-We were proud of that result. The integrity audit showed that conclusion was wrong.
-
-## Why the conformance suite is the referee that matters
-
-A tool does not get to grade its own homework. Every type checker author will tell you their tool is excellent. That is not evidence.
-
-The [python/typing conformance suite](https://github.com/python/typing/tree/main/conformance) is the closest thing the Python ecosystem has to an objective referee. It is maintained by the Python Typing community, it encodes the actual typing specification as a set of test files, and it runs every participating checker through the same tests with the same harness. Nobody grades themselves. The suite grades all of them, together, on one run.
-
-We treated that as making the result meaningful. The suite did produce the number with its shared harness, but our code had been fitted to exact fixture text. The measurement therefore did not support the conclusion we drew from it.
-
-Basilisk was added to that run in [python/typing pull request #2316](https://github.com/python/typing/pull/2316), "Add Basilisk to conformance results," merged on July 6, 2026. We later requested removal after retracting the result, and Basilisk no longer appears in the live table.
-
-## The historical board snapshot we published
-
-The following is the leaderboard snapshot that accompanied the original announcement on {{ conformanceOfficial.historical.snapshot.dateLabel }}. It is not current, and Basilisk's row is withdrawn. Use the [live official results]({{ conformanceOfficial.historical.snapshot.source }}) for tools that remain listed.
-
-| Rank | Type checker | Backed by | Conformance |
-|---|---|---|---|
-{%- for t in conformanceOfficial.historical.ranked %}
-| {{ t.rank }} | [{{ t.name }} {{ t.version }}]({{ t.resultsUrl }}) | {{ t.org | default("Independent") }} | **{{ t.pct }}%** ({{ t.passLabel }}/{{ t.total }}) |
-{%- endfor %}
-
-A few things are worth saying plainly about that table, because the company Basilisk is keeping is serious.
-
-[Pyright](https://github.com/microsoft/pyright) is developed by Microsoft. [Pyrefly](https://github.com/facebook/pyrefly) is built by Meta. [ty](https://github.com/astral-sh/ty) is built by Astral, the team behind Ruff and uv, which [has agreed to join OpenAI](https://openai.com/index/openai-to-acquire-astral/) (a deal announced in March 2026 and, at announcement, still subject to regulatory approval and customary closing conditions). [mypy](https://github.com/python/mypy) is the original, created by Jukka Lehtosalo and developed heavily at Dropbox. [zuban](https://github.com/zubanls/zuban) is written by David Halter, the author of Jedi. [pycroscope](https://github.com/JelleZijlstra/pycroscope) is maintained by CPython core developer Jelle Zijlstra.
-
-At publication, we used this snapshot to place Basilisk above the other tools. That comparison is withdrawn because Basilisk's result was not robust.
-
-The original post presented the snapshot as proof that a small independent tool sat at the top of a board containing much larger teams. That presentation is part of the withdrawn claim.
-
-## What 100% does and does not mean
-
-Here is the part where we argue against our own headline, because you deserve the honest version.
-
-The python/typing maintainers put a caveat right at the top of the results page, and we agree with it completely:
-
-> "While specification conformance is important for the ecosystem, we don't recommend using it as the primary basis for choosing a type checker. It is not representative of many of the things users typically care about." ([python/typing conformance results]({{ conformanceOfficial.historical.snapshot.source }}))
-
-Read that twice. The people who built the suite are telling you not to treat their own scoreboard as the only thing that matters. That is the right position, and we are not going to pretend otherwise to make Basilisk look better.
-
-The original post tried to explain what we believed a perfect conformance score meant. The integrity audit invalidated the central claim.
-
-**What we claimed it was:** proof that Basilisk judged code correctly against the typing specification. That inference was wrong. Passing the exact suite did not establish a general implementation when parts of the checker matched the fixtures' text.
-
-**What it is not:** a claim that Basilisk is automatically the best choice for your project. Conformance does not measure how fast a checker runs, how good its error messages are, how well it integrates with your editor, or how mature its ecosystem is. Those things matter enormously, and on some of them the older tools have years of head start.
-
-Conformance is one input. It happens to be the input that decides whether you can trust the rest. But it is not the whole decision, and anyone who tells you a single number settles the question is selling something.
-
-## Why we chased the number anyway
-
-If conformance is not the whole story, why did we make 100% a hard requirement rather than a nice-to-have?
-
-Because the alternative is a checker that is confidently wrong some of the time, and a checker that is confidently wrong is worse than no checker at all. The problem with Python typing was never the syntax. The problem was enforcement. A type hint that is never checked is a comment. A type hint that is checked by a tool with gaps is a comment that occasionally lies to you.
-
-Basilisk enables its typing-spec rules by default, with no `--strict` flag to remember. We claimed the old score proved those rules implemented the specification correctly. It did not, and the rules that cannot show they analyse code are being deleted rather than repaired.
-
-## How the withdrawn score was produced
-
-We measure this the boring, reproducible way, because that is the only kind of measurement worth publishing.
-
-The withdrawn number came from the suite's unmodified harness, run against the default-configuration Basilisk CLI with every specification rule enabled. That procedure reproduced the result, but it could not reveal that parts of the implementation were fitted to the exact tests. Future publication will therefore require both the official harness and mutation-based robustness checks.
-
-That is a deliberate design choice, and it maps to a rule we hold for everything we ship: self-measured metrics are only worth anything if they are reproducible and measured by a neutral party. The conformance suite is that neutral party. We just make sure our tool shows up and runs.
-
-## Try it, and try to break it
-
-You can read the current correction and remediation plan on our [conformance page](/docs/conformance/), and see that Basilisk is no longer listed on the [python/typing results page]({{ conformanceOfficial.historical.snapshot.source }}).
-
-Point Basilisk at your own code and report disagreements on [GitHub](https://github.com/Nimblesite/Basilisk/issues). The old score cannot stand in for that real-world scrutiny. A replacement result will be published only after the clean implementation survives broader regression cases and semantics-preserving mutations.
-
-Python's type system has been good enough to trust for a while. Now the tooling can be too.
+> ## This post is retracted in full
+>
+> It originally announced a Python typing conformance result for Basilisk and used
+> that result to compare Basilisk with other type checkers. **Every one of those
+> claims is withdrawn** — the score, the ranking, the pass counts, the category
+> breakdowns, and every conclusion drawn from them. None of it should be cited,
+> quoted, or relied on by anyone, including us.
+>
+> We have **removed the original text** rather than leave it sitting under a notice.
+> A banner on top of an intact announcement still republishes the claim to everyone
+> who lands on the page, and search engines and language models keep lifting the
+> number back out of it. What follows is the correction.
+
+## What the original post claimed
+
+It announced that Basilisk had been added to the official `python/typing` conformance
+results, reported the figure from that run, and presented a leaderboard placing
+Basilisk above established checkers from Microsoft, Meta, Astral, and others. It
+argued that the result proved Basilisk implemented the typing specification correctly.
+
+All of that is withdrawn. The comparison with other tools is withdrawn along with it:
+it was built on our own unreliable row, and it was not a fair statement about anyone
+else's work.
+
+## Why the claim was wrong
+
+Parts of Basilisk's checker decided their answers from the **spelling** of source code
+rather than its meaning, and that logic had been shaped around the exact contents of
+the conformance test files.
+
+The clearest example: type-alias validation ran prefix and substring tests against raw
+source text, including a special case for the literal `eval(` — a spelling that has no
+standing anywhere in the typing specification and appeared only because one test file
+used it. `int("3")` is the identical specification violation and was accepted silently.
+Rename an import or reformat a file and the answer changed, even though the typing
+behaviour under test had not.
+
+A suite result produced by code developed against that suite's exact fixtures does not
+measure specification conformance. The file passes and the rule is not implemented.
+That is why the correction is deletion rather than a better score.
+
+## This was a mistake, not an attempt to game the suite
+
+We didn't set out to defeat the conformance suite, and nothing was concealed from
+`python/typing`. The submission ran the suite's own unmodified harness, with
+Basilisk's default configuration and every specification rule enabled. When the defect was demonstrated, we asked for our own removal.
+
+Our development process named the conformance score as the thing to build against, and matching source text raises that score faster than real analysis does — so that is the direction the code drifted, one plausible-looking rule at a time. Then we published and submitted the result on the
+strength of a green run, without ever running the one check that would have exposed
+it: does this rule still hold when the same program is spelled differently?
+
+That check did not exist. It still doesn't — building it is part of the remediation.
+The conformance suite cannot catch this class of defect by construction, because it is
+the very artefact the code was fitted to, so every green run reinforced a conclusion
+we had no basis for. We believed the number meant what we said it meant. We were
+wrong, and we were wrong because we did not verify it, not because we were trying to
+get away with something.
+
+It was also **not us who found it**. It was reported from outside, in
+[issue #379](https://github.com/Nimblesite/Basilisk/issues/379), from a public
+reproduction. That is its own finding, and we record it as one.
+
+Basilisk's author has written a
+[personal account and apology](https://www.christianfindlay.com/blog/basilisk-conformance-apology)
+taking responsibility in his own words — that he did not set out to publish a number he
+knew was false, did not understand how the rules were passing until it was demonstrated,
+and did not verify the code, which he calls his own failure.
+
+## What we have done
+
+- Withdrew the conformance claim and the published benchmark figures.
+- Asked `python/typing` to remove Basilisk from the official results
+ ([python/typing#2330](https://github.com/python/typing/pull/2330), reverting
+ [#2316](https://github.com/python/typing/pull/2316)). It has been removed.
+- Published the full
+ [integrity audit](https://github.com/Nimblesite/Basilisk/blob/main/docs/CONFORMANCE-INTEGRITY-AUDIT.md)
+ — the defect, the method used to find the rest, reproduction commands, and the parts
+ still broken — with the remaining work tracked as public issues.
+- Started auditing every rule and **deleting** the ones that decide from source text,
+ leaving a failing test behind each removal so the gap is visible rather than hidden.
+
+## What is true today
+
+**We publish no conformance figure, and we ask that none be attributed to us.** Not the
+withdrawn one, not a revised one. A number is not what is wrong here, and putting a new
+one out before the audit finishes would repeat the mistake.
+
+**Do not put Basilisk's type checker in your pipeline yet.** It still contains code that
+isn't doing real type checking, so it can be wrong in both directions — a false error on
+correct code, or silence where there is a genuine bug.
+
+**The rest of Basilisk is unaffected.** The language server, refactoring, formatting,
+integrated debugging, and profiling do not rest on the rules under audit. Sharpening
+those, and removing anything that could hand you a misleading result, is the work in
+front of us.
+
+Basilisk is MIT-licensed and provided **as is, without warranty of any kind**.
+
+## If you cited this post
+
+Please update or remove the citation. If you are holding a screenshot, a ranking table,
+or a figure that traces back here, it is withdrawn. We would rather carry the cost of
+the correction than have the number keep circulating.
+
+Read the [full correction](/docs/conformance/) for the audit scope, the deletion policy,
+and the bar any future result has to clear before we publish anything again.
diff --git a/website/src/docs/conformance.md b/website/src/docs/conformance.md
index 11a6bd6a..8f7e0774 100644
--- a/website/src/docs/conformance.md
+++ b/website/src/docs/conformance.md
@@ -19,6 +19,14 @@ We found checker logic fitted to the exact contents of conformance test files ra
A passing result from code developed against the exact fixtures is not evidence, so the fix is not a better score.
+## This was a mistake, not an attempt to game the suite
+
+We didn't set out to defeat the conformance suite. What actually happened is duller. Our development process named the conformance score as the thing to build against, and matching source text raises that score faster than real analysis does, so that is the direction the code drifted — one plausible-looking rule at a time. We then published and submitted on the strength of a green run, without ever running the one check that would have exposed it: does this rule still hold when the same program is spelled differently? That check did not exist, and building it is part of the remediation below. The suite cannot catch this class of defect by construction, because it is the artefact the code was fitted to, so every green run reinforced a conclusion we had no basis for.
+
+We believed the number meant what we said it meant. We were wrong, and we were wrong because we failed to verify it. Basilisk's author has published a [personal account and apology](https://www.christianfindlay.com/blog/basilisk-conformance-apology) taking responsibility in his own words.
+
+We also did not find this ourselves. It was reported from outside, in [issue #379](https://github.com/Nimblesite/Basilisk/issues/379), with a public reproduction. That is its own finding, and we record it as one.
+
## What we are doing
**We are auditing every rule and deleting the ones that don't do real type checking.** Not rewriting them, not patching them, not marking them TODO — deleting them, and leaving a failing test behind so the gap is visible rather than hidden. A rule stays only if it decides from the resolved syntax tree and returns the same diagnostics when the same program is spelled differently.
@@ -31,7 +39,9 @@ The consequences are deliberate, and we would rather state them up front than ha
What is left will be code that is honest about what it does — nothing else.
-Whether the deleted analysis gets rebuilt from the specification, or the extension ends up driven by an established open-source checker, is a decision we have not made yet. Either way, no replacement percentage gets published until it survives the robustness testing described below.
+**Where a rule can't be made reliable in a straightforward way, we will depend on a different, established type checker rather than ship our own unreliable version of it.** An answer from an engine that has earned trust is worth more to you than a Basilisk-branded one that hasn't. No replacement percentage gets published until it survives the robustness testing described below.
+
+Type checking is one part of Basilisk. The rest — language server, refactoring, formatting, integrated debugging, profiling, and the editor extensions — does not rest on the rules under audit, and that is what we are sharpening while the audit runs: make the parts that are genuinely useful solid, and remove anything that could hand you a misleading result.
Current python/typing results ↗ diff --git a/website/src/docs/releases.njk b/website/src/docs/releases.njk index f050aafc..d31ece6d 100644 --- a/website/src/docs/releases.njk +++ b/website/src/docs/releases.njk @@ -17,10 +17,12 @@ permalink: /docs/releases/
- Historical release notes may contain withdrawn conformance or benchmark claims. - Those claims are preserved verbatim as part of the changelog, but they are not current evidence and - must not be relied on. Basilisk's conformance percentage is temporarily unknown and its benchmark - figures are under review. Read the conformance correction and + Withdrawn conformance claims in these historical notes are redacted. + Older entries announced a conformance result that we have since retracted, so any changelog line + quoting a conformance figure is replaced here with a marked redaction rather than republished. + Nothing is hidden: each release heading links to the unmodified release on GitHub. Basilisk's + conformance percentage is temporarily unknown and we publish no figure; its benchmark numbers are + also under review. Read the conformance correction and benchmark notice for the current status.
diff --git a/website/src/index.njk b/website/src/index.njk index 65ac9055..cdab1a06 100644 --- a/website/src/index.njk +++ b/website/src/index.njk @@ -19,11 +19,15 @@ permalink: /
+ Don't put the type checker in your pipeline yet. It still contains code
+ that isn't doing real type checking, so it is not yet trustworthy.
We have withdrawn both our former conformance claim and our published benchmark figures,
and Basilisk was removed from the official python/typing results at our request.
We are now auditing every rule and deleting the ones that match source text
instead of doing real type checking. Expect Basilisk to get smaller before it gets better;
- what is left will be code that is honest about what it does.
+ what is left will be code that is honest about what it does. Our current conformance
+ percentage is temporarily unknown, and no replacement figure gets published until it
+ survives robustness and mutation testing.
- Both sets of figures are withdrawn while the audit runs. + Both sets of figures are withdrawn while the audit runs. This was a mistake and a failure to verify, not an attempt to game the suite. Read the conformance correction → + Read the author’s apology → Read the benchmark notice →
diff --git a/website/src/zh/blog/basilisk-100-percent-python-typing-conformance.md b/website/src/zh/blog/basilisk-100-percent-python-typing-conformance.md index 16ea9c4b..3a81a4b9 100644 --- a/website/src/zh/blog/basilisk-100-percent-python-typing-conformance.md +++ b/website/src/zh/blog/basilisk-100-percent-python-typing-conformance.md @@ -1,107 +1,117 @@ --- layout: layouts/blog.njk -title: "已撤回：Basilisk 此前的类型符合性结果" -description: "撤回 Basilisk 此前的 Python typing 符合性声明，说明该结果为何不可信，以及受影响的规则为何是被删除而不是被修补。" +title: "撤回声明：我们已撤回的 Python typing 符合性声明" +description: "我们全文撤回本文此前作出的 Python typing 符合性声明。该结果出自针对测试用例适配的检查器逻辑，并不能证明规范符合性，不应被引用。" date: 2026-07-11 -dateModified: 2026-08-06 +dateModified: 2026-08-08 author: Christian Findlay image: /assets/images/og-image.png -imageAlt: "Basilisk Python 类型检查器与语言服务器；结果正在接受完整性审查" +imageAlt: "Basilisk Python 类型检查器与语言服务器；符合性声明已撤回" imageWidth: 1200 imageHeight: 630 tags: - Python typing category: announcements lang: zh -excerpt: "Basilisk 已撤回此前的符合性声明，并请求从官方结果中移除。本文仅作为已撤回公告的历史记录保留。" -keywords: python类型检查器, python类型符合性, python/typing符合性结果, basilisk, mypy, pyright, ty, pyrefly, zuban, pep符合性, 严格类型 +excerpt: "本文原本发布了 Basilisk 的一项符合性结果。该结果现已全文撤回；我们删除了原文，而不是在其上方加一条提示——因为把原文留在那里，等于每次都在重新发布一个我们已知站不住脚的声明。" +keywords: basilisk 撤回声明, python typing 符合性更正, basilisk 符合性已撤回 faq: - - q: "哪个 Python 类型检查器的符合性分数最高？" - a: "Basilisk 目前不在官方 python/typing 结果中。此前的结果已撤回；在完成规则审计、删除依据源文本判断的规则之前，实际百分比暂时未知。其他工具请查看官方实时结果表。" - - q: "什么是 python/typing 符合性测试套件？" - a: "这是由 Python Typing 社区维护的官方测试套件。它使用自己的评分工具记录检查器在确切测试用例上的行为。这是有价值的证据，但原始套件结果本身不能证明完整规范已被忠实实现；还必须通过保持语义的变异和独立的套件外用例验证。" - - q: "100% 的符合性分数是否等于是最好的类型检查器？" - a: "不。套件分数只描述被覆盖的测试用例；正如 Basilisk 此次撤回所证明的，它本身不能证明规范实现正确。它也无法反映编辑器集成、错误信息质量、生态系统支持或经过独立验证的性能。" - - q: "Basilisk 的符合性分数是如何测量的？" - a: "Basilisk 目前没有可发布的符合性分数。只有在审计完成、依据源文本而非已解析符号判断的规则被删除之后，未来结果才可能发布，并且必须同时通过未经修改的 python/typing 评分工具、保持语义的变异测试，以及依据规范独立设计的套件外用例。" + - q: "Basilisk 目前在官方 python/typing 符合性结果中吗？" + a: "不在。我们主动请求将 Basilisk 从官方结果表中移除，并且没有申请重新列入。在审计完成、保持语义的变异测试全部通过、并且由项目之外的人完成复核之前，我们不会再次提交。" + - q: "Basilisk 当前的符合性百分比是多少？" + a: "我们不发布任何数字，也请不要把任何数字归到我们名下。审计之前我们得到的任何数字，都是针对我们自己的代码已经适配过的测试用例测出来的，因此它并没有衡量我们所声称的东西。我们预计诚实的数字会低于此前撤回的那个；等到能够正确测量时，我们会如实说明。" + - q: "当初的结果是不是有意操纵符合性测试套件？" + a: "不是。这是一次验证失败。我们的开发流程把套件分数定为要去达成的目标，而匹配源文本比真正做分析更快地提高这个分数，代码于是滑向了那个方向。没有人试图去糊弄测试套件，也没有向 python/typing 隐瞒任何东西——提交使用的是套件自己未经修改的评分工具。我们在发布之前没有做那项本可以发现问题的检查，这是我们的责任。Basilisk 作者已在 https://www.christianfindlay.com/blog/basilisk-conformance-apology 发表了个人说明。" + - q: "我应该在 CI 中使用 Basilisk 的类型检查器吗？" + a: "暂时不要。在审计完成之前，请不要用它作为 CI 门禁，不要用它阻断合并，也不要把一次干净的运行结果当作代码库是干净的。Basilisk 的其余部分——语言服务器、重构、格式化、调试与性能分析——并不依赖正在审计的这些规则。" --- -> **撤回说明——2026 年 8 月 6 日：**我们撤回本文中的所有符合性声明。Basilisk 源码中存在针对确切符合性测试用例实现的逻辑，因此此前的满分结果不能证明规范符合性。我们请求从官方结果表中移除 Basilisk，现已完成移除。在删除有问题的实现、根据规范重新构建并通过保持语义的变异测试之前，当前百分比暂时未知。下方原文仅作为公开历史记录保留；其中的得分、排名、通过数量和结论均不可依赖。请阅读[完整更正](/zh/docs/conformance/)。 +> ## 本文已全文撤回 +> +> 本文原本发布了 Basilisk 的一项 Python typing 符合性结果，并用该结果将 Basilisk +> 与其他类型检查器作比较。**其中的每一项声明均已撤回** —— 包括分数、排名、通过数量、 +> 分类明细，以及由此得出的全部结论。任何人都不应再引用、转述或依赖这些内容，我们自己也不会。 +> +> 我们**删除了原文**，而不是在原文上方加一条提示。把撤回声明贴在一篇完好的公告上方， +> 每一个访问该页面的人仍然会再次读到那个声明，搜索引擎和语言模型也会继续把那个数字 +> 从中提取出来。以下是更正内容。 -Python 现在拥有一个真正出色的类型系统，而大多数开发者仍然没有意识到这一点。Python 类型检查器的工作方式很像 TypeScript 编译器。经过类型检查的 Python 之于普通 Python，就如同 TypeScript 之于 JavaScript。类型注解已经存在于语言中十年了，规范已经成熟，工具也已经跟上。 +## 原文当时声称了什么 -从来悬而未决的问题从不是 Python 的类型系统是否足够好。而是任何一个具体工具究竟对它的实现有多忠实。 +原文宣布 Basilisk 已被加入官方 `python/typing` 符合性结果，报告了那次运行的数字，并给出 +一份排行榜，把 Basilisk 排在来自 Microsoft、Meta、Astral 等团队的成熟检查器之上；文章据此 +论证该结果证明 Basilisk 正确实现了类型规范。 -发布本文时，我们以为 Basilisk 得到了一个客观答案。它被添加进这份[官方 python/typing 符合性结果的固定快照]({{ conformanceOfficial.historical.snapshot.snapshotUrl }})，当时的运行报告了 {{ conformanceOfficial.historical.basilisk.pct }}%（{{ conformanceOfficial.historical.basilisk.total }} 项测试中通过 {{ conformanceOfficial.historical.basilisk.passLabel }} 项）。该结果现已撤回。 +以上全部撤回。与其他工具的比较也一并撤回：它建立在我们自己那一行不可靠的数据之上， +对别人的工作而言并不是一个公平的陈述。 -我们曾为此感到自豪。完整性审计证明这个结论是错误的。 +## 这个声明为什么是错的 -## 为什么符合性测试套件才是那个重要的裁判 +Basilisk 检查器中的一部分逻辑，依据的是源代码的**写法**而不是它的含义，而且这些逻辑是围绕 +符合性测试文件的确切内容塑造出来的。 -工具不能给自己的作业打分。每个类型检查器的作者都会告诉你他们的工具很出色。这不是证据。 +最清楚的例子：类型别名验证对原始源代码文本执行前缀和子字符串判断，其中甚至专门判断了 +字面量 `eval(` —— 这个写法在类型规范中没有任何依据，出现在代码里只是因为某个测试文件用了它。 +`int("3")` 是完全相同的规范违规，却被静默接受。改一个导入别名或重新格式化文件，结论就会变， +尽管被测试的类型行为并没有改变。 -[python/typing 符合性测试套件](https://github.com/python/typing/tree/main/conformance)是 Python 生态系统中最接近客观裁判的东西。它由 Python Typing 社区维护，将真正的类型规范编码为一组测试文件，并用同一套评分工具、同样的测试来运行每一个参与的检查器。没有人给自己打分。是这个套件在一次运行中把它们全部一起评分。 +针对某个套件的确切测试用例开发出来的代码，即使在该套件上通过，也不能衡量规范符合性。 +真实情况是：文件通过了，而规则并没有实现。这正是为什么我们的纠正措施是删除，而不是去拿一个更好的分数。 -我们曾认为这使结果具有意义。套件确实使用共享评分工具产出了该数字，但我们的代码针对确切测试文本进行了适配，因此该测量无法支撑我们得出的结论。 +## 这是一个错误，而不是有意操纵测试套件 -Basilisk 是通过 [python/typing 拉取请求 #2316](https://github.com/python/typing/pull/2316)（"Add Basilisk to conformance results"）加入那次运行的，该请求于 2026 年 7 月 6 日合并。撤回结果后，我们请求将其移除；Basilisk 已不再出现在实时结果表中。 +我们希望在这一点上毫不含糊，因为事件的先后顺序容易让人得出一个比事实更糟的解读。 -## 当时发布的历史排行榜快照 +**没有人试图去糊弄符合性测试套件，也没有向 `python/typing` 隐瞒任何东西。** 提交时运行的是套件 +自己未经修改的评分工具，使用 Basilisk 的默认配置并启用全部规范规则。没有针对评分工具做特殊处理， +没有伪造输出，没有禁用规则，也没有手工编辑结果文件。当缺陷被演示出来之后，我们主动请求移除自己。 -以下是原公告在 {{ conformanceOfficial.historical.snapshot.dateLabel }} 使用的排行榜快照。它不是当前结果，Basilisk 这一行也已撤回。当前仍列出的工具请查看[官方实时结果]({{ conformanceOfficial.historical.snapshot.source }})。 +真实经过要平淡得多，也更普通。我们的开发流程把符合性分数定为要去达成的目标，而匹配源文本比 +真正做分析更快地提高这个分数 —— 于是代码就朝那个方向滑了过去，一条看起来合理的规则接着一条。 +然后我们仅凭一次全绿的运行就发布并提交了结果，却始终没有做那项本可以揭露问题的检查： +同一个程序换一种写法时，这条规则是否依然成立？ -| 排名 | 类型检查器 | 背后团队 | 符合性 | -|---|---|---|---| -{%- for t in conformanceOfficial.historical.ranked %} -| {{ t.rank }} | [{{ t.name }} {{ t.version }}]({{ t.resultsUrl }}) | {{ t.org | default("独立") }} | **{{ t.pct }}%** ({{ t.passLabel }}/{{ t.total }}) | -{%- endfor %} +这项检查当时并不存在，现在也还没有 —— 把它建起来正是补救工作的一部分。符合性测试套件从构造上 +就无法发现这一类缺陷，因为它正是代码被适配过去的那个对象，所以每一次全绿运行都在强化一个 +我们根本没有依据的结论。我们当时相信那个数字就是我们所说的意思。我们错了；我们之所以错， +是因为我们没有去验证它，而不是因为我们想蒙混过关。 -关于那张表格，有几点值得直白地说清楚，因为 Basilisk 所处的阵容非常有分量。 +发现问题的**也不是我们**。它是由外部报告的，见 +[issue #379](https://github.com/Nimblesite/Basilisk/issues/379)，并附有公开的复现。 +这件事本身也是一项发现，我们如实记录。 -[Pyright](https://github.com/microsoft/pyright) 由微软开发。[Pyrefly](https://github.com/facebook/pyrefly) 由 Meta 构建。[ty](https://github.com/astral-sh/ty) 由 Astral 构建，也就是 Ruff 和 uv 背后的团队，该团队[已同意加入 OpenAI](https://openai.com/index/openai-to-acquire-astral/)（该交易于 2026 年 3 月宣布，在宣布时仍需监管批准和惯例性交割条件）。[mypy](https://github.com/python/mypy) 是最早的那个，由 Jukka Lehtosalo 创建，并在 Dropbox 大量开发。[zuban](https://github.com/zubanls/zuban) 由 Jedi 的作者 David Halter 编写。[pycroscope](https://github.com/JelleZijlstra/pycroscope) 由 CPython 核心开发者 Jelle Zijlstra 维护。 +Basilisk 的作者已就此事的经过发表了[个人说明与致歉](https://www.christianfindlay.com/blog/basilisk-conformance-apology)， +并承担责任。他在文中写道，他并没有打算发布一个自己明知是错的数字，也没有理解那些规则究竟是怎么通过的， +直到有人把问题演示出来；没有验证代码是他的失误。 -发布时，我们用该快照将 Basilisk 排在其他工具之前。由于 Basilisk 的结果缺乏稳健性，这项比较现已撤回。 +## 我们已经做了什么 -原文将该快照描述为一个小型独立工具位居大型团队之前的证明。这个说法属于已撤回的声明。 +- 撤回了符合性声明和已发布的基准测试数字。 +- 请求 `python/typing` 将 Basilisk 从官方结果中移除 + （[python/typing#2330](https://github.com/python/typing/pull/2330)，撤销 + [#2316](https://github.com/python/typing/pull/2316)）。现已完成移除。 +- 公开了完整的[完整性审计](https://github.com/Nimblesite/Basilisk/blob/main/docs/CONFORMANCE-INTEGRITY-AUDIT.md) + —— 包括缺陷本身、用于排查其余部分的方法、复现命令，以及仍然没有修好的部分；剩余工作以公开 + issue 跟踪。 +- 开始逐条审计规则，**删除**那些依据源文本判断的规则，并在每一次删除之后留下一个失败的测试， + 让缺口可见而不是被掩盖。 -## 100% 意味着什么，又不意味着什么 +## 现在的真实情况 -接下来这部分我们要反驳自己的标题，因为你值得听到诚实的版本。 +**我们不发布任何符合性数字，也请不要把任何数字归到我们名下。** 撤回的那个不行，修订过的也不行。 +这里出问题的并不是某个数字；在审计完成之前发布一个新数字，只会重复同样的错误。 -python/typing 的维护者在结果页面的顶部就放了一个提醒，我们完全同意： +**请暂时不要把 Basilisk 的类型检查器放进你的流水线。** 其中仍有没有做真正类型检查的代码， +因此它在两个方向上都可能出错 —— 既可能对正确的代码报出虚假错误，也可能对真实的缺陷保持沉默。 -> "虽然规范符合性对生态系统很重要，但我们不建议将其作为选择类型检查器的主要依据。它并不能代表用户通常关心的许多方面。"（[python/typing 符合性结果]({{ conformanceOfficial.historical.snapshot.source }})） +**Basilisk 的其余部分不受影响。** 语言服务器、重构、格式化、集成调试和性能分析都不依赖正在审计的 +这些规则。把这些部分打磨扎实，并移除任何可能给出误导性结果的东西，正是我们眼下要做的工作。 -请读两遍。构建这个套件的人正在告诉你，不要把他们自己的记分牌当作唯一重要的东西。这是正确的立场，我们不会为了让 Basilisk 看起来更好而假装不是这样。 +Basilisk 采用 MIT 许可证，并按**“原样”提供，不附带任何形式的担保**。 -原文试图解释我们当时认为满分意味着什么。完整性审计推翻了核心结论。 +## 如果你引用过本文 -**我们曾声称它是什么：**证明 Basilisk 能根据类型规范正确判断代码。这个推论是错误的。如果检查器的一部分匹配了测试文本，通过确切套件并不能证明通用实现。 +请更新或删除该引用。如果你手上有截图、排行榜表格，或任何可以追溯到这里的数字，它们均已撤回。 +我们宁愿承担更正的代价，也不愿让那个数字继续流传。 -**它不是什么：**它不是一个声称 Basilisk 自动就是你项目最佳选择的说法。符合性不衡量检查器运行有多快、错误信息有多好、与你的编辑器集成得有多好，或者其生态系统有多成熟。这些方面极其重要，而在其中一些方面，较老的工具有多年的先发优势。 - -符合性是一个输入。它恰好是决定你能否信任其余一切的那个输入。但它不是全部决定因素，任何告诉你单一数字就能定论的人都在推销某种东西。 - -## 我们为什么还是追求了这个数字 - -如果符合性不是全部，我们为什么把 100% 当作一个硬性要求，而不是一个锦上添花的东西？ - -因为另一种情况是一个有时会自信地出错的检查器，而一个自信地出错的检查器比没有检查器更糟。Python 类型的问题从来不在于语法。问题在于强制执行。一个从不被检查的类型提示只是一句注释。一个被有漏洞的工具检查的类型提示，是一句偶尔会对你撒谎的注释。 - -Basilisk 默认启用类型规范规则，无需记住 `--strict` 标志。我们曾声称旧分数证明这些规则正确实现了规范。事实并非如此；相关实现正在重新构建和验证。 - -## 已撤回分数如何产生 - -我们用那种枯燥、可复现的方式来测量它，因为那是唯一值得发布的测量方式。 - -已撤回的数字来自套件自己未经修改的评分工具，针对默认配置的 Basilisk CLI 运行并开启所有规范规则。该过程可以复现数字，却无法揭示部分实现针对确切测试进行了适配。因此，未来发布必须同时通过官方评分工具和基于变异的稳健性检查。 - -这是一个刻意的设计选择，它对应着我们对所交付的一切都坚持的一条规则：自我测量的指标只有在可复现、且由中立方测量时才有价值。符合性套件就是那个中立方。我们只是确保我们的工具出现并运行。 - -## 试用它，并试着让它出错 - -你可以在[符合性页面](/zh/docs/conformance/)阅读当前更正和修复计划，并在 [python/typing 结果页面]({{ conformanceOfficial.historical.snapshot.source }})上看到 Basilisk 已不再列出。 - -请把 Basilisk 指向你自己的代码，并在 [GitHub](https://github.com/Nimblesite/Basilisk/issues) 上报告分歧。旧得分不能替代这种真实检验。只有在审计完成、留下的代码通过更广泛的回归用例和保持语义的变异后，我们才会发布替代结果。 - -Python 的类型系统已经足够好、值得信任有一段时间了。现在，工具也可以了。 +关于审计范围、删除策略，以及未来任何结果在发布之前必须达到的标准，请阅读[完整更正](/zh/docs/conformance/)。 diff --git a/website/src/zh/docs/conformance.md b/website/src/zh/docs/conformance.md index aab0fefc..015c4f54 100644 --- a/website/src/zh/docs/conformance.md +++ b/website/src/zh/docs/conformance.md @@ -15,6 +15,18 @@ lang: zh 针对固定测试用例开发出的代码即使通过，也不能作为证据；因此，解决办法不是拿到一个更好的分数。 +## 这是一个错误，而不是有意操纵测试套件 + +我们希望在这一点上毫不含糊，因为事件的先后顺序容易让人得出一个比事实更糟的解读。 + +**没有人试图去糊弄符合性测试套件，也没有向 `python/typing` 隐瞒任何东西。** 提交时运行的是套件自己未经修改的评分工具，使用 Basilisk 的默认配置并启用全部规范规则 —— 没有针对评分工具做特殊处理，没有伪造输出，没有禁用规则，也没有手工编辑结果文件。当缺陷被演示给我们之后，我们主动请求移除自己。 + +真实经过要平淡得多。我们的开发流程把符合性分数定为要去达成的目标，而匹配源文本比真正做分析更快地提高这个分数，于是代码就朝那个方向滑了过去 —— 一条看起来合理的规则接着一条。然后我们仅凭一次全绿的运行就发布并提交了结果，却始终没有做那项本可以揭露问题的检查：同一个程序换一种写法时，这条规则是否依然成立？这项检查当时并不存在，把它建起来正是下文补救工作的一部分。符合性测试套件从构造上就无法发现这一类缺陷，因为它正是代码被适配过去的那个对象，所以每一次全绿运行都在强化一个我们根本没有依据的结论。 + +我们当时相信那个数字就是我们所说的意思。我们错了；我们之所以错，是因为我们没有去验证它。Basilisk 作者已就此发表[个人说明与致歉](https://www.christianfindlay.com/blog/basilisk-conformance-apology)，并承担责任。 + +发现问题的也不是我们。它是由外部报告的，见 [issue #379](https://github.com/Nimblesite/Basilisk/issues/379)，并附有公开的复现。这件事本身也是一项发现，我们如实记录。 + ## 我们正在做什么 **我们正在逐条审计规则，并删除那些没有做真正类型检查的规则。** 不是重写，不是打补丁，也不是标一个 TODO —— 是删除，并留下一个失败的测试，让这个缺口可见而不是被掩盖。一条规则只有在依据已解析的语法树做判断、并且同一个程序换一种写法时给出相同诊断的情况下，才会保留。 @@ -27,7 +39,9 @@ lang: zh 留下来的，将是对自己所做之事诚实的代码 —— 仅此而已。 -被删掉的分析是按规范重建，还是让扩展由一个成熟的开源检查器驱动，我们尚未决定。无论走哪条路，在通过下文所述的稳健性验证之前，都不会发布替代百分比。 +**如果一条规则无法以直截了当的方式做到可靠，我们会转而依赖另一个成熟的类型检查器，而不是端出我们自己那份不可靠的实现。** 一个已经赢得信任的引擎给出的答案，对你而言比一个挂着 Basilisk 名号却没有赢得信任的答案更有价值。在通过下文所述的稳健性验证之前，我们不会发布任何替代百分比。 + +类型检查只是 Basilisk 的一部分。其余部分 —— 语言服务器、重构、格式化、集成调试、性能分析，以及各个编辑器扩展 —— 都不建立在正在接受审计的规则之上；审计期间，这些正是我们着力打磨的地方：把真正有用的部分做扎实，并移除任何可能给出误导性结果的东西。当前 python/typing 结果 ↗ diff --git a/website/src/zh/index.njk b/website/src/zh/index.njk index d1d840e3..59868ad0 100644 --- a/website/src/zh/index.njk +++ b/website/src/zh/index.njk @@ -20,10 +20,13 @@ permalink: /zh/
+ 请暂时不要把类型检查器放进你的流水线。其中仍有没有做真正
+ 类型检查的代码，因此它目前还不值得信任。
我们已撤回此前的符合性声明和公开的基准测试数据，应我们的请求，Basilisk
已从官方 python/typing 结果中移除。我们正在逐条审计规则，
并删除那些只匹配源文本、并未做真正类型检查的规则。
Basilisk 会先变小，再变好；留下来的，将是对自己所做之事诚实的代码。
+ 当前符合性百分比暂时未知；在通过稳健性与变异测试之前，我们不会发布任何替代数字。
- 审计期间，两组数据均已撤回。 + 审计期间，两组数据均已撤回。这是一个错误、一次验证上的失职，而不是有意操纵测试套件。 阅读符合性更正 → + 阅读作者的致歉 → 阅读基准测试说明 →