Reorient the docs from conformance score to accuracy, and from repair to deletion - #421
Merged
Conversation
The change-scope classifier routes by path, so a branch that mixes documentation with an unrelated tooling change still runs the full Rust matrix, the live conformance clone, and the mutation shards. Add an explicit per-PR opt-out: a docs-only label makes the classifier emit every scope as false. A label rather than a path rule keeps the decision visible on the PR and attributable to whoever made it, and leaves every unlabelled PR untouched.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TLDR
Reorients every agent-facing doc, spec, and plan away from optimising the conformance score and toward accuracy on unseen Python, and makes the public message "we are auditing and deleting code that doesn't do real type checking."
What Was Added?
[CHKARCH-TEXT-MATCHED-LOGIC]in the checker architecture spec — the normative failing-test → delete → report rule for any code that decides from source text instead of resolved symbols, with the detection signatures and an explicit "do not fix it, do not rewrite it, do not leave a TODO."[CHKARCH-TESTING-SEMANTIC-MUTATION]— the accuracy gate: aliased imports, import form, reformatting, quote style, consistent renaming, statement reordering, comment churn → byte-identical diagnostics. Marked specified, not built, with the build item owned by[LINESCANPLAN-SEMANTIC-MUTATION]. Until it exists, no rule may be described as spec-implementing on the strength of a fixture.docs-onlyCI label — the change-scope classifier emits every scope as false when a PR carries it. A label rather than a path rule keeps the decision visible on the PR and leaves every unlabelled PR untouched.scripts/publish_zed_registry.py+ tests — automates the Zed registry PR on tagged releases ([ZED-MIRROR]), carried over from earlier work on this branch.What Was Changed or Deleted?
Signals that told agents to protect the number:
CHECKER-TYPE-NARROWING-INFERENCE-PLAN.mdsaid "Never delete, disable, or unregister a rule. Not one. The rule survives; its guts get replaced" and "A migration that drops the number is reverted, not negotiated." An agent finding a text-matching rule was forbidden from removing it. Replaced with a boundary of intent and disclosure.CHECKER-ELIMINATE-LINE-SCANNING-PLAN.mdrequired "Keep conformance at 141/141" and "Preserve the exact diagnostics" while migrating. Rewritten around disposal; the AST replacements are kept as evidence of what the deleted code failed to do, not as work items.ROADMAP-NEXT-STEPS-PLAN.mdopened "Conformance remains the prime directive and both ratchets stand." Now: accuracy is the prime directive, the suite is an overfitted downstream sample.CHECKER-ARCHITECTURE-SPEC.mdcarried "Target: 100%", "The path to 100%", a pass-percentage ratchet, "Gaming the number is a punishable offence", and the withdrawn figure stamped inline. All removed. The last<!--g:score-->markers are gone repo-wide, sogen_conformance_reference.pynow stamps nothing.CHECKER-TYPE-INFERENCE-SPEC.mdjustified comparingNamedtypes by the source text before the[— solist[int]acceptslist[str]— on the grounds that "the conformance gate holdsmax_false_positivesat zero." Now described as what it is: unimplemented generic-argument compatibility, and text matching.CHECKER-STUB-RESOLUTION-SPEC.mdrouted advisories off the diagnostic stream "so it can NEVER create a false positive." Rewritten to turn on what a diagnostic means rather than what it costs a measurement.NARROWPLAN-SCOREBOARDmandated beating five competitors on five axes with "falling behind on a led axis is a build failure" — the same self-authored-and-self-graded shape. Demoted to recorded-not-gated.Testing and mutation regime (
[CHKARCH-TESTING-MUTATION-RATCHET]) — three defects documented as corrections owed rather than conventions to preserve:#[mutation_safe], so the pool is self-selected: the committed baseline is 161 mutants at 100% against a crate of ~82k LOC across 307 files. The comment inmutants_report.pyclaimed the run "mutates the WHOLE crate (no code excluded)" — false, corrected in place.detected = caught + timeoutcredits unevaluated mutants, and timeouts are structurally biased toward survivors.Public message — README (×5 generated), homepage, and
/docs/conformance/in both languages now lead with "We are auditing the checker and deleting what doesn't hold up", state plainly that Basilisk gets smaller first and that the conformance number will fall, and commit to reporting each drop. Every "being rebuilt" across the blog, comparison, docs index and quick-start became audit-and-delete.How Do The Automated Tests Prove It Works?
This is a documentation change and carries no behavioural test. What was verified:
python3 scripts/gen_readmes.py --check→ "READMEs are in sync with docs/readme/" — the five generated READMEs match their two sources.python3 scripts/gen_conformance_reference.py --check→ "conformance docs up to date" — confirms no stamp marker survives anywhere, so the stamper is a no-op rather than silently stale.npx @11ty/eleventy --dryrunbuilds the site clean after the EN and ZH rewrites.docs/andCLAUDE.md.coverage-thresholds.jsonre-parses and every threshold value is byte-identical — only the two_docstrings changed.mutation_testing/mutants_report.pycompiles; the edit is a comment block.Spec / Doc Changes
Summarised above.
CLAUDE.md,CONTRIBUTING.md, andCONTRIBUTING.zh.mdcarry the same three-step protocol and the same standard: a failing test that pins real incorrect behaviour is worth more than a passing fixture carried by logic that does not analyse code.One deliberate non-change.
coverage-thresholds.jsonstill gates the conformance pass percentage at 100 with zero false positives, so the first honest deletion will failmake test. That floor is what §6.3 of the integrity audit identifies as the incentive behind the fitted predicates, but removing it is a scoreboard edit and not an agent's call. The contradiction is recorded in[CHKARCH-CONFORMANCE],[COVERAGE-THRESHOLDS-JSON-CONFORMANCE],CLAUDE.md, and the file's own_doc, each with the same instruction: delete anyway, report the drop and the failing gate, stop there.Refs #379, #408, #409, #410, #411, #412
Breaking Changes