conformance integrity: alias-rule validation is a 1:1 transcription of the conformance test files, not a type-expression validator

[MelbourneDeveloper]

Summary

The alias rules do not implement PEP 613 / PEP 695 type-expression validation. They implement a set of textual predicates that is in 1:1 correspondence with the lines of the conformance test files and generalises to essentially nothing else.

aliases_implicit.py and aliases_type_statement.py are recorded as PASS in conformance/conformance_status.csv. Those two PASS results are not evidence that Basilisk implements the rule they claim to test. This issue exists to say so publicly and to scope the remediation.

#379 reports the same function, but frames it as a false-negative bug in one file, discovered from the outside. This issue is the integrity question underneath it: why does that predicate set exist in that exact shape, and what else shares it. Refs #379.

Evidence 1 — the predicate set is a transcription of the test file

is_invalid_rhs on main, duplicated verbatim in two rules:

• crates/basilisk-checker/src/rules/aliases_type_statement.rs:44
• crates/basilisk-checker/src/rules/aliases_implicit.rs:97

Every branch maps onto exactly one line of conformance/tests/aliases_type_statement.py (the BadTypeAlias* block of aliases_implicit.py is the same set):

Conformance test line	Branch that catches it
eval("".join(map(chr, [105, 110, 116])))	rhs.starts_with("eval(")
[int, str]	rhs.starts_with('[')
((int, str),)	starts_with('(') && paren_has_top_level_comma
[int for i in range(1)]	rhs.starts_with('[')
{"a": "b"}	rhs.starts_with('{')
(lambda: int)()	rhs.contains("lambda")
[int][0]	rhs.starts_with('[')
int if 1 < 3 else str	has_top_level_token(rhs, " if ")
var1	is_non_type_name / collect_runtime_var_names
True	rhs == "True" || rhs == "False"
1	rhs.chars().next().is_ascii_digit()
list or set	has_top_level_token(rhs, " or ")
f"{'int'}"	rhs.starts_with("f\"")

Coverage of the test file: 13/13. Content in the predicate set that is not required by the test file: three items — False, " and ", and the negative-number branch — each the trivial symmetric twin of a branch that was required.

There is no branch for anything else in the type-expression grammar: no call other than the literal prefix eval(, no +/-/* binary operator, no comparison, no not, no unary op, no bytes literal, no starred or walrus expression, no attribute access on a subscript. type A = "the" + "thing", type B = list["of genshin"], and type D = list[int].attr all pass silently (this is the #379 reproduction).

Evidence 2 — starts_with("eval(")

eval is not a typing special form. It has no standing in PEP 613 or PEP 695. The only reason to name that one builtin, as a source-text prefix, is that BadTypeAlias1 in the conformance suite happens to be spelled eval(...).

It appears in three separate files:

• aliases_type_statement.rs:82
• aliases_implicit.rs:157
• annotations_forward_refs/type_checks.rs:113

int("3") as an alias RHS is the identical spec violation and is accepted by all three.

Evidence 3 — the same shape in the parameterization checks

Not confined to is_invalid_rhs. In aliases_implicit.rs, against aliases_implicit.py:76-81:

• :687 — the ParamSpec check emits a hard error on the guess all_simple && args.len() > 1, with the comment "the ParamSpec arg is probably wrong". It fires on GoodTypeAlias9[int, int] and is not a ParamSpec check.
• :751 — is_assignable_to_bound implements int/float/complex and returns true (accept) for every other bound. The numeric tower is exactly and only what GoodTypeAlias12[str] needs. Every non-numeric TypeVar bound is unchecked.
• :407 — a module variable is treated as an implicit type alias only if its name starts with an uppercase ASCII letter. This works on the test file because its aliases are named GoodTypeAlias* / ListAlias.
• :76 — TypeAlias as X import aliases are recovered by match_indices("TypeAlias as ") over raw import source text, against the repo rule "avoid regex to parse anything, use ruff".

Consequences

1. The scoreboard overstates these rules. aliases_implicit.py (22 caught), aliases_type_statement.py (24 caught), and the two other files that list aliases_implicit among their rules pass on predicates fitted to their own contents. The errors are caught; the rule is not implemented.
2. False positives on valid code. contains("lambda") matches any identifier containing that substring. has_top_level_token tracks bracket depth but not string state, so a string RHS containing if / or / and is misread as code.
3. CONTRIBUTING/CLAUDE constraints already forbade this. "Avoid regex to parse anything, use ruff." The Ruff AST for these nodes is already parsed and available on ResolvedModule.

Remediation

• aliases_type_statement.rs — replace with type-expression grammar validation over the StmtTypeAlias value node. In flight on bidirectionaltype-inference; the structural rewrite is written and passes the same conformance files. Refs type-statement RHS validation is substring matching on source text, so most invalid type expressions pass silently #379.
• aliases_implicit.rs — same treatment; currently untouched and not listed in CHECKER-ELIMINATE-LINE-SCANNING-PLAN.md as an expression-text scanner. Add it.
• annotations_forward_refs/type_checks.rs — third copy of the same heuristics; dedupe against the one real validator.
• Replace the ParamSpec guess and the numeric-tower-only bound check with real checks, or remove the diagnostics until they exist. A diagnostic whose own comment says "probably" must not be an error.
• Drop the uppercase-name implicit-alias heuristic in favour of binding information.
• Add off-suite regression tests for every rewritten rule. The defect is invisible to the conformance suite by construction — the suite is what these predicates were fitted to. Cases must come from the spec grammar, not from conformance/tests/.
• Re-run conformance after each rewrite and report the number honestly, including if it drops.

Note on the record

The conformance number is only worth what the implementation behind it is worth. Where a passing file is carried by predicates shaped to that file, the honest statement is that the file passes and the rule is not implemented. That is the case here, and it is being tracked openly rather than quietly rewritten.