Check the transcriptions against the vendored upstream sources

The keyword table, the %fallback set and "every parse function names its
rule" were all things only a careful reading could confirm, and the sort
of thing that quietly rots when the pin advances. Three tests now check
them mechanically:

- token rebuilds the keyword table from mkkeywordhash.c (now vendored
  alongside parse.y and tokenize.c) and compares all 147 entries and
  their TK_ codes;
- token rebuilds the %fallback set from parse.y and compares all 73,
  both resolved for the pinned build's feature flags;
- parser checks that every one of parse.y's 133 nonterminals is named
  somewhere in the parser, which is the repository rule read from the
  side that catches an omission.

All three passed on the first run, which is the answer I wanted but not
the one worth relying on next time.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JzBeCg7rjweVW3uGPg5G7T

Author: claude

CLAUDE.md

@@ -9,14 +9,18 @@ https://sqlite.org/lang.html.
 ## Rules
 
 - **Zero dependencies.** `go.mod` must never gain a `require` line.
+- The keyword table and the `%fallback` set are transcriptions, and tests in
+  `token` rebuild both from the vendored upstream sources to keep them
+  honest. Advancing the pin surfaces a keyword change as a failure.
 - **No parser generators.** Everything is hand-written recursive descent.
 - **Never edit `parser/testdata/*.test` by hand.** Corpus files are produced
   by `cmd/regenerate-parse` from SQLite's own test suite plus a real SQLite
   build (the oracle). `*.metadata.json` sidecars are updated by tooling
   (`go test ./parser -check-parse`), not by hand. The hand-written snapshot
   inputs under `parser/testdata/ast/` are the exception — see below.
 - Every nontrivial parse function carries a comment naming the `parse.y`
-  rule(s) it implements.
+  rule(s) it implements. `TestEveryRuleIsNamed` enforces this from the other
+  side: no nonterminal of the vendored grammar may go unmentioned.
 - Error messages must match SQLite's parser byte-for-byte
   (`near "X": syntax error`, `unrecognized token: "X"`, `incomplete input`).
 

internal/reference/README.md

@@ -1,16 +1,29 @@
 # Vendored SQLite sources (documentation only)
 
-These files are the grammar and tokenizer meyer is a port of, kept here so
-that the `parse.y` rule named in a parser comment can be read without a
-SQLite checkout. **Nothing in the repository processes them**: no parser
-generator runs, no build step reads them, and no test depends on them. They
-are reference material for humans, and the directory deliberately contains
+These files are the grammar, tokenizer and keyword table meyer is a port of,
+kept here so that the `parse.y` rule named in a parser comment can be read
+without a SQLite checkout. **No tool processes them**: no parser generator
+runs and no build step reads them, and the directory deliberately contains
 no Go file, so it is not a package and `./...` does not reach it.
 
+Two tests do read them, to check transcriptions that would otherwise only
+ever be verified by hand:
+
+- `token`'s tests rebuild the keyword table and the `%fallback` set from
+  `mkkeywordhash.c` and `parse.y` and compare them with meyer's, resolved
+  for the pinned build's feature flags.
+- `parser`'s tests check that every nonterminal in `parse.y` is named by a
+  comment somewhere in the parser, which is how the repository's "every
+  nontrivial parse function names its rule" rule is enforced.
+
+Advancing the pin therefore surfaces a grammar or keyword change as a test
+failure rather than as a silent divergence.
+
 | file | upstream path |
 |---|---|
 | `parse.y` | `src/parse.y` |
 | `tokenize.c` | `src/tokenize.c` |
+| `mkkeywordhash.c` | `tool/mkkeywordhash.c` |
 
 Both come from SQLite 3.53.4, the release pinned in
 `cmd/regenerate-parse/main.go` and recorded in