llms: close the evidence and identity gaps; row the rest (external-AI probe)#99
Merged
Conversation
… probe)
WORKSTREAM A, shipped. Three assistants were asked to describe Columna; the desk graded the results.
Every reported gap VERIFIED before editing — llms.txt contained ZERO occurrences of: benchmark,
Zenodo, papers, positions, Atlas, independent, Huayin, Irena, Apache, and "semantic layer".
A1 — THE EVIDENCE GAP. One assistant described the mechanism accurately and the evidence not at all,
because llms.txt named none of it. New "The evidence" section after "Start here": the six papers
chronological with DOIs, the public benchmark kit (frozen key, raw answers, scoring driver, errata),
/case with its recorded transcripts, /positions, /atlas.
A2 — THE IDENTITY GAP. A second assistant reported "the company behind it is datumwise.ai". New "Who"
section, ratified copy, placed HIGH (before the mechanism) so an assistant that stops reading early
still gets the identity right.
A3 — THE CATEGORY LEAK. Two assistants filed Columna as "a semantic layer". Ratified clause appended
to the summary block: it declares meaning, adjudicates at publish, and executes queries itself —
which is why it can refuse.
A4 — llms-full sync VERIFIED, not assumed: it reads llms.txt verbatim at build, and all three
additions were confirmed present in the built dist/llms-full.txt.
A5 — ROWED, not fixed: columna-core's PyPI summary is off-voice ("the column-foundation analytic
framework (multi-table, transport-based, correctness-governed)"), matching no other surface. Folded
into AW-5 — both it and the README only reach PyPI with a release.
ROWS (do not fix here):
· OF-18 gains the FORM-PRIMACY note — Chapter 26 presents long forms as primary and the shipped
short forms as parentheticals, so a CAREFUL reader learns the form the parser rejects. That is the
defect an external reader tripped on (MANIFOLD without VERSION, LEVEL without `= <column>`,
MEASURE without `FROM … AS …`) while getting HIERARCHY RIGHT — the §26.6 correction working in the
wild, and the control in the experiment. The audit must cover form primacy, not only status marks.
· OF-21 NEW — should the map layer live in a separate FILE from the logical Manifold? Insulation is
enforced today at the DESCRIBE boundary (test-enforced); Huayin's intent is the DOCUMENT boundary
(structural). For/against recorded; the desk drafts the fork document.
· OF-22 NEW — a meta-refresh stub returns HTTP 200, so crawlers can hold retired copy indefinitely.
Two of three assistants quoted PURGED "metrics engine" positioning from the retired /launch page;
nothing on the live site says it (verified — the byte-preserved archive is unserved). This upgrades
the pending 308 from a nicety to a live misinformation vector.
WORKSTREAM B — PROPOSAL ONLY, no code: specs/grammar_page_proposal_v0_1.md. A grammar page generated
from the parser (keywords from `_KW`, signatures from its own patterns, examples the build FAILS on if
they do not parse). It carries the desk's CRITICAL CORRECTION: the page must NOT say "physical binding
is part of the declaration" — the architecture is the opposite and the code enforces it (§2b / C-2;
describe emits logical names only; the standing insulation test asserts physical identifiers never
cross). The page teaches syntax AND layering: a .cml co-locates logical declarations (what agents see)
with MAP clauses (`= <column>`, `FROM … AS …`, `VIA t(a,b)`, `REJECT …`) that are engine-visible only,
each labelled as such, with §2b as the page's frame rather than a footnote. Recommended route
/docs/grammar (sibling of the three manuals, not inside one — generated output should not make a
ratified manual's build fail), with four open questions for the desk.
Build green (29 pages); prose-coherence green; purged-grammar guard green.
Co-Authored-By: Claude <noreply@anthropic.com>
…the parser
Workstream B, built on Huayin's ATTR ruling. Route RATIFIED as /docs/grammar — a sibling of the three
manuals, not a section inside one: generated output must not be able to fail a ratified manual's build
on a package change, and ratifiable prose stays distinct from recomputed content.
THE TRIO, three properties from three deliberately different artifacts:
1. STRUCTURE ← the parser's MODULE DOCSTRING grammar block (already teaching shape: statements at
top level, inline clauses nested — ATTR under LEVEL, M_ANCHOR/FAMILY under MEASURE).
2. EXAMPLES ← executed against the shipped parser; the generator EXITS NON-ZERO if any fails.
3. COMPLETENESS ← _KW, used ONLY to assert coverage: every token must appear as a statement or an
inline clause, or the build fails.
A LEXING SET IS NOT A GRAMMAR TAXONOMY — recorded as a build-doc line. _KW says what may BEGIN A LINE;
structuring a page from it would present ATTR as a false peer of MEASURE (faithful to the parser,
misleading to the reader). Generated docs must NAME the artifact they derive structure from, and why —
this page states all three sources on its face.
§2b IS THE PAGE'S FRAME, not a footnote, because this is exactly where a reader forms the wrong mental
model. The page states that a .cml CO-LOCATES two layers — logical declarations (what agents see over
describe) and MAP clauses (engine-visible only) — and LABELS every map clause as such. The desk's
earlier framing ("physical binding is part of the declaration") is the inverse of the architecture and
does not appear anywhere on the page.
TWO THINGS THE MECHANISM CAUGHT DURING THE BUILD, both mine:
· The completeness assertion fired on ATTR — my documented-set scan read only nested clauses, not
SIGNATURE text, and ATTR lives inline in LEVEL's signature. Fixed the scan rather than
supplementing ATTR, which would have restored the false-peer framing the ruling exists to prevent.
· The example verifier rejected my own UNIVERSE example — a standalone universe with no LEVELs. All
examples are now COMPLETE, VALID documents, which is right on the merits too: the probe's failure
was a whole document the parser rejected, so a page of fragments would not have caught it.
DOCSTRING GAP, found by the assertion on first run: ASSERT is in _KW but ABSENT from the docstring's
grammar block (8 of 9 tokens present). Rather than fail forever on a package-side omission, ASSERT is
covered by an explicit, visible SUPPLEMENT entry with its signatures read from the parser's own
patterns and a note saying why. The guarantee holds — a token in NEITHER the docstring nor the
supplement still fails the build — so the exception is declared, not silent. Upstream fix rowed.
BOTH FAIL-CLOSED PATHS VERIFIED BY PROVOKING THEM, not by reading:
· an example the parser rejects → exit 1, "grammar-page examples DO NOT PARSE"
· a _KW token documented nowhere → exit 1, "COMPLETENESS ASSERTION FAILED: ['NEWKEYWORD']"
Wired into BOTH website jobs beside the other generators, so the page describes the pinned shipped
package on the deploy path. /learn lists it with the manuals; llms.txt + llms-full carry the
"read this FIRST" pointer, naming that Chapter 26 presents long forms as primary.
Build green (30 pages); prose-coherence green.
Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Workstream A ships here. Workstream B is a proposal only — no code.
Every reported gap verified before editing
llms.txtcontained zero occurrences of: benchmark, Zenodo, papers, positions, Atlas, independent, Huayin, Irena, Apache, and "semantic layer". The diagnosis was exactly right.A1 · The evidence gap
New "The evidence" section after "Start here": six papers chronological with DOIs, the benchmark kit (frozen key, raw answers, scoring driver, errata),
/casewith recorded transcripts,/positions,/atlas.A2 · The identity gap
New "Who" section, ratified copy — placed high, before the mechanism, so an assistant that stops reading early still gets the identity right. (An assistant had reported "the company behind it is datumwise.ai".)
A3 · The category leak
Ratified clause appended to the summary block: Columna declares meaning, adjudicates at publish, and executes queries itself — which is why it can refuse.
A4 · llms-full sync — verified, not assumed
It reads
llms.txtverbatim at build; all three additions confirmed present in the builtdist/llms-full.txt.A5 · Rowed, not fixed
columna-core's PyPI summary is off-voice — "the column-foundation analytic framework (multi-table, transport-based, correctness-governed)" — matching no other surface. Folded into AW-5: both it and the README only reach PyPI with a release.Rows
HIERARCHYwas the one construct the model got right — the §26.6 correction working in the wild five days after landing, and the control in the experiment./launch, and nothing on the live site says it (verified; the byte-preserved archive is unserved). This upgrades the pending 308 from a nicety to a live misinformation vector.Workstream B — proposal only
specs/grammar_page_proposal_v0_1.md. A grammar page generated from the parser — keywords from_KW, signatures from its own patterns, and examples the build FAILS on if they do not parse (the property the manuals lacked: nothing in CI ever read Chapter 26's examples).It carries your critical correction: the page must not say physical binding is part of the declaration. The architecture is the opposite and the code enforces it (§2b / C-2;
describeemits logical names only; the standing insulation test asserts physical identifiers never cross). So the page teaches syntax and layering together — a.cmlco-locates logical declarations (what agents see) with map clauses (= <column>,FROM … AS …,VIA t(a,b),REJECT …) that are engine-visible only, each labelled as such, with §2b as the page's frame, not a footnote — because the page is exactly where a reader forms the wrong mental model.Recommended route:
/docs/grammar, a sibling of the three manuals rather than a section inside one — generated output shouldn't make a ratified manual's build fail on a package change, and it keeps ratifiable prose distinct from recomputed content. Four open questions for the desk, including howATTRshould be listed (it's in_KWbut is an inline clause ofLEVEL— accurate to the parser, misleading to a reader) and whether OF-21 should settle first.Gates
Build green (29 pages) · prose-coherence green · purged-grammar guard green.
🤖 Generated with Claude Code