From 7f350f631b0a7b354b0f4a59b6d4fe1977900f5a Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Tue, 11 Aug 2026 12:05:19 -0500 Subject: [PATCH 1/2] ledger: file #1221 -- a doc lint crashes when printing a hit that contains a glyph The coordinator asked whether this belongs under #1030 or needs its own number. It needs its own, and the reason is measured rather than argued. #1030 is 'keep the corpus cp1252-clean, because per-file gating lets the class recur'. THAT CANNOT FIX THIS, because the corpus this lint reads is REQUIRED to contain cp1252-unsafe characters: asvs_tally_lint walks rglob('*.md') over docs/ (:245) and explicitly reads docs/BACKLOG.md (:63-64), whose sanctioned banner alphabet is CLAUDE.md section 11's one machine-parsed holdout and every glyph of which is cp1252-unsafe. Its own comment at :153 records it already redding on backlog content. So a clean corpus is not achievable HERE even in principle. #1030 is about the CONTENT; this is about the INSTRUMENT. Both are needed and neither substitutes for the other. The defect shape is the interesting half: it fires ONLY on the failure path -- printing the hits -- so it is INVISIBLE WHILE THE LINT IS GREEN, and it converts a legible FAIL into a traceback at exactly the moment someone needs to read which document tripped it. A gate that is correct until it has something to say, and then unreadable. The item states the real decision rather than prescribing a patch: force UTF-8 on the script's own stdout -- an idiom already proven in-tree, since messagefoundry/__main__.py hardens stdout/stderr for the whole CLI and the harness and a bench script carry the same remedy -- or treat the crash as the corpus's problem. PYTHONIOENCODING=utf-8 is a workaround, not a fix, because CI and a developer shell will not both set it. And it names where to prove the fix: MAKE THE LINT FIRE on a document containing a glyph and read the output. A green run exercises none of it, which is the whole point. Fourth cp1252 surface in one day -- a U+21D2 in a ci.yml comment, three pre-existing U+2192 already on main in that same file, one in the coordinator playbook, and now this. The first three are #1030's subject; this one is not. Gates: 485 items each declaring exactly one status; ledger_check clean. cp1252 dogfood on my own added lines: the only unsafe character is U+1F522, the sanctioned banner glyph, which is the exemption recorded under #1030. --- docs/BACKLOG.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/docs/BACKLOG.md b/docs/BACKLOG.md index a2e6954d..bd058ad8 100644 --- a/docs/BACKLOG.md +++ b/docs/BACKLOG.md @@ -7671,3 +7671,18 @@ gate is the wrong shape, validation of the walk is the right one. **Cluster:** Connections and Transports. **Priority:** P3. **Verdict:** build. **Severity:** minor. +## 1221. A doc lint crashes when printing a hit that contains a glyph, so it is unreadable exactly when it fires + +> 🔢 **Filed 2026-08-11 -- found by Session B while fixing an unrelated false positive in the same file.** Value **5/10** -- Difficulty **2/10** -- _quick win_. `scripts/docs/asvs_tally_lint.py` raises `UnicodeEncodeError: 'charmap' codec can't encode character` on a stock Windows **cp1252** console when it prints a hit whose text contains a glyph. It fires **only on the failure path** -- printing the hits -- so it is **INVISIBLE WHILE THE LINT IS GREEN**, and it converts a legible FAIL into a traceback at exactly the moment somebody needs to read which document tripped it. **A gate that is correct until it has something to say, and then unreadable.** + +> **WHY THIS IS NOT MORE EVIDENCE UNDER #1030, WHICH WAS THE OBVIOUS CALL AND IS WRONG.** #1030 is *"keep the corpus cp1252-clean, because per-file gating lets the class recur"*. **That cannot fix this, because the corpus this lint reads is REQUIRED to contain cp1252-unsafe characters.** Measured: it walks `base.rglob("*.md")` over `docs/` (`:245`) and explicitly reads `docs/BACKLOG.md` (`:63-64`) -- the file whose **sanctioned banner alphabet** is CLAUDE.md §11's one machine-parsed holdout, every glyph of which is cp1252-unsafe. Its own comment at `:153` records it already redding on backlog content. So a clean corpus is not achievable *here* even in principle, and only making the tool robust addresses it. **#1030 is about the content; this is about the instrument.** Both are needed and neither substitutes. + +> **The decision this needs, and it is a real one rather than a patch.** Either a repo script **forces UTF-8 on its own stdout** (the engine CLI already does exactly this -- `messagefoundry/__main__.py` hardens `sys.stdout`/`sys.stderr`, and the harness and a bench script carry the same remedy), **or** the crash is treated as the corpus's problem and the script stays naive. Those have different blast radii: the first makes every repo script robust and is a one-line idiom already proven in-tree; the second is unachievable for any tool that reads the ledger. `PYTHONIOENCODING=utf-8` works around it either way and is **not** a fix, because CI and a developer shell will not both set it. + +> **FOURTH SURFACE IN ONE DAY**, which is the argument for doing it structurally rather than one file at a time: a `U+21D2` in a `ci.yml` comment, **three pre-existing `U+2192` already on `main` in that same file**, a `U+21D2` in the coordinator playbook, and now this. The first three are #1030's subject; this one is not. + +> **Scope note:** the fix belongs with whichever of the two answers is chosen, and the failure path is the place to prove it -- **make the lint FIRE on a document containing a glyph and read the output**, because a green run exercises none of this. That is the whole point of the item. + +**Cluster:** Developer tooling / CI. **Priority:** P3. **Verdict:** build. **Severity:** minor -- no product effect; it degrades a working gate into an unreadable one at the moment it matters. + + From 72b1d7132b044b85f5843fff01c875086e088b90 Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Tue, 11 Aug 2026 12:10:44 -0500 Subject: [PATCH 2/2] ledger: #1221 gains the measured census -- 140 arrows in the file the lint names The coordinator measured the corpus after I filed, and it moves the item from 'fires on the failure path in principle' to 'fires on a large fraction of possible failures in the primary corpus'. main's docs/BACKLOG.md -- the file asvs_tally_lint names at :63-64 -- carries 799 cp1252-unsafe codepoints across 29 distinct characters, including U+2192 x140. So the crash is reachable from any hit whose line carries an arrow, and there are 140 chances from that character alone. I re-derived the census rather than transcribing it, and that surfaced a decoding correction now recorded in the item: the first pass reported U+D83D x193 and U+DD22 x171, which are UTF-16 SURROGATE HALVES rather than characters. Read as codepoints they are U+1F522 x171 and U+1F6A7 x16. Same underlying data, but anyone grepping for U+D83D would find nothing and reasonably conclude the census was wrong. Count codepoints, not UTF-16 units. The item also now carries the in-tree hardening precedent with its edge cases, which is worth more than the idiom alone: __main__.py:52-55 uses getattr(stream, 'reconfigure', None) and calls it with errors='replace', with a comment naming the environments that lack or reject reconfigure -- PYTHONLEGACYWINDOWSSTDIO and pytest capture. Whoever builds this inherits that scar tissue instead of rediscovering it. Gates: 485 items each declaring exactly one status; #1221 still open. --- docs/BACKLOG.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/BACKLOG.md b/docs/BACKLOG.md index bd058ad8..e8ce4ff9 100644 --- a/docs/BACKLOG.md +++ b/docs/BACKLOG.md @@ -7679,6 +7679,8 @@ gate is the wrong shape, validation of the walk is the right one. > **The decision this needs, and it is a real one rather than a patch.** Either a repo script **forces UTF-8 on its own stdout** (the engine CLI already does exactly this -- `messagefoundry/__main__.py` hardens `sys.stdout`/`sys.stderr`, and the harness and a bench script carry the same remedy), **or** the crash is treated as the corpus's problem and the script stays naive. Those have different blast radii: the first makes every repo script robust and is a one-line idiom already proven in-tree; the second is unachievable for any tool that reads the ledger. `PYTHONIOENCODING=utf-8` works around it either way and is **not** a fix, because CI and a developer shell will not both set it. +> **MEASURED, AND IT IS NOT AN EDGE CASE WAITING FOR AN UNLUCKY DOCUMENT.** `main`'s `docs/BACKLOG.md` -- the file this lint names at `:63-64` -- carries **799 cp1252-unsafe codepoints across 29 distinct characters**, including **`U+2192` x140**, `U+1F522` x171, `U+2705` x138, `U+26A0` x91 and `U+26D4` x79. So the crash is reachable from any hit whose line carries one, and there are **140 chances from the arrow alone**. That is a large fraction of the possible failures in the primary corpus, not a corner. *(Count codepoints, not UTF-16 units: a first pass reported `U+D83D` x193, which is a SURROGATE HALF and not a character present in the file -- grepping for it finds nothing.)* + > **FOURTH SURFACE IN ONE DAY**, which is the argument for doing it structurally rather than one file at a time: a `U+21D2` in a `ci.yml` comment, **three pre-existing `U+2192` already on `main` in that same file**, a `U+21D2` in the coordinator playbook, and now this. The first three are #1030's subject; this one is not. > **Scope note:** the fix belongs with whichever of the two answers is chosen, and the failure path is the place to prove it -- **make the lint FIRE on a document containing a glyph and read the output**, because a green run exercises none of this. That is the whole point of the item.