From f8adcc8b6d49679cee09c2fdc34242611cd41ad7 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Thu, 30 Jul 2026 01:30:23 -0700 Subject: [PATCH 1/9] =?UTF-8?q?Divide=20on=20the=20=E9=97=B4=E9=9A=94?= =?UTF-8?q?=E5=8F=B7=20and=20record=20its=20offsets=20(#298)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Each dot is read by its own typography's standard. The Japanese nakaguro (U+30FB/U+FF65) is roster/divider typography whose pieces the script license already reads correctly (#272: 高橋・一郎 family-first, マイケル・ジャクソン positional) -- it stays an unconditional pure separator that records nothing. Only the Chinese 间隔号 U+00B7 carries the transcription meaning (威廉·莎士比亚 keeps source order), so only it records, as interpunct_offsets mirroring comma_offsets: main-region only, gated by the same flag that keeps nickname commas out of comma_offsets (record_commas renamed record_offsets). U+00B7 cannot separate unconditionally -- it is also the Catalan punt volat, interior to legitimate names (Gal·la) -- so it is the first context-sensitive separator: it divides only between classified-script neighbors (region-local flanks; an edge B7 stays token text), handled in _tokenize_region where the index exists. The consumers (order and segmentation suppression) arrive in the following commits. Co-Authored-By: Claude Fable 5 --- AGENTS.md | 2 +- docs/release_log.rst | 2 +- nameparser/_pipeline/_state.py | 5 +- nameparser/_pipeline/_tokenize.py | 79 ++++++++++++++++++++++-------- tests/v2/pipeline/test_state.py | 6 ++- tests/v2/pipeline/test_tokenize.py | 44 +++++++++++++++++ 6 files changed, 112 insertions(+), 26 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 790d70d..4b3c35e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -75,7 +75,7 @@ logging.getLogger('HumanName').setLevel(logging.DEBUG) The library has two layers: `nameparser/config/` (data) and `nameparser/parser.py` (logic). -**Design philosophy — positional and language-agnostic.** The parser assigns parts by *position* plus small sets of words that join to neighbors; it never detects language. A name's language can't be reliably inferred from Latin-script transliteration ("Ali" is Arabic or Italian; "Van"/"Della"/"Bin" are first names in some cultures, particles in others), so language-specific rules belong in opt-in `Constants` config, never global defaults. Many "wrong for language X" reports (#133, #150, #130, #85, #103, #146, #83) are irreducible ambiguities — e.g. `de Mesnil` (want last name) vs `Van Johnson` (want first name) are the same `[prefix][word]` shape. Before adding a rule, confirm it doesn't break the opposite case (run the full suite — Portuguese and "Van Johnson" tests are the usual canaries). **The one scoped exception (2.1, #271/#272): script-conditional behavior is permitted exactly where the SCRIPT ITSELF — not statistics about it — determines the convention.** The never-detect-language rule above is about Latin *transliteration*, where the signal genuinely is destroyed; native script is a different question, and it is answered per behavior rather than per script. Four defaults fall out of it. Wholly-Han, wholly-Hangul and kana-licensed names read family-first (`Policy.script_orders`) — no language detection needed, because zh and ja both write family-first in native script, so order cannot be misread even though the language is unknowable. Unspaced hangul splits into surname + given name (`config/surnames.py` ships the Korean census list as DEFAULT vocabulary) — nothing but Korean is written in hangul and the surnames are a closed census set, and the vocabulary is self-selecting besides: a hangul entry can only ever match hangul text. Hiragana licenses Japanese (#272) — a name whose characters stay inside Han∪kana while carrying at least one kana cannot be Chinese (the kana rules it out) and is not a transcription (foreign names are transcribed in katakana ALONE, マイケル has no kanji), so 高橋みなみ and 山田 エミ read family-first too; mechanically they resolve to the HIRAGANA entry, the license's carrier key. PURE katakana is excluded and keeps the positional default: マイケル・ジャクソン is a transcribed foreign name in its source order. The nakaguro belongs to the same doctrine but is decided a layer down: U+30FB and its halfwidth twin U+FF65 separate tokens like whitespace, unconditionally and in tokenize, so neither policy opt-out (`script_orders={}`, `segment_scripts=()`) reaches it — the codepoints are CJK-only and appear in no other script's names, which is what licenses a tokenize-level rule where U+00B7 (also the Catalan punt volat, interior to Gal·la) would need a guard. Han segmentation stays OPT-IN (`locales.ZH` for Chinese, `locales.JA` for Japanese) — a zh surname list corrupts Japanese kanji names, since 高 is a common Chinese surname and 高橋一郎 would split 高+橋一郎 where the correct reading is 高橋+一郎; no surname list divides a kanji name at all, so `locales.JA` activates the stage and a pluggable `Parser(segmenter=...)` does the dividing. Latin-script input is never touched by any of this: "Kim Min-jun" is genuinely order-ambiguous and stays governed by `name_order` and opt-in packs. Before adding a script-conditional rule, work out which of the three it is — certain, certain for this one behavior only, or a statistical guess wearing a script's clothes. +**Design philosophy — positional and language-agnostic.** The parser assigns parts by *position* plus small sets of words that join to neighbors; it never detects language. A name's language can't be reliably inferred from Latin-script transliteration ("Ali" is Arabic or Italian; "Van"/"Della"/"Bin" are first names in some cultures, particles in others), so language-specific rules belong in opt-in `Constants` config, never global defaults. Many "wrong for language X" reports (#133, #150, #130, #85, #103, #146, #83) are irreducible ambiguities — e.g. `de Mesnil` (want last name) vs `Van Johnson` (want first name) are the same `[prefix][word]` shape. Before adding a rule, confirm it doesn't break the opposite case (run the full suite — Portuguese and "Van Johnson" tests are the usual canaries). **The one scoped exception (2.1, #271/#272): script-conditional behavior is permitted exactly where the SCRIPT ITSELF — not statistics about it — determines the convention.** The never-detect-language rule above is about Latin *transliteration*, where the signal genuinely is destroyed; native script is a different question, and it is answered per behavior rather than per script. Four defaults fall out of it. Wholly-Han, wholly-Hangul and kana-licensed names read family-first (`Policy.script_orders`) — no language detection needed, because zh and ja both write family-first in native script, so order cannot be misread even though the language is unknowable. Unspaced hangul splits into surname + given name (`config/surnames.py` ships the Korean census list as DEFAULT vocabulary) — nothing but Korean is written in hangul and the surnames are a closed census set, and the vocabulary is self-selecting besides: a hangul entry can only ever match hangul text. Hiragana licenses Japanese (#272) — a name whose characters stay inside Han∪kana while carrying at least one kana cannot be Chinese (the kana rules it out) and is not a transcription (foreign names are transcribed in katakana ALONE, マイケル has no kanji), so 高橋みなみ and 山田 エミ read family-first too; mechanically they resolve to the HIRAGANA entry, the license's carrier key. PURE katakana is excluded and keeps the positional default: マイケル・ジャクソン is a transcribed foreign name in its source order. The nakaguro belongs to the same doctrine but is decided a layer down: U+30FB and its halfwidth twin U+FF65 separate tokens like whitespace, unconditionally and in tokenize, so neither policy opt-out (`script_orders={}`, `segment_scripts=()`) reaches it — the codepoints are CJK-only and appear in no other script's names, which is what licenses a tokenize-level rule where U+00B7 (also the Catalan punt volat, interior to Gal·la) needs the flanked-by-classified-script guard `_tokenize_region` gives it (#298). Han segmentation stays OPT-IN (`locales.ZH` for Chinese, `locales.JA` for Japanese) — a zh surname list corrupts Japanese kanji names, since 高 is a common Chinese surname and 高橋一郎 would split 高+橋一郎 where the correct reading is 高橋+一郎; no surname list divides a kanji name at all, so `locales.JA` activates the stage and a pluggable `Parser(segmenter=...)` does the dividing. Latin-script input is never touched by any of this: "Kim Min-jun" is genuinely order-ambiguous and stays governed by `name_order` and opt-in packs. Before adding a script-conditional rule, work out which of the three it is — certain, certain for this one behavior only, or a statistical guess wearing a script's clothes. ### Configuration layer (`nameparser/config/`) diff --git a/docs/release_log.rst b/docs/release_log.rst index 50a39ad..d6306fc 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -23,7 +23,7 @@ Release Log - Fix unspaced Korean names not splitting: the census surname list now ships as default vocabulary (``Lexicon.surnames``) with hangul segmentation on by default (``Policy.segment_scripts``), so ``"김민준"`` parses family ``김``, given ``민준`` where 1.x returned the whole string as ``first``. Rendering follows the split, so ``str(HumanName("김민준"))`` is now ``"민준 김"`` where 1.x echoed the input back unchanged. Nothing but Korean is written in hangul and its surnames are a closed census set, which is what makes the split safe as a default rather than a pack. **Default-on**, and it reaches ``HumanName`` too. ``Policy(segment_scripts=())`` turns the split off; ``Policy(script_orders={})`` separately restores the positional reading; clearing both restores 2.0 behavior exactly (#271) - Fix Japanese names carrying kana parsing given-first: the family-first rule above extends to any name whose characters stay within kanji and kana while carrying at least one kana character, so ``"高橋 みなみ"`` gives family ``高橋`` and ``"山田 エミ"`` family ``山田`` where 1.x read both the other way round, and a lone such token (``"高橋みなみ"``, ``"みなみ"``) lands in ``family``/``last`` where 1.x put it in ``given``/``first``. The reasoning is the one hangul already uses: hiragana never transcribes a foreign name, and a transcription is kana ALONE, so kanji-plus-kana is a Japanese person's name written in Japanese order. A name written **wholly in katakana** is deliberately excluded and stays positional — it is predominantly a transcribed foreign name (``"マイケル ジャクソン"``) already in given-first order. **Default-on: changes parse output for kana-bearing Japanese names**, through ``HumanName`` as well as the 2.0 API, and the spaced forms change what the name renders as (``str(HumanName("高橋 みなみ"))`` is now ``"みなみ 高橋"``). ``Policy(script_orders={})`` clears this entry along with the Han and Hangul ones (#272) - Fix names containing 〆 (U+3006, the shime mark that opens Japanese surnames like 〆木 and 〆谷) parsing given-first: the script classifier now counts 〆 as Han, extending the 々 entry the table already carries, and going a step further than it — 々 is Script=Han and merely outside the ideograph blocks, while 〆 is Script=Common — so these names take the East Asian family-first reading like any other wholly-Han name. ``Policy(script_orders={})`` restores the positional reading for these names exactly as for other wholly-Han names (#303) - - Fix the katakana middle dot ``・`` (U+30FB, and its halfwidth twin U+FF65) being read as part of a name rather than as the divider it is: it now separates tokens exactly as a space does. A transcribed foreign name therefore divides into its parts and, being wholly katakana, keeps its source order — ``"マイケル・ジャクソン"`` gives given ``マイケル``, family ``ジャクソン``, where 1.x left the whole string in ``first`` — while a kanji pair written the same way takes the family-first rule (``"高橋・一郎"`` → family ``高橋``). Native Japanese names never contain this character and no other script's names use it, so the separation is unconditional, which also means it is **not** covered by the two policy opt-outs. Rendering follows, the way the Korean split's does: the dot comes back as a space, so ``str(HumanName("マイケル・ジャクソン"))`` is now ``"マイケル ジャクソン"``, and that reaches delimited content too — the nickname in ``"山田 太郎 (マイケル・ジャクソン)"`` renders ``"マイケル ジャクソン"``. The Chinese interpunct ``·`` is deliberately NOT included: U+00B7 is also the Catalan punt volat and appears inside legitimate names (``Gal·la``) (#272) + - Fix the katakana middle dot ``・`` (U+30FB, and its halfwidth twin U+FF65) being read as part of a name rather than as the divider it is: it now separates tokens exactly as a space does. A transcribed foreign name therefore divides into its parts and, being wholly katakana, keeps its source order — ``"マイケル・ジャクソン"`` gives given ``マイケル``, family ``ジャクソン``, where 1.x left the whole string in ``first`` — while a kanji pair written the same way takes the family-first rule (``"高橋・一郎"`` → family ``高橋``). Native Japanese names never contain this character and no other script's names use it, so the separation is unconditional, which also means it is **not** covered by the two policy opt-outs. Rendering follows, the way the Korean split's does: the dot comes back as a space, so ``str(HumanName("マイケル・ジャクソン"))`` is now ``"マイケル ジャクソン"``, and that reaches delimited content too — the nickname in ``"山田 太郎 (マイケル・ジャクソン)"`` renders ``"マイケル ジャクソン"``. The Chinese interpunct ``·`` is not an unconditional separator like these two: U+00B7 is also the Catalan punt volat and appears inside legitimate names (``Gal·la``), so it divides only between classified-script characters — the transcription treatment it marks is #298's (#272) - Fix NFD-decomposed input missing the East Asian defaults entirely: script classification now normalizes to NFC before deciding, so a Korean or Japanese name typed on macOS — where decomposed text is routine — gets the same order rule as its composed twin, which it silently did not before. Segmentation MATCHING deliberately stays raw, so an unspaced NFD hangul name is ordered correctly but not split, rather than being split in the wrong place. One gotcha worth stating plainly: parse output preserves the encoding it was given, so for NFD input ``name.family == "김"`` is ``False`` even though it is the same name — compare NFC-normalized text when comparing across encodings (#272) * 2.0.0 - July 27, 2026 diff --git a/nameparser/_pipeline/_state.py b/nameparser/_pipeline/_state.py index 6cf05bd..d524e4e 100644 --- a/nameparser/_pipeline/_state.py +++ b/nameparser/_pipeline/_state.py @@ -66,7 +66,9 @@ class ParseState: """Carried through the stage fold. Frozen; stages return copies via dataclasses.replace. Fields are filled progressively: extract_delimited -> extracted/masked; tokenize -> tokens (span- - sorted)/comma_offsets; segment -> segments/structure; + sorted)/comma_offsets/interpunct_offsets (the 间隔号 offsets the + order and segmentation decisions consult, #298; the nakaguro + separators record NOTHING); segment -> segments/structure; script_segment -> tokens and segments again (the one stage that changes the token COUNT: an unspaced CJK token splits into n+1 pieces, still as sub-slices of the original, and every later index @@ -94,6 +96,7 @@ class ParseState: masked: tuple[Span, ...] = () tokens: tuple[WorkToken, ...] = () comma_offsets: tuple[int, ...] = () + interpunct_offsets: tuple[int, ...] = () segments: tuple[tuple[int, ...], ...] = () structure: Structure = Structure.NO_COMMA # pieces[s][p] = run of token indices: piece p of segment s. diff --git a/nameparser/_pipeline/_tokenize.py b/nameparser/_pipeline/_tokenize.py index 0d4643d..1fdb1d2 100644 --- a/nameparser/_pipeline/_tokenize.py +++ b/nameparser/_pipeline/_tokenize.py @@ -3,7 +3,8 @@ Consumes: original, masked (regions to skip), extracted (regions that tokenize with a pre-set role). Produces: tokens (span-sorted WorkTokens; text always == original -slice), comma_offsets (segmentation points; never tokens). +slice), comma_offsets (segmentation points; never tokens), +interpunct_offsets (间隔号 transcription markers, #298; never tokens). Reads: Policy.strip_emoji, Policy.strip_bidi. There is NO text-rewriting normalize stage: whitespace collapsing, @@ -12,7 +13,10 @@ separators and never enter a token, so spans always index the original exactly as given. Whitespace and the name-dot are unconditional; emoji/bidi stripping alone is policy-gated -(Policy.strip_emoji/strip_bidi). +(Policy.strip_emoji/strip_bidi). The Chinese interpunct U+00B7 is the +one context-sensitive separator: it divides (and records) only +between classified-script neighbors, so its rule lives in +_tokenize_region, which has the index _ignorable lacks. v1's squash_emoji/squash_bidi REMOVED the char and joined neighbors ('A\U0001f600B' -> 'AB'); here an ignorable char is a SEPARATOR @@ -28,6 +32,7 @@ from nameparser._pipeline._state import ( COMMA_CHARS, ParseState, WorkToken, ) +from nameparser._policy import _SCRIPT_RANGES, _script_matcher from nameparser._types import Role, Span # Ported from v1 (nameparser/config/regexes.py, "emoji" and "bidi") -- @@ -45,14 +50,20 @@ # a foreign name transcribed into katakana (マイケル・ジャクソン) -- # native names never contain them, so they separate unconditionally, # like whitespace (amendment 2026-07-29 section 1b). The Chinese -# interpunct U+00B7 (威廉·莎士比亚) is deliberately NOT here: it is -# also the Catalan punt volat, which sits INSIDE a single name piece -# (Gal·la), so separating on it unconditionally would break names it -# has no business touching. Admitting it would need a flanked-by-CJK -# guard, which is a different rule from this set's "these codepoints -# are always separators". +# interpunct U+00B7 is context-sensitive instead -- see _INTERPUNCT +# below. _NAME_DOT_SEPARATORS = frozenset({"\u30FB", "\uFF65"}) +_INTERPUNCT = "\u00B7" +# Per-CHAR classifier for the interpunct's flank guard: a one-char +# string is wholly-classified iff the character is. U+00B7 cannot be +# an unconditional separator like the name dots above -- it is also +# the Catalan punt volat, INTERIOR to legitimate names (Gal\u00B7la) -- so +# it divides only between classified-script characters: the first +# context-sensitive separator rule, which is why it lives in +# _tokenize_region (where the index exists) and not in _ignorable. +_classified_char = _script_matcher(*_SCRIPT_RANGES, whole=True) + def _ignorable(ch: str, state: ParseState) -> bool: if ch.isspace(): @@ -64,11 +75,16 @@ def _ignorable(ch: str, state: ParseState) -> bool: return False # unconditional, like whitespace -- not policy-gated, so this sits # ahead of the policy checks rather than in _tokenize_region beside - # COMMA_CHARS (commas RECORD an offset; these dots must not). The - # only load-bearing constraint is "after the isascii fast path" - # (both dots are non-ASCII); being ahead of the bidi/emoji checks - # below is NOT load-bearing -- the three sets are disjoint, so a - # future edit is free to reorder them. + # COMMA_CHARS (commas RECORD an offset; these dots must not: the + # nakaguro is Japanese roster/divider typography whose pieces the + # script license already reads correctly -- #272, 高橋・一郎 + # family-first, マイケル・ジャクソン positional -- while only the + # Chinese 间隔号 carries the transcription meaning and records, + # #298, handled in _tokenize_region). The only load-bearing + # constraint is "after the isascii fast path" (both dots are + # non-ASCII); being ahead of the bidi/emoji checks below is NOT + # load-bearing -- the three sets are disjoint, so a future edit is + # free to reorder them. if ch in _NAME_DOT_SEPARATORS: return True if state.policy.strip_bidi and _BIDI.match(ch): @@ -80,19 +96,36 @@ def _ignorable(ch: str, state: ParseState) -> bool: def _tokenize_region(state: ParseState, start: int, end: int, - role: Role | None, record_commas: bool, - tokens: list[WorkToken], commas: list[int]) -> None: + role: Role | None, record_offsets: bool, + tokens: list[WorkToken], commas: list[int], + interpuncts: list[int]) -> None: text = state.original tok_start: int | None = None for i in range(start, end): ch = text[i] - if ch in COMMA_CHARS or _ignorable(ch, state): + is_separator = ch in COMMA_CHARS or _ignorable(ch, state) + if not is_separator and ch == _INTERPUNCT: + # flanks are region-local: a B7 at a region edge has a + # masked or absent neighbor and stays token text. + # Region-local on purpose and defensively: under the + # default delimiters a masked span's edge character is + # never classified, so the two bounds are indistinguishable + # -- but a custom delimiter ending in a classified + # character would otherwise let a B7 split and record + # across a mask seam. + is_separator = (start < i < end - 1 + and _classified_char(text[i - 1]) + and _classified_char(text[i + 1])) + if is_separator: if tok_start is not None: tokens.append(WorkToken(text[tok_start:i], Span(tok_start, i), role=role)) tok_start = None - if ch in COMMA_CHARS and record_commas: - commas.append(i) + if record_offsets: + if ch in COMMA_CHARS: + commas.append(i) + elif ch == _INTERPUNCT: + interpuncts.append(i) continue if tok_start is None: tok_start = i @@ -104,17 +137,20 @@ def _tokenize_region(state: ParseState, start: int, end: int, def tokenize(state: ParseState) -> ParseState: tokens: list[WorkToken] = [] commas: list[int] = [] + interpuncts: list[int] = [] # main stream: everything outside masked regions boundaries = [0] for m in state.masked: boundaries.extend((m.start, m.end)) boundaries.append(len(state.original)) for start, end in zip(boundaries[::2], boundaries[1::2]): - _tokenize_region(state, start, end, None, True, tokens, commas) - # extracted regions: pre-set role, commas are mere separators + _tokenize_region(state, start, end, None, True, tokens, commas, + interpuncts) + # extracted regions: pre-set role, commas and interpuncts are mere + # separators for role, inner in state.extracted: _tokenize_region(state, inner.start, inner.end, role, False, - tokens, commas) + tokens, commas, interpuncts) tokens.sort(key=lambda t: t.span) # extract_delimited runs before tokens exist, so its ambiguities # carry a character offset instead of an index. Resolve them now @@ -147,4 +183,5 @@ def _containing(offset: int) -> tuple[int, ...]: for a in ambiguities) return dataclasses.replace(state, tokens=tuple(tokens), comma_offsets=tuple(sorted(commas)), + interpunct_offsets=tuple(sorted(interpuncts)), ambiguities=ambiguities) diff --git a/tests/v2/pipeline/test_state.py b/tests/v2/pipeline/test_state.py index fe5c167..03676a3 100644 --- a/tests/v2/pipeline/test_state.py +++ b/tests/v2/pipeline/test_state.py @@ -15,7 +15,8 @@ def test_state_defaults_are_empty() -> None: assert s.tokens == () and s.segments == () and s.pieces == () assert s.structure is Structure.NO_COMMA assert s.ambiguities == () and s.extracted == () and s.masked == () - assert s.comma_offsets == () and s.dropped == () and s.piece_tags == () + assert s.comma_offsets == () and s.interpunct_offsets == () + assert s.dropped == () and s.piece_tags == () assert s.segmenter is None @@ -50,7 +51,8 @@ def test_stage_field_ownership() -> None: # tokenize also rewrites ambiguities: extract_delimited runs # before tokens exist, so its UNBALANCED_DELIMITER entries carry # a character offset that tokenize resolves to a token index - "tokenize": {"tokens", "comma_offsets", "ambiguities"}, + "tokenize": {"tokens", "comma_offsets", "interpunct_offsets", + "ambiguities"}, "segment": {"segments", "structure", "ambiguities"}, # script_segment splits one unspaced CJK token into n+1 pieces # (n = 1 from the vocabulary, any n from a segmenter), so it diff --git a/tests/v2/pipeline/test_tokenize.py b/tests/v2/pipeline/test_tokenize.py index 96b18c4..ed72655 100644 --- a/tests/v2/pipeline/test_tokenize.py +++ b/tests/v2/pipeline/test_tokenize.py @@ -113,6 +113,50 @@ def test_nakaguro_next_to_a_real_comma_still_only_the_comma_offsets() -> None: assert state.comma_offsets == (5,) +def test_interpunct_splits_between_classified_characters() -> None: + state = _tokenized("威廉·莎士比亚") + assert [t.text for t in state.tokens] == ["威廉", "莎士比亚"] + assert state.interpunct_offsets == (2,) + + +def test_interpunct_stays_in_latin_names() -> None: + # the Catalan punt volat is INTERIOR to names; Latin neighbors + # never qualify + state = _tokenized("Gal·la Marcet") + assert [t.text for t in state.tokens] == ["Gal·la", "Marcet"] + assert state.interpunct_offsets == () + + +def test_interpunct_between_kana_splits() -> None: + # sloppy-IME katakana transcription + state = _tokenized("マイケル·ジャクソン") + assert [t.text for t in state.tokens] == ["マイケル", "ジャクソン"] + + +def test_interpunct_at_edges_stays() -> None: + # no classified neighbor on one side: not a divider position + state = _tokenized("·威廉") + assert [t.text for t in state.tokens] == ["·威廉"] + + +def test_interpunct_in_delimited_regions_splits_but_does_not_record() -> None: + # a B7 inside a nickname must not mark the OUTER name as a + # transcription -- same flag that keeps nickname commas out of + # comma_offsets + state = _tokenized("山田 太郎 (マイケル·ジャクソン)") + assert state.interpunct_offsets == () + assert "マイケル" in [t.text for t in state.tokens] + + +def test_nakaguro_never_records() -> None: + # the Japanese dot is roster/divider typography, not the + # transcription marker: it separates (#272) and records nothing -- + # 高橋・一郎 keeps its family-first roster reading downstream + state = _tokenized("威廉・莎士比亚") + assert [t.text for t in state.tokens] == ["威廉", "莎士比亚"] + assert state.interpunct_offsets == () + + def test_halfwidth_nakaguro_separates_too() -> None: # U+FF65 between halfwidth katakana -- build the string from # escapes and VERIFY the codepoints (the U+F900 homoglyph lesson): From e49d1d76deb30065bf2cc30d119a42b660b4f23f Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Thu, 30 Jul 2026 01:55:35 -0700 Subject: [PATCH 2/9] =?UTF-8?q?=E9=97=B4=E9=9A=94=E5=8F=B7-divided=20names?= =?UTF-8?q?=20keep=20source=20order=20(#298)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A B7-divided name is a transcription: the dot plays for Han exactly the role pure katakana plays in the kana license -- orthography naming the convention -- so the family-first license (Policy.script_orders) yields to the positional name_order. The check sits before the script_orders lookup in _effective_order because the suppression outranks the license. Codepoint-scoped per spec decision 5 (2026-07-30 design doc): only U+00B7 records interpunct_offsets. The Japanese nakaguro (U+30FB) is roster typography whose divided pieces the license already reads correctly, so #272's family-first reading of 高橋・一郎 is byte-identical. Co-Authored-By: Claude Fable 5 --- nameparser/_pipeline/_assign.py | 19 +++++++++++++++--- tests/v2/pipeline/test_assign.py | 33 ++++++++++++++++++++++++++++++++ 2 files changed, 49 insertions(+), 3 deletions(-) diff --git a/nameparser/_pipeline/_assign.py b/nameparser/_pipeline/_assign.py index 3672341..70ddbe6 100644 --- a/nameparser/_pipeline/_assign.py +++ b/nameparser/_pipeline/_assign.py @@ -82,11 +82,14 @@ def _peel_leading_titles(pieces: tuple[tuple[int, ...], ...], def _effective_order(policy: Policy, pieces: list[tuple[int, ...]], - tokens: list[WorkToken]) -> tuple[Role, Role, Role]: + tokens: list[WorkToken], + *, dot_divided: bool) -> tuple[Role, Role, Role]: """script_orders resolution (#271): when every name piece is written wholly in ONE script that has an entry, that script's order governs the positional read; anything else -- Latin, mixed - scripts, no entry -- falls back to name_order. Piece-level, after + scripts, no entry -- falls back to name_order. A 间隔号-divided + name (`dot_divided`, #298) suppresses the whole lookup first: the + dot marks a transcription, which keeps name_order. Piece-level, after title/suffix peeling: 'Dr. 毛泽东' is a wholly-Han NAME under a Latin title. Kana-licensed tokens (高橋みなみ, #272) resolve to HIRAGANA the same way a wholly-Han or wholly-Hangul token resolves @@ -104,6 +107,15 @@ def _effective_order(policy: Policy, resolves the SCRIPT for a single token. This function calls that one per token below. """ + # #298: a 间隔号-divided name is a transcription -- the B7 plays + # for Han exactly the role pure katakana plays in the kana license + # (orthography naming the convention), so the family-first license + # yields to the positional read. Checked before the script_orders + # lookup: the suppression outranks the license. Codepoint-scoped: + # only U+00B7 records (spec 2026-07-30 decision 5) -- the nakaguro + # is roster typography whose pieces the license reads correctly. + if dot_divided: + return policy.name_order if not policy.script_orders: return policy.name_order # Collect every token's script rather than comparing pairwise as @@ -232,7 +244,8 @@ def _assign_main(seg_idx: int, state: ParseState, # pieces only, so a Latin title or suffix ('Dr. 毛 泽东', '毛 泽东, # PhD') cannot make a wholly-CJK name look mixed-script. order = _effective_order(state.policy, - [pieces[i] for i in name_pieces], tokens) + [pieces[i] for i in name_pieces], tokens, + dot_divided=bool(state.interpunct_offsets)) roles = _name_positions(order, len(name_pieces)) for pos, piece_idx in enumerate(name_pieces): _set_roles(tokens, pieces[piece_idx], roles[pos]) diff --git a/tests/v2/pipeline/test_assign.py b/tests/v2/pipeline/test_assign.py index 69544c1..a7c47f9 100644 --- a/tests/v2/pipeline/test_assign.py +++ b/tests/v2/pipeline/test_assign.py @@ -210,6 +210,39 @@ def test_pure_katakana_piece_falls_back_to_name_order() -> None: assert _by_role(out, Role.FAMILY) == "ジャクソン" +def test_interpunct_divided_name_reads_positionally() -> None: + # 威廉·莎士比亚 is William Shakespeare: a 间隔号-divided name is a + # transcription and keeps source order (spec 2026-07-30) -- the + # B7 is the marker, playing the role pure katakana plays in the + # kana license + out = _assigned("威廉·莎士比亚") + assert _by_role(out, Role.GIVEN) == "威廉" + assert _by_role(out, Role.FAMILY) == "莎士比亚" + + +def test_interpunct_suppression_yields_to_explicit_name_order() -> None: + # the suppression falls back to name_order, it does not force + # given-first: an explicit FAMILY_FIRST governs the transcription + out = _assigned("威廉·莎士比亚", Policy(name_order=FAMILY_FIRST)) + assert _by_role(out, Role.FAMILY) == "威廉" + assert _by_role(out, Role.GIVEN) == "莎士比亚" + + +def test_nakaguro_divided_kanji_keeps_the_roster_reading() -> None: + # the Japanese dot is roster formatting (姓・名), NOT the + # transcription marker -- #272's family-first reading stands + # (codepoint scope, spec decision 5) + out = _assigned("高橋・一郎") + assert _by_role(out, Role.FAMILY) == "高橋" + assert _by_role(out, Role.GIVEN) == "一郎" + + +def test_undotted_han_still_reads_family_first() -> None: + out = _assigned("毛 泽东") + assert _by_role(out, Role.FAMILY) == "毛" + assert _by_role(out, Role.GIVEN) == "泽东" + + def test_script_with_no_table_entry_falls_back() -> None: # A single, well-defined script the table simply does not list: # resolution must fall through to name_order rather than pick an From 67526672e3068954ba742f0aa3b7fafbd56313a2 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Thu, 30 Jul 2026 10:20:32 -0700 Subject: [PATCH 3/9] =?UTF-8?q?=E9=97=B4=E9=9A=94=E5=8F=B7-divided=20names?= =?UTF-8?q?=20skip=20segmentation=20(#298)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The transcription doctrine's second consumer, after _effective_order: a B7-divided name's pieces are foreign-name syllable groups, not surname+given, so the segmentation stage has no surname site to find -- neither the vocabulary nor a configured segmenter applies. Reachable today: script_segment takes the FIRST script token of the name segment with no single-token gate, so 马丁·路德·金 (Martin Luther King) would split 马 + 丁 under a zh vocabulary listing 马. The gate sits directly beside the FAMILY_COMMA one -- both are the-writer-already-decided opt-outs that run before either split mechanism is consulted. Codepoint-scoped like the rest of #298: the Japanese nakaguro records no offsets and therefore gates nothing. The gate is state-global, matching the order twin: a marker anywhere in the name suppresses segmentation of every token, so an un-dotted hangul token beside a dotted Han one stays whole too ("김민준 马丁·路德" no longer splits 김 + 민준) -- a name carrying the transcription marker reads as a transcription listing throughout. Co-Authored-By: Claude Fable 5 --- nameparser/_pipeline/_script_segment.py | 12 ++++++- tests/v2/pipeline/test_script_segment.py | 46 ++++++++++++++++++++++++ 2 files changed, 57 insertions(+), 1 deletion(-) diff --git a/nameparser/_pipeline/_script_segment.py b/nameparser/_pipeline/_script_segment.py index 0365a76..37e01c2 100644 --- a/nameparser/_pipeline/_script_segment.py +++ b/nameparser/_pipeline/_script_segment.py @@ -1,6 +1,6 @@ """Stage: script_segment (#271, #272). -Consumes: tokens, segments, structure, segmenter. +Consumes: tokens, segments, structure, interpunct_offsets, segmenter. Produces: tokens (the first activated-script token of the name segment split into n+1 sub-slices), segments (index runs remapped past the insertions), ambiguities (indices likewise remapped, plus a @@ -207,6 +207,16 @@ def script_segment(state: ParseState) -> ParseState: return state if state.structure is Structure.FAMILY_COMMA: return state # the comma already drew the boundary + if state.interpunct_offsets: + # #298: a 间隔号-divided name is a transcription -- its pieces + # are syllable groups, not surname+given, so neither the + # vocabulary nor the segmenter applies (codepoint-scoped: the + # nakaguro records nothing and gates nothing, spec decision 5). + # State-global like the FAMILY_COMMA gate above: a marker + # anywhere in the name reads the WHOLE name as a transcription + # listing, so even an un-dotted hangul token beside a dotted + # one stays whole. + return state # segments[0] is the NAME part under both remaining structures # (everything, under NO_COMMA); later segments are suffixes. Its # members are main-stream token indices by construction, so diff --git a/tests/v2/pipeline/test_script_segment.py b/tests/v2/pipeline/test_script_segment.py index bb14d77..80f88fc 100644 --- a/tests/v2/pipeline/test_script_segment.py +++ b/tests/v2/pipeline/test_script_segment.py @@ -399,6 +399,52 @@ def test_a_neighbour_in_an_UNACTIVATED_script_still_blocks_the_consult() -> None assert out.ambiguities == (), name +# -- the 间隔号 gate: a divided name is a transcription (#298) ---------- + + +def test_interpunct_divided_name_never_segments() -> None: + # 马丁·路德·金 (Martin Luther King): 马 is a listed surname, but a + # 间隔号-divided name is a transcription -- its pieces are syllable + # groups, not surname+given, so the dot gates the stage off + # entirely, vocabulary AND segmenter (spec 2026-07-30). Built + # through the real tokenize so interpunct_offsets is the field's + # own producer's, not hand-set. + state = segment(tokenize(ParseState( + original="马丁·路德·金", + lexicon=Lexicon(surnames=frozenset({"马"})), policy=_HAN))) + assert state.interpunct_offsets # tokenize recorded the dots + out = script_segment(state) + assert out.tokens == state.tokens # nothing split: no 马 + 丁 + assert out.ambiguities == () + + +def test_interpunct_divided_name_never_consults_the_segmenter() -> None: + # the gate's other consumer, pinned separately: today the + # second-script-token precondition would decline this consult + # anyway, but the transcription doctrine must not depend on it -- + # the dot gates BEFORE either mechanism is reached + asked, seg = _capture() + state = segment(tokenize(ParseState( + original="马丁·路德·金", lexicon=Lexicon.empty(), policy=_HAN, + segmenter=seg))) + out = script_segment(state) + assert out.tokens == state.tokens + assert asked == [] + + +def test_interpunct_gates_the_whole_name_not_just_dotted_tokens() -> None: + # state-global on purpose: the marker reads the WHOLE name as a + # transcription listing, so the un-dotted hangul token stays whole + # too -- the baseline shows it segments without the B7 elsewhere + assert _texts(_run("김민준 马丁 路德", policy=_HANGUL)) == [ + "김", "민준", "马丁", "路德"] + state = segment(tokenize(ParseState( + original="김민준 马丁·路德", lexicon=_LEX, policy=_HANGUL))) + assert state.interpunct_offsets + out = script_segment(state) + assert out.tokens == state.tokens + + def test_the_segmenter_is_handed_the_gated_token_only() -> None: # It is asked where ONE token divides, so it gets that token's text # and nothing else -- not the original, not the name part joined From 96832ce8a69cb5ae1dd611f1b22ea600d817684a Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Thu, 30 Jul 2026 10:30:05 -0700 Subject: [PATCH 4/9] Pin the interpunct behavior: case rows, corpus, differential guards (#298) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Five case rows land in tests/v2/cases.py, in the table's canonical given/family/middle spelling: - zh_interpunct_transcription_source_order: 威廉·莎士比亚 divides on the B7 and keeps source order (fix(#298)). - zh_interpunct_nakaguro_typed_stays_roster: the SAME transcription typed with the Japanese nakaguro reads as a 姓・名 roster pair, family-first (fix(#272)). A limitation row, chosen not accidental: the design is codepoint-scoped (spec 2026-07-30 decision 5), so cross-convention input reads by the convention of the codepoint it was typed with -- only the Chinese B7 marks a transcription. - ja_interpunct_b7_katakana: sloppy-IME B7 between katakana divides like the nakaguro; order was already positional (fix(#298)). - latin_punt_volat_is_name_interior: Gal·la Marcet, the flank guard's reason -- Latin-flanked B7 is interior, never a divider (parity). - zh_interpunct_suppresses_segmentation: 马丁·路德·金 under zh -- the dot gates segmentation off, 马丁 stays whole (fix(#298)). The differential CJK rule's character class gains U+00B7, the second sanctioned extra beside U+FF65: both dots change parses without being classified by any Script (B7 doubles as Catalan's punt volat). test_differential_cjk_rule_matches_the_script_ranges pins the pair as sanctioned_extras; the compound rule's lookahead copies classified spans only and is untouched. corpus_cjk.jsonl regenerates 31 -> 35: the four CJK-bearing rows enter (Gal·la Marcet correctly does not -- no classified codepoint). Harness hand-run: 689 names, 48 intentional diffs, zero unexplained; the CJK rule classifies 27 including all the B7 names. Co-Authored-By: Claude Fable 5 --- tests/v2/cases.py | 37 ++++++++++++++++++++++++ tests/v2/test_regex_sync.py | 32 +++++++++++--------- tools/differential/corpus_cjk.jsonl | 4 +++ tools/differential/expected_changes.toml | 29 +++++++++++-------- 4 files changed, 77 insertions(+), 25 deletions(-) diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 62f7ced..40a2155 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -651,4 +651,41 @@ def __post_init__(self) -> None: notes="delimited content tokenizes under the same separator " "rules: the dot renders back as a space in the nickname " "join -- a decision, not an accident"), + Case("zh_interpunct_transcription_source_order", "威廉·莎士比亚", + {"given": "威廉", "family": "莎士比亚"}, + classification="fix(#298)", + notes="间隔号-divided Han is a transcribed foreign name and " + "keeps source order -- the B7 is the transcription " + "marker, playing the role pure katakana plays in the " + "kana license; it divides only between classified " + "characters"), + Case("zh_interpunct_nakaguro_typed_stays_roster", "威廉・莎士比亚", + {"given": "莎士比亚", "family": "威廉"}, + classification="fix(#272)", + notes="the SAME transcription typed with the Japanese " + "nakaguro reads as the dot's own typography says -- a " + "姓・名 roster pair, family-first (#272) -- because the " + "nakaguro records nothing (spec 2026-07-30 decision 5: " + "codepoint-scoped; only the Chinese B7 marks a " + "transcription). A limitation row: chosen, not " + "accidental -- cross-convention input reads by the " + "convention of the codepoint it was typed with"), + Case("ja_interpunct_b7_katakana", "マイケル·ジャクソン", + {"given": "マイケル", "family": "ジャクソン"}, + classification="fix(#298)", + notes="sloppy-IME B7 between katakana divides like the " + "nakaguro; katakana was never licensed, so order was " + "already positional -- the split is the fix"), + Case("latin_punt_volat_is_name_interior", "Gal·la Marcet", + {"given": "Gal·la", "family": "Marcet"}, + notes="the Catalan punt volat: U+00B7 with Latin neighbors " + "is interior to the name, never a divider -- the flank " + "guard's whole reason"), + Case("zh_interpunct_suppresses_segmentation", "马丁·路德·金", + {"given": "马丁", "middle": "路德", "family": "金"}, + locale="zh", + classification="fix(#298)", + notes="马 is a listed zh surname, but a 间隔号-divided name " + "is a transcription: the dot gates segmentation off, " + "so 马丁 stays whole and the positional read stands"), ) diff --git a/tests/v2/test_regex_sync.py b/tests/v2/test_regex_sync.py index 7c2ddd2..943dc29 100644 --- a/tests/v2/test_regex_sync.py +++ b/tests/v2/test_regex_sync.py @@ -192,16 +192,20 @@ def test_differential_cjk_rule_matches_the_script_ranges() -> None: it, see the comment there -- so the comparison runs over the BMP spans only. - The rule is also WIDER than the table by exactly one span, which + The rule is also WIDER than the table by exactly two spans, which the equality has to know about or it would just fail forever. The halfwidth middle dot U+FF65 changes parses without being classified as anything: tokenize separates on it, so a halfwidth transcription splits where 1.4 kept one token, while halfwidth - kana stays out of _SCRIPT_RANGES on purpose. Naming that span here - rather than relaxing the comparison to a subset check is what - keeps the pin honest in both directions: a THIRD source of - divergence still fails, and the one sanctioned difference has to - be written down to exist. + kana stays out of _SCRIPT_RANGES on purpose. U+00B7 -- the + context-sensitive 间隔号 (#298) -- is the same situation: it + divides between classified characters and pins source order, + changing parses, yet doubles as Catalan's punt volat and so can + never itself be classified. Naming those spans here rather than + relaxing the comparison to a subset check is what keeps the pin + honest in both directions: an unsanctioned source of divergence + still fails, and each sanctioned difference has to be written + down to exist. """ toml_path = (Path(__file__).parents[2] / "tools" / "differential" / "expected_changes.toml") @@ -222,16 +226,18 @@ def test_differential_cjk_rule_matches_the_script_ranges() -> None: (int(lo, 16), int(hi, 16)) for lo, hi in re.findall(r"\\u([0-9A-Fa-f]{4})-\\u([0-9A-Fa-f]{4})", matched[0]["name_regex"])} - # the one span the rule carries that no Script claims (see above) - halfwidth_dot = (0xFF65, 0xFF65) - assert not any(lo <= halfwidth_dot[0] <= hi - for spans in _policy._SCRIPT_RANGES.values() - for lo, hi in spans), ( - "U+FF65 is classified now; drop it from the sanctioned extras") + # the two spans the rule carries that no Script claims (see above) + sanctioned_extras = {(0xFF65, 0xFF65), (0xB7, 0xB7)} + for extra, _ in sorted(sanctioned_extras): + assert not any(lo <= extra <= hi + for spans in _policy._SCRIPT_RANGES.values() + for lo, hi in spans), ( + f"U+{extra:04X} is classified now; drop it from the " + "sanctioned extras") expected = {span for spans in _policy._SCRIPT_RANGES.values() for span in spans - if span[1] <= 0xFFFF} | {halfwidth_dot} + if span[1] <= 0xFFFF} | sanctioned_extras assert declared == expected, ( f"{toml_path.name}'s CJK name_regex declares {sorted(declared)}; " f"_SCRIPT_RANGES' BMP spans are {sorted(expected)}") diff --git a/tools/differential/corpus_cjk.jsonl b/tools/differential/corpus_cjk.jsonl index a6ce564..6900cac 100644 --- a/tools/differential/corpus_cjk.jsonl +++ b/tools/differential/corpus_cjk.jsonl @@ -6,10 +6,13 @@ "みなみ" "マイケル" "マイケル ジャクソン" +"マイケル·ジャクソン" "マイケル・ジャクソン" "佐々木 太郎" "司马相如" "夏侯惇" +"威廉·莎士比亚" +"威廉・莎士比亚" "山田 エミ" "山田 太郎 (マイケル・ジャクソン)" "山田「タロ」太郎" @@ -20,6 +23,7 @@ "田中『ハナ』花子" "諸葛亮" "阿明" +"马丁·路德·金" "高橋 みなみ" "高橋みなみ" "高橋・一郎" diff --git a/tools/differential/expected_changes.toml b/tools/differential/expected_changes.toml index cf03b8b..e0fa4ca 100644 --- a/tools/differential/expected_changes.toml +++ b/tools/differential/expected_changes.toml @@ -32,17 +32,22 @@ issue = "fix(#271/#272) native-script CJK: family-first order, hangul segmentati # supplementary-plane kana is hentaigana and other archaic forms no # modern name uses, so _SCRIPT_RANGES does not list it either. # -# U+FF65 is the one span here that _SCRIPT_RANGES does NOT have, and -# it is deliberate: the halfwidth middle dot separates tokens like its -# fullwidth twin, so 'ﾏｲｹﾙ･ｼﾞｬｸｿﾝ' splits where 1.4 left one token, -# but halfwidth kana is excluded from CLASSIFICATION on purpose (a -# separate normalization problem). The rest of the halfwidth block -# (U+FF66-U+FF9F) is left out for the usual tightness reason, and it -# was measured rather than assumed: a dotless halfwidth name such as -# 'ﾏｲｹﾙ ｼﾞｬｸｿﾝ' is byte-identical on both sides, so covering the -# block would pre-excuse a future regression on a shape that is -# parity today. The dot is the whole mechanism; the dot is the whole -# span. +# U+FF65 and U+00B7 are the two spans here that _SCRIPT_RANGES does +# NOT have, and both are deliberate. The halfwidth middle dot +# separates tokens like its fullwidth twin, so 'ﾏｲｹﾙ･ｼﾞｬｸｿﾝ' splits +# where 1.4 left one token, but halfwidth kana is excluded from +# CLASSIFICATION on purpose (a separate normalization problem). The +# rest of the halfwidth block (U+FF66-U+FF9F) is left out for the +# usual tightness reason, and it was measured rather than assumed: a +# dotless halfwidth name such as 'ﾏｲｹﾙ ｼﾞｬｸｿﾝ' is byte-identical on +# both sides, so covering the block would pre-excuse a future +# regression on a shape that is parity today. U+00B7, the +# context-sensitive 间隔号 (#298), is FF65's situation again: it +# divides only between classified-script characters and pins source +# order, so '威廉·莎士比亚' splits where 1.4 kept one token -- but a +# codepoint that doubles as Catalan's punt volat can never itself be +# classified as a CJK script. The dots are the whole mechanism; the +# dots are the whole spans. # # corpus_cjk.jsonl (#295) exists so this rule fires in every real # run: build_cjk_corpus.py harvests every CJK-bearing text in @@ -52,7 +57,7 @@ issue = "fix(#271/#272) native-script CJK: family-first order, hangul segmentati # two corpora have by construction: v1's banks had no reason to test # CJK, and build_issues_corpus.py requires an internal space, which # unspaced names never have. -name_regex = "[\\u3005-\\u3006\\u3040-\\u309F\\u30A0-\\u30FF\\u3400-\\u4DBF\\u4E00-\\u9FFF\\uF900-\\uFAFF\\uAC00-\\uD7A3\\uFF65-\\uFF65]" +name_regex = "[\\u3005-\\u3006\\u3040-\\u309F\\u30A0-\\u30FF\\u3400-\\u4DBF\\u4E00-\\u9FFF\\uF900-\\uFAFF\\uAC00-\\uD7A3\\uFF65-\\uFF65\\u00B7-\\u00B7]" fields = ["first", "middle", "last"] [[change]] From 7df9c7052c8a254e228209ec8d924ea083a675f0 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Thu, 30 Jul 2026 10:34:57 -0700 Subject: [PATCH 5/9] Document the interpunct transcription reading (#298) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit docs/release_log.rst gains the 2.1.0 Behavior Changes bullet for U+00B7: the 间隔号 is a token separator between characters of a classified script, the name it divides keeps its source order and is never segmented, the Catalan punt volat is untouched, the nakaguro is deliberately not a transcription marker, and rendering stays space-joined with a custom string_format reinstating the dot. Placed directly after the katakana-dot bullet, whose closing sentence hands off to #298. docs/usage.rst gains a paragraph in the East Asian section's Chinese flow — the 间隔号 convention as background, then the separator behavior and the never-segmented reading, with the nakaguro distinguished — and the Boundaries limits paragraph states the two limitations: the spaced form (威廉 莎士比亚) and the nakaguro-typed transcription (威廉・莎士比亚) both keep family-first, because only the Chinese dot marks a transcription. No new doctest blocks; the case rows carry the examples. docs/migrate.rst completes the "one exception" sentence: two dots are now decided in tokenization rather than by policy — the katakana dot unconditionally, the interpunct only between classified-script characters. The nakaguro doctest and its explanation are unchanged. docs/concepts.rst extends the first/last decider list: the interpunct walks the script_orders entry back, marking a transcription that keeps its source order. docs/customize.rst needs no edit: its tokenization-not-policy sentence is scoped to the nakaguro and stays true, and under script_orders={} everything is positional already, so the dot adds nothing switchable. Co-Authored-By: Claude Fable 5 --- docs/concepts.rst | 4 +++- docs/migrate.rst | 11 +++++++---- docs/release_log.rst | 1 + docs/usage.rst | 26 ++++++++++++++++++++++++-- 4 files changed, 35 insertions(+), 7 deletions(-) diff --git a/docs/concepts.rst b/docs/concepts.rst index 25b612b..0da1d14 100644 --- a/docs/concepts.rst +++ b/docs/concepts.rst @@ -58,7 +58,9 @@ layer, which assigns purely by where a word sits: the first unclaimed word is the given name, the last is the family name, and anything between them is the middle name. ``name_order``, an explicit comma, and — for a name written wholly in one East Asian script — -``script_orders`` change what "first" and "last" mean here; nothing +``script_orders`` change what "first" and "last" mean here; the +Chinese interpunct ``·`` dividing such a name walks the last of those +back, marking a transcription that keeps its source order; nothing else does. This is the whole parser in two sentences, and it explains its diff --git a/docs/migrate.rst b/docs/migrate.rst index 3ede387..23dc2c4 100644 --- a/docs/migrate.rst +++ b/docs/migrate.rst @@ -429,7 +429,10 @@ now. surface is frozen for 2.x — so the way out is the 2.0 API: ``Parser(policy=Policy(script_orders={}, segment_scripts=()))`` restores 1.4's reading of every shape above that turns on order or -splitting. The middle dot is the one exception: it is decided in -tokenization rather than by policy, so a name written with one still -divides at the dot, and still renders with a space, whatever those two -fields are set to. +splitting. The middle dots are the exception: both the katakana dot +and the Chinese interpunct ``·`` (U+00B7, dividing a transcription +like ``威廉·莎士比亚``, and a separator only between characters of a +classified script where the katakana dot is unconditional) are decided +in tokenization rather than by policy, so a name written with either +still divides at the dot, and still renders with a space, whatever +those two fields are set to. diff --git a/docs/release_log.rst b/docs/release_log.rst index d6306fc..afd6221 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -24,6 +24,7 @@ Release Log - Fix Japanese names carrying kana parsing given-first: the family-first rule above extends to any name whose characters stay within kanji and kana while carrying at least one kana character, so ``"高橋 みなみ"`` gives family ``高橋`` and ``"山田 エミ"`` family ``山田`` where 1.x read both the other way round, and a lone such token (``"高橋みなみ"``, ``"みなみ"``) lands in ``family``/``last`` where 1.x put it in ``given``/``first``. The reasoning is the one hangul already uses: hiragana never transcribes a foreign name, and a transcription is kana ALONE, so kanji-plus-kana is a Japanese person's name written in Japanese order. A name written **wholly in katakana** is deliberately excluded and stays positional — it is predominantly a transcribed foreign name (``"マイケル ジャクソン"``) already in given-first order. **Default-on: changes parse output for kana-bearing Japanese names**, through ``HumanName`` as well as the 2.0 API, and the spaced forms change what the name renders as (``str(HumanName("高橋 みなみ"))`` is now ``"みなみ 高橋"``). ``Policy(script_orders={})`` clears this entry along with the Han and Hangul ones (#272) - Fix names containing 〆 (U+3006, the shime mark that opens Japanese surnames like 〆木 and 〆谷) parsing given-first: the script classifier now counts 〆 as Han, extending the 々 entry the table already carries, and going a step further than it — 々 is Script=Han and merely outside the ideograph blocks, while 〆 is Script=Common — so these names take the East Asian family-first reading like any other wholly-Han name. ``Policy(script_orders={})`` restores the positional reading for these names exactly as for other wholly-Han names (#303) - Fix the katakana middle dot ``・`` (U+30FB, and its halfwidth twin U+FF65) being read as part of a name rather than as the divider it is: it now separates tokens exactly as a space does. A transcribed foreign name therefore divides into its parts and, being wholly katakana, keeps its source order — ``"マイケル・ジャクソン"`` gives given ``マイケル``, family ``ジャクソン``, where 1.x left the whole string in ``first`` — while a kanji pair written the same way takes the family-first rule (``"高橋・一郎"`` → family ``高橋``). Native Japanese names never contain this character and no other script's names use it, so the separation is unconditional, which also means it is **not** covered by the two policy opt-outs. Rendering follows, the way the Korean split's does: the dot comes back as a space, so ``str(HumanName("マイケル・ジャクソン"))`` is now ``"マイケル ジャクソン"``, and that reaches delimited content too — the nickname in ``"山田 太郎 (マイケル・ジャクソン)"`` renders ``"マイケル ジャクソン"``. The Chinese interpunct ``·`` is not an unconditional separator like these two: U+00B7 is also the Catalan punt volat and appears inside legitimate names (``Gal·la``), so it divides only between classified-script characters — the transcription treatment it marks is #298's (#272) + - Fix 间隔号-divided transcriptions parsing as one unsplit token: U+00B7 — the interpunct Chinese text divides a transcribed foreign name with, ``威廉·莎士比亚`` for William Shakespeare — is now a token separator between characters of a classified script, and a name it divides keeps its source order and is never segmented: the dot is the transcription marker, playing the role pure katakana plays in the kana license. It divides ONLY between classified-script characters, so the Catalan punt volat interior to names like ``Gal·la`` is untouched. The Japanese nakaguro is deliberately not a transcription marker — ``高橋・一郎`` is roster formatting, 姓・名, and keeps its family-first reading — so a transcription typed with the wrong dot (``威廉・莎士比亚``) reads by the convention of the codepoint it was typed with; like the spaced form, only the Chinese dot rescues it. Rendering stays space-joined (``str(HumanName("威廉·莎士比亚"))`` is ``"威廉 莎士比亚"``); a custom ``string_format`` such as ``"{first}·{last}"`` reinstates the dot (#298) - Fix NFD-decomposed input missing the East Asian defaults entirely: script classification now normalizes to NFC before deciding, so a Korean or Japanese name typed on macOS — where decomposed text is routine — gets the same order rule as its composed twin, which it silently did not before. Segmentation MATCHING deliberately stays raw, so an unspaced NFD hangul name is ordered correctly but not split, rather than being split in the wrong place. One gotcha worth stating plainly: parse output preserves the encoding it was given, so for NFD input ``name.family == "김"`` is ``False`` even though it is the same name — compare NFC-normalized text when comparing across encodings (#272) * 2.0.0 - July 27, 2026 diff --git a/docs/usage.rst b/docs/usage.rst index 61f249a..634f412 100644 --- a/docs/usage.rst +++ b/docs/usage.rst @@ -188,6 +188,21 @@ language is up to you. When you know the data is Chinese, apply the >>> parser_for(locales.ZH).parse("毛泽东").family '毛' +Chinese also has a transcription convention, and it is written in the +punctuation: a foreign name transcribed into Han characters keeps its +source order and divides its parts with the 间隔号, the interpunct +``·`` (U+00B7) — 威廉·莎士比亚 is William Shakespeare, given name +first. The dot itself is the marker, so nameparser reads U+00B7 as a +token separator when it sits between characters of a classified +script, and a name it divides keeps the order it was written in and is +never segmented — the role pure katakana plays for Japanese +transcriptions, played here by the divider instead of the script. +Because only a classified script on both sides makes it a divider, the +same codepoint interior to a Latin-script name — the Catalan punt +volat in ``Gal·la`` — is untouched. The Japanese middle dot ・ +(covered next) is a different mark carrying a different convention, +and keeps its own reading. + Japanese ~~~~~~~~~ @@ -266,8 +281,15 @@ positional rules — order genuinely varies in romanized data, so nothing script-based applies. A name written wholly in katakana stays positional for the reason given above, pack or no pack: it is predominantly a transcription, and a transcription is already in the -order it should be read in. And a comma disables the script behaviors -entirely, on the reasoning ``name_order`` already follows: whoever +order it should be read in. A Han transcription written with a space +instead of the 间隔号 (威廉 莎士比亚) carries nothing to distinguish it +from a native two-token name and keeps the family-first reading — the +dot is the marker, and without it there is no signal. The same holds +for a transcription typed with the Japanese middle dot (威廉・莎士比亚): +each dot carries its own script's convention, the nakaguro's is +Japanese roster formatting rather than transcription, so only the +Chinese dot rescues the source order. And a comma disables the script +behaviors entirely, on the reasoning ``name_order`` already follows: whoever wrote the comma has already said where the family name ends. A division the parser had to choose is reported rather than hidden. From 9fdb3f2711ac77a2b0410840cff7f91f7c41d130 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Thu, 30 Jul 2026 10:43:06 -0700 Subject: [PATCH 6/9] Polish: the CJK rule slug owns #298; generator docstring tense The differential rule now explains #298's B7 diffs, so its issue slug says so -- compare.py output labels a B7 diff with the issue that caused it. The sync test's substring selection (#271/#272) still matches this one rule uniquely. And the corpus generator's docstring spoke of #298 as future work it has since harvested. Co-Authored-By: Claude Fable 5 --- tools/differential/build_cjk_corpus.py | 5 +++-- tools/differential/expected_changes.toml | 2 +- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/tools/differential/build_cjk_corpus.py b/tools/differential/build_cjk_corpus.py index 2858546..4e0c8f6 100644 --- a/tools/differential/build_cjk_corpus.py +++ b/tools/differential/build_cjk_corpus.py @@ -15,8 +15,9 @@ Selection is by the shipped table (nameparser._policy._SCRIPT_RANGES, through the same _script_matcher the parser uses), so a script added to the table widens the harvest on the next regeneration, and a case -row added for future CJK work (#298's 间隔号 forms, say) enters the -corpus by being written down in the table everyone already reviews. +row added for new CJK work enters the corpus by being written down in +the table everyone already reviews (#298's 间隔号 forms arrived +exactly this way). Regenerate after editing CJK case rows: diff --git a/tools/differential/expected_changes.toml b/tools/differential/expected_changes.toml index e0fa4ca..7d25c8d 100644 --- a/tools/differential/expected_changes.toml +++ b/tools/differential/expected_changes.toml @@ -7,7 +7,7 @@ # ahead of `fields`-only ones before matching. [[change]] -issue = "fix(#271/#272) native-script CJK: family-first order, hangul segmentation, the kana license and the nakaguro" +issue = "fix(#271/#272/#298) native-script CJK: family-first order, hangul segmentation, the kana license and the dots" # '毛 泽东', '김민준': script_orders flips first/last for a name written # wholly in Han or Hangul, and the Korean surnames that now ship as # default vocabulary additionally split an unspaced hangul token into From 96b4a9a56c7b7f4e0577120c6e5d8175d4e191ed Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Thu, 30 Jul 2026 11:13:22 -0700 Subject: [PATCH 7/9] Stripped invisibles are transparent to the interpunct flank guard MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The B7 flank guard read text[i-1]/text[i+1] raw, so a policy-stripped invisible adjacent to the dot defeated it: parse("威廉‏·莎士比亚") left the literal dot in the family field, lost the order suppression, and the polluted token defeated the script license too. That input is the real-world case strip_bidi exists for -- RTL documents quoting Chinese transcriptions get U+200E/200F inserted at script transitions, in text visually identical to the clean form. The guard now walks each flank past characters the strip policy removes (bidi when strip_bidi, emoji when strip_emoji) before classifying: a character that vanishes from the token stream cannot occupy a flank position either. Whitespace, commas, and the name-dot separators remain guard-defeating -- a dot beside a space is not between characters -- and the region-local bounds are unchanged, so the pinned edge behavior stays. Also pins the guard's edge coverage the surviving mutants demanded: the trailing-dot upper bound (whose loss is an IndexError, not a wrong answer), the minimal one-char flanks, the one-sided flank (the AND, and that no spurious offset records), and hangul flanks (마틴·루터·킹 -- Korean writes interpunct transcriptions too). Co-Authored-By: Claude Fable 5 --- nameparser/_pipeline/_tokenize.py | 54 +++++++++++++++++++++++------- tests/v2/pipeline/test_tokenize.py | 52 ++++++++++++++++++++++++++++ 2 files changed, 93 insertions(+), 13 deletions(-) diff --git a/nameparser/_pipeline/_tokenize.py b/nameparser/_pipeline/_tokenize.py index 1fdb1d2..680e267 100644 --- a/nameparser/_pipeline/_tokenize.py +++ b/nameparser/_pipeline/_tokenize.py @@ -65,6 +65,29 @@ _classified_char = _script_matcher(*_SCRIPT_RANGES, whole=True) +def _is_emoji(ch: str) -> bool: + cp = ord(ch) + return any(lo <= cp <= hi for lo, hi in _EMOJI_RANGES) + + +def _flank(text: str, i: int, step: int, limit: int, + state: ParseState) -> str | None: + """The nearest flank character the strip policy would keep, or + None at the region bound. Stripped invisibles (bidi, emoji) are + TRANSPARENT here: they vanish from the token stream, so they must + not occupy a flank position either -- an RTL document quoting a + transcription puts U+200F beside the dot in visually identical + text. Whitespace and the other separators stay guard-defeating: + a dot beside a space is not between characters.""" + while i != limit: + ch = text[i] + if not (state.policy.strip_bidi and _BIDI.match(ch)) \ + and not (state.policy.strip_emoji and _is_emoji(ch)): + return ch + i += step + return None + + def _ignorable(ch: str, state: ParseState) -> bool: if ch.isspace(): return True @@ -90,8 +113,7 @@ def _ignorable(ch: str, state: ParseState) -> bool: if state.policy.strip_bidi and _BIDI.match(ch): return True if state.policy.strip_emoji: - cp = ord(ch) - return any(lo <= cp <= hi for lo, hi in _EMOJI_RANGES) + return _is_emoji(ch) return False @@ -105,17 +127,23 @@ def _tokenize_region(state: ParseState, start: int, end: int, ch = text[i] is_separator = ch in COMMA_CHARS or _ignorable(ch, state) if not is_separator and ch == _INTERPUNCT: - # flanks are region-local: a B7 at a region edge has a - # masked or absent neighbor and stays token text. - # Region-local on purpose and defensively: under the - # default delimiters a masked span's edge character is - # never classified, so the two bounds are indistinguishable - # -- but a custom delimiter ending in a classified - # character would otherwise let a B7 split and record - # across a mask seam. - is_separator = (start < i < end - 1 - and _classified_char(text[i - 1]) - and _classified_char(text[i + 1])) + # flanks are region-local (the _flank walks stop at the + # region bounds): a B7 at a region edge has a masked or + # absent neighbor and stays token text. Region-local on + # purpose and defensively: under the default delimiters a + # masked span's edge character is never classified, so + # the two bounds are indistinguishable -- but a custom + # delimiter ending in a classified character would + # otherwise let a B7 split and record across a mask seam. + # The flank scan reads raw text like segmentation matching + # does (#272's stance): NFD hangul degrades to no-split, + # never wrong-split -- classification NFC-normalizes but + # the guard does not. + left = _flank(text, i - 1, -1, start - 1, state) + right = _flank(text, i + 1, +1, end, state) + is_separator = (left is not None and right is not None + and _classified_char(left) + and _classified_char(right)) if is_separator: if tok_start is not None: tokens.append(WorkToken(text[tok_start:i], diff --git a/tests/v2/pipeline/test_tokenize.py b/tests/v2/pipeline/test_tokenize.py index ed72655..ad2dd7f 100644 --- a/tests/v2/pipeline/test_tokenize.py +++ b/tests/v2/pipeline/test_tokenize.py @@ -139,6 +139,58 @@ def test_interpunct_at_edges_stays() -> None: assert [t.text for t in state.tokens] == ["·威廉"] +def test_stripped_bidi_mark_is_transparent_to_the_flank_guard() -> None: + # an RTL document quoting a transcription: U+200F beside the dot, + # visually identical to the clean form -- the stripped mark must + # not occupy a flank position + state = _tokenized("威廉‏·莎士比亚") + assert [t.text for t in state.tokens] == ["威廉", "莎士比亚"] + assert state.interpunct_offsets != () + + +def test_stripped_emoji_is_transparent_to_the_flank_guard() -> None: + state = _tokenized("威廉\U0001f600·莎士比亚") + assert [t.text for t in state.tokens] == ["威廉", "莎士比亚"] + + +def test_unstripped_bidi_mark_defeats_the_guard() -> None: + # the contrast: with strip_bidi off the mark is real token text, + # an unclassified flank -- the dot stays interior + state = _tokenized("威廉‏·莎士比亚", + policy=Policy(strip_bidi=False)) + assert state.interpunct_offsets == () + + +def test_interpunct_trailing_edge_stays() -> None: + # the upper bound: a trailing dot has no right flank -- and + # dropping the bound is an IndexError, not a wrong answer + state = _tokenized("威廉·") + assert [t.text for t in state.tokens] == ["威廉·"] + + +def test_interpunct_minimal_flanks_split() -> None: + state = _tokenized("威·廉") + assert [t.text for t in state.tokens] == ["威", "廉"] + assert state.interpunct_offsets == (1,) + + +def test_interpunct_one_sided_flank_stays() -> None: + # the AND in the guard: one classified neighbor is not enough, + # and no offset records (a spurious record would fake the + # transcription marker downstream) + state = _tokenized("Anna·威") + assert [t.text for t in state.tokens] == ["Anna·威"] + assert state.interpunct_offsets == () + + +def test_hangul_flanks_split_and_record() -> None: + # Korean writes transcribed foreign names with the interpunct too + # (마틴·루터·킹) -- hangul flanks are classified flanks + state = _tokenized("김·민준") + assert [t.text for t in state.tokens] == ["김", "민준"] + assert state.interpunct_offsets == (1,) + + def test_interpunct_in_delimited_regions_splits_but_does_not_record() -> None: # a B7 inside a nickname must not mark the OUTER name as a # transcription -- same flag that keeps nickname commas out of From c4ab1ca1ea3293b89999c7abbffb86ed0bc3794a Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Thu, 30 Jul 2026 11:27:57 -0700 Subject: [PATCH 8/9] Retighten the differential rule; pin the review-found gaps (#298) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The B7 sanctioned-extra span was a pure loss, removed: the flank guard guarantees every name the dot can change contains a classified character, so B7 names already matched the CJK rule through their flanks -- the span's only actual effect was letting the rule claim first/middle/last diffs on punt-volat Latin names (Gal·la Marcet), pre-excusing a regression on exactly the input class the guard protects. The sync test goes back to one sanctioned extra, and its overlap loop now checks whole spans instead of discarding each span's high end. Three case rows close review-found gaps: the Korean transcription 마이클·잭슨 (마 is a listed census surname, and the spaced form really does mis-split 마|이클 today -- the dot is what rescues it), the suffix-comma composition 威廉·莎士比亚, PhD, and the half-flanked 王·Smith parity row. The suffix-comma row's corpus entry surfaced a genuine compound diff -- comma routing (title to suffix) union the CJK flip (first/last) -- that neither single-change rule may claim, each excluding the other's fields on purpose; the new fix(cjk-comma-compound) rule takes exactly that union, gated by the same two-lookahead shape as the delimiter compound, and the compound sync pin is parametrized over both slugs so neither span copy goes unpinned. Docs: usage.rst and migrate.rst state the per-flank judgment (a hangul character beside a Han one qualifies), the state-global breadth (a dot anywhere reads the whole name as a transcription listing), and the half-flanked outcome (the undivided dot remains part of the word); AGENTS.md's doctrine paragraph counts five defaults and names the 间隔号 marker among them. Co-Authored-By: Claude Fable 5 --- AGENTS.md | 2 +- docs/migrate.rst | 5 +- docs/usage.rst | 17 +++--- tests/v2/cases.py | 19 +++++++ tests/v2/test_regex_sync.py | 66 +++++++++++++----------- tools/differential/corpus_cjk.jsonl | 3 ++ tools/differential/expected_changes.toml | 54 +++++++++++++------ 7 files changed, 110 insertions(+), 56 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 4b3c35e..537bd09 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -75,7 +75,7 @@ logging.getLogger('HumanName').setLevel(logging.DEBUG) The library has two layers: `nameparser/config/` (data) and `nameparser/parser.py` (logic). -**Design philosophy — positional and language-agnostic.** The parser assigns parts by *position* plus small sets of words that join to neighbors; it never detects language. A name's language can't be reliably inferred from Latin-script transliteration ("Ali" is Arabic or Italian; "Van"/"Della"/"Bin" are first names in some cultures, particles in others), so language-specific rules belong in opt-in `Constants` config, never global defaults. Many "wrong for language X" reports (#133, #150, #130, #85, #103, #146, #83) are irreducible ambiguities — e.g. `de Mesnil` (want last name) vs `Van Johnson` (want first name) are the same `[prefix][word]` shape. Before adding a rule, confirm it doesn't break the opposite case (run the full suite — Portuguese and "Van Johnson" tests are the usual canaries). **The one scoped exception (2.1, #271/#272): script-conditional behavior is permitted exactly where the SCRIPT ITSELF — not statistics about it — determines the convention.** The never-detect-language rule above is about Latin *transliteration*, where the signal genuinely is destroyed; native script is a different question, and it is answered per behavior rather than per script. Four defaults fall out of it. Wholly-Han, wholly-Hangul and kana-licensed names read family-first (`Policy.script_orders`) — no language detection needed, because zh and ja both write family-first in native script, so order cannot be misread even though the language is unknowable. Unspaced hangul splits into surname + given name (`config/surnames.py` ships the Korean census list as DEFAULT vocabulary) — nothing but Korean is written in hangul and the surnames are a closed census set, and the vocabulary is self-selecting besides: a hangul entry can only ever match hangul text. Hiragana licenses Japanese (#272) — a name whose characters stay inside Han∪kana while carrying at least one kana cannot be Chinese (the kana rules it out) and is not a transcription (foreign names are transcribed in katakana ALONE, マイケル has no kanji), so 高橋みなみ and 山田 エミ read family-first too; mechanically they resolve to the HIRAGANA entry, the license's carrier key. PURE katakana is excluded and keeps the positional default: マイケル・ジャクソン is a transcribed foreign name in its source order. The nakaguro belongs to the same doctrine but is decided a layer down: U+30FB and its halfwidth twin U+FF65 separate tokens like whitespace, unconditionally and in tokenize, so neither policy opt-out (`script_orders={}`, `segment_scripts=()`) reaches it — the codepoints are CJK-only and appear in no other script's names, which is what licenses a tokenize-level rule where U+00B7 (also the Catalan punt volat, interior to Gal·la) needs the flanked-by-classified-script guard `_tokenize_region` gives it (#298). Han segmentation stays OPT-IN (`locales.ZH` for Chinese, `locales.JA` for Japanese) — a zh surname list corrupts Japanese kanji names, since 高 is a common Chinese surname and 高橋一郎 would split 高+橋一郎 where the correct reading is 高橋+一郎; no surname list divides a kanji name at all, so `locales.JA` activates the stage and a pluggable `Parser(segmenter=...)` does the dividing. Latin-script input is never touched by any of this: "Kim Min-jun" is genuinely order-ambiguous and stays governed by `name_order` and opt-in packs. Before adding a script-conditional rule, work out which of the three it is — certain, certain for this one behavior only, or a statistical guess wearing a script's clothes. +**Design philosophy — positional and language-agnostic.** The parser assigns parts by *position* plus small sets of words that join to neighbors; it never detects language. A name's language can't be reliably inferred from Latin-script transliteration ("Ali" is Arabic or Italian; "Van"/"Della"/"Bin" are first names in some cultures, particles in others), so language-specific rules belong in opt-in `Constants` config, never global defaults. Many "wrong for language X" reports (#133, #150, #130, #85, #103, #146, #83) are irreducible ambiguities — e.g. `de Mesnil` (want last name) vs `Van Johnson` (want first name) are the same `[prefix][word]` shape. Before adding a rule, confirm it doesn't break the opposite case (run the full suite — Portuguese and "Van Johnson" tests are the usual canaries). **The one scoped exception (2.1, #271/#272): script-conditional behavior is permitted exactly where the SCRIPT ITSELF — not statistics about it — determines the convention.** The never-detect-language rule above is about Latin *transliteration*, where the signal genuinely is destroyed; native script is a different question, and it is answered per behavior rather than per script. Five defaults fall out of it. Wholly-Han, wholly-Hangul and kana-licensed names read family-first (`Policy.script_orders`) — no language detection needed, because zh and ja both write family-first in native script, so order cannot be misread even though the language is unknowable. Unspaced hangul splits into surname + given name (`config/surnames.py` ships the Korean census list as DEFAULT vocabulary) — nothing but Korean is written in hangul and the surnames are a closed census set, and the vocabulary is self-selecting besides: a hangul entry can only ever match hangul text. Hiragana licenses Japanese (#272) — a name whose characters stay inside Han∪kana while carrying at least one kana cannot be Chinese (the kana rules it out) and is not a transcription (foreign names are transcribed in katakana ALONE, マイケル has no kanji), so 高橋みなみ and 山田 エミ read family-first too; mechanically they resolve to the HIRAGANA entry, the license's carrier key. PURE katakana is excluded and keeps the positional default: マイケル・ジャクソン is a transcribed foreign name in its source order. And the 间隔号 U+00B7 (#298) is the transcription marker for scripts that HAVE no transcription script: a name it divides (威廉·莎士比亚 — flanked by classified characters on both sides, so Catalan's Gal·la is untouched) keeps its source order and never segments — the orthography names the convention, exactly as pure katakana does, with the divider carrying the signal instead of the script. The nakaguro belongs to the same doctrine but is decided a layer down: U+30FB and its halfwidth twin U+FF65 separate tokens like whitespace, unconditionally and in tokenize, so neither policy opt-out (`script_orders={}`, `segment_scripts=()`) reaches it — the codepoints are CJK-only and appear in no other script's names, which is what licenses a tokenize-level rule where U+00B7 (also the Catalan punt volat, interior to Gal·la) needs the flanked-by-classified-script guard `_tokenize_region` gives it (#298). Han segmentation stays OPT-IN (`locales.ZH` for Chinese, `locales.JA` for Japanese) — a zh surname list corrupts Japanese kanji names, since 高 is a common Chinese surname and 高橋一郎 would split 高+橋一郎 where the correct reading is 高橋+一郎; no surname list divides a kanji name at all, so `locales.JA` activates the stage and a pluggable `Parser(segmenter=...)` does the dividing. Latin-script input is never touched by any of this: "Kim Min-jun" is genuinely order-ambiguous and stays governed by `name_order` and opt-in packs. Before adding a script-conditional rule, work out which of the three it is — certain, certain for this one behavior only, or a statistical guess wearing a script's clothes. ### Configuration layer (`nameparser/config/`) diff --git a/docs/migrate.rst b/docs/migrate.rst index 23dc2c4..5474858 100644 --- a/docs/migrate.rst +++ b/docs/migrate.rst @@ -431,8 +431,9 @@ surface is frozen for 2.x — so the way out is the 2.0 API: restores 1.4's reading of every shape above that turns on order or splitting. The middle dots are the exception: both the katakana dot and the Chinese interpunct ``·`` (U+00B7, dividing a transcription -like ``威廉·莎士比亚``, and a separator only between characters of a -classified script where the katakana dot is unconditional) are decided +like ``威廉·莎士比亚``, and a separator only between classified-script +characters — each side judged on its own — where the katakana dot is +unconditional) are decided in tokenization rather than by policy, so a name written with either still divides at the dot, and still renders with a space, whatever those two fields are set to. diff --git a/docs/usage.rst b/docs/usage.rst index 634f412..a77a40f 100644 --- a/docs/usage.rst +++ b/docs/usage.rst @@ -193,15 +193,18 @@ punctuation: a foreign name transcribed into Han characters keeps its source order and divides its parts with the 间隔号, the interpunct ``·`` (U+00B7) — 威廉·莎士比亚 is William Shakespeare, given name first. The dot itself is the marker, so nameparser reads U+00B7 as a -token separator when it sits between characters of a classified -script, and a name it divides keeps the order it was written in and is +token separator when it sits between classified-script characters +(each side judged on its own — a hangul character beside a Han one +qualifies), and a dot anywhere in the name reads the whole name as a +transcription listing: it keeps the order it was written in and is never segmented — the role pure katakana plays for Japanese transcriptions, played here by the divider instead of the script. -Because only a classified script on both sides makes it a divider, the -same codepoint interior to a Latin-script name — the Catalan punt -volat in ``Gal·la`` — is untouched. The Japanese middle dot ・ -(covered next) is a different mark carrying a different convention, -and keeps its own reading. +Because only classified characters on both sides make it a divider, +the same codepoint interior to a Latin-script name — the Catalan punt +volat in ``Gal·la`` — is untouched, and a dot with a classified +character on just one side (``王·Smith``) stays part of the word +undivided. The Japanese middle dot ・ (covered next) is a different +mark carrying a different convention, and keeps its own reading. Japanese ~~~~~~~~~ diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 40a2155..753485d 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -688,4 +688,23 @@ def __post_init__(self) -> None: notes="马 is a listed zh surname, but a 间隔号-divided name " "is a transcription: the dot gates segmentation off, " "so 马丁 stays whole and the positional read stands"), + Case("ko_interpunct_transcription_source_order", "마이클·잭슨", + {"given": "마이클", "family": "잭슨"}, + classification="fix(#298)", + notes="Korean writes transcribed foreign names with the " + "interpunct too: the dot suppresses the hangul " + "family-first entry AND the default segmentation -- 마 " + "is a listed census surname, and the spaced form " + "마이클 잭슨 really does mis-split 마|이클 today, so " + "the dot is what rescues the ko transcription"), + Case("zh_interpunct_with_suffix_comma", "威廉·莎士比亚, PhD", + {"given": "威廉", "family": "莎士比亚", "suffix": "PhD"}, + classification="fix(#298)", + notes="the transcription reading composes with a suffix " + "comma: the marker is structure-independent"), + Case("zh_interpunct_half_flanked_stays", "王·Smith", + {"given": "王·Smith"}, + notes="one classified neighbor is not enough: the guard " + "requires both, so the undivided dot remains part of " + "the word -- declining, not deciding"), ) diff --git a/tests/v2/test_regex_sync.py b/tests/v2/test_regex_sync.py index 943dc29..11f89f5 100644 --- a/tests/v2/test_regex_sync.py +++ b/tests/v2/test_regex_sync.py @@ -192,20 +192,22 @@ def test_differential_cjk_rule_matches_the_script_ranges() -> None: it, see the comment there -- so the comparison runs over the BMP spans only. - The rule is also WIDER than the table by exactly two spans, which + The rule is also WIDER than the table by exactly one span, which the equality has to know about or it would just fail forever. The halfwidth middle dot U+FF65 changes parses without being classified as anything: tokenize separates on it, so a halfwidth transcription splits where 1.4 kept one token, while halfwidth kana stays out of _SCRIPT_RANGES on purpose. U+00B7 -- the - context-sensitive 间隔号 (#298) -- is the same situation: it - divides between classified characters and pins source order, - changing parses, yet doubles as Catalan's punt volat and so can - never itself be classified. Naming those spans here rather than - relaxing the comparison to a subset check is what keeps the pin - honest in both directions: an unsanctioned source of divergence - still fails, and each sanctioned difference has to be written - down to exist. + context-sensitive 间隔号 (#298) -- also changes parses yet is + deliberately NOT an extra: its flank guard means every name it + can change matches the class through a flanking character + already, and a B7 span's only actual effect would be letting the + rule claim diffs on punt-volat Latin names (Gal·la), pre-excusing + a regression on exactly the guarded class. Naming the sanctioned + span here rather than relaxing the comparison to a subset check + is what keeps the pin honest in both directions: an unsanctioned + source of divergence still fails, and each sanctioned difference + has to be written down to exist. """ toml_path = (Path(__file__).parents[2] / "tools" / "differential" / "expected_changes.toml") @@ -227,13 +229,13 @@ def test_differential_cjk_rule_matches_the_script_ranges() -> None: for lo, hi in re.findall(r"\\u([0-9A-Fa-f]{4})-\\u([0-9A-Fa-f]{4})", matched[0]["name_regex"])} # the two spans the rule carries that no Script claims (see above) - sanctioned_extras = {(0xFF65, 0xFF65), (0xB7, 0xB7)} - for extra, _ in sorted(sanctioned_extras): - assert not any(lo <= extra <= hi + sanctioned_extras = {(0xFF65, 0xFF65)} + for xlo, xhi in sanctioned_extras: + assert not any(lo <= xhi and xlo <= hi for spans in _policy._SCRIPT_RANGES.values() for lo, hi in spans), ( - f"U+{extra:04X} is classified now; drop it from the " - "sanctioned extras") + f"U+{xlo:04X}-U+{xhi:04X} is classified now; drop it " + "from the sanctioned extras") expected = {span for spans in _policy._SCRIPT_RANGES.values() for span in spans @@ -243,29 +245,35 @@ def test_differential_cjk_rule_matches_the_script_ranges() -> None: f"_SCRIPT_RANGES' BMP spans are {sorted(expected)}") -def test_differential_compound_rule_matches_the_script_ranges() -> None: - """The fix(cjk-delimited-nickname) rule (#295) carries the SECOND - hand copy of the script spans in the toml: its require-a-classified- - codepoint lookahead exists so the delimiter set alone cannot claim a - Latin name's first/last regression ('John 「Jack」 Kennedy' -- the - corner brackets sit outside every classified span). Pin that copy to - the table exactly as the canonical CJK rule's is, and pin the - delimiter set itself, so widening either (to ASCII quotes, say) is - an explicit decision here rather than a silent absorption change. +@pytest.mark.parametrize("slug", ["cjk-delimited-nickname", + "cjk-comma-compound"]) +def test_differential_compound_rules_match_the_script_ranges( + slug: str) -> None: + """The compound rules carry FURTHER hand copies of the script + spans in the toml: each one's require-a-classified-codepoint + lookahead exists so its trigger set alone (delimiters; a comma) + cannot claim a Latin name's regression ('John 「Jack」 Kennedy', + 'Smith, Jr.' -- neither contains a classified character). Pin each + copy to the table exactly as the canonical CJK rule's is, plus the + nickname rule's delimiter set itself, so widening either is an + explicit decision here rather than a silent absorption change. Selection note for future rule authors: the canonical-CJK-rule pin above selects by the literal '#271'/'#272' substrings and asserts - uniqueness -- this rule's slug avoids them on purpose, and any new - rule's must too. + uniqueness -- the compound slugs avoid them on purpose, and any + new rule's must too (and its slug joins this parametrize list, or + its span copy goes unpinned). """ toml_path = (Path(__file__).parents[2] / "tools" / "differential" / "expected_changes.toml") rules = tomllib.loads(toml_path.read_text())["change"] - matched = [r for r in rules if "cjk-delimited-nickname" in r["issue"]] + matched = [r for r in rules if slug in r["issue"]] assert len(matched) == 1 regex = matched[0]["name_regex"] - assert "[「」『』・･]" in regex, ( - "the compound rule's delimiter set changed; decide deliberately") + if slug == "cjk-delimited-nickname": + assert "[「」『』・･]" in regex, ( + "the compound rule's delimiter set changed; decide " + "deliberately") declared = { (int(lo, 16), int(hi, 16)) for lo, hi in re.findall(r"\\u([0-9A-Fa-f]{4})-\\u([0-9A-Fa-f]{4})", @@ -275,7 +283,7 @@ def test_differential_compound_rule_matches_the_script_ranges() -> None: for span in spans if span[1] <= 0xFFFF} | {(0xFF65, 0xFF65)} assert declared == expected, ( - f"compound rule's codepoint lookahead declares {sorted(declared)}; " + f"{slug}'s codepoint lookahead declares {sorted(declared)}; " f"_SCRIPT_RANGES' BMP spans are {sorted(expected)}") diff --git a/tools/differential/corpus_cjk.jsonl b/tools/differential/corpus_cjk.jsonl index 6900cac..8d9381e 100644 --- a/tools/differential/corpus_cjk.jsonl +++ b/tools/differential/corpus_cjk.jsonl @@ -12,6 +12,7 @@ "司马相如" "夏侯惇" "威廉·莎士比亚" +"威廉·莎士比亚, PhD" "威廉・莎士比亚" "山田 エミ" "山田 太郎 (マイケル・ジャクソン)" @@ -20,6 +21,7 @@ "毛 泽东" "毛 김" "毛泽东" +"王·Smith" "田中『ハナ』花子" "諸葛亮" "阿明" @@ -33,3 +35,4 @@ "남궁" "남궁민수" "남궁민수, 지훈" +"마이클·잭슨" diff --git a/tools/differential/expected_changes.toml b/tools/differential/expected_changes.toml index 7d25c8d..74a5c69 100644 --- a/tools/differential/expected_changes.toml +++ b/tools/differential/expected_changes.toml @@ -32,22 +32,23 @@ issue = "fix(#271/#272/#298) native-script CJK: family-first order, hangul segme # supplementary-plane kana is hentaigana and other archaic forms no # modern name uses, so _SCRIPT_RANGES does not list it either. # -# U+FF65 and U+00B7 are the two spans here that _SCRIPT_RANGES does -# NOT have, and both are deliberate. The halfwidth middle dot -# separates tokens like its fullwidth twin, so 'ﾏｲｹﾙ･ｼﾞｬｸｿﾝ' splits -# where 1.4 left one token, but halfwidth kana is excluded from -# CLASSIFICATION on purpose (a separate normalization problem). The -# rest of the halfwidth block (U+FF66-U+FF9F) is left out for the -# usual tightness reason, and it was measured rather than assumed: a -# dotless halfwidth name such as 'ﾏｲｹﾙ ｼﾞｬｸｿﾝ' is byte-identical on -# both sides, so covering the block would pre-excuse a future -# regression on a shape that is parity today. U+00B7, the -# context-sensitive 间隔号 (#298), is FF65's situation again: it -# divides only between classified-script characters and pins source -# order, so '威廉·莎士比亚' splits where 1.4 kept one token -- but a -# codepoint that doubles as Catalan's punt volat can never itself be -# classified as a CJK script. The dots are the whole mechanism; the -# dots are the whole spans. +# U+FF65 is the one span here that _SCRIPT_RANGES does NOT have, and +# it is deliberate. The halfwidth middle dot separates tokens like +# its fullwidth twin, so 'ﾏｲｹﾙ･ｼﾞｬｸｿﾝ' splits where 1.4 left one +# token, but halfwidth kana is excluded from CLASSIFICATION on +# purpose (a separate normalization problem). The rest of the +# halfwidth block (U+FF66-U+FF9F) is left out for the usual tightness +# reason, and it was measured rather than assumed: a dotless +# halfwidth name such as 'ﾏｲｹﾙ ｼﾞｬｸｿﾝ' is byte-identical on both +# sides, so covering the block would pre-excuse a future regression +# on a shape that is parity today. U+00B7, the context-sensitive +# 间隔号 (#298), deliberately gets NO span even though it too changes +# parses: its flank guard divides only between classified-script +# characters, so every name it can change already matches this class +# through a flanking character -- and a B7 span's only actual effect +# would be letting this rule claim first/middle/last diffs on +# punt-volat Latin names ('Gal·la Marcet'), pre-excusing a regression +# on exactly the input class the guard exists to protect. # # corpus_cjk.jsonl (#295) exists so this rule fires in every real # run: build_cjk_corpus.py harvests every CJK-bearing text in @@ -57,7 +58,7 @@ issue = "fix(#271/#272/#298) native-script CJK: family-first order, hangul segme # two corpora have by construction: v1's banks had no reason to test # CJK, and build_issues_corpus.py requires an internal space, which # unspaced names never have. -name_regex = "[\\u3005-\\u3006\\u3040-\\u309F\\u30A0-\\u30FF\\u3400-\\u4DBF\\u4E00-\\u9FFF\\uF900-\\uFAFF\\uAC00-\\uD7A3\\uFF65-\\uFF65\\u00B7-\\u00B7]" +name_regex = "[\\u3005-\\u3006\\u3040-\\u309F\\u30A0-\\u30FF\\u3400-\\u4DBF\\u4E00-\\u9FFF\\uF900-\\uFAFF\\uAC00-\\uD7A3\\uFF65-\\uFF65]" fields = ["first", "middle", "last"] [[change]] @@ -157,6 +158,25 @@ issue = "fix(cjk-delimited-nickname) delimiter recognition compounds with the CJ name_regex = "(?s)(?=.*[「」『』・･])(?=.*[\\u3005-\\u3006\\u3040-\\u309F\\u30A0-\\u30FF\\u3400-\\u4DBF\\u4E00-\\u9FFF\\uF900-\\uFAFF\\uAC00-\\uD7A3\\uFF65-\\uFF65])" fields = ["first", "last", "nickname"] +[[change]] +issue = "fix(cjk-comma-compound) comma routing compounds with the CJK order flip" +# '威廉·莎士比亚, PhD': one name, two intended changes at once -- the +# post-comma lone-piece routing (fix(comma-family): v1 read PhD as +# title; 2.x routes it to suffix) and the CJK source-order/family- +# first movement in first/middle/last on the pre-comma name. Neither +# single-change rule may claim the union: comma-family's fields +# exclude last, the CJK rule's exclude title/suffix -- each on +# purpose, so a lone regression in the other's fields stays loud. +# Both lookaheads are required, by the same reasoning as the +# delimiter compound above: a comma alone matches every Latin +# 'Smith, Jr.' in the corpus, and the classified-codepoint lookahead +# confines the rule to names the CJK behaviors can actually reach. +# The class is the same hand copy of _SCRIPT_RANGES the rules above +# carry, pinned by the same sync test. The slug avoids the literal +# #271/#272 substrings the canonical-rule pin selects by. +name_regex = "(?s)(?=.*,)(?=.*[\\u3005-\\u3006\\u3040-\\u309F\\u30A0-\\u30FF\\u3400-\\u4DBF\\u4E00-\\u9FFF\\uF900-\\uFAFF\\uAC00-\\uD7A3\\uFF65-\\uFF65])" +fields = ["first", "middle", "last", "title", "suffix"] + [[change]] issue = "feat(#269) non-Latin titles/conjunctions recognized" # 'г-н Иван Петров' (Cyrillic title), 'Хосе И Мария Сантос' (Cyrillic From d65dc003c6d468848588fe4ede4a00bc5fd8c62e Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Thu, 30 Jul 2026 15:39:09 -0700 Subject: [PATCH 9/9] Simplify: single-source the strip set, auto-discover span pins, trim accretion MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The strip-policy set is now defined ONCE (_stripped, shared by _ignorable and _flank), making the flank-transparency invariant structural: a new strip class added to the policy stays transparent to the interpunct guard by construction instead of silently re-opening the RTL-quote bug. _flank takes a range and the policy (three params instead of five; the bound arithmetic lives at the call sites where the region is in scope), and the right flank is now walked only after the left one classifies -- half the guard work on every Catalan punt-volat name. The dead 'not is_separator' conjunct on the interpunct test is gone (unfalsifiable: B7 is no comma, no space, and _ignorable cannot claim it). The differential span pins single-source through _SANCTIONED_EXTRAS, _declared_spans and _expected_bmp_spans, and the compound-rule pin auto-discovers every span-bearing rule that intersects the table instead of reading a hand-maintained slug roster whose own docstring admitted its failure mode ('or its span copy goes unpinned') -- a third compound rule is now pinned by existing. Review-round comment accretion trimmed to single homes: the nakaguro roster doctrine lives at _NAME_DOT_SEPARATORS (with the 高橋・一郎 example), the placement rationale at _INTERPUNCT, the suppression rationale in _effective_order's docstring. Two duplicate tests deleted: test_undotted_han_still_reads_family_first was byte-identical to test_script_order_applies_when_every_piece_is_one_script, and the assign-level nakaguro roster test restated the pre-existing ja_nakaguro_han_takes_the_han_order case row plus the tokenize-level non-recording pin. Co-Authored-By: Claude Fable 5 --- nameparser/_pipeline/_assign.py | 13 ++-- nameparser/_pipeline/_tokenize.py | 101 ++++++++++++------------- tests/v2/pipeline/test_assign.py | 15 ---- tests/v2/test_regex_sync.py | 118 ++++++++++++++++++------------ 4 files changed, 123 insertions(+), 124 deletions(-) diff --git a/nameparser/_pipeline/_assign.py b/nameparser/_pipeline/_assign.py index 70ddbe6..0407736 100644 --- a/nameparser/_pipeline/_assign.py +++ b/nameparser/_pipeline/_assign.py @@ -89,7 +89,9 @@ def _effective_order(policy: Policy, order governs the positional read; anything else -- Latin, mixed scripts, no entry -- falls back to name_order. A 间隔号-divided name (`dot_divided`, #298) suppresses the whole lookup first: the - dot marks a transcription, which keeps name_order. Piece-level, after + dot marks a transcription -- playing the role pure katakana plays + in the kana license, orthography naming the convention -- so the + license yields to name_order. Piece-level, after title/suffix peeling: 'Dr. 毛泽东' is a wholly-Han NAME under a Latin title. Kana-licensed tokens (高橋みなみ, #272) resolve to HIRAGANA the same way a wholly-Han or wholly-Hangul token resolves @@ -107,13 +109,8 @@ def _effective_order(policy: Policy, resolves the SCRIPT for a single token. This function calls that one per token below. """ - # #298: a 间隔号-divided name is a transcription -- the B7 plays - # for Han exactly the role pure katakana plays in the kana license - # (orthography naming the convention), so the family-first license - # yields to the positional read. Checked before the script_orders - # lookup: the suppression outranks the license. Codepoint-scoped: - # only U+00B7 records (spec 2026-07-30 decision 5) -- the nakaguro - # is roster typography whose pieces the license reads correctly. + # #298 transcription marker -- see the docstring; codepoint-scoped + # (only U+00B7 records, spec 2026-07-30 decision 5) if dot_divided: return policy.name_order if not policy.script_orders: diff --git a/nameparser/_pipeline/_tokenize.py b/nameparser/_pipeline/_tokenize.py index 680e267..682ae19 100644 --- a/nameparser/_pipeline/_tokenize.py +++ b/nameparser/_pipeline/_tokenize.py @@ -13,10 +13,8 @@ separators and never enter a token, so spans always index the original exactly as given. Whitespace and the name-dot are unconditional; emoji/bidi stripping alone is policy-gated -(Policy.strip_emoji/strip_bidi). The Chinese interpunct U+00B7 is the -one context-sensitive separator: it divides (and records) only -between classified-script neighbors, so its rule lives in -_tokenize_region, which has the index _ignorable lacks. +(Policy.strip_emoji/strip_bidi). The Chinese interpunct U+00B7 is +context-sensitive -- see _INTERPUNCT below. v1's squash_emoji/squash_bidi REMOVED the char and joined neighbors ('A\U0001f600B' -> 'AB'); here an ignorable char is a SEPARATOR @@ -32,7 +30,7 @@ from nameparser._pipeline._state import ( COMMA_CHARS, ParseState, WorkToken, ) -from nameparser._policy import _SCRIPT_RANGES, _script_matcher +from nameparser._policy import Policy, _SCRIPT_RANGES, _script_matcher from nameparser._types import Role, Span # Ported from v1 (nameparser/config/regexes.py, "emoji" and "bidi") -- @@ -49,9 +47,11 @@ # The katakana middle dot and its halfwidth twin divide the parts of # a foreign name transcribed into katakana (マイケル・ジャクソン) -- # native names never contain them, so they separate unconditionally, -# like whitespace (amendment 2026-07-29 section 1b). The Chinese -# interpunct U+00B7 is context-sensitive instead -- see _INTERPUNCT -# below. +# like whitespace (amendment 2026-07-29 section 1b). They record no +# offset: the nakaguro also divides kanji roster pairs (高橋・一郎, +# 姓・名 -- read family-first by the script license, #272), so it is +# not a transcription marker. Only the Chinese 间隔号 is (#298), and +# it is context-sensitive -- see _INTERPUNCT below. _NAME_DOT_SEPARATORS = frozenset({"\u30FB", "\uFF65"}) _INTERPUNCT = "\u00B7" @@ -70,21 +70,28 @@ def _is_emoji(ch: str) -> bool: return any(lo <= cp <= hi for lo, hi in _EMOJI_RANGES) -def _flank(text: str, i: int, step: int, limit: int, - state: ParseState) -> str | None: +def _stripped(ch: str, policy: Policy) -> bool: + """True when the strip policy removes `ch` from the token stream. + The ONE definition of that set, shared by _ignorable and _flank: a + character that vanishes from tokens must not occupy a flank + position either, so a new strip class added here stays transparent + to the interpunct guard by construction.""" + if policy.strip_bidi and _BIDI.match(ch): + return True + return bool(policy.strip_emoji and _is_emoji(ch)) + + +def _flank(text: str, indices: range, policy: Policy) -> str | None: """The nearest flank character the strip policy would keep, or - None at the region bound. Stripped invisibles (bidi, emoji) are - TRANSPARENT here: they vanish from the token stream, so they must - not occupy a flank position either -- an RTL document quoting a - transcription puts U+200F beside the dot in visually identical - text. Whitespace and the other separators stay guard-defeating: - a dot beside a space is not between characters.""" - while i != limit: + None if the range exhausts. Stripped invisibles are TRANSPARENT + here: an RTL document quoting a transcription puts U+200F beside + the dot in visually identical text. Whitespace and the other + separators stay guard-defeating: a dot beside a space is not + between characters.""" + for i in indices: ch = text[i] - if not (state.policy.strip_bidi and _BIDI.match(ch)) \ - and not (state.policy.strip_emoji and _is_emoji(ch)): + if not _stripped(ch, policy): return ch - i += step return None @@ -97,24 +104,14 @@ def _ignorable(ch: str, state: ParseState) -> bool: # skip the checks below for every ASCII letter return False # unconditional, like whitespace -- not policy-gated, so this sits - # ahead of the policy checks rather than in _tokenize_region beside - # COMMA_CHARS (commas RECORD an offset; these dots must not: the - # nakaguro is Japanese roster/divider typography whose pieces the - # script license already reads correctly -- #272, 高橋・一郎 - # family-first, マイケル・ジャクソン positional -- while only the - # Chinese 间隔号 carries the transcription meaning and records, - # #298, handled in _tokenize_region). The only load-bearing - # constraint is "after the isascii fast path" (both dots are - # non-ASCII); being ahead of the bidi/emoji checks below is NOT - # load-bearing -- the three sets are disjoint, so a future edit is - # free to reorder them. + # ahead of the policy check rather than in _tokenize_region beside + # COMMA_CHARS (commas RECORD an offset; these dots must not -- + # only the 间隔号 marks a transcription and records, #298, see + # _NAME_DOT_SEPARATORS above). The only load-bearing constraint is + # "after the isascii fast path" (both dots are non-ASCII). if ch in _NAME_DOT_SEPARATORS: return True - if state.policy.strip_bidi and _BIDI.match(ch): - return True - if state.policy.strip_emoji: - return _is_emoji(ch) - return False + return _stripped(ch, state.policy) def _tokenize_region(state: ParseState, start: int, end: int, @@ -126,24 +123,20 @@ def _tokenize_region(state: ParseState, start: int, end: int, for i in range(start, end): ch = text[i] is_separator = ch in COMMA_CHARS or _ignorable(ch, state) - if not is_separator and ch == _INTERPUNCT: - # flanks are region-local (the _flank walks stop at the - # region bounds): a B7 at a region edge has a masked or - # absent neighbor and stays token text. Region-local on - # purpose and defensively: under the default delimiters a - # masked span's edge character is never classified, so - # the two bounds are indistinguishable -- but a custom - # delimiter ending in a classified character would - # otherwise let a B7 split and record across a mask seam. - # The flank scan reads raw text like segmentation matching - # does (#272's stance): NFD hangul degrades to no-split, - # never wrong-split -- classification NFC-normalizes but - # the guard does not. - left = _flank(text, i - 1, -1, start - 1, state) - right = _flank(text, i + 1, +1, end, state) - is_separator = (left is not None and right is not None - and _classified_char(left) - and _classified_char(right)) + if ch == _INTERPUNCT: + # Region-local flanks: a B7 at a region edge stays token + # text, and the bound also stops a custom delimiter's + # classified edge character from acting as a flank across + # a mask seam. The scan reads raw text like segmentation + # matching does (#272's stance): NFD hangul degrades to + # no-split, never wrong-split -- classification + # NFC-normalizes but the guard does not. + left = _flank(text, range(i - 1, start - 1, -1), + state.policy) + if left is not None and _classified_char(left): + right = _flank(text, range(i + 1, end), state.policy) + is_separator = (right is not None + and _classified_char(right)) if is_separator: if tok_start is not None: tokens.append(WorkToken(text[tok_start:i], diff --git a/tests/v2/pipeline/test_assign.py b/tests/v2/pipeline/test_assign.py index a7c47f9..5c59a8b 100644 --- a/tests/v2/pipeline/test_assign.py +++ b/tests/v2/pipeline/test_assign.py @@ -228,21 +228,6 @@ def test_interpunct_suppression_yields_to_explicit_name_order() -> None: assert _by_role(out, Role.GIVEN) == "莎士比亚" -def test_nakaguro_divided_kanji_keeps_the_roster_reading() -> None: - # the Japanese dot is roster formatting (姓・名), NOT the - # transcription marker -- #272's family-first reading stands - # (codepoint scope, spec decision 5) - out = _assigned("高橋・一郎") - assert _by_role(out, Role.FAMILY) == "高橋" - assert _by_role(out, Role.GIVEN) == "一郎" - - -def test_undotted_han_still_reads_family_first() -> None: - out = _assigned("毛 泽东") - assert _by_role(out, Role.FAMILY) == "毛" - assert _by_role(out, Role.GIVEN) == "泽东" - - def test_script_with_no_table_entry_falls_back() -> None: # A single, well-defined script the table simply does not list: # resolution must fall through to name_order rather than pick an diff --git a/tests/v2/test_regex_sync.py b/tests/v2/test_regex_sync.py index 11f89f5..209c759 100644 --- a/tests/v2/test_regex_sync.py +++ b/tests/v2/test_regex_sync.py @@ -164,6 +164,33 @@ def test_comma_char_matches_the_pipeline_comma_set() -> None: assert set(_render._COMMA_CHAR.pattern.strip("[]")) == set(COMMA_CHARS) +# The one sanctioned divergence between the differential rules' +# character classes and _SCRIPT_RANGES: the halfwidth middle dot +# separates tokens without being classified (halfwidth kana stays out +# of the table on purpose). U+00B7 is deliberately NOT here -- its +# flank guard means every name it can change matches through a +# classified flanking character already. Single-sourced: both span +# pins below read this set. +_SANCTIONED_EXTRAS = frozenset({(0xFF65, 0xFF65)}) + + +def _declared_spans(name_regex: str) -> set[tuple[int, int]]: + """The \\uXXXX-\\uXXXX span pairs a rule's character class declares.""" + return { + (int(lo, 16), int(hi, 16)) + for lo, hi in re.findall(r"\\u([0-9A-Fa-f]{4})-\\u([0-9A-Fa-f]{4})", + name_regex)} + + +def _expected_bmp_spans() -> set[tuple[int, int]]: + """What a full CJK character class in the toml must declare: the + table's BMP spans plus the sanctioned extras.""" + return {span + for spans in _policy._SCRIPT_RANGES.values() + for span in spans + if span[1] <= 0xFFFF} | set(_SANCTIONED_EXTRAS) + + def test_differential_cjk_rule_matches_the_script_ranges() -> None: """The CJK rule in tools/differential/expected_changes.toml hand- copies the script spans from _policy._SCRIPT_RANGES into a character @@ -224,67 +251,64 @@ def test_differential_cjk_rule_matches_the_script_ranges() -> None: "a Script joined _SCRIPT_RANGES: decide whether the " f"differential rule in {toml_path.name} should cover it, then " "update this assertion") - declared = { - (int(lo, 16), int(hi, 16)) - for lo, hi in re.findall(r"\\u([0-9A-Fa-f]{4})-\\u([0-9A-Fa-f]{4})", - matched[0]["name_regex"])} - # the two spans the rule carries that no Script claims (see above) - sanctioned_extras = {(0xFF65, 0xFF65)} - for xlo, xhi in sanctioned_extras: + declared = _declared_spans(matched[0]["name_regex"]) + # the extras must stay UNclassified, or they belong in the table + for xlo, xhi in _SANCTIONED_EXTRAS: assert not any(lo <= xhi and xlo <= hi for spans in _policy._SCRIPT_RANGES.values() for lo, hi in spans), ( f"U+{xlo:04X}-U+{xhi:04X} is classified now; drop it " - "from the sanctioned extras") - expected = {span - for spans in _policy._SCRIPT_RANGES.values() - for span in spans - if span[1] <= 0xFFFF} | sanctioned_extras + "from _SANCTIONED_EXTRAS") + expected = _expected_bmp_spans() assert declared == expected, ( f"{toml_path.name}'s CJK name_regex declares {sorted(declared)}; " f"_SCRIPT_RANGES' BMP spans are {sorted(expected)}") -@pytest.mark.parametrize("slug", ["cjk-delimited-nickname", - "cjk-comma-compound"]) -def test_differential_compound_rules_match_the_script_ranges( - slug: str) -> None: - """The compound rules carry FURTHER hand copies of the script - spans in the toml: each one's require-a-classified-codepoint - lookahead exists so its trigger set alone (delimiters; a comma) - cannot claim a Latin name's regression ('John 「Jack」 Kennedy', - 'Smith, Jr.' -- neither contains a classified character). Pin each - copy to the table exactly as the canonical CJK rule's is, plus the - nickname rule's delimiter set itself, so widening either is an - explicit decision here rather than a silent absorption change. - - Selection note for future rule authors: the canonical-CJK-rule pin +def test_every_span_bearing_rule_matches_the_script_ranges() -> None: + """Auto-discovered pin for every FURTHER hand copy of the script + spans in the toml: any rule whose character class declares spans + intersecting _SCRIPT_RANGES must declare the whole expected class + (table BMP spans + sanctioned extras). The compound rules' + require-a-classified-codepoint lookaheads exist so their trigger + sets alone (delimiters; a comma) cannot claim a Latin name's + regression -- and each such lookahead is a copy nothing else + checks. Discovery replaces a hand-maintained slug roster: a new + compound rule's copy is pinned by existing here, not by an author + remembering to enroll it. Rules whose spans touch OTHER scripts + (Cyrillic, say) are out of scope and skipped by the intersection + test. + + Selection note for future rule authors: the canonical-rule pin above selects by the literal '#271'/'#272' substrings and asserts - uniqueness -- the compound slugs avoid them on purpose, and any - new rule's must too (and its slug joins this parametrize list, or - its span copy goes unpinned). + uniqueness -- compound slugs must avoid them. """ toml_path = (Path(__file__).parents[2] / "tools" / "differential" / "expected_changes.toml") rules = tomllib.loads(toml_path.read_text())["change"] - matched = [r for r in rules if slug in r["issue"]] - assert len(matched) == 1 - regex = matched[0]["name_regex"] - if slug == "cjk-delimited-nickname": - assert "[「」『』・･]" in regex, ( - "the compound rule's delimiter set changed; decide " - "deliberately") - declared = { - (int(lo, 16), int(hi, 16)) - for lo, hi in re.findall(r"\\u([0-9A-Fa-f]{4})-\\u([0-9A-Fa-f]{4})", - regex)} - expected = {span - for spans in _policy._SCRIPT_RANGES.values() - for span in spans - if span[1] <= 0xFFFF} | {(0xFF65, 0xFF65)} - assert declared == expected, ( - f"{slug}'s codepoint lookahead declares {sorted(declared)}; " - f"_SCRIPT_RANGES' BMP spans are {sorted(expected)}") + table_spans = _expected_bmp_spans() + checked = [] + for rule in rules: + regex = rule.get("name_regex") + if not isinstance(regex, str): + continue + declared = _declared_spans(regex) + if not declared & table_spans: + continue + checked.append(rule["issue"]) + assert declared == table_spans, ( + f"{rule['issue']!r} declares {sorted(declared)}; expected " + f"{sorted(table_spans)}") + # the canonical rule plus both compound rules, today -- if this + # count drops, a hand copy fell out of discovery's reach + assert len(checked) >= 3, checked + # the delimiter compound's trigger set is its own decision surface + nickname = [r for r in rules + if "cjk-delimited-nickname" in r["issue"]] + assert len(nickname) == 1 + assert "[\u300C\u300D\u300E\u300F\u30FB\uFF65]" in nickname[0][ + "name_regex"] or "[「」『』・･]" in nickname[0]["name_regex"], ( + "the compound rule's delimiter set changed; decide deliberately") def test_cjk_corpus_matches_the_case_table() -> None: