diff --git a/AGENTS.md b/AGENTS.md index 790d70d..537bd09 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -75,7 +75,7 @@ logging.getLogger('HumanName').setLevel(logging.DEBUG) The library has two layers: `nameparser/config/` (data) and `nameparser/parser.py` (logic). -**Design philosophy — positional and language-agnostic.** The parser assigns parts by *position* plus small sets of words that join to neighbors; it never detects language. A name's language can't be reliably inferred from Latin-script transliteration ("Ali" is Arabic or Italian; "Van"/"Della"/"Bin" are first names in some cultures, particles in others), so language-specific rules belong in opt-in `Constants` config, never global defaults. Many "wrong for language X" reports (#133, #150, #130, #85, #103, #146, #83) are irreducible ambiguities — e.g. `de Mesnil` (want last name) vs `Van Johnson` (want first name) are the same `[prefix][word]` shape. Before adding a rule, confirm it doesn't break the opposite case (run the full suite — Portuguese and "Van Johnson" tests are the usual canaries). **The one scoped exception (2.1, #271/#272): script-conditional behavior is permitted exactly where the SCRIPT ITSELF — not statistics about it — determines the convention.** The never-detect-language rule above is about Latin *transliteration*, where the signal genuinely is destroyed; native script is a different question, and it is answered per behavior rather than per script. Four defaults fall out of it. Wholly-Han, wholly-Hangul and kana-licensed names read family-first (`Policy.script_orders`) — no language detection needed, because zh and ja both write family-first in native script, so order cannot be misread even though the language is unknowable. Unspaced hangul splits into surname + given name (`config/surnames.py` ships the Korean census list as DEFAULT vocabulary) — nothing but Korean is written in hangul and the surnames are a closed census set, and the vocabulary is self-selecting besides: a hangul entry can only ever match hangul text. Hiragana licenses Japanese (#272) — a name whose characters stay inside Han∪kana while carrying at least one kana cannot be Chinese (the kana rules it out) and is not a transcription (foreign names are transcribed in katakana ALONE, マイケル has no kanji), so 高橋みなみ and 山田 エミ read family-first too; mechanically they resolve to the HIRAGANA entry, the license's carrier key. PURE katakana is excluded and keeps the positional default: マイケル・ジャクソン is a transcribed foreign name in its source order. The nakaguro belongs to the same doctrine but is decided a layer down: U+30FB and its halfwidth twin U+FF65 separate tokens like whitespace, unconditionally and in tokenize, so neither policy opt-out (`script_orders={}`, `segment_scripts=()`) reaches it — the codepoints are CJK-only and appear in no other script's names, which is what licenses a tokenize-level rule where U+00B7 (also the Catalan punt volat, interior to Gal·la) would need a guard. Han segmentation stays OPT-IN (`locales.ZH` for Chinese, `locales.JA` for Japanese) — a zh surname list corrupts Japanese kanji names, since 高 is a common Chinese surname and 高橋一郎 would split 高+橋一郎 where the correct reading is 高橋+一郎; no surname list divides a kanji name at all, so `locales.JA` activates the stage and a pluggable `Parser(segmenter=...)` does the dividing. Latin-script input is never touched by any of this: "Kim Min-jun" is genuinely order-ambiguous and stays governed by `name_order` and opt-in packs. Before adding a script-conditional rule, work out which of the three it is — certain, certain for this one behavior only, or a statistical guess wearing a script's clothes. +**Design philosophy — positional and language-agnostic.** The parser assigns parts by *position* plus small sets of words that join to neighbors; it never detects language. A name's language can't be reliably inferred from Latin-script transliteration ("Ali" is Arabic or Italian; "Van"/"Della"/"Bin" are first names in some cultures, particles in others), so language-specific rules belong in opt-in `Constants` config, never global defaults. Many "wrong for language X" reports (#133, #150, #130, #85, #103, #146, #83) are irreducible ambiguities — e.g. `de Mesnil` (want last name) vs `Van Johnson` (want first name) are the same `[prefix][word]` shape. Before adding a rule, confirm it doesn't break the opposite case (run the full suite — Portuguese and "Van Johnson" tests are the usual canaries). **The one scoped exception (2.1, #271/#272): script-conditional behavior is permitted exactly where the SCRIPT ITSELF — not statistics about it — determines the convention.** The never-detect-language rule above is about Latin *transliteration*, where the signal genuinely is destroyed; native script is a different question, and it is answered per behavior rather than per script. Five defaults fall out of it. Wholly-Han, wholly-Hangul and kana-licensed names read family-first (`Policy.script_orders`) — no language detection needed, because zh and ja both write family-first in native script, so order cannot be misread even though the language is unknowable. Unspaced hangul splits into surname + given name (`config/surnames.py` ships the Korean census list as DEFAULT vocabulary) — nothing but Korean is written in hangul and the surnames are a closed census set, and the vocabulary is self-selecting besides: a hangul entry can only ever match hangul text. Hiragana licenses Japanese (#272) — a name whose characters stay inside Han∪kana while carrying at least one kana cannot be Chinese (the kana rules it out) and is not a transcription (foreign names are transcribed in katakana ALONE, マイケル has no kanji), so 高橋みなみ and 山田 エミ read family-first too; mechanically they resolve to the HIRAGANA entry, the license's carrier key. PURE katakana is excluded and keeps the positional default: マイケル・ジャクソン is a transcribed foreign name in its source order. And the 间隔号 U+00B7 (#298) is the transcription marker for scripts that HAVE no transcription script: a name it divides (威廉·莎士比亚 — flanked by classified characters on both sides, so Catalan's Gal·la is untouched) keeps its source order and never segments — the orthography names the convention, exactly as pure katakana does, with the divider carrying the signal instead of the script. The nakaguro belongs to the same doctrine but is decided a layer down: U+30FB and its halfwidth twin U+FF65 separate tokens like whitespace, unconditionally and in tokenize, so neither policy opt-out (`script_orders={}`, `segment_scripts=()`) reaches it — the codepoints are CJK-only and appear in no other script's names, which is what licenses a tokenize-level rule where U+00B7 (also the Catalan punt volat, interior to Gal·la) needs the flanked-by-classified-script guard `_tokenize_region` gives it (#298). Han segmentation stays OPT-IN (`locales.ZH` for Chinese, `locales.JA` for Japanese) — a zh surname list corrupts Japanese kanji names, since 高 is a common Chinese surname and 高橋一郎 would split 高+橋一郎 where the correct reading is 高橋+一郎; no surname list divides a kanji name at all, so `locales.JA` activates the stage and a pluggable `Parser(segmenter=...)` does the dividing. Latin-script input is never touched by any of this: "Kim Min-jun" is genuinely order-ambiguous and stays governed by `name_order` and opt-in packs. Before adding a script-conditional rule, work out which of the three it is — certain, certain for this one behavior only, or a statistical guess wearing a script's clothes. ### Configuration layer (`nameparser/config/`) diff --git a/docs/concepts.rst b/docs/concepts.rst index 25b612b..0da1d14 100644 --- a/docs/concepts.rst +++ b/docs/concepts.rst @@ -58,7 +58,9 @@ layer, which assigns purely by where a word sits: the first unclaimed word is the given name, the last is the family name, and anything between them is the middle name. ``name_order``, an explicit comma, and — for a name written wholly in one East Asian script — -``script_orders`` change what "first" and "last" mean here; nothing +``script_orders`` change what "first" and "last" mean here; the +Chinese interpunct ``·`` dividing such a name walks the last of those +back, marking a transcription that keeps its source order; nothing else does. This is the whole parser in two sentences, and it explains its diff --git a/docs/migrate.rst b/docs/migrate.rst index 3ede387..5474858 100644 --- a/docs/migrate.rst +++ b/docs/migrate.rst @@ -429,7 +429,11 @@ now. surface is frozen for 2.x — so the way out is the 2.0 API: ``Parser(policy=Policy(script_orders={}, segment_scripts=()))`` restores 1.4's reading of every shape above that turns on order or -splitting. The middle dot is the one exception: it is decided in -tokenization rather than by policy, so a name written with one still -divides at the dot, and still renders with a space, whatever those two -fields are set to. +splitting. The middle dots are the exception: both the katakana dot +and the Chinese interpunct ``·`` (U+00B7, dividing a transcription +like ``威廉·莎士比亚``, and a separator only between classified-script +characters — each side judged on its own — where the katakana dot is +unconditional) are decided +in tokenization rather than by policy, so a name written with either +still divides at the dot, and still renders with a space, whatever +those two fields are set to. diff --git a/docs/release_log.rst b/docs/release_log.rst index 50a39ad..afd6221 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -23,7 +23,8 @@ Release Log - Fix unspaced Korean names not splitting: the census surname list now ships as default vocabulary (``Lexicon.surnames``) with hangul segmentation on by default (``Policy.segment_scripts``), so ``"김민준"`` parses family ``김``, given ``민준`` where 1.x returned the whole string as ``first``. Rendering follows the split, so ``str(HumanName("김민준"))`` is now ``"민준 김"`` where 1.x echoed the input back unchanged. Nothing but Korean is written in hangul and its surnames are a closed census set, which is what makes the split safe as a default rather than a pack. **Default-on**, and it reaches ``HumanName`` too. ``Policy(segment_scripts=())`` turns the split off; ``Policy(script_orders={})`` separately restores the positional reading; clearing both restores 2.0 behavior exactly (#271) - Fix Japanese names carrying kana parsing given-first: the family-first rule above extends to any name whose characters stay within kanji and kana while carrying at least one kana character, so ``"高橋 みなみ"`` gives family ``高橋`` and ``"山田 エミ"`` family ``山田`` where 1.x read both the other way round, and a lone such token (``"高橋みなみ"``, ``"みなみ"``) lands in ``family``/``last`` where 1.x put it in ``given``/``first``. The reasoning is the one hangul already uses: hiragana never transcribes a foreign name, and a transcription is kana ALONE, so kanji-plus-kana is a Japanese person's name written in Japanese order. A name written **wholly in katakana** is deliberately excluded and stays positional — it is predominantly a transcribed foreign name (``"マイケル ジャクソン"``) already in given-first order. **Default-on: changes parse output for kana-bearing Japanese names**, through ``HumanName`` as well as the 2.0 API, and the spaced forms change what the name renders as (``str(HumanName("高橋 みなみ"))`` is now ``"みなみ 高橋"``). ``Policy(script_orders={})`` clears this entry along with the Han and Hangul ones (#272) - Fix names containing 〆 (U+3006, the shime mark that opens Japanese surnames like 〆木 and 〆谷) parsing given-first: the script classifier now counts 〆 as Han, extending the 々 entry the table already carries, and going a step further than it — 々 is Script=Han and merely outside the ideograph blocks, while 〆 is Script=Common — so these names take the East Asian family-first reading like any other wholly-Han name. ``Policy(script_orders={})`` restores the positional reading for these names exactly as for other wholly-Han names (#303) - - Fix the katakana middle dot ``・`` (U+30FB, and its halfwidth twin U+FF65) being read as part of a name rather than as the divider it is: it now separates tokens exactly as a space does. A transcribed foreign name therefore divides into its parts and, being wholly katakana, keeps its source order — ``"マイケル・ジャクソン"`` gives given ``マイケル``, family ``ジャクソン``, where 1.x left the whole string in ``first`` — while a kanji pair written the same way takes the family-first rule (``"高橋・一郎"`` → family ``高橋``). Native Japanese names never contain this character and no other script's names use it, so the separation is unconditional, which also means it is **not** covered by the two policy opt-outs. Rendering follows, the way the Korean split's does: the dot comes back as a space, so ``str(HumanName("マイケル・ジャクソン"))`` is now ``"マイケル ジャクソン"``, and that reaches delimited content too — the nickname in ``"山田 太郎 (マイケル・ジャクソン)"`` renders ``"マイケル ジャクソン"``. The Chinese interpunct ``·`` is deliberately NOT included: U+00B7 is also the Catalan punt volat and appears inside legitimate names (``Gal·la``) (#272) + - Fix the katakana middle dot ``・`` (U+30FB, and its halfwidth twin U+FF65) being read as part of a name rather than as the divider it is: it now separates tokens exactly as a space does. A transcribed foreign name therefore divides into its parts and, being wholly katakana, keeps its source order — ``"マイケル・ジャクソン"`` gives given ``マイケル``, family ``ジャクソン``, where 1.x left the whole string in ``first`` — while a kanji pair written the same way takes the family-first rule (``"高橋・一郎"`` → family ``高橋``). Native Japanese names never contain this character and no other script's names use it, so the separation is unconditional, which also means it is **not** covered by the two policy opt-outs. Rendering follows, the way the Korean split's does: the dot comes back as a space, so ``str(HumanName("マイケル・ジャクソン"))`` is now ``"マイケル ジャクソン"``, and that reaches delimited content too — the nickname in ``"山田 太郎 (マイケル・ジャクソン)"`` renders ``"マイケル ジャクソン"``. The Chinese interpunct ``·`` is not an unconditional separator like these two: U+00B7 is also the Catalan punt volat and appears inside legitimate names (``Gal·la``), so it divides only between classified-script characters — the transcription treatment it marks is #298's (#272) + - Fix 间隔号-divided transcriptions parsing as one unsplit token: U+00B7 — the interpunct Chinese text divides a transcribed foreign name with, ``威廉·莎士比亚`` for William Shakespeare — is now a token separator between characters of a classified script, and a name it divides keeps its source order and is never segmented: the dot is the transcription marker, playing the role pure katakana plays in the kana license. It divides ONLY between classified-script characters, so the Catalan punt volat interior to names like ``Gal·la`` is untouched. The Japanese nakaguro is deliberately not a transcription marker — ``高橋・一郎`` is roster formatting, 姓・名, and keeps its family-first reading — so a transcription typed with the wrong dot (``威廉・莎士比亚``) reads by the convention of the codepoint it was typed with; like the spaced form, only the Chinese dot rescues it. Rendering stays space-joined (``str(HumanName("威廉·莎士比亚"))`` is ``"威廉 莎士比亚"``); a custom ``string_format`` such as ``"{first}·{last}"`` reinstates the dot (#298) - Fix NFD-decomposed input missing the East Asian defaults entirely: script classification now normalizes to NFC before deciding, so a Korean or Japanese name typed on macOS — where decomposed text is routine — gets the same order rule as its composed twin, which it silently did not before. Segmentation MATCHING deliberately stays raw, so an unspaced NFD hangul name is ordered correctly but not split, rather than being split in the wrong place. One gotcha worth stating plainly: parse output preserves the encoding it was given, so for NFD input ``name.family == "김"`` is ``False`` even though it is the same name — compare NFC-normalized text when comparing across encodings (#272) * 2.0.0 - July 27, 2026 diff --git a/docs/usage.rst b/docs/usage.rst index 61f249a..a77a40f 100644 --- a/docs/usage.rst +++ b/docs/usage.rst @@ -188,6 +188,24 @@ language is up to you. When you know the data is Chinese, apply the >>> parser_for(locales.ZH).parse("毛泽东").family '毛' +Chinese also has a transcription convention, and it is written in the +punctuation: a foreign name transcribed into Han characters keeps its +source order and divides its parts with the 间隔号, the interpunct +``·`` (U+00B7) — 威廉·莎士比亚 is William Shakespeare, given name +first. The dot itself is the marker, so nameparser reads U+00B7 as a +token separator when it sits between classified-script characters +(each side judged on its own — a hangul character beside a Han one +qualifies), and a dot anywhere in the name reads the whole name as a +transcription listing: it keeps the order it was written in and is +never segmented — the role pure katakana plays for Japanese +transcriptions, played here by the divider instead of the script. +Because only classified characters on both sides make it a divider, +the same codepoint interior to a Latin-script name — the Catalan punt +volat in ``Gal·la`` — is untouched, and a dot with a classified +character on just one side (``王·Smith``) stays part of the word +undivided. The Japanese middle dot ・ (covered next) is a different +mark carrying a different convention, and keeps its own reading. + Japanese ~~~~~~~~~ @@ -266,8 +284,15 @@ positional rules — order genuinely varies in romanized data, so nothing script-based applies. A name written wholly in katakana stays positional for the reason given above, pack or no pack: it is predominantly a transcription, and a transcription is already in the -order it should be read in. And a comma disables the script behaviors -entirely, on the reasoning ``name_order`` already follows: whoever +order it should be read in. A Han transcription written with a space +instead of the 间隔号 (威廉 莎士比亚) carries nothing to distinguish it +from a native two-token name and keeps the family-first reading — the +dot is the marker, and without it there is no signal. The same holds +for a transcription typed with the Japanese middle dot (威廉・莎士比亚): +each dot carries its own script's convention, the nakaguro's is +Japanese roster formatting rather than transcription, so only the +Chinese dot rescues the source order. And a comma disables the script +behaviors entirely, on the reasoning ``name_order`` already follows: whoever wrote the comma has already said where the family name ends. A division the parser had to choose is reported rather than hidden. diff --git a/nameparser/_pipeline/_assign.py b/nameparser/_pipeline/_assign.py index 3672341..0407736 100644 --- a/nameparser/_pipeline/_assign.py +++ b/nameparser/_pipeline/_assign.py @@ -82,11 +82,16 @@ def _peel_leading_titles(pieces: tuple[tuple[int, ...], ...], def _effective_order(policy: Policy, pieces: list[tuple[int, ...]], - tokens: list[WorkToken]) -> tuple[Role, Role, Role]: + tokens: list[WorkToken], + *, dot_divided: bool) -> tuple[Role, Role, Role]: """script_orders resolution (#271): when every name piece is written wholly in ONE script that has an entry, that script's order governs the positional read; anything else -- Latin, mixed - scripts, no entry -- falls back to name_order. Piece-level, after + scripts, no entry -- falls back to name_order. A 间隔号-divided + name (`dot_divided`, #298) suppresses the whole lookup first: the + dot marks a transcription -- playing the role pure katakana plays + in the kana license, orthography naming the convention -- so the + license yields to name_order. Piece-level, after title/suffix peeling: 'Dr. 毛泽东' is a wholly-Han NAME under a Latin title. Kana-licensed tokens (高橋みなみ, #272) resolve to HIRAGANA the same way a wholly-Han or wholly-Hangul token resolves @@ -104,6 +109,10 @@ def _effective_order(policy: Policy, resolves the SCRIPT for a single token. This function calls that one per token below. """ + # #298 transcription marker -- see the docstring; codepoint-scoped + # (only U+00B7 records, spec 2026-07-30 decision 5) + if dot_divided: + return policy.name_order if not policy.script_orders: return policy.name_order # Collect every token's script rather than comparing pairwise as @@ -232,7 +241,8 @@ def _assign_main(seg_idx: int, state: ParseState, # pieces only, so a Latin title or suffix ('Dr. 毛 泽东', '毛 泽东, # PhD') cannot make a wholly-CJK name look mixed-script. order = _effective_order(state.policy, - [pieces[i] for i in name_pieces], tokens) + [pieces[i] for i in name_pieces], tokens, + dot_divided=bool(state.interpunct_offsets)) roles = _name_positions(order, len(name_pieces)) for pos, piece_idx in enumerate(name_pieces): _set_roles(tokens, pieces[piece_idx], roles[pos]) diff --git a/nameparser/_pipeline/_script_segment.py b/nameparser/_pipeline/_script_segment.py index 0365a76..37e01c2 100644 --- a/nameparser/_pipeline/_script_segment.py +++ b/nameparser/_pipeline/_script_segment.py @@ -1,6 +1,6 @@ """Stage: script_segment (#271, #272). -Consumes: tokens, segments, structure, segmenter. +Consumes: tokens, segments, structure, interpunct_offsets, segmenter. Produces: tokens (the first activated-script token of the name segment split into n+1 sub-slices), segments (index runs remapped past the insertions), ambiguities (indices likewise remapped, plus a @@ -207,6 +207,16 @@ def script_segment(state: ParseState) -> ParseState: return state if state.structure is Structure.FAMILY_COMMA: return state # the comma already drew the boundary + if state.interpunct_offsets: + # #298: a 间隔号-divided name is a transcription -- its pieces + # are syllable groups, not surname+given, so neither the + # vocabulary nor the segmenter applies (codepoint-scoped: the + # nakaguro records nothing and gates nothing, spec decision 5). + # State-global like the FAMILY_COMMA gate above: a marker + # anywhere in the name reads the WHOLE name as a transcription + # listing, so even an un-dotted hangul token beside a dotted + # one stays whole. + return state # segments[0] is the NAME part under both remaining structures # (everything, under NO_COMMA); later segments are suffixes. Its # members are main-stream token indices by construction, so diff --git a/nameparser/_pipeline/_state.py b/nameparser/_pipeline/_state.py index 6cf05bd..d524e4e 100644 --- a/nameparser/_pipeline/_state.py +++ b/nameparser/_pipeline/_state.py @@ -66,7 +66,9 @@ class ParseState: """Carried through the stage fold. Frozen; stages return copies via dataclasses.replace. Fields are filled progressively: extract_delimited -> extracted/masked; tokenize -> tokens (span- - sorted)/comma_offsets; segment -> segments/structure; + sorted)/comma_offsets/interpunct_offsets (the 间隔号 offsets the + order and segmentation decisions consult, #298; the nakaguro + separators record NOTHING); segment -> segments/structure; script_segment -> tokens and segments again (the one stage that changes the token COUNT: an unspaced CJK token splits into n+1 pieces, still as sub-slices of the original, and every later index @@ -94,6 +96,7 @@ class ParseState: masked: tuple[Span, ...] = () tokens: tuple[WorkToken, ...] = () comma_offsets: tuple[int, ...] = () + interpunct_offsets: tuple[int, ...] = () segments: tuple[tuple[int, ...], ...] = () structure: Structure = Structure.NO_COMMA # pieces[s][p] = run of token indices: piece p of segment s. diff --git a/nameparser/_pipeline/_tokenize.py b/nameparser/_pipeline/_tokenize.py index 0d4643d..682ae19 100644 --- a/nameparser/_pipeline/_tokenize.py +++ b/nameparser/_pipeline/_tokenize.py @@ -3,7 +3,8 @@ Consumes: original, masked (regions to skip), extracted (regions that tokenize with a pre-set role). Produces: tokens (span-sorted WorkTokens; text always == original -slice), comma_offsets (segmentation points; never tokens). +slice), comma_offsets (segmentation points; never tokens), +interpunct_offsets (间隔号 transcription markers, #298; never tokens). Reads: Policy.strip_emoji, Policy.strip_bidi. There is NO text-rewriting normalize stage: whitespace collapsing, @@ -12,7 +13,8 @@ separators and never enter a token, so spans always index the original exactly as given. Whitespace and the name-dot are unconditional; emoji/bidi stripping alone is policy-gated -(Policy.strip_emoji/strip_bidi). +(Policy.strip_emoji/strip_bidi). The Chinese interpunct U+00B7 is +context-sensitive -- see _INTERPUNCT below. v1's squash_emoji/squash_bidi REMOVED the char and joined neighbors ('A\U0001f600B' -> 'AB'); here an ignorable char is a SEPARATOR @@ -28,6 +30,7 @@ from nameparser._pipeline._state import ( COMMA_CHARS, ParseState, WorkToken, ) +from nameparser._policy import Policy, _SCRIPT_RANGES, _script_matcher from nameparser._types import Role, Span # Ported from v1 (nameparser/config/regexes.py, "emoji" and "bidi") -- @@ -44,15 +47,53 @@ # The katakana middle dot and its halfwidth twin divide the parts of # a foreign name transcribed into katakana (マイケル・ジャクソン) -- # native names never contain them, so they separate unconditionally, -# like whitespace (amendment 2026-07-29 section 1b). The Chinese -# interpunct U+00B7 (威廉·莎士比亚) is deliberately NOT here: it is -# also the Catalan punt volat, which sits INSIDE a single name piece -# (Gal·la), so separating on it unconditionally would break names it -# has no business touching. Admitting it would need a flanked-by-CJK -# guard, which is a different rule from this set's "these codepoints -# are always separators". +# like whitespace (amendment 2026-07-29 section 1b). They record no +# offset: the nakaguro also divides kanji roster pairs (高橋・一郎, +# 姓・名 -- read family-first by the script license, #272), so it is +# not a transcription marker. Only the Chinese 间隔号 is (#298), and +# it is context-sensitive -- see _INTERPUNCT below. _NAME_DOT_SEPARATORS = frozenset({"\u30FB", "\uFF65"}) +_INTERPUNCT = "\u00B7" +# Per-CHAR classifier for the interpunct's flank guard: a one-char +# string is wholly-classified iff the character is. U+00B7 cannot be +# an unconditional separator like the name dots above -- it is also +# the Catalan punt volat, INTERIOR to legitimate names (Gal\u00B7la) -- so +# it divides only between classified-script characters: the first +# context-sensitive separator rule, which is why it lives in +# _tokenize_region (where the index exists) and not in _ignorable. +_classified_char = _script_matcher(*_SCRIPT_RANGES, whole=True) + + +def _is_emoji(ch: str) -> bool: + cp = ord(ch) + return any(lo <= cp <= hi for lo, hi in _EMOJI_RANGES) + + +def _stripped(ch: str, policy: Policy) -> bool: + """True when the strip policy removes `ch` from the token stream. + The ONE definition of that set, shared by _ignorable and _flank: a + character that vanishes from tokens must not occupy a flank + position either, so a new strip class added here stays transparent + to the interpunct guard by construction.""" + if policy.strip_bidi and _BIDI.match(ch): + return True + return bool(policy.strip_emoji and _is_emoji(ch)) + + +def _flank(text: str, indices: range, policy: Policy) -> str | None: + """The nearest flank character the strip policy would keep, or + None if the range exhausts. Stripped invisibles are TRANSPARENT + here: an RTL document quoting a transcription puts U+200F beside + the dot in visually identical text. Whitespace and the other + separators stay guard-defeating: a dot beside a space is not + between characters.""" + for i in indices: + ch = text[i] + if not _stripped(ch, policy): + return ch + return None + def _ignorable(ch: str, state: ParseState) -> bool: if ch.isspace(): @@ -63,36 +104,49 @@ def _ignorable(ch: str, state: ParseState) -> bool: # skip the checks below for every ASCII letter return False # unconditional, like whitespace -- not policy-gated, so this sits - # ahead of the policy checks rather than in _tokenize_region beside - # COMMA_CHARS (commas RECORD an offset; these dots must not). The - # only load-bearing constraint is "after the isascii fast path" - # (both dots are non-ASCII); being ahead of the bidi/emoji checks - # below is NOT load-bearing -- the three sets are disjoint, so a - # future edit is free to reorder them. + # ahead of the policy check rather than in _tokenize_region beside + # COMMA_CHARS (commas RECORD an offset; these dots must not -- + # only the 间隔号 marks a transcription and records, #298, see + # _NAME_DOT_SEPARATORS above). The only load-bearing constraint is + # "after the isascii fast path" (both dots are non-ASCII). if ch in _NAME_DOT_SEPARATORS: return True - if state.policy.strip_bidi and _BIDI.match(ch): - return True - if state.policy.strip_emoji: - cp = ord(ch) - return any(lo <= cp <= hi for lo, hi in _EMOJI_RANGES) - return False + return _stripped(ch, state.policy) def _tokenize_region(state: ParseState, start: int, end: int, - role: Role | None, record_commas: bool, - tokens: list[WorkToken], commas: list[int]) -> None: + role: Role | None, record_offsets: bool, + tokens: list[WorkToken], commas: list[int], + interpuncts: list[int]) -> None: text = state.original tok_start: int | None = None for i in range(start, end): ch = text[i] - if ch in COMMA_CHARS or _ignorable(ch, state): + is_separator = ch in COMMA_CHARS or _ignorable(ch, state) + if ch == _INTERPUNCT: + # Region-local flanks: a B7 at a region edge stays token + # text, and the bound also stops a custom delimiter's + # classified edge character from acting as a flank across + # a mask seam. The scan reads raw text like segmentation + # matching does (#272's stance): NFD hangul degrades to + # no-split, never wrong-split -- classification + # NFC-normalizes but the guard does not. + left = _flank(text, range(i - 1, start - 1, -1), + state.policy) + if left is not None and _classified_char(left): + right = _flank(text, range(i + 1, end), state.policy) + is_separator = (right is not None + and _classified_char(right)) + if is_separator: if tok_start is not None: tokens.append(WorkToken(text[tok_start:i], Span(tok_start, i), role=role)) tok_start = None - if ch in COMMA_CHARS and record_commas: - commas.append(i) + if record_offsets: + if ch in COMMA_CHARS: + commas.append(i) + elif ch == _INTERPUNCT: + interpuncts.append(i) continue if tok_start is None: tok_start = i @@ -104,17 +158,20 @@ def _tokenize_region(state: ParseState, start: int, end: int, def tokenize(state: ParseState) -> ParseState: tokens: list[WorkToken] = [] commas: list[int] = [] + interpuncts: list[int] = [] # main stream: everything outside masked regions boundaries = [0] for m in state.masked: boundaries.extend((m.start, m.end)) boundaries.append(len(state.original)) for start, end in zip(boundaries[::2], boundaries[1::2]): - _tokenize_region(state, start, end, None, True, tokens, commas) - # extracted regions: pre-set role, commas are mere separators + _tokenize_region(state, start, end, None, True, tokens, commas, + interpuncts) + # extracted regions: pre-set role, commas and interpuncts are mere + # separators for role, inner in state.extracted: _tokenize_region(state, inner.start, inner.end, role, False, - tokens, commas) + tokens, commas, interpuncts) tokens.sort(key=lambda t: t.span) # extract_delimited runs before tokens exist, so its ambiguities # carry a character offset instead of an index. Resolve them now @@ -147,4 +204,5 @@ def _containing(offset: int) -> tuple[int, ...]: for a in ambiguities) return dataclasses.replace(state, tokens=tuple(tokens), comma_offsets=tuple(sorted(commas)), + interpunct_offsets=tuple(sorted(interpuncts)), ambiguities=ambiguities) diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 62f7ced..753485d 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -651,4 +651,60 @@ def __post_init__(self) -> None: notes="delimited content tokenizes under the same separator " "rules: the dot renders back as a space in the nickname " "join -- a decision, not an accident"), + Case("zh_interpunct_transcription_source_order", "威廉·莎士比亚", + {"given": "威廉", "family": "莎士比亚"}, + classification="fix(#298)", + notes="间隔号-divided Han is a transcribed foreign name and " + "keeps source order -- the B7 is the transcription " + "marker, playing the role pure katakana plays in the " + "kana license; it divides only between classified " + "characters"), + Case("zh_interpunct_nakaguro_typed_stays_roster", "威廉・莎士比亚", + {"given": "莎士比亚", "family": "威廉"}, + classification="fix(#272)", + notes="the SAME transcription typed with the Japanese " + "nakaguro reads as the dot's own typography says -- a " + "姓・名 roster pair, family-first (#272) -- because the " + "nakaguro records nothing (spec 2026-07-30 decision 5: " + "codepoint-scoped; only the Chinese B7 marks a " + "transcription). A limitation row: chosen, not " + "accidental -- cross-convention input reads by the " + "convention of the codepoint it was typed with"), + Case("ja_interpunct_b7_katakana", "マイケル·ジャクソン", + {"given": "マイケル", "family": "ジャクソン"}, + classification="fix(#298)", + notes="sloppy-IME B7 between katakana divides like the " + "nakaguro; katakana was never licensed, so order was " + "already positional -- the split is the fix"), + Case("latin_punt_volat_is_name_interior", "Gal·la Marcet", + {"given": "Gal·la", "family": "Marcet"}, + notes="the Catalan punt volat: U+00B7 with Latin neighbors " + "is interior to the name, never a divider -- the flank " + "guard's whole reason"), + Case("zh_interpunct_suppresses_segmentation", "马丁·路德·金", + {"given": "马丁", "middle": "路德", "family": "金"}, + locale="zh", + classification="fix(#298)", + notes="马 is a listed zh surname, but a 间隔号-divided name " + "is a transcription: the dot gates segmentation off, " + "so 马丁 stays whole and the positional read stands"), + Case("ko_interpunct_transcription_source_order", "마이클·잭슨", + {"given": "마이클", "family": "잭슨"}, + classification="fix(#298)", + notes="Korean writes transcribed foreign names with the " + "interpunct too: the dot suppresses the hangul " + "family-first entry AND the default segmentation -- 마 " + "is a listed census surname, and the spaced form " + "마이클 잭슨 really does mis-split 마|이클 today, so " + "the dot is what rescues the ko transcription"), + Case("zh_interpunct_with_suffix_comma", "威廉·莎士比亚, PhD", + {"given": "威廉", "family": "莎士比亚", "suffix": "PhD"}, + classification="fix(#298)", + notes="the transcription reading composes with a suffix " + "comma: the marker is structure-independent"), + Case("zh_interpunct_half_flanked_stays", "王·Smith", + {"given": "王·Smith"}, + notes="one classified neighbor is not enough: the guard " + "requires both, so the undivided dot remains part of " + "the word -- declining, not deciding"), ) diff --git a/tests/v2/pipeline/test_assign.py b/tests/v2/pipeline/test_assign.py index 69544c1..5c59a8b 100644 --- a/tests/v2/pipeline/test_assign.py +++ b/tests/v2/pipeline/test_assign.py @@ -210,6 +210,24 @@ def test_pure_katakana_piece_falls_back_to_name_order() -> None: assert _by_role(out, Role.FAMILY) == "ジャクソン" +def test_interpunct_divided_name_reads_positionally() -> None: + # 威廉·莎士比亚 is William Shakespeare: a 间隔号-divided name is a + # transcription and keeps source order (spec 2026-07-30) -- the + # B7 is the marker, playing the role pure katakana plays in the + # kana license + out = _assigned("威廉·莎士比亚") + assert _by_role(out, Role.GIVEN) == "威廉" + assert _by_role(out, Role.FAMILY) == "莎士比亚" + + +def test_interpunct_suppression_yields_to_explicit_name_order() -> None: + # the suppression falls back to name_order, it does not force + # given-first: an explicit FAMILY_FIRST governs the transcription + out = _assigned("威廉·莎士比亚", Policy(name_order=FAMILY_FIRST)) + assert _by_role(out, Role.FAMILY) == "威廉" + assert _by_role(out, Role.GIVEN) == "莎士比亚" + + def test_script_with_no_table_entry_falls_back() -> None: # A single, well-defined script the table simply does not list: # resolution must fall through to name_order rather than pick an diff --git a/tests/v2/pipeline/test_script_segment.py b/tests/v2/pipeline/test_script_segment.py index bb14d77..80f88fc 100644 --- a/tests/v2/pipeline/test_script_segment.py +++ b/tests/v2/pipeline/test_script_segment.py @@ -399,6 +399,52 @@ def test_a_neighbour_in_an_UNACTIVATED_script_still_blocks_the_consult() -> None assert out.ambiguities == (), name +# -- the 间隔号 gate: a divided name is a transcription (#298) ---------- + + +def test_interpunct_divided_name_never_segments() -> None: + # 马丁·路德·金 (Martin Luther King): 马 is a listed surname, but a + # 间隔号-divided name is a transcription -- its pieces are syllable + # groups, not surname+given, so the dot gates the stage off + # entirely, vocabulary AND segmenter (spec 2026-07-30). Built + # through the real tokenize so interpunct_offsets is the field's + # own producer's, not hand-set. + state = segment(tokenize(ParseState( + original="马丁·路德·金", + lexicon=Lexicon(surnames=frozenset({"马"})), policy=_HAN))) + assert state.interpunct_offsets # tokenize recorded the dots + out = script_segment(state) + assert out.tokens == state.tokens # nothing split: no 马 + 丁 + assert out.ambiguities == () + + +def test_interpunct_divided_name_never_consults_the_segmenter() -> None: + # the gate's other consumer, pinned separately: today the + # second-script-token precondition would decline this consult + # anyway, but the transcription doctrine must not depend on it -- + # the dot gates BEFORE either mechanism is reached + asked, seg = _capture() + state = segment(tokenize(ParseState( + original="马丁·路德·金", lexicon=Lexicon.empty(), policy=_HAN, + segmenter=seg))) + out = script_segment(state) + assert out.tokens == state.tokens + assert asked == [] + + +def test_interpunct_gates_the_whole_name_not_just_dotted_tokens() -> None: + # state-global on purpose: the marker reads the WHOLE name as a + # transcription listing, so the un-dotted hangul token stays whole + # too -- the baseline shows it segments without the B7 elsewhere + assert _texts(_run("김민준 马丁 路德", policy=_HANGUL)) == [ + "김", "민준", "马丁", "路德"] + state = segment(tokenize(ParseState( + original="김민준 马丁·路德", lexicon=_LEX, policy=_HANGUL))) + assert state.interpunct_offsets + out = script_segment(state) + assert out.tokens == state.tokens + + def test_the_segmenter_is_handed_the_gated_token_only() -> None: # It is asked where ONE token divides, so it gets that token's text # and nothing else -- not the original, not the name part joined diff --git a/tests/v2/pipeline/test_state.py b/tests/v2/pipeline/test_state.py index fe5c167..03676a3 100644 --- a/tests/v2/pipeline/test_state.py +++ b/tests/v2/pipeline/test_state.py @@ -15,7 +15,8 @@ def test_state_defaults_are_empty() -> None: assert s.tokens == () and s.segments == () and s.pieces == () assert s.structure is Structure.NO_COMMA assert s.ambiguities == () and s.extracted == () and s.masked == () - assert s.comma_offsets == () and s.dropped == () and s.piece_tags == () + assert s.comma_offsets == () and s.interpunct_offsets == () + assert s.dropped == () and s.piece_tags == () assert s.segmenter is None @@ -50,7 +51,8 @@ def test_stage_field_ownership() -> None: # tokenize also rewrites ambiguities: extract_delimited runs # before tokens exist, so its UNBALANCED_DELIMITER entries carry # a character offset that tokenize resolves to a token index - "tokenize": {"tokens", "comma_offsets", "ambiguities"}, + "tokenize": {"tokens", "comma_offsets", "interpunct_offsets", + "ambiguities"}, "segment": {"segments", "structure", "ambiguities"}, # script_segment splits one unspaced CJK token into n+1 pieces # (n = 1 from the vocabulary, any n from a segmenter), so it diff --git a/tests/v2/pipeline/test_tokenize.py b/tests/v2/pipeline/test_tokenize.py index 96b18c4..ad2dd7f 100644 --- a/tests/v2/pipeline/test_tokenize.py +++ b/tests/v2/pipeline/test_tokenize.py @@ -113,6 +113,102 @@ def test_nakaguro_next_to_a_real_comma_still_only_the_comma_offsets() -> None: assert state.comma_offsets == (5,) +def test_interpunct_splits_between_classified_characters() -> None: + state = _tokenized("威廉·莎士比亚") + assert [t.text for t in state.tokens] == ["威廉", "莎士比亚"] + assert state.interpunct_offsets == (2,) + + +def test_interpunct_stays_in_latin_names() -> None: + # the Catalan punt volat is INTERIOR to names; Latin neighbors + # never qualify + state = _tokenized("Gal·la Marcet") + assert [t.text for t in state.tokens] == ["Gal·la", "Marcet"] + assert state.interpunct_offsets == () + + +def test_interpunct_between_kana_splits() -> None: + # sloppy-IME katakana transcription + state = _tokenized("マイケル·ジャクソン") + assert [t.text for t in state.tokens] == ["マイケル", "ジャクソン"] + + +def test_interpunct_at_edges_stays() -> None: + # no classified neighbor on one side: not a divider position + state = _tokenized("·威廉") + assert [t.text for t in state.tokens] == ["·威廉"] + + +def test_stripped_bidi_mark_is_transparent_to_the_flank_guard() -> None: + # an RTL document quoting a transcription: U+200F beside the dot, + # visually identical to the clean form -- the stripped mark must + # not occupy a flank position + state = _tokenized("威廉‏·莎士比亚") + assert [t.text for t in state.tokens] == ["威廉", "莎士比亚"] + assert state.interpunct_offsets != () + + +def test_stripped_emoji_is_transparent_to_the_flank_guard() -> None: + state = _tokenized("威廉\U0001f600·莎士比亚") + assert [t.text for t in state.tokens] == ["威廉", "莎士比亚"] + + +def test_unstripped_bidi_mark_defeats_the_guard() -> None: + # the contrast: with strip_bidi off the mark is real token text, + # an unclassified flank -- the dot stays interior + state = _tokenized("威廉‏·莎士比亚", + policy=Policy(strip_bidi=False)) + assert state.interpunct_offsets == () + + +def test_interpunct_trailing_edge_stays() -> None: + # the upper bound: a trailing dot has no right flank -- and + # dropping the bound is an IndexError, not a wrong answer + state = _tokenized("威廉·") + assert [t.text for t in state.tokens] == ["威廉·"] + + +def test_interpunct_minimal_flanks_split() -> None: + state = _tokenized("威·廉") + assert [t.text for t in state.tokens] == ["威", "廉"] + assert state.interpunct_offsets == (1,) + + +def test_interpunct_one_sided_flank_stays() -> None: + # the AND in the guard: one classified neighbor is not enough, + # and no offset records (a spurious record would fake the + # transcription marker downstream) + state = _tokenized("Anna·威") + assert [t.text for t in state.tokens] == ["Anna·威"] + assert state.interpunct_offsets == () + + +def test_hangul_flanks_split_and_record() -> None: + # Korean writes transcribed foreign names with the interpunct too + # (마틴·루터·킹) -- hangul flanks are classified flanks + state = _tokenized("김·민준") + assert [t.text for t in state.tokens] == ["김", "민준"] + assert state.interpunct_offsets == (1,) + + +def test_interpunct_in_delimited_regions_splits_but_does_not_record() -> None: + # a B7 inside a nickname must not mark the OUTER name as a + # transcription -- same flag that keeps nickname commas out of + # comma_offsets + state = _tokenized("山田 太郎 (マイケル·ジャクソン)") + assert state.interpunct_offsets == () + assert "マイケル" in [t.text for t in state.tokens] + + +def test_nakaguro_never_records() -> None: + # the Japanese dot is roster/divider typography, not the + # transcription marker: it separates (#272) and records nothing -- + # 高橋・一郎 keeps its family-first roster reading downstream + state = _tokenized("威廉・莎士比亚") + assert [t.text for t in state.tokens] == ["威廉", "莎士比亚"] + assert state.interpunct_offsets == () + + def test_halfwidth_nakaguro_separates_too() -> None: # U+FF65 between halfwidth katakana -- build the string from # escapes and VERIFY the codepoints (the U+F900 homoglyph lesson): diff --git a/tests/v2/test_regex_sync.py b/tests/v2/test_regex_sync.py index 7c2ddd2..209c759 100644 --- a/tests/v2/test_regex_sync.py +++ b/tests/v2/test_regex_sync.py @@ -164,6 +164,33 @@ def test_comma_char_matches_the_pipeline_comma_set() -> None: assert set(_render._COMMA_CHAR.pattern.strip("[]")) == set(COMMA_CHARS) +# The one sanctioned divergence between the differential rules' +# character classes and _SCRIPT_RANGES: the halfwidth middle dot +# separates tokens without being classified (halfwidth kana stays out +# of the table on purpose). U+00B7 is deliberately NOT here -- its +# flank guard means every name it can change matches through a +# classified flanking character already. Single-sourced: both span +# pins below read this set. +_SANCTIONED_EXTRAS = frozenset({(0xFF65, 0xFF65)}) + + +def _declared_spans(name_regex: str) -> set[tuple[int, int]]: + """The \\uXXXX-\\uXXXX span pairs a rule's character class declares.""" + return { + (int(lo, 16), int(hi, 16)) + for lo, hi in re.findall(r"\\u([0-9A-Fa-f]{4})-\\u([0-9A-Fa-f]{4})", + name_regex)} + + +def _expected_bmp_spans() -> set[tuple[int, int]]: + """What a full CJK character class in the toml must declare: the + table's BMP spans plus the sanctioned extras.""" + return {span + for spans in _policy._SCRIPT_RANGES.values() + for span in spans + if span[1] <= 0xFFFF} | set(_SANCTIONED_EXTRAS) + + def test_differential_cjk_rule_matches_the_script_ranges() -> None: """The CJK rule in tools/differential/expected_changes.toml hand- copies the script spans from _policy._SCRIPT_RANGES into a character @@ -197,11 +224,17 @@ def test_differential_cjk_rule_matches_the_script_ranges() -> None: halfwidth middle dot U+FF65 changes parses without being classified as anything: tokenize separates on it, so a halfwidth transcription splits where 1.4 kept one token, while halfwidth - kana stays out of _SCRIPT_RANGES on purpose. Naming that span here - rather than relaxing the comparison to a subset check is what - keeps the pin honest in both directions: a THIRD source of - divergence still fails, and the one sanctioned difference has to - be written down to exist. + kana stays out of _SCRIPT_RANGES on purpose. U+00B7 -- the + context-sensitive 间隔号 (#298) -- also changes parses yet is + deliberately NOT an extra: its flank guard means every name it + can change matches the class through a flanking character + already, and a B7 span's only actual effect would be letting the + rule claim diffs on punt-volat Latin names (Gal·la), pre-excusing + a regression on exactly the guarded class. Naming the sanctioned + span here rather than relaxing the comparison to a subset check + is what keeps the pin honest in both directions: an unsanctioned + source of divergence still fails, and each sanctioned difference + has to be written down to exist. """ toml_path = (Path(__file__).parents[2] / "tools" / "differential" / "expected_changes.toml") @@ -218,59 +251,64 @@ def test_differential_cjk_rule_matches_the_script_ranges() -> None: "a Script joined _SCRIPT_RANGES: decide whether the " f"differential rule in {toml_path.name} should cover it, then " "update this assertion") - declared = { - (int(lo, 16), int(hi, 16)) - for lo, hi in re.findall(r"\\u([0-9A-Fa-f]{4})-\\u([0-9A-Fa-f]{4})", - matched[0]["name_regex"])} - # the one span the rule carries that no Script claims (see above) - halfwidth_dot = (0xFF65, 0xFF65) - assert not any(lo <= halfwidth_dot[0] <= hi - for spans in _policy._SCRIPT_RANGES.values() - for lo, hi in spans), ( - "U+FF65 is classified now; drop it from the sanctioned extras") - expected = {span - for spans in _policy._SCRIPT_RANGES.values() - for span in spans - if span[1] <= 0xFFFF} | {halfwidth_dot} + declared = _declared_spans(matched[0]["name_regex"]) + # the extras must stay UNclassified, or they belong in the table + for xlo, xhi in _SANCTIONED_EXTRAS: + assert not any(lo <= xhi and xlo <= hi + for spans in _policy._SCRIPT_RANGES.values() + for lo, hi in spans), ( + f"U+{xlo:04X}-U+{xhi:04X} is classified now; drop it " + "from _SANCTIONED_EXTRAS") + expected = _expected_bmp_spans() assert declared == expected, ( f"{toml_path.name}'s CJK name_regex declares {sorted(declared)}; " f"_SCRIPT_RANGES' BMP spans are {sorted(expected)}") -def test_differential_compound_rule_matches_the_script_ranges() -> None: - """The fix(cjk-delimited-nickname) rule (#295) carries the SECOND - hand copy of the script spans in the toml: its require-a-classified- - codepoint lookahead exists so the delimiter set alone cannot claim a - Latin name's first/last regression ('John 「Jack」 Kennedy' -- the - corner brackets sit outside every classified span). Pin that copy to - the table exactly as the canonical CJK rule's is, and pin the - delimiter set itself, so widening either (to ASCII quotes, say) is - an explicit decision here rather than a silent absorption change. - - Selection note for future rule authors: the canonical-CJK-rule pin +def test_every_span_bearing_rule_matches_the_script_ranges() -> None: + """Auto-discovered pin for every FURTHER hand copy of the script + spans in the toml: any rule whose character class declares spans + intersecting _SCRIPT_RANGES must declare the whole expected class + (table BMP spans + sanctioned extras). The compound rules' + require-a-classified-codepoint lookaheads exist so their trigger + sets alone (delimiters; a comma) cannot claim a Latin name's + regression -- and each such lookahead is a copy nothing else + checks. Discovery replaces a hand-maintained slug roster: a new + compound rule's copy is pinned by existing here, not by an author + remembering to enroll it. Rules whose spans touch OTHER scripts + (Cyrillic, say) are out of scope and skipped by the intersection + test. + + Selection note for future rule authors: the canonical-rule pin above selects by the literal '#271'/'#272' substrings and asserts - uniqueness -- this rule's slug avoids them on purpose, and any new - rule's must too. + uniqueness -- compound slugs must avoid them. """ toml_path = (Path(__file__).parents[2] / "tools" / "differential" / "expected_changes.toml") rules = tomllib.loads(toml_path.read_text())["change"] - matched = [r for r in rules if "cjk-delimited-nickname" in r["issue"]] - assert len(matched) == 1 - regex = matched[0]["name_regex"] - assert "[「」『』・･]" in regex, ( + table_spans = _expected_bmp_spans() + checked = [] + for rule in rules: + regex = rule.get("name_regex") + if not isinstance(regex, str): + continue + declared = _declared_spans(regex) + if not declared & table_spans: + continue + checked.append(rule["issue"]) + assert declared == table_spans, ( + f"{rule['issue']!r} declares {sorted(declared)}; expected " + f"{sorted(table_spans)}") + # the canonical rule plus both compound rules, today -- if this + # count drops, a hand copy fell out of discovery's reach + assert len(checked) >= 3, checked + # the delimiter compound's trigger set is its own decision surface + nickname = [r for r in rules + if "cjk-delimited-nickname" in r["issue"]] + assert len(nickname) == 1 + assert "[\u300C\u300D\u300E\u300F\u30FB\uFF65]" in nickname[0][ + "name_regex"] or "[「」『』・･]" in nickname[0]["name_regex"], ( "the compound rule's delimiter set changed; decide deliberately") - declared = { - (int(lo, 16), int(hi, 16)) - for lo, hi in re.findall(r"\\u([0-9A-Fa-f]{4})-\\u([0-9A-Fa-f]{4})", - regex)} - expected = {span - for spans in _policy._SCRIPT_RANGES.values() - for span in spans - if span[1] <= 0xFFFF} | {(0xFF65, 0xFF65)} - assert declared == expected, ( - f"compound rule's codepoint lookahead declares {sorted(declared)}; " - f"_SCRIPT_RANGES' BMP spans are {sorted(expected)}") def test_cjk_corpus_matches_the_case_table() -> None: diff --git a/tools/differential/build_cjk_corpus.py b/tools/differential/build_cjk_corpus.py index 2858546..4e0c8f6 100644 --- a/tools/differential/build_cjk_corpus.py +++ b/tools/differential/build_cjk_corpus.py @@ -15,8 +15,9 @@ Selection is by the shipped table (nameparser._policy._SCRIPT_RANGES, through the same _script_matcher the parser uses), so a script added to the table widens the harvest on the next regeneration, and a case -row added for future CJK work (#298's 间隔号 forms, say) enters the -corpus by being written down in the table everyone already reviews. +row added for new CJK work enters the corpus by being written down in +the table everyone already reviews (#298's 间隔号 forms arrived +exactly this way). Regenerate after editing CJK case rows: diff --git a/tools/differential/corpus_cjk.jsonl b/tools/differential/corpus_cjk.jsonl index a6ce564..8d9381e 100644 --- a/tools/differential/corpus_cjk.jsonl +++ b/tools/differential/corpus_cjk.jsonl @@ -6,10 +6,14 @@ "みなみ" "マイケル" "マイケル ジャクソン" +"マイケル·ジャクソン" "マイケル・ジャクソン" "佐々木 太郎" "司马相如" "夏侯惇" +"威廉·莎士比亚" +"威廉·莎士比亚, PhD" +"威廉・莎士比亚" "山田 エミ" "山田 太郎 (マイケル・ジャクソン)" "山田「タロ」太郎" @@ -17,9 +21,11 @@ "毛 泽东" "毛 김" "毛泽东" +"王·Smith" "田中『ハナ』花子" "諸葛亮" "阿明" +"马丁·路德·金" "高橋 みなみ" "高橋みなみ" "高橋・一郎" @@ -29,3 +35,4 @@ "남궁" "남궁민수" "남궁민수, 지훈" +"마이클·잭슨" diff --git a/tools/differential/expected_changes.toml b/tools/differential/expected_changes.toml index cf03b8b..74a5c69 100644 --- a/tools/differential/expected_changes.toml +++ b/tools/differential/expected_changes.toml @@ -7,7 +7,7 @@ # ahead of `fields`-only ones before matching. [[change]] -issue = "fix(#271/#272) native-script CJK: family-first order, hangul segmentation, the kana license and the nakaguro" +issue = "fix(#271/#272/#298) native-script CJK: family-first order, hangul segmentation, the kana license and the dots" # '毛 泽东', '김민준': script_orders flips first/last for a name written # wholly in Han or Hangul, and the Korean surnames that now ship as # default vocabulary additionally split an unspaced hangul token into @@ -33,16 +33,22 @@ issue = "fix(#271/#272) native-script CJK: family-first order, hangul segmentati # modern name uses, so _SCRIPT_RANGES does not list it either. # # U+FF65 is the one span here that _SCRIPT_RANGES does NOT have, and -# it is deliberate: the halfwidth middle dot separates tokens like its -# fullwidth twin, so 'ﾏｲｹﾙ･ｼﾞｬｸｿﾝ' splits where 1.4 left one token, -# but halfwidth kana is excluded from CLASSIFICATION on purpose (a -# separate normalization problem). The rest of the halfwidth block -# (U+FF66-U+FF9F) is left out for the usual tightness reason, and it -# was measured rather than assumed: a dotless halfwidth name such as -# 'ﾏｲｹﾙ ｼﾞｬｸｿﾝ' is byte-identical on both sides, so covering the -# block would pre-excuse a future regression on a shape that is -# parity today. The dot is the whole mechanism; the dot is the whole -# span. +# it is deliberate. The halfwidth middle dot separates tokens like +# its fullwidth twin, so 'ﾏｲｹﾙ･ｼﾞｬｸｿﾝ' splits where 1.4 left one +# token, but halfwidth kana is excluded from CLASSIFICATION on +# purpose (a separate normalization problem). The rest of the +# halfwidth block (U+FF66-U+FF9F) is left out for the usual tightness +# reason, and it was measured rather than assumed: a dotless +# halfwidth name such as 'ﾏｲｹﾙ ｼﾞｬｸｿﾝ' is byte-identical on both +# sides, so covering the block would pre-excuse a future regression +# on a shape that is parity today. U+00B7, the context-sensitive +# 间隔号 (#298), deliberately gets NO span even though it too changes +# parses: its flank guard divides only between classified-script +# characters, so every name it can change already matches this class +# through a flanking character -- and a B7 span's only actual effect +# would be letting this rule claim first/middle/last diffs on +# punt-volat Latin names ('Gal·la Marcet'), pre-excusing a regression +# on exactly the input class the guard exists to protect. # # corpus_cjk.jsonl (#295) exists so this rule fires in every real # run: build_cjk_corpus.py harvests every CJK-bearing text in @@ -152,6 +158,25 @@ issue = "fix(cjk-delimited-nickname) delimiter recognition compounds with the CJ name_regex = "(?s)(?=.*[「」『』・･])(?=.*[\\u3005-\\u3006\\u3040-\\u309F\\u30A0-\\u30FF\\u3400-\\u4DBF\\u4E00-\\u9FFF\\uF900-\\uFAFF\\uAC00-\\uD7A3\\uFF65-\\uFF65])" fields = ["first", "last", "nickname"] +[[change]] +issue = "fix(cjk-comma-compound) comma routing compounds with the CJK order flip" +# '威廉·莎士比亚, PhD': one name, two intended changes at once -- the +# post-comma lone-piece routing (fix(comma-family): v1 read PhD as +# title; 2.x routes it to suffix) and the CJK source-order/family- +# first movement in first/middle/last on the pre-comma name. Neither +# single-change rule may claim the union: comma-family's fields +# exclude last, the CJK rule's exclude title/suffix -- each on +# purpose, so a lone regression in the other's fields stays loud. +# Both lookaheads are required, by the same reasoning as the +# delimiter compound above: a comma alone matches every Latin +# 'Smith, Jr.' in the corpus, and the classified-codepoint lookahead +# confines the rule to names the CJK behaviors can actually reach. +# The class is the same hand copy of _SCRIPT_RANGES the rules above +# carry, pinned by the same sync test. The slug avoids the literal +# #271/#272 substrings the canonical-rule pin selects by. +name_regex = "(?s)(?=.*,)(?=.*[\\u3005-\\u3006\\u3040-\\u309F\\u30A0-\\u30FF\\u3400-\\u4DBF\\u4E00-\\u9FFF\\uF900-\\uFAFF\\uAC00-\\uD7A3\\uFF65-\\uFF65])" +fields = ["first", "middle", "last", "title", "suffix"] + [[change]] issue = "feat(#269) non-Latin titles/conjunctions recognized" # 'г-н Иван Петров' (Cyrillic title), 'Хосе И Мария Сантос' (Cyrillic