Retighten the differential rule; pin the review-found gaps (#298)

The B7 sanctioned-extra span was a pure loss, removed: the flank
guard guarantees every name the dot can change contains a classified
character, so B7 names already matched the CJK rule through their
flanks -- the span's only actual effect was letting the rule claim
first/middle/last diffs on punt-volat Latin names (Gal·la Marcet),
pre-excusing a regression on exactly the input class the guard
protects. The sync test goes back to one sanctioned extra, and its
overlap loop now checks whole spans instead of discarding each
span's high end.

Three case rows close review-found gaps: the Korean transcription
마이클·잭슨 (마 is a listed census surname, and the spaced form
really does mis-split 마|이클 today -- the dot is what rescues it),
the suffix-comma composition 威廉·莎士比亚, PhD, and the half-flanked
王·Smith parity row. The suffix-comma row's corpus entry surfaced a
genuine compound diff -- comma routing (title to suffix) union the
CJK flip (first/last) -- that neither single-change rule may claim,
each excluding the other's fields on purpose; the new
fix(cjk-comma-compound) rule takes exactly that union, gated by the
same two-lookahead shape as the delimiter compound, and the compound
sync pin is parametrized over both slugs so neither span copy goes
unpinned.

Docs: usage.rst and migrate.rst state the per-flank judgment (a
hangul character beside a Han one qualifies), the state-global
breadth (a dot anywhere reads the whole name as a transcription
listing), and the half-flanked outcome (the undivided dot remains
part of the word); AGENTS.md's doctrine paragraph counts five
defaults and names the 间隔号 marker among them.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Author: derek73

AGENTS.md

@@ -75,7 +75,7 @@ logging.getLogger('HumanName').setLevel(logging.DEBUG)
 
 The library has two layers: `nameparser/config/` (data) and `nameparser/parser.py` (logic).
 
-**Design philosophy — positional and language-agnostic.** The parser assigns parts by *position* plus small sets of words that join to neighbors; it never detects language. A name's language can't be reliably inferred from Latin-script transliteration ("Ali" is Arabic or Italian; "Van"/"Della"/"Bin" are first names in some cultures, particles in others), so language-specific rules belong in opt-in `Constants` config, never global defaults. Many "wrong for language X" reports (#133, #150, #130, #85, #103, #146, #83) are irreducible ambiguities — e.g. `de Mesnil` (want last name) vs `Van Johnson` (want first name) are the same `[prefix][word]` shape. Before adding a rule, confirm it doesn't break the opposite case (run the full suite — Portuguese and "Van Johnson" tests are the usual canaries). **The one scoped exception (2.1, #271/#272): script-conditional behavior is permitted exactly where the SCRIPT ITSELF — not statistics about it — determines the convention.** The never-detect-language rule above is about Latin *transliteration*, where the signal genuinely is destroyed; native script is a different question, and it is answered per behavior rather than per script. Four defaults fall out of it. Wholly-Han, wholly-Hangul and kana-licensed names read family-first (`Policy.script_orders`) — no language detection needed, because zh and ja both write family-first in native script, so order cannot be misread even though the language is unknowable. Unspaced hangul splits into surname + given name (`config/surnames.py` ships the Korean census list as DEFAULT vocabulary) — nothing but Korean is written in hangul and the surnames are a closed census set, and the vocabulary is self-selecting besides: a hangul entry can only ever match hangul text. Hiragana licenses Japanese (#272) — a name whose characters stay inside Han∪kana while carrying at least one kana cannot be Chinese (the kana rules it out) and is not a transcription (foreign names are transcribed in katakana ALONE, マイケル has no kanji), so 高橋みなみ and 山田 エミ read family-first too; mechanically they resolve to the HIRAGANA entry, the license's carrier key. PURE katakana is excluded and keeps the positional default: マイケル・ジャクソン is a transcribed foreign name in its source order. The nakaguro belongs to the same doctrine but is decided a layer down: U+30FB and its halfwidth twin U+FF65 separate tokens like whitespace, unconditionally and in tokenize, so neither policy opt-out (`script_orders={}`, `segment_scripts=()`) reaches it — the codepoints are CJK-only and appear in no other script's names, which is what licenses a tokenize-level rule where U+00B7 (also the Catalan punt volat, interior to Gal·la) needs the flanked-by-classified-script guard `_tokenize_region` gives it (#298). Han segmentation stays OPT-IN (`locales.ZH` for Chinese, `locales.JA` for Japanese) — a zh surname list corrupts Japanese kanji names, since 高 is a common Chinese surname and 高橋一郎 would split 高+橋一郎 where the correct reading is 高橋+一郎; no surname list divides a kanji name at all, so `locales.JA` activates the stage and a pluggable `Parser(segmenter=...)` does the dividing. Latin-script input is never touched by any of this: "Kim Min-jun" is genuinely order-ambiguous and stays governed by `name_order` and opt-in packs. Before adding a script-conditional rule, work out which of the three it is — certain, certain for this one behavior only, or a statistical guess wearing a script's clothes.
+**Design philosophy — positional and language-agnostic.** The parser assigns parts by *position* plus small sets of words that join to neighbors; it never detects language. A name's language can't be reliably inferred from Latin-script transliteration ("Ali" is Arabic or Italian; "Van"/"Della"/"Bin" are first names in some cultures, particles in others), so language-specific rules belong in opt-in `Constants` config, never global defaults. Many "wrong for language X" reports (#133, #150, #130, #85, #103, #146, #83) are irreducible ambiguities — e.g. `de Mesnil` (want last name) vs `Van Johnson` (want first name) are the same `[prefix][word]` shape. Before adding a rule, confirm it doesn't break the opposite case (run the full suite — Portuguese and "Van Johnson" tests are the usual canaries). **The one scoped exception (2.1, #271/#272): script-conditional behavior is permitted exactly where the SCRIPT ITSELF — not statistics about it — determines the convention.** The never-detect-language rule above is about Latin *transliteration*, where the signal genuinely is destroyed; native script is a different question, and it is answered per behavior rather than per script. Five defaults fall out of it. Wholly-Han, wholly-Hangul and kana-licensed names read family-first (`Policy.script_orders`) — no language detection needed, because zh and ja both write family-first in native script, so order cannot be misread even though the language is unknowable. Unspaced hangul splits into surname + given name (`config/surnames.py` ships the Korean census list as DEFAULT vocabulary) — nothing but Korean is written in hangul and the surnames are a closed census set, and the vocabulary is self-selecting besides: a hangul entry can only ever match hangul text. Hiragana licenses Japanese (#272) — a name whose characters stay inside Han∪kana while carrying at least one kana cannot be Chinese (the kana rules it out) and is not a transcription (foreign names are transcribed in katakana ALONE, マイケル has no kanji), so 高橋みなみ and 山田 エミ read family-first too; mechanically they resolve to the HIRAGANA entry, the license's carrier key. PURE katakana is excluded and keeps the positional default: マイケル・ジャクソン is a transcribed foreign name in its source order. And the 间隔号 U+00B7 (#298) is the transcription marker for scripts that HAVE no transcription script: a name it divides (威廉·莎士比亚 — flanked by classified characters on both sides, so Catalan's Gal·la is untouched) keeps its source order and never segments — the orthography names the convention, exactly as pure katakana does, with the divider carrying the signal instead of the script. The nakaguro belongs to the same doctrine but is decided a layer down: U+30FB and its halfwidth twin U+FF65 separate tokens like whitespace, unconditionally and in tokenize, so neither policy opt-out (`script_orders={}`, `segment_scripts=()`) reaches it — the codepoints are CJK-only and appear in no other script's names, which is what licenses a tokenize-level rule where U+00B7 (also the Catalan punt volat, interior to Gal·la) needs the flanked-by-classified-script guard `_tokenize_region` gives it (#298). Han segmentation stays OPT-IN (`locales.ZH` for Chinese, `locales.JA` for Japanese) — a zh surname list corrupts Japanese kanji names, since 高 is a common Chinese surname and 高橋一郎 would split 高+橋一郎 where the correct reading is 高橋+一郎; no surname list divides a kanji name at all, so `locales.JA` activates the stage and a pluggable `Parser(segmenter=...)` does the dividing. Latin-script input is never touched by any of this: "Kim Min-jun" is genuinely order-ambiguous and stays governed by `name_order` and opt-in packs. Before adding a script-conditional rule, work out which of the three it is — certain, certain for this one behavior only, or a statistical guess wearing a script's clothes.
 
 ### Configuration layer (`nameparser/config/`)
 

docs/migrate.rst

@@ -431,8 +431,9 @@ surface is frozen for 2.x — so the way out is the 2.0 API:
 restores 1.4's reading of every shape above that turns on order or
 splitting. The middle dots are the exception: both the katakana dot
 and the Chinese interpunct ``·`` (U+00B7, dividing a transcription
-like ``威廉·莎士比亚``, and a separator only between characters of a
-classified script where the katakana dot is unconditional) are decided
+like ``威廉·莎士比亚``, and a separator only between classified-script
+characters — each side judged on its own — where the katakana dot is
+unconditional) are decided
 in tokenization rather than by policy, so a name written with either
 still divides at the dot, and still renders with a space, whatever
 those two fields are set to.

docs/usage.rst

@@ -193,15 +193,18 @@ punctuation: a foreign name transcribed into Han characters keeps its
 source order and divides its parts with the 间隔号, the interpunct
 ``·`` (U+00B7) — 威廉·莎士比亚 is William Shakespeare, given name
 first. The dot itself is the marker, so nameparser reads U+00B7 as a
-token separator when it sits between characters of a classified
-script, and a name it divides keeps the order it was written in and is
+token separator when it sits between classified-script characters
+(each side judged on its own — a hangul character beside a Han one
+qualifies), and a dot anywhere in the name reads the whole name as a
+transcription listing: it keeps the order it was written in and is
 never segmented — the role pure katakana plays for Japanese
 transcriptions, played here by the divider instead of the script.
-Because only a classified script on both sides makes it a divider, the
-same codepoint interior to a Latin-script name — the Catalan punt
-volat in ``Gal·la`` — is untouched. The Japanese middle dot ・
-(covered next) is a different mark carrying a different convention,
-and keeps its own reading.
+Because only classified characters on both sides make it a divider,
+the same codepoint interior to a Latin-script name — the Catalan punt
+volat in ``Gal·la`` — is untouched, and a dot with a classified
+character on just one side (``王·Smith``) stays part of the word
+undivided. The Japanese middle dot ・ (covered next) is a different
+mark carrying a different convention, and keeps its own reading.
 
 Japanese
 ~~~~~~~~~

tests/v2/cases.py

@@ -688,4 +688,23 @@ def __post_init__(self) -> None:
          notes="马 is a listed zh surname, but a 间隔号-divided name "
                "is a transcription: the dot gates segmentation off, "
                "so 马丁 stays whole and the positional read stands"),
+    Case("ko_interpunct_transcription_source_order", "마이클·잭슨",
+         {"given": "마이클", "family": "잭슨"},
+         classification="fix(#298)",
+         notes="Korean writes transcribed foreign names with the "
+               "interpunct too: the dot suppresses the hangul "
+               "family-first entry AND the default segmentation -- 마 "
+               "is a listed census surname, and the spaced form "
+               "마이클 잭슨 really does mis-split 마|이클 today, so "
+               "the dot is what rescues the ko transcription"),
+    Case("zh_interpunct_with_suffix_comma", "威廉·莎士比亚, PhD",
+         {"given": "威廉", "family": "莎士比亚", "suffix": "PhD"},
+         classification="fix(#298)",
+         notes="the transcription reading composes with a suffix "
+               "comma: the marker is structure-independent"),
+    Case("zh_interpunct_half_flanked_stays", "王·Smith",
+         {"given": "王·Smith"},
+         notes="one classified neighbor is not enough: the guard "
+               "requires both, so the undivided dot remains part of "
+               "the word -- declining, not deciding"),
 )

tests/v2/test_regex_sync.py

@@ -192,20 +192,22 @@ def test_differential_cjk_rule_matches_the_script_ranges() -> None:
     it, see the comment there -- so the comparison runs over the BMP
     spans only.
 
-    The rule is also WIDER than the table by exactly two spans, which
+    The rule is also WIDER than the table by exactly one span, which
     the equality has to know about or it would just fail forever. The
     halfwidth middle dot U+FF65 changes parses without being
     classified as anything: tokenize separates on it, so a halfwidth
     transcription splits where 1.4 kept one token, while halfwidth
     kana stays out of _SCRIPT_RANGES on purpose. U+00B7 -- the
-    context-sensitive 间隔号 (#298) -- is the same situation: it
-    divides between classified characters and pins source order,
-    changing parses, yet doubles as Catalan's punt volat and so can
-    never itself be classified. Naming those spans here rather than
-    relaxing the comparison to a subset check is what keeps the pin
-    honest in both directions: an unsanctioned source of divergence
-    still fails, and each sanctioned difference has to be written
-    down to exist.
+    context-sensitive 间隔号 (#298) -- also changes parses yet is
+    deliberately NOT an extra: its flank guard means every name it
+    can change matches the class through a flanking character
+    already, and a B7 span's only actual effect would be letting the
+    rule claim diffs on punt-volat Latin names (Gal·la), pre-excusing
+    a regression on exactly the guarded class. Naming the sanctioned
+    span here rather than relaxing the comparison to a subset check
+    is what keeps the pin honest in both directions: an unsanctioned
+    source of divergence still fails, and each sanctioned difference
+    has to be written down to exist.
     """
     toml_path = (Path(__file__).parents[2] / "tools" / "differential"
                  / "expected_changes.toml")
@@ -227,13 +229,13 @@ def test_differential_cjk_rule_matches_the_script_ranges() -> None:
         for lo, hi in re.findall(r"\\u([0-9A-Fa-f]{4})-\\u([0-9A-Fa-f]{4})",
                                  matched[0]["name_regex"])}
     # the two spans the rule carries that no Script claims (see above)
-    sanctioned_extras = {(0xFF65, 0xFF65), (0xB7, 0xB7)}
-    for extra, _ in sorted(sanctioned_extras):
-        assert not any(lo <= extra <= hi
+    sanctioned_extras = {(0xFF65, 0xFF65)}
+    for xlo, xhi in sanctioned_extras:
+        assert not any(lo <= xhi and xlo <= hi
                        for spans in _policy._SCRIPT_RANGES.values()
                        for lo, hi in spans), (
-            f"U+{extra:04X} is classified now; drop it from the "
-            "sanctioned extras")
+            f"U+{xlo:04X}-U+{xhi:04X} is classified now; drop it "
+            "from the sanctioned extras")
     expected = {span
                 for spans in _policy._SCRIPT_RANGES.values()
                 for span in spans
@@ -243,29 +245,35 @@ def test_differential_cjk_rule_matches_the_script_ranges() -> None:
         f"_SCRIPT_RANGES' BMP spans are {sorted(expected)}")
 
 
-def test_differential_compound_rule_matches_the_script_ranges() -> None:
-    """The fix(cjk-delimited-nickname) rule (#295) carries the SECOND
-    hand copy of the script spans in the toml: its require-a-classified-
-    codepoint lookahead exists so the delimiter set alone cannot claim a
-    Latin name's first/last regression ('John 「Jack」 Kennedy' -- the
-    corner brackets sit outside every classified span). Pin that copy to
-    the table exactly as the canonical CJK rule's is, and pin the
-    delimiter set itself, so widening either (to ASCII quotes, say) is
-    an explicit decision here rather than a silent absorption change.
+@pytest.mark.parametrize("slug", ["cjk-delimited-nickname",
+                                  "cjk-comma-compound"])
+def test_differential_compound_rules_match_the_script_ranges(
+        slug: str) -> None:
+    """The compound rules carry FURTHER hand copies of the script
+    spans in the toml: each one's require-a-classified-codepoint
+    lookahead exists so its trigger set alone (delimiters; a comma)
+    cannot claim a Latin name's regression ('John 「Jack」 Kennedy',
+    'Smith, Jr.' -- neither contains a classified character). Pin each
+    copy to the table exactly as the canonical CJK rule's is, plus the
+    nickname rule's delimiter set itself, so widening either is an
+    explicit decision here rather than a silent absorption change.
 
     Selection note for future rule authors: the canonical-CJK-rule pin
     above selects by the literal '#271'/'#272' substrings and asserts
-    uniqueness -- this rule's slug avoids them on purpose, and any new
-    rule's must too.
+    uniqueness -- the compound slugs avoid them on purpose, and any
+    new rule's must too (and its slug joins this parametrize list, or
+    its span copy goes unpinned).
     """
     toml_path = (Path(__file__).parents[2] / "tools" / "differential"
                  / "expected_changes.toml")
     rules = tomllib.loads(toml_path.read_text())["change"]
-    matched = [r for r in rules if "cjk-delimited-nickname" in r["issue"]]
+    matched = [r for r in rules if slug in r["issue"]]
     assert len(matched) == 1
     regex = matched[0]["name_regex"]
-    assert "[「」『』・･]" in regex, (
-        "the compound rule's delimiter set changed; decide deliberately")
+    if slug == "cjk-delimited-nickname":
+        assert "[「」『』・･]" in regex, (
+            "the compound rule's delimiter set changed; decide "
+            "deliberately")
     declared = {
         (int(lo, 16), int(hi, 16))
         for lo, hi in re.findall(r"\\u([0-9A-Fa-f]{4})-\\u([0-9A-Fa-f]{4})",
@@ -275,7 +283,7 @@ def test_differential_compound_rule_matches_the_script_ranges() -> None:
                 for span in spans
                 if span[1] <= 0xFFFF} | {(0xFF65, 0xFF65)}
     assert declared == expected, (
-        f"compound rule's codepoint lookahead declares {sorted(declared)}; "
+        f"{slug}'s codepoint lookahead declares {sorted(declared)}; "
         f"_SCRIPT_RANGES' BMP spans are {sorted(expected)}")
 
 

tools/differential/corpus_cjk.jsonl

@@ -12,6 +12,7 @@
 "司马相如"
 "夏侯惇"
 "威廉·莎士比亚"
+"威廉·莎士比亚, PhD"
 "威廉・莎士比亚"
 "山田 エミ"
 "山田 太郎 (マイケル・ジャクソン)"
@@ -20,6 +21,7 @@
 "毛 泽东"
 "毛 김"
 "毛泽东"
+"王·Smith"
 "田中『ハナ』花子"
 "諸葛亮"
 "阿明"
@@ -33,3 +35,4 @@
 "남궁"
 "남궁민수"
 "남궁민수, 지훈"
+"마이클·잭슨"