From bcb443badae8a830ae32446248e18761de82bbf6 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Thu, 30 Jul 2026 19:07:12 -0700 Subject: [PATCH 1/2] Recognize spaced CJK postnominal honorifics as suffixes (#307) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CJK honorifics and degrees FOLLOW the name, and under the 2.1 family-first defaults an unrecognized one confidently became a name part: 王小明 先生 read given=先生, 김민준 씨 put the honorific in middle. Thirteen curated entries join SUFFIX_NOT_ACRONYMS beside the Hebrew post-nominal block -- ko 씨/박사/선생님/교수님/군/양 (standardly spaced in Korean orthography, the strongest case), zh 先生/女士/小姐/博士/教授, ja 様/氏, with 先生/博士/教授 shared Han serving both languages. Self-selecting like the Korean surnames: a Han or hangul entry can only ever match CJK text. Vetted per the file's мл/ст standard; 氏 and the ko single-syllable pair 군/양 are argued in the comment, and the trailing-only position gate keeps surname-position 양 (양 미선) untouched. The order interaction needed no mechanism: suffix classification runs after script_segment and before the positional read, so the rows pass on vocabulary alone, through both the core and facade runners. Review found the scope claim wrong in the good direction: glued HANGUL honorifics are reached too, because default segmentation splits 김씨 into 김 + 씨 before suffix classification -- a partial delivery of #308 that falls out of stage order (glued Han needs the zh pack the same way; glued kana stays #308). Docs and the config comment state it, and case rows pin the glued and post-comma shapes alongside the four spaced ones. Differential: the new fix(cjk-honorific-suffix) rule takes the compound diffs (honorific recognition x CJK flip), keyed on the honorific alternation per the maiden-marker precedent and anchored to a WHOLE trailing token -- a glued ending must not match, or a genuine given name ending in 양/군 could have a real regression absorbed. The alternation is a hand copy of the config's CJK entries, pinned by a sync test that DERIVES the expected set from SUFFIX_NOT_ACRONYMS by script membership. Glued-hangul diffs fall to fields-only suffix-routing by their shape, noted in the rule. Harness: 698 names, 56 intentional, 0 unexplained. Closes #307 Co-Authored-By: Claude Fable 5 --- docs/release_log.rst | 1 + docs/usage.rst | 9 ++++++ nameparser/config/suffixes.py | 30 +++++++++++++++++ tests/v2/cases.py | 41 ++++++++++++++++++++++++ tests/v2/test_regex_sync.py | 30 +++++++++++++++++ tools/differential/corpus_cjk.jsonl | 6 ++++ tools/differential/expected_changes.toml | 22 +++++++++++++ 7 files changed, 139 insertions(+) diff --git a/docs/release_log.rst b/docs/release_log.rst index afd6221..5d20936 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -25,6 +25,7 @@ Release Log - Fix names containing 〆 (U+3006, the shime mark that opens Japanese surnames like 〆木 and 〆谷) parsing given-first: the script classifier now counts 〆 as Han, extending the 々 entry the table already carries, and going a step further than it — 々 is Script=Han and merely outside the ideograph blocks, while 〆 is Script=Common — so these names take the East Asian family-first reading like any other wholly-Han name. ``Policy(script_orders={})`` restores the positional reading for these names exactly as for other wholly-Han names (#303) - Fix the katakana middle dot ``・`` (U+30FB, and its halfwidth twin U+FF65) being read as part of a name rather than as the divider it is: it now separates tokens exactly as a space does. A transcribed foreign name therefore divides into its parts and, being wholly katakana, keeps its source order — ``"マイケル・ジャクソン"`` gives given ``マイケル``, family ``ジャクソン``, where 1.x left the whole string in ``first`` — while a kanji pair written the same way takes the family-first rule (``"高橋・一郎"`` → family ``高橋``). Native Japanese names never contain this character and no other script's names use it, so the separation is unconditional, which also means it is **not** covered by the two policy opt-outs. Rendering follows, the way the Korean split's does: the dot comes back as a space, so ``str(HumanName("マイケル・ジャクソン"))`` is now ``"マイケル ジャクソン"``, and that reaches delimited content too — the nickname in ``"山田 太郎 (マイケル・ジャクソン)"`` renders ``"マイケル ジャクソン"``. The Chinese interpunct ``·`` is not an unconditional separator like these two: U+00B7 is also the Catalan punt volat and appears inside legitimate names (``Gal·la``), so it divides only between classified-script characters — the transcription treatment it marks is #298's (#272) - Fix 间隔号-divided transcriptions parsing as one unsplit token: U+00B7 — the interpunct Chinese text divides a transcribed foreign name with, ``威廉·莎士比亚`` for William Shakespeare — is now a token separator between characters of a classified script, and a name it divides keeps its source order and is never segmented: the dot is the transcription marker, playing the role pure katakana plays in the kana license. It divides ONLY between classified-script characters, so the Catalan punt volat interior to names like ``Gal·la`` is untouched. The Japanese nakaguro is deliberately not a transcription marker — ``高橋・一郎`` is roster formatting, 姓・名, and keeps its family-first reading — so a transcription typed with the wrong dot (``威廉・莎士比亚``) reads by the convention of the codepoint it was typed with; like the spaced form, only the Chinese dot rescues it. Rendering stays space-joined (``str(HumanName("威廉·莎士比亚"))`` is ``"威廉 莎士比亚"``); a custom ``string_format`` such as ``"{first}·{last}"`` reinstates the dot (#298) + - Fix spaced CJK postnominal honorifics parsing as name parts: 씨, 박사, 선생님, 교수님, 군, 양 (Korean — standardly written as their own token), 先生, 女士, 小姐, 博士, 教授 (Chinese), and 様, 氏 (Japanese) now route to ``suffix``, so ``王小明 先生`` reads family ``王小明`` where the family-first default had confidently made 先生 the given name. Whole-token matching — which reaches glued hangul too, since default segmentation splits ``김씨`` into 김 + 씨 before the suffix is classified (and glued Han the same way under ``locales.ZH``); a glued kana honorific (``山田太郎様``, ``田中さん``) stays out of reach, tracked as #308 (closes #307) - Fix NFD-decomposed input missing the East Asian defaults entirely: script classification now normalizes to NFC before deciding, so a Korean or Japanese name typed on macOS — where decomposed text is routine — gets the same order rule as its composed twin, which it silently did not before. Segmentation MATCHING deliberately stays raw, so an unspaced NFD hangul name is ordered correctly but not split, rather than being split in the wrong place. One gotcha worth stating plainly: parse output preserves the encoding it was given, so for NFD input ``name.family == "김"`` is ``False`` even though it is the same name — compare NFC-normalized text when comparing across encodings (#272) * 2.0.0 - July 27, 2026 diff --git a/docs/usage.rst b/docs/usage.rst index a77a40f..707a492 100644 --- a/docs/usage.rst +++ b/docs/usage.rst @@ -295,6 +295,15 @@ Chinese dot rescues the source order. And a comma disables the script behaviors entirely, on the reasoning ``name_order`` already follows: whoever wrote the comma has already said where the family name ends. +Honorifics and degrees follow a CJK name, and the spaced forms are +recognized as suffixes — ``王小明 先生`` reads family 王小明 with 先生 +in ``suffix``, and Korean's standardly-spaced 씨 routes the same way. +Glued hangul forms are reached too — default segmentation splits +김씨 into 김 + 씨 before the suffix is classified — and glued Han +forms the same way under the zh pack's segmentation. A glued kana +honorific (山田太郎様, 田中さん) is beyond whole-token matching and +currently stays part of the name. + A division the parser had to choose is reported rather than hidden. When an unspaced name has more than one vocabulary-supported split — ``남궁민수`` is 남궁 + 민수 by the two-syllable surname but 남 + 궁민수 diff --git a/nameparser/config/suffixes.py b/nameparser/config/suffixes.py index 43d1f4b..c56bbb2 100644 --- a/nameparser/config/suffixes.py +++ b/nameparser/config/suffixes.py @@ -36,6 +36,36 @@ 'ז״ל', # same, U+05F4 gershayim 'שליט"א', # honorific for a living rabbi, ASCII quote 'שליט״א', # same, U+05F4 gershayim + + # #307: CJK postnominal honorifics and degrees, whole-token + # matched. That reaches the spaced forms directly -- and glued + # HANGUL forms too (김씨), because default segmentation splits the + # token before suffix classification runs; glued Han forms need + # the zh pack's segmentation the same way (王先生), and glued kana + # (山田太郎様, 田中さん) stays out of reach, tracked as #308. + # Self-selecting like the Korean + # surnames: a Han or hangul entry can only ever match CJK text. + # Vetting per the мл/ст standard above: none of these is a given + # name or surname in trailing whole-token position -- the two + # worth naming are 氏 (a rare Japanese surname reading exists, but + # a bare trailing 氏 after a name is the news-style honorific, and + # a 氏-surnamed person writes it FIRST) and the single-syllable + # ko pair 군/양, which trail a name only as honorifics (a + # single-syllable GIVEN name in final position is already + # vanishingly rare, and neither 군 nor 양 is one in practice). + '씨', # ko Mr./Ms. -- standardly spaced in Korean orthography + '박사', # ko doctorate holder ("Dr.") + '선생님', # ko teacher/respected elder + '교수님', # ko professor (honorific form) + '군', # ko young man ("Master") + '양', # ko young woman ("Miss") + '先生', # zh Mr. / ja teacher-master -- honorific in both + '女士', # zh Ms./Madam + '小姐', # zh Miss + '博士', # zh+ja doctorate holder, shared Han + '教授', # zh+ja professor, shared Han + '様', # ja formal Mr./Ms. (the mail-addressing honorific) + '氏', # ja news-style Mr. (田中氏) } """ diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 753485d..902457f 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -707,4 +707,45 @@ def __post_init__(self) -> None: notes="one classified neighbor is not enough: the guard " "requires both, so the undivided dot remains part of " "the word -- declining, not deciding"), + Case("zh_honorific_suffix_spaced", "王小明 先生", + {"family": "王小明", "suffix": "先生"}, + classification="fix(#307)", + notes="CJK honorifics FOLLOW the name; a spaced 先生 (Mr.) is " + "a suffix, and recognizing it must come before the " + "family-first order hands it a role -- unrecognized it " + "read as the GIVEN name under the 2.1 defaults"), + Case("ko_honorific_ssi", "김민준 씨", + {"family": "김", "given": "민준", "suffix": "씨"}, + classification="fix(#307)", + notes="Korean orthography standardly SPACES 씨, so the " + "whole-token suffix machinery reaches it; the name " + "still segments (suffix classification runs after the " + "script_segment stage, which only ever saw 김민준)"), + Case("ko_degree_baksa", "김민준 박사", + {"family": "김", "given": "민준", "suffix": "박사"}, + classification="fix(#307)", + notes="박사 (doctorate) is the ko analogue of a trailing " + "PhD: fix(suffix-routing)'s two-token shape, one " + "script over"), + Case("ja_sama_spaced", "田中 太郎 様", + {"family": "田中", "given": "太郎", "suffix": "様"}, + classification="fix(#307)", + notes="the spaced 様 of forms and databases; the glued " + "mail-addressing form 山田太郎様 is #308's mechanism, " + "out of reach of whole-token matching"), + Case("ko_honorific_glued_via_segmentation", "김씨", + {"family": "김", "suffix": "씨"}, + classification="fix(#307)", + notes="glued HANGUL honorifics are reached after all: " + "default segmentation splits 김씨 into 김 + 씨 BEFORE " + "suffix classification, manufacturing the trailing " + "whole token -- a partial delivery of #308 that falls " + "out of stage order, hangul-only (glued Han needs the " + "zh pack; glued kana stays #308)"), + Case("ko_honorific_after_comma", "김민준, 씨", + {"family": "김민준", "suffix": "씨"}, + classification="fix(#307)", + notes="the post-comma lenient gate admits the honorific too; " + "the comma disables segmentation per the comma " + "doctrine, so 김민준 stays whole"), ) diff --git a/tests/v2/test_regex_sync.py b/tests/v2/test_regex_sync.py index 209c759..7d314ef 100644 --- a/tests/v2/test_regex_sync.py +++ b/tests/v2/test_regex_sync.py @@ -334,3 +334,33 @@ def test_cjk_corpus_matches_the_case_table() -> None: assert checked_in == module.selected_names(), ( "corpus_cjk.jsonl is stale: regenerate with " "`uv run python tools/differential/build_cjk_corpus.py`") + + +def test_differential_honorific_rule_matches_the_suffix_vocabulary() -> None: + """The fix(cjk-honorific-suffix) rule's alternation is a hand copy + of SUFFIX_NOT_ACRONYMS' CJK entries (#307) -- the toml cannot + import them. The expected set is DERIVED from the config by script + membership (a classified codepoint anywhere in the entry), so + adding a CJK honorific without widening the rule -- or widening + the rule with something the vocabulary does not ship -- fails + here. The span-bearing pins above skip this rule on purpose: its + trigger is the alternation, not a character class. + """ + from nameparser.config.suffixes import SUFFIX_NOT_ACRONYMS + + toml_path = (Path(__file__).parents[2] / "tools" / "differential" + / "expected_changes.toml") + rules = tomllib.loads(toml_path.read_text())["change"] + matched = [r for r in rules if "cjk-honorific-suffix" in r["issue"]] + assert len(matched) == 1 + regex = matched[0]["name_regex"] + prefix = "(?:^| )(?:" + assert regex.startswith(prefix) and regex.endswith(")$"), ( + "the honorific rule's shape changed; update this parser") + declared = set(regex[len(prefix):-2].split("|")) + has_classified = _policy._script_matcher(*_policy._SCRIPT_RANGES) + expected = {entry for entry in SUFFIX_NOT_ACRONYMS + if has_classified(entry)} + assert declared == expected, ( + f"rule declares {sorted(declared)}; the config's CJK suffix " + f"entries are {sorted(expected)}") diff --git a/tools/differential/corpus_cjk.jsonl b/tools/differential/corpus_cjk.jsonl index 8d9381e..8f2428e 100644 --- a/tools/differential/corpus_cjk.jsonl +++ b/tools/differential/corpus_cjk.jsonl @@ -22,6 +22,8 @@ "毛 김" "毛泽东" "王·Smith" +"王小明 先生" +"田中 太郎 様" "田中『ハナ』花子" "諸葛亮" "阿明" @@ -32,6 +34,10 @@ "高橋一郎" "김 민준" "김민준" +"김민준 박사" +"김민준 씨" +"김민준, 씨" +"김씨" "남궁" "남궁민수" "남궁민수, 지훈" diff --git a/tools/differential/expected_changes.toml b/tools/differential/expected_changes.toml index 74a5c69..e19fb0a 100644 --- a/tools/differential/expected_changes.toml +++ b/tools/differential/expected_changes.toml @@ -177,6 +177,28 @@ issue = "fix(cjk-comma-compound) comma routing compounds with the CJK order flip name_regex = "(?s)(?=.*,)(?=.*[\\u3005-\\u3006\\u3040-\\u309F\\u30A0-\\u30FF\\u3400-\\u4DBF\\u4E00-\\u9FFF\\uF900-\\uFAFF\\uAC00-\\uD7A3\\uFF65-\\uFF65])" fields = ["first", "middle", "last", "title", "suffix"] +[[change]] +issue = "fix(cjk-honorific-suffix) postnominal honorifics recognized, compounding with the CJK order flip" +# '王小明 先生', '김민준 씨', '田中 太郎 様': #307 ships the spaced CJK +# honorifics as suffix vocabulary, so a trailing 先生/씨/様 moves to +# `suffix` where 1.4 read it as a name part -- and the remaining +# wholly-CJK name simultaneously takes the family-first flip, so a +# three-token name moves `middle` too, past the fields-only +# suffix-routing rule. Keyed on the honorific alternation itself +# (the maiden-marker rule's pattern): the trigger tokens are wholly +# CJK characters, so the rule cannot touch a Latin name, and the +# alternation is a hand copy of SUFFIX_NOT_ACRONYMS' CJK entries -- +# pinned by tests/v2/test_regex_sync.py, which derives the expected +# set from the config by script membership. Anchored to a WHOLE +# trailing token ((?:^| )...$): a glued ending must not match, or a +# genuine given name ending in 양/군 (김지양) could have a real +# regression absorbed as intentional. Glued-hangul diffs (김씨 -- +# reached via segmentation) fall to the fields-only suffix-routing +# rule by their {first, suffix} shape, which is field-honest even +# though that rule's prose describes the two-token Latin case. +name_regex = "(?:^| )(?:씨|박사|선생님|교수님|군|양|先生|女士|小姐|博士|教授|様|氏)$" +fields = ["first", "middle", "last", "suffix"] + [[change]] issue = "feat(#269) non-Latin titles/conjunctions recognized" # 'г-н Иван Петров' (Cyrillic title), 'Хосе И Мария Сантос' (Cyrillic From 4ebb4c414323a67cd5df569cdfe95502d1d866f9 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Thu, 30 Jul 2026 20:02:27 -0700 Subject: [PATCH 2/2] Scope the glued-reach story to its real boundary; pin what the prose claims MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The PR review's convergent finding: 'glued hangul is reached' was overbroad in four places. Segmentation splits off ONE leading surname, so a glued honorific is reached in exactly the surname+honorific shape (김씨, 王先生 under zh) -- a glued honorific after a GIVEN name (김민준씨, 王小明先生), the more common written form, stays out of reach and belongs to #308. Every surface now states the boundary, and six new case rows pin it and the other prose-only claims: the two glued contrast pairs, whole-token matching (김지양 -- 지양 ENDS with 양 and must not peel, the parser-side twin of the differential anchor), surname-position 양 (양 미선 keeps family Yang), and the trailing honorific run (김민준 박사 씨 peels both, the Latin multi-suffix loop). The vetting comment gains its third named entry -- 博士 is an attested Japanese given name (ひろし, Hiroshi), argued rather than omitted -- plus the 양-surname overlap the 씨 argument covered only implicitly, and a sentence recording why bare hangul 선생/교수 deliberately do not ship. The differential rule comment corrects the 김씨 diff shape ({first, last, suffix}: v1 held the whole token in first, so last moves too), states the mixed-Latin shadow ('Wang Xiaoming 先生') as accepted rather than denied, and explains the anchor at the name-string level where it actually operates. The release log recovers the shared-Han fact (先生/博士/教授 serve Japanese too). Harness: 704 names, 62 intentional, 0 unexplained. Co-Authored-By: Claude Fable 5 --- docs/release_log.rst | 2 +- docs/usage.rst | 11 +++--- nameparser/config/suffixes.py | 39 +++++++++++-------- tests/v2/cases.py | 49 ++++++++++++++++++++---- tools/differential/corpus_cjk.jsonl | 6 +++ tools/differential/expected_changes.toml | 21 ++++++---- 6 files changed, 93 insertions(+), 35 deletions(-) diff --git a/docs/release_log.rst b/docs/release_log.rst index 5d20936..4be088b 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -25,7 +25,7 @@ Release Log - Fix names containing 〆 (U+3006, the shime mark that opens Japanese surnames like 〆木 and 〆谷) parsing given-first: the script classifier now counts 〆 as Han, extending the 々 entry the table already carries, and going a step further than it — 々 is Script=Han and merely outside the ideograph blocks, while 〆 is Script=Common — so these names take the East Asian family-first reading like any other wholly-Han name. ``Policy(script_orders={})`` restores the positional reading for these names exactly as for other wholly-Han names (#303) - Fix the katakana middle dot ``・`` (U+30FB, and its halfwidth twin U+FF65) being read as part of a name rather than as the divider it is: it now separates tokens exactly as a space does. A transcribed foreign name therefore divides into its parts and, being wholly katakana, keeps its source order — ``"マイケル・ジャクソン"`` gives given ``マイケル``, family ``ジャクソン``, where 1.x left the whole string in ``first`` — while a kanji pair written the same way takes the family-first rule (``"高橋・一郎"`` → family ``高橋``). Native Japanese names never contain this character and no other script's names use it, so the separation is unconditional, which also means it is **not** covered by the two policy opt-outs. Rendering follows, the way the Korean split's does: the dot comes back as a space, so ``str(HumanName("マイケル・ジャクソン"))`` is now ``"マイケル ジャクソン"``, and that reaches delimited content too — the nickname in ``"山田 太郎 (マイケル・ジャクソン)"`` renders ``"マイケル ジャクソン"``. The Chinese interpunct ``·`` is not an unconditional separator like these two: U+00B7 is also the Catalan punt volat and appears inside legitimate names (``Gal·la``), so it divides only between classified-script characters — the transcription treatment it marks is #298's (#272) - Fix 间隔号-divided transcriptions parsing as one unsplit token: U+00B7 — the interpunct Chinese text divides a transcribed foreign name with, ``威廉·莎士比亚`` for William Shakespeare — is now a token separator between characters of a classified script, and a name it divides keeps its source order and is never segmented: the dot is the transcription marker, playing the role pure katakana plays in the kana license. It divides ONLY between classified-script characters, so the Catalan punt volat interior to names like ``Gal·la`` is untouched. The Japanese nakaguro is deliberately not a transcription marker — ``高橋・一郎`` is roster formatting, 姓・名, and keeps its family-first reading — so a transcription typed with the wrong dot (``威廉・莎士比亚``) reads by the convention of the codepoint it was typed with; like the spaced form, only the Chinese dot rescues it. Rendering stays space-joined (``str(HumanName("威廉·莎士比亚"))`` is ``"威廉 莎士比亚"``); a custom ``string_format`` such as ``"{first}·{last}"`` reinstates the dot (#298) - - Fix spaced CJK postnominal honorifics parsing as name parts: 씨, 박사, 선생님, 교수님, 군, 양 (Korean — standardly written as their own token), 先生, 女士, 小姐, 博士, 教授 (Chinese), and 様, 氏 (Japanese) now route to ``suffix``, so ``王小明 先生`` reads family ``王小明`` where the family-first default had confidently made 先生 the given name. Whole-token matching — which reaches glued hangul too, since default segmentation splits ``김씨`` into 김 + 씨 before the suffix is classified (and glued Han the same way under ``locales.ZH``); a glued kana honorific (``山田太郎様``, ``田中さん``) stays out of reach, tracked as #308 (closes #307) + - Fix spaced CJK postnominal honorifics parsing as name parts: 씨, 박사, 선생님, 교수님, 군, 양 (Korean — standardly written as their own token), 先生, 女士, 小姐, 博士, 教授 (Chinese, with 先生/博士/教授 shared with Japanese), and 様, 氏 (Japanese) now route to ``suffix``, so ``王小明 先生`` reads family ``王小明`` where the family-first default had confidently made 先生 the given name. Whole-token matching — which also reaches a glued surname+honorific token, since segmentation splits off the surname first (``김씨`` reads family 김, suffix 씨; ``王先生`` the same under ``locales.ZH``); a glued honorific after a given name (``김민준씨``) and glued kana (``山田太郎様``) stay out of reach, tracked as #308 (closes #307) - Fix NFD-decomposed input missing the East Asian defaults entirely: script classification now normalizes to NFC before deciding, so a Korean or Japanese name typed on macOS — where decomposed text is routine — gets the same order rule as its composed twin, which it silently did not before. Segmentation MATCHING deliberately stays raw, so an unspaced NFD hangul name is ordered correctly but not split, rather than being split in the wrong place. One gotcha worth stating plainly: parse output preserves the encoding it was given, so for NFD input ``name.family == "김"`` is ``False`` even though it is the same name — compare NFC-normalized text when comparing across encodings (#272) * 2.0.0 - July 27, 2026 diff --git a/docs/usage.rst b/docs/usage.rst index 707a492..4e1b4aa 100644 --- a/docs/usage.rst +++ b/docs/usage.rst @@ -298,11 +298,12 @@ wrote the comma has already said where the family name ends. Honorifics and degrees follow a CJK name, and the spaced forms are recognized as suffixes — ``王小明 先生`` reads family 王小明 with 先生 in ``suffix``, and Korean's standardly-spaced 씨 routes the same way. -Glued hangul forms are reached too — default segmentation splits -김씨 into 김 + 씨 before the suffix is classified — and glued Han -forms the same way under the zh pack's segmentation. A glued kana -honorific (山田太郎様, 田中さん) is beyond whole-token matching and -currently stays part of the name. +A glued honorific is reached in exactly one shape — surname + +honorific, where segmentation splits off the surname and the +honorific is what remains: 김씨 reads family 김 with suffix 씨, and +王先生 the same way under the zh pack. A glued honorific after a +given name (김민준씨, 王小明先生) and glued kana (山田太郎様, 田中さん) +stay part of the name. A division the parser had to choose is reported rather than hidden. When an unspaced name has more than one vocabulary-supported split — diff --git a/nameparser/config/suffixes.py b/nameparser/config/suffixes.py index c56bbb2..c592b8d 100644 --- a/nameparser/config/suffixes.py +++ b/nameparser/config/suffixes.py @@ -38,21 +38,30 @@ 'שליט״א', # same, U+05F4 gershayim # #307: CJK postnominal honorifics and degrees, whole-token - # matched. That reaches the spaced forms directly -- and glued - # HANGUL forms too (김씨), because default segmentation splits the - # token before suffix classification runs; glued Han forms need - # the zh pack's segmentation the same way (王先生), and glued kana - # (山田太郎様, 田中さん) stays out of reach, tracked as #308. - # Self-selecting like the Korean - # surnames: a Han or hangul entry can only ever match CJK text. - # Vetting per the мл/ст standard above: none of these is a given - # name or surname in trailing whole-token position -- the two - # worth naming are 氏 (a rare Japanese surname reading exists, but - # a bare trailing 氏 after a name is the news-style honorific, and - # a 氏-surnamed person writes it FIRST) and the single-syllable - # ko pair 군/양, which trail a name only as honorifics (a - # single-syllable GIVEN name in final position is already - # vanishingly rare, and neither 군 nor 양 is one in practice). + # matched. That reaches the spaced forms directly, and glued + # forms in exactly ONE shape: surname+honorific (김씨; 王先生 + # under the zh pack), where segmentation splits off the surname + # before suffix classification and the honorific is what remains. + # A glued honorific after a GIVEN name (김민준씨, 王小明先生) and + # glued kana (山田太郎様; 田中さん additionally ships no さん + # entry at all) stay out of reach, tracked as #308. + # Self-selecting like the Korean surnames: a Han or hangul entry + # can only ever match CJK text. Vetting per the мл/ст standard + # above -- three entries worth naming: + # - 氏: a rare Japanese surname reading exists, but a bare + # trailing 氏 after a name is the news-style honorific, and a + # 氏-surnamed person writes it FIRST. + # - 양/군: 양 is also a top-tier surname (Yang) -- but a surname + # LEADS, so the trailing-only suffix gate never sees it there + # (양 미선 keeps family 양); as single-syllable GIVEN names in + # final position both are vanishingly rare in practice. + # - 博士: an attested Japanese given name (ひろし, Hiroshi) -- + # the doctorate reading vastly dominates the spaced trailing + # position this set matches, and a glued 田中博士 is untouched. + # Bare hangul 선생/교수 are deliberately absent (only the -님 + # honorific forms ship): the bare forms read as common nouns as + # readily as address terms. Further candidates (여사, 太太, 殿) + # wait on the same case-by-case argument. '씨', # ko Mr./Ms. -- standardly spaced in Korean orthography '박사', # ko doctorate holder ("Dr.") '선생님', # ko teacher/respected elder diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 902457f..167f980 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -736,16 +736,51 @@ def __post_init__(self) -> None: Case("ko_honorific_glued_via_segmentation", "김씨", {"family": "김", "suffix": "씨"}, classification="fix(#307)", - notes="glued HANGUL honorifics are reached after all: " - "default segmentation splits 김씨 into 김 + 씨 BEFORE " - "suffix classification, manufacturing the trailing " - "whole token -- a partial delivery of #308 that falls " - "out of stage order, hangul-only (glued Han needs the " - "zh pack; glued kana stays #308)"), + notes="glued hangul is reached ONLY in the surname+honorific " + "shape: default segmentation splits off the surname, " + "and the honorific is what remains -- a partial " + "delivery of #308 that falls out of stage order. A " + "glued honorific after a GIVEN name (김민준씨, the row " + "below) stays out of reach"), Case("ko_honorific_after_comma", "김민준, 씨", {"family": "김민준", "suffix": "씨"}, classification="fix(#307)", notes="the post-comma lenient gate admits the honorific too; " "the comma disables segmentation per the comma " - "doctrine, so 김민준 stays whole"), + "doctrine, so 김민준 stays whole"), Case("ko_honorific_glued_given_stays", "김민준씨", + {"family": "김", "given": "민준씨"}, + notes="the boundary of the glued reach: segmentation splits " + "off 김 and the REMAINDER is 민준씨, not a listed " + "honorific token -- the common full-name glued shape " + "is #308's mechanism, pinned here so the docs' scoped " + "claim stays true"), + Case("zh_honorific_glued_surname", "王先生", + {"family": "王", "suffix": "先生"}, + locale="zh", + classification="fix(#307)", + notes="the Han twin of 김씨: the zh pack's segmentation " + "splits off the surname and the remaining 先生 is the " + "honorific token"), + Case("zh_honorific_glued_given_stays", "王小明先生", + {"family": "王", "given": "小明先生"}, + locale="zh", + notes="the Han boundary twin: 小明先生 is the remainder, not " + "an honorific token -- glued full names stay #308"), + Case("ko_suffix_matching_is_whole_token", "김지양", + {"family": "김", "given": "지양"}, + notes="지양 ENDS with the honorific 양 but is a given name: " + "suffix matching is whole-token, never endswith -- the " + "pin the differential rule's anchor mirrors at the " + "name-string level"), + Case("ko_surname_yang_leads", "양 미선", + {"family": "양", "given": "미선"}, + notes="양 is both a top-tier surname (Yang) and a shipped " + "honorific: position decides, and a surname LEADS -- " + "the trailing-only suffix gate never sees it here"), + Case("ko_honorific_stack", "김민준 박사 씨", + {"family": "김", "given": "민준", "suffix": "박사, 씨"}, + classification="fix(#307)", + notes="a trailing RUN of honorifics peels whole, like " + "'Smith PhD MD' -- the multi-suffix loop the peel " + "shares with Latin suffixes"), ) diff --git a/tools/differential/corpus_cjk.jsonl b/tools/differential/corpus_cjk.jsonl index 8f2428e..4de386d 100644 --- a/tools/differential/corpus_cjk.jsonl +++ b/tools/differential/corpus_cjk.jsonl @@ -22,7 +22,9 @@ "毛 김" "毛泽东" "王·Smith" +"王先生" "王小明 先生" +"王小明先生" "田中 太郎 様" "田中『ハナ』花子" "諸葛亮" @@ -35,10 +37,14 @@ "김 민준" "김민준" "김민준 박사" +"김민준 박사 씨" "김민준 씨" "김민준, 씨" +"김민준씨" "김씨" +"김지양" "남궁" "남궁민수" "남궁민수, 지훈" "마이클·잭슨" +"양 미선" diff --git a/tools/differential/expected_changes.toml b/tools/differential/expected_changes.toml index e19fb0a..3e4fea9 100644 --- a/tools/differential/expected_changes.toml +++ b/tools/differential/expected_changes.toml @@ -186,16 +186,23 @@ issue = "fix(cjk-honorific-suffix) postnominal honorifics recognized, compoundin # three-token name moves `middle` too, past the fields-only # suffix-routing rule. Keyed on the honorific alternation itself # (the maiden-marker rule's pattern): the trigger tokens are wholly -# CJK characters, so the rule cannot touch a Latin name, and the +# CJK characters, so the rule matches only a name whose TRAILING +# token is a listed honorific -- a mostly-Latin name with one +# ('Wang Xiaoming 先生') is inside its shadow, accepted because the +# recognized honorific is the diff's cause there too, and the # alternation is a hand copy of SUFFIX_NOT_ACRONYMS' CJK entries -- # pinned by tests/v2/test_regex_sync.py, which derives the expected # set from the config by script membership. Anchored to a WHOLE -# trailing token ((?:^| )...$): a glued ending must not match, or a -# genuine given name ending in 양/군 (김지양) could have a real -# regression absorbed as intentional. Glued-hangul diffs (김씨 -- -# reached via segmentation) fall to the fields-only suffix-routing -# rule by their {first, suffix} shape, which is field-honest even -# though that rule's prose describes the two-token Latin case. +# trailing token ((?:^| )...$), judged on the NAME STRING: without +# it, any name string merely ENDING in 양/군 -- the glued given name +# 김지양, the spaced 김 지양 -- would match and could have a real +# regression absorbed as intentional (the case table pins the +# parser-side twin: suffix matching is whole-token, never endswith). +# Glued surname+honorific diffs (김씨 -- reached via segmentation) +# fall to the fields-only suffix-routing rule by their +# {first, last, suffix} shape (v1 held the whole token in first, so +# last moves too), which is field-honest even though that rule's +# prose describes the two-token Latin case. name_regex = "(?:^| )(?:씨|박사|선생님|교수님|군|양|先生|女士|小姐|博士|教授|様|氏)$" fields = ["first", "middle", "last", "suffix"]