Skip to content

Commit bcb443b

Browse files
derek73claude
andcommitted
Recognize spaced CJK postnominal honorifics as suffixes (#307)
CJK honorifics and degrees FOLLOW the name, and under the 2.1 family-first defaults an unrecognized one confidently became a name part: 王小明 先生 read given=先生, 김민준 씨 put the honorific in middle. Thirteen curated entries join SUFFIX_NOT_ACRONYMS beside the Hebrew post-nominal block -- ko 씨/박사/선생님/교수님/군/양 (standardly spaced in Korean orthography, the strongest case), zh 先生/女士/小姐/博士/教授, ja 様/氏, with 先生/博士/教授 shared Han serving both languages. Self-selecting like the Korean surnames: a Han or hangul entry can only ever match CJK text. Vetted per the file's мл/ст standard; 氏 and the ko single-syllable pair 군/양 are argued in the comment, and the trailing-only position gate keeps surname-position 양 (양 미선) untouched. The order interaction needed no mechanism: suffix classification runs after script_segment and before the positional read, so the rows pass on vocabulary alone, through both the core and facade runners. Review found the scope claim wrong in the good direction: glued HANGUL honorifics are reached too, because default segmentation splits 김씨 into 김 + 씨 before suffix classification -- a partial delivery of #308 that falls out of stage order (glued Han needs the zh pack the same way; glued kana stays #308). Docs and the config comment state it, and case rows pin the glued and post-comma shapes alongside the four spaced ones. Differential: the new fix(cjk-honorific-suffix) rule takes the compound diffs (honorific recognition x CJK flip), keyed on the honorific alternation per the maiden-marker precedent and anchored to a WHOLE trailing token -- a glued ending must not match, or a genuine given name ending in 양/군 could have a real regression absorbed. The alternation is a hand copy of the config's CJK entries, pinned by a sync test that DERIVES the expected set from SUFFIX_NOT_ACRONYMS by script membership. Glued-hangul diffs fall to fields-only suffix-routing by their shape, noted in the rule. Harness: 698 names, 56 intentional, 0 unexplained. Closes #307 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent d09ac7f commit bcb443b

7 files changed

Lines changed: 139 additions & 0 deletions

File tree

docs/release_log.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@ Release Log
2525
- Fix names containing 〆 (U+3006, the shime mark that opens Japanese surnames like 〆木 and 〆谷) parsing given-first: the script classifier now counts 〆 as Han, extending the 々 entry the table already carries, and going a step further than it — 々 is Script=Han and merely outside the ideograph blocks, while 〆 is Script=Common — so these names take the East Asian family-first reading like any other wholly-Han name. ``Policy(script_orders={})`` restores the positional reading for these names exactly as for other wholly-Han names (#303)
2626
- Fix the katakana middle dot ``・`` (U+30FB, and its halfwidth twin U+FF65) being read as part of a name rather than as the divider it is: it now separates tokens exactly as a space does. A transcribed foreign name therefore divides into its parts and, being wholly katakana, keeps its source order — ``"マイケル・ジャクソン"`` gives given ``マイケル``, family ``ジャクソン``, where 1.x left the whole string in ``first`` — while a kanji pair written the same way takes the family-first rule (``"高橋・一郎"`` → family ``高橋``). Native Japanese names never contain this character and no other script's names use it, so the separation is unconditional, which also means it is **not** covered by the two policy opt-outs. Rendering follows, the way the Korean split's does: the dot comes back as a space, so ``str(HumanName("マイケル・ジャクソン"))`` is now ``"マイケル ジャクソン"``, and that reaches delimited content too — the nickname in ``"山田 太郎 (マイケル・ジャクソン)"`` renders ``"マイケル ジャクソン"``. The Chinese interpunct ``·`` is not an unconditional separator like these two: U+00B7 is also the Catalan punt volat and appears inside legitimate names (``Gal·la``), so it divides only between classified-script characters — the transcription treatment it marks is #298's (#272)
2727
- Fix 间隔号-divided transcriptions parsing as one unsplit token: U+00B7 — the interpunct Chinese text divides a transcribed foreign name with, ``威廉·莎士比亚`` for William Shakespeare — is now a token separator between characters of a classified script, and a name it divides keeps its source order and is never segmented: the dot is the transcription marker, playing the role pure katakana plays in the kana license. It divides ONLY between classified-script characters, so the Catalan punt volat interior to names like ``Gal·la`` is untouched. The Japanese nakaguro is deliberately not a transcription marker — ``高橋・一郎`` is roster formatting, 姓・名, and keeps its family-first reading — so a transcription typed with the wrong dot (``威廉・莎士比亚``) reads by the convention of the codepoint it was typed with; like the spaced form, only the Chinese dot rescues it. Rendering stays space-joined (``str(HumanName("威廉·莎士比亚"))`` is ``"威廉 莎士比亚"``); a custom ``string_format`` such as ``"{first}·{last}"`` reinstates the dot (#298)
28+
- Fix spaced CJK postnominal honorifics parsing as name parts: 씨, 박사, 선생님, 교수님, 군, 양 (Korean — standardly written as their own token), 先生, 女士, 小姐, 博士, 教授 (Chinese), and 様, 氏 (Japanese) now route to ``suffix``, so ``王小明 先生`` reads family ``王小明`` where the family-first default had confidently made 先生 the given name. Whole-token matching — which reaches glued hangul too, since default segmentation splits ``김씨`` into 김 + 씨 before the suffix is classified (and glued Han the same way under ``locales.ZH``); a glued kana honorific (``山田太郎様``, ``田中さん``) stays out of reach, tracked as #308 (closes #307)
2829
- Fix NFD-decomposed input missing the East Asian defaults entirely: script classification now normalizes to NFC before deciding, so a Korean or Japanese name typed on macOS — where decomposed text is routine — gets the same order rule as its composed twin, which it silently did not before. Segmentation MATCHING deliberately stays raw, so an unspaced NFD hangul name is ordered correctly but not split, rather than being split in the wrong place. One gotcha worth stating plainly: parse output preserves the encoding it was given, so for NFD input ``name.family == "김"`` is ``False`` even though it is the same name — compare NFC-normalized text when comparing across encodings (#272)
2930

3031
* 2.0.0 - July 27, 2026

docs/usage.rst

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -295,6 +295,15 @@ Chinese dot rescues the source order. And a comma disables the script
295295
behaviors entirely, on the reasoning ``name_order`` already follows: whoever
296296
wrote the comma has already said where the family name ends.
297297

298+
Honorifics and degrees follow a CJK name, and the spaced forms are
299+
recognized as suffixes — ``王小明 先生`` reads family 王小明 with 先生
300+
in ``suffix``, and Korean's standardly-spaced 씨 routes the same way.
301+
Glued hangul forms are reached too — default segmentation splits
302+
김씨 into 김 + 씨 before the suffix is classified — and glued Han
303+
forms the same way under the zh pack's segmentation. A glued kana
304+
honorific (山田太郎様, 田中さん) is beyond whole-token matching and
305+
currently stays part of the name.
306+
298307
A division the parser had to choose is reported rather than hidden.
299308
When an unspaced name has more than one vocabulary-supported split —
300309
``남궁민수`` is 남궁 + 민수 by the two-syllable surname but 남 + 궁민수

nameparser/config/suffixes.py

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,36 @@
3636
'ז״ל', # same, U+05F4 gershayim
3737
'שליט"א', # honorific for a living rabbi, ASCII quote
3838
'שליט״א', # same, U+05F4 gershayim
39+
40+
# #307: CJK postnominal honorifics and degrees, whole-token
41+
# matched. That reaches the spaced forms directly -- and glued
42+
# HANGUL forms too (김씨), because default segmentation splits the
43+
# token before suffix classification runs; glued Han forms need
44+
# the zh pack's segmentation the same way (王先生), and glued kana
45+
# (山田太郎様, 田中さん) stays out of reach, tracked as #308.
46+
# Self-selecting like the Korean
47+
# surnames: a Han or hangul entry can only ever match CJK text.
48+
# Vetting per the мл/ст standard above: none of these is a given
49+
# name or surname in trailing whole-token position -- the two
50+
# worth naming are 氏 (a rare Japanese surname reading exists, but
51+
# a bare trailing 氏 after a name is the news-style honorific, and
52+
# a 氏-surnamed person writes it FIRST) and the single-syllable
53+
# ko pair 군/양, which trail a name only as honorifics (a
54+
# single-syllable GIVEN name in final position is already
55+
# vanishingly rare, and neither 군 nor 양 is one in practice).
56+
'씨', # ko Mr./Ms. -- standardly spaced in Korean orthography
57+
'박사', # ko doctorate holder ("Dr.")
58+
'선생님', # ko teacher/respected elder
59+
'교수님', # ko professor (honorific form)
60+
'군', # ko young man ("Master")
61+
'양', # ko young woman ("Miss")
62+
'先生', # zh Mr. / ja teacher-master -- honorific in both
63+
'女士', # zh Ms./Madam
64+
'小姐', # zh Miss
65+
'博士', # zh+ja doctorate holder, shared Han
66+
'教授', # zh+ja professor, shared Han
67+
'様', # ja formal Mr./Ms. (the mail-addressing honorific)
68+
'氏', # ja news-style Mr. (田中氏)
3969
}
4070
"""
4171

tests/v2/cases.py

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -707,4 +707,45 @@ def __post_init__(self) -> None:
707707
notes="one classified neighbor is not enough: the guard "
708708
"requires both, so the undivided dot remains part of "
709709
"the word -- declining, not deciding"),
710+
Case("zh_honorific_suffix_spaced", "王小明 先生",
711+
{"family": "王小明", "suffix": "先生"},
712+
classification="fix(#307)",
713+
notes="CJK honorifics FOLLOW the name; a spaced 先生 (Mr.) is "
714+
"a suffix, and recognizing it must come before the "
715+
"family-first order hands it a role -- unrecognized it "
716+
"read as the GIVEN name under the 2.1 defaults"),
717+
Case("ko_honorific_ssi", "김민준 씨",
718+
{"family": "김", "given": "민준", "suffix": "씨"},
719+
classification="fix(#307)",
720+
notes="Korean orthography standardly SPACES 씨, so the "
721+
"whole-token suffix machinery reaches it; the name "
722+
"still segments (suffix classification runs after the "
723+
"script_segment stage, which only ever saw 김민준)"),
724+
Case("ko_degree_baksa", "김민준 박사",
725+
{"family": "김", "given": "민준", "suffix": "박사"},
726+
classification="fix(#307)",
727+
notes="박사 (doctorate) is the ko analogue of a trailing "
728+
"PhD: fix(suffix-routing)'s two-token shape, one "
729+
"script over"),
730+
Case("ja_sama_spaced", "田中 太郎 様",
731+
{"family": "田中", "given": "太郎", "suffix": "様"},
732+
classification="fix(#307)",
733+
notes="the spaced 様 of forms and databases; the glued "
734+
"mail-addressing form 山田太郎様 is #308's mechanism, "
735+
"out of reach of whole-token matching"),
736+
Case("ko_honorific_glued_via_segmentation", "김씨",
737+
{"family": "김", "suffix": "씨"},
738+
classification="fix(#307)",
739+
notes="glued HANGUL honorifics are reached after all: "
740+
"default segmentation splits 김씨 into 김 + 씨 BEFORE "
741+
"suffix classification, manufacturing the trailing "
742+
"whole token -- a partial delivery of #308 that falls "
743+
"out of stage order, hangul-only (glued Han needs the "
744+
"zh pack; glued kana stays #308)"),
745+
Case("ko_honorific_after_comma", "김민준, 씨",
746+
{"family": "김민준", "suffix": "씨"},
747+
classification="fix(#307)",
748+
notes="the post-comma lenient gate admits the honorific too; "
749+
"the comma disables segmentation per the comma "
750+
"doctrine, so 김민준 stays whole"),
710751
)

tests/v2/test_regex_sync.py

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -334,3 +334,33 @@ def test_cjk_corpus_matches_the_case_table() -> None:
334334
assert checked_in == module.selected_names(), (
335335
"corpus_cjk.jsonl is stale: regenerate with "
336336
"`uv run python tools/differential/build_cjk_corpus.py`")
337+
338+
339+
def test_differential_honorific_rule_matches_the_suffix_vocabulary() -> None:
340+
"""The fix(cjk-honorific-suffix) rule's alternation is a hand copy
341+
of SUFFIX_NOT_ACRONYMS' CJK entries (#307) -- the toml cannot
342+
import them. The expected set is DERIVED from the config by script
343+
membership (a classified codepoint anywhere in the entry), so
344+
adding a CJK honorific without widening the rule -- or widening
345+
the rule with something the vocabulary does not ship -- fails
346+
here. The span-bearing pins above skip this rule on purpose: its
347+
trigger is the alternation, not a character class.
348+
"""
349+
from nameparser.config.suffixes import SUFFIX_NOT_ACRONYMS
350+
351+
toml_path = (Path(__file__).parents[2] / "tools" / "differential"
352+
/ "expected_changes.toml")
353+
rules = tomllib.loads(toml_path.read_text())["change"]
354+
matched = [r for r in rules if "cjk-honorific-suffix" in r["issue"]]
355+
assert len(matched) == 1
356+
regex = matched[0]["name_regex"]
357+
prefix = "(?:^| )(?:"
358+
assert regex.startswith(prefix) and regex.endswith(")$"), (
359+
"the honorific rule's shape changed; update this parser")
360+
declared = set(regex[len(prefix):-2].split("|"))
361+
has_classified = _policy._script_matcher(*_policy._SCRIPT_RANGES)
362+
expected = {entry for entry in SUFFIX_NOT_ACRONYMS
363+
if has_classified(entry)}
364+
assert declared == expected, (
365+
f"rule declares {sorted(declared)}; the config's CJK suffix "
366+
f"entries are {sorted(expected)}")

tools/differential/corpus_cjk.jsonl

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,8 @@
2222
"毛 김"
2323
"毛泽东"
2424
"王·Smith"
25+
"王小明 先生"
26+
"田中 太郎 様"
2527
"田中『ハナ』花子"
2628
"諸葛亮"
2729
"阿明"
@@ -32,6 +34,10 @@
3234
"高橋一郎"
3335
"김 민준"
3436
"김민준"
37+
"김민준 박사"
38+
"김민준 씨"
39+
"김민준, 씨"
40+
"김씨"
3541
"남궁"
3642
"남궁민수"
3743
"남궁민수, 지훈"

tools/differential/expected_changes.toml

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -177,6 +177,28 @@ issue = "fix(cjk-comma-compound) comma routing compounds with the CJK order flip
177177
name_regex = "(?s)(?=.*,)(?=.*[\\u3005-\\u3006\\u3040-\\u309F\\u30A0-\\u30FF\\u3400-\\u4DBF\\u4E00-\\u9FFF\\uF900-\\uFAFF\\uAC00-\\uD7A3\\uFF65-\\uFF65])"
178178
fields = ["first", "middle", "last", "title", "suffix"]
179179

180+
[[change]]
181+
issue = "fix(cjk-honorific-suffix) postnominal honorifics recognized, compounding with the CJK order flip"
182+
# '王小明 先生', '김민준 씨', '田中 太郎 様': #307 ships the spaced CJK
183+
# honorifics as suffix vocabulary, so a trailing 先生/씨/様 moves to
184+
# `suffix` where 1.4 read it as a name part -- and the remaining
185+
# wholly-CJK name simultaneously takes the family-first flip, so a
186+
# three-token name moves `middle` too, past the fields-only
187+
# suffix-routing rule. Keyed on the honorific alternation itself
188+
# (the maiden-marker rule's pattern): the trigger tokens are wholly
189+
# CJK characters, so the rule cannot touch a Latin name, and the
190+
# alternation is a hand copy of SUFFIX_NOT_ACRONYMS' CJK entries --
191+
# pinned by tests/v2/test_regex_sync.py, which derives the expected
192+
# set from the config by script membership. Anchored to a WHOLE
193+
# trailing token ((?:^| )...$): a glued ending must not match, or a
194+
# genuine given name ending in 양/군 (김지양) could have a real
195+
# regression absorbed as intentional. Glued-hangul diffs (김씨 --
196+
# reached via segmentation) fall to the fields-only suffix-routing
197+
# rule by their {first, suffix} shape, which is field-honest even
198+
# though that rule's prose describes the two-token Latin case.
199+
name_regex = "(?:^| )(?:씨|박사|선생님|교수님|군|양|先生|女士|小姐|博士|教授|様|氏)$"
200+
fields = ["first", "middle", "last", "suffix"]
201+
180202
[[change]]
181203
issue = "feat(#269) non-Latin titles/conjunctions recognized"
182204
# 'г-н Иван Петров' (Cyrillic title), 'Хосе И Мария Сантос' (Cyrillic

0 commit comments

Comments
 (0)