Skip to content

Commit 4ebb4c4

Browse files
derek73claude
andcommitted
Scope the glued-reach story to its real boundary; pin what the prose claims
The PR review's convergent finding: 'glued hangul is reached' was overbroad in four places. Segmentation splits off ONE leading surname, so a glued honorific is reached in exactly the surname+honorific shape (김씨, 王先生 under zh) -- a glued honorific after a GIVEN name (김민준씨, 王小明先生), the more common written form, stays out of reach and belongs to #308. Every surface now states the boundary, and six new case rows pin it and the other prose-only claims: the two glued contrast pairs, whole-token matching (김지양 -- 지양 ENDS with 양 and must not peel, the parser-side twin of the differential anchor), surname-position 양 (양 미선 keeps family Yang), and the trailing honorific run (김민준 박사 씨 peels both, the Latin multi-suffix loop). The vetting comment gains its third named entry -- 博士 is an attested Japanese given name (ひろし, Hiroshi), argued rather than omitted -- plus the 양-surname overlap the 씨 argument covered only implicitly, and a sentence recording why bare hangul 선생/교수 deliberately do not ship. The differential rule comment corrects the 김씨 diff shape ({first, last, suffix}: v1 held the whole token in first, so last moves too), states the mixed-Latin shadow ('Wang Xiaoming 先生') as accepted rather than denied, and explains the anchor at the name-string level where it actually operates. The release log recovers the shared-Han fact (先生/博士/教授 serve Japanese too). Harness: 704 names, 62 intentional, 0 unexplained. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent bcb443b commit 4ebb4c4

6 files changed

Lines changed: 93 additions & 35 deletions

File tree

docs/release_log.rst

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@ Release Log
2525
- Fix names containing 〆 (U+3006, the shime mark that opens Japanese surnames like 〆木 and 〆谷) parsing given-first: the script classifier now counts 〆 as Han, extending the 々 entry the table already carries, and going a step further than it — 々 is Script=Han and merely outside the ideograph blocks, while 〆 is Script=Common — so these names take the East Asian family-first reading like any other wholly-Han name. ``Policy(script_orders={})`` restores the positional reading for these names exactly as for other wholly-Han names (#303)
2626
- Fix the katakana middle dot ``・`` (U+30FB, and its halfwidth twin U+FF65) being read as part of a name rather than as the divider it is: it now separates tokens exactly as a space does. A transcribed foreign name therefore divides into its parts and, being wholly katakana, keeps its source order — ``"マイケル・ジャクソン"`` gives given ``マイケル``, family ``ジャクソン``, where 1.x left the whole string in ``first`` — while a kanji pair written the same way takes the family-first rule (``"高橋・一郎"`` → family ``高橋``). Native Japanese names never contain this character and no other script's names use it, so the separation is unconditional, which also means it is **not** covered by the two policy opt-outs. Rendering follows, the way the Korean split's does: the dot comes back as a space, so ``str(HumanName("マイケル・ジャクソン"))`` is now ``"マイケル ジャクソン"``, and that reaches delimited content too — the nickname in ``"山田 太郎 (マイケル・ジャクソン)"`` renders ``"マイケル ジャクソン"``. The Chinese interpunct ``·`` is not an unconditional separator like these two: U+00B7 is also the Catalan punt volat and appears inside legitimate names (``Gal·la``), so it divides only between classified-script characters — the transcription treatment it marks is #298's (#272)
2727
- Fix 间隔号-divided transcriptions parsing as one unsplit token: U+00B7 — the interpunct Chinese text divides a transcribed foreign name with, ``威廉·莎士比亚`` for William Shakespeare — is now a token separator between characters of a classified script, and a name it divides keeps its source order and is never segmented: the dot is the transcription marker, playing the role pure katakana plays in the kana license. It divides ONLY between classified-script characters, so the Catalan punt volat interior to names like ``Gal·la`` is untouched. The Japanese nakaguro is deliberately not a transcription marker — ``高橋・一郎`` is roster formatting, 姓・名, and keeps its family-first reading — so a transcription typed with the wrong dot (``威廉・莎士比亚``) reads by the convention of the codepoint it was typed with; like the spaced form, only the Chinese dot rescues it. Rendering stays space-joined (``str(HumanName("威廉·莎士比亚"))`` is ``"威廉 莎士比亚"``); a custom ``string_format`` such as ``"{first}·{last}"`` reinstates the dot (#298)
28-
- Fix spaced CJK postnominal honorifics parsing as name parts: 씨, 박사, 선생님, 교수님, 군, 양 (Korean — standardly written as their own token), 先生, 女士, 小姐, 博士, 教授 (Chinese), and 様, 氏 (Japanese) now route to ``suffix``, so ``王小明 先生`` reads family ``王小明`` where the family-first default had confidently made 先生 the given name. Whole-token matching — which reaches glued hangul too, since default segmentation splits ``김씨`` into 김 + 씨 before the suffix is classified (and glued Han the same way under ``locales.ZH``); a glued kana honorific (``山田太郎様``, ``田中さん``) stays out of reach, tracked as #308 (closes #307)
28+
- Fix spaced CJK postnominal honorifics parsing as name parts: 씨, 박사, 선생님, 교수님, 군, 양 (Korean — standardly written as their own token), 先生, 女士, 小姐, 博士, 教授 (Chinese, with 先生/博士/教授 shared with Japanese), and 様, 氏 (Japanese) now route to ``suffix``, so ``王小明 先生`` reads family ``王小明`` where the family-first default had confidently made 先生 the given name. Whole-token matching — which also reaches a glued surname+honorific token, since segmentation splits off the surname first (``김씨`` reads family 김, suffix 씨; ``王先生`` the same under ``locales.ZH``); a glued honorific after a given name (``김민준씨``) and glued kana (``山田太郎様``) stay out of reach, tracked as #308 (closes #307)
2929
- Fix NFD-decomposed input missing the East Asian defaults entirely: script classification now normalizes to NFC before deciding, so a Korean or Japanese name typed on macOS — where decomposed text is routine — gets the same order rule as its composed twin, which it silently did not before. Segmentation MATCHING deliberately stays raw, so an unspaced NFD hangul name is ordered correctly but not split, rather than being split in the wrong place. One gotcha worth stating plainly: parse output preserves the encoding it was given, so for NFD input ``name.family == "김"`` is ``False`` even though it is the same name — compare NFC-normalized text when comparing across encodings (#272)
3030

3131
* 2.0.0 - July 27, 2026

docs/usage.rst

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -298,11 +298,12 @@ wrote the comma has already said where the family name ends.
298298
Honorifics and degrees follow a CJK name, and the spaced forms are
299299
recognized as suffixes — ``王小明 先生`` reads family 王小明 with 先生
300300
in ``suffix``, and Korean's standardly-spaced 씨 routes the same way.
301-
Glued hangul forms are reached too — default segmentation splits
302-
김씨 into 김 + 씨 before the suffix is classified — and glued Han
303-
forms the same way under the zh pack's segmentation. A glued kana
304-
honorific (山田太郎様, 田中さん) is beyond whole-token matching and
305-
currently stays part of the name.
301+
A glued honorific is reached in exactly one shape — surname +
302+
honorific, where segmentation splits off the surname and the
303+
honorific is what remains: 김씨 reads family 김 with suffix 씨, and
304+
王先生 the same way under the zh pack. A glued honorific after a
305+
given name (김민준씨, 王小明先生) and glued kana (山田太郎様, 田中さん)
306+
stay part of the name.
306307

307308
A division the parser had to choose is reported rather than hidden.
308309
When an unspaced name has more than one vocabulary-supported split —

nameparser/config/suffixes.py

Lines changed: 24 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -38,21 +38,30 @@
3838
'שליט״א', # same, U+05F4 gershayim
3939

4040
# #307: CJK postnominal honorifics and degrees, whole-token
41-
# matched. That reaches the spaced forms directly -- and glued
42-
# HANGUL forms too (김씨), because default segmentation splits the
43-
# token before suffix classification runs; glued Han forms need
44-
# the zh pack's segmentation the same way (王先生), and glued kana
45-
# (山田太郎様, 田中さん) stays out of reach, tracked as #308.
46-
# Self-selecting like the Korean
47-
# surnames: a Han or hangul entry can only ever match CJK text.
48-
# Vetting per the мл/ст standard above: none of these is a given
49-
# name or surname in trailing whole-token position -- the two
50-
# worth naming are 氏 (a rare Japanese surname reading exists, but
51-
# a bare trailing 氏 after a name is the news-style honorific, and
52-
# a 氏-surnamed person writes it FIRST) and the single-syllable
53-
# ko pair 군/양, which trail a name only as honorifics (a
54-
# single-syllable GIVEN name in final position is already
55-
# vanishingly rare, and neither 군 nor 양 is one in practice).
41+
# matched. That reaches the spaced forms directly, and glued
42+
# forms in exactly ONE shape: surname+honorific (김씨; 王先生
43+
# under the zh pack), where segmentation splits off the surname
44+
# before suffix classification and the honorific is what remains.
45+
# A glued honorific after a GIVEN name (김민준씨, 王小明先生) and
46+
# glued kana (山田太郎様; 田中さん additionally ships no さん
47+
# entry at all) stay out of reach, tracked as #308.
48+
# Self-selecting like the Korean surnames: a Han or hangul entry
49+
# can only ever match CJK text. Vetting per the мл/ст standard
50+
# above -- three entries worth naming:
51+
# - 氏: a rare Japanese surname reading exists, but a bare
52+
# trailing 氏 after a name is the news-style honorific, and a
53+
# 氏-surnamed person writes it FIRST.
54+
# - 양/군: 양 is also a top-tier surname (Yang) -- but a surname
55+
# LEADS, so the trailing-only suffix gate never sees it there
56+
# (양 미선 keeps family 양); as single-syllable GIVEN names in
57+
# final position both are vanishingly rare in practice.
58+
# - 博士: an attested Japanese given name (ひろし, Hiroshi) --
59+
# the doctorate reading vastly dominates the spaced trailing
60+
# position this set matches, and a glued 田中博士 is untouched.
61+
# Bare hangul 선생/교수 are deliberately absent (only the -님
62+
# honorific forms ship): the bare forms read as common nouns as
63+
# readily as address terms. Further candidates (여사, 太太, 殿)
64+
# wait on the same case-by-case argument.
5665
'씨', # ko Mr./Ms. -- standardly spaced in Korean orthography
5766
'박사', # ko doctorate holder ("Dr.")
5867
'선생님', # ko teacher/respected elder

tests/v2/cases.py

Lines changed: 42 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -736,16 +736,51 @@ def __post_init__(self) -> None:
736736
Case("ko_honorific_glued_via_segmentation", "김씨",
737737
{"family": "김", "suffix": "씨"},
738738
classification="fix(#307)",
739-
notes="glued HANGUL honorifics are reached after all: "
740-
"default segmentation splits 김씨 into 김 + 씨 BEFORE "
741-
"suffix classification, manufacturing the trailing "
742-
"whole token -- a partial delivery of #308 that falls "
743-
"out of stage order, hangul-only (glued Han needs the "
744-
"zh pack; glued kana stays #308)"),
739+
notes="glued hangul is reached ONLY in the surname+honorific "
740+
"shape: default segmentation splits off the surname, "
741+
"and the honorific is what remains -- a partial "
742+
"delivery of #308 that falls out of stage order. A "
743+
"glued honorific after a GIVEN name (김민준씨, the row "
744+
"below) stays out of reach"),
745745
Case("ko_honorific_after_comma", "김민준, 씨",
746746
{"family": "김민준", "suffix": "씨"},
747747
classification="fix(#307)",
748748
notes="the post-comma lenient gate admits the honorific too; "
749749
"the comma disables segmentation per the comma "
750-
"doctrine, so 김민준 stays whole"),
750+
"doctrine, so 김민준 stays whole"), Case("ko_honorific_glued_given_stays", "김민준씨",
751+
{"family": "김", "given": "민준씨"},
752+
notes="the boundary of the glued reach: segmentation splits "
753+
"off 김 and the REMAINDER is 민준씨, not a listed "
754+
"honorific token -- the common full-name glued shape "
755+
"is #308's mechanism, pinned here so the docs' scoped "
756+
"claim stays true"),
757+
Case("zh_honorific_glued_surname", "王先生",
758+
{"family": "王", "suffix": "先生"},
759+
locale="zh",
760+
classification="fix(#307)",
761+
notes="the Han twin of 김씨: the zh pack's segmentation "
762+
"splits off the surname and the remaining 先生 is the "
763+
"honorific token"),
764+
Case("zh_honorific_glued_given_stays", "王小明先生",
765+
{"family": "王", "given": "小明先生"},
766+
locale="zh",
767+
notes="the Han boundary twin: 小明先生 is the remainder, not "
768+
"an honorific token -- glued full names stay #308"),
769+
Case("ko_suffix_matching_is_whole_token", "김지양",
770+
{"family": "김", "given": "지양"},
771+
notes="지양 ENDS with the honorific 양 but is a given name: "
772+
"suffix matching is whole-token, never endswith -- the "
773+
"pin the differential rule's anchor mirrors at the "
774+
"name-string level"),
775+
Case("ko_surname_yang_leads", "양 미선",
776+
{"family": "양", "given": "미선"},
777+
notes="양 is both a top-tier surname (Yang) and a shipped "
778+
"honorific: position decides, and a surname LEADS -- "
779+
"the trailing-only suffix gate never sees it here"),
780+
Case("ko_honorific_stack", "김민준 박사 씨",
781+
{"family": "김", "given": "민준", "suffix": "박사, 씨"},
782+
classification="fix(#307)",
783+
notes="a trailing RUN of honorifics peels whole, like "
784+
"'Smith PhD MD' -- the multi-suffix loop the peel "
785+
"shares with Latin suffixes"),
751786
)

tools/differential/corpus_cjk.jsonl

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,9 @@
2222
"毛 김"
2323
"毛泽东"
2424
"王·Smith"
25+
"王先生"
2526
"王小明 先生"
27+
"王小明先生"
2628
"田中 太郎 様"
2729
"田中『ハナ』花子"
2830
"諸葛亮"
@@ -35,10 +37,14 @@
3537
"김 민준"
3638
"김민준"
3739
"김민준 박사"
40+
"김민준 박사 씨"
3841
"김민준 씨"
3942
"김민준, 씨"
43+
"김민준씨"
4044
"김씨"
45+
"김지양"
4146
"남궁"
4247
"남궁민수"
4348
"남궁민수, 지훈"
4449
"마이클·잭슨"
50+
"양 미선"

tools/differential/expected_changes.toml

Lines changed: 14 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -186,16 +186,23 @@ issue = "fix(cjk-honorific-suffix) postnominal honorifics recognized, compoundin
186186
# three-token name moves `middle` too, past the fields-only
187187
# suffix-routing rule. Keyed on the honorific alternation itself
188188
# (the maiden-marker rule's pattern): the trigger tokens are wholly
189-
# CJK characters, so the rule cannot touch a Latin name, and the
189+
# CJK characters, so the rule matches only a name whose TRAILING
190+
# token is a listed honorific -- a mostly-Latin name with one
191+
# ('Wang Xiaoming 先生') is inside its shadow, accepted because the
192+
# recognized honorific is the diff's cause there too, and the
190193
# alternation is a hand copy of SUFFIX_NOT_ACRONYMS' CJK entries --
191194
# pinned by tests/v2/test_regex_sync.py, which derives the expected
192195
# set from the config by script membership. Anchored to a WHOLE
193-
# trailing token ((?:^| )...$): a glued ending must not match, or a
194-
# genuine given name ending in 양/군 (김지양) could have a real
195-
# regression absorbed as intentional. Glued-hangul diffs (김씨 --
196-
# reached via segmentation) fall to the fields-only suffix-routing
197-
# rule by their {first, suffix} shape, which is field-honest even
198-
# though that rule's prose describes the two-token Latin case.
196+
# trailing token ((?:^| )...$), judged on the NAME STRING: without
197+
# it, any name string merely ENDING in 양/군 -- the glued given name
198+
# 김지양, the spaced 김 지양 -- would match and could have a real
199+
# regression absorbed as intentional (the case table pins the
200+
# parser-side twin: suffix matching is whole-token, never endswith).
201+
# Glued surname+honorific diffs (김씨 -- reached via segmentation)
202+
# fall to the fields-only suffix-routing rule by their
203+
# {first, last, suffix} shape (v1 held the whole token in first, so
204+
# last moves too), which is field-honest even though that rule's
205+
# prose describes the two-token Latin case.
199206
name_regex = "(?:^| )(?:씨|박사|선생님|교수님|군|양|先生|女士|小姐|博士|教授|様|氏)$"
200207
fields = ["first", "middle", "last", "suffix"]
201208

0 commit comments

Comments
 (0)