Peel glued CJK honorifics off the end of a name (#308) - #311
Merged
Conversation
The neighbour precondition exempted any recognized post-nominal, which reached well past the case it was written for: it broke the SPACED forms, whose writers drew an actual boundary. Under the JA pack "佐藤 氏" divided 佐 + 藤, "田中 様" 田 + 中, "鈴木 先生" 鈴 + 木, "中村 教授" 中 + 村 -- while glued 佐藤氏, which never peels (氏 is deliberately not a tail), stayed correct. That inverts the branch's own doctrine: the spaced position is a token boundary its writer drew. Narrow the carve-out to the tail the peel MANUFACTURED. The peel tags the token it cuts off and the neighbour test reads that tag instead of asking the vocabulary -- which cannot separate the two spellings, since they put the same word in the same place and only the provenance differs. A glued honorific means the writer drew no boundary anywhere; a spaced one means they drew one and wrote the name beside it as a unit. The motivating fix is untouched: glued 山田太郎様 still reaches the segmenter as 山田太郎, and 田中さん still divides under the pack. _is_post_nominal loses a caller and says so.
선생님, 교수님 and 박사님 are the three standard -님 professional honorifics; the first two shipped in both sets and the third in neither, which broke both of its spellings. Glued, 김민준박사님 peeled 님 alone and stranded 박사 in the given name; spaced, 김민준 박사님 was cut by the peel -- the token is not a whole-token suffix word -- and came back as two suffixes for one honorific. Both now give suffix 박사님. The case row that used 김민준박사님 to pin the one-peel/no-recursion rule can no longer demonstrate it, and no shipped entry pair has the shape that could: it needs a tail whose remainder ends in a DIFFERENT tail, the two not forming a listed entry themselves. The pin stays at stage level, where a synthetic lexicon can build that shape, and the row is repointed at the behavior it now shows.
Nothing here changes behavior; each is a statement a later reader would have built on. - The ASCII bail justified itself purely in terms of scripts, which was true when only the surname half existed. It now also short-circuits the peel, which has no script gate, so a configured ASCII tail never fires and an accent on an unrelated token switches it on. Both stated at the bail and at the public field, matching rather than changed: correct for the CJK vocabulary that ships, and changing it needs its own vetting pass. - The peel's maiden paragraph documented the direction that fires and not the one that does not: "김민준씨 née 박" still gives given 민준씨, and "김민준씨 née 박씨" routes the maiden's honorific while leaving the person's own. Stated as a limit, with why chasing it here is not possible (classify has not run). - _longest_entry's cache is keyed by the frozenset VALUE, so lexicons sharing KOREAN_SURNAMES collapse to one entry; the "times the two fields" multiplication overstated it. - _SUBSET_FIELDS' header generalized "each marker NARROWS its base", which honorific_tails inverts -- it widens where a suffix word may match. Its base is also narrower than what classifies the peeled piece (suffix_as_written ORs in the non-ambiguous acronyms); that is a trade, recorded so nobody fixes it. - GLUED_HONORIFICS' exclusion list was headed "these ship spaced only" while its last bullet named two entries that ship in neither set. Split the two kinds. - The kana-vs-kanji vetting note claimed hiragana spells no name character, true only within its implicit "in Chinese" -- this project's own case table pins the hiragana given name 高橋みなみ.
honorific_tails appeared nowhere in tests/v2/test_config_shim.py, and the snapshot-equality test passes VACUOUSLY here -- with the default config the GLUED_HONORIFICS ∩ suffix_words intersection is a no-op, which is exactly the failure mode AGENTS.md warns about. Pin the case the translation decides: deleting a CJK honorific from Constants.suffix_not_acronyms, the only v1 knob that reaches the field, turns the peel off rather than raising the subset error or leaving a tail that peels into a suffix field no longer recognizing it. AGENTS.md's roster of shim transformations said four and named four; this branch added the fifth. Same-commit rule applied one commit late.
Two overclaims in the user docs and four stale notes in the tests. usage.rst said the honorific is split off "before anything else reads the name". Comma structure and interpunct detection both run first and both gate the peel: "田中さん, 太郎" keeps the whole 田中さん as the family name, pinned by test_family_comma_skips_the_peel and documented nowhere. Said now, next to the comma rule it belongs to. Both usage.rst and the release log promised that the spaced spelling "composes the same way" and warned that a spaced family-name-only address would start dividing. The previous commit that narrowed the carve-out makes both false: spaced forms are unaffected, and the release log's ja-pack paragraph now says which spelling divides and which does not, and offers the spaced spelling as the opt-out it now is. The case-table notes: the lone-씨 row pointed at the length cap when the peel-site scan is what holds it, ja_sama_spaced still forward- pointed to work that has landed, the lone-さん row named its twin by position and got the wrong row, and test_lexicon's rationale credited an ambiguous subtraction that happens only in the v1 shim.
A mutation pass found one live survivor on a fully-covered line: replacing the peel's `state.segments[0]` with `range(len(state.tokens))` -- the natural simplification, since under NO_COMMA the two runs are usually identical -- changes real output and fails nothing. Extracted delimited content is still in the token stream at this stage with only its role set, so "김민준씨 (Jimmy)" would hand the scan-back Jimmy as its site: no post-nominal to step over, no tail to peel, and the peel lost entirely. The comment above the segment lookup made it worse by saying extracted content is unreachable and that no input can produce it -- true of the surname half it was written for, actively misleading for the peel, where the choice is what keeps Jimmy out. Case row plus both comments. Two boundaries a reader could otherwise meet by surprise: - "田中さん, PhD" keeps the whole 田中さん as the family name while "田中さん PhD" gives family 田中 and suffix "さん, PhD". A one-word name part before a comma is FAMILY_COMMA, which opts the stage out by doctrine -- the one spelling disagreement #308 does not remove, where the 김민준씨 Jr. pair got two rows for removing exactly that. A case row, so it is a recorded decision. - The off-switch docs/customize.rst promises was verified and untested: removing a tail leaves the glued name unsplit while the spaced spelling still routes the honorific. Pinned at parse level, against a peel-is-on baseline so it cannot pass vacuously. And a clause recording why _is_post_nominal is strict rather than lenient: the initial veto makes "田中さん V." decline where "田中さん II" peels, which agrees with what classify does with "V." downstream. Swapping the predicate fails no test; the clause is the record instead.
The property layer draws honorific_tails and repairs the draw, but
_names_using only manufactured `surname + 민준`, so no generated token
ever ENDED in a drawn tail and the peel never fired under a drawn
lexicon. Instrumented before and after: 14 fires across
test_properties.py, all 14 under Lexicon.default() and 0 under a drawn
one; deriving the glued token from the draw moves that to 2. Both are
ASCII tails on a non-Latin stem ('민준de', '민준and'), which is the one
shape that reaches an ASCII tail at all -- the stage bails on a wholly
ASCII original.
Same failure mode the pool's own comment records having measured and
fixed for surnames. The comment now covers both derivations, and no
longer states a per-run fire count for the surname half as fact: the
same instrumentation shows zero surname-half splits under drawn
lexicons in this run, so the honest note is to instrument before
concluding either half is exercised.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #311 +/- ##
==========================================
+ Coverage 98.45% 98.46% +0.01%
==========================================
Files 41 41
Lines 2776 2803 +27
==========================================
+ Hits 2733 2760 +27
Misses 43 43 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
The clause read "ONE peel, never recursive: 김민준박사님 gives up its
님 and keeps its glued 박사". Shipping 박사님 falsified it in the same
branch: longest-first now reaches the whole honorific, and that input
measures {'given': '민준', 'family': '김', 'suffix': '박사님'}.
Every other home of the claim was updated when 박사님 landed -- the
case row renamed and re-expected, the stage test moved to the
synthetic _TAILS that deliberately omits 박사님, both docs. This
docstring was the survivor, and it is the one a maintainer reads while
editing the peel.
State the rule with an honest witness instead: no shipped pair has the
shape the pin needs, so the synthetic lexicon in
test_one_peel_never_a_stack is the only one left.
The row for "田中さん, PhD" was classified fix(#271) with the note "the fields here are the CJK order flip's". The observation behind it was right -- the only deviation from 1.4.0 is first -> family -- but the attribution was untested, and it is wrong. Measured, 1.4.0 against HEAD: Smith, PhD first Smith -> family Smith Smith, Dr. first Smith -> family Smith 田中さん, PhD first 田中さん -> family 田中さん "Smith, PhD" is pure Latin, so #271's script-conditional rules cannot be touching it, and it makes the identical move. The behavior is FAMILY_COMMA's and predates #271; the table already has the slug for it, used by family_comma_lone_title. Also rename ko_ -> ja_: the input is Han plus hiragana, and the table prefixes rows by their input's script.
The narrowing was explained in four places as "its writer drew that
boundary and chose to write 山田太郎 as a unit". That is false for
exactly the input it describes: in "山田太郎 様" the unit the writer
drew is the whole name, honorific and all. The neighbour precondition
exists to detect a boundary WITHIN the name ("山田 太郎"), and a
trailing honorific is not one in either spelling.
What the code actually relies on is weaker: by position a spaced
honorific is indistinguishable from a spaced name element, so the
precondition conservatively counts it. State that, and the measured
trade that makes it worth taking -- exempting spaced honorifics would
divide 佐藤 氏, 田中 様, 鈴木 先生 and 中村 教授 (all four divide bare
under the JA pack) to win the single division 山田太郎 様.
Fixed in the module docstring, the inline comment above the
precondition, docs/usage.rst and the release log's #308 bullet.
usage.rst promised that "writing the honorific spaced is therefore
also how you ask for a family name to be kept whole, short of
declining the pack", and the release log carried the same closing
sentence. Both generalize a fact about the SEGMENTER path to a
paragraph a Korean- or Chinese-data reader also reads.
Measured, the vocabulary path does not part company at all:
김민준 씨 -> family 김, given 민준, suffix 씨 김민준씨 identical
王小明 先生 -> family 王, given 小明, suffix 先生 王小明先生 identical (zh)
Spacing buys nothing there, and "short of declining the pack" is not
even available for hangul, whose segmentation is a default.
Narrow both, and pin the boundary with a test, since the claim is
falsifiable prose.
Swapping is_suffix_lenient into _is_post_nominal still failed zero
tests. The previous round closed that with a docstring clause -- but
the clause NAMES the discriminating input under the default lexicon,
and when a comment can name the input, the input belongs in a test.
Measured, default lexicon:
田中さん V. strict given 田中さん, family 'V.' (no peel)
田中さん V. lenient given 田中, middle さん, family 'V.'
田中さん II strict family 田中, suffix 'さん, II'
田中さん II lenient identical
The V. row is 1.4.0 parity (first 田中さん / last 'V.'), so it lands as
a parity row; the II row is a #308 deviation and is classified as one.
Together they are the pin: only the strict/lenient choice separates
them, and a future "unify the two suffix predicates" refactor would
have changed a user-visible parse silently.
corpus_cjk.jsonl regenerated (it derives from the case table).
test_ja_does_not_divide_a_spaced_surname_under_an_honorific asserted that 佐藤 氏 keeps family 佐藤, but nothing in it showed the pack WOULD divide 佐藤 otherwise -- it passed vacuously for any reason the segmenter went unconsulted, and the contrast is what the doc sentence it supports rests on. Assert the bare division too (佐 + 藤, 田 + 中, 鈴 + 木, 中 + 村), and add 中村 教授 so the four cases the prose now cites as the measured price of this rule are the four the test runs. The comment's "the writer drew this boundary: 佐藤 is a unit" goes with them, for the reason the previous commit gives.
- _script_segment's module header Reads: line omitted the suffix
vocabulary the stage now reads through _is_post_nominal ->
_vocab.is_suffix_strict; _segment.py sets the convention for naming
it. Produces: likewise omitted the peel's _PEELED_TAG.
- The same docstring's gate list ("gated by the FAMILY comma and by
间隔号 but NOT by segment_scripts") read as exhaustive and omitted
the ASCII bail -- the gate that actually decides a caller-configured
LATIN tail. The bail's own comment had it right; the header had not
caught up.
- config/suffixes.py said "Two more are in NEITHER set" and then named
three strings (君, 선생, 교수). The list above it counts entries.
- AGENTS.md's shim-transformation roster was one short: maiden_delimiters
minus nickname_delimiters carries the same v1-reachability comment
and is pinned by test_snapshot_overlap_keeps_v1_nickname_precedence.
Verified: with parens in both v1 buckets, 1.4.0 and HEAD both parse
nickname, while Policy alone gives maiden. It sits on the Policy half
of _snapshot(), which is why a Lexicon-only sweep missed it.
- test_properties.py's "neither derivation guarantees a fire" reads as
"sometimes fires". Measured over the 250 examples: peel 2, surname
half 0. Give the number and the structural reason -- a non-hangul
surname makes `w + "민준"` a mixed-script token whose
effective_script is None, so it is never an activated site; the one
hangul token drawn ('남궁민준') landed in an example whose policy had
HANGUL out of segment_scripts.
The peel called _split and then rebuilt the token tuple slice-wise to set one tag on the piece _split had just created, re-deriving where its own tail landed after handing the index arithmetic off -- an idiom no other stage uses. _split now takes an optional tail_tag and applies it to the last piece, which is the tail by construction for the one caller that passes one. The surname path passes nothing and the default leaves it unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
_LEX_TAILS re-declared _LEX's six-surname frozenset and its "jr" suffix word verbatim three lines below it, so an edit to _LEX -- whose comment explains why 欧/欧阳 and 남/남궁 are there -- would silently miss the peel tests that exercise the same surname half. Building it with add() produces an identical lexicon and keeps one declaration. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
test_default_lexicon_honorific_tails_are_all_suffix_words justified itself as closing a python -O gap in config/suffixes.py's import-time assert, but Lexicon.__post_init__ enforces the same invariant with a raise, which -O does not strip: Lexicon.default() can never construct a violating instance, so the assert line is unreachable as a failure and the constructor on the line above would raise first. Neither sibling subset field has such a test either; the shared coverage is the parametrized test_subset_error_names_the_fix. The useful half of the justification -- that the runtime guard, unlike the config assert, survives -O -- moves to the honorific_tails row in _SUBSET_FIELDS. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every other glued-set exclusion has a case row holding it -- 氏 by zh_glued_shi_stays, 博士 by ja_glued_degree_stays, 君 by zh_glued_jun_stays, 양 by ko_suffix_matching_is_whole_token -- while 殿, which has the longest argument behind it, had none: adding it back to GLUED_HONORIFICS passed the entire suite. The discriminating input is the lone surname 鵜殿 (Udono), which a peeled 殿 would report as family 鵜 plus suffix 殿; a two-token 真殿 太郎 puts the site scan on 太郎 and never consults the exclusion. Verified by adding 殿 back: the new row and its facade twin are the only tests that then fail. Classified fix(#271) like its neighbours -- 1.4.0 gives first 鵜殿 and no last, so it is the CJK order flip that moves the field. corpus_cjk.jsonl regenerated; differential stays 0 unexplained. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every unit in _SHAPES was pure ASCII, so script_segment returned at its isascii() bail before either the #308 peel or the #271 surname site ran: the file's own docstring warns the absolute tests are structurally blind to a complexity regression, and for this stage the scaling test was blind too. A repeated 씨 walks the peel's site scan-back over the whole run -- every token is a post-nominal -- and measures 4.0-4.3 at base 800, inside the clean column the ASCII shapes already occupy, so _MAX_RATIO is unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The peel costs a couple of microseconds per non-ASCII parse and is not gated on segment_scripts -- deliberate, and the prose already says so, but a performance-sensitive caller reaches for that field first. Emptying honorific_tails hits the peel's first-line guard and recovers the cost; customize.rst already documents that removal as the BEHAVIORAL off-switch, so one clause there covers the other reason to reach for it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
CJK honorifics written glued to a name — 田中さん, 山田太郎様, 김민준씨 — were absorbed into the name, because suffix matching is whole-token.
script_segmentnow splits a listed honorific off the end of the name part's last non-post-nominal token, as a split (sub-slices of the original, spans unchanged), so ordinary suffix classification claims it and the stage's existing surname-split and segmenter machinery see the name without it.GLUED_HONORIFICSvocabulary (15 entries) andLexicon.honorific_tailscarrying it, guarded by a_SUBSET_FIELDSinvariant — every tail must also be a suffix word, or the peel strands a fragment inside the name._is_post_nominalpredicate governs two decisions with deliberately different verdicts: the peel site scans back over trailing post-nominals, while the surname site declines when the leading script-written token is one. Skipping there split양 지훈— Yang Jihoon — into given 지 / middle 훈.山田太郎様divides. Spaced forms are deliberately untouched:佐藤 氏keeps its surname whole, because there the writer did draw the boundary.Default-on. Reaches
HumanNameas well as the 2.0 API. Release-log entries cover the upgrade surface, including that glued田中さんnow divides to 田・中 under thejapack.Test Plan
uv run --extra ja pytest -q— 2844 passed, 16 skipped, 11 xfaileduv run mypy— clean, 96 filesuv run ruff check nameparser/— cleansphinx-build -b htmland-b doctest— clean, 207 doctests, 0 failurestools/differential/compare.pyvs nameparser 1.4.0 — 726 names, 84 intentional diffs, 0 unexplainedReview notes
Two design calls were made, then reversed on measured evidence — both reversals are their own commits so the reasoning survives:
17f9c39→be97aec: the surname site skipped post-nominals and fell through to the given name.양 지훈is why it declines instead.59aac1f→063fd1a: the segmenter carve-out exempted any recognized honorific, which took four correct spaced outputs (佐藤 氏,田中 様,鈴木 先生,中村 教授) and broke them to win one. Narrowed to the tail the peel manufactures.Also corrected in passing: two case-table rows labelled
paritythat diverge from 1.4.0 (verified against the real wheel) are nowfix(#271).Closes #308.
🤖 Generated with Claude Code