Skip to content

Commit e6a6b69

Browse files
committed
Correct five accuracy nits around the peel (#308)
- _script_segment's module header Reads: line omitted the suffix vocabulary the stage now reads through _is_post_nominal -> _vocab.is_suffix_strict; _segment.py sets the convention for naming it. Produces: likewise omitted the peel's _PEELED_TAG. - The same docstring's gate list ("gated by the FAMILY comma and by 间隔号 but NOT by segment_scripts") read as exhaustive and omitted the ASCII bail -- the gate that actually decides a caller-configured LATIN tail. The bail's own comment had it right; the header had not caught up. - config/suffixes.py said "Two more are in NEITHER set" and then named three strings (君, 선생, 교수). The list above it counts entries. - AGENTS.md's shim-transformation roster was one short: maiden_delimiters minus nickname_delimiters carries the same v1-reachability comment and is pinned by test_snapshot_overlap_keeps_v1_nickname_precedence. Verified: with parens in both v1 buckets, 1.4.0 and HEAD both parse nickname, while Policy alone gives maiden. It sits on the Policy half of _snapshot(), which is why a Lexicon-only sweep missed it. - test_properties.py's "neither derivation guarantees a fire" reads as "sometimes fires". Measured over the 250 examples: peel 2, surname half 0. Give the number and the structural reason -- a non-hangul surname makes `w + "민준"` a mixed-script token whose effective_script is None, so it is never an activated site; the one hangul token drawn ('남궁민준') landed in an example whose policy had HANGUL out of segment_scripts.
1 parent b8a9252 commit e6a6b69

4 files changed

Lines changed: 31 additions & 14 deletions

File tree

AGENTS.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -131,7 +131,7 @@ The 2.0 rewrite lands as underscore-private modules alongside the v1 code. These
131131
- **Parser owns config-dependent conveniences**: `Parser.matches`/`Parser.capitalized`/`Parser.revise` exist because the `ParsedName` equivalents fall back to DEFAULT config for str/omitted arguments (documented loudly in both docstrings). `revise` harvests tokens from a full sub-parse of each replacement value (tags kept minus `FOLDED_TAG`, roles forced, ambiguities discarded); the merge tail is shared with `replace()` via `ParsedName._with_field_tokens`. `Parser.capitalized` delegates through `name.capitalized(self.lexicon)` specifically so `_parser` never imports `_render` — keep it that way.
132132
- **Per-word vocabulary fields warn on multi-word entries** (`_normset`/`_normpairs` via `_warn_dead_entry`, UserWarning, never a raise — see the given_name_titles Gotcha for why raising is wrong). `given_name_titles` is the one multi-word-matched field and is exempt; `_edit` passes `warn=False` (add() warns once via the new instance's `__post_init__`; remove() stores nothing). The default vocabulary and every locale pack must stay warning-free (`test_default_lexicon_builds_warning_free`, `test_pack_vocabulary_entries_are_single_words`).
133133
- **Invariants guard harm, not no-ops**: add a constructor check when violating it produces a *wrong parse*, not when it produces *nothing*. A false positive costs a working configuration; a true positive on an inert condition costs the user nothing, so that trade is never worth taking. `suffix_acronyms_ambiguous ∩ suffix_words` is guarded because the overlap loses a family name; `given_name_titles` is not, because an unreachable entry is simply never consulted (see Gotchas). Before adding one, construct the config it forbids and check what actually breaks.
134-
- **The shim TRANSLATES; it never raises on a config v1 accepted, and never silently changes the parse**: `Constants._snapshot()` is a translation boundary between v1's model and v2's invariants, and every transformation there carries its v1-reachability argument in a comment. Five exist today — `first_name_titles` re-folded per word (v1 joins-then-`lc`, v2 normalizes-then-joins), `suffix_acronyms_ambiguous ∩ acronyms` (a provable no-op), `suffix_words − ambiguous` (v1 already accepts the word via the acronym branch, so the addition is inert there), `particles_ambiguous ∪ (bound ∩ particles)` (a pinned deviation, `test_bound_never_given_prefix_deviates_on_two_pieces`), and `honorific_tails = GLUED_HONORIFICS ∩ suffix_words` (#308 behavior with no v1 manager of its own, so the one v1 knob that reaches it is deleting the suffix word — which turns the peel off, `test_snapshot_removing_a_honorific_word_turns_the_peel_off`). When a v1 config cannot satisfy a v2 invariant, work out what v1 actually *does* with it — usually nothing — and reproduce that; weakening the invariant or letting the raise through are both wrong. **Test the case the translation decides**, not one where both branches agree: a test using an input v1 parses identically with and without the config pins nothing.
134+
- **The shim TRANSLATES; it never raises on a config v1 accepted, and never silently changes the parse**: `Constants._snapshot()` is a translation boundary between v1's model and v2's invariants, and every transformation there carries its v1-reachability argument in a comment. Six exist today — `first_name_titles` re-folded per word (v1 joins-then-`lc`, v2 normalizes-then-joins), `suffix_acronyms_ambiguous ∩ acronyms` (a provable no-op), `suffix_words − ambiguous` (v1 already accepts the word via the acronym branch, so the addition is inert there), `particles_ambiguous ∪ (bound ∩ particles)` (a pinned deviation, `test_bound_never_given_prefix_deviates_on_two_pieces`), `honorific_tails = GLUED_HONORIFICS ∩ suffix_words` (#308 behavior with no v1 manager of its own, so the one v1 knob that reaches it is deleting the suffix word — which turns the peel off, `test_snapshot_removing_a_honorific_word_turns_the_peel_off`), and `maiden_delimiters − nickname_delimiters` on the POLICY half of the same method (v1 precedence: a pair in both v1 buckets parses as a nickname, while `Policy` resolves the overlap the other way, so the subtraction is what keeps the facade at v1 behavior, `test_snapshot_overlap_keeps_v1_nickname_precedence`). Note that last one is on the `Policy`, not the `Lexicon` — the roster is per-`_snapshot()`, not per-vocabulary-field, so a sweep that only reads the `Lexicon(...)` call misses it. When a v1 config cannot satisfy a v2 invariant, work out what v1 actually *does* with it — usually nothing — and reproduce that; weakening the invariant or letting the raise through are both wrong. **Test the case the translation decides**, not one where both branches agree: a test using an input v1 parses identically with and without the config pins nothing.
135135
- **Reprs are bounded**: render which fields deviate from a named baseline and by how much, never contents (`Lexicon(default + titles: +2)`). `PolicyPatch`'s repr shows only set (non-UNSET) fields; `_order_repr` must never raise even on an unvalidated patch's garbage `name_order` (PolicyPatch defers validation to apply time); the sweep test in `tests/v2/test_reprs.py` pins that no config repr leaks the UNSET sentinel.
136136
- **Typing/docs**: `from __future__ import annotations`; `frozen=True, slots=True` on every public dataclass; strict-profile mypy flags via per-module overrides in pyproject (`strict = true` itself is not valid per-module). Docstrings state contracts in prose with **no doctest blocks**`--doctest-modules` makes every example a test; behavior examples go to unit tests per the lean-docs rule. **Document the positive direction of a partial property**: "a non-empty `ambiguities` is a signal to act on" is checkable, while "an empty one means no fork occurred" is a universal negative needing exhaustive verification -- that claim was written twice and falsified twice, at sites the author had not audited.
137137
- **The segmenter contract**: the optional `Parser(segmenter=...)` hook is parse-totality's ONE exception (locales spec section 4). Everything inside that exception is a bug in USER CODE, never a fact about the name, so it is surfaced rather than absorbed: the segmenter's own exceptions propagate, and the two protocol violations the stage can detect for itself — an answer of the wrong type, and one cutting at or past the end of the token it was handed — raise `TypeError`/`ValueError` from `_script_segment` for the same reason. The line to hold when adding a check there: a protocol violation by the segmenter's AUTHOR raises, while an adapter's defense against its own third-party library (`locales/ja.py`'s repertoire, length, reconstruction and score guards) declines with `None`, because what those catch is a fact about the content.

nameparser/_pipeline/_script_segment.py

Lines changed: 18 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -3,13 +3,16 @@
33
Consumes: tokens, segments, structure, interpunct_offsets, segmenter.
44
Produces: tokens, by two independent splits into sub-slices -- a
55
listed honorific peeled off the END of the name part's last
6-
non-post-nominal token, and the first activated-script token of the
7-
name segment split into n+1 -- segments (index runs remapped past the
8-
insertions), ambiguities (indices likewise remapped, plus a
9-
SEGMENTATION report when more than one split was vocabulary-supported,
10-
or when a segmenter's answer scored under the confidence floor).
6+
non-post-nominal token (whose tail token also carries this module's
7+
_PEELED_TAG), and the first activated-script token of the name segment
8+
split into n+1 -- segments (index runs remapped past the insertions),
9+
ambiguities (indices likewise remapped, plus a SEGMENTATION report
10+
when more than one split was vocabulary-supported, or when a
11+
segmenter's answer scored under the confidence floor).
1112
Reads: Policy.segment_scripts, Lexicon.surnames,
12-
Lexicon.honorific_tails, ParseState.segmenter.
13+
Lexicon.honorific_tails, ParseState.segmenter, and Lexicon suffix
14+
vocabulary (via _vocab.is_suffix_strict) -- both the peel's scan-back
15+
and the surname site ask whether a token is a post-nominal.
1316
1417
Unspaced CJK names give tokenize no separator to find, so this stage
1518
inserts the missing token boundary by vocabulary: the first token
@@ -25,11 +28,15 @@
2528
peeled off as its own token -- 田中さん -> 田中 + さん -- so that
2629
suffix classification can claim it and the surname match or segmenter
2730
consult below sees the name rather than the name plus an honorific. It
28-
is gated by the FAMILY comma and by 间隔号 but NOT by segment_scripts:
29-
the vocabulary of tails is licensed by the entries themselves, each of
30-
which can never end a name, so no per-script trust question arises. A
31-
suffix comma does not gate it -- "Dr 김민준씨, Jr." peels within its
32-
name part like any other.
31+
is gated by the stage's own ASCII bail, by the FAMILY comma and by
32+
间隔号, but NOT by segment_scripts: the vocabulary of tails is licensed
33+
by the entries themselves, each of which can never end a name, so no
34+
per-script trust question arises. The ASCII bail is the gate a caller
35+
adding a LATIN tail meets -- it sits above everything here, so such a
36+
tail fires only on a name carrying at least one non-ASCII character
37+
(see the bail's own comment, and honorific_tails' field note). A
38+
suffix comma gates nothing -- "Dr 김민준씨, Jr." peels within its name
39+
part like any other.
3340
3441
Where the VOCABULARY declines -- no prefix matched -- an optional
3542
Parser(segmenter=...) gets the token (#272 amendment 2026-07-29).

nameparser/config/suffixes.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -145,7 +145,7 @@
145145
family name in two. Spaced 殿 is safe for the reason 양/군 are: a
146146
殿-surnamed person's name LEADS, and the suffix gate is trailing-only.
147147
148-
Two more are in NEITHER set, so neither spelling is recognized. 君: 王君 is
148+
Three more are in NEITHER set, so neither spelling is recognized. 君: 王君 is
149149
a complete Chinese name (君 is a common given-name final), so the honorific
150150
reading never gets the benefit of the doubt -- while its kana spelling くん
151151
ships glued, above. Bare 선생 and 교수: they read as common nouns as readily

tests/v2/test_properties.py

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -332,7 +332,17 @@ def _names_using(draw: st.DrawFn, lexicon: Lexicon) -> str:
332332
# words, and a bare drawn surname standing earlier in the name takes
333333
# the surname site before the unspaced token can. Instrument before
334334
# concluding either half is exercised -- the counts above are what
335-
# that costs to find out.
335+
# that costs to find out, and the two halves are NOT in the same
336+
# state. Measured over this test's 250 examples: the peel fires
337+
# twice, the surname half ZERO times -- currently inert. Structural,
338+
# not luck: `w + "민준"` on a non-hangul surname is a MIXED-script
339+
# token, whose effective_script is None, so it can never be an
340+
# activated surname site at all; only a drawn HANGUL surname makes
341+
# one, and across the run exactly one such token was sampled
342+
# ('남궁민준'), into an example whose drawn policy had HANGUL out of
343+
# segment_scripts. Deriving the token was still the right move --
344+
# it took the half off structurally-unreachable -- but a fix worth
345+
# having would derive it in the script the policy activated.
336346
# sorted for the same reason `vocab` above is: frozenset iteration
337347
# order is not stable across runs, and an unsorted pool shifts
338348
# every index sampled_from draws -- which would defeat

0 commit comments

Comments
 (0)