Skip to content

Transcribed foreign names in Chinese (威廉·莎士比亚) parse reversed or not at all #298

Description

@derek73

Chinese has no dedicated script for foreign names. Where Japanese writes transcriptions in katakana, Chinese writes them in ordinary Han characters — 威廉·莎士比亚 is William Shakespeare — and marks the boundary between name parts with the 间隔号, an interpunct written as U+00B7 (·), U+30FB (・), or U+FF65 (・) depending on the input method. Transcribed names keep their source order: 威廉 (William) is the given name. The interpunct is the only orthographic signal that distinguishes a transcription from a native Han name.

The 2.1 CJK defaults get these names wrong in two different ways, depending on which codepoint the dot happens to be:

>>> parse("威廉・莎士比亚").family   # U+30FB — splits, then reversed
'威廉'
>>> parse("威廉·莎士比亚").given    # U+00B7 — not a separator at all
'威廉·莎士比亚'
  • U+30FB / U+FF65 (added as separators in Japanese name segmentation via optional dependency (nameparser[ja]) #272 for katakana names like マイケル・ジャクソン): the name splits into two wholly-Han tokens, script_orders licenses them family-first, and the parts come out reversed. The katakana case is protected because pure katakana is deliberately excluded from the order license; a Han transcription has no such exemption.
  • U+00B7: not a separator, and it falls outside every script range, so the token is never licensed and the whole string lands in given unsplit.

Two halves to a fix, and the second is the important one:

  1. Order suppression. A name whose tokens were divided by an interpunct is a transcription and should keep source order — the dot plays the same role for Han that pure katakana plays in the ja rule, so this extends the existing "the orthography itself settles the convention" principle rather than adding a heuristic. The implementation cost is that separators are erased during tokenization; the "this boundary was a name dot" fact has to survive to where the order decision is made.
  2. U+00B7 as a separator, guarded. U+00B7 is also the Catalan punt volat, interior to legitimate names (Gal·la), so it can only separate when flanked by CJK on both sides. Separator classification is currently per-character and context-free; this adds the first context-sensitive rule.

Out of scope: the space-separated form. parse("威廉 莎士比亚") also reads family-first today, but nothing distinguishes it by script from a native name like 毛 泽东, and family-first is right for the overwhelmingly more common native case. That limitation should be documented, not fixed.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions