Skip to content

fix(encodings): use Cyrillic combining macron for long ā in Russian scheme - #8

Open
tym83 wants to merge 1 commit into
kosperun:masterfrom
tym83:fix/rus-long-a-combining-macron
Open

fix(encodings): use Cyrillic combining macron for long ā in Russian scheme#8
tym83 wants to merge 1 commit into
kosperun:masterfrom
tym83:fix/rus-long-a-combining-macron

Conversation

@tym83

@tym83 tym83 commented Jul 13, 2026

Copy link
Copy Markdown

Summary

Fixes #7. Long ā / Ā pasted from Russian VedaBase was not converted to Gaura Times, and the macron drifted onto the following letter.

Root cause

Russian VedaBase writes long ā as Cyrillic а / А (U+0430 / U+0410) + COMBINING MACRON (U+0304) — Cyrillic А has no precomposed macron form in Unicode. The RUS table stored positions 0 and 37 as the Latin look-alikes ā / Ā (U+0101 / U+0100), so real input never matched: the Cyrillic base passed straight through and the orphaned macron was left floating over the neighbouring letter.

Every other special letter in RUS is already stored as Cyrillic base + combining mark (e.g. т̣, ш́, х̣), which is exactly why only long ā was affected — И and У have precomposed Cyrillic macron forms (Ӣ, Ӯ), but А does not, so the table fell back to Latin.

Changes

  • src/encoding_mappings.pyRUS positions 0 and 37 now use the Cyrillic base + combining macron that VedaBase actually emits, with a comment documenting the codepoints and the reason.
  • tests/test_services.py — regression tests built from verbatim vedabase.io/ru strings, asserting that no combining marks survive a RUS → Gaura Times conversion, plus a RUS ↔ IAST round-trip for long ā.

Testing

All 90 tests pass (86 existing + 4 new). The existing round-trip tests are unaffected because they compare against the tuple itself.

Deliberately out of scope (follow-ups noted in #7)

  • An NFC normalization pass in convert() to make matching robust to combining-mark order (e.g. , whose order in the table is non-canonical) and to NFD-form sources. This is a broader change and is left for separate discussion.
  • The Cyrillic (Ukrainian) UKR table appears to have the same Latin-vs-combining gap on long ā (positions 0 and 37) and is worth checking — not touched here to keep this PR focused on the reported bug.

…cheme

Russian VedaBase writes long "ā"/"Ā" as Cyrillic "а"/"А" (U+0430/U+0410)
followed by a COMBINING MACRON (U+0304), because Cyrillic А has no precomposed
macron form in Unicode. The RUS table stored these as the Latin look-alikes
"ā"/"Ā" (U+0101/U+0100), so real VedaBase input never matched: the Cyrillic
base passed through unchanged and the orphaned macron was left floating over
the neighbouring letter.

Store positions 0 and 37 as the Cyrillic base + combining macron that VedaBase
actually emits. Add regression tests using verbatim vedabase.io/ru strings that
assert no combining marks survive a RUS -> Gaura Times conversion.

Fixes kosperun#7.

Signed-off-by: tym83 <6355522@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

a with diacrtitic from Cyrillic Russian to Gaura Times on Linux and Windows

1 participant