Script data is missing combining marks in some scripts

[moyogo]

The script data is missing combining marks in some scripts. For example in Latin, static-lang-word-lists/data/diffenator/Latin.txt only have composed characters and doesn’t contain any combining marks but plenty are used in practice. For example p̱ is used Uduk, where U+0331 COMBINING MACRON BELOW may be below low composed characters.

[RickyDaMa]

Hey Denis,

The word list you highlight originates from the diffenator2 repo. I wonder if perhaps Wikipedia normalised to composed characters (which is the root source for the diffenator word lists), but I don't know for certain. @m4rc1e or @simoncozens may know more as they were the people behind these word lists.

I'd be happy to update the word lists here if it turns out the post-processing that's done (and there is a fair amount done to get the word lists down to a manageable size) removes them inadvertently.

[moyogo]

Wikipedia normalizes to NFC, which doesn’t discard combining marks.