[Lexer] Add Unicode identifier and whitespace recognition#23

Open

kuilpd wants to merge 9 commits intodil-mainfrom

DIL-add-utf8-identifiers

Commits on Feb 24, 2025

Commits on Feb 25, 2025

Refactor code & add unicode whitespace skipping
kuilpd
committed

Commits on Feb 27, 2025

Add test for an identifier with a right-to-left script
kuilpd
committed

Commits on Mar 3, 2025

Commits on Mar 7, 2025

Manually count identifier and whitespace width using charWidth
kuilpd
committed

Commits on Mar 10, 2025

Remove Unicode whitespaces recognition
kuilpd
committed

Commits on Mar 11, 2025

Replace unicode codepoint conversion with UTF8 validity check
kuilpd
committed