[uchardet] Encoding detection error for Chinese document containing Chinese and Japanese text

[yfdyh000]

Notepad3 (x64) 6.26.227.1 beta

This is a condensed sample for a larger document.

Reproduction steps:
Enable the [Settings2] DevDebugMode=1 in Notepad3.ini. (Optional)
Save the following text to a file with Chinese GB18030. 寄生少女ゆうちゃん
Reopen the file with Notepad3.
Incorrect encoding and content. UCD='EUC-JP' Conf=95% (reliable (90%)) [Invalid UTF-8] - OS-CP='ANSI(CP-936) in the title bar.

This sometimes happens for files that contain something similar, but not always, depending on the content. e.g. the file with following text are correctly identified: 图书 寄生少女ゆうちゃん. But once you remove the "图" or "书" character, the error occurs.

EmEditor always does the correct and identifies the file as GB2312.
Any ideas to change?

[RaiKoHoff]

Inadequate/insufficient training of UCHARDET Encoding detection (Machine Learning, ML). The UCHARDET in NP3 is trained to find a lot more ANSI Encodings than standard UCHARDET training sets, unfortunately that reduces the probability for common encodings like GB2312.
Providing more characters in the document are sufficient to find the right encoding.
Best solution: do not use GB2312 but UTF-8 (https://utf8everywhere.org/zh-cn). It is by far superior over GB2312.
This also the reason, why we will not retrain the UCHARDET detector.

Ed.: If you must stay with GB2312 encoding, bypass UCHARDET detection by providing a encoding Tag encoding:GB2312 in the first line of the document and enable parsing of encoding tags:

[RaiKoHoff]

UTF-8 superior over GB2312:

[hpwamr]

Hello @yfdyh000 , see in Notepad3.ini the new Settings2 key-value:

UchardetLanguageFilter=31

;  Bitmask controlling which CJK charset probers are active in uchardet:
;    Bit 0 (0x01/1)  = Chinese Simplified (GB18030)
;    Bit 1 (0x02/2)  = Chinese Traditional (Big5, EUC-TW)
;    Bit 2 (0x04/4)  = Japanese (Shift_JIS, EUC-JP)
;    Bit 3 (0x08/8)  = Korean (EUC-KR)
;    Bit 4 (0x10/16) = Non-CJK (single-byte encodings)
;  Common values:
;    31 = All probers (default)
;    27 = Exclude Japanese (fix GB18030 misdetected as EUC-JP)
;    17 = Chinese Simplified + Non-CJK only
;     3 = Chinese only (simplified + traditional)

---

Feel free to test the "BETA PortableApps", version "Notepad3Portable_7.26.311.1_beta.paf" or newer, see 1st list in issue #1129.

• Notepad3Portable_7.26.311.1_beta.paf.exe.7z -s

"Notepad3Portable BETA PortableApps" version can be used with or without ".7z" extension.

Also, feel free to test the "BETA Setup", version "Notepad3_7.26.311.1_beta_Setup" or newer, see the 2nd list in issue #1129.

Comments and suggestions are welcome... 😃