Japanese characters in metadata on (u)pLaTeX

[aminophen]

I'm not sure this is the right place, sorry.

After loading \RequirePackage{pdfmanagement-testphase}, adding Japanese characters into PDF metadata does not work at all on (u)pLaTeX + dvipdfmx.

upLaTeX + dvipdfmx

Nowadays we use the following syntax.

\documentclass[dvipdfmx]{ujarticle}
\usepackage{hyperref}
\usepackage{pxjahyper}
\hypersetup{pdftitle={日本語}}
\begin{document}
\section{はじめに}
あいうえお。
\end{document}

If we add

\RequirePackage{pdfmanagement-testphase}
\DeclareDocumentMetadata
  {
    backend = dvipdfmx
  }

then an error happens:

! LaTeX3 Error: Invalid UTF-8 string: missing continuation byte (x3).

It seems that "l3str-convert" does not support Japanese characters. (maybe similar to latex3/latex3#939; you should consider better handing of Japanese tokens; just pass it as-is, literally untouched, is ok for us.)

pLaTeX + dvipdfmx

Nowadays we use the following syntax.

\documentclass[dvipdfmx]{jarticle}
\usepackage{hyperref}
\usepackage{pxjahyper}
\hypersetup{pdftitle={日本語}}
\begin{document}
\section{はじめに}
あいうえお。
\end{document}

If we add

\RequirePackage{pdfmanagement-testphase}
\DeclareDocumentMetadata
  {
    backend = dvipdfmx
  }

then an error happens:

! Package pxjahyper Error: The hyperref 'unicode' mode is not supported
(pxjahyper)                on the pTeX engine.

The error itself is natural enough, since pLaTeX engine cannot handle Unicode at all. Therefore, "hyperref" should not enable "unicode" mode on pLaTeX.

FYI: What the "pxjahyper" package does?

When using (u)pLaTeX, we need to give the correct "encoding conversion rule" to dvipdfmx by using pdf:tounicode special. The rule is provided in the ToUnicode CMap developed by Adobe or other contributers (for upLaTeX "UTF8-UTF16" or "UTF8-UCS2", for pLaTeX "EUC-UCS2" or "90ms-RKSJ-UCS2"), but we need to select the correct one depending on "which engine is running (pLaTeX or upLaTeX)" and "which encoding is used (mostly Shift-JIS on win32 / EUC-JP on Unix)". The "pxjahyper" package automatically does it.

[aminophen]

Handling Japanese characters on (u)pTeX is a big deal, due to complicated traditional encoding rules. It is almost impossible to support such a complicated conversion rule properly, even for us Japanese, without knowing the historical reasons. So, we recommend you "just leave it to us, without any conversion." The halfway conversion makes things more complicated, so pass it literally to us!

[u-fischer]

I'm not sure this is the right place, sorry.

It's fine, one can always move if needed.

It seems that "l3str-convert" does not support Japanese characters.

yes I guess it this problem, so we will have to solve that first.

The error itself is natural enough, since pLaTeX engine cannot handle Unicode at all. Therefore, "hyperref" should not enable "unicode" mode on pLaTeX.

Well the generic driver used by hyperref if the pdfmanagement is used is "generic", that means it doesn't contain any engine tests. And yes it force the unicode option. But the option mostly declares how hyperref writes out strings to the pdf or dvi, so I'm not sure why ptex cares about it.

Side question: which input encoding is used with ptex? utf8 or something else?

[aminophen]

But the option mostly declares how hyperref writes out strings to the pdf or dvi,
so I'm not sure why ptex cares about it.

The check is done by the pxjahyper package (not by pTeX), as the package knows it cannot work as expected when the strings are written out in unicode.

Side question: which input encoding is used with ptex? utf8 or something else?

It depends on users ;-) I think many users are writing in UTF-8 these days, but in some situations people still choose to write in Shift-JIS or EUC-JP for historical reasons. The pTeX engine accepts all of these encodings, by using -kanji=<enc> runtime option.

[josephwright]

@u-fischer Move to the latex3 repo?

[u-fischer]

The check is done by the pxjahyper package (not by pTeX), as the package knows it cannot work as expected when the strings are written out in unicode.

But strings are not written out "in unicode". I mean all in the PDF is ascii. hyperref writes out strings like (\376\377\000g\000r\000\374\000\337\000e). They may look curious, but they are ascii, so why does pTeX chokes on them?

[u-fischer]

@josephwright I think there is already an issue about str-convert in latex3, so this can stay here for now.

[FrankMittelbach]

But strings are not written out "in unicode". I mean all in the PDF is ascii. hyperref writes out strings like (\376\377\000g\000r\000\374\000\337\000e). They may look curious, but they are ascii, so why does pTeX chokes on them?

because they still are unicode only written in octal or not? Point is (I may be mistaken) if ptex doesn't support unicode but Shift-JIS or EUC-JP it probably translates to that on input, i.e. it is like inputenc translates to some LICR internally and then works with that. But your output isn't Shift-JIS it is unicode and so it chokes ... my rough guess

[u-fischer]

@FrankMittelbach but I'm writing that out in specials, so only the driver sees it. And this is dvips or something like that. Isn't it?

[FrankMittelbach]

maybe because of this?

When using (u)pLaTeX, we need to give the correct "encoding conversion rule" to dvipdfmx by using pdf:tounicode special. The rule is provided in the ToUnicode CMap developed by Adobe or other contributers (for upLaTeX "UTF8-UTF16" or "UTF8-UCS2", for pLaTeX "EUC-UCS2" or "90ms-RKSJ-UCS2"), but we need to select the correct one depending on "which engine is running (pLaTeX or upLaTeX)" and "which encoding is used (mostly Shift-JIS on win32 / EUC-JP on Unix)". The "pxjahyper" package automatically does it.

the Japanese char ends up in your octals and so the to UnicodeMapping stops working? (just guessing)

[davidcarlisle]

@u-fischer, @aminophen wil correct me if I'm wrong, but I think the old model for ptex was that the specials contained shift-jis or whatever in the special and the dvi driver did the conversion to Unicode in the final stage. Given that \ is Yen in Shift-jis I would guess things go badly wrong if \ -quoted octal utf-8 gets interpreted that way.

[aminophen]

the old model for ptex was that the specials contained shift-jis or whatever in the special and the dvi driver did the conversion to Unicode in the final stage.

Correct.

Given that \ is Yen in Shift-jis I would guess things go badly wrong if \ -quoted octal utf-8 gets interpreted that way.

"\ looks like Yen in shift-jis" is unrelated. The problem here is that "how the octal should be decoded." (using shift-jis? or euc-jp? or utf-8?). The encoding is pre-determined by (u)pTeX engine (not by a macro layer), so you cannot disable such a encoding conversion. Instead, you have to know "how (u)pTeX engine will encode it" and keep consistency with it.

[u-fischer]

@aminophen

I think I can remove the unicode settings from the generic driver as pdftex use now unicode by default anyway.
This will avoid the one error (! Package pxjahyper Error: The hyperref 'unicode' mode is not supported).

It will not solve everything as the driver doesn't always use hyperref commands for the conversion, so one will have to check if there are more wrong conversions somewhere, but this requires at first that l3str-convert works correctly with japanese.

[aminophen]

Unfortunately, Japanese devs found out that "it is almost impossible to support Japanese characters within the current behavior of pTeX, as long as l3str-convert uses \tl_to_str:n (= \detokenize) first."

As described in TUGboat article "Distinguishing 8-bit characters and Japanese characters in (u)pTeX" by H.Kitagawa, pTeX becomes confused among Latin and Japanese character tokens during "stringization". The "stringization" procedure occurs in \meaning, \message and \detokenize and related primitives, thus it becomes impossible to distinguish between Latin and Japanese characters after the token is processed by \detokenize. The proposed improvement of pTeX is quite reasonable, but the patches are so huge that they are not fully tested yet ...

The current behavior is considered unnatural today; however, at the time pTeX was designed and developed, Unicode 1.0 didn't even exist and 8-bit character inputs were rare, so the problem had never been exposed at all for years. Changing the behavior at this stage needs design change and requires lots of testing, so it would take much time.

[u-fischer]

Unfortunately, Japanese devs found out that "it is almost impossible to support Japanese characters within the
current behavior of pTeX, as long as l3str-convert uses \tl_to_str:n (= \detokenize) first."

The question is if and how one can sanitize the input first. That means if you get an input like

\tl_set:Nx \l_tmpa_tl {\text_purify:n{abcはじめにgrüße}}

can you convert \l_tmpa_tl into a form that one can safely fed to str_convert?

As described in TUGboat article "Distinguishing 8-bit characters and Japanese characters in (u)pTeX"

well I would say that while one can look for macro solutions, engine changes are really needed here ;-) But we quite understand that large changes aren't done fast.