UTF-8 changes

[taniwallach]

• Replace
  • encode_utf8($x) by Encode::encode("UTF-8",$x), and
  • decode_utf8($x) by Encode::decode("UTF-8",$x)
    as the prior is not a proper conversion, and the documentation at https://perldoc.perl.org/Encode#encode_utf8
    state that it should not be used for data exchange.
• In several places use Encode::encode() explicitly, instead of just encode() after a use Encode qw(encode).
• Replace binmode(STDOUT, ":utf8"); with binmode(STDOUT, ":encoding(UTF-8)"); as Perl 5.30 makes sysread and syswrite on ":utf8" filehandles fatal (it was deprecated in 5.24).
  See: https://perldoc.perl.org/perl5300delta#Previously-deprecated-sysread()/syswrite()-on-:utf8-handles-is-now-fatal

So far I have not thoroughly tested the changes.

Motivation:

• UTF-8 email issue discussed at https://webwork.maa.org/moodle/mod/forum/discuss.php?d=4916 and the issue at Problems with accented characters in feedback emails on some systems #1144 related to that post.
• Error messages I found in my Apache error.log files:

syswrite() is deprecated on :utf8 handles. This will be a fatal error in Perl 5.30 at /usr/share/perl/5.26/sigtrap.pm line 86.
Caught a SIGsyswrite() is deprecated on :utf8 handles. This will be a fatal error in Perl 5.30 at /usr/share/perl/5.26/sigtrap.pm line 87.
PIPEsyswrite() is deprecated on :utf8 handles. This will be a fatal error in Perl 5.30 at /usr/share/perl/5.26/sigtrap.pm line 88.
 at syswrite() is deprecated on :utf8 handles. This will be a fatal error in Perl 5.30 at /usr/share/perl/5.26/sigtrap.pm line 90.
-esyswrite() is deprecated on :utf8 handles. This will be a fatal error in Perl 5.30 at /usr/share/perl/5.26/sigtrap.pm line 91.
 line syswrite() is deprecated on :utf8 handles. This will be a fatal error in Perl 5.30 at /usr/share/perl/5.26/sigtrap.pm line 92.
0syswrite() is deprecated on :utf8 handles. This will be a fatal error in Perl 5.30 at /usr/share/perl/5.26/sigtrap.pm line 93.
syswrite() is deprecated on :utf8 handles. This will be a fatal error in Perl 5.30 at /usr/share/perl/5.26/sigtrap.pm line 119.

[drgrice1]

This looks good. I am not able to do any extensive testing on this as my usage rarely deals with anything that will really stress test the utf8 encoding and decoding.

I noticed that there is one instance of encode_utf8 that you did not convert to the Encode::encode("UTF-8", ...) method, and this is in lib/WeBWorK/EquationCache.pm. It may be it is not needed there. Just pointing this out.

Also, both Encode::decode_utf8 and Encode::encode_utf8 are used in PGcore.pm in the PG code. binmode(STDOUT, ":utf8") is also used there. Do those need to be changed also?

[taniwallach]

I noticed that there is one instance of encode_utf8 that you did not convert to the Encode::encode("UTF-8", ...) method, and this is in lib/WeBWorK/EquationCache.pm. It may be it is not needed there. Just pointing this out.

I'll look into that one soon.

---

Also, both Encode::decode_utf8 and Encode::encode_utf8 are used in PGcore.pm in the PG code. binmode(STDOUT, ":utf8") is also used there. Do those need to be changed also?

Ideally yes, practically - probably not now.

WWSafe interferes with Encode::encode("UTF-8", ...).

• See Investigate WWSafe.pm vs. Perl's standard Safe.pm - UTF-8 motivation #982 and the older discussions it references, and
  -the comment at https://github.com/openwebwork/pg/blob/a284c3b31a26fb316c2fa6b34246df8ba2895a3e/lib/WeBWorK/PG/IO.pm#L162
• As a result, several compromises exist in PG.
• Apparently, more recent versions of Safe.pm have approaches to allow Encode to pre-load and work, but WWSafe was forked of an earlier version.
• That was the reason I once discussed with @mgage how feasible it would be to update WWSafe or go back to Safe.pm

[drgrice1]

I didn't see that issue.

[taniwallach]

I noticed that there is one instance of encode_utf8 that you did not convert to the Encode::encode("UTF-8", ...) method, and this is in lib/WeBWorK/EquationCache.pm. It may be it is not needed there. Just pointing this out.

I'll look into that one soon.

That is pg/lib/WeBWorK/EquationCache.pm and our experience is that WWSafe does not let Encode::encode("UTF-8", ...) work properly in PG.

The use is in my $md5 = md5_hex(encode_utf8($tex)); so it is only using Perl's lax UTF-8 to encode a string which is them passed into md5_hex. I do not think that any of the potential security risks related to Perl's lax UTF-8 code can cause any significant problems in such a setting.

---

From some research 2 years ago, the issues were related to the autoloading mechanisms used by Encode and that more modern versions of the regular Perl Safe.pm had changes to allow Encode to behave properly inside Safe.

See:

• New develop candidate multilingual pg#390 (comment)
• Investigate WWSafe.pm vs. Perl's standard Safe.pm - UTF-8 motivation #982
• openwebwork/pg@3d94218
• https://github.com/openwebwork/pg/blob/a284c3b31a26fb316c2fa6b34246df8ba2895a3e/lib/WeBWorK/PG/IO.pm#L162
• Develop multi ling 03 10 2019 no mb4 pg pg#402 (comment)

[drgrice1]

Yeah, that probably is fine. Furthermore, does anyone even use images for equations anymore? MathJax is so much better.

[mgage]

There are situations where it is a good backup. Some labs still don’t enable javaScript. I periodically run into students who for one reason or another can’t get the javascript to work on their computer/browser. At the very least the image provides a check that everything else is working.

…

On Mar 11, 2021, at 6:36 PM, Glenn Rice ***@***.***> wrote: Yeah, that probably is fine. Furthermore, does anyone even use images for equations anymore? MathJax is so much better. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openwebwork_webwork2_pull_1154-23issuecomment-2D797122215&d=DwMCaQ&c=kbmfwr1Yojg42sGEpaQh5ofMHBeTl9EI2eaqQZhHbOU&r=C6Pt5AGtImanmAdcooarL-JZO8M5dSFPfs3VweYXYkE&m=Eylhgnhfnva3Zl2gyQGOsKkZSlqxKPJGigtvsHP3w-k&s=AFcMhbNPLl5QUNSYt5KzMODZAnzmYSILlmkiVT73U2w&e=>, or unsubscribe <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AAJF27HTTBIRJ65XNFN65H3TDFAXXANCNFSM4TH5OIIA&d=DwMCaQ&c=kbmfwr1Yojg42sGEpaQh5ofMHBeTl9EI2eaqQZhHbOU&r=C6Pt5AGtImanmAdcooarL-JZO8M5dSFPfs3VweYXYkE&m=Eylhgnhfnva3Zl2gyQGOsKkZSlqxKPJGigtvsHP3w-k&s=enU-Fz7K7e9aCS9_nkaNoGagxFzOZq5DgxlM5jUX6aY&e=>.

[drgrice1]

Yeah, I am not saying to remove the code for that. I am just saying it is probably not used much, and as such, it is probably not a serious concern if utf8 encoding is proper (if that is even needed for equations in any case).

I am betting that the instances in which students have trouble with the javascript working are becoming increasingly infrequent (at least in regards to MathJax). I have never had that happen for my students as far as I know.

[taniwallach]

I periodically run into students who for one reason or another can’t get the javascript to work on their computer/browser.

I have run into a few such cases. Often asking the student to use a different browser than their default browser will overcome the problem. A few cooperative students who disabled browser extensions/plugins and bothered to respond after trying that reported success with extensions/plugins disabled. (There are several types of plugins which block content or modify content in ways the the user's don't really understand, but have some feature they like.) I have never got a student to test enough and write back telling me which extension/plugin caused their problem. The students typically just want to do the homework and move on.

[taniwallach]

if utf8 encoding is proper (if that is even needed for equations in any case).

Submitted answers can contain UTF-8 character (even if they are not suitable), and almost everything gets rendered inside a math element in the "submitted answers" column in the results table. When using images equation display mode without the UTF-8 encoding an error page would be displayed (see the details in openwebwork/pg#443 (comment)), as the MD5 hash used to create file names cannot process Unicode characters from the internal Perl encoding of strings. The patch was made to encode into UTF-8, and the the image fails to be generated as LaTeX can't actually make the image so also fails appear in the rendered image, but the tool-top text does work. Having something fail to appear on screen, is far better than there being a WeBWorK error page.