need a version of encode_base64 and decode_base64 that handles utf8mb4

[mgage]

[mgage]

I've found that encoding things in base64 and then decoding them causes a known problem.
See #932. It causes things like knowls to fail (which means that Solutions also fails). Most likely requires changes in both perl encode/decode and javascript encode/decode.

The standard MIME::encode_base64 which we use does not handle multibyte characters.

Here are some references about perl. The last looks like it should give a good solution but might require new versions of perl.

• https://perldoc.perl.org/MIME/Base64.html
• https://www.perlmonks.org/?node_id=1180100
• https://metacpan.org/pod/distribution/Encode/lib/Encode/PerlIO.pod

for javascript we have the same problem

• https://developer.mozilla.org/en-US/docs/Web/API/WindowBase64/Base64_encoding_and_decoding
• https://developer.mozilla.org/en-US/docs/Web/JavaScript/Base64_encoding_and_decoding
• https://stackoverflow.com/questions/26721893/convert-buffer-base64-utf8-encoding-node-js
  -(seems to indicate that Buffer.from() is the command to investigate but this may only work for node.js and not javaScript)

[taniwallach]

lib/WeBWorK/Utils.pm already has new functions

• decode_utf8_base64
• encode_utf8_base64
  which are intended to make sure that base64 encoding is being applied to utf8 octet sequences and not to sequences of Perl characters which can be UniCode characters which require more than a single byte. These functions seem to be widely used in the webwork2 codebase of Develop multi ling 03 14 2019 mb4 #932 .

@mgage - I have at least one problem I translated from the OPL to Hebrew has a solution, and I don't recall any problem with the solution, but I don't think I have any other use of knowls. Using an admin user, I can certainly open and close the solution. Please explain more about what the problem is, and how it is triggered.

Looking for Perl code which still uses the standard encode_base64 or decode_base64 I found only a few such isolated uses, none of which seem to be very likely to be related to what you mentioned. The files where these non-utf8 base64 functions are still in use are:

• clients/hello_world_apps/webwork_xmlrpc_client.pl
• clients/hello_world_apps/webwork_soap_client.pl
• clients/uribase64_encode.pl
• lib/WebworkWebservice/MathTranslators.pm (for texString in sub tex2pdf)
• lib/WeBWorK/ContentGenerator.pm (in sub feedbackMacro_form)
• lib/FormatRenderedProblem.pm (for WARNINGS)

About use of base64 inside the JavaScript code, I looked at the references you mentioned, and there is certainly a potential problem. However, I am not sure where WW is making use of base64 encoding in the JavaScript code, and to what extent such code already has Unicode support.

On the other hand, while looking into this now, I think there may be a possible UTF8 support issue with

• nfreeze_base64
• thaw_base64
  which do not seem to encode/decode strings into/from utf8 so that base64 is only be used to code/decode sequences of octets. These functions seem to only be used by "Achievements":
• lib/WeBWorK/AchievementItems.pm
• lib/WeBWorK/AchievementEvaluator.pm
  and it is quite possible that no one has actually used the achievement code in a non-English course where utf8 support would be needed here.

[mgage]

Here is what it looks like. I can't reproduce this in all of the different sites I have running. This is picture is from within docker using the original db instead of the mb4 db. That could be the problem.

The point where I thought base64 was involved is at https://github.com/openwebwork/pg/blob/develop/macros/PGbasicmacros.pl#L1288 through line 1315. PGcore uses the MIME::Base64
routines which according to what I've read don't handle multibyte characters.

It might be a few days before I get back to this so let me know if you find anything out. What you've listed here is already useful.

[taniwallach]

I don't think the encoding inside the database can have any such effect on the rendering of text from the PG files.

Sorry, I did not look at the PG side of things before...

From what I see in pg/.lib/PGcore.pm the encode_base64() function is already calling encode_utf8() before it calls MIME::Base64::encode_base64() so it should work fine if the solution string was a string of valid Unicode characters before it got sent to encode_base64().

I suspect that the specific PG file is not really valid UTF-8, but that Perl can handle it properly when it does not need to pass the string through encode_utf8, but when this string gets passed via encode_utf8 the invalid Unicode is making trouble.

Notice that the garbled characters are from an accented character but the next two characters were also "swallowed up" in the base64 encode/decode (which called encode_utf8 on the way).

Is it possible that the accented characters making trouble are actually encoded in this PG file in their latin-1 8-bit location (with the high bit set)?

If I recall, Perl can treat such issues laxly when ":utf8" is used, but not when the stricter "UTF-8" via the Encode module is used.

Maybe post (and email me a copy) of the specific PG file itself.

[mgage]

it's possible that it is not utf-8 but I've been retyping the accented characters directly into the PGproblemeditor using the spanish keyboard provided with my macOS.

I've attached the file -- you'll have to change the extension to .pg to use it.

testspanish.txt

[mgage]

my version of PGcore doesn't appear to use encode_utf8 or decode_utf8

Am I missing some pull requests?

[mgage]

Adding:

sub decode_base64 ($) {
	my $self = shift;
	my $str = shift;
	#MIME::Base64::decode_base64($str);
	decode_utf8( MIME::Base64::decode_base64($str) );
}

sub encode_base64 ($;$) {
	my $self = shift;
	my $str  = shift;
	my $option = shift;
	#MIME::Base64::encode_base64($str);
	MIME::Base64::encode_base64( encode_utf8($str) );
}

to PGcore has fixed the problem. I'll update the repos at github.com/mgage/pg (branch develoop_.....03.......) in a couple of hours. I have some errands first.

[taniwallach]

my version of PGcore doesn't appear to use encode_utf8 or decode_utf8

Am I missing some pull requests?

@mgage - Probably due to juggling too many things, you were running a version of pg without the localization support in your Docker setup at the time you had this problem.

The relevant change to lib/PGcore.pm date back to commit d2d636c1f79a66ccd0d719c9f576db0820a041d6 by @goehle from July 2016. This is one of the oldest commits in the localization support, and seems to be in all the pull requests for utf8 support I could find.

• UTF8 Support pg#278
  • openwebwork/pg@d2d636c
• Develop utf8 ver2 pg#319
  • openwebwork/pg@d2d636c
• Develop international pg#367
  • openwebwork/pg@d2d636c
• New develop candidate multilingual pg#390
  • openwebwork/pg@d2d636c
• Develop multi ling 03 10 2019 no mb4 pg pg#402
  • openwebwork/pg@d2d636c
• Multilingual pg#404
  • openwebwork/pg@d2d636c

[mgage]

You were right Tani I had an old copy of pg (I believe the master branch) in my docker box. Using any of the precursors to multilingual I don't see any problem with spanish language or chinese language characters.

I'll close this issue but open a new one listing the remaining places to check that base64 translations are handled correctly.