Develop multi ling 03 10 2019 no mb4

[mgage]

This replaces develop_candidate_02_24_2019 PR #927 . This has additional tweaks to
encode(UTF-8) and :utf8 layers for input/output. It is set up so that while it uses utf8 it
does not the force the usage of utf8mb4 so it can work with mysql older than 5.3 with
legacy courses. Once we require utf8mb4 the code will no longer be compatible with
older versions of mysql/mariadb.

The reading of the PG version has been tweaked so that it is read directly from the file pg/VERSION.
The file webwork2/PG_VERSION is no longer used. I think this will improve compatibility with docker on some machines where linking PG_VERSION to pg/VERSION caused troubles.

[taniwallach]

@mgage I think that the substitution $data =~ s/\r\n?/\n/g; which is run before the data is converted from a string of bytes which may be "raw" UTF8 to a string of characters may be what is sometimes corrupting UTF8 files uploaded in "Text" format.

[mgage]

It's actually been working for me. I'm pretty sure however that $type is never equal to 'Text' in my examples. The files uploaded have been identified in binary when the mode is automatic. (you reported failures with the automatic mode, but I didn't see that. ) The isText subroutine has been experimented with for other reasons (in general the current hacks for testing for text need to be rethought with the advent of ift8mb4)

sub isText {
my $string = shift;

#	return $string !~ m/[^\s\x20-\x7E]{4}/;
return utf8::is_utf8($string);
# return $string !~ m/[\x00-\x08\x0B\x0C\x0E-\x1F\x7F]{2}/;

}

Is what I have in my current copy of FileManager.pm

If I force uploading in text mode I got errors but forcing the print command to use :utf8 fixed the problem. (open(UPLOAD,">:encoding(UTF-8)",$file)).

[taniwallach]

Regarding FileManager uploads mangling UTF8 files:

1. I apologize - I tested again and "automatic" is not causing trouble. I probably confused the file sizes. The bigger file size is the damaged file and the smaller file was the good file. I think I may have accidentally assumed the opposite.

2. I did some testing on a very small file I called small_utf8.pg which has just the special characters from the file I was testing with separated by a blank line. The file contents were:

é

¿

á

ó

á

ñ

When uploaded in the 2 different modes - the resulting file had different sizes:

-rw-r--r-- 1 tani tani 23 Mar 14 22:39 small_utf8_uploaded_binary.pg
-rw-r--r-- 1 tani tani 35 Mar 14 22:39 small_utf8_uploaded_text.pg

The original file had 23 bytes and was the same as the result of the binary upload.

Using Unix od we can see the data in each file.

> od -a -t x1 small_utf8_uploaded_binary.pg

0000000   C   )  nl  nl   B   ?  nl  nl   C   !  nl  nl   C   3  nl  nl
         c3  a9  0a  0a  c2  bf  0a  0a  c3  a1  0a  0a  c3  b3  0a  0a
0000020   C   !  nl  nl   C   1  nl
         c3  a1  0a  0a  c3  b1  0a
0000027

> od -a -t x1 small_utf8_uploaded_text.pg 

0000000   C etx   B   )  nl  nl   C stx   B   ?  nl  nl   C etx   B   !
         c3  83  c2  a9  0a  0a  c3  82  c2  bf  0a  0a  c3  83  c2  a1
0000020  nl  nl   C etx   B   3  nl  nl   C etx   B   !  nl  nl   C etx
         0a  0a  c3  83  c2  b3  0a  0a  c3  83  c2  a1  0a  0a  c3  83
0000040   B   1  nl
         c2  b1  0a
0000043

Notice how the six 2-byte special characters were each replaced by a 4-byte sequence. That added 12 extra bytes to the files, and when I try to edit the mangled file I see:

é

¿

á

ó

á

ñ

which corresponds to the sort of manged text seen in the mangled version of the original pg file.

3. I think the issue is that the uploaded data is a sequence of bytes (octets), so it should not be saved using either

• >:utf8 as in
  • if (open(UPLOAD,">:utf8",$file)) {print UPLOAD $data; close(UPLOAD)}
• or >:encoding(UTF-8) as in
  • if (open(UPLOAD,">:encoding(UTF-8)",$file)) {print UPLOAD $data; close(UPLOAD)}
• as those are meant for when the Perl string is already a sequence of Unicode characters which need to be converted to a sequence of UTF-8 octets.
• Instead it should just be written out to a file "as is" using the old fashioned:
  • if (open(UPLOAD,">$file")) {print UPLOAD $data; close(UPLOAD)}
• which should not modify the sequence of bytes.
• For me, that change leads to text mode giving an unmangled file.

4. The question remains whether the substitution to change DOS/Mac line-endings to Unix line-endings can mess up multi-byte characters.

• According to https://en.wikipedia.org/wiki/UTF-8 multi-byte UTF-8 characters have the high-order bit set in all the bytes, so bytes with the high-order bit off must be an ASCII character in the regular location.
• In particular, the octet \r = CR (Carriage return, 0x0D, 13 in decimal) cannot appear in a valid UTF-8 file with any meaning other than carriage return.
• Thus, I conclude that the line-end substitution cannot cause any harm.
• Sorry, I did not do my homework before raising the suspicion that the line-end substitution might be related to the problem.

[taniwallach]

I'm repeating much of what I just posted at #915 as it bears on this issue also.

I did some more reading/review about Perl and Unicode.

In short:

• Any "text" which arrives from outside Perl should be "decoded" (from whatever transfer encoding it was sent in) and thereby converted to a sequence of Perl characters (a "text" string) when a Perl "text string" is needed.
• Any "text" being sent out of Perl should be "encoded" from a sequence of Perl characters (a "text" string) to whatever transfer encoding is expected on the other side of the line.

When FileManager.pm is handling a file upload, the upload data is just a sequence of bytes from the file being uploaded. Those bytes should essentially just be written to the file being created. When the data is either in an 8-bit encoding or in UTF-8 encoding, the line-end substitution code is safe (as explained in the prior comment), but no encoding conversion code should be applied.

Bottom line - for this situation we want: if (open(UPLOAD,">$file")) {print UPLOAD $data; close(UPLOAD)} without any encoding layer involved.

References:

• https://mojolicious.org/perldoc/perlunitut
• https://mojolicious.org/perldoc/perlunifaq
• https://mojolicious.org/perldoc/perluniintro

[taniwallach]

@mgage

Maybe we can add a setting in defaults.conf which controls whether the (problematic) 3-byte max mysql utf8 charset is used instead of the full 4-byte max utf8mb4. (This difference exists only at the SQL level: in terms of the charset mySQL is storing, and in terms of the charset for the communications between WW and the SQL server).

Then anyone who needs to support a (old) version of mySQL which does not support utf8mb4 can flip the setting, while modern installations can just use utf8mb4 and avoid any possible issues with 4-byte characters?

[taniwallach]

Should this be ":encoding(UTF-8)" with a colon?

[mgage]

Yes. Believe there is supposed to be a colon in front of encoding.

I agree about creating a switch. The places I've identified where utf8mb4 is specifically mentioned and causes trouble for my old mysql installation (pre 5.3) which can't handle mb4 are:

• set up utf8mb4 switch???
• control line 95,99 in OPL-update
• LocationAddresses.pm ine 31
• SQL.pm line 272

The use of encoding(UTF-8) above didn't cause problems.

[taniwallach]

Also:

• lib/WeBWorK/Utils/ListingDB.pm line 519 (SET NAMES for DB connection).
• bin/update-OPL-statistics - line 85 (CHARACTER SET utf8mb4 vs. ascii)

I think all the changes can be found in the following 2 commits from my tree.
taniwallach@b840e28
taniwallach@472e837
from March 6 2019 commits to branch: https://github.com/taniwallach/webwork2/commits/tani_develop_candidate_02_24_2019_utf8mb4

Note the column sizes in the various database tables which exceed the index limit for utf8mb4 encoding can be made smaller already, which may save trouble for relatively new tables when conversion needs to be done.

[mgage]

I'm closing this since nearly everything in this pull request is also in #932 and can be made compatible with early versions of mysql by setting $ENABLE_UTF8MB4=0 in site.conf. The one set of files missing is the themes/math4-ar directory. Can someone remind me what this directory contains? Is it still needed?

[taniwallach]

The one set of files missing is the themes/math4-ar directory. Can someone remind me what this directory contains? Is it still needed?

It was an early attempt (by @heiderich) to allow using Arabic and the right-to-left text direction in a course, but required a special set of templates. The main change in math4-ar was setting lang="ar" dir="rtl" in the main HTML tag of 4 files.

This was replaced in 5758400 (and e4b6358) by a modification of the main math4 files to use a new method to add a lang and when needed a dir tag as needed based on the language set for the course: <!--#output_course_lang_and_dir-->

That approach is currently supporting Hebrew courses, should support Arabic also with dir=rtl set, and other course languages without triggering right-to-left text direction. Adding support to turn on RTL mode for other right to left languages can be done in the future by adding suitable cases to the output_course_lang_and_dir method in lib/WeBWorK/ContentGenerator.pm.