Investigate WWSafe.pm vs. Perl's standard Safe.pm - UTF-8 motivation

[taniwallach]

WeBWorK's lib/WWSafe.pm is based on $Safe::VERSION = "2.16" and does not allow the use of the Perl Encode module fully inside the processing of PG files. Perl 5.24.1 has $Safe::VERSION = "2.39"; and that version has additional code to enable the Perl Encode module to work better within "Safe". Updating lib/WWSafe.pm will require great care and expertise.

As a result some code of the limitation of lib/WWSafe.pm some parts of PG use <:utf8 instead of the stricter and safer <:encoding(UTF-8).

See the discussions at:

• New develop candidate multilingual pg#390
• Develop multi ling 03 10 2019 no mb4 pg pg#402
• Develop multi ling 03 10 2019 no mb4 #930

Somewhat related:

• New develop candidate 01 01 2019 #908

Files (in pg):

• lib/WeBWorK/PG/IO.pm
  • read_whole_file reads a file in raw mode and then uses utf8::decode($string);
  • we would prefer to read the file using <:encoding(UTF-8)
  • there is a comment about Safe not letting <:encoding(UTF-8) work
• lib/PGloadfiles.pm
  • compile_file reads a file in raw mode and then uses utf8::decode($string);
  • we would prefer to read the file using <:encoding(UTF-8)
• lib/WeBWorK/PG/Translator.pm
  • use binmode(STDOUT, ":utf8")
  • and not something like binmode(":encoding(UTF-8)")
  • or binmode(INPUT,":encoding(UTF-8)");
• lib/PGcore.pm
  • uses binmode(STDOUT, ":utf8");
  • and not something like binmode(":encoding(UTF-8)")

[taniwallach]

I think that this has been ignored for too long to want to consider it as something we are going to handle for WW 2.16.
@mgage @drgrice1 - What do you think?

[drgrice1]

That is probably best unless someone has the time to work on it.

[drdrew42]

I am currently in the process of resolving this (related?) issue with the Safe module. I believe that I have successfully resolved the issue itself (confirmed that PGCore, Translator, PG::Local, CourseEnvironment, myriad MOs, etc... are destroyed), but it seems that our code is dependent on the fact that this bug exists (i.e. our code uses references to certain contents of the safe after it is 'supposed' to have been cleaned).

The aforementioned issue with Safe is still present in the latest version, and the upgrading of Safe will require additional attention. This is also on my list, and I believe I have a handle on what will be required (at least initially).

I will be continuing my effort on this task, and I really hope to have some pushable code before our anticipated release candidate.

[taniwallach]

I am currently in the process of resolving this (related?) issue with the Safe module. I believe that I have successfully resolved the issue itself (confirmed that PGCore, Translator, PG::Local, CourseEnvironment, myriad MOs, etc... are destroyed), but it seems that our code is dependent on the fact that this bug exists (i.e. our code uses references to certain contents of the safe after it is 'supposed' to have been cleaned).

The aforementioned issue with Safe is still present in the latest version, and the upgrading of Safe will require additional attention. This is also on my list, and I believe I have a handle on what will be required (at least initially).

I will be continuing my effort on this task, and I really hope to have some pushable code before our anticipated release candidate.

@drdrew42 You are a brave man. I still recommend we leave this aside, at least until we have a release candidate ready for @Alex-Jordan to install in about 2.5 weeks. Any changes to Safe / WWSafe can be considered for later addition before we advance to a final release, but will certainly require extensive testing. That is why I changed the project and the mile-stone. If sufficient progress is made after the release candidate is ready - we can change them back.