UTF8 issue with permissions saved to simple.conf

[taniwallach]

There is an issue with the code of the permission editor / the code which writes to simple.conf which is triggered when changing between languages.

(The issue was mentioned in #948 (comment) on Aug 24, 2019 but was not a reason to hold back on the WW 2.15 release.)

The problem is not fully diagnosed, but under certain circumstances the value save for $permissionLevels{type} is being set using the foreign language value rather than the English value, which breaks the behavior.

A sample procedure to trigger the issue is as follows:

1. Set simple.conf in some course to be the contents of the file provided at the bottom of the issue.
   • Maybe use a different value of the language (which has translated names of the permission levels) instead of Hebrew so you can follow the steps below with more ease.
   • However, Hebrew definitely triggers the problem.
2. Use the menu to open the course configuration page - two lines above the ?-mark icon in the menu. (Hebrew phrase is תצורת הקורס, in the current PO file)
3. Change the language setting from "heb" to "en_us".
4. Click the save button on the bottom of the page.
5. Example the value of $permissionLevels{change_password} in the saved simple.conf. It will show (based on my tests) the value מרצה which is the Hebrew term for professor in use.
6. Use the left menu to change to a different part of the system and them back to the configuration page. That will get the language change to take effect.
7. Now open the "Permissions" tab again. It will show the value guest as the value מרצה from the file cannot be read in.

---

Note: When the permissions are edited and saved while using a foreign language, and the language setting is not changes, the values saved in simple.conf are the proper English names of the relevant permission levels. The problem is apparently only triggered when a save is occurring which changes from a foreign language back to English

---

Note: In the pull-down lists, the <option> items have a value which is the standard English name of the permission levels, but the text shown for the options is the translated phrase. When the permissions page is saved in a foreign language - the form values sent are the English values from the value field.

---

The probably cause is a call to maketext which is not needed or some value already saved as the "translated" value rather than the English value.

---

A prior issue with related code is at: #798 (comment)

---

The sample simple.conf used to trigger the issue

#!perl
# This file is automatically generated by WeBWorK's web-based
# configuration module.  Do not make changes directly to this
# file.  It will be overwritten the next time configuration
# changes are saved.

$language = 'heb';
$permissionLevels{change_password} = 'professor';

---

The value of simple.conf after step 5:

#!perl
# This file is automatically generated by WeBWorK's web-based
# configuration module.  Do not make changes directly to this
# file.  It will be overwritten the next time configuration
# changes are saved.

$language = 'en_us';
$hardcopyTheme = 'oneColumn';
$permissionLevels{change_password} = 'מרצה';

[taniwallach]

A side effect of this problem, is that when simple.conf has an invalid (non-English) value for $permissionLevels{change_password} the Classlist Editor does not display the "Password" tab.

[taniwallach]

Editing "Permissions" and saving removes/replaces the invalid setting from simple.conf - so this is a workaround which can help until the bug is fixed.

[taniwallach]

This bug triggers also without a language change... I just enabled MathQuill in a Hebrew course, and when the setting was saved - the permissions settings became corrupted with the same

$permissionLevels{change_password} = 'מרצה';

setting replacing the prior permissions setting.

This is probably a hint that the issue may be triggered whenever a "tab" of the settings are saved, except for the permissions tab. Saving the permissions tab does not trigger any corruption of simple.conf.

[cucucucumber]

Hi @taniwallach , I attempted to solve this issue by forcing UTF-8 encoding when reading and writing configuration files. I have tested the cases you mentioned above in Chinese and they worked just fine. But since I have scanty knowledge of Hebrew, more testings in Hebrew are still needed to see if the changes proposed really solve this issue.

[taniwallach]

Hi @taniwallach , I attempted to solve this issue by forcing UTF-8 encoding when reading and writing configuration files. I have tested the cases you mentioned above in Chinese and they worked just fine. But since I have scanty knowledge of Hebrew, more testings in Hebrew are still needed to see if the changes proposed really solve this issue.

@cucucucumber Thanks for working on this. It was a minor inconvenience - so did not get to the top of the to-do list. I will try to test your changes relatively soon. In the meanwhile I added some comments to your PR.

[robert-marik]

Hello, thank you for working on better UTF-8 support. Anyway, I think that the conf file should be independent on the language of the course (this means that the conf file should be in English). Also with current setting the roles are sometimes written in English, sometimes in Hebrew, depending on the fact which tab is saved. This is something which could be source of troubles for future.

[robert-marik]

To diagnose the problem consider the following.

1. Put warn $newval; and warn $inlinevarname; into the definition of the function convert_newval_source in the file Config.pm. The definition starts on line 57 and you should add these lines near the final return few lines below. This will show the values of the variables on the page.
2. Then switch to Hebrew (or perhaps any language which translates roles) and on various "tabs" in the config editor always save the configuration and check the warnings and the content of the simple.conf file. On the tab with roles you get English roles in the configuration. On the other tabs the roles are translated. (Perhaps with the possible exception of the first saving after language change).

I think that this proves that the problem is in the file Config.pm. The construction

        $newval = $self->comparison_value(eval('$self->{Module}->{r}->ce->'.
					       $inlinevarname));

takes the name of the variable and outputs the value of this variable translated to the local language. However it should return English string, since this is the string which is saved in the simple.conf file . If you save the tab with permission setting, the values are taken from the form and are in English. In this case the output is fine. However in the other cases the values are aparently taken from the eval function and IMO this is the source of the troubles. I do not understand this $self->{Module}->{r}->ce-> construction and was not able to find the right fix.

This code is 16 years old, this means before the i18n of WeBWorK.

[robert-marik]

I have another approach to this issue: just remove maketext from the variables to be saved. I believe that to keep the conf file in English is the correct approach. Without this patch the configuration is broken is someone changes the language of the course or if the translated strings change. And this may occur.

The patch robert-marik@939fe2b tries to do this. Did not test deeply yet though. The first thing is to make decision whether non-English terms in config file are fine or not. My vote is not, but there are may others, @taniwallach and others. Here is the explanation for my opinion to this: Localization is intended to help users, but sometimes the localization produces more harm than good. In localized software we many times fight with poor or unclear translation where the names of the strings do not match documentation. And you cannot use Google to find some hints, since the rest of the Internet speaks English.

Currently Hungarian, French and Hebrew translated these roles. The tests should be in English and one of these langauges.

The patch is simple but since the config file is and important part of WeBWorK I would ask skilled developers to look at it. No pull request yet.