Develop candidate 02 24 2019#927
Closed
mgage wants to merge 99 commits into
Closed
Conversation
… and mess up the database. Note: We do not need decode from thaw because as sequences of bytes nothing changes. (I think.)
…ch/webwork2 into locbug Conflicts: courses.dist/modelCourse/course.conf
… suggested by goehle
added [qw(Encode::Encoding)] to ${pg}{modules}) in defaults.config as…
…lop_uft8_ver2 # Conflicts: # lib/WeBWorK/ContentGenerator/Instructor/SendMail.pm # lib/WeBWorK/Utils.pm
On the other hand if the file contains a character (such as @) which is an invalid utf8 character then issue a warning to the html page. (Since course.conf often has an incorrect @ instead of © this results in an error for each course until those files are fixed.)
using © instead of © (the latter doesn't alwayswork with utf8 )
this change was undone in some some places during recent merges.
also have a "home directory" printout for debugging.
…pty_passwords_during_checkPassword PR 911+910+904 hotfixes to mgage new_develop_candidate_01_01_2019
…base use the utf8mb4 charset. The new file will be added to the running Docker MariabDB 10.1 container using changes to be made to docker-compose.yml in a following commit.
mysql_enable_utf8 ==> mysql_enable_utf8mb4 "SET NAMES 'utf8'" ==> SET NAMES 'utf8mb4'" modify the length of several VARCHAR columns so key length < 1000 as utf8mb4 reserves 4 bytes per character. These changes sufficed to allow (in a Docker test system): creating and using the admin course in a new MariaDB 10.0 database which was set to use utf8mb4 as the default charset, running OPL_update, creating a test course and creating a homework set with the question setMAAtutorial/hello.pg in it, creating a user whose name needs UTF8 characters, and submitting a UTF8 encoded string as an answer to setMAAtutorial/hello.pg and storing the UTF8 answer in the past_answer table. Note: The Docker test was done after the docker-compose.yml was modified, which is in the following commit.
database to use utf8mb4. This needs to be done BEFORE the Docker database storage volume is created. A special version called docker-compose.yml.specialUTF8MB4-db-storage can be used to replace docker-compose.yml, and will change the name of the database container, the database named storage volume, and make related settings. Replace docker-compose.yml with the file will force Docker to run the containers in a manner which will not intefere with the non-utf8mb4 database used by the prior configuration. That approach would allow one to switch between the use of Docker containers using a utf8mb4 database and using an old style database, just by using the "correct" settings in docker-compose.yml for each "case".
To allow the 1000 byte limit for keys, the OPL_local_statistics was
changed to use ENGINE=MyISAM. That change allowed reducing the
source_file column from varchar(255) to varchar(245), while the
767 bytes key limit of ENGINE=InnoDB as was used by this table and
by OPL_global_statistics.sql (at present) would have limited this to
about 191 characters.
The OPL_problem_user table had 3 keys changed from using 100 characters
each to using only 80 characters each. The same changes were already
made to lib/WeBWorK/DB/Record/PastAnswer.pm for the same reason. Only
the first 245 characters of source_file are used as a key. Finally,
the CHARACTER SET of OPL_problem_user was explicitly changed from ascii
to utf8mb4.
The saved version of OPL_global_statistics.sql needs to be hand modified
as follows, until the GitHub master branch has these changes made. Only
after those changes can the file be loaded into a utf8mb4 database.
utf8 => utf8mb4
255 => 245
ENGINE=InnoDB => ENGINE=MyISAM
=================================
10c10
< /*!40101 SET NAMES utf8mb4 */;
---
> /*!40101 SET NAMES utf8 */;
24c24
< /*!40101 SET character_set_client = utf8mb4 */;
---
> /*!40101 SET character_set_client = utf8 */;
26c26
< `source_file` varchar(245) NOT NULL,
---
> `source_file` varchar(255) NOT NULL,
31c31
< ) ENGINE=MyISAM DEFAULT CHARSET=utf8mb4;
---
> ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Member
|
I made a PR to @mgage tree, to get things to work (at least in Docker) when a uft8mb4 database is used. There is a special With those changes (and the hacks to bypass the missing XML::Simple module in the current Docker image) I could (in a Docker test system):
The testing was done when PG was using the branch from openwebwork/pg#390 Overall, this seems a pretty good indication that things basically work with this set of UTF8 support for both webwork2 and pg. I have been running a development server using the older (early 2019) branches with good success. In my opinion, we won't find many more of the issues with the UTF8 support until more people are testing the code on servers which get a reasonable amount of usage. |
Note: When I tried to include XML::Simple near the start of the first "cpanm install" line, there was an error: Building and testing XMLRPC-Lite-0.717 ... ! Installing XMLRPC::Lite failed. See /root/.cpanm/work/1551887935.125/build.log for details. Retry with --force to force install it. so it was put into a second "cpanm install" line.
This was referenced Mar 6, 2019
…2019_utf8mb4 Tani develop candidate 02 24 2019 utf8mb4
Member
Author
|
This is replaced by PR #930 |
Member
Author
|
replaced by pr #930 |
Member
Author
|
this is replaced by PR #930 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This should replace PR #908. It changes fewer files since I have already committed the bulk of the changes in to the copyright symbol.