Some fixes for MySQL and PostgreSQL files in TestLink 1.9.20 (branch testlink_1_9_20_fixed)

[jmjjg]

Hello,
following my previous pull request, here is some more.

• Ubuntu 16.04.6 LTS
• mysql Ver 15.1 Distrib 10.0.38-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2
• psql (PostgreSQL) 9.5.19

I wrote a little bash script (attached and explained below) to help me install, update and compare testlink database schemas from 1.9.19 updated to 1.9.20 versus a stock 19.20 installation (outputting the structure to an SQL file and using diff).

Proposed fixes in this pull request

I made a change in install/sql/alter_tables/1.9.20/mysql/DB.1.9.20/step1/db_schema_update.sql so that the diff would return no difference, although I believe the difference wouldn't break anything.

I also made changes to install/sql/alter_tables/1.9.20/postgres/DB.1.9.20/step1/db_schema_update.sql and install/sql/postgres/testlink_create_tables.sql so the SQL script won't break at execution and the diff would return no difference either.

Explanations on the differences can be found below.

Homemade bash script (bonus)

Here is the bash script that helps me with installing, updating and comparing TestLink databases from the command-line. It is not perfect yet, but if you want, you can include it in the TestLink codebase (or I could do another pull request for that if you prefer).

tl_database.sh.txt

Sample usage

Get sources from github in directory /tmp/github.com for

• the official TestLink account on branch testlink_1_9_20_fixed
• a proposed solution from the Libriciel SCOP account on branch 1.9.20_tl-database_script_fix_sql_schemas

# Global variables
TL_DATABASE_SCRIPT="/home/cbuffin/Projets/www/gitlab.libriciel.fr/cbuffin/ls-testlink/tl_database.sh"
GITHUB_LOCAL_ROOT="/tmp/github.com"
TL_ORIGINAL_CODE="${GITHUB_LOCAL_ROOT}/TestLinkOpenSourceTRMS/testlink-code"
TL_ORIGINAL_BRANCH_1_9_20_FIXED="testlink_1_9_20_fixed"
LS_CORRECTION_CODE="${GITHUB_LOCAL_ROOT}/libriciel/testlink-code"
LS_CORRECTION_BRANCH="1.9.20_tl-database_script_fix_sql_schemas"
(
    set -o errexit
    set -o nounset
    set -o pipefail
    set -o xtrace

    # Get up-to-date with TL original code
    if [ ! -d "${TL_ORIGINAL_CODE}" ] ; then
        mkdir -p "`dirname "${TL_ORIGINAL_CODE}"`"
        cd "`dirname "${TL_ORIGINAL_CODE}"`"
        git clone https://github.com/TestLinkOpenSourceTRMS/testlink-code.git
    fi
    # Original code
    cd "${TL_ORIGINAL_CODE}"
    git checkout ${TL_ORIGINAL_BRANCH_1_9_20_FIXED}
    # Get up-to-date with TL proposed branch
    if [ ! -d "${LS_CORRECTION_CODE}" ] ; then
        mkdir -p "`dirname "${LS_CORRECTION_CODE}"`"
        cd "`dirname "${LS_CORRECTION_CODE}"`"
        git clone https://github.com/libriciel/testlink-code.git
        cd "${LS_CORRECTION_CODE}"
        git remote add upstream https://github.com/TestLinkOpenSourceTRMS/testlink-code.git
    fi
    cd "${LS_CORRECTION_CODE}"
    git fetch upstream
    git checkout ${LS_CORRECTION_BRANCH}
    git merge upstream/${TL_ORIGINAL_BRANCH_1_9_20_FIXED}
)

MySQL

Check the original code

# Try MySQL install at tag 1.9.19 and update in 1.9.20_fixed
( \
    cd ${TL_ORIGINAL_CODE} \
    && git checkout 1.9.19 \
    && ${TL_DATABASE_SCRIPT} reinstall --database testlink_from_1_9_19 \
    && git checkout ${TL_ORIGINAL_BRANCH_1_9_20_FIXED} \
    && ${TL_DATABASE_SCRIPT} update --database testlink_from_1_9_19 \
    && ${TL_DATABASE_SCRIPT} reinstall --database testlink_from_1_9_20 \
    && ${TL_DATABASE_SCRIPT} compare --database testlink_from_1_9_19 --reference testlink_from_1_9_20 \
) && echo "Success" || echo "Error"

Check the proposed solution

# Try MySQL install at tag 1.9.19 and update in 1.9.20_tl-database_script_fix_sql_schemas
( \
    cd ${LS_CORRECTION_CODE} \
    && git checkout 1.9.19 \
    && ${TL_DATABASE_SCRIPT} reinstall --database testlink_from_1_9_19 \
    && git checkout ${LS_CORRECTION_BRANCH} \
    && ${TL_DATABASE_SCRIPT} update --database testlink_from_1_9_19 \
    && ${TL_DATABASE_SCRIPT} reinstall --database testlink_from_1_9_20 \
    && ${TL_DATABASE_SCRIPT} compare --database testlink_from_1_9_19 --reference testlink_from_1_9_20 \
) && echo "Success" || echo "Error"

Explanations

1. Inconsistent unsigned usage

<   `enable_on_design` tinyint(1) unsigned NOT NULL DEFAULT '0',
<   `enable_on_execution` tinyint(1) unsigned NOT NULL DEFAULT '1',
---
>   `enable_on_design` tinyint(1) NOT NULL DEFAULT '0',
>   `enable_on_execution` tinyint(1) NOT NULL DEFAULT '1',

Solution

Update lines 25 and 26 from install/sql/alter_tables/1.9.20/mysql/DB.1.9.20/step1/db_schema_update.sql to use tinyint(1) unsigned

ALTER TABLE /*prefix*/platforms ADD COLUMN  enable_on_design tinyint(1) unsigned NOT NULL default '0';
ALTER TABLE /*prefix*/platforms ADD COLUMN  enable_on_execution tinyint(1) unsigned NOT NULL default '1';

Result

2020-02-03 15:11:55 INFO     OK, database structures are the same
2020-02-03 15:11:55 DEBUG    Deleting compare directory "/tmp/tmp.Apb0GtItg3"
2020-02-03 15:11:55 INFO     Process 23078 exited normally

PostgreSQL

Check the original code

# Try PostgreSQL install at tag 1.9.19 and update in 1.9.20_fixed
( \
    cd ${TL_ORIGINAL_CODE} \
    && git checkout 1.9.19 \
    && ${TL_DATABASE_SCRIPT} reinstall --database testlink_from_1_9_19 --type postgres \
    && git checkout ${TL_ORIGINAL_BRANCH_1_9_20_FIXED} \
    && ${TL_DATABASE_SCRIPT} update --database testlink_from_1_9_19 --type postgres \
    && ${TL_DATABASE_SCRIPT} reinstall --database testlink_from_1_9_20 --type postgres \
    && ${TL_DATABASE_SCRIPT} compare --database testlink_from_1_9_19 --reference testlink_from_1_9_20 --type postgres \
) && echo "Success" || echo "Error"

Check the proposed solution

# Try PostgreSQL install at tag 1.9.19 and update in 1.9.20_tl-database_script_fix_sql_schemas
( \
    cd ${LS_CORRECTION_CODE} \
    && git checkout 1.9.19 \
    && ${TL_DATABASE_SCRIPT} reinstall --database testlink_from_1_9_19 --type postgres \
    && git checkout ${LS_CORRECTION_BRANCH} \
    && ${TL_DATABASE_SCRIPT} update --database testlink_from_1_9_19 --type postgres \
    && ${TL_DATABASE_SCRIPT} reinstall --database testlink_from_1_9_20 --type postgres \
    && ${TL_DATABASE_SCRIPT} compare --database testlink_from_1_9_19 --reference testlink_from_1_9_20 --type postgres \
) && echo "Success" || echo "Error"

Explanations

1. Unique name for indexes in install/sql/postgres/testlink_create_tables.sql

2020-02-03 15:51:14 INFO     \i ./install/sql/alter_tables/1.9.20/postgres/DB.1.9.20/step1/db_schema_update.sql
ERROR:  relation "udx1" already exists

Solution

Update line 928 and [941,942] from install/sql/postgres/testlink_create_tables.sql to use unique index names

CREATE UNIQUE INDEX /*prefix*/udx1_baseline_l1l2_context ON /*prefix*/baseline_l1l2_context ("testplan_id","platform_id","creation_ts");

CREATE UNIQUE INDEX /*prefix*/udx1_baseline_l1l2_details
ON /*prefix*/baseline_l1l2_details ("context_id","top_tsuite_id","child_tsuite_id","status");

2. Unique name for indexes in install/sql/alter_tables/1.9.20/postgres/DB.1.9.20/step1/db_schema_update.sql

2020-02-03 15:57:58 INFO     \i ./install/sql/alter_tables/1.9.20/postgres/DB.1.9.20/step1/db_schema_update.sql
ERROR:  relation "udx1" already exists

Update line 45 and [58,59] from install/sql/alter_tables/1.9.20/postgres/DB.1.9.20/step1/db_schema_update.sql to use unique index names

CREATE UNIQUE INDEX /*prefix*/udx1_baseline_l1l2_context ON /*prefix*/baseline_l1l2_context ("testplan_id","platform_id","creation_ts");

CREATE UNIQUE INDEX /*prefix*/udx1_baseline_l1l2_details
ON /*prefix*/baseline_l1l2_details ("context_id","top_tsuite_id","child_tsuite_id","status");

3. INT unsigned should be INT

2020-02-03 11:17:59 INFO     \i ./install/sql/postgres/testlink_create_tables.sql
ERROR:  syntax error at or near "unsigned"
LINE 938:   "qty" INT unsigned NOT NULL DEFAULT '0',
# Will be the same for next line: "total_tc" INT unsigned NOT NULL DEFAULT '0',

Solution

Update lines [937,938] from install/sql/postgres/testlink_create_tables.sql to not use unsigned, since they are
not supported by PostgreSQL.

  "qty" INT NOT NULL DEFAULT '0',
  "total_tc" INT NOT NULL DEFAULT '0',

4. Missing table execution_tcsteps_wip in install/sql/alter_tables/1.9.20/postgres/DB.1.9.20/step1/db_schema_update.sql

2020-02-03 16:00:48 INFO     tl_database.sh: compare postgres database "testlink_from_1_9_19" with username "testlink"
2020-02-03 16:00:48 INFO     Comparing database structure of "testlink_from_1_9_19" with respect to "testlink_from_1_9_20" in directory "/tmp/tmp.xfmLRRwKWh"
567,602d566
< -- Name: execution_tcsteps_wip; Type: TABLE; Schema: public; Owner: -
< --
< 
< CREATE TABLE public.execution_tcsteps_wip (
...
< --
< -- Name: execution_tcsteps_wip_testplan_id_fkey; Type: FK CONSTRAINT; Schema: public; Owner: -
< --
< 
< ALTER TABLE ONLY public.execution_tcsteps_wip
<     ADD CONSTRAINT execution_tcsteps_wip_testplan_id_fkey FOREIGN KEY (testplan_id) REFERENCES public.testplans(id);
2020-02-03 16:00:49 CRITICAL Process 572 exited with error code 1

Solution

Add the table definition code in install/sql/alter_tables/1.9.20/postgres/DB.1.9.20/step1/db_schema_update.sql around line 23

--
-- Table structure for table "execution_tcsteps_wip"
--
CREATE TABLE /*prefix*/execution_tcsteps_wip (
    "id" BIGSERIAL NOT NULL ,
    "tcstep_id" INTEGER NOT NULL DEFAULT '0' REFERENCES  /*prefix*/tcsteps (id),
    "testplan_id" INTEGER NOT NULL DEFAULT '0' REFERENCES  /*prefix*/testplans (id),
    "platform_id" INTEGER NOT NULL DEFAULT '0',
    "build_id" INTEGER NOT NULL DEFAULT '0',
    "tester_id" BIGINT NULL DEFAULT NULL,
    "creation_ts" TIMESTAMP NOT NULL DEFAULT now(),
    "notes" TEXT NULL DEFAULT NULL,
    "status" CHAR(1) NULL DEFAULT NULL,
    PRIMARY KEY ("id")
);
CREATE UNIQUE INDEX /*prefix*/execution_tcsteps_wip_uidx1 ON  /*prefix*/execution_tcsteps_wip ("tcstep_id","testplan_id","platform_id","build_id");

[squash-labs]

Manage this branch in Squash

Test this branch here: https://libriciel1920-tl-database-scri-f9iua.squash.io

[kovacs-andras]

Very many thanks!
I just upgraded an older install and without your postgresql fix I was not able to browse the tests tree.
THIS PR MUST BE MERGED!

[neiesc]

@fmancardi up to this

[strawmelonjuice]

Can I just compliment your bash file, seeing that alone was a blessing for my eyes, so well-structured