Updated installation scripts and docker install. Fixed samplesheet parse and using application folder samba param

.gitignore

@@ -3,7 +3,6 @@
 *.bin
 *.gz
 *_BAK
-migrations/
 tmp/
 logs/
 /static/

CHANGELOG.md

@@ -23,6 +23,12 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Included thorough description for wetlab API's update_lab() method [#361](https://github.com/BU-ISCIII/iskylims/pull/361)
 - Included new API function lab-request-mapping to get LabRequest fields ontology map [#377](https://github.com/BU-ISCIII/iskylims/pull/377)
 - Improved responses in API create-sample-data by adding ERROR messages and data [#377](https://github.com/BU-ISCIII/iskylims/pull/377)
+- Added support for `script-before` and `script-after` hooks in install script. [#389](https://github.com/BU-ISCIII/iskylims/pull/389)
+- Committed baseline migration files and added migrations for develop changes [#389](https://github.com/BU-ISCIII/iskylims/pull/389)
+- Added developer notes on how to create and manage migration files [#389](https://github.com/BU-ISCIII/iskylims/pull/389)
+- Added documentation describing migration scripts and their related versions [#389](https://github.com/BU-ISCIII/iskylims/pull/389)
+- Enabled Docker internal networking for local test installation [#389](https://github.com/BU-ISCIII/iskylims/pull/389)
+- Opened Docker network to allow localhost MySQL connection when required [#389](https://github.com/BU-ISCIII/iskylims/pull/389)
 
 #### Fixes
 
@@ -59,6 +65,11 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Adapted update_lab serializer call to new serializer update method [#361](https://github.com/BU-ISCIII/iskylims/pull/361)
 - Leave missing submitting_fields as empty string instead of crashing in wetlab.api.create_sample_data [#363](https://github.com/BU-ISCIII/iskylims/pull/363)
 - Fixed wetlab API create-sample-data error when submitting_institution fields were not provided [#377](https://github.com/BU-ISCIII/iskylims/pull/377)
+- Fixed incorrect git revision propagation to Dockerfile during build [#389](https://github.com/BU-ISCIII/iskylims/pull/389)
+- Fixed configuration file accessibility from within Docker container [#389](https://github.com/BU-ISCIII/iskylims/pull/389)
+- Fixed disk utilization check to correctly resolve application folder path [#389](https://github.com/BU-ISCIII/iskylims/pull/389)
+- Fixed incorrect application folder path resolution in crontab scripts [#389](https://github.com/BU-ISCIII/iskylims/pull/389)
+- Fixed samplesheet parsing error [#389](https://github.com/BU-ISCIII/iskylims/pull/389)
 
 #### Changed
 
@@ -71,10 +82,15 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Improved query performance and excluded rejected/archived services from ongoing list. (#299)
 - Updated sample metadata fields with standardized ontology mappings and schema alignment.[#358](https://github.com/BU-ISCIII/iskylims/pull/358)
 - API create-sample-data Lab data mapping moved to core_config.LAB_REQUEST_ONTOLOGY_MAP [#377](https://github.com/BU-ISCIII/iskylims/pull/377)
+- Renamed `docker-compose.yml` to `docker-compose.test.yml` and `docker-compose.prod.yml` for test clarity [#389](https://github.com/BU-ISCIII/iskylims/pull/389)
+- Refactored Docker runtime handling when path is outside repository [#389](https://github.com/BU-ISCIII/iskylims/pull/389)
+- Updated `docker_install.sh` with multiple reliability improvements [#389](https://github.com/BU-ISCIII/iskylims/pull/389)
+- Updated upgrade scripts documentation to include execution order information and docker upgrade clarifications [#389](https://github.com/BU-ISCIII/iskylims/pull/389)
 
 #### Removed
 
 - Dummy fix in usage line [#271](https://github.com/BU-ISCIII/iskylims/pull/271)
+- Removed migrations from `.gitignore` and app-level `.gitignore` to ensure version control of schema changes [#389](https://github.com/BU-ISCIII/iskylims/pull/389)
 
 #### Requirements
 

Dockerfile

@@ -30,7 +30,7 @@ ENV PATH="/usr/sbin/cron:$PATH"
 # Set default install type
 ARG INSTALL_TYPE=dep
 ARG GIT_REVISION=main
-ARG INSTALL_CONF=conf/docker_install_settings.txt
+ARG INSTALL_CONF=conf/docker_test_settings.txt
 
 # Execute the dependency stage only; app migrations run when the container is up.
 RUN /bin/bash install.sh --install dep --git_revision $GIT_REVISION --conf $INSTALL_CONF --skip_apache_restart

LEAME.md

@@ -18,6 +18,7 @@ De acuerdo con la infraestructura existente, la secuenciacion se realiza en un i
     - [Contenedor local de pruebas](#contenedor-local-de-pruebas)
     - [Contenedor de produccion](#contenedor-de-produccion)
     - [Actualizacion del despliegue Docker](#actualizacion-del-despliegue-docker)
+    - [Actualizacion del despliegue Docker v3.0.0 a 3.1.0](#actualizacion-del-despliegue-docker-v300-a-310)
   - [Despliegue bare-metal (Ubuntu/CentOS)](#despliegue-bare-metal-ubuntucentos)
     - [Instalacion](#instalacion)
       - [Requisitos previos](#requisitos-previos)
@@ -70,6 +71,8 @@ Levanta el sistema completo (base de datos, Samba y app) con fixtures y datos de
 bash docker_install.sh --test
 ```
 
+Esto usa `docker-compose.test.yml` por defecto.
+
 Puedes personalizar los valores por defecto:
 
 - `--demo_data /ruta/a/iskylims_demo_data.tar.gz` para reutilizar un archivo local (si no, se descarga).
@@ -116,6 +119,22 @@ bash docker_install.sh --install_conf conf/my_prod_settings.txt --action upgrade
 
 La actualizacion reconstruye/reinicia el contenedor y ejecuta `install.sh` dentro del contenedor, que regenera migraciones, las aplica con `--fake-initial` y evita cargar superusuario/datos demo/prueba.
 
+### Actualizacion del despliegue Docker v3.0.0 a 3.1.0
+
+Antes de actualizar, asegurate de tener una copia completa de la base de datos y de los datos en volumenes que uses. Confirma que `conf/my_prod_settings.txt` tenga el host/usuario/password de la base de datos de produccion, la URL/IP del servidor, correo y ajustes de logging usados por el contenedor.
+
+Para 3.0.0 -> 3.1.0, exporta primero el mapeo de LibraryPool y luego ejecuta la actualizacion con scripts pre/post:
+
+```bash
+mysql --user=<db_user> --password=<db_password> --host=<db_server_ip> --port=<db_port> iskylims \
+  -e "SELECT id, run_process_id_id FROM wetlab_library_pool" \
+  > /tmp/library_pool_run_process.tsv
+
+bash docker_install.sh --install_conf conf/my_prod_settings.txt --action upgrade \
+  --script_before convert_rawtop_counter_to_int \
+  --script_after library_pool_to_many_relation,/tmp/library_pool_run_process.tsv
+```
+
 ## Despliegue bare-metal (Ubuntu/CentOS)
 
 ### Instalacion
@@ -233,6 +252,15 @@ bash install.sh --upgrade app --git_revision main --tables
 
 # ejemplo ejecutando un script de migracion en la actualizacion
 bash install.sh --upgrade app --script migrate_optional_values --git_revision main --tables
+
+# 3.0.0 -> 3.1.0 (scripts de datos pre/post)
+mysql --user=<db_user> --password=<db_password> --host=<db_server_ip> --port=<db_port> iskylims \
+  -e "SELECT id, run_process_id_id FROM wetlab_library_pool" \
+  > /tmp/library_pool_run_process.tsv
+
+bash install.sh --upgrade app --git_revision main \
+  --script_before convert_rawtop_counter_to_int \
+  --script_after library_pool_to_many_relation,/tmp/library_pool_run_process.tsv
 ```
 
 O ejecuta todo en un unico comando:

README.md

@@ -19,22 +19,27 @@ Application servers run web applications for bioinformatics analysis (GALAXY), t
     - [Local test stack](#local-test-stack)
     - [Production container](#production-container)
     - [Upgrade docker deployment](#upgrade-docker-deployment)
+    - [Upgrade docker deployment v3.0.0 to 3.1.0](#upgrade-docker-deployment-v300-to-310)
+      - [Back up first](#back-up-first)
+      - [Refresh code and settings](#refresh-code-and-settings)
   - [Bare-metal deployment (Ubuntu/CentOS)](#bare-metal-deployment-ubuntucentos)
     - [Install](#install)
       - [Prerequisites](#prerequisites)
       - [Clone the repository](#clone-the-repository)
       - [Prepare the database](#prepare-the-database)
       - [Configure install\_settings.txt](#configure-install_settingstxt)
       - [Run install.sh](#run-installsh)
-    - [Upgrade (3.0.x to 3.1.x)](#upgrade-30x-to-31x)
-      - [Back up first](#back-up-first)
-      - [Refresh code and settings](#refresh-code-and-settings)
+    - [Upgrade (3.0.0 to 3.1.0)](#upgrade-300-to-310)
+      - [Back up first](#back-up-first-1)
+      - [Refresh code and settings](#refresh-code-and-settings-1)
       - [Run upgrade steps requiring root](#run-upgrade-steps-requiring-root)
       - [Run upgrade steps without root](#run-upgrade-steps-without-root)
   - [What to do if something fails](#what-to-do-if-something-fails)
   - [Final configuration steps](#final-configuration-steps)
     - [SAMBA configurarion](#samba-configurarion)
     - [Email verification](#email-verification)
+  - [Developer notes](#developer-notes)
+    - [Django migrations workflow](#django-migrations-workflow)
     - [Configure Apache server](#configure-apache-server)
     - [Verification of the installation](#verification-of-the-installation)
   - [iSkyLIMS documentation](#iskylims-documentation)
@@ -71,6 +76,8 @@ Bring up a full test stack (database, Samba, app) plus fixtures and demo data:
 bash docker_install.sh --test
 ```
 
+This uses `docker-compose.test.yml` by default.
+
 Defaults can be customised:
 
 - `--demo_data /path/to/iskylims_demo_data.tar.gz` to reuse a local demo archive (otherwise it is downloaded).
@@ -117,6 +124,38 @@ bash docker_install.sh --install_conf conf/my_prod_settings.txt --action upgrade
 
 The upgrade path rebuilds/restarts the container and runs `install.sh` inside the app container, which regenerates migrations, applies them with `--fake-initial`, and skips superuser/demo/test data loading.
 
+### Upgrade docker deployment v3.0.0 to 3.1.0
+
+#### Back up first
+
+- Full backup of the `iskylims` database.
+- Full backup of the logs folder and the documents folder.
+
+For 3.0.0 -> 3.1.0, export the LibraryPool mapping first, then run the upgrade with pre/post scripts:
+
+```bash
+mysql --user=<db_user> --password=<db_password> --host=<db_server_ip> --port=<db_port> iskylims \
+  -e "SELECT id, run_process_id_id FROM wetlab_library_pool" \
+  > /tmp/library_pool_run_process.tsv
+```
+
+#### Refresh code and settings
+
+```bash
+cd <your working directory>/iskylims
+git pull
+cp conf/docker_production_settings.txt myprod_settings.txt
+sudo nano myprod_settings.txt
+```
+
+Ensure the file uses Linux-friendly encoding (UTF-8/ASCII) if you edit it on Windows.
+
+```bash
+bash docker_install.sh --install_conf my_prod_settings.txt --action upgrade \
+  --script_before convert_rawtop_counter_to_int \
+  --script_after library_pool_to_many_relation,/tmp/library_pool_run_process.tsv
+```
+
 ## Bare-metal deployment (Ubuntu/CentOS)
 
 ### Install
@@ -185,7 +224,7 @@ sudo bash install.sh --install full --git_revision main --tables
 
 - If Apache is managed elsewhere, skip the automatic restart with `--skip_apache_restart`.
 
-### Upgrade (3.0.x to 3.1.x)
+### Upgrade (3.0.0 to 3.1.0)
 
 Follow these steps to move from version 3.0.0 to the 3.1.x series.
 
@@ -196,8 +235,9 @@ Follow these steps to move from version 3.0.0 to the 3.1.x series.
 - If you use library pools, export them before upgrading:
 
   ```bash
-  mysql --user=<db_user> --password=<db_password> --host=<db_server_ip> --port=<db_port> iskylims \
-    -e "SELECT * FROM wetlab_library_pool" > <backup_folder>/backup_lib_pool.sql
+mysql --user=<db_user> --password=<db_password> --host=<db_server_ip> --port=<db_port> iskylims \
+  -e "SELECT id, run_process_id_id FROM wetlab_library_pool" \
+  > /tmp/library_pool_run_process.tsv
   ```
 
 #### Refresh code and settings
@@ -227,19 +267,9 @@ Upgrade the application code and database:
 
 ```bash
 # with library pool restore
-bash install.sh --upgrade app --script <backup_folder>/backup_lib_pool.sql --git_revision main --tables
-
-# without library pool restore
-bash install.sh --upgrade app --git_revision main --tables
-
-# example running a migration script during upgrade
-bash install.sh --upgrade app --script migrate_optional_values --git_revision main --tables
-```
-
-Or run everything in one go:
-
-```bash
-sudo bash install.sh --upgrade full --git_revision main --tables
+bash install.sh --upgrade app --git_revision main \
+  --script_before convert_rawtop_counter_to_int \
+  --script_after library_pool_to_many_relation,/tmp/library_pool_run_process.tsv
 ```
 
 Upgrades regenerate migrations and apply them with `--fake-initial` so existing tables remain intact, matching the Docker workflow.
@@ -277,6 +307,19 @@ mysql -u iskylims -p -h dmysqlps.isciiides.es iskylims < /home/dadmin/backup_pro
 - Go to Configuration -> Email configuration
 - Fill the form with the needed params for your email configuration and try to send a test email.
 
+## Developer notes
+
+### Django migrations workflow
+
+Migrations are committed to the repo. Do not run `makemigrations` during install/upgrade.
+
+Baseline + upgrade flow for new releases:
+
+1. Generate baseline migrations from the last stable tag (example 3.0.0).
+2. Commit the baseline migrations.
+3. Generate new migrations on `develop` for schema changes and commit them.
+4. Upgrades run `migrate --fake-initial` once to align existing tables, then `migrate` to apply the new migration files.
+
 ### Configure Apache server
 
 Copy the apache configuration file according to your distribution inside the apache configutation directory and rename it to iskylims.conf

UPGRADE_SCRIPTS.md

@@ -0,0 +1,36 @@
+# Upgrade scripts
+
+This file lists data migration scripts and the version range they apply to.
+Run them with:
+
+```bash
+python manage.py runscript <script_name>
+```
+
+## 3.0.0 -> 3.1.0
+
+- Run order:
+  1. Export LibraryPool run_process_id data (before migrations).
+  2. Run `convert_rawtop_counter_to_int` before migrations.
+  3. Run migrations.
+  4. Run `library_pool_to_many_relation` after migrations with the exported file.
+
+- Export example:
+  ```bash
+  mysql -u <db_user> -p -h <db_host> -D <db_name> \
+    -e "SELECT id, run_process_id_id FROM wetlab_library_pool" \
+    > /tmp/library_pool_run_process.tsv
+  ```
+
+- `wetlab/scripts/convert_rawtop_counter_to_int.py` (script name: `convert_rawtop_counter_to_int`)
+- `wetlab/scripts/library_pool_to_many_relation.py` (script name: `library_pool_to_many_relation`)
+
+## 2.3.0 -> 3.0.0
+
+- `core/scripts/rename_app_name.py` (script name: `rename_app_name`)
+- `core/scripts/migrate_sample_type.py` (script name: `migrate_sample_type`)
+- `core/scripts/migrate_optional_values.py` (script name: `migrate_optional_values`)
+
+## 2.3.0 -> 2.3.1
+
+- `drylab/scripts/drylab_service_state_migration.py` (script name: `drylab_service_state_migration`)