Merge 2.3.0 changes from develop -> main

.github/workflows/trivy-scan.yml

@@ -4,6 +4,7 @@ on:
   pull_request:
     branches:
       - develop
+      - develop-*
       - master
 
   workflow_dispatch:  # Allows manual runs from the GitHub UI
@@ -61,4 +62,4 @@ jobs:
     - name: Output vulnerabilities
       if: failure()  # Run only if the previous step fails
       run: |
-        jq '.Results[]?.Vulnerabilities[] | select(.Severity == "HIGH" or .Severity == "CRITICAL") | "\(.VulnerabilityID): \(.PkgName) - Severity: \(.Severity) - Fixed Version: \(.FixedVersion)"' trivy_report.json
+        jq '.Results[]?.Vulnerabilities[] | select(.Severity == "HIGH" or .Severity == "CRITICAL") | "\(.VulnerabilityID): \(.PkgName) - Severity: \(.Severity) - Installed Version: \(.InstalledVersion) - Fixed Version: \(.FixedVersion)"' trivy_report.json

CHANGELOG.md

@@ -4,11 +4,25 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
 and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
 
+## [2.3.0.RELEASE] - 2025-12-20
+### Changed
+- Various minor bug fixes and typo corrections
+- FHIR improvements (resource _history, improved parameter support, POST call support for adhoc value sets, set dates properly on ConceptMap)
+- On Reindex, handle "updates" to latest versions only once rather than for each terminology one at a time
+- Ensure old mapsets get removed on reindex
+- Update unit test data to newer versions and update corresponding sampling files
+- Address CodeQL, dependabot, and trivy reported vulnerability issues
+- Support viewable maps the NCI Thesaurus mappings
+- Bring online support for CTCAE6
+- Improve term form to support attachments in the style of the CDISC template (and improve error messages)
+- Better handling of NPO complex definitions
+- Swagger version update
+
 ## [2.2.0.RELEASE] - 2025-06-10
 ### Changed
 - FHIR improvements (improved param support, improved swagger, improved metadata, more tests)
 - Improvements to computed childhood neoplasm subsets
-- Improved handling of complex defintions
+- Improved handling of complex definitions
 - Improved handling of NCI Thesaurus cumulative history
 
 ## [2.1.1.RELEASE] - 2025-04-30
@@ -98,7 +112,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 ### Added
 - Added an additional "value" parameter that works with "property" and can also work with "term" 
 - Added "has terminology hierarchy" info to terminology metadata to allow setting hierarchy links for terms that have a valid hierarchy
-- Loading/Indexing of the Chebi terminologyI.
+- Loading/Indexing of the ChEBI terminology.
 - Loading/Indexing of the HGNC terminology.
 - Loading/Indexing of the GO terminology.
 - Add API endpoint for terminology "welcome text"
@@ -120,3 +134,5 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 ## [1.5.0-SNAPSHOT and prior]
 ### No change log for earlier releases
+
+

OPENSEARCH.md

@@ -2,8 +2,12 @@
 
 Information on downloading and using OPENSEARCH with EVSRESTAPI.
 
-Verify the $OS_DIR variable points to the correct path set up in previous steps. If not, set it now: </br>`export set
-OS_DIR=c:$dir/opensearch/data`
+Verify the $OS_DIR variable points to the correct path set up in previous steps. If not, set it now.
+
+Examples:
+
+- Bash (Git Bash / WSL): `export OS_DIR="$dir/opensearch/data"`
+- Windows CMD: `set OS_DIR=%dir%\\opensearch\\data`
 
 See https://github.com/opensearch-project/spring-data-opensearch for setting up the project
 
@@ -40,7 +44,7 @@ curl -X PUT $ES/_cluster/settings -H "Content-type: application/json" \
   -d '{  "persistent": { "index.refresh_interval": "60s" } }
 ```
 
-A call like this in AWS needs to be done from a machine with an IAM role l
+A call like this in AWS needs to be done from a machine with an IAM role.
 
 ```
 {

README.md

@@ -9,10 +9,12 @@ Information on the build and deployment process for the EVSRESTAPI project
 * Before cloning the repo, make sure that the command `git config core.autocrlf` returns `false`. Change it to `false` using `git config --global core.autocrlf false` if necessary
 * Clone the project - [https://github.com/NCIEVS/evsrestapi](https://github.com/NCIEVS/evsrestapi)
 * Create a local data directory and set a `$dir` variable in your terminal. This `$dir` variable will be referenced multiple times in upcoming steps.
-  * `export set dir=C:/Users/carlsenbr/eclipse-workspace/data`
+  * Bash (Git Bash / WSL): `export dir="/c/Users/carlsenbr/eclipse-workspace/data"`
+  * Windows CMD: `set dir=C:\Users\carlsenbr\eclipse-workspace\data`
 * Execute `mkdir -p $dir/opensearch/data`
-* * Set a new variable `$ES_DIR` to the new directory just created. This `$ES_DIR` will be referenced in upcoming steps.
-* `export set ES_DIR=$dir/opensearch/data`
+- Set a new variable `$ES_DIR` to the new directory just created. This `$ES_DIR` will be referenced in upcoming steps.
+  * Bash: `export ES_DIR="$dir/opensearch/data"`
+  * Windows CMD: `set ES_DIR=%dir%\\opensearch\\data`
 * Download the "Unit Test Data" folder from <https://drive.google.com/drive/u/1/folders/11RcXLTsbOZ34_7ofKdVxLKHp_8aJGgTI>.  Unpack it to your `$dir` folder (so that `$dir/UnitTestData` exists)
   * run `prep.sh`
 
@@ -36,7 +38,7 @@ Information on the build and deployment process for the EVSRESTAPI project
   * MAIL_AUTH=true
   * MAIL_TLS=true
   * RECAPTCHA_KEY=<SITE_KEY_FROM_RECAPTCHA>
-  * {RECAPTCHA_SECRET=<SECRET_KEY_FROM_RECAPTCHA>
+  * RECAPTCHA_SECRET=<SECRET_KEY_FROM_RECAPTCHA>
 
 * Load the UnitTestData set by running `prep.sh`
   * Make sure that you can run all programs in `bash`, especially if on Mac which defaults to `zsh`
@@ -65,7 +67,7 @@ Information on the build and deployment process for the EVSRESTAPI project
 * Set the "Main Class" to `gov.nih.nci.evs.api.Application`
 * In the "Arguments" tab, add to "VM Arguments" the value `-Dspring.profiles.active=local`
 * In the Environment variables add the email credentials and settings for testing:
-  * e.g. `AUTH=true;MAIL_USER=<testUserEmail@domain.com;TLS=true;MAIL_PASSWORD=#########;RECAPTCHA_KEY:########;RECAPTCHA_SECRET=#######`
+  * e.g. `AUTH=true;MAIL_USER=<testUserEmail@domain.com>;TLS=true;MAIL_PASSWORD=#########;RECAPTCHA_KEY=########;RECAPTCHA_SECRET=########`
 * Test that it's up by looking for swagger docs: [http://localhost:8082/swagger-ui/index.html#/](http://localhost:8082/swagger-ui/index.html#/)
 
 ### Run application in IntelliJ
@@ -80,4 +82,4 @@ Information on the build and deployment process for the EVSRESTAPI project
 * Test that it's up by looking for swagger docs: [http://localhost:8082/swagger-ui/index.html#/](http://localhost:8082/swagger-ui/index.html#/)
 
 ### Run application from command line
-* Run with `java -Xmx4096 -Dspring.profiles.active=local -jar build/libs/evsrestapi*jar`
+* Run with `java -Xmx4096 -Dspring.profiles.active=local -jar build/libs/evsrestapi*.jar`

build.gradle

@@ -1,6 +1,6 @@
  buildscript {
   ext {
-    springBootVersion = '3.3.11'
+    springBootVersion = '3.4.12'
     // Remove this line to see what version is favored by springBootVersion
     // and the version brought in by spring-data-elasticsearch
     set('elasticsearch.version', '7.12.1')
@@ -23,7 +23,7 @@ plugins {
   id 'eclipse'
   id 'war'
   id 'org.springframework.boot' version "${springBootVersion}"
-  id 'io.spring.dependency-management' version '1.1.6'
+  id 'io.spring.dependency-management' version '1.1.7'
   id 'maven-publish'
   // google java format gradle plugin
   id "com.diffplug.spotless" version "6.25.0"
@@ -38,7 +38,8 @@ repositories {
   mavenCentral()
 }
 
-ext['log4j2.version'] = '2.24.1'
+// Avoid specifying version unless needed
+// ext['log4j2.version'] = '2.25.0'
 
 // Extra Properties
 ext {
@@ -56,7 +57,7 @@ ext {
 
 /* Version info */
 group = "gov.nih.nci.evs.api"
-version = "2.2.0.RELEASE"
+version = "2.3.0.RELEASE"
 
 sourceCompatibility = 17
 targetCompatibility = 17
@@ -81,19 +82,25 @@ dependencies {
 
   // Use version 2 with spring boot 3
   // custom location => springdoc.swagger-ui.path=/swagger-ui/index.html
-  implementation "org.springdoc:springdoc-openapi-starter-webmvc-ui:2.6.0"
+  implementation "org.springdoc:springdoc-openapi-starter-webmvc-ui:2.7.0"
 
   implementation "org.springframework.boot:spring-boot-devtools"
   implementation "org.springframework.boot:spring-boot-starter-actuator"
+  // Explicit patch for spring-core to mitigate CVE-2025-41249
+  implementation 'org.springframework:spring-core:6.2.11'
   implementation "org.springframework.boot:spring-boot-starter-cache"
   implementation "org.springframework.boot:spring-boot-properties-migrator"
   implementation "org.springframework:spring-aop"
 
-  // For vulnerabilities
-  implementation "org.apache.tomcat.embed:tomcat-embed-core:10.1.42"
+  // For vulnerabilities - may not be needed with Spring Boot 3.4.10
+  // implementation "org.apache.tomcat.embed:tomcat-embed-core:10.1.44"
 
   //implementation "org.springframework.data:spring-data-elasticsearch:4.2.12"
-  implementation "org.opensearch.client:spring-data-opensearch-starter:1.5.3"
+  implementation "org.opensearch.client:spring-data-opensearch-starter:1.7.1"
+
+  // Explicit override for vulnerability still present in spring-data-opensearch-starter:1.7.1
+  // and spring-data-opensearch-starter:1.8.2, but both use 2.19.x as a basis
+  implementation "org.opensearch:opensearch-common:2.19.4"
 
   //Java Mail Sender dependency
   implementation "org.springframework.boot:spring-boot-starter-mail"
@@ -164,15 +171,16 @@ dependencies {
   testImplementation("org.springframework.boot:spring-boot-starter-test") {
     exclude group: "net.minidev", module: "json-smart"
   }
-  testImplementation "org.opensearch.client:spring-data-opensearch-test-autoconfigure:1.5.3"
-  testImplementation "org.junit.jupiter:junit-jupiter-params:5.10.5"
-  testImplementation "org.junit.jupiter:junit-jupiter-api:5.10.5"
-  testRuntimeOnly "org.junit.jupiter:junit-jupiter-engine:5.10.5"
-  testRuntimeOnly "org.junit.platform:junit-platform-commons:1.10.0"
-  testRuntimeOnly "org.junit.platform:junit-platform-runner:1.10.0"
+  // Use spring-boot-starter-test defaults
+  // testImplementation "org.opensearch.client:spring-data-opensearch-test-autoconfigure:1.6.0"
+  // testImplementation "org.junit.jupiter:junit-jupiter-params:5.10.5"
+  // testImplementation "org.junit.jupiter:junit-jupiter-api:5.10.5"
+  // testRuntimeOnly "org.junit.jupiter:junit-jupiter-engine:5.10.5"
+  // testRuntimeOnly "org.junit.platform:junit-platform-commons:1.10.0"
+  // testRuntimeOnly "org.junit.platform:junit-platform-runner:1.10.0"
   // (avoid using lombok for now)
-  //testCompileOnly "org.projectlombok:lombok:1.18.32"
-  //testAnnotationProcessor "org.projectlombok:lombok:1.18.32"
+  // testCompileOnly "org.projectlombok:lombok:1.18.32"
+  // testAnnotationProcessor "org.projectlombok:lombok:1.18.32"
 
 }
 
@@ -213,7 +221,7 @@ spotless {
 }
 
 // # for this, the cuis with listed sources will have hierarchies computed
-// ./gradlew rrfSample -Pterminology=NCIMTH,ICD10CM,ICD9CM,SNOMEDCT_US,RXNORM,LNC,MDR,RADLEX -PlistFile=src/main/resources/cuis.txt -PinputPath=../data/NCIM
+// ./gradlew rrfSample -Pterminology=NCIMTH,MDR,ICD10CM,ICD9CM,LNC,SNOMEDCT_US,RADLEX,PDQ,ICD10,HL7V3.0 -PlistFile=src/main/resources/cuis-202508.txt -PinputPath=../data/NCIM/META
 tasks.register('rrfSample', JavaExec) {
   classpath = sourceSets.main.runtimeClasspath
   main = "gov.nih.nci.evs.api.service.SamplingApplication"

gradle/wrapper/gradle-wrapper.properties

@@ -1,5 +1,5 @@
 distributionBase=GRADLE_USER_HOME
 distributionPath=wrapper/dists
-distributionUrl=https\://services.gradle.org/distributions/gradle-8.2.1-bin.zip
+distributionUrl=https\://services.gradle.org/distributions/gradle-8.14.2-bin.zip
 zipStoreBase=GRADLE_USER_HOME
 zipStorePath=wrapper/dists

src/main/bin/devreset.sh

@@ -22,7 +22,7 @@ if [ $help == 1 ] || [ ${#arr[@]} -ne 1 ]; then
 fi
 dir=${arr[0]}
 # Hardcode the history file
-historyFile=$dir/cumulative_history_21.06e.txt
+historyFile=$dir/NCIT/cumulative_history_25.06e.txt
 
 
 databases=("NCIT2" "CTRP")
@@ -78,41 +78,48 @@ fi
 
 # Check NCIM
 echo "    check NCIM"
-ct=`ls $dir/NCIM | grep RRF | wc -l`
-if [[ $ct -le 20 ]]; then
-    echo "ERROR: unexpectedly small number of NCIM/*RRF files = $ct"
+ct=`ls $dir/NCIM | egrep "(AUI|COLS|CONSO|CUI|DEF|DOC|FILES|HIER|MAP|RANK|REL|SAB|SAT|STY)*RRF" | wc -l`
+if [[ $ct -le 14 ]]; then
+    echo "ERROR: unexpectedly missing NCIM/*RRF files = $dir/NCIM"
+    exit 1
+fi
+# Check NCIM2
+echo "    check NCIM2"
+ct=`ls $dir/NCIM2 | egrep "(AUI|COLS|CONSO|CUI|DEF|DOC|FILES|HIER|MAP|RANK|REL|SAB|SAT|STY)*RRF" | wc -l`
+if [[ ! $ct -eq 14 ]]; then
+    echo "ERROR: unexpectedly missing NCIM/*RRF files = $dir/NCIM"
     exit 1
 fi
 # Check NCIt weekly
 echo "    check NCIt weekly"
-if [[ ! -e "$dir/ThesaurusInferred_+1weekly.owl" ]]; then
+if [[ ! -e "$dir/NCIT/ThesaurusInferred_+1weekly.owl" ]]; then
     echo "ERROR: unexpectedly ThesaurusInferred_+1weekly.owl file"
     exit 1
 fi
 # Check NCIt monthly
 echo "    check NCIt monthly"
-if [[ ! -e "$dir/ThesaurusInferred_monthly.owl" ]]; then
+if [[ ! -e "$dir/NCIT/ThesaurusInferred_monthly.owl" ]]; then
     echo "ERROR: unexpectedly ThesaurusInferred_monthly.owl file"
     exit 1
 fi
 # Check GO monthly
 echo "    check GO monthly"
-if [[ ! -e "$dir/GO/go.2022-07-01.owl" ]]; then
-    echo "ERROR: unexpectedly missing GO/go.2022-07-01.owl file"
+if [[ ! -e "$dir/GO/GO.20250601.owl" ]]; then
+    echo "ERROR: unexpectedly missing GO/GO.20250601.owl file"
     exit 1
 fi
 
 # Check HGNC monthly
 echo "    check HGNC monthly"
-if [[ ! -e "$dir/HGNC/HGNC_202209.owl" ]]; then
-    echo "ERROR: unexpectedly missing HGNC/HGNC_202209.owl file"
+if [[ ! -e "$dir/HGNC/HGNC.202507.owl" ]]; then
+    echo "ERROR: unexpectedly missing HGNC/HGNC.202507.owl file"
     exit 1
 fi
 
 # Check ChEBI monthly
 echo "    check ChEBI monthly"
-if [[ ! -e "$dir/ChEBI/chebi_213.owl" ]]; then
-    echo "ERROR: unexpectedly missing ChEBI/chebi_213.owl file"
+if [[ ! -e "$dir/ChEBI/chebi_241.owl" ]]; then
+    echo "ERROR: unexpectedly missing ChEBI/chebi_241.owl file"
     exit 1
 fi
 
@@ -144,6 +151,13 @@ if [[ ! -e "$dir/NDFRT/NDFRT_Public_2018.02.05_Inferred.owl" ]]; then
     exit 1
 fi
 
+# Check CTCAE6
+echo "    check CTCAE6"
+if [[ ! -e "$dir/CTCAE/ctcae6.owl" ]]; then
+    echo "ERROR: unexpectedly missing CTCAE/ctcae6.owl file"
+    exit 1
+fi
+
 # check status
 check_status() {
     local retval=$1
@@ -243,6 +257,29 @@ reindex_ncim(){
   fi
 }
 
+# Reindex ncim - individual terminologies
+reindex_ncim2(){
+  echo "  Reindexing ncim2 ...`/bin/date`"
+  for t in MDR ICD10CM ICD9CM LNC SNOMEDCT_US RADLEX PDQ ICD10 HL7V3.0; do
+      # Keep the NCIM folder around while we run
+      echo "  Load $t (from ncim2 downloaded data) ...`/bin/date`"
+      src/main/bin/ncim-part.sh --noconfig $dir/NCIM2 --keep --terminology $t > /tmp/x.$$.txt 2>&1
+      if [[ $? -ne 0 ]]; then
+          cat /tmp/x.$$.txt | sed 's/^/    /'
+          echo "ERROR: loading $t"
+          exit 1
+      fi
+  done
+  # Reindex ncim - must run after the prior section so that maps can connect to loaded terminologies
+  echo "  Reindex ncim from ncim2 ...`/bin/date`"
+  src/main/bin/ncim-part.sh --noconfig $dir/NCIM2 > /tmp/x.$$.txt 2>&1
+  if [[ $? -ne 0 ]]; then
+      cat /tmp/x.$$.txt | sed 's/^/    /'
+      echo "ERROR: problem running ncim-part.sh"
+      exit 1
+  fi
+}
+
 drop_databases(){
   for db in "${databases[@]}"
   do
@@ -279,6 +316,8 @@ load_terminology_data_in_transaction(){
 }
 
 load_terminology_data(){
+  # e.g. curl -X POST -H 'Content-Type: application/rdf+xml' -T '../data/UnitTestData/UmlsSemNet/umlssemnet.owl' http://localhost:3030/NCIT2/data?graph=http://UmlsSemNet
+  # e.g. curl -X POST -H 'Content-Type: application/rdf+xml' -T '../data/UnitTestData/Mouse_Anatomy/ma_07_27_2016.owl' http://localhost:3030/NCIT2/data?graph=http://MA
   echo "    Loading $3 into $1 ...`/bin/date`"
   echo "      curl -X POST -H 'Content-Type: application/rdf+xml' -T '$dir/$3' http://${GRAPH_DB_HOST}:${GRAPH_DB_PORT}/$1/data?graph=$2"
   $curl_cmd -X POST -H "Content-Type: application/rdf+xml" -T "$dir/$3" "http://${GRAPH_DB_HOST}:${GRAPH_DB_PORT}/$1/data?graph=$2" 2> /dev/null > /tmp/x.$$
@@ -290,16 +329,17 @@ load_terminology_data(){
 }
 
 load_data(){
-    load_terminology_data CTRP http://NCI_T_weekly ThesaurusInferred_+1weekly.owl
-    load_terminology_data CTRP http://NCI_T_monthly ThesaurusInferred_monthly.owl
-    load_terminology_data NCIT2 http://NCI_T_monthly ThesaurusInferred_monthly.owl
-    load_terminology_data NCIT2 http://GO_monthly GO/go.2022-07-01.owl
-    load_terminology_data NCIT2 http://HGNC_monthly HGNC/HGNC_202209.owl
-    load_terminology_data NCIT2 http://ChEBI_monthly ChEBI/chebi_213.owl
+    load_terminology_data CTRP http://NCI_T_weekly NCIT/ThesaurusInferred_+1weekly.owl
+    load_terminology_data CTRP http://NCI_T_monthly NCIT/ThesaurusInferred_monthly.owl
+    load_terminology_data NCIT2 http://NCI_T_monthly NCIT/ThesaurusInferred_monthly.owl
+    load_terminology_data NCIT2 http://GO_monthly GO/GO.20250601.owl
+    load_terminology_data NCIT2 http://HGNC_monthly HGNC/HGNC.202507.owl
+    load_terminology_data NCIT2 http://ChEBI_monthly ChEBI/chebi_241.owl
     load_terminology_data NCIT2 http://UmlsSemNet UmlsSemNet/umlssemnet.owl
-    load_terminology_data NCIT2 http://MEDRT MED-RT/medrt.owl
-    load_terminology_data NCIT2 http://Canmed CanMed/canmed.owl
+    load_terminology_data NCIT2 http://Canmed CanMed/CANMED.202506.owl
+    load_terminology_data NCIT2 http://MEDRT MED-RT/MEDRT.2025-06-02.owl
     load_terminology_data NCIT2 http://CTCAE CTCAE/ctcae5.owl
+    load_terminology_data NCIT2 http://CTCAE6 CTCAE/ctcae6.owl
     load_terminology_data NCIT2 http://DUO_monthly DUO/duo_Feb21.owl
     load_terminology_data NCIT2 http://DUO_monthly DUO/iao_Dec20.owl
     load_terminology_data NCIT2 http://OBI_monthly OBI/obi_2022_07.owl
@@ -330,6 +370,7 @@ drop_databases
 create_databases
 remove_elasticsearch_indexes
 reindex_ncim
+reindex_ncim2
 load_data
 # This will load maps from github evsrestapi-operations
 reindex