Raise runtime error if raw data stream is closed unexpectedly.

.github/PULL_REQUEST_TEMPLATE.md

@@ -20,13 +20,12 @@
 - [ ] Refactoring / documentation update (non-breaking change which does not change functionality)
 - [ ] Bug fix (non-breaking change which fixes an issue)
 - [ ] New feature (non-breaking change which adds functionality)
-- [ ] Breaking change (fix, feature, or deprecation that would cause existing functionality to change)
+- [ ] Breaking change (fix, feature, or removal that would cause existing functionality to change)
 - [ ] Release preparation
 
 ## Checklist:
 <!--- Go over all the following points and make sure they have all been completed -->
 <!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
-- [ ] `xdmod_data/__version__.py` has been updated to the next development version
 - [ ] The milestone is set correctly on the pull request
 - [ ] The appropriate labels have been added to the pull request
 - [ ] Updates have been made to the `xdmod-notebooks` repository as necessary, and the notebooks all run successfully

docs/developing.md

@@ -116,8 +116,6 @@ git+https://github.com/username/xdmod-data.git@branch-name
         1. In the `README.md`, update the Open XDMoD compatibility matrix.
         1. Get the PR approved and merged.
 1. In a PR to the same branch you just released:
-    1. In `xdmod_data/__version__.py`, make sure the version number is updated
-       to a development pre-release of the next version, e.g., `3.0.0.dev01`.
     1. In `README.md`:
         1. If you just released a new major version:
             1. Under the main heading, in the sentence that begins `This
@@ -141,5 +139,6 @@ git+https://github.com/username/xdmod-data.git@branch-name
             1. Add the expected compatibility with Open XDMoD versions.
         1. In `README.md`, add the new version under development to the Open XDMoD
            compatibility matrix.
-        1. In `xdmod_data/__version__.py`, change the version number to the new
-           version under development.
+        1. In `xdmod_data/__version__.py`, change the version number to a
+           development release of the new version under development, e.g.,
+           `3.0.0.dev1`.

tests/ci/scripts/run-tests.sh

@@ -20,103 +20,112 @@ declare -a xdmod_containers=$(yq '.services | keys | .[] | select(. == "xdmod-*"
 # with Flake 8, and install the package and its testing
 # dependencies.
 for python_container in $python_containers; do
-  docker cp $PROJECT_DIR/. $python_container:/home/circleci/project
-  docker exec $python_container bash -c 'sudo chown -R circleci:circleci /home/circleci/project'
-  docker exec -w /home/circleci/project $python_container bash -c 'python3 -m pip install --upgrade pip'
-  docker exec -w /home/circleci/project $python_container bash -c 'python3 -m pip install --upgrade flake8 flake8-commas flake8-quotes'
-  docker exec -w /home/circleci/project $python_container bash -c 'python3 -m flake8 . --max-complexity=10 --show-source --exclude __init__.py'
-  docker exec -w /home/circleci/project $python_container bash -c 'python3 -m pip install -e .'
-  docker exec -w /home/circleci/project $python_container bash -c 'python3 -m pip install --upgrade python-dotenv pytest coverage'
-  # The minimum version of each dependency should be tested in the
-  # container with the minimum Python version.
-  if [ "$python_container" = "python-min" ]; then
-    min_dependency_versions=$(awk \
-      '/install_requires/ {flag=1} flag && !/install_requires/ && NF {print $0} flag && /^\[.*\]$/ {flag=0}' \
-      setup.cfg | tr -d '\n' | sed 's/ >= /==/g'
-    )
-    docker exec -w /home/circleci/project $python_container bash -c "python3 -m pip install --force-reinstall $min_dependency_versions"
-  fi
+    docker cp $PROJECT_DIR/. $python_container:/home/circleci/project
+    docker exec $python_container bash -c 'sudo chown -R circleci:circleci /home/circleci/project'
+    docker exec -w /home/circleci/project $python_container bash -c 'python3 -m pip install --upgrade pip'
+    docker exec -w /home/circleci/project $python_container bash -c 'python3 -m pip install --upgrade flake8 flake8-commas flake8-quotes'
+    docker exec -w /home/circleci/project $python_container bash -c 'python3 -m flake8 . --max-complexity=10 --show-source --exclude __init__.py'
+    docker exec -w /home/circleci/project $python_container bash -c 'python3 -m pip install -e .'
+    docker exec -w /home/circleci/project $python_container bash -c 'python3 -m pip install --upgrade python-dotenv pytest coverage'
+    # The minimum version of each dependency should be tested in the
+    # container with the minimum Python version.
+    if [ "$python_container" = "python-min" ]; then
+        min_dependency_versions=$(awk \
+            '/install_requires/ {flag=1} flag && !/install_requires/ && NF {print $0} flag && /^\[.*\]$/ {flag=0}' \
+            setup.cfg | tr -d '\n' | sed 's/ >= /==/g'
+        )
+        docker exec -w /home/circleci/project $python_container bash -c "python3 -m pip install --force-reinstall $min_dependency_versions"
+    fi
 done
 
 # Set up XDMoD web server containers.
 for xdmod_container in $xdmod_containers; do
-  # Generate OpenSSL key and certificate.
-  docker exec $xdmod_container bash -c "openssl genrsa -rand /proc/cpuinfo:/proc/filesystems:/proc/interrupts:/proc/ioports:/proc/uptime 2048 > /etc/pki/tls/private/$xdmod_container.key"
-  docker exec $xdmod_container bash -c "openssl req -new -key /etc/pki/tls/private/$xdmod_container.key -x509 -sha256 -days 365 -set_serial $RANDOM -extensions v3_req -out /etc/pki/tls/certs/$xdmod_container.crt -subj '/C=XX/L=Default City/O=Default Company Ltd/CN=$xdmod_container' -addext 'subjectAltName=DNS:$xdmod_container'"
-  # Update the server hostnames and certificates so the Python
-  # containers can make requests to them.
-  docker exec $xdmod_container bash -c "sed -i \"s/localhost/$xdmod_container/g\" /etc/httpd/conf.d/xdmod.conf"
-  if [[ "$xdmod_container" =~ 'xdmod-*-dev' ]]; then
-    if [ "$xdmod_container" = 'xdmod-main-dev' ]; then
-      branch='main'
-    else
-      branch="xdmod$(echo $xdmod_container | sed 's/xdmod-\(.*\)-dev/\1/' | sed 's/-/./')"
+    # Generate OpenSSL key and certificate.
+    docker exec $xdmod_container bash -c "openssl genrsa -rand /proc/cpuinfo:/proc/filesystems:/proc/interrupts:/proc/ioports:/proc/uptime 2048 > /etc/pki/tls/private/$xdmod_container.key"
+    docker exec $xdmod_container bash -c "openssl req -new -key /etc/pki/tls/private/$xdmod_container.key -x509 -sha256 -days 365 -set_serial $RANDOM -extensions v3_req -out /etc/pki/tls/certs/$xdmod_container.crt -subj '/C=XX/L=Default City/O=Default Company Ltd/CN=$xdmod_container' -addext 'subjectAltName=DNS:$xdmod_container'"
+    # For the containers with the development versions of the XDMoD web server,
+    if [[ "$xdmod_container" =~ xdmod-.+-dev ]]; then
+        # Install and run the latest development version of the XDMoD
+        # web server.
+        if [ "$xdmod_container" = 'xdmod-main-dev' ]; then
+            branch='main'
+        else
+            branch="xdmod$(echo $xdmod_container | sed 's/xdmod-\(.*\)-dev/\1/' | sed 's/-/./')"
+        fi
+        docker exec $xdmod_container bash -c "git clone --depth=1 --branch=$branch https://github.com/ubccr/xdmod.git /root/xdmod"
+        docker exec -w /root/xdmod $xdmod_container bash -c 'composer install'
+        docker exec -w /root/xdmod $xdmod_container bash -c '/root/bin/buildrpm xdmod'
+        # Work around the fact that the 11.0 version of bootstrap.sh contains
+        # an extra prompt in xdmod-upgrade.tcl for upgrading from 10.5 to 11.0,
+        # and in this case we are upgrading from an earlier version of 11.0 to
+        # the latest development version of 11.0, so that prompt should not be
+        # expected.
+        if [ "$xdmod_container" = 'xdmod-11-0-dev' ]; then
+            docker exec -w /root/xdmod $xdmod_container bash -c 'sed -i "/^confirmResourceSpecs/d" tests/ci/scripts/xdmod-upgrade.tcl'
+        fi
+        docker exec -w /root/xdmod $xdmod_container bash -c 'XDMOD_TEST_MODE=upgrade bash -x ./tests/ci/bootstrap.sh'
+        docker exec -w /root/xdmod $xdmod_container bash -c './tests/ci/validate.sh'
     fi
-    # Install and run the latest development version of the XDMoD
-    # web server.
-    docker exec $xdmod_container bash -c 'git clone --depth=1 --branch=$branch https://github.com/ubccr/xdmod.git /root/xdmod'
-    docker exec -w /root/xdmod $xdmod_container bash -c 'composer install'
-    docker exec -w /root/xdmod $xdmod_container bash -c '/root/bin/buildrpm xdmod'
-    docker exec -w /root/xdmod $xdmod_container bash -c 'XDMOD_TEST_MODE=upgrade ./tests/ci/bootstrap.sh'
-    docker exec -w /root/xdmod $xdmod_container bash -c './tests/ci/validate.sh'
-  elif [[ "$xdmod_container" =~ xdmod-* ]]; then
-    # Run the XDMoD web server.
-    docker exec $xdmod_container bash -c '/root/bin/services start'
-  fi
-  # Copy the 10,000 users file into the container and shred it.
-  # We use this file so we can test filters with more than 10,000
-  # values and date ranges that span multiple quarters.
-  docker cp $PROJECT_DIR/tests/ci/artifacts/10000users.log $xdmod_container:.
-  docker exec $xdmod_container xdmod-shredder -r frearson -f slurm -i 10000users.log
-  # Ingest and aggregate.
-  date=$(date --utc +%Y-%m-%d)
-  docker exec $xdmod_container xdmod-ingestor --ingest
-  docker exec $xdmod_container xdmod-ingestor --aggregate=job --last-modified-start-date $date
-  # Copy certificate (for doing requests) from the XDMoD container.
-  docker cp $xdmod_container:/etc/pki/tls/certs/$xdmod_container.crt $PROJECT_DIR
-  # Copy certificate to one of the Python containers and get an
-  # XDMoD API token for the XDMoD container.
-  docker cp $xdmod_container.crt $python_container:/home/circleci/project
-  rest_token=$(docker exec \
-    -e CURL_CA_BUNDLE="/home/circleci/project/$xdmod_container.crt" \
-    $python_container \
-    bash -c "curl \
-      -sS \
-      -X POST \
-      -c xdmod.cookie \
-      -d 'username=normaluser&password=normaluser' \
-      https://$xdmod_container/rest/auth/login \
-      | jq -r '.results.token'"
-  )
-  docker exec $python_container bash -c 'echo -n "XDMOD_API_TOKEN="' > ${xdmod_container}-token
-  docker exec \
-    -e CURL_CA_BUNDLE="/home/circleci/project/$xdmod_container.crt" \
-    $python_container \
-    bash -c "curl \
-      -sS \
-      -X POST \
-      -b xdmod.cookie \
-      https://$xdmod_container/rest/users/current/api/token?token=$rest_token \
-      | jq -r '.data.token'" \
-      >> ${xdmod_container}-token
+    # Update the server hostnames and certificates so the Python containers can
+    # make requests to them.
+    docker exec $xdmod_container bash -c "sed -i 's/localhost/$xdmod_container/g' /etc/httpd/conf.d/xdmod.conf"
+    # (Re)start the XDMoD-related services.
+    docker exec $xdmod_container bash -c '/root/bin/services restart'
+    # Copy the 10,000 users file into the container and shred it.
+    # We use this file so we can test filters with more than 10,000
+    # values and date ranges that span multiple quarters.
+    docker cp $PROJECT_DIR/tests/ci/artifacts/10000users.log $xdmod_container:.
+    docker exec $xdmod_container xdmod-shredder -r frearson -f slurm -i 10000users.log
+    # Ingest and aggregate.
+    date=$(date --utc +%Y-%m-%d)
+    docker exec $xdmod_container xdmod-ingestor --ingest
+    docker exec $xdmod_container xdmod-ingestor --aggregate=job --last-modified-start-date $date
+    # Copy certificate (for doing requests) from the XDMoD container.
+    docker cp $xdmod_container:/etc/pki/tls/certs/$xdmod_container.crt $PROJECT_DIR
+    # For each of the python containers,
+    for python_container in $python_containers; do
+        # Copy the certificate to the container.
+        docker cp $xdmod_container.crt $python_container:/home/circleci/project
+    done
+    # Use one of the Python containers to make requests to the XDMoD container
+    # to obtain an API token that will be used by all the Python containers.
+    rest_token=$(docker exec \
+        -e CURL_CA_BUNDLE="/home/circleci/project/$xdmod_container.crt" \
+        $python_container \
+        bash -c "curl \
+            -sS \
+            -X POST \
+            -c xdmod.cookie \
+            -d 'username=normaluser&password=normaluser' \
+            https://$xdmod_container/rest/auth/login \
+            | jq -r '.results.token'"
+    )
+    api_token=$(docker exec \
+        -e CURL_CA_BUNDLE="/home/circleci/project/$xdmod_container.crt" \
+        $python_container \
+        bash -c "curl \
+            -sS \
+            -X POST \
+            -b xdmod.cookie \
+            https://$xdmod_container/rest/users/current/api/token?token=$rest_token \
+            | jq -r '.data.token'"
+    )
+    echo "XDMOD_API_TOKEN=$api_token" > ${xdmod_container}-token
 done
 
 # Run the tests against each XDMoD web server.
 for python_container in $python_containers; do
-  for xdmod_container in $xdmod_containers; do
-    # Copy certificate (for doing requests) to the Python
-    # container.
-    docker cp $xdmod_container.crt $python_container:/home/circleci/project
-    # Copy XDMoD API token to the Python container.
-    docker cp $PROJECT_DIR/${xdmod_container}-token $python_container:/home/circleci/.xdmod-data-token
-    # Run tests in the Python container.
-    docker exec \
-      -e CURL_CA_BUNDLE="/home/circleci/project/$xdmod_container.crt" \
-      -e XDMOD_HOST="https://$xdmod_container" \
-      -e XDMOD_VERSION="$xdmod_container" \
-      $python_container \
-      bash -c 'python3 -m coverage run --branch --append -m pytest -vvs -o log_cli=true tests/'
-  done
-  # Make sure 100% test coverage.
-  docker exec $python_container bash -c 'python3 -m coverage report -m --fail-under=100'
+    for xdmod_container in $xdmod_containers; do
+        # Copy XDMoD API token to the Python container.
+        docker cp $PROJECT_DIR/${xdmod_container}-token $python_container:/home/circleci/.xdmod-data-token
+        # Run tests in the Python container.
+        docker exec \
+            -e CURL_CA_BUNDLE="/home/circleci/project/$xdmod_container.crt" \
+            -e XDMOD_HOST="https://$xdmod_container" \
+            -e XDMOD_VERSION="$xdmod_container" \
+            $python_container \
+            bash -c 'python3 -m coverage run --branch --append -m pytest -vvs -o log_cli=true tests/'
+    done
+    # Make sure 100% test coverage.
+    docker exec $python_container bash -c 'python3 -m coverage report -m --fail-under=100'
 done

tests/regression/data/default/jobs-dimensions.csv

@@ -9,9 +9,12 @@ jobwalltime,Job Wall Time,A categorization of jobs into discrete groups based on
 nodecount,Node Count,A categorization of jobs into discrete groups based on node count.
 pi,PI,The principal investigator of a project.
 fieldofscience,PI Group,PI Group
+pi_institution,PI Institution,Organizations that have PIs with allocations.
 qos,QOS,The quality of service of the job
 queue,Queue,Queue pertains to the low level job queues on each resource.
 resource,Resource,A resource is a remote computer that can run jobs.
 resource_type,Resource Type,A categorization of resources into by their general capabilities.
+provider,Service Provider,A service provider is an institution that hosts resources.
 username,System Username,The specific system username of the users who ran jobs.
 person,User,"A person who is on a PIs allocation, hence able to run jobs on resources."
+institution,User Institution,Organizations that have users with allocations.

tests/regression/data/default/jobs-metrics.csv

@@ -6,7 +6,6 @@ avg_gpu_hours,GPU Hours: Per Job,"The average GPU hours (number of GPU cores x w
 total_gpu_hours,GPU Hours: Total,"The total GPU hours (number of GPUs x wall time hours) used by Screwdriver jobs.<br/>For each job, the GPU usage is aggregated. For example, if a job used 1000 GPUs for one minute, it would be aggregated as 1000 GPU minutes or 16.67 GPU hours."
 max_processors,Job Size: Max (Core Count),The maximum size Screwdriver job in number of cores.<br/><i>Job Size: </i>The total number of processor cores used by a (parallel) job.
 min_processors,Job Size: Min (Core Count),The minimum size Screwdriver job in number of cores.<br/><i>Job Size: </i>The total number of processor cores used by a (parallel) job.
-normalized_avg_processors,Job Size: Normalized (% of Total Cores),"The percentage average size Screwdriver job divided by the total number of cores in the resource where the job ran. The job normalization calculation assumes that the resource size is constant. This statistic should not be used with a time range where the resource size changes, because the statistic will be incorrect.<br><i>Normalized Job Size: </i>The ratio of the total number of processor cores used by a (parallel) job over the total number of cores on the resource."
 avg_processors,Job Size: Per Job (Core Count),The average job size per Screwdriver job.<br><i>Job Size: </i>The number of processor cores used by a (parallel) job.
 avg_job_size_weighted_by_cpu_hours,Job Size: Weighted By CPU Hours (Core Count),The average Screwdriver job size weighted by CPU Hours. Defined as <br><i>Average Job Size Weighted By CPU Hours: </i> sum(i = 0 to n){ job i core count * job i cpu hours}/sum(i =  0 to n){job i cpu hours}
 avg_job_size_weighted_by_gpu_hours,Job Size: Weighted By GPU Hours (GPU Count),The average Screwdriver job size weighted by GPU Hours. Defined as <br><i>Average Job Size Weighted By GPU Hours: </i> sum(i = 0 to n){ job i core count * job i gpu hours}/sum(i =  0 to n){job i gpu hours}