UV package intergration by kamalcou · Pull Request #70 · CIROH-UA/forcingprocessor

kamalcou · 2026-03-11T16:06:07Z

[Short description explaining the high-level reason for the pull request]
UV package integration

requirements.txt
readme.md -- uv installation
.gitignore -- add !requirements.txt
pyproject.toml-- modified
uv.lock
configs/conf_fp.json -- add relative path

Additions

Removals

Changes

Testing

Screenshots

Notes

Resolves Python Version Update #14
Resolves Python Package Versions #24
Resolves There are two "Set up Python" steps that conflict with each other. #36
Resolves pytests failing on ubuntu/x86 #68
Resolves Workflow triggers #71

Todos

Checklist

Testing checklist

Target Environment support

Windows
Linux
Browser

Accessibility

Keyboard friendly
Screen reader friendly

Other

- Bump Python from 3.9 to 3.11 to match requires-python >= 3.11 - Update actions/checkout from v3 to v4 - Update actions/setup-python from v3 to v5 - Add pyproject.toml, setup.cfg, .python-version to workflow trigger paths (fixes #71) - Remove duplicate broken setup-python step in forcingprocessor_nrds.yaml

This workflow installs Python dependencies, configures AWS, and sets up testing for the forcing processor module.

Updated pytest commands to run with 'uv run' for better compatibility.

h5py 3.15.1 and netcdf4 1.7.4 bundle conflicting HDF5 C libraries, causing RuntimeError in H5DSget_num_scales when both are loaded. h5py 3.14.0 does not exhibit this conflict. See h5py/h5py#2453.

- Dockerfile.forcingprocessor-deps: python3.9 -> python3.11, install uv, replace all pip/pip3 calls with uv pip install --system - Dockerfile.forcingprocessor: pip3 install -> uv pip install --system - Add pyproject.toml, setup.cfg, .python-version to Docker workflow trigger paths (build_test_fp_x86, build_test_fp_arm, build_push_fp, integration_fp_ds_x86, integration_fp_ds_arm64) - Remove redundant uv_test_workflow.yml (covered by existing workflows)

Amazon Linux 2023 ships with Python 3.9 as the default python3. Installing python3.11 adds it as a separate binary but does not change the default. Use alternatives to set python3.11 as the default so uv resolves the correct Python version.

The alternatives command changed the default python3 to 3.11, which broke dnf on ARM since dnf depends on the system Python 3.9. Instead, use uv's --python python3.11 flag to target the correct interpreter without changing the system default.

Buildx stores images in its own cache instead of the local Docker daemon, causing forcingprocessor build to pull the old deps image from Docker Hub (without uv) instead of the locally-built one.

Packages are installed under python3.11 but datastream invokes scripts with python3 (which defaults to 3.9 on Amazon Linux 2023). This symlink is safe in the forcingprocessor image since no dnf operations run here.

Auto-cancels older runs when a new push arrives, preventing concurrent Terraform apply/destroy conflicts on shared resources.

Self-hosted runners accumulate old Docker images (21GB+) and /tmp is a 3.9GB tmpfs. docker save fails silently when space runs out. - Add docker system prune at start of build and test jobs - Use runner.temp (on disk) instead of /tmp (RAM-based tmpfs)

JordanLaserGit

Great work @kamalcou and @harshavemula-ua!

Added instructions for creating a Python virtual environment using 'uv'.

Copilot

Pull request overview

This PR integrates the uv Python tooling into the repository, updating dependency management/packaging metadata, Docker build steps, example configs, and CI workflow triggers to align with Python 3.11 and uv-based installs.

Changes:

Add a pinned requirements.txt and uv.lock, and migrate pyproject.toml to a uv build backend.
Update Dockerfiles to install/use uv and move base images to Python 3.11.
Expand GitHub Actions workflow paths filters and bump workflow Python versions to 3.11; update example config and README instructions.

Reviewed changes

Copilot reviewed 18 out of 21 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
requirements.txt	Adds a pinned requirements set for uv/pip installs.
pyproject.toml	Introduces PEP 621 project metadata + uv build backend + script entry point.
docker/Dockerfile.forcingprocessor-deps	Switches base deps image to Python 3.11 and installs/uses uv.
docker/Dockerfile.forcingprocessor	Uses uv to install forcingprocessor + pytest in container.
configs/conf_fp.json	Updates example config to use repo-relative paths and local output directory.
README.md	Documents uv installation and uv-based run instructions.
.python-version	Pins repo Python version to 3.11.
.gitignore	Ignores generated `data/forcing` outputs.
.github/workflows/integration_fp_ds_x86.yaml	Expands workflow path triggers and adds concurrency control.
.github/workflows/integration_fp_ds_arm64.yaml	Expands workflow path triggers and adds concurrency control.
.github/workflows/forcingprocessor_weights.yaml	Updates CI to Python 3.11 and expands path triggers.
.github/workflows/forcingprocessor_sources_nwm_google.yaml	Updates CI to Python 3.11 and expands path triggers.
.github/workflows/forcingprocessor_sources_nwm_aws.yaml	Updates CI to Python 3.11 and expands path triggers.
.github/workflows/forcingprocessor_sources_nomads.yaml	Updates CI to Python 3.11 and expands path triggers.
.github/workflows/forcingprocessor_plotting.yaml	Updates CI to Python 3.11 and expands path triggers.
.github/workflows/forcingprocessor_output_opts.yaml	Updates CI to Python 3.11 and expands path triggers.
.github/workflows/forcingprocessor_nrds.yaml	Updates CI to Python 3.11 and removes conflicting setup-python usage.
.github/workflows/build_test_fp_x86.yaml	Expands workflow path triggers for packaging-related files.
.github/workflows/build_test_fp_arm.yaml	Expands workflow path triggers and improves cleanup/artifact paths.
.github/workflows/build_push_fp.yaml	Expands workflow path triggers for packaging-related files.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

pyproject.toml

@@ -1,6 +1,105 @@
+[project]
+name = "forcingprocessor"
+version = "0.1.0"


pyproject.toml

+requires-python = ">=3.11"
+dependencies = [
+    "affine==2.4.0",
+    "aiohappyeyeballs==2.6.1",
+    "aiohttp==3.13.3",
+    "aiosignal==1.4.0",
+    "attrs==25.4.0",
+    "boto3==1.42.49",
+    "botocore==1.42.49",
+    "certifi==2026.1.4",
+    "cffi==2.0.0",
+    "cftime==1.6.5",
+    "charset-normalizer==3.4.4",
+    "click==8.3.1",
+    "click-plugins==1.1.1.2",
+    "cligj==0.7.2",
+    "cloudpickle==3.1.2",
+    "contourpy==1.3.3",
+    "cryptography==46.0.5",
+    "cycler==0.12.1",
+    "dask==2026.1.2",
+    "decorator==5.2.1",
+    "exactextract==0.3.0",
+    "fonttools==4.61.1",
+    "frozenlist==1.8.0",
+    "fsspec==2026.2.0",
+    "gcsfs==2026.2.0",
+    "geopandas==1.1.2",
+    "google-api-core==2.29.0",
+    "google-auth==2.48.0",
+    "google-auth-oauthlib==1.2.4",
+    "google-cloud-core==2.5.0",
+    "google-cloud-storage==3.9.0",
+    "google-cloud-storage-control==1.10.0",


docker/Dockerfile.forcingprocessor-deps

+RUN curl -LsSf https://astral.sh/uv/install.sh | sh
+ENV PATH="/root/.local/bin:${PATH}"


README.md

+## install requirement files

-## Run the forcingprocessor
 ```
-python src/forcingprocessor/processor.py ./configs/conf.json
+   uv pip install -r requirements.txt
+```
+## make directory for output data
+```


.github/workflows/build_test_fp_x86.yaml

    paths:
-      - 'docker/**' 
+      - 'docker/**'
      - 'src/forcingprocessor/**'
      - '!src/forcingprocessor/README.md'
+      - 'pyproject.toml'
+      - 'setup.cfg'
+      - '.python-version'


pyproject.toml

+description = "Tool to convert nwm forcing netcdfs to ngen compatible files"
+readme ="README.md"
+authors = [
+    { name = "Jordan J. Laser", email = " jlaser@lynker.com" }


pyproject.toml

+[project.scripts]
+forcingprocessor = "forcingprocessor:main"
+


docker/Dockerfile.forcingprocessor-deps

 FROM amazonlinux:2023 AS forcingprocessor_base

-RUN dnf -y install pigz tar git python3.9 python3-pip
+RUN dnf -y install pigz tar git python3.11 python3.11-pip


UV package intergration

fe5cb0e

JordanLaserGit mentioned this pull request Mar 11, 2026

Workflow triggers #71

Open

harshavemula-ua and others added 11 commits March 11, 2026 14:46

Add GitHub Actions workflow for forcing processor tests

fd5c0fa

This workflow installs Python dependencies, configures AWS, and sets up testing for the forcing processor module.

Rename workflow for processor output testing

7e5a53b

Update UV installation and workflow steps-Kamal

d51d901

Modify pytest commands to use 'uv run'

3eb4019

Updated pytest commands to run with 'uv run' for better compatibility.

Downgrade h5py from 3.15.1 to 3.14.0 to fix HDF5 library conflict

cde8694

h5py 3.15.1 and netcdf4 1.7.4 bundle conflicting HDF5 C libraries, causing RuntimeError in H5DSget_num_scales when both are loaded. h5py 3.14.0 does not exhibit this conflict. See h5py/h5py#2453.

Update requirements.txt to match h5py downgrade in pyproject.toml

9f345e6

Regenerate uv.lock for h5py 3.14.0 downgrade

d3eb056

JordanLaserGit mentioned this pull request Mar 12, 2026

pinned h5py version #69

Closed

12 tasks

harshavemula-ua added 4 commits March 12, 2026 14:12

Remove unnecessary QEMU and buildx from x86 integration workflow

57ea630

Buildx stores images in its own cache instead of the local Docker daemon, causing forcingprocessor build to pull the old deps image from Docker Hub (without uv) instead of the locally-built one.

Symlink python3 to python3.11 in forcingprocessor image

6552f46

Packages are installed under python3.11 but datastream invokes scripts with python3 (which defaults to 3.9 on Amazon Linux 2023). This symlink is safe in the forcingprocessor image since no dnf operations run here.

Add concurrency groups to integration workflows

e6f1b18

Auto-cancels older runs when a new push arrives, preventing concurrent Terraform apply/destroy conflicts on shared resources.

JordanLaserGit requested review from JordanLaserGit and benlee0423 March 13, 2026 15:13

JordanLaserGit approved these changes Mar 13, 2026

View reviewed changes

kamalcou added 2 commits March 13, 2026 11:18

Add Python virtual environment setup instructions

22252e5

Added instructions for creating a Python virtual environment using 'uv'.

Correct capitalization in README for virtual environment

a0fc41a

JordanLaserGit mentioned this pull request Mar 13, 2026

Python Package Versions CIROH-UA/ngen-datastream#198

Open

arpita0911patel requested a review from Copilot March 13, 2026 21:01

Copilot started reviewing on behalf of arpita0911patel March 13, 2026 21:01 View session

Copilot AI reviewed Mar 13, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

UV package intergration#70

UV package intergration#70
kamalcou wants to merge 18 commits intomainfrom
UV-package

kamalcou commented Mar 11, 2026 •

edited by JordanLaserGit

Loading

Uh oh!

JordanLaserGit left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		RUN curl -LsSf https://astral.sh/uv/install.sh \| sh
		ENV PATH="/root/.local/bin:${PATH}"

Conversation

kamalcou commented Mar 11, 2026 • edited by JordanLaserGit Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Additions

Removals

Changes

Testing

Screenshots

Notes

Todos

Checklist

Testing checklist

Target Environment support

Accessibility

Other

Uh oh!

JordanLaserGit left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kamalcou commented Mar 11, 2026 •

edited by JordanLaserGit

Loading