Skip to content

Add NOAA AQM (Air Quality Forecast Capability) ingest#588

Merged
cloneofghosts merged 5 commits into
add-silam-aqifrom
copilot/sub-pr-465
Mar 24, 2026
Merged

Add NOAA AQM (Air Quality Forecast Capability) ingest#588
cloneofghosts merged 5 commits into
add-silam-aqifrom
copilot/sub-pr-465

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Mar 24, 2026

Adds ingest for NOAA's Air Quality Model (AQFC), which provides CONUS hourly PM2.5 and Ozone forecasts at 12km resolution, updated at 06Z and 12Z (both 72-hour forecasts). Complements the existing global SILAM ingest for US locations where AQM has higher accuracy.

Ingest script (API/NOAA_AQM_Local_Ingest.py)

  • Downloads PM2.5 (PMTF, units: Particulate matter (fine) [10^-6g/m^3] = µg/m³, no conversion needed) and O3 (OZCON, units: Ozone Concentration [ppb]) GRIB2 files from NOMADS; tries grid IDs 148 and 227 for cross-version compatibility
  • OZCON is always in ppb; convert_o3_to_ug_m3 converts ppb → µg/m³ and uses ppb as the fallback for any unrecognised unit string
  • Data availability delay is ~4.5 hours after each model run time
  • Calculates and stores AQI from PM2.5 + O3 only; AQM does not provide NO2/SO2/CO
  • Uses if __name__ == '__main__': guard so pure helpers (get_latest_aqm_run, build_aqm_url, convert_o3_to_ug_m3) are importable without side effects

AQI blending design

SILAM stores a full 6-pollutant global AQI; AQM only covers PM2.5 + O3 for CONUS. Taking the max of two separate AQI values across different pollutant subsets is misleading. The recommended path when the API layer integrates these sources:

  • Use AQM concentrations for PM2.5 and O3 (where available)
  • Use SILAM concentrations for NO2, SO2, CO
  • Feed all six into a single calculate_aqi() call — the averaging windows (NowCast, 8h, 1h) are already handled internally

Tests (tests/test_noaa_aqm_ingest.py) — 31 tests

  • O3 unit conversion: ppb→µg/m³ (including the real GRIB2 string "Ozone Concentration [ppb]"), ppm→µg/m³, µg/m³ pass-through, unknown-units treated as ppb
  • build_aqm_url: correct NOMADS URL structure for both variables and both grid IDs
  • get_latest_aqm_run: always returns 06Z or 12Z, never a future timestamp
  • AQI with PM2.5+O3 subset: clean air, high-PM2.5 dominance, high-O3 dominance, spatial variation, NowCast path

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Copy link
Copy Markdown
Contributor Author

Copilot AI commented Mar 24, 2026

Just as a heads up, I was blocked by some firewall rules while working on your feedback. Expand below for details.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • www.nco.ncep.noaa.gov
    • Triggering command: /home/REDACTED/work/_temp/ghcca-node/node/bin/node /home/REDACTED/work/_temp/ghcca-node/node/bin/node --enable-source-maps /home/REDACTED/work/_temp/copilot-developer-action-main/dist/index.js (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Copilot AI changed the title [WIP] Add FMI Silam air quality ingest and wildfire smoke ingest Add NOAA AQM (Air Quality Forecast Capability) ingest Mar 24, 2026
Copilot AI requested a review from cloneofghosts March 24, 2026 16:03
@cloneofghosts
Copy link
Copy Markdown
Collaborator

@copilot Comments:

  1. I'm pretty sure both runs go out 72h just from looking at the file sizes.
  2. Both products are roughly 4.5 hours delayed. The 12Z run was uploaded to the nomads server roughly 10m ago (16:33Z) so we will need a longer delay.
  3. PM2.5 name is PMTF and units are Particulate matter (fine) [10^-6g/m^3]
  4. Ozone name is OZCON and units are Ozone Concentration [ppb]

… and O3 always-ppb handling

Co-authored-by: cloneofghosts <10248058+cloneofghosts@users.noreply.github.com>
Agent-Logs-Url: https://github.com/Pirate-Weather/pirate-weather-code/sessions/d174c63d-0e16-4cde-aee0-9259e9dfa53e
Copy link
Copy Markdown
Contributor Author

Copilot AI commented Mar 24, 2026

@copilot Comments:

  1. I'm pretty sure both runs go out 72h just from looking at the file sizes.
    ...

Fixed in 9f8b923:

  1. Forecast length — updated header comment to "both 72-hour forecasts"
  2. Availability delay — changed from timedelta(hours=2) to timedelta(hours=4, minutes=30)
  3. PM2.5 units — updated comment to show the actual GRIB2 unit string "Particulate matter (fine) [10^-6g/m^3]" (= µg/m³, no conversion needed)
  4. O3 always ppbconvert_o3_to_ug_m3 now checks for ppb first and uses it as the fallback for unknown units (previously fell through to a µg/m³ pass-through). Added a test that exercises the real GRIB2 unit string "Ozone Concentration [ppb]".

@cloneofghosts cloneofghosts marked this pull request as ready for review March 24, 2026 16:55
@cloneofghosts
Copy link
Copy Markdown
Collaborator

@copilot Tests are failing because of this issue: ModuleNotFoundError: No module named 'cartopy'

@cloneofghosts cloneofghosts merged commit 93e4507 into add-silam-aqi Mar 24, 2026
8 of 9 checks passed
@cloneofghosts cloneofghosts deleted the copilot/sub-pr-465 branch March 24, 2026 17:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants