Fix MRMS RadarOnly fallback when system publishes files late within the hour

[Copilot]

RadarOnly_QPE_01H is published at 2-minute intervals. When the MRMS system starts late, the file for a given hour lands at e.g. 20210916-220800 instead of 20210916-220000. The existing code tries only the exact HH:00:00 filename, gets a 404, and writes NaN — even though valid data for that hour exists.

Investigation confirmed this via S3 directory listings and rasterio inspection of the off-minute files (e.g. 220800, 001000, 181800, 190800), which all contain valid precipitation data. The remaining missing timestamps are genuine MRMS system outages where all three products are absent from S3.

Changes

• download.py: Add s3_list_first_key_with_prefix(base_url, key_prefix) — uses the S3 XML list API to return the first key matching a prefix, or None if empty. Prefix is URL-encoded.

• noaa/mrms/.../region_job.py: In _download_from_source, after a FileNotFoundError on an S3 RadarOnly_QPE_* download, list the hour's directory prefix and retry with the first file found. Non-RadarOnly products and non-S3 sources re-raise immediately without the listing roundtrip.

# Before: tries only HH:00:00, raises FileNotFoundError if missing
gz_path = http_download_to_disk(coord.get_url(source="s3"), self.dataset_id)

# After: on 404 for RadarOnly, find first file in same hour window
except FileNotFoundError:
    if source != "s3" or not coord.product.startswith("RadarOnly_QPE_"):
        raise
    first_key = s3_list_first_key_with_prefix(_NOAA_MRMS_S3_BASE_URL, key_prefix)
    if first_key is None:
        raise
    gz_path = http_download_to_disk(f"{_NOAA_MRMS_S3_BASE_URL}/{first_key}", ...)

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• mirrors.aliyun.com
  • Triggering command: /home/REDACTED/work/reformatters/reformatters/.venv/bin/prek /home/REDACTED/work/reformatters/reformatters/.venv/bin/prek hook-impl --hook-dir /home/REDACTED/work/reformatters/reformatters/.git/hooks --script-version 4 --hook-type=pre-commit -- k/reformatters/runame k/reformatters/r-p k/reformatters/reformatters/.ven--noprofile k/reformatters/reformatters/.venv/lib/python3.13/site-packages/k/tmp/copilot-tool-output-1772496512497-zgbz0h.txt k/re�� k/reformatters/reformatters/.venv/lib/python3.13/site-packages/kubernetes/client/models/v1_priorsh k/reformatters/reformatters/.venv/lib/python3.13/site-packages/kubernetes/client/models/v1_node_-c bash /region_job.py (dns block)
• mirrors.cloud.tencent.com
  • Triggering command: /home/REDACTED/work/reformatters/reformatters/.venv/bin/prek /home/REDACTED/work/reformatters/reformatters/.venv/bin/prek hook-impl --hook-dir /home/REDACTED/work/reformatters/reformatters/.git/hooks --script-version 4 --hook-type=pre-commit -- k/reformatters/runame k/reformatters/r-p k/reformatters/reformatters/.ven--noprofile k/reformatters/reformatters/.venv/lib/python3.13/site-packages/k/tmp/copilot-tool-output-1772496512497-zgbz0h.txt k/re�� k/reformatters/reformatters/.venv/lib/python3.13/site-packages/kubernetes/client/models/v1_priorsh k/reformatters/reformatters/.venv/lib/python3.13/site-packages/kubernetes/client/models/v1_node_-c bash /region_job.py (dns block)
• mtarchive.geol.iastate.edu
  • Triggering command: /home/REDACTED/work/reformatters/reformatters/.venv/bin/python3 /home/REDACTED/work/reformatters/reformatters/.venv/bin/python3 t.py (dns block)
  • Triggering command: /home/REDACTED/work/reformatters/reformatters/.venv/bin/python /home/REDACTED/work/reformatters/reformatters/.venv/bin/python -u -c import sys;exec(eval(sys.stdin.readline())) (dns block)
• pypi.tuna.tsinghua.edu.cn
  • Triggering command: /home/REDACTED/work/reformatters/reformatters/.venv/bin/prek /home/REDACTED/work/reformatters/reformatters/.venv/bin/prek hook-impl --hook-dir /home/REDACTED/work/reformatters/reformatters/.git/hooks --script-version 4 --hook-type=pre-commit -- k/reformatters/runame k/reformatters/r-p k/reformatters/reformatters/.ven--noprofile k/reformatters/reformatters/.venv/lib/python3.13/site-packages/k/tmp/copilot-tool-output-1772496512497-zgbz0h.txt k/re�� k/reformatters/reformatters/.venv/lib/python3.13/site-packages/kubernetes/client/models/v1_priorsh k/reformatters/reformatters/.venv/lib/python3.13/site-packages/kubernetes/client/models/v1_node_-c bash /region_job.py (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.