Serve weather data from GHCNh through a generic load_data api

[travis-recurve]

Your checklist for this pull request

Please review the guidelines for contributing to this repository.

• Make sure you are requesting to pull a feature/bugfix branch (right side). Don't request your master!
• Make sure tests pass and coverage has not fallen (177 tests, fully offline; coverage 99%).
• Update the CHANGELOG.md to describe your changes in a bulleted list under the "Development" section at the top of the changelog.
• Code style: PEP 008. (Not blackened, per maintainer preference.)
• New functions and classes have numpy-style docstrings. The sphinx docs are retired in this pull request; documentation moves to opendsm.energy.
• All git commits have the "Signed-off-by" message for the Developer Certificate of Origin.

Description

Serves observed weather data from NOAA GHCNh, resolving #100. ISD and GSOD stopped receiving data 2025-08-27; GHCNh serves the same stations through the same NCEI access api (endpoint identified by @k-wolfe99 in #100) with history back decades and current data. Based on #102; the commits unique to this pull request start at "Delete dead FTP fetch path".

Because this is a breaking release, the api is consolidated rather than aliased:

• load_data(usaf_id, start, end, frequency, variables) returns a DataFrame and a warnings list, replacing the load_isd_*/load_gsod_* family; daily values are means of hourly values. GHCNh variables beyond temperature (dew point, relative humidity, wind speed, ...) are available through variables
• ISDStation is renamed WeatherStation, with ghcn_id, ghcn_first_year, and ghcn_last_year attributes
• Caching stores under new keys, so ISD-era cache entries are never served; the deleted ISD/GSOD code paths, filename helpers, and CLI commands are removed rather than deprecated

Station registry:

• Every station maps to its GHCNh id (ICAO join → USW000{wban} pattern → nearest neighbor within 5 km, with a 50 km sanity guard that caught one ICAO reuse). 349 stations with no GHCNh counterpart are removed; 99.8% of high-quality stations map
• One longstanding registry error surfaced by the mapping is corrected: the isd-history longitude for 725374 (Ann Arbor Municipal, KARB) is off by 4 degrees, placing the station in Lake Michigan
• Quality ratings are computed from the GHCNh inventory with the same thresholds as before; ratings can also be computed for the period being requested (rank_stations(..., rating_period=...), get_station_quality), rating the five calendar years ending two years after the request's last date
• ghcn_first_year/ghcn_last_year record each station's data era; about a fifth of the registry, nearly all low quality, has no observations after 2023

Validation:

• scripts/validate_ghcnh_against_isd.py compares both sources through the library's pipeline over a stratified sample of 210 stations (every state and territory, all quality tiers) × 4 years: over 572 comparable station-years, the median hourly deviation is 0.0001 °C and the largest annual-mean deviation is 0.14 °C. GHCNh hourly coverage matches or exceeds ISD's at the median

[k-wolfe99]

Good thinking on using a coordinate distance search to map GHCNh stations to ISD equivalents when they don't match by ICAO, WBAN, or USAF identifiers. I've been working on a tool to pull data from the GHCNh into SkySpark, and I found a station search endpoint on the NOAA search service API that lets you search for weather stations within a bounding box of coordinates. It returns the weather parameters available at each site and their coverage over the date span you provide in the request. I've found it useful for searching for weather stations and getting live data availability. Here's an example search around SFO:

https://www.ncei.noaa.gov/access/services/search/v1/data?dataset=global-historical-climatology-network-hourly&startDate=2022-01-01&endDate=2100-12-31&bbox=38.070,-122.946,37.172,-121.812&limit=10000

You folks seem to have a pretty robust process for evaluating data quality and ranking stations, so probably not a useful addition here. I was just getting tired of depending on NOAA to update their station inventory file, which hasn't been updated since February. It also provides per-parameter coverage which was important to me.

[travis-recurve]

Thanks, that's a useful find. For the id mapping itself we're in good shape: the ICAO/WBAN/coordinate mapping in this PR was cross-validated against isd-history coordinates and an overlap comparison of served data, so the search endpoint would mostly confirm what the packaged registry already encodes.

Where it looks genuinely valuable is the per-station-year dataTypes listing in its responses. The registry's ghcn_inventory table carries total monthly observation counts per station, with no per-variable dimension, so station quality ratings and ranking are effectively temperature-anchored, and a request for something like variables=("wind_speed",) can return an all-NaN column from a station that simply never reports it. Presence-by-variable-by-year from this endpoint is exactly the information needed to close that: it could ship in a future registry update and feed station selection for non-temperature variables.

[k-wolfe99]

Yep, exactly my thoughts. You've already validated the mapping by cross comparison of data during overlapping coverage dates so no need to re-invent the wheel there, that analysis was thoroughly convincing. As you mentioned, only potential area of added value from this endpoint is for improving the detail of station quality ratings.

Great work so far!

[jason-recurve]

same calculation is written out separately in 3 spots: here, get_station_qualities(), and _compute_station_quality_from_ghcnh() in database.py

should consolidate them or at least share constants if it's an optimization issue