Map SILAM OPeNDAP variables, add unit conversions, and comprehensive tests for air quality ingest

[Copilot]

The SILAM ingest was attempting to use generic variable names that don't exist in the actual SILAM OPeNDAP dataset. Updated to use the correct SILAM v6.1 variable names with proper unit conversions and added comprehensive tests to validate the conversion logic.

Variable Mapping

• Particulate matter: cnc_PM2_5, cnc_PM10 (provided in kg/m³, converted to µg/m³)
• Gas species: vmr_O3_gas, vmr_NO2_gas, vmr_SO2_gas, vmr_CO_gas (volume mixing ratios in mole/mole)
• Air density: air_dens (kg/m³, used for volume mixing ratio conversions)

Unit Conversions

# PM variables: kg/m³ → µg/m³
cnc_PM2_5 = silam_data["cnc_PM2_5"] * KG_M3_TO_UG_M3  # 1e9

# Gas variables: volume mixing ratio → concentration (µg/m³)
def convert_vmr_to_concentration(vmr, air_density, molar_mass):
    return vmr * air_density * (molar_mass / MOLAR_MASS_AIR) * KG_M3_TO_UG_M3

Note: SILAM's vmr_*_gas variables are true volume mixing ratios (mole pollutant/mole air) as confirmed by the SILAM metadata (units: mole/mole, silam_amount_unit: mole). The conversion requires both the pollutant's molar mass and air's molar mass (0.02897 kg/mole for dry air).

Molar Masses

From SILAM metadata:

• O3: 0.048 kg/mole
• NO2: 0.046 kg/mole
• SO2: 0.064 kg/mole
• CO: 0.028 kg/mole

Code Quality

• Removed redundant variable mapping dictionary
• Extracted conversion constants to module level
• Correct VMR conversion formula accounting for molar mass ratio
• Created API/silam_conversion.py module for testable conversion utilities

Testing

Added comprehensive test coverage (8 new tests, 41 total tests passing):

VMR Conversion Tests (5 unit tests):

• Basic conversion validation (1 ppm O3 → 2030 µg/m³)
• Zero concentration handling
• All gas species (O3, NO2, SO2, CO)
• Numpy array support
• Realistic tropospheric O3 levels (50 ppb → 101.5 µg/m³)

AQI Integration Tests (3 tests):

• AQI calculation with VMR-converted gases at moderate pollution levels
• High O3 scenario (120 ppb → unhealthy for sensitive groups)
• Spatial variation with varying VMR values

All tests validate that VMR to concentration conversion is mathematically correct and that AQI calculations integrate properly with VMR-converted data.

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[cloneofghosts]

@copilot Here is the definition for each of the vmr gasses:

_FillValue: -1.74483046E9
cell_methods: hybrid: mean time: mean
long_name: Volume mixing ratio CO_gas
missing_value: -9.9999898E14
mode_distribution_type: GAS_PHASE
mode_name: GAS
mode_solubility: 0
molar_mass: 0.0280000 kg/mole
quantum: 2.9802322E-8
silam_amount_unit: mole
substance_name: CO
units: mole/mole
_ChunkSizes: 10, 224, 449

_FillValue: -1.74483046E9
cell_methods: hybrid: mean time: mean
long_name: Volume mixing ratio NO2_gas
missing_value: -9.9999898E14
mode_distribution_type: GAS_PHASE
mode_name: GAS
mode_solubility: 0
molar_mass: 0.0460000 kg/mole
quantum: 2.910383E-11
silam_amount_unit: mole
substance_name: NO2
units: mole/mole
_ChunkSizes: 10, 224, 449

_FillValue: -1.74483046E9
cell_methods: hybrid: mean time: mean
long_name: Volume mixing ratio O3_gas
missing_value: -9.9999898E14
mode_distribution_type: GAS_PHASE
mode_name: GAS
mode_solubility: 0
molar_mass: 0.0480000 kg/mole
quantum: 2.910383E-11
silam_amount_unit: mole
substance_name: O3
units: mole/mole
_ChunkSizes: 10, 224, 449

_FillValue: -1.74483046E9
cell_methods: hybrid: mean time: mean
long_name: Volume mixing ratio SO2_gas
missing_value: -9.9999898E14
mode_distribution_type: GAS_PHASE
mode_name: GAS
mode_solubility: 0
molar_mass: 0.0640000 kg/mole
quantum: 2.910383E-11
silam_amount_unit: mole
substance_name: SO2
units: mole/mole
_ChunkSizes: 10, 224, 449

[Copilot]

@copilot Here is the definition for each of the vmr gasses:

_FillValue: -1.74483046E9
...

You're absolutely right - the metadata confirms these are volume mixing ratios (units: mole/mole), not mass mixing ratios as I initially assumed.

I've corrected the conversion formula in commit cbf4ea9:

Updated conversion:

concentration (µg/m³) = VMR (mole/mole) × air_density (kg/m³) × (molar_mass_pollutant / molar_mass_air) × 1e9

Using the molar masses from the SILAM metadata you provided:

• O3: 0.048 kg/mole
• NO2: 0.046 kg/mole
• SO2: 0.064 kg/mole
• CO: 0.028 kg/mole
• Air: 0.02897 kg/mole (dry air)

All tests passing and conversion is now physically correct.

[cloneofghosts]

@copilot Can you add some tests to make sure everything is working properly? Maybe some can include AQI calculations with the other variables to make sure it gives expected values. Can add to the existing AQI tests file.

This might be outside the scope of this PR but how does the OPeNDAP download work? I've tried looking at the code and can't figure that part out.

[Copilot]

@copilot Can you add some tests to make sure everything is working properly? Maybe some can include AQI calculations with the other variables to make sure it gives expected values. Can add to the exis...

I've added comprehensive tests in commit 9acabd1:

VMR Conversion Tests (5 tests):

• Basic conversion validation (1 ppm O3 → 2030 µg/m³)
• Zero concentration handling
• All gas species (O3, NO2, SO2, CO)
• Numpy array support
• Realistic tropospheric levels (50 ppb O3 → 101.5 µg/m³)

AQI Integration Tests (3 tests):

• AQI with VMR-converted gases (moderate conditions)
• High O3 scenario (120 ppb → unhealthy for sensitive groups)
• Spatial variation with varying VMR values

All 41 tests passing (33 existing + 8 new).

Regarding OPeNDAP download: The code uses xr.open_dataset(opendap_url, engine="netcdf4") at line 200 in FMI_Silam_Local_Ingest.py. This leverages xarray's built-in OPeNDAP support via the netCDF4 library, which opens the remote dataset without downloading the entire file - it streams data on-demand. The URL pattern is constructed from the THREDDS catalog at thredds.silam.fmi.fi based on the model run time.