add restriction variables and modified ioda-converters originally from Praveen.Kumar

[HyundeokChoi-NOAA]

This PR have 2 updates.

1. Enable extraction of redistribution‑restriction metadata by adding restrictionFlag (RSRD) and restrictionExpiration (EXPRSRD) to the BUFR→IODA conversion. This ensures downstream restriction‑filter logic has the required fields available in MetaData.
2. Add modified IODA-Converters that originally from Praveen in PRs (YAML and Python script for the AHI CSR BUFR #45, YAML and Python script for the SEV ASR BUFR #55, YAMLs and Python script for the GOES CSR BUFR #61, YAMLs and Python script for the GOES ASR BUFR #62, YAMLs and Python script for the AVHRR BUFR #69).

[CoryMartin-NOAA]

Thanks @HyundeokChoi-NOAA. Doing it in batch form like this is fine, but it makes it hard to see any additional changes you made to Praveen's initial PRs. Can you post here in a comment what you had to change in his PRs (or I misunderstood from our meeting on Monday, but I thought there were things you had to add)?

[Copilot]

Pull request overview

This PR expands BUFR→IODA conversion support by (a) exposing redistribution restriction metadata in several existing PrepBUFR/GNSSRO configs and (b) adding a set of additional atmosphere radiance/ozone converter YAMLs + Python entrypoints (largely ported from earlier work).

Changes:

• Add restrictionFlag (RSRD) and restrictionExpiration (EXPRSRD) to multiple PrepBUFR and GNSSRO encoder configs as MetaData/*.
• Add multiple new radiance converter YAMLs and corresponding ObsBuilder scripts (e.g., SEV ASR, GOES CSR/ASR, AHI CSR, AMSR2, ATMS/IASI/CrIS variants).
• Add an OMPS ozone retrieval config/script entrypoint (retrieval_ozone_ompslp).

Reviewed changes

Copilot reviewed 29 out of 29 changed files in this pull request and generated 17 comments.

Show a summary per file

File	Description
dump/scripts/atmosphere/retrieval_ozone_ompslp.py	New OMPS ozone retrieval script entrypoint using BufrOzoneObsBuilder.
dump/scripts/atmosphere/radiance_sevasr.py	New SEVIRI ASR builder; derives and injects sensorChannelNumber.
dump/scripts/atmosphere/radiance_iasidb.py	New IASI “db” radiance builder entrypoint.
dump/scripts/atmosphere/radiance_gsrcsr.py	New GOES CSR builder; derives and injects sensorChannelNumber.
dump/scripts/atmosphere/radiance_gsrasr.py	New GOES ASR builder; derives and injects sensorChannelNumber.
dump/scripts/atmosphere/radiance_gome.py	New “gome” radiance builder entrypoint.
dump/scripts/atmosphere/radiance_esiasi.py	New “esiasi” radiance builder entrypoint.
dump/scripts/atmosphere/radiance_esatms.py	New “esatms” radiance builder entrypoint.
dump/scripts/atmosphere/radiance_crsfdb.py	New CrIS “db” radiance builder entrypoint.
dump/scripts/atmosphere/radiance_atmsdb.py	New ATMS “db” radiance builder entrypoint.
dump/scripts/atmosphere/radiance_amsr2.py	New AMSR2 radiance builder entrypoint.
dump/scripts/atmosphere/radiance_ahicsr.py	New AHI CSR builder; derives and injects sensorChannelNumber.
dump/config/atmosphere/retrieval_ozone_ompslp.yaml	New OMPS ozone retrieval YAML mapping + encoder definition.
dump/config/atmosphere/radiance_sevasr.yaml	New SEVIRI ASR YAML mapping + encoder definition.
dump/config/atmosphere/radiance_iasidb.yaml	New IASI “db” YAML mapping + encoder definition.
dump/config/atmosphere/radiance_gsrcsr.yaml	New GOES CSR YAML mapping + encoder definition.
dump/config/atmosphere/radiance_gsrasr.yaml	New GOES ASR YAML mapping + encoder definition.
dump/config/atmosphere/radiance_gome.yaml	New “gome” YAML mapping + encoder definition.
dump/config/atmosphere/radiance_esiasi.yaml	New “esiasi” YAML mapping + encoder definition.
dump/config/atmosphere/radiance_esatms.yaml	New “esatms” YAML mapping + encoder definition.
dump/config/atmosphere/radiance_crsfdb.yaml	New CrIS “db” YAML mapping + encoder definition.
dump/config/atmosphere/radiance_atmsdb.yaml	New ATMS “db” YAML mapping + encoder definition.
dump/config/atmosphere/radiance_amsr2.yaml	New AMSR2 YAML mapping + encoder definition.
dump/config/atmosphere/radiance_ahicsr.yaml	New AHI CSR YAML mapping + encoder definition.
dump/config/atmosphere/prepbufr_sfcshp.yaml	Add restriction fields to PrepBUFR SFCSHP output as MetaData/*.
dump/config/atmosphere/prepbufr_aircraft.yaml	Add restriction fields to PrepBUFR aircraft output as MetaData/*.
dump/config/atmosphere/prepbufr_adpupa.yaml	Add restriction fields to PrepBUFR ADPUPA output as MetaData/*.
dump/config/atmosphere/prepbufr_adpsfc.yaml	Add restriction fields to PrepBUFR ADPSFC output as MetaData/*.
dump/config/atmosphere/gnssro.yaml	Add restriction fields to GNSSRO output as MetaData/*.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The docstring says the channel numbers are [1, 2, ..., nchannels], but the implementation starts at 7 (np.arange(7, ...)). Please update the docstring (and/or parameter naming) so it matches the actual channel numbering being generated.

Suggested change

	Creates a 2D array of sensor channel numbers with shape (nlocs, nchannels),
	where each row is [1, 2, ..., nchannels].
	Args:
	nlocs (int): Number of observation locations.
	nchannels (int): Number of channels.
	Returns:
	np.ndarray: 2D array with shape (nlocs, nchannels).
	Creates a 2D array of sensor channel numbers with shape (nlocs, nchan_out),
	where each row is [7, 8, ..., nchannels].
	Here, `nchannels` is the highest channel number to include (inclusive), and
	the number of output channels is `nchan_out = nchannels - 7 + 1`.
	Args:
	nlocs (int): Number of observation locations.
	nchannels (int): Highest channel number to include (inclusive).
	Returns:
	np.ndarray: 2D array with shape (nlocs, nchan_out).

[Copilot]

The builder class is named BufrAtmsObsBuilder, but this script maps radiance_gome.yaml. Renaming the class (and the add_main_functions(...) call) to something gome-specific would avoid confusion when reading logs/debugging and keeps naming consistent with other radiance scripts.

Suggested change

	class BufrAtmsObsBuilder(ObsBuilder):
	def __init__(self):
	super().__init__(MAPPING_PATH, log_name=os.path.basename(__file__))
	add_main_functions(BufrAtmsObsBuilder)
	class BufrGomeObsBuilder(ObsBuilder):
	def __init__(self):
	super().__init__(MAPPING_PATH, log_name=os.path.basename(__file__))
	add_main_functions(BufrGomeObsBuilder)

[Copilot]

The sensorLongDescription says "Advanced Baseline Imager (ATMS)", but ABI is a different instrument (GOES). This looks like a copy/paste error; update the description to the correct ATMS expansion/description so global metadata is accurate.

Suggested change

	value: "Advanced Baseline Imager (ATMS) - 22 channels including the 54 and 183 GHz bands; cross-track scanning"
	value: "Advanced Technology Microwave Sounder (ATMS) - 22-channel cross-track microwave sounder including bands near 23, 31, 50–57, and 183 GHz"

[Copilot]

source: variable/solarElevation looks like a typo; other entries use variables/<name>. As written, the encoder likely won't resolve this source path and MetaData/solarElevation may be missing or cause a config error. Update to source: variables/solarElevation.

Suggested change

	source: variable/solarElevation
	source: variables/solarElevation

[CoryMartin-NOAA]

This seems like it should be fixed. Do the output files include solarElevation? I'm surprised there isn't an error from this

[HyundeokChoi-NOAA]

It was fixed and solarElevation is in there.
float solarElevation(Location) ;
solarElevation:_FillValue = 3.402823e+38f ;
solarElevation:long_name = "SOLAR ELEVATION" ;

[HyundeokChoi-NOAA]

@CoryMartin-NOAA All BUFR‑based YAML mapping files were renamed from the bufr_.yaml pattern to radiance_.yaml, and the corresponding Python scripts were updated to reference the new filenames. In addition, several builder class names were corrected to match their respective products, ensuring that each script uses the appropriate ObsBuilder subclass and CLI entry point.

radiance_ahicsr.py
Updated mapping reference
bufr_ahicsr.yaml → radiance_ahicsr.yaml

radiance_sevasr.py
Updated mapping reference
bufr_sevasr.yaml → radiance_sevasr.yaml

Corrected builder class name
BufrAhicsrObsBuilder → BufrSevasrObsBuilder

Updated CLI entry point
add_main_functions(BufrAhicsrObsBuilder) → add_main_functions(BufrSevasrObsBuilder)

radiance_gsrcsr.py
Updated mapping reference
bufr_gsrcsr.yaml → radiance_gsrcsr.yaml

radiance_gsrasr.py
Updated mapping reference
bufr_gsrasr.yaml → radiance_gsrasr.yaml

radiance_avhrr.py
Updated mapping reference
bufr_avhrr.yaml → radiance_avhrr.yaml

Corrected builder class name
BufrGsrasrObsBuilder → BufrAvhrrObsBuilder

Updated CLI entry point
add_main_functions(BufrGsrasrObsBuilder) → add_main_functions(BufrAvhrrObsBuilder)

[CoryMartin-NOAA]

thanks @HyundeokChoi-NOAA

[CoryMartin-NOAA]

@HyundeokChoi-NOAA I know these aren't your typos, but can you address the typos that copilot found? You should be able to do it by clicking through on GitHub rather than doing them manually.

[nicholasesposito]

Some comments but the vast majority looks fine. I definitely agree with some of the things copilot said regarding "variables/" and some naming conventions, so once those are fixed, I'll approve.

[nicholasesposito]

sensorID is not commented earlier in the yaml file. Is it supposed to be an output?

[HyundeokChoi-NOAA]

@nicholasesposito The sensorID variable is defined because the BUFR message contains the SIID field, but I followed the pattern used in radiance_cris-fsr.yaml, where sensorId is also defined but the corresponding encoder output is intentionally commented out.

[nicholasesposito]

There are a number of yamls that have a double end parentheses "))" after NCEP. I won't tag them all, but recommend you update to just one parenthesis ")"

[CoryMartin-NOAA]

Perhaps we open an issue to attack this later? I think we should also remove NCEP from the thing entirely.

[HyundeokChoi-NOAA]

@CoryMartin-NOAA Okay, will do that, but let me make sure those are correct.

[CoryMartin-NOAA]

@HyundeokChoi-NOAA are the AVHRR files supposed to be included here? The PR description says they should be

[HyundeokChoi-NOAA]

@CoryMartin-NOAA Hmm, they should be included here. Let me revise this PR.

[ADCollard]

Sorry for the slow reponse to this. I am trying to decide how we should address EARS/RARS and DBNET data, especially for ATMS (so eatms and atmsdb). In read_atms.f90 in GSI we loop over the three data sources (global, eatms and atmsdb) and combine them before we do the spatial averaging, rather than having three separate converters. We want to do that in obsforge too.

There is also a secondary issue with the EARS and DBNET data that is the subject of a draft PR from @emilyhcliu.

@xincjin-NOAA has done a lot of work on this, so I am adding him to the list of reviewers. Hopefully he can correct any errors in what I said.

But for now I think we need to hold off on the EARS, RARS and DBNet converters.

[xincjin-NOAA]

Sorry for the slow reponse to this. I am trying to decide how we should address EARS/RARS and DBNET data, especially for ATMS (so eatms and atmsdb). In read_atms.f90 in GSI we loop over the three data sources (global, eatms and atmsdb) and combine them before we do the spatial averaging, rather than having three separate converters. We want to do that in obsforge too.

There is also a secondary issue with the EARS and DBNET data that is the subject of a draft PR from @emilyhcliu.

@xincjin-NOAA has done a lot of work on this, so I am adding him to the list of reviewers. Hopefully he can correct any errors in what I said.

But for now I think we need to hold off on the EARS, RARS and DBNet converters.

Agree with @ADCollard. We could combine them into a single file if needed. One thing to check is whether all files use the same unit for brightnessTemperature. For example, in AMSU-A we may need to convert between db2bt or the reverse.

[ADCollard]

Agree with @ADCollard. We could combine them into a single file if needed. One thing to check is whether all files use the same unit for brightnessTemperature. For example, in AMSU-A we may need to convert between db2bt or the reverse.

@xincjin-NOAA This should not be an issue as ATMS BUFR has separate entries for TA and TB. We should ensure the same correction is being applied though.

[xincjin-NOAA]

@xincjin-NOAA This should not be an issue as ATMS BUFR has separate entries for TA and TB. We should ensure the same correction is being applied though.

Got it. Thanks

[CoryMartin-NOAA]

@HyundeokChoi-NOAA can you address @xincjin-NOAA 's comments?

[xincjin-NOAA]

Looks good to me.

Thanks.

[HyundeokChoi-NOAA]

I’ve encountered two issues in the radiance converters that need attention.

• radiance_gome: Bad parameter: Number of chunk sizes does not match the number of dimensions for variable ObsValue/brightnessTemperature
• radiance_gmi: Conversion failed

Details are in the log:
/lfs/h2/emc/da/noscrub/Hyundeok.Choi/obsForge/COMROOT/obsforge/logs/2026040600/gdas_atmos_bufr_dump_prep_error.log

Other than these two issues, the rest of the radiance processing is working fine.

[xincjin-NOAA]

• radiance_gmi: Conversion failed

bufr file is empty:

/lfs/h1/ops/prod/com/obsproc/v1.2/gdas.20260406/00/atmos/gdas.t00z.gmi1cr.tm00.bufr_d

[xincjin-NOAA]

• radiance_gome: Bad parameter: Number of chunk sizes does not match the number of dimensions for variable ObsValue/brightnessTemperature

If the file is not too large, you could try removing the chunk setting from the YAML file.

[HyundeokChoi-NOAA]

@xincjin-NOAA I didn’t make any changes to the radiance_gmi converter. For radiance_gome, after removing the chunks setting as suggested, everything runs correctly now.

[xincjin-NOAA]

looks good to me!

[HyundeokChoi-NOAA]

Thank you all!!