Skip to content

HYRAX-2043, make the build_dmrpp_h4 always save the eos projection du… #1282

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kyang2014 merged 8 commits into from
Mar 31, 2026
Merged

HYRAX-2043, make the build_dmrpp_h4 always save the eos projection du… #1282

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
a47d7f0
HYRAX-2043, make the build_dmrpp_h4 always save the eos projection du…
kyang2014 Mar 25, 2026
5ea3750
HYRAX-2043, update the baseline file and correct a typo
kyang2014 Mar 25, 2026
7ffd8f6
HYRAX-2043, 1) Make the build_dmrpp write the 1-byte eos grid variabl…
kyang2014 Mar 26, 2026
7363650
just check if macos build can get passed.
kyang2014 Mar 26, 2026
e35ee09
HYRAX-2043, update comments and see if travis gets passed.
kyang2014 Mar 30, 2026
51e3a32
Merge branch 'master' into hyrax-2043
kyang2014 Mar 31, 2026
ee376aa
HYRAX-2043, update the baselines to keep the dmrVersion consistent wi…
kyang2014 Mar 31, 2026
92fa180
HYRAX-2043, attempt to make github build-OS happy.
kyang2014 Mar 31, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 18 additions & 17 deletions modules/dmrpp_module/build_dmrpp_h4/build_dmrpp_util_h4.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1399,7 +1399,7 @@ bool get_chunks_for_an_array(const string& filename, int32 sd_id, int32 file_id,
}

// Currently this function is only for CF grid_mapping dummy variable.
bool handle_chunks_for_none_array(BaseType *btp, bool disable_missing_data, string &err_msg) {
bool handle_chunks_for_none_array(BaseType *btp, string &err_msg) {

bool ret_value = false;

Expand All @@ -1410,25 +1410,26 @@ bool handle_chunks_for_none_array(BaseType *btp, bool disable_missing_data, stri

auto attr = d4_attrs->find("eos_cf_grid_mapping");

if (disable_missing_data == false) {

// Here we don't bother to check the attribute value since this CF grid variable value is dummy.
if (attr) {

auto db = dynamic_cast<DmrppByte *>(btp);
if (!db) {
err_msg = "Expected to find a DmrppByte instance but did not in handle_chunks_for_none_array";
return false;
}
// Since this is just a one-byte variable, regardless the disable_missing data flag,
// we always save the value in the dmrpp file rather than in the missing data side car file.

VERBOSE(cerr<<"For none_array cf dummy grid variable: var name: "<<btp->name() <<endl);
// Here we don't bother to check the attribute value since this CF grid variable value is dummy.
if (attr) {

char buf='p';
db->set_missing_data(true);
db->set_value((dods_byte)buf);
db->set_read_p(true);

auto db = dynamic_cast<DmrppByte *>(btp);
if (!db) {
err_msg = "Expected to find a DmrppByte instance but did not in handle_chunks_for_none_array";
return false;
}

VERBOSE(cerr<<"For none_array cf dummy grid variable: var name: "<<btp->name() <<endl);

char buf='p';
db->set_missing_data(true);
db->set_value((dods_byte)buf);
db->set_read_p(true);

}
ret_value = true;
}
Expand Down Expand Up @@ -1456,7 +1457,7 @@ bool get_chunks_for_a_variable(const string& filename,int32 sd_id, int32 file_id
return get_chunks_for_an_array(filename,sd_id,file_id, btp,disable_missing_data);
default: {
string err_msg;
bool ret_value = handle_chunks_for_none_array(btp,disable_missing_data,err_msg);
bool ret_value = handle_chunks_for_none_array(btp,err_msg);
if (ret_value == false) {
if (err_msg.empty() == false) {
close_hdf4_file_ids(sd_id,file_id);
Expand Down
42 changes: 35 additions & 7 deletions modules/dmrpp_module/build_dmrpp_util.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@
#include "DmrppD4Group.h"
#include "DmrppArray.h"
#include "DmrppStructure.h"
#include "DmrppByte.h"
#include "D4ParserSax2.h"

#include "UnsupportedTypeException.h"
Expand Down Expand Up @@ -2188,19 +2189,25 @@
dataset = H5Dopen2(file, real_path_candidate.c_str(), H5P_DEFAULT);
}

// Here we need to handle a special case for a dmr file generated by the EnableCF option in the HDF5 handler.
// The netCDF-4's pure dimension is mapped to a fake coordinate by the EnableCF option and is ignored by the default option.
// However, netCDF-4 still stores this dimension as an HDF5 variable with 0 values. The EnableCF option replaces
// those 0 values with 0,1,2... as the fake coordinate. So here we need to ignore this kind of variables and let the
// code after this call to handle it as the EnableCF required.

VERBOSE(cerr << prolog << "Working on: " << FQN << endl);
// Note for the majority of cases, we haven't called H5Dopen2 to open this HDF5 dataset(variable) yet, so we need to do it.
if (dataset < 0) {
dataset = H5Dopen2(file, FQN.c_str(), H5P_DEFAULT);

// Here we need to handle a special case for a dmr file generated by the EnableCF option in the HDF5 handler.
// The netCDF-4's pure dimension is mapped to a fake coordinate by the EnableCF option and is ignored by the default option.
// However, netCDF-4 still stores this dimension as an HDF5 variable with 0 values. The EnableCF option replaces
// those 0 values with 0,1,2... as the fake coordinate. So here we need to ignore this kind of variables and let the
// code after this call to handle it as the EnableCF required.

// It is possible that this is a variable that contains data outside the original HDF5 file.
// That means this variable is added to the dmr not based on the original HDF5 file. If this is the case, it should also be handled later.
if (dataset < 0) {
VERBOSE(cerr << prolog << "WARNING: HDF5 dataset '" << FQN << "' cannot be opened." << endl);
VERBOSE(cerr << prolog << "However, this maybe is a variable that contains the missing data." << endl);
VERBOSE(cerr << prolog << "Check the generated dmrpp file." << endl);
}
else if(check_enable_cf_fake_cv(btp, FQN) == true)
else if(check_enable_cf_fake_cv(btp, FQN) == true)// Here dataset = -1 is not an error; it will be handled later.
dataset = -1;
}
}
Expand Down Expand Up @@ -2301,6 +2308,27 @@

if (d4_attrs->empty() == false) {

if (btp->type() == dods_byte_c) {

Check failure on line 2311 in modules/dmrpp_module/build_dmrpp_util.cc

View check run for this annotation

SonarQubeCloud / SonarCloud Code Analysis

Refactor this code to not nest more than 3 if|for|do|while|switch statements. 


See more on https://sonarcloud.io/project/issues?id=opendap-bes-modules&issues=AZ0rbCoDGCUUzfFYUR5D&open=AZ0rbCoDGCUUzfFYUR5D&pullRequest=1282

auto attr = d4_attrs->find("grid_mapping_name");

// Since this is just a one-byte variable, we always save the value in the dmrpp file rather than in the missing data side car file.
if (attr) {

auto db = dynamic_cast<DmrppByte *>(btp);
if (!db) {
string err_msg = "Expected to find a DmrppByte instance but did not.";
throw BESInternalError(err_msg, __FILE__, __LINE__);
}
VERBOSE(cerr<<"For none_array cf dummy grid variable: var name: "<<btp->name() <<endl);

char buf='p';
db->set_missing_data(true);
db->set_value((dods_byte)buf);
db->set_read_p(true);

}
}
D4Attribute *attr = d4_attrs->find("units");
if (attr) {
string attr_value = attr->value(0);
Expand Down
19 changes: 12 additions & 7 deletions modules/dmrpp_module/get_dmrpp/unit-tests/gen_dmrpp_side_car_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,10 @@ def test_gen_dmrpp_side_car(self):
# Hacky removal of lines that otherwise show spurious failure
# due to test brittleness. (Better fix would be to run same version
# stripping as on non-python tests)
dmrpp_lines_after_79.pop(220)
baseline_lines_after_79.pop(220)
# Although the above statement may be true,
# the following two lines may cause IndexError,however, github macOS build keeps throwing errors.
dmrpp_lines_after_79.pop(216)
baseline_lines_after_79.pop(216)

self.assertEqual(dmrpp_lines_after_79 ,baseline_lines_after_79)

Expand All @@ -56,8 +58,10 @@ def test_gen_dmrpp_side_car2(self):
# Hacky removal of lines that otherwise show spurious failure
# due to test brittleness. (Better fix would be to run same version
# stripping as on non-python tests)
dmrpp_lines_after_19.pop(221)
baseline_lines_after_19.pop(221)
# The following two lines may cause IndexError and they are not necessary since the testing file should not be changed.
# If the testing file is changed, the right way is to update the baseline file.
#dmrpp_lines_after_19.pop(221)
#baseline_lines_after_19.pop(221)

self.assertEqual(dmrpp_lines_after_19 ,baseline_lines_after_19)

Expand All @@ -68,7 +72,7 @@ def test_gen_dmrpp_side_car_h4_nsc(self):
if not os.environ.get('PRESERVE_TEST_ASSETS'):
self.addCleanup(os.remove, "grid_1_2d.hdf.dmrpp")

# Since we also add the dmrpp metadata generation informatio for the HDF4 files,
# Since we also add the dmrpp metadata generation information for the HDF4 files,
# we need to ignore those information when doing comparision.
with open('grid_1_2d.hdf.dmrpp') as f:
dmrpp_lines_after_54 = f.readlines()[54:]
Expand All @@ -95,8 +99,9 @@ def test_gen_dmrpp_side_car_h5_cf(self):
# Hacky removal of lines that otherwise show spurious failure
# due to test brittleness. (Better fix would be to run same version
# stripping as on non-python tests)
dmrpp_minus_18_lines.pop(62)
baseline_minus_18_lines.pop(62)
# Although the above statement may be true, the following lines may cause IndexError.So comment them out.
#dmrpp_minus_18_lines.pop(62)
#baseline_minus_18_lines.pop(62)

self.assertEqual(dmrpp_minus_18_lines ,baseline_minus_18_lines)

Expand Down
28 changes: 12 additions & 16 deletions modules/dmrpp_module/get_dmrpp/unit-tests/grid_2_2d_ps.hdf.dmrpp.baseline
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ END
</Attribute>
<Attribute name="build_dmrpp_metadata" type="Container">
<Attribute name="created" type="String">
<Value>2025-06-16T11:59:01Z</Value>
<Value>2026-03-25T20:48:03Z</Value>
</Attribute>
<Attribute name="build_dmrpp" type="String">
<Value>3.21.1</Value>
Expand All @@ -73,7 +73,7 @@ END
<Value>libdap-3.21.1</Value>
</Attribute>
<Attribute name="invocation" type="String">
<Value>build_dmrpp_h4 -f /Users/myang6/work/opendap/hyrax-1781/bes/modules/dmrpp_module/get_dmrpp/unit-tests/grid_2_2d_ps.hdf -r grid_2_2d_ps.hdf.dmr -u OPeNDAP_DMRpp_DATA_ACCESS_URL -D -M</Value>
<Value>build_dmrpp_h4 -f /Users/myang6/work/opendap/hyrax-2043/bes/modules/dmrpp_module/get_dmrpp/unit-tests/grid_2_2d_ps.hdf -r grid_2_2d_ps.hdf.dmr -u OPeNDAP_DMRpp_DATA_ACCESS_URL -D -M</Value>
</Attribute>
</Attribute>
<Group name="NPGrid">
Expand Down Expand Up @@ -102,9 +102,7 @@ END
<Attribute name="eos_cf_grid_mapping" type="String">
<Value>NPGrid eos_cf_projection</Value>
</Attribute>
<dmrpp:chunks fillValue="0" byteOrder="LE">
<dmrpp:chunk offset="9786" nBytes="1" href="grid_2_2d_ps.hdf_mvs.h5" />
</dmrpp:chunks>
<dmrpp:missingdata>cA==</dmrpp:missingdata>
</Byte>
<Float64 name="XDim_NPGrid">
<Dim name="/XDim_NPGrid"/>
Expand All @@ -125,7 +123,7 @@ END
</Attribute>
<dmrpp:chunks compressionType="deflate" deflateLevel="4" fillValue="0" byteOrder="LE">
<dmrpp:chunkDimensionSizes>4</dmrpp:chunkDimensionSizes>
<dmrpp:chunk offset="9787" nBytes="33" chunkPositionInArray="[0]" href="grid_2_2d_ps.hdf_mvs.h5" />
<dmrpp:chunk offset="21396" nBytes="33" chunkPositionInArray="[0]" href="grid_2_2d_ps.hdf_mvs.h5" />
</dmrpp:chunks>
</Float64>
<Float64 name="YDim_NPGrid">
Expand All @@ -147,7 +145,7 @@ END
</Attribute>
<dmrpp:chunks compressionType="deflate" deflateLevel="4" fillValue="0" byteOrder="LE">
<dmrpp:chunkDimensionSizes>5</dmrpp:chunkDimensionSizes>
<dmrpp:chunk offset="9820" nBytes="38" chunkPositionInArray="[0]" href="grid_2_2d_ps.hdf_mvs.h5" />
<dmrpp:chunk offset="21429" nBytes="38" chunkPositionInArray="[0]" href="grid_2_2d_ps.hdf_mvs.h5" />
</dmrpp:chunks>
</Float64>
<Float64 name="Latitude">
Expand All @@ -161,7 +159,7 @@ END
</Attribute>
<dmrpp:chunks compressionType="deflate" deflateLevel="4" fillValue="0" byteOrder="LE">
<dmrpp:chunkDimensionSizes>5 4</dmrpp:chunkDimensionSizes>
<dmrpp:chunk offset="9858" nBytes="171" chunkPositionInArray="[0,0]" href="grid_2_2d_ps.hdf_mvs.h5" />
<dmrpp:chunk offset="21467" nBytes="171" chunkPositionInArray="[0,0]" href="grid_2_2d_ps.hdf_mvs.h5" />
</dmrpp:chunks>
</Float64>
<Float64 name="Longitude">
Expand All @@ -175,7 +173,7 @@ END
</Attribute>
<dmrpp:chunks compressionType="deflate" deflateLevel="4" fillValue="0" byteOrder="LE">
<dmrpp:chunkDimensionSizes>5 4</dmrpp:chunkDimensionSizes>
<dmrpp:chunk offset="10029" nBytes="171" chunkPositionInArray="[0,0]" href="grid_2_2d_ps.hdf_mvs.h5" />
<dmrpp:chunk offset="21638" nBytes="171" chunkPositionInArray="[0,0]" href="grid_2_2d_ps.hdf_mvs.h5" />
</dmrpp:chunks>
</Float64>
<Group name="Data_Fields">
Expand Down Expand Up @@ -224,9 +222,7 @@ END
<Attribute name="eos_cf_grid_mapping" type="String">
<Value>SPGrid eos_cf_projection</Value>
</Attribute>
<dmrpp:chunks fillValue="0" byteOrder="LE">
<dmrpp:chunk offset="10200" nBytes="1" href="grid_2_2d_ps.hdf_mvs.h5" />
</dmrpp:chunks>
<dmrpp:missingdata>cA==</dmrpp:missingdata>
</Byte>
<Float64 name="XDim_SPGrid">
<Dim name="/XDim_SPGrid"/>
Expand All @@ -247,7 +243,7 @@ END
</Attribute>
<dmrpp:chunks compressionType="deflate" deflateLevel="4" fillValue="0" byteOrder="LE">
<dmrpp:chunkDimensionSizes>3</dmrpp:chunkDimensionSizes>
<dmrpp:chunk offset="10201" nBytes="26" chunkPositionInArray="[0]" href="grid_2_2d_ps.hdf_mvs.h5" />
<dmrpp:chunk offset="31233" nBytes="26" chunkPositionInArray="[0]" href="grid_2_2d_ps.hdf_mvs.h5" />
</dmrpp:chunks>
</Float64>
<Float64 name="YDim_SPGrid">
Expand All @@ -269,7 +265,7 @@ END
</Attribute>
<dmrpp:chunks compressionType="deflate" deflateLevel="4" fillValue="0" byteOrder="LE">
<dmrpp:chunkDimensionSizes>4</dmrpp:chunkDimensionSizes>
<dmrpp:chunk offset="10227" nBytes="32" chunkPositionInArray="[0]" href="grid_2_2d_ps.hdf_mvs.h5" />
<dmrpp:chunk offset="31259" nBytes="32" chunkPositionInArray="[0]" href="grid_2_2d_ps.hdf_mvs.h5" />
</dmrpp:chunks>
</Float64>
<Float64 name="Latitude">
Expand All @@ -283,7 +279,7 @@ END
</Attribute>
<dmrpp:chunks compressionType="deflate" deflateLevel="4" fillValue="0" byteOrder="LE">
<dmrpp:chunkDimensionSizes>4 3</dmrpp:chunkDimensionSizes>
<dmrpp:chunk offset="10259" nBytes="83" chunkPositionInArray="[0,0]" href="grid_2_2d_ps.hdf_mvs.h5" />
<dmrpp:chunk offset="31291" nBytes="83" chunkPositionInArray="[0,0]" href="grid_2_2d_ps.hdf_mvs.h5" />
</dmrpp:chunks>
</Float64>
<Float64 name="Longitude">
Expand All @@ -297,7 +293,7 @@ END
</Attribute>
<dmrpp:chunks compressionType="deflate" deflateLevel="4" fillValue="0" byteOrder="LE">
<dmrpp:chunkDimensionSizes>4 3</dmrpp:chunkDimensionSizes>
<dmrpp:chunk offset="10342" nBytes="62" chunkPositionInArray="[0,0]" href="grid_2_2d_ps.hdf_mvs.h5" />
<dmrpp:chunk offset="31374" nBytes="62" chunkPositionInArray="[0,0]" href="grid_2_2d_ps.hdf_mvs.h5" />
</dmrpp:chunks>
</Float64>
<Group name="Data_Fields">
Expand Down
Loading
Loading