bufr-query as a bundle

[Dooruk]

No description provided.

[Dooruk]

@mathomp4 we could really use your help building bufr-query as part of jedi_bundle.

My two attempts are as below:

1. With the following, I had to run make -j64 to generate bufr-query relevant executables, after already running jedi_bundle make build.yaml:
   /discover/nobackup/projects/gmao/soca/swell/JediBundles/ioda_02182026

I suspect we are missing the CMAKE_INSTALL_PREFIX and/or Python3_EXECUTABLE flags shown in the installation instructions.. Would that make sense? Or is RECURSIVE necessary?

2. For the following build, I ran configure and make and certain executables such as show_queries.x is missing. So If I run make -j64 command the executables would be generated:
   /discover/nobackup/projects/gmao/soca/swell/JediBundles/ioda_02232026

/discover/nobackup/projects/gmao/soca/swell/JediBundles/ioda_02232026/CMakelists.txt looks like the following:

# Preamble
# --------
cmake_minimum_required( VERSION 3.12 )
project( jedi VERSION 0.0.0 LANGUAGES C CXX Fortran )
find_package( ecbuild 3.3.2 REQUIRED)
include( ecbuild_bundle )

# Packages
# --------
ecbuild_bundle_initialize()

if(DEFINED ENV{jedi_cmake_ROOT})
  include( $ENV{jedi_cmake_ROOT}/share/jedicmake/Functions/git_functions.cmake )
else()
  ecbuild_bundle( PROJECT jedicmake  GIT "https://github.com/JCSDA-internal/jedi-cmake"      BRANCH develop UPDATE RECURSIVE)
  include( jedicmake/cmake/Functions/git_functions.cmake )
endif()

ecbuild_bundle( PROJECT oops       GIT "https://github.com/JCSDA-internal/oops"            BRANCH develop UPDATE          )
ecbuild_bundle( PROJECT bufr-query GIT "https://github.com/NOAA-EMC/bufr-query"            TAG    v0.0.5                  )
ecbuild_bundle( PROJECT ioda-data  GIT "https://github.com/JCSDA-internal/ioda-data"       BRANCH develop UPDATE          )
ecbuild_bundle( PROJECT ioda       GIT "https://github.com/JCSDA-internal/ioda"            BRANCH develop UPDATE          )
ecbuild_bundle( PROJECT iodaconv   GIT "https://github.com/JCSDA-internal/ioda-converters" BRANCH develop UPDATE          )

ecbuild_bundle_finalize()

ecbuild.log shows for the installation:

[mathomp4]

@mathomp4 we could really use your help building bufr-query as part of jedi_bundle.

@Dooruk Can you teach me how to build what you currently have? It's been a while since I've done jedi_bundle

[Dooruk]

@Dooruk Can you teach me how to build what you currently have? It's been a while since I've done jedi_bundle

Sure, could you follow the instructions here, with two changes:

1. load sles15_skylab9_test (this branch) instead of sles15_skylab9 version
2. Modify your build.yaml to look like the following (with path locations different, obviosuly 😄 ):

clone_options:
  bundles:
  - iodaconv
  - ioda
  crtm_tag_or_branch: v2.4.1-jedi.2
  download_lfs_data: true
  extra_repos:
  - bufr-query
  github_orgs:
  - JCSDA-internal
  - JCSDA
  - GEOS-ESM
  - NOAA-EMC
  path_to_source: /gpfsm/dnb07/projects/p138/swell/JediBundles/ioda_02232026
  user_branch: ''
configure_options:
  cmake_build_type: release
  custom_configure_options: '-diag-disable=10441'
  external_modules: false
  modules: intel
  path_to_build: /gpfsm/dnb07/projects/p138/swell/JediBundles/ioda_02232026/build-intel-release
  platform: nccs_discover_sles15
make_options:
  cores_to_use_for_make: 64

[Dooruk]

'-diag-disable=10441' is to supress some ifx warnings but I don't think that worked.

[mathomp4]

Huh. I'm getting:

INFO JediBundle: Gathering repository information...
INFO JediBundle: Found commit at https://github.com/JCSDA/jedi-cmake
INFO JediBundle: Found commit at https://github.com/JCSDA/oops
INFO JediBundle: Found commit at https://github.com/NOAA-EMC/bufr-query
INFO JediBundle: Found commit at https://github.com/JCSDA-internal/ioda-data
ABORT JediBundle: No matching branch for 'ioda' was found in any organizations. ABORTING...

And then if I try again I get:

INFO JediBundle: Gathering repository information...
ABORT JediBundle: No matching branch for 'jedicmake' was found in any organizations. ABORTING...

Maybe my environment is...not happy?

[Dooruk]

Huh. I'm getting:

INFO JediBundle: Gathering repository information...
INFO JediBundle: Found commit at https://github.com/JCSDA/jedi-cmake
INFO JediBundle: Found commit at https://github.com/JCSDA/oops
INFO JediBundle: Found commit at https://github.com/NOAA-EMC/bufr-query
INFO JediBundle: Found commit at https://github.com/JCSDA-internal/ioda-data
ABORT JediBundle: No matching branch for 'ioda' was found in any organizations. ABORTING...

And then if I try again I get:

INFO JediBundle: Gathering repository information...
ABORT JediBundle: No matching branch for 'jedicmake' was found in any organizations. ABORTING...

Maybe my environment is...not happy?

Hmm, looks like only public repos are being cloned. Are you sure you have JCSDA-internal access and git-credentials are set properly? github-orgs should have JCSDA-internal first...

[mathomp4]

Hmm, looks like only public repos are being cloned. Are you sure you have JCSDA-internal access and git-credentials are set properly? github-orgs should have JCSDA-internal first...

I can do git clone https://github.com/JCSDA-internal/ioda-data just fine, so I should be able to see JCSDA-internal

[Dooruk]

After deliberating with Matt, I realized make stage only runs for bundles, not for extra repos so I had to handle bufr-query, similar to how Michael had it originally.

I'm very close to finish line. The only tests that are failing for me in ioda are:

The following tests FAILED:
        495 - test_ioda_bufr_python_encoder (Failed)
        496 - test_ioda_bufr_python_parallel (Failed)

due to:

Exception: oops::TimeIodaIO<IODA> terminating...
Exception: level: 0
ModuleNotFoundError: No module named 'bufr'

But the module is loaded: 37) bufr/12.1.0

@cohen-seth (or others) do you have any idea/suggestion why this might be?

This is one of the failing tests:

❯ cat iodatest_bufr_python_encoder.py

import bufr
from pyioda.ioda.Engines.Bufr import Encoder

def create_obs_group(input_path):
    YAML_PATH = "./testinput/iodatest_bufr_mhs_mapping.yaml"
    container = bufr.Parser(input_path, YAML_PATH).parse()

    encoder = Encoder(YAML_PATH)
    data = encoder.encode(container)

    return data[('metop-b', )]

[Dooruk]

tagging @jemccurr to see if he had this same issue in:

#113 (comment)

Might be a silly mistake on my part. I'm wondering though if ioda repo is set for utilizing an updated version of bufr-query(0.05 instead of 0.0.4)?

[jemccurr]

tagging @jemccurr to see if he had this same issue in:

#113 (comment)

Might be a silly mistake on my part. I'm wondering though if ioda repo is set for utilizing an updated version of bufr-query(0.05 instead of 0.0.4)?

I never tried building bufr-query as part of jedi so haven't had the same issue, but I do know that bufr-query has a directory inside site-packages named bufr which contains the python binding libraries. The .so is named bufr_python but it's just brought in with 'import bufr' - so that's independent of the actual NCEPlibs bufr module. I wonder if the bufr-query install might be conflicting with another site-package named 'bufr'?

[Dooruk]

I never tried building bufr-query as part of jedi so haven't had the same issue, but I do know that bufr-query has a directory inside site-packages named bufr which contains the python binding libraries. The .so is named bufr_python but it's just brought in with 'import bufr' - so that's independent of the actual NCEPlibs bufr module. I wonder if the bufr-query install might be conflicting with another site-package named 'bufr'?

Hmm, you are right, I see bufr under site-packages here with my build:

/discover/nobackup/projects/gmao/soca/swell/JediBundles/ioda_02242026/build-intel-release/lib

I just couldn't figure out proper $PYTHONPATH command for the encoders to work.

Perhaps following ctest is only meant to work with a module type bufr. For it to pass we need to edit the CMakelists.txt, especially $PYTHONPATH section:

ecbuild_add_test( TARGET  test_ioda_bufr_python_encoder
        TYPE    SCRIPT
        COMMAND bash
        ARGS    ${CMAKE_BINARY_DIR}/bin/ioda_compare.sh
        netcdf
        "${CMAKE_BINARY_DIR}/bin/time_IodaIO.x testinput/iodatest_bufr_python_encoder.yaml"
        iodatest_bufr_mhs.nc
        ENVIRONMENT PYTHONPATH=${CMAKE_BINARY_DIR}/lib/python${Python3_VERSION_MAJOR}.${Python3_VERSION_MINOR}:$ENV{PYTHONPATH}
        TEST_DEPENDS test_ioda_bufr_mhs )

[mranst]

I tried modifying the pythonpath of the test in the ioda source before configuring and building, I changed it to this:

ecbuild_add_test( TARGET  test_ioda_bufr_python_encoder
        TYPE    SCRIPT
        COMMAND bash
        ARGS    ${CMAKE_BINARY_DIR}/bin/ioda_compare.sh
        netcdf
        "${CMAKE_BINARY_DIR}/bin/time_IodaIO.x testinput/iodatest_bufr_python_encoder.yaml"
        iodatest_bufr_mhs.nc
        ENVIRONMENT PYTHONPATH=${CMAKE_BINARY_DIR}/lib/python${Python3_VERSION_MAJOR}${Python3_VERSION_MINOR}/site-packages:${CMAKE_BINARY_DIR}/lib/python${Python3_VERSION_MAJOR}${Python3_VERSION_MINOR}:$ENV{PYTHONPATH}
        TEST_DEPENDS test_ioda_bufr_mhs )

But now I am getting undefined symbol and segmentation fault type errors...

[Dooruk]

Obsolete because of #114