[Do not merge] For cmssw PR testing

[smuzaffar]

No description provided.

[cmsbuild]

A new Pull Request was created by @smuzaffar for branch master.

@aandvalenzuela, @cmsbuild, @iarspider, @smuzaffar can you please review it and eventually sign? Thanks.
@antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy you are the release manager for this.
cms-bot commands are listed here

[cmsbuild]

cms-bot internal usage

[smuzaffar]

please test for CMSSW_15_0_DEVEL_X

[smuzaffar]

please test for CMSSW_15_0_DEVEL_X

[cmsbuild]

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f4a6c3/43249/summary.html
COMMIT: 8341a0f
CMSSW: CMSSW_15_0_DEVEL_X_2024-12-04-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cms-bot/2392/43249/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found 1 errors in the following unit tests:

---> test TestFWCoreReflectionClassVersionUpdate had ERRORS

Comparison Summary

There are some workflows for which there are errors in the baseline:
2024.202001 step 1
The results for the comparisons for these workflows could be incomplete
This means most likely that the IB is having errors in the relvals.The error does NOT come from this pull request

Summary:

• You potentially added 127 lines to the logs
• Reco comparison results: 5 differences found in the comparisons
• DQMHistoTests: Total files compared: 45
• DQMHistoTests: Total histograms compared: 3393039
• DQMHistoTests: Total failures: 426
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3392593
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 44 files compared)
• Checked 198 log files, 168 edm output root files, 45 DQM output files
• TriggerResults: no differences found

[smuzaffar]

please test for CMSSW_15_1_X/el8_amd64_gcc12

[smuzaffar]

please test for CMSSW_15_1_X/el8_amd64_gcc13

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f4a6c3/44370/summary.html
COMMIT: 8341a0f
CMSSW: CMSSW_15_1_X_2025-02-12-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cms-bot/2392/44370/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially removed 370 lines from the logs
• Reco comparison results: 8 differences found in the comparisons
• DQMHistoTests: Total files compared: 49
• DQMHistoTests: Total histograms compared: 3905420
• DQMHistoTests: Total failures: 66
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3905334
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
• Checked 214 log files, 184 edm output root files, 49 DQM output files
• TriggerResults: no differences found

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f4a6c3/44371/summary.html
COMMIT: 8341a0f
CMSSW: CMSSW_15_1_X_2025-02-12-2300/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cms-bot/2392/44371/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially removed 561 lines from the logs
• ROOTFileChecks: Some differences in event products or their sizes found
• Reco comparison results: 74618 differences found in the comparisons
• DQMHistoTests: Total files compared: 49
• DQMHistoTests: Total histograms compared: 3905420
• DQMHistoTests: Total failures: 466974
• DQMHistoTests: Total nulls: 300
• DQMHistoTests: Total successes: 3438126
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.827 KiB( 48 files compared)
• DQMHistoSizes: changed ( 10224.0 ): 0.352 KiB SiStrip/MechanicalView
• DQMHistoSizes: changed ( 13034.0 ): 0.713 KiB SiStrip/MechanicalView
• DQMHistoSizes: changed ( 145.604 ): 0.051 KiB JetMET/SUSYDQM
• DQMHistoSizes: changed ( 17034.0 ): -5.459 KiB SiStrip/MechanicalView
• DQMHistoSizes: changed ( 250202.181 ): -0.420 KiB SiStrip/MechanicalView
• DQMHistoSizes: changed ( 25202.0 ): 0.592 KiB SiStrip/MechanicalView
• DQMHistoSizes: changed ( 7.3 ): 6.046 KiB SiStrip/MechanicalView
• DQMHistoSizes: changed ( 8.0 ): -1.048 KiB SiStrip/MechanicalView
• Checked 214 log files, 184 edm output root files, 49 DQM output files
• TriggerResults: found differences in 15 / 47 workflows

[smuzaffar]

please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f4a6c3/44676/summary.html
COMMIT: 8341a0f
CMSSW: CMSSW_15_1_X_2025-02-25-2300/el8_amd64_gcc12
Additional Tests: PROFILING
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cms-bot/2392/44676/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 6 differences found in the comparisons
• DQMHistoTests: Total files compared: 49
• DQMHistoTests: Total histograms compared: 3920300
• DQMHistoTests: Total failures: 6
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3920274
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
• Checked 214 log files, 184 edm output root files, 49 DQM output files
• TriggerResults: no differences found

[cmsbuild]

Pull request #2392 was updated.

[smuzaffar]

please test

[cmsbuild]

-1

Failed Tests: RelVals-ROCM rocmUnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f4a6c3/47102/summary.html
COMMIT: abfd04f
CMSSW: CMSSW_15_1_X_2025-07-06-2300/el8_amd64_gcc12
Additional Tests: ROCM
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cms-bot/2392/47102/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals-ROCM

• 17034.41217034.412_TTbar_14TeV+2025PU_Patatrack_ECALOnlyAlpaka/step2_TTbar_14TeV+2025PU_Patatrack_ECALOnlyAlpaka.log

ROCm Unit Tests

I found 0 errors in the following unit tests:

Comparison Summary

Summary:

• You potentially added 5 lines to the logs
• Reco comparison results: 2 differences found in the comparisons
• DQMHistoTests: Total files compared: 50
• DQMHistoTests: Total histograms compared: 4067201
• DQMHistoTests: Total failures: 43
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 4067138
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
• Checked 215 log files, 184 edm output root files, 50 DQM output files
• TriggerResults: no differences found

[smuzaffar]

please test using addpkg Utilities/DavixAdaptor

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f4a6c3/47438/summary.html
COMMIT: abfd04f
CMSSW: CMSSW_15_1_X_2025-07-28-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cms-bot/2392/47438/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially added 3 lines to the logs
• Reco comparison results: 8 differences found in the comparisons
• DQMHistoTests: Total files compared: 50
• DQMHistoTests: Total histograms compared: 4076162
• DQMHistoTests: Total failures: 27
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 4076115
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
• Checked 215 log files, 184 edm output root files, 50 DQM output files
• TriggerResults: no differences found

[smuzaffar]

please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f4a6c3/47553/summary.html
COMMIT: abfd04f
CMSSW: CMSSW_15_1_X_2025-08-06-1100/el8_amd64_gcc12
Additional Tests: CUDA,ROCM
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cms-bot/2392/47553/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially removed 4 lines from the logs
• Reco comparison results: 12 differences found in the comparisons
• DQMHistoTests: Total files compared: 50
• DQMHistoTests: Total histograms compared: 4076122
• DQMHistoTests: Total failures: 68
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 4076034
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
• Checked 215 log files, 184 edm output root files, 50 DQM output files
• TriggerResults: no differences found

CUDA Comparison Summary

Summary:

• You potentially removed 2 lines from the logs
• Reco comparison results: 63 differences found in the comparisons
• DQMHistoTests: Total files compared: 7
• DQMHistoTests: Total histograms compared: 53444
• DQMHistoTests: Total failures: 7172
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 46272
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
• Checked 24 log files, 30 edm output root files, 7 DQM output files
• TriggerResults: no differences found

ROCM Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 71 differences found in the comparisons
• DQMHistoTests: Total files compared: 7
• DQMHistoTests: Total histograms compared: 53444
• DQMHistoTests: Total failures: 6810
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 46634
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
• Checked 24 log files, 30 edm output root files, 7 DQM output files
• TriggerResults: no differences found

[smuzaffar]

please test for CMSSW_16_0_PY312_X

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f4a6c3/49396/summary.html
COMMIT: abfd04f
CMSSW: CMSSW_16_0_PY312_X_2025-11-10-2300/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cms-bot/2392/49396/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially added 194 lines to the logs
• ROOTFileChecks: Some differences in event products or their sizes found
• Reco comparison results: 873 differences found in the comparisons
• Reco comparison had 2 failed jobs
• DQMHistoTests: Total files compared: 51
• DQMHistoTests: Total histograms compared: 3939953
• DQMHistoTests: Total failures: 3685
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3936248
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 50 files compared)
• Checked 218 log files, 188 edm output root files, 51 DQM output files
• TriggerResults: no differences found

[smuzaffar]

please test with cms-sw/cmssw#49829,cms-sw/cmssw#49859 for CMSSW_16_1_CPP23_X

[cmsbuild]

-1

Failed Tests: Build
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f4a6c3/50742/summary.html
COMMIT: abfd04f
CMSSW: CMSSW_16_1_CPP23_X_2026-01-19-1100/el8_amd64_gcc14
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cms-bot/2392/50742/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed Build

I found compilation error when building:

>> Compiling edm plugin src/RecoMET/METPUSubtraction/plugins/PFMETProducerMVA.cc
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/gcc/14.3.1-724da22786638848892aa9ded8fcd995/bin/c++ -c -DCMS_MICRO_ARCH='x86-64-v3' -DGNU_GCC -D_GNU_SOURCE -DTBB_USE_GLIBCXX_VERSION=140301 -DTBB_SUPPRESS_DEPRECATED_MESSAGES -DTBB_PREVIEW_RESUMABLE_TASKS=1 -DTBB_PREVIEW_TASK_GROUP_EXTENSIONS=1 -DKTDOUBLEPRECISION -DBOOST_SPIRIT_THREADSAFE -DPHOENIX_THREADSAFE -DBOOST_MATH_DISABLE_STD_FPCLASSIFY -DBOOST_UUID_RANDOM_PROVIDER_FORCE_POSIX -DBOOST_MPL_IGNORE_PARENTHESES_WARNING -DGNU_GCC -DG4V9 -DGNU_GCC -DG4V9 -DDD4HEP_USE_GEANT4_UNITS=1 -DCMSSW_GIT_HASH='CMSSW_16_1_CPP23_X_2026-01-19-1100' -DPROJECT_NAME='CMSSW' -DPROJECT_VERSION='CMSSW_16_1_CPP23_X_2026-01-19-1100' -Isrc -Ipoison -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/cms/cmssw/CMSSW_16_1_CPP23_X_2026-01-19-1100/src -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/cms/coral/CORAL_2_3_21-60e5363d94954915db11e3660662559d/include/LCG -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/dd4hep/v01-31-0x-e549d95c2247023fc5ee0b4813fa64ab/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/triton-inference-client/2.25.0-060fc97b3dc73e2fac520a2426bbed47/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/geant4/11.2.2-319a0976e3acdb214f396b84d16c79fe/include/Geant4 -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/geant4/11.2.2-319a0976e3acdb214f396b84d16c79fe/include -isystem/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/grpc/1.35.0-7bdfaa433c5391d1eebe8e225cf86206/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/libpng/1.6.37-d149b20a025a88b1a5c4f4a0d0cc87b1/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/pcre/8.43-dd87967f31bc038bf6a8abe2be06d878/include -isystem/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/boost/1.80.0-c41698204bbd8fe6b779e12574786ae4/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/bz2lib/1.0.6-a45a4a64bd925bd0ba8954d249304f19/include -isystem/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/clhep/2.4.7.2-9a88560cbc77d65e8c5bdb5b231d7b9c/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/curl/7.79.0-18f444b37b15728f7417a39b1eaaf992/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/expat/2.4.8-060db54c1f39d08974e07bbd10d73002/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/fastjet/3.4.1-b39d92230d0b9f717c03a6b5dd21fb4b/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/giflib/5.2.1-aff0bf47c2621a4b2d1f8118a074e245/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/gsl/2.6-cf0afcaa612018cc1c6b9101f0b38cc8/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/hepmc/2.06.10-883dcb26a06b098c5e01527ea760aabf/include -isystem/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/hls/2025.05-83833cfa3ba518c9932b44149362ee78/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/ktjet/1.06-7f8f41b44fd3076d10fda452b3723eda/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/libjpeg-turbo/2.0.2-3395aaa291eb009cb56fec2fd9beba68/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/libuuid/2.34-767b094004db7bc5bb18e2c4125785ec/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/protobuf/3.21.9-a596ab8f795bc24a6d33407d76b5f949/include -isystem/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/lcg/root/6.36.07-a4d1fa890c372e508af854bfb8620c41/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/sqlite/3.48.0-f51709baf9192743a55feccef84f90ae/include -isystem/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/tbb/v2022.3.0-b08dd18838015457cfeb2ffa9192bd18/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/cms/vdt/0.4.3-ca4b1070e42ff42a0bfc9044d6f5e087/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/vecgeom/v1.2.11-ad9661e13c1a00b610aa87817a18b852/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/vecgeom/v1.2.11-ad9661e13c1a00b610aa87817a18b852/include/VecGeom -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/xerces-c/3.1.3-6a59f41333d09eeb85768a284ddbf37a/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/xz/5.6.4-d44dfda29fc121dff35f6cb17845463b/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/zlib/1.2.13-8abc0804c1703682302a8801f051fa9d/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/abseil-cpp/20230802.3-f2016b7577eafd43ad476d16a418721d/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/alpaka/2.1.1-3caaac8d71f39d400ab2511b2403675a/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/c-ares/1.15.0-5a8c1cdd67d26ec915592da7d8941af8/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/cuda/12.9.1-b4d725e7aa767ba4d6804e4d023e7850/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/eigen/3bb6a48d8c171cf20b5f8e48bfb4e424fbd4f79e-c086936d4081784c5fb2001ad42fea6c/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/eigen/3bb6a48d8c171cf20b5f8e48bfb4e424fbd4f79e-c086936d4081784c5fb2001ad42fea6c/include/eigen3 -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/fastjet-contrib/1.101-3ad89cdb5a5363d6056ab88124af9a3c/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/fmt/10.2.1-e55d990bd8eb36851eb054a189006b12/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/hepmc3/3.2.7-c98b31f971da7980ff629442a06de22a/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/md5/1.0.0-4c04fac6c42256d53fbdabb58aa703f3/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/OpenBLAS/0.3.27-a355b8d8018f8027c9f5496a9d9beedf/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/re2/2021-06-01-ce65e95e12e2879f99e3522b2e14de6b/include -isystem/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/tensorflow/2.12.0-4f55e9b2911d6a7b66b9118d4ea33a3f/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/tinyxml2/6.2.0-776d06aaa678176c86223e685b2807b3/include -O3 -pthread -pipe -Werror=main -Werror=pointer-arith -Werror=overlength-strings -Wno-vla -Werror=overflow -std=c++23 -ftree-vectorize -Werror=array-bounds -Werror=format-contains-nul -Werror=type-limits -fvisibility-inlines-hidden -fno-math-errno --param vect-max-version-for-alias-checks=50 -Xassembler --compress-debug-sections -Wno-error=array-bounds -Warray-bounds -fuse-ld=bfd -march=x86-64-v3 -felide-constructors -fmessage-length=0 -Wall -Wno-non-template-friend -Wno-long-long -Wreturn-type -Wextra -Wpessimizing-move -Wclass-memaccess -Wno-cast-function-type -Wno-unused-but-set-parameter -Wno-ignored-qualifiers -Wno-unused-parameter -Wunused -Wparentheses -Werror=return-type -Werror=unused-value -Werror=unused-label -Werror=address -Werror=format -Werror=sign-compare -Werror=write-strings -Werror=delete-non-virtual-dtor -Werror=strict-aliasing -Werror=narrowing -Werror=unused-but-set-variable -Werror=reorder -Werror=unused-variable -Werror=conversion-null -Werror=return-local-addr -Wnon-virtual-dtor -Werror=switch -fdiagnostics-show-option -Wno-unused-local-typedefs -Wno-attributes -Wno-psabi -DEIGEN_DONT_PARALLELIZE -DEIGEN_MAX_ALIGN_BYTES=64 -DALPAKA_DEFAULT_HOST_MEMORY_ALIGNMENT=128 -DALPAKA_DISABLE_VENDOR_RNG -DALPAKA_HAS_STD_ATOMIC_REF -Wno-error=unused-variable -ftls-model=global-dynamic -pthread -DBOOST_DISABLE_ASSERTS -flto=auto -fipa-icf -flto-odr-type-merging -fno-fat-lto-objects -Wodr -fPIC -MMD -MF tmp/el8_amd64_gcc14/src/RecoMET/METPUSubtraction/plugins/RecoMETMETPUSubtraction_plugins/PFMETProducerMVA.cc.d src/RecoMET/METPUSubtraction/plugins/PFMETProducerMVA.cc -o tmp/el8_amd64_gcc14/src/RecoMET/METPUSubtraction/plugins/RecoMETMETPUSubtraction_plugins/PFMETProducerMVA.cc.o
>> Building edm plugin tmp/el8_amd64_gcc14/src/RecoMET/METPUSubtraction/plugins/RecoMETMETPUSubtraction_plugins/libRecoMETMETPUSubtraction_plugins.so
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/gcc/14.3.1-724da22786638848892aa9ded8fcd995/bin/c++ -O3 -pthread -pipe -Werror=main -Werror=pointer-arith -Werror=overlength-strings -Wno-vla -Werror=overflow -std=c++23 -ftree-vectorize -Werror=array-bounds -Werror=format-contains-nul -Werror=type-limits -fvisibility-inlines-hidden -fno-math-errno --param vect-max-version-for-alias-checks=50 -Xassembler --compress-debug-sections -Wno-error=array-bounds -Warray-bounds -fuse-ld=bfd -march=x86-64-v3 -felide-constructors -fmessage-length=0 -Wall -Wno-non-template-friend -Wno-long-long -Wreturn-type -Wextra -Wpessimizing-move -Wclass-memaccess -Wno-cast-function-type -Wno-unused-but-set-parameter -Wno-ignored-qualifiers -Wno-unused-parameter -Wunused -Wparentheses -Werror=return-type -Werror=unused-value -Werror=unused-label -Werror=address -Werror=format -Werror=sign-compare -Werror=write-strings -Werror=delete-non-virtual-dtor -Werror=strict-aliasing -Werror=narrowing -Werror=unused-but-set-variable -Werror=reorder -Werror=unused-variable -Werror=conversion-null -Werror=return-local-addr -Wnon-virtual-dtor -Werror=switch -fdiagnostics-show-option -Wno-unused-local-typedefs -Wno-attributes -Wno-psabi -DEIGEN_DONT_PARALLELIZE -DEIGEN_MAX_ALIGN_BYTES=64 -DALPAKA_DEFAULT_HOST_MEMORY_ALIGNMENT=128 -DALPAKA_DISABLE_VENDOR_RNG -DALPAKA_HAS_STD_ATOMIC_REF -Wno-error=unused-variable -ftls-model=global-dynamic -pthread -DBOOST_DISABLE_ASSERTS -flto=auto -fipa-icf -flto-odr-type-merging -fno-fat-lto-objects -Wodr -shared -Wl,-E    -Wl,-z,defs     tmp/el8_amd64_gcc14/src/RecoMET/METPUSubtraction/plugins/RecoMETMETPUSubtraction_plugins/DeepMETProducer.cc.o tmp/el8_amd64_gcc14/src/RecoMET/METPUSubtraction/plugins/RecoMETMETPUSubtraction_plugins/DeepMETSonicProducer.cc.o tmp/el8_amd64_gcc14/src/RecoMET/METPUSubtraction/plugins/RecoMETMETPUSubtraction_plugins/JVFJetIdProducer.cc.o tmp/el8_amd64_gcc14/src/RecoMET/METPUSubtraction/plugins/RecoMETMETPUSubtraction_plugins/NoPileUpPFMEtProducer.cc.o tmp/el8_amd64_gcc14/src/RecoMET/METPUSubtraction/plugins/RecoMETMETPUSubtraction_plugins/PFMETProducerMVA.cc.o -o tmp/el8_amd64_gcc14/src/RecoMET/METPUSubtraction/plugins/RecoMETMETPUSubtraction_plugins/libRecoMETMETPUSubtraction_plugins.so -Wl,-E -Wl,--hash-style=gnu -Wl,--as-needed -Wl,-z,noexecstack -Wl,-no-as-needed -lpthread -Wl,-as-needed -L/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_16_1_CPP23_X_2026-01-19-1100/biglib/el8_amd64_gcc14 -L/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_16_1_CPP23_X_2026-01-19-1100/lib/el8_amd64_gcc14 -L/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/cms/cmssw/CMSSW_16_1_CPP23_X_2026-01-19-1100/biglib/el8_amd64_gcc14 -L/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/cms/cmssw/CMSSW_16_1_CPP23_X_2026-01-19-1100/lib/el8_amd64_gcc14 -L/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/cms/cmssw/CMSSW_16_1_CPP23_X_2026-01-19-1100/external/el8_amd64_gcc14/lib -L/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/cuda/12.9.1-b4d725e7aa767ba4d6804e4d023e7850/lib64/stubs -L/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/cuda/12.9.1-b4d725e7aa767ba4d6804e4d023e7850/lib64 -L/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_16_1_CPP23_X_2026-01-19-1100/static/el8_amd64_gcc14 -L/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/cms/cmssw/CMSSW_16_1_CPP23_X_2026-01-19-1100/static/el8_amd64_gcc14 -lRecoMETMETPUSubtraction -lRecoMETMETAlgorithms -lRecoMuonTrackingTools -lRecoVertexKalmanVertexFit -lRecoVertexLinearizationPointFinders -lRecoVertexVertexTools -lRecoVertexVertexPrimitives -lTrackingToolsTrackRefitter -lRecoJetsJetProducers -lTrackingToolsTransientTrack -lDataFormatsPatCandidates -lDataFormatsHLTReco -lJetMETCorrectionsModules -lRecoJetsJetAlgorithms -lDataFormatsBTauReco -lDataFormatsJetMatching -lDataFormatsL1TParticleFlow -lDataFormatsMETReco -lDataFormatsTauReco -lJetMETCorrectionsJetCorrector -lTrackingToolsKalmanUpdators -lDataFormatsJetReco -lRecoTrackerTransientTrackingRecHit -lTrackingToolsGsfTools -lTrackingToolsTrackFitters -lDataFormatsParticleFlowCandidate -lRecoEcalEgammaCoreTools -lRecoLocalTrackerSiStripRecHitConverter -lRecoMuonNavigation -lRecoMuonTransientTrackingRecHit -lRecoParticleFlowPFClusterTools -lTrackingToolsPatternTools -lTrackingToolsRecoGeometry -lDataFormatsParticleFlowReco -lRecoMTDDetLayers -lRecoMuonDetLayers -lRecoTrackerTkDetLayers -lTrackingToolsTrackAssociator -lTrackingToolsTransientTrackingRecHit -lDataFormatsEgammaCandidates -lDataFormatsHcalIsolatedTrack -lDataFormatsL1TCorrelator -lDataFormatsL1TMuonPhase2 -lDataFormatsMuonReco -lRecoLocalTrackerPhase2TrackerRecHits -lTrackPropagationSteppingHelixPropagator -lTrackingToolsDetLayers -lDataFormatsL1TMuon -lDataFormatsRecoCandidate -lRecoLocalTrackerClusterParameterEstimator -lTrackingToolsGeomPropagators -lDataFormatsCSCDigi -lDataFormatsEgammaReco -lDataFormatsGsfTrackReco -lDataFormatsVertexReco -lSimDataFormatsTrackingAnalysis -lTrackingToolsTrajectoryState -lDataFormatsGEMDigi -lDataFormatsTrackReco -lRecoTrackerRecord -lDataFormatsGEMRecHit -lDataFormatsTrackCandidate -lDataFormatsTrackerRecHit2D -lTrackingToolsRecords -lCondFormatsSiPhase2TrackerObjects -lDataFormatsCSCRecHit -lDataFormatsDTRecHit -lDataFormatsFTLRecHit -lDataFormatsRPCRecHit -lDataFormatsTrajectorySeed -lGeometryHcalTowerAlgo -lGeometryMTDGeometryBuilder -lRecoLocalTrackerRecords -lCalibFormatsSiStripObjects -lCalibTrackerRecords -lDataFormatsL1Trigger -lDataFormatsTrackingRecHit -lGeometryCSCGeometry -lGeometryCaloTopology -lGeometryDTGeometry -lGeometryEcalAlgo -lGeometryGEMGeometry -lGeometryMTDNumberingBuilder -lGeometryRPCGeometry -lGeometryTrackerGeometryBuilder -lCalibCalorimetryEcalLaserCorrection -lCommonToolsMVAUtils -lCondFormatsSiStripObjects -lDataFormatsL1TrackTrigger -lGeometryEcalCommonData -lGeometryHcalCommonData -lMagneticFieldRecords -lMagneticFieldVolumeBasedEngine -lRecoLocalCaloEcalRecAlgos -lRecoMuonRecords -lCondFormatsDataRecord -lDataFormatsCastorReco -lDataFormatsTrackerCommon -lDetectorDescriptionRegressionTest -lGeometryCaloGeometry -lGeometryCommonTopologies -lGeometryHGCalTBCommonData -lMagneticFieldLayers -lRecoMTDRecords -lCondCoreDBOutputService -lCondFormatsAlignment -lCondFormatsEcalObjects -lDataFormatsBeamSpot -lDataFormatsCaloTowers -lDataFormatsEcalRecHit -lDataFormatsGeometryCommonDetAlgo -lDataFormatsHcalRecHit -lDataFormatsHepMCCandidate -lDataFormatsSiStripCluster -lDetectorDescriptionParser -lGeometryHGCalCommonData -lGeometryRecords -lGeometryTrackerNumberingBuilder -lHeterogeneousCoreSonicTriton -lMagneticFieldVolumeGeometry -lTrackingToolsAnalyticalJacobians -lCommonToolsStatistics -lCondCoreCondDB -lCondFormatsAlignmentRecord -lCondFormatsESObjects -lCondFormatsJetMETObjects -lDataFormatsCandidate -lDataFormatsDTDigi -lDataFormatsEcalDigi -lDataFormatsGeometrySurface -lDataFormatsHcalDigi -lDataFormatsL1GlobalCaloTrigger -lDataFormatsSiStripCommon -lDataFormatsTrajectoryState -lDetectorDescriptionCore -lDetectorDescriptionDDCMS -lGeometryMTDCommonData -lHeterogeneousCoreCUDACore -lHeterogeneousCoreSonicCore -lMagneticFieldEngine -lPhysicsToolsTensorFlow -lCommonToolsUtils -lCondFormatsGeometryObjects -lDataFormatsCLHEP -lDataFormatsCaloRecHit -lDataFormatsEcalDetId -lDataFormatsForwardDetId -lDataFormatsGeometryVector -lDataFormatsHcalDetId -lDataFormatsL1CaloTrigger -lDataFormatsL1GlobalTrigger -lDataFormatsMuonDetId -lDataFormatsPhase2TrackerCluster -lDataFormatsSiPixelDetId -lDataFormatsSiStripDetId -lFWCoreFramework -lHeterogeneousCoreCUDAServices -lSimDataFormatsCaloHit -lSimDataFormatsTrack -lSimDataFormatsVertex -lCUDADataFormatsCommon -lCondFormatsCommon -lDataFormatsDetId -lDataFormatsFEDRawData -lDataFormatsL1GlobalMuonTrigger -lDataFormatsMath -lDataFormatsPhase2TrackerDigi -lDataFormatsScouting -lDataFormatsSiPixelCluster -lDataFormatsSiStripDigi -lFWCoreCommon -lFWCoreServiceRegistry -lSimDataFormatsGeneratorProducts -lCalibCalorimetryEcalLaserAnalyzer -lCondFormatsPhysicsToolsObjects -lDataFormatsCommon -lFWCoreParameterSet -lHeterogeneousCoreCUDAUtilities -lFWCoreAbstractServices -lFWCoreMessageLogger -lDataFormatsProvenance -lCondFormatsGBRForest -lFWCoreConcurrency -lFWCorePluginManager -lFWCoreReflection -lHeterogeneousCoreAlpakaInterface -lTrackingToolsTrajectoryParametrization -lCondFormatsSerialization -lFWCoreUtilities -lFWCoreVersion -lSimDataFormatsEncodedEventId -lUtilitiesBinningTools -lUtilitiesGeneral -lUtilitiesOpenSSL -lUtilitiesXerces -llcg_CoralCommon -llcg_RelationalAccess -llcg_CoralKernel -llcg_CoralBase -lDDAlign -lDDCond -lTMVA -lDDCore -lDDParsers -lMLP -lTreePlayer -lGraf3d -lPostscript -lMinuit -lGpad -lGraf -lPhysics -lHist -lMatrix -lGenVector -lMathMore -lTree -lNet -lGeom -lThread -lboost_filesystem -lG4digits_hits -lG4error_propagation -lG4event -lG4geometry -lG4global -lG4graphics_reps -lG4intercoms -lG4interfaces -lG4materials -lG4parmodels -lG4particles -lG4geomtext -lG4mctruth -lG4gdml -lG4physicslists -lG4processes -lG4readout -lG4run -lG4tracking -lG4track -lG4analysis -lG4ptl -lMathCore -lRIO -lSmatrix -lgrpcclient -ltritoncommonmodelconfig -lboost_iostreams -lboost_program_options -lboost_regex -lboost_serialization -lboost_system -lgrpc -lgrpc++ -lgrpc++_reflection -lCore -ltensorflow_cc -lboost_thread -lboost_date_time -lCLHEP -lcudart -lcudadevrt -lnvidia-ml -lHepMCfio -lHepMC -lpng -lpcre -ltensorflow_framework -lvecgeom -lbz2 -lcuda -lcurl -lexpat -lfastjetplugins -lfastjettools -lsiscone -lsiscone_spherical -lfastjet -lgif -lgsl -lKtEvent -ljpeg -lturbojpeg -luuid -lprotobuf -lsqlite3 -ltbb -lxerces-c -llzma -lz -labsl_bad_any_cast_impl -labsl_bad_optional_access -labsl_bad_variant_access -labsl_base -labsl_city -labsl_civil_time -labsl_cord -labsl_debugging_internal -labsl_demangle_internal -labsl_examine_stack -labsl_exponential_biased -labsl_failure_signal_handler -labsl_flags_commandlineflag_internal -labsl_flags_commandlineflag -labsl_flags_config -labsl_flags_internal -labsl_flags_marshalling -labsl_flags_parse -labsl_flags_private_handle_accessor -labsl_flags_program_name -labsl_flags_reflection -labsl_flags -labsl_flags_usage_internal -labsl_flags_usage -labsl_graphcycles_internal -labsl_hash -labsl_hashtablez_sampler -labsl_int128 -labsl_leak_check -labsl_log_severity -labsl_malloc_internal -labsl_periodic_sampler -labsl_random_distributions -labsl_random_internal_distribution_test_util -labsl_random_internal_platform -labsl_random_internal_pool_urbg -labsl_random_internal_randen_hwaes_impl -labsl_random_internal_randen_hwaes -labsl_random_internal_randen_slow -labsl_random_internal_randen -labsl_random_internal_seed_material -labsl_random_seed_gen_exception -labsl_random_seed_sequences -labsl_raw_hash_set -labsl_raw_logging_internal -labsl_scoped_set_env -labsl_spinlock_wait -labsl_stacktrace -labsl_statusor -labsl_status -labsl_strerror -labsl_str_format_internal -labsl_strings_internal -labsl_strings -labsl_symbolize -labsl_synchronization -labsl_throw_delegate -labsl_time -labsl_time_zone -lcares -lfastjetcontribfragile -lfmt -lHepMC3 -lHepMC3search -lcms-md5 -lopenblas -lssl -lcrypto -lre2 -lcrypt -ldl -lrt -lstdc++fs -ltinyxml2
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/gcc/14.3.1-724da22786638848892aa9ded8fcd995/bin/../lib/gcc/x86_64-redhat-linux-gnu/14.3.1/../../../../x86_64-redhat-linux-gnu/bin/ld.bfd: cannot find -lHeterogeneousCoreSonicTriton: No such file or directory
collect2: error: ld returned 1 exit status
gmake: *** [tmp/el8_amd64_gcc14/src/RecoMET/METPUSubtraction/plugins/RecoMETMETPUSubtraction_plugins/libRecoMETMETPUSubtraction_plugins.so] Error 1
Leaving library rule at src/RecoMET/METPUSubtraction/plugins
Entering library rule at RecoMET/METPUSubtraction
>> Compiling  src/RecoMET/METPUSubtraction/src/DeepMETHelper.cc
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/gcc/14.3.1-724da22786638848892aa9ded8fcd995/bin/c++ -c -DCMS_MICRO_ARCH='x86-64-v3' -DGNU_GCC -D_GNU_SOURCE -DTBB_USE_GLIBCXX_VERSION=140301 -DTBB_SUPPRESS_DEPRECATED_MESSAGES -DTBB_PREVIEW_RESUMABLE_TASKS=1 -DTBB_PREVIEW_TASK_GROUP_EXTENSIONS=1 -DBOOST_SPIRIT_THREADSAFE -DPHOENIX_THREADSAFE -DBOOST_MATH_DISABLE_STD_FPCLASSIFY -DBOOST_UUID_RANDOM_PROVIDER_FORCE_POSIX -DBOOST_MPL_IGNORE_PARENTHESES_WARNING -DGNU_GCC -DG4V9 -DGNU_GCC -DG4V9 -DDD4HEP_USE_GEANT4_UNITS=1 -DCMSSW_GIT_HASH='CMSSW_16_1_CPP23_X_2026-01-19-1100' -DPROJECT_NAME='CMSSW' -DPROJECT_VERSION='CMSSW_16_1_CPP23_X_2026-01-19-1100' -Isrc -Ipoison -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/cms/cmssw/CMSSW_16_1_CPP23_X_2026-01-19-1100/src -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/cms/coral/CORAL_2_3_21-60e5363d94954915db11e3660662559d/include/LCG -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/dd4hep/v01-31-0x-e549d95c2247023fc5ee0b4813fa64ab/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/geant4/11.2.2-319a0976e3acdb214f396b84d16c79fe/include/Geant4 -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/geant4/11.2.2-319a0976e3acdb214f396b84d16c79fe/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/libpng/1.6.37-d149b20a025a88b1a5c4f4a0d0cc87b1/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/pcre/8.43-dd87967f31bc038bf6a8abe2be06d878/include -isystem/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/boost/1.80.0-c41698204bbd8fe6b779e12574786ae4/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/bz2lib/1.0.6-a45a4a64bd925bd0ba8954d249304f19/include -isystem/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/clhep/2.4.7.2-9a88560cbc77d65e8c5bdb5b231d7b9c/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/curl/7.79.0-18f444b37b15728f7417a39b1eaaf992/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/expat/2.4.8-060db54c1f39d08974e07bbd10d73002/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/giflib/5.2.1-aff0bf47c2621a4b2d1f8118a074e245/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/gsl/2.6-cf0afcaa612018cc1c6b9101f0b38cc8/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/hepmc/2.06.10-883dcb26a06b098c5e01527ea760aabf/include -isystem/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/hls/2025.05-83833cfa3ba518c9932b44149362ee78/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/libjpeg-turbo/2.0.2-3395aaa291eb009cb56fec2fd9beba68/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/libuuid/2.34-767b094004db7bc5bb18e2c4125785ec/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/protobuf/3.21.9-a596ab8f795bc24a6d33407d76b5f949/include -isystem/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/lcg/root/6.36.07-a4d1fa890c372e508af854bfb8620c41/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/sqlite/3.48.0-f51709baf9192743a55feccef84f90ae/include -isystem/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/tbb/v2022.3.0-b08dd18838015457cfeb2ffa9192bd18/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/cms/vdt/0.4.3-ca4b1070e42ff42a0bfc9044d6f5e087/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/vecgeom/v1.2.11-ad9661e13c1a00b610aa87817a18b852/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/vecgeom/v1.2.11-ad9661e13c1a00b610aa87817a18b852/include/VecGeom -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/xerces-c/3.1.3-6a59f41333d09eeb85768a284ddbf37a/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/xz/5.6.4-d44dfda29fc121dff35f6cb17845463b/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/zlib/1.2.13-8abc0804c1703682302a8801f051fa9d/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/alpaka/2.1.1-3caaac8d71f39d400ab2511b2403675a/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/cuda/12.9.1-b4d725e7aa767ba4d6804e4d023e7850/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/eigen/3bb6a48d8c171cf20b5f8e48bfb4e424fbd4f79e-c086936d4081784c5fb2001ad42fea6c/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/eigen/3bb6a48d8c171cf20b5f8e48bfb4e424fbd4f79e-c086936d4081784c5fb2001ad42fea6c/include/eigen3 -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/fmt/10.2.1-e55d990bd8eb36851eb054a189006b12/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/hepmc3/3.2.7-c98b31f971da7980ff629442a06de22a/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/md5/1.0.0-4c04fac6c42256d53fbdabb58aa703f3/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/OpenBLAS/0.3.27-a355b8d8018f8027c9f5496a9d9beedf/include -isystem/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/tensorflow/2.12.0-4f55e9b2911d6a7b66b9118d4ea33a3f/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/tinyxml2/6.2.0-776d06aaa678176c86223e685b2807b3/include -O3 -pthread -pipe -Werror=main -Werror=pointer-arith -Werror=overlength-strings -Wno-vla -Werror=overflow -std=c++23 -ftree-vectorize -Werror=array-bounds -Werror=format-contains-nul -Werror=type-limits -fvisibility-inlines-hidden -fno-math-errno --param vect-max-version-for-alias-checks=50 -Xassembler --compress-debug-sections -Wno-error=array-bounds -Warray-bounds -fuse-ld=bfd -march=x86-64-v3 -felide-constructors -fmessage-length=0 -Wall -Wno-non-template-friend -Wno-long-long -Wreturn-type -Wextra -Wpessimizing-move -Wclass-memaccess -Wno-cast-function-type -Wno-unused-but-set-parameter -Wno-ignored-qualifiers -Wno-unused-parameter -Wunused -Wparentheses -Werror=return-type -Werror=unused-value -Werror=unused-label -Werror=address -Werror=format -Werror=sign-compare -Werror=write-strings -Werror=delete-non-virtual-dtor -Werror=strict-aliasing -Werror=narrowing -Werror=unused-but-set-variable -Werror=reorder -Werror=unused-variable -Werror=conversion-null -Werror=return-local-addr -Wnon-virtual-dtor -Werror=switch -fdiagnostics-show-option -Wno-unused-local-typedefs -Wno-attributes -Wno-psabi -DEIGEN_DONT_PARALLELIZE -DEIGEN_MAX_ALIGN_BYTES=64 -DALPAKA_DEFAULT_HOST_MEMORY_ALIGNMENT=128 -DALPAKA_DISABLE_VENDOR_RNG -DALPAKA_HAS_STD_ATOMIC_REF -Wno-error=unused-variable -ftls-model=global-dynamic -pthread -DBOOST_DISABLE_ASSERTS -flto=auto -fipa-icf -flto-odr-type-merging -fno-fat-lto-objects -Wodr -fPIC -MMD -MF tmp/el8_amd64_gcc14/src/RecoMET/METPUSubtraction/src/RecoMETMETPUSubtraction/DeepMETHelper.cc.d src/RecoMET/METPUSubtraction/src/DeepMETHelper.cc -o tmp/el8_amd64_gcc14/src/RecoMET/METPUSubtraction/src/RecoMETMETPUSubtraction/DeepMETHelper.cc.o


[smuzaffar]

please test with cms-sw/cmssw#49829 using full cmssw for CMSSW_16_1_CPP23_X

[smuzaffar]

please test with cms-sw/cmssw#49829,cms-sw/cmssw#49872 using full cmssw for CMSSW_16_1_CPP23_X

[cmsbuild]

-1

Failed Tests: Build
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f4a6c3/50748/summary.html
COMMIT: abfd04f
CMSSW: CMSSW_16_1_CPP23_X_2026-01-19-2300/el8_amd64_gcc14
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cms-bot/2392/50748/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed Build

I found compilation error when building:

/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/gcc/14.3.1-724da22786638848892aa9ded8fcd995/bin/c++ -c -DCMS_MICRO_ARCH='x86-64-v3' -DGNU_GCC -D_GNU_SOURCE -DTBB_USE_GLIBCXX_VERSION=140301 -DTBB_SUPPRESS_DEPRECATED_MESSAGES -DTBB_PREVIEW_RESUMABLE_TASKS=1 -DTBB_PREVIEW_TASK_GROUP_EXTENSIONS=1 -DBOOST_SPIRIT_THREADSAFE -DPHOENIX_THREADSAFE -DBOOST_MATH_DISABLE_STD_FPCLASSIFY -DBOOST_UUID_RANDOM_PROVIDER_FORCE_POSIX -DBOOST_MPL_IGNORE_PARENTHESES_WARNING -DCMSSW_GIT_HASH='CMSSW_16_1_CPP23_X_2026-01-19-2300' -DPROJECT_NAME='CMSSW' -DPROJECT_VERSION='CMSSW_16_1_CPP23_X_2026-01-19-2300' -Isrc -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/pcre/8.43-dd87967f31bc038bf6a8abe2be06d878/include -isystem/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/boost/1.80.0-c41698204bbd8fe6b779e12574786ae4/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/bz2lib/1.0.6-a45a4a64bd925bd0ba8954d249304f19/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/libuuid/2.34-767b094004db7bc5bb18e2c4125785ec/include -isystem/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/lcg/root/6.36.07-a4d1fa890c372e508af854bfb8620c41/include -isystem/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/tbb/v2022.3.0-b08dd18838015457cfeb2ffa9192bd18/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/xz/5.6.4-d44dfda29fc121dff35f6cb17845463b/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/zlib/1.2.13-8abc0804c1703682302a8801f051fa9d/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/cuda/12.9.1-b4d725e7aa767ba4d6804e4d023e7850/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/eigen/3bb6a48d8c171cf20b5f8e48bfb4e424fbd4f79e-c086936d4081784c5fb2001ad42fea6c/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/eigen/3bb6a48d8c171cf20b5f8e48bfb4e424fbd4f79e-c086936d4081784c5fb2001ad42fea6c/include/eigen3 -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/fmt/10.2.1-e55d990bd8eb36851eb054a189006b12/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/md5/1.0.0-4c04fac6c42256d53fbdabb58aa703f3/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/tinyxml2/6.2.0-776d06aaa678176c86223e685b2807b3/include -O3 -pthread -pipe -Werror=main -Werror=pointer-arith -Werror=overlength-strings -Wno-vla -Werror=overflow -std=c++23 -ftree-vectorize -Werror=array-bounds -Werror=format-contains-nul -Werror=type-limits -fvisibility-inlines-hidden -fno-math-errno --param vect-max-version-for-alias-checks=50 -Xassembler --compress-debug-sections -Wno-error=array-bounds -Warray-bounds -fuse-ld=bfd -march=x86-64-v3 -felide-constructors -fmessage-length=0 -Wall -Wno-non-template-friend -Wno-long-long -Wreturn-type -Wextra -Wpessimizing-move -Wclass-memaccess -Wno-cast-function-type -Wno-unused-but-set-parameter -Wno-ignored-qualifiers -Wno-unused-parameter -Wunused -Wparentheses -Werror=return-type -Werror=unused-value -Werror=unused-label -Werror=address -Werror=format -Werror=sign-compare -Werror=write-strings -Werror=delete-non-virtual-dtor -Werror=strict-aliasing -Werror=narrowing -Werror=unused-but-set-variable -Werror=reorder -Werror=unused-variable -Werror=conversion-null -Werror=return-local-addr -Wnon-virtual-dtor -Werror=switch -fdiagnostics-show-option -Wno-unused-local-typedefs -Wno-attributes -Wno-psabi -DEIGEN_DONT_PARALLELIZE -DEIGEN_MAX_ALIGN_BYTES=64 -Wno-error=unused-variable -DBOOST_DISABLE_ASSERTS -flto=auto -fipa-icf -flto-odr-type-merging -fno-fat-lto-objects -Wodr -fPIC -MMD -MF tmp/el8_amd64_gcc14/src/HeterogeneousCore/CUDAServices/plugins/HeterogeneousCoreCUDAServicesPlugins/NVProfilerService.cc.d src/HeterogeneousCore/CUDAServices/plugins/NVProfilerService.cc -o tmp/el8_amd64_gcc14/src/HeterogeneousCore/CUDAServices/plugins/HeterogeneousCoreCUDAServicesPlugins/NVProfilerService.cc.o
>> Building edm plugin tmp/el8_amd64_gcc14/src/HeterogeneousCore/CUDAServices/plugins/HeterogeneousCoreCUDAServicesPlugins/libHeterogeneousCoreCUDAServicesPlugins.so
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/gcc/14.3.1-724da22786638848892aa9ded8fcd995/bin/c++ -O3 -pthread -pipe -Werror=main -Werror=pointer-arith -Werror=overlength-strings -Wno-vla -Werror=overflow -std=c++23 -ftree-vectorize -Werror=array-bounds -Werror=format-contains-nul -Werror=type-limits -fvisibility-inlines-hidden -fno-math-errno --param vect-max-version-for-alias-checks=50 -Xassembler --compress-debug-sections -Wno-error=array-bounds -Warray-bounds -fuse-ld=bfd -march=x86-64-v3 -felide-constructors -fmessage-length=0 -Wall -Wno-non-template-friend -Wno-long-long -Wreturn-type -Wextra -Wpessimizing-move -Wclass-memaccess -Wno-cast-function-type -Wno-unused-but-set-parameter -Wno-ignored-qualifiers -Wno-unused-parameter -Wunused -Wparentheses -Werror=return-type -Werror=unused-value -Werror=unused-label -Werror=address -Werror=format -Werror=sign-compare -Werror=write-strings -Werror=delete-non-virtual-dtor -Werror=strict-aliasing -Werror=narrowing -Werror=unused-but-set-variable -Werror=reorder -Werror=unused-variable -Werror=conversion-null -Werror=return-local-addr -Wnon-virtual-dtor -Werror=switch -fdiagnostics-show-option -Wno-unused-local-typedefs -Wno-attributes -Wno-psabi -DEIGEN_DONT_PARALLELIZE -DEIGEN_MAX_ALIGN_BYTES=64 -Wno-error=unused-variable -DBOOST_DISABLE_ASSERTS -flto=auto -fipa-icf -flto-odr-type-merging -fno-fat-lto-objects -Wodr -shared -Wl,-E    -Wl,-z,defs     tmp/el8_amd64_gcc14/src/HeterogeneousCore/CUDAServices/plugins/HeterogeneousCoreCUDAServicesPlugins/CUDAMonitoringService.cc.o tmp/el8_amd64_gcc14/src/HeterogeneousCore/CUDAServices/plugins/HeterogeneousCoreCUDAServicesPlugins/CUDAService.cc.o tmp/el8_amd64_gcc14/src/HeterogeneousCore/CUDAServices/plugins/HeterogeneousCoreCUDAServicesPlugins/NVProfilerService.cc.o -o tmp/el8_amd64_gcc14/src/HeterogeneousCore/CUDAServices/plugins/HeterogeneousCoreCUDAServicesPlugins/libHeterogeneousCoreCUDAServicesPlugins.so -Wl,-E -Wl,--hash-style=gnu -Wl,--as-needed -Wl,-z,noexecstack -L/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_16_1_CPP23_X_2026-01-19-2300/biglib/el8_amd64_gcc14 -L/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_16_1_CPP23_X_2026-01-19-2300/lib/el8_amd64_gcc14 -L/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_16_1_CPP23_X_2026-01-19-2300/external/el8_amd64_gcc14/lib -L/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/cuda/12.9.1-b4d725e7aa767ba4d6804e4d023e7850/lib64/stubs -L/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02925/el8_amd64_gcc14/external/cuda/12.9.1-b4d725e7aa767ba4d6804e4d023e7850/lib64 -L/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_16_1_CPP23_X_2026-01-19-2300/static/el8_amd64_gcc14 -lHLTriggerTimer -lFWCoreFramework -lHeterogeneousCoreCUDAServices -lFWCoreCommon -lFWCoreServiceRegistry -lDataFormatsCommon -lFWCoreParameterSet -lHeterogeneousCoreCUDAUtilities -lFWCoreAbstractServices -lFWCoreMessageLogger -lDataFormatsProvenance -lFWCoreConcurrency -lFWCorePluginManager -lFWCoreReflection -lFWCoreUtilities -lFWCoreVersion -lTree -lNet -lThread -lboost_chrono -lMathCore -lRIO -lboost_program_options -lboost_system -lCore -lboost_thread -lboost_date_time -lcudart -lcudadevrt -lnvidia-ml -lpcre -lbz2 -lcuda -luuid -ltbb -llzma -lz -lfmt -lcms-md5 -lcrypt -ldl -lrt -lstdc++fs -ltinyxml2
Leaving library rule at src/HeterogeneousCore/CUDAServices/plugins
@@@@ Running EDM write config for HeterogeneousCoreCUDAServicesPlugins
error: edmWriteConfigs caught an exception while loading a plugin library.
The executable will return success (0) so scram will continue,
but no cfi files will be written.
An exception of category 'PluginLibraryLoadError' occurred.
Exception Message:
unable to load /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_16_1_CPP23_X_2026-01-19-2300/tmp/el8_amd64_gcc14/src/HeterogeneousCore/CUDAServices/plugins/HeterogeneousCoreCUDAServicesPlugins/libHeterogeneousCoreCUDAServicesPlugins.so because libnvidia-ml.so.1: cannot open shared object file: No such file or directory


[smuzaffar]

please test workflow 136.721

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f4a6c3/51378/summary.html
COMMIT: abfd04f
CMSSW: CMSSW_16_1_X_2026-02-17-1100/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cms-bot/2392/51378/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially removed 169 lines from the logs
• Reco comparison results: 4 differences found in the comparisons
• DQMHistoTests: Total files compared: 54
• DQMHistoTests: Total histograms compared: 4228167
• DQMHistoTests: Total failures: 3
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 4228142
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 53 files compared)
• Checked 231 log files, 201 edm output root files, 54 DQM output files
• TriggerResults: no differences found

[smuzaffar]

please test for CMSSW_16_1_PY312_X/el8_amd64_gcc13

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f4a6c3/51608/summary.html
COMMIT: abfd04f
CMSSW: CMSSW_16_1_PY312_X_2026-02-24-2300/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cms-bot/2392/51608/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially added 26 lines to the logs
• ROOTFileChecks: Some differences in event products or their sizes found
• Reco comparison results: 2964 differences found in the comparisons
• DQMHistoTests: Total files compared: 53
• DQMHistoTests: Total histograms compared: 4172678
• DQMHistoTests: Total failures: 7036
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 4165622
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 52 files compared)
• Checked 227 log files, 198 edm output root files, 53 DQM output files
• TriggerResults: no differences found