Split torch tool for cpu, cuda and rocm

[smuzaffar]

This PR proposes to build Torch separately for cpu, cuda and rocm. For now torch-rocm is not enabled as it requires full distribution of rocm (nearly 11GB)[a]. We should be able to build most/all of [a] from sources using https://github.com/ROCm/rocm-libraries .

For now this PR proposes to split torch in cpu and cuda part as a proof of concept that we can dynamically select env depending on available resources. Currently this PR proposes the following

• py3-torch: Only cpu c++ API and python interface
• py3-torch-cuda: cuda and cpu c++ API and python interface
• py3-torch-<extensions> (e.g. cluster, scatter and sparse): cpu only python interface
• py3-torch-<extensions>-cuda (e.g. cluster, scatter and sparse): cuda and cpu python interface

As we do not have py3-torch-rocm yet ( which needs full 11GB of rocm distribution), on amdgpu hosts we fall back to cpu only interface. SCRAM will dynamically set the env e.g. on

• cpu only host:
  • LD_LIBRARY_PATH: <path>/lib
  • PYTHON3PATH:<path>/lib/python3.12/site-packages
• host with cuda:
  • LD_LIBRARY_PATH: <path>/lib/scram_cuda:<path>/lib(where /lib/scram_cuda contains libs for cuda and cpu torch)
  • PYTHON3PATH: <path>/lib/python3.12/site-packages/scram_cuda:<path>/lib/python3.12/site-packages

for rocm, for now we fall back to cpu only libs/python modules.

I also have added triton and aotriton spec so that if we have to build the torch-rocm fully from sources then we can use aotirton. For now torch-rocm downloads prebuild aotriton bundle at build time

This need cms-sw/cmssw-config#117 too

[a]

 hipblas-common-devel
  miopen-hip miopen-hip-devel
  hipfft hipfft-devel
  hipsparse hipsparse-devel
  rocprim-devel hipcub-devel
  rocthrust-devel
  hipsolver hipsolver-devel
  rocsolver  rocsolver-devel
  hipblaslt hipblaslt-devel
  hipsparselt hipsparselt-devel
  roctrace roctrac-devel
  rocblas rocblas-devel

[smuzaffar]

enable gpu

[cmsbuild]

A new Pull Request was created by @smuzaffar for branch IB/CMSSW_16_1_X/py312.

@akritkbehera, @cmsbuild, @iarspider, @raoatifshad, @smuzaffar can you please review it and eventually sign? Thanks.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.
cms-bot commands are listed here

[cmsbuild]

cms-bot internal usage

[smuzaffar]

please test with cms-sw/cmssw-config#117 using full cmssw

[cmsbuild]

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b5dec0/51700/summary.html
COMMIT: af9e68a
CMSSW: CMSSW_16_1_PY312_X_2026-03-01-2300/el8_amd64_gcc13
Additional Tests: GPU,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/10390/51700/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed External Build

I found compilation error when building:

set_property could not find TARGET torch_cuda.  Perhaps it has not yet been
created.


-- Configuring incomplete, errors occurred!
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.GwcHyT (%build)

RPM build errors:
Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.GwcHyT (%build)

* The action "build-install-external+pytorch-cluster+1.6.3-24509dcecfe3ecca5dcdbeba91f58f58" was not completed successfully because The following dependencies could not complete:


[cmsbuild]

Pull request #10390 was updated.

[smuzaffar]

please test

[cmsbuild]

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b5dec0/51750/summary.html
COMMIT: 0739911
CMSSW: CMSSW_16_1_PY312_X_2026-03-03-2300/el8_amd64_gcc13
Additional Tests: GPU,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/10390/51750/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed External Build

I found compilation error when building:

+ DUP_BIN=torchrun
+ set +x
torchrun:py3-torch
torchrun:py3-torch-cuda
ERROR: Duplicate python binaries found. Please cleanup and make sure only one binary is available.
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.MUT1X4 (%install)

RPM build errors:
Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.MUT1X4 (%install)

* The action "build-srpm-cms+cmssw-tools+6.0-87f515c1e140fba3b0dce87520911fe2" was not completed successfully because The following dependencies could not complete:


[cmsbuild]

Pull request #10390 was updated.

[smuzaffar]

please test with cms-sw/cmssw-config#117 using full cmssw

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b5dec0/51850/summary.html
COMMIT: aef3014
CMSSW: CMSSW_16_1_PY312_X_2026-03-08-2300/el8_amd64_gcc13
Additional Tests: GPU,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/10390/51850/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially added 42 lines to the logs
• ROOTFileChecks: Some differences in event products or their sizes found
• Reco comparison results: 2976 differences found in the comparisons
• DQMHistoTests: Total files compared: 53
• DQMHistoTests: Total histograms compared: 4181048
• DQMHistoTests: Total failures: 6556
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 4174472
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 52 files compared)
• Checked 227 log files, 198 edm output root files, 53 DQM output files
• TriggerResults: no differences found

[cmsbuild]

Pull request #10390 was updated.

[cmsbuild]

Pull request #10390 was updated.

[smuzaffar]

please test with cms-sw/cms-bot#2702 using full cmssw

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b5dec0/51878/summary.html
COMMIT: 72b83fd
CMSSW: CMSSW_16_1_PY312_X_2026-03-09-2300/el8_amd64_gcc13
Additional Tests: GPU,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/10390/51878/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

• @ghyls Add the an option to disable the TrivialSerialisation mechanism in MPI modules cmssw#50335
• @JanGerritSchulz Fix CopyToDevice of TrackingRecHitsSoACollection cmssw#50318
• @bsunanda Phase2-hgx365U Correct the HGCal TB2023 scenario definitions to work with the current definitions of HGCal software cmssw#50355
• @cms-sw Added support for gpu backend specific packages #10408
• @cms-sw llvm: create clang cfg files before the build so that llvm tests can use it too #10406
• @cms-sw Update build rules tag V09-09-00 which supports env selection based on available nvidia/amd gpu #10407

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b5dec0/51878/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b5dec0/51878/git-merge-result

Comparison Summary

Summary:

• You potentially added 20 lines to the logs
• Reco comparison results: 2968 differences found in the comparisons
• DQMHistoTests: Total files compared: 53
• DQMHistoTests: Total histograms compared: 4181048
• DQMHistoTests: Total failures: 6578
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 4174450
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 52 files compared)
• Checked 227 log files, 198 edm output root files, 53 DQM output files
• TriggerResults: no differences found

[smuzaffar]

+externals

good to go in Python 3.12 based IBs ( i.e. PY312 and GCC15 IBs)

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next IB/CMSSW_16_1_X/py312 IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @mandrenguyen, @sextonkennedy, @ftenchini (and backports should be raised in the release meeting by the corresponding L2)
Notice This PR was tested with additional Pull Request(s), please also merge them if necessary: cms-sw/cms-bot#2702