Run PR gpu utests/relvals on both CUDA and ROCm GPUs

[iarspider]

Additional changes :

• PR Unit test job (https://cmssdt.cern.ch/jenkins/job/ib-run-pr-unittests) - add GPU_FLAVOR parameter, select node based on it.

• PR Relval job (https://cmssdt.cern.ch/jenkins/job/ib-run-pr-relvals/) - select node based on TEST_FLAVOR

• Monitoring job (https://cmssdt.cern.ch/jenkins/job/jenkins-monitor-queue/) needs to be changed - kill pr test jobs that are stuck due to lumi node being offline

• If ROCm job is killed due to node being offline, mark commit status!

[cmsbuild]

cms-bot internal usage

[iarspider]

please test with cms-sw/cmssw#46579

to check that cpu tests are not broken

[cmsbuild]

-1

Failed Tests: ClangBuild
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b157ff/43916/summary.html
COMMIT: 118dd7e
CMSSW: CMSSW_15_0_X_2025-01-22-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cms-bot/2418/43916/install.sh to create a dev area with all the needed externals and cmssw changes.

CMS deprecated warnings: 1 CMS deprecated warnings found, see summary page for details.

Clang Build

I found compilation warning while trying to compile with clang. Command used:

USER_CUDA_FLAGS='--expt-relaxed-constexpr' USER_CXXFLAGS='-Wno-register -fsyntax-only' /usr/bin/time -v scram build -k -j 32 COMPILER='llvm compile'

See details on the summary page.

[iarspider]

please test with cms-sw/cmssw#47163

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b157ff/43917/summary.html
COMMIT: 118dd7e
CMSSW: CMSSW_15_0_X_2025-01-22-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cms-bot/2418/43917/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially added 1 lines to the logs
• Reco comparison results: 1664 differences found in the comparisons
• DQMHistoTests: Total files compared: 49
• DQMHistoTests: Total histograms compared: 3819085
• DQMHistoTests: Total failures: 149
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3818916
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
• Checked 214 log files, 184 edm output root files, 49 DQM output files
• TriggerResults: found differences in 1 / 47 workflows

[iarspider]

test parameters:

• enable = gpu
• workflows_gpu = 141.044406,141.044408,141.044412,141.044414,141.044422,141.044424,160.03502

[iarspider]

please test

[cmsbuild]

Pull request #2418 was updated.

[iarspider]

please test

[cmsbuild]

Pull request #2418 was updated.

[cmsbuild]

Pull request #2418 was updated.

[cmsbuild]

Pull request #2418 was updated.

[cmsbuild]

Pull request #2418 was updated.

[iarspider]

please test

[cmsbuild]

-1

Failed Tests: RelVals-GPU
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b157ff/44546/summary.html
COMMIT: cb924cc
CMSSW: CMSSW_15_1_X_2025-02-20-1100/el8_amd64_gcc12
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cms-bot/2418/44546/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals-GPU

ValueError: Undefined workflows: 12834.403, 12834.422, 12834.406, 12834.423
ValueError: Undefined workflows: 160.03502, 141.044406, 141.044408, 141.044422, 141.044414, 141.044412, 141.044424

Comparison Summary

Summary:

• You potentially removed 1 lines from the logs
• Reco comparison results: 6 differences found in the comparisons
• DQMHistoTests: Total files compared: 49
• DQMHistoTests: Total histograms compared: 3920300
• DQMHistoTests: Total failures: 61
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3920219
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
• Checked 214 log files, 184 edm output root files, 49 DQM output files
• TriggerResults: no differences found

[cmsbuild]

Pull request #2418 was updated.

[cmsbuild]

Pull request #2418 was updated.

[cmsbuild]

Pull request #2418 was updated.

[iarspider]

please test

[cmsbuild]

-1

Failed Tests: RelVals-GPU
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b157ff/44561/summary.html
COMMIT: fff835b
CMSSW: CMSSW_15_1_X_2025-02-21-1100/el8_amd64_gcc12
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cms-bot/2418/44561/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals-GPU

ValueError: Undefined workflows: 12834.403, 12834.422, 12834.406, 12834.423
ValueError: Undefined workflows: 160.03502, 141.044406, 141.044408, 141.044422, 141.044414, 141.044412, 141.044424

Comparison Summary

Summary:

• You potentially removed 2 lines from the logs
• Reco comparison results: 4 differences found in the comparisons
• DQMHistoTests: Total files compared: 49
• DQMHistoTests: Total histograms compared: 3920300
• DQMHistoTests: Total failures: 48
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3920232
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
• Checked 214 log files, 184 edm output root files, 49 DQM output files
• TriggerResults: no differences found