[GCC13] Enable frame-pointer; sync latest changes from GCC14 branch #9901

smuzaffar · 2025-05-30T14:48:17Z

Enable frame-pointers by default for GCC 13. As this change rebuilds GCC package which results in rebuilding full software stack so I propose to get the latest changes from GCC14 branch back to GCC13 too. This will reduce the changes between gcc14 and gcc13 branch.

smuzaffar · 2025-05-30T14:48:35Z

test parameters:

full_cmssw = true

cmsbuild · 2025-05-30T14:48:45Z

A new Pull Request was created by @smuzaffar for branch IB/CMSSW_15_1_X/g13.

@cmsbuild, @iarspider, @smuzaffar can you please review it and eventually sign? Thanks.
@antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy you are the release manager for this.
cms-bot commands are listed here

cmsbuild · 2025-05-30T14:48:47Z

cms-bot internal usage

smuzaffar · 2025-05-30T14:48:51Z

please test

smuzaffar · 2025-05-30T14:58:54Z

please test for CMSSW_15_1_X/el8_amd64_gcc13

cmsbuild · 2025-05-30T23:39:33Z

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-556a5f/46460/summary.html
COMMIT: 09e8c29
CMSSW: CMSSW_15_1_X_2025-05-29-2300/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/9901/46460/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation error when building:

+ i=/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc13/./etc/profile.d/init.sh
+ '[' -f /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc13/./etc/profile.d/init.sh ']'
+ CFLAGS='-Wno-error=return-mismatch -Wno-error=implicit-int -Wno-error=implicit-function-declaration'
+ ./configure --prefix=/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/BUILDROOT/8b841c314bd0137e1c514a85064415a1/opt/cmssw/el8_amd64_gcc13/external/glimpse/4.18.7-6-8b841c314bd0137e1c514a85064415a1
checking for gcc... gcc
checking for C compiler default output... configure: error: C compiler cannot create executables
See `config.log' for more details.
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.Ci8wgV (%build)

RPM build errors:
Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.Ci8wgV (%build)

cmsbuild · 2025-05-31T07:19:45Z

Pull request #9901 was updated.

smuzaffar · 2025-05-31T07:20:05Z

please test for CMSSW_15_1_X/el8_amd64_gcc13

smuzaffar · 2025-05-31T07:51:59Z

please test for CMSSW_15_1_X/el9_amd64_gcc13

cmsbuild · 2025-05-31T17:22:05Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-556a5f/46468/summary.html
COMMIT: 8263f39
CMSSW: CMSSW_15_1_X_2025-05-30-2300/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/9901/46468/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found 1 errors in the following unit tests:

---> test TestDQMGUIUpload had ERRORS

Comparison Summary

Summary:

You potentially removed 43 lines from the logs
Reco comparison results: 73862 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 4041063
DQMHistoTests: Total failures: 451441
DQMHistoTests: Total nulls: 394
DQMHistoTests: Total successes: 3589208
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 10.774000000000001 KiB( 49 files compared)
DQMHistoSizes: changed ( 10224.0 ): 0.415 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 13034.0 ): 3.887 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 145.604 ): 0.051 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 17034.0 ): -1.113 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 250202.181 ): 0.064 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 25202.0 ): 0.186 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 7.3 ): 2.490 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 8.0 ): 4.794 KiB SiStrip/MechanicalView
Checked 215 log files, 184 edm output root files, 50 DQM output files
TriggerResults: found differences in 13 / 48 workflows

smuzaffar · 2025-05-31T19:58:55Z

please test for CMSSW_15_1_X/el8_aarch64_gcc13

cmsbuild · 2025-06-01T00:56:50Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-556a5f/46469/summary.html
COMMIT: 8263f39
CMSSW: CMSSW_15_1_X_2025-05-30-2300/el9_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/9901/46469/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-556a5f/46469/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-556a5f/46469/git-merge-result

Unit Tests

I found 1 errors in the following unit tests:

---> test TestDQMGUIUpload had ERRORS

Comparison Summary

Summary:

You potentially added 1082 lines to the logs
Reco comparison results: 82845 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 4041063
DQMHistoTests: Total failures: 564110
DQMHistoTests: Total nulls: 438
DQMHistoTests: Total successes: 3476495
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 43.47699999999999 KiB( 49 files compared)
DQMHistoSizes: changed ( 10224.0 ): 0.603 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 13034.0 ): 0.044 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 140.045,... ): -0.004 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 141.042,... ): -0.012 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 145.5 ): -0.023 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 145.604 ): 0.055 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 17034.0 ): -0.293 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 24834.911,... ): 6.824 KiB MTD/BTL
DQMHistoSizes: changed ( 250202.181 ): 0.191 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 25202.0 ): 0.303 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 7.3 ): ...
Checked 215 log files, 184 edm output root files, 50 DQM output files

cmsbuild · 2025-06-01T14:49:39Z

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-556a5f/46479/summary.html
COMMIT: 8263f39
CMSSW: CMSSW_15_1_X_2025-05-30-2300/el8_aarch64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/9901/46479/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation error when building:

[1617/1741] /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_aarch64_gcc13/external/cuda/12.9.0-bd75e525bda7b42f6cfd95d860bdb4ff/bin/nvcc -forward-unknown-to-host-compiler -DAT_BUILD_ARM_VEC256_WITH_SLEEF -DAT_PER_OPERATOR_HEADERS -DCAFFE2_PERF_WITH_SVE=1 -DFLASHATTENTION_DISABLE_ALIBI -DFMT_HEADER_ONLY=1 -DHAVE_MALLOC_USABLE_SIZE=1 -DHAVE_MMAP=1 -DHAVE_SHM_OPEN=1 -DHAVE_SHM_UNLINK=1 -DMINIZ_DISABLE_ZIP_READER_CRC32_CHECKS -DONNXIFI_ENABLE_EXT=1 -DONNX_ML=1 -DONNX_NAMESPACE=onnx_torch -DPROTOBUF_USE_DLLS -DTORCH_CUDA_BUILD_MAIN_LIB -DTORCH_CUDA_USE_NVTX3 -DUSE_CUDA -DUSE_EXTERNAL_MZCRC -DUSE_FLASH_ATTENTION -DUSE_MEM_EFF_ATTENTION -D_FILE_OFFSET_BITS=64 -Dtorch_cuda_EXPORTS -DTORCH_ASSERT_NO_OPERATORS -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/build/aten/src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/aten/src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/build -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0 -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/third_party/onnx -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/build/third_party/onnx -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/nlohmann -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/aten/src/THC -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/aten/src/ATen/cuda -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/aten/src/ATen/../../../third_party/cutlass/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/aten/src/ATen/../../../third_party/cutlass/tools/util/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/build/caffe2/aten/src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/aten/src/ATen/.. -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/c10/cuda/../.. -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/c10/.. -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/torch/csrc/api -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/torch/csrc/api/include -isystem /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_aarch64_gcc13/external/protobuf/3.21.9-75d10f9fa19400086a7e6b0af7776af5/include -isystem /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_aarch64_gcc13/external/OpenBLAS/0.3.27-b625e6b07d07859d9c5c91d31c55e065/include -isystem /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_aarch64_gcc13/external/eigen/3bb6a48d8c171cf20b5f8e48bfb4e424fbd4f79e-6dcfc9f2d1f3614b2e6856de400c8e93/include/eigen3 -isystem /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_aarch64_gcc13/external/cuda/12.9.0-bd75e525bda7b42f6cfd95d860bdb4ff/include -isystem /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/INTERFACE -isystem /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/third_party/nlohmann/include -isystem /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_aarch64_gcc13/external/fmt/10.2.1-df9b57de4d22becc29b7004da9cf4a4f/include -isystem /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_aarch64_gcc13/external/cudnn/9.9.0.52-7cc097f13f4a0ed946482d1e751acce9/include -isystem /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/cmake/../third_party/cudnn_frontend/include -DLIBCUDACXX_ENABLE_SIMPLIFIED_COMPLEX_OPERATIONS -D_GLIBCXX_USE_CXX11_ABI=1 -Xfatbin -compress-all -DONNX_NAMESPACE=onnx_torch -gencode arch=compute_60,code=sm_60 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_75,code=sm_75 -gencode arch=compute_80,code=sm_80 -gencode arch=compute_89,code=sm_89 -gencode arch=compute_90,code=sm_90 -gencode arch=compute_90,code=compute_90 -Xcudafe --diag_suppress=cc_clobber_ignored,--diag_suppress=field_without_dll_interface,--diag_suppress=base_class_has_different_dll_interface,--diag_suppress=dll_interface_conflict_none_assumed,--diag_suppress=dll_interface_conflict_dllexport_assumed,--diag_suppress=bad_friend_decl --expt-relaxed-constexpr --expt-extended-lambda  -Wno-deprecated-gpu-targets --expt-extended-lambda -DCUB_WRAPPED_NAMESPACE=at_cuda_detail -DCUDA_HAS_FP16=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -O3 -DNDEBUG -std=c++17 -Xcompiler=-fPIC -D__NEON__ -Xcompiler -Wall -Wextra -Wdeprecated -Wno-unused-parameter -Wno-missing-field-initializers -Wno-array-bounds -Wno-unknown-pragmas -Wno-strict-overflow -Wno-strict-aliasing -Wunused-function -Wunused-variable -Wunused-but-set-variable -Wno-maybe-uninitialized -MD -MT caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/transformers/cuda/flash_attn/kernels/flash_fwd_hdim160_fp16_sm80.cu.o -MF caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/transformers/cuda/flash_attn/kernels/flash_fwd_hdim160_fp16_sm80.cu.o.d -x cu -c /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/aten/src/ATen/native/transformers/cuda/flash_attn/kernels/flash_fwd_hdim160_fp16_sm80.cu -o caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/transformers/cuda/flash_attn/kernels/flash_fwd_hdim160_fp16_sm80.cu.o
[1618/1741] /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_aarch64_gcc13/external/cuda/12.9.0-bd75e525bda7b42f6cfd95d860bdb4ff/bin/nvcc -forward-unknown-to-host-compiler -DAT_BUILD_ARM_VEC256_WITH_SLEEF -DAT_PER_OPERATOR_HEADERS -DCAFFE2_PERF_WITH_SVE=1 -DFLASHATTENTION_DISABLE_ALIBI -DFMT_HEADER_ONLY=1 -DHAVE_MALLOC_USABLE_SIZE=1 -DHAVE_MMAP=1 -DHAVE_SHM_OPEN=1 -DHAVE_SHM_UNLINK=1 -DMINIZ_DISABLE_ZIP_READER_CRC32_CHECKS -DONNXIFI_ENABLE_EXT=1 -DONNX_ML=1 -DONNX_NAMESPACE=onnx_torch -DPROTOBUF_USE_DLLS -DTORCH_CUDA_BUILD_MAIN_LIB -DTORCH_CUDA_USE_NVTX3 -DUSE_CUDA -DUSE_EXTERNAL_MZCRC -DUSE_FLASH_ATTENTION -DUSE_MEM_EFF_ATTENTION -D_FILE_OFFSET_BITS=64 -Dtorch_cuda_EXPORTS -DTORCH_ASSERT_NO_OPERATORS -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/build/aten/src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/aten/src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/build -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0 -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/third_party/onnx -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/build/third_party/onnx -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/nlohmann -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/aten/src/THC -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/aten/src/ATen/cuda -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/aten/src/ATen/../../../third_party/cutlass/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/aten/src/ATen/../../../third_party/cutlass/tools/util/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/build/caffe2/aten/src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/aten/src/ATen/.. -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/c10/cuda/../.. -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/c10/.. -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/torch/csrc/api -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/torch/csrc/api/include -isystem /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_aarch64_gcc13/external/protobuf/3.21.9-75d10f9fa19400086a7e6b0af7776af5/include -isystem /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_aarch64_gcc13/external/OpenBLAS/0.3.27-b625e6b07d07859d9c5c91d31c55e065/include -isystem /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_aarch64_gcc13/external/eigen/3bb6a48d8c171cf20b5f8e48bfb4e424fbd4f79e-6dcfc9f2d1f3614b2e6856de400c8e93/include/eigen3 -isystem /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_aarch64_gcc13/external/cuda/12.9.0-bd75e525bda7b42f6cfd95d860bdb4ff/include -isystem /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/INTERFACE -isystem /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/third_party/nlohmann/include -isystem /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_aarch64_gcc13/external/fmt/10.2.1-df9b57de4d22becc29b7004da9cf4a4f/include -isystem /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_aarch64_gcc13/external/cudnn/9.9.0.52-7cc097f13f4a0ed946482d1e751acce9/include -isystem /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/cmake/../third_party/cudnn_frontend/include -DLIBCUDACXX_ENABLE_SIMPLIFIED_COMPLEX_OPERATIONS -D_GLIBCXX_USE_CXX11_ABI=1 -Xfatbin -compress-all -DONNX_NAMESPACE=onnx_torch -gencode arch=compute_60,code=sm_60 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_75,code=sm_75 -gencode arch=compute_80,code=sm_80 -gencode arch=compute_89,code=sm_89 -gencode arch=compute_90,code=sm_90 -gencode arch=compute_90,code=compute_90 -Xcudafe --diag_suppress=cc_clobber_ignored,--diag_suppress=field_without_dll_interface,--diag_suppress=base_class_has_different_dll_interface,--diag_suppress=dll_interface_conflict_none_assumed,--diag_suppress=dll_interface_conflict_dllexport_assumed,--diag_suppress=bad_friend_decl --expt-relaxed-constexpr --expt-extended-lambda  -Wno-deprecated-gpu-targets --expt-extended-lambda -DCUB_WRAPPED_NAMESPACE=at_cuda_detail -DCUDA_HAS_FP16=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -O3 -DNDEBUG -std=c++17 -Xcompiler=-fPIC -D__NEON__ -Xcompiler -Wall -Wextra -Wdeprecated -Wno-unused-parameter -Wno-missing-field-initializers -Wno-array-bounds -Wno-unknown-pragmas -Wno-strict-overflow -Wno-strict-aliasing -Wunused-function -Wunused-variable -Wunused-but-set-variable -Wno-maybe-uninitialized -MD -MT caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/transformers/cuda/flash_attn/kernels/flash_fwd_hdim224_fp16_sm80.cu.o -MF caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/transformers/cuda/flash_attn/kernels/flash_fwd_hdim224_fp16_sm80.cu.o.d -x cu -c /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/aten/src/ATen/native/transformers/cuda/flash_attn/kernels/flash_fwd_hdim224_fp16_sm80.cu -o caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/transformers/cuda/flash_attn/kernels/flash_fwd_hdim224_fp16_sm80.cu.o
[1619/1741] /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_aarch64_gcc13/external/cuda/12.9.0-bd75e525bda7b42f6cfd95d860bdb4ff/bin/nvcc -forward-unknown-to-host-compiler -DAT_BUILD_ARM_VEC256_WITH_SLEEF -DAT_PER_OPERATOR_HEADERS -DCAFFE2_PERF_WITH_SVE=1 -DFLASHATTENTION_DISABLE_ALIBI -DFMT_HEADER_ONLY=1 -DHAVE_MALLOC_USABLE_SIZE=1 -DHAVE_MMAP=1 -DHAVE_SHM_OPEN=1 -DHAVE_SHM_UNLINK=1 -DMINIZ_DISABLE_ZIP_READER_CRC32_CHECKS -DONNXIFI_ENABLE_EXT=1 -DONNX_ML=1 -DONNX_NAMESPACE=onnx_torch -DPROTOBUF_USE_DLLS -DTORCH_CUDA_BUILD_MAIN_LIB -DTORCH_CUDA_USE_NVTX3 -DUSE_CUDA -DUSE_EXTERNAL_MZCRC -DUSE_FLASH_ATTENTION -DUSE_MEM_EFF_ATTENTION -D_FILE_OFFSET_BITS=64 -Dtorch_cuda_EXPORTS -DTORCH_ASSERT_NO_OPERATORS -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/build/aten/src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/aten/src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/build -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0 -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/third_party/onnx -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/build/third_party/onnx -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/nlohmann -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/aten/src/THC -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/aten/src/ATen/cuda -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/aten/src/ATen/../../../third_party/cutlass/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/aten/src/ATen/../../../third_party/cutlass/tools/util/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/build/caffe2/aten/src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/aten/src/ATen/.. -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/c10/cuda/../.. -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/c10/.. -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/torch/csrc/api -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/torch/csrc/api/include -isystem /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_aarch64_gcc13/external/protobuf/3.21.9-75d10f9fa19400086a7e6b0af7776af5/include -isystem /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_aarch64_gcc13/external/OpenBLAS/0.3.27-b625e6b07d07859d9c5c91d31c55e065/include -isystem /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_aarch64_gcc13/external/eigen/3bb6a48d8c171cf20b5f8e48bfb4e424fbd4f79e-6dcfc9f2d1f3614b2e6856de400c8e93/include/eigen3 -isystem /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_aarch64_gcc13/external/cuda/12.9.0-bd75e525bda7b42f6cfd95d860bdb4ff/include -isystem /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/INTERFACE -isystem /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/third_party/nlohmann/include -isystem /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_aarch64_gcc13/external/fmt/10.2.1-df9b57de4d22becc29b7004da9cf4a4f/include -isystem /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_aarch64_gcc13/external/cudnn/9.9.0.52-7cc097f13f4a0ed946482d1e751acce9/include -isystem /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/cmake/../third_party/cudnn_frontend/include -DLIBCUDACXX_ENABLE_SIMPLIFIED_COMPLEX_OPERATIONS -D_GLIBCXX_USE_CXX11_ABI=1 -Xfatbin -compress-all -DONNX_NAMESPACE=onnx_torch -gencode arch=compute_60,code=sm_60 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_75,code=sm_75 -gencode arch=compute_80,code=sm_80 -gencode arch=compute_89,code=sm_89 -gencode arch=compute_90,code=sm_90 -gencode arch=compute_90,code=compute_90 -Xcudafe --diag_suppress=cc_clobber_ignored,--diag_suppress=field_without_dll_interface,--diag_suppress=base_class_has_different_dll_interface,--diag_suppress=dll_interface_conflict_none_assumed,--diag_suppress=dll_interface_conflict_dllexport_assumed,--diag_suppress=bad_friend_decl --expt-relaxed-constexpr --expt-extended-lambda  -Wno-deprecated-gpu-targets --expt-extended-lambda -DCUB_WRAPPED_NAMESPACE=at_cuda_detail -DCUDA_HAS_FP16=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -O3 -DNDEBUG -std=c++17 -Xcompiler=-fPIC -D__NEON__ -Xcompiler -Wall -Wextra -Wdeprecated -Wno-unused-parameter -Wno-missing-field-initializers -Wno-array-bounds -Wno-unknown-pragmas -Wno-strict-overflow -Wno-strict-aliasing -Wunused-function -Wunused-variable -Wunused-but-set-variable -Wno-maybe-uninitialized -MD -MT caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/transformers/cuda/flash_attn/kernels/flash_fwd_hdim256_bf16_sm80.cu.o -MF caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/transformers/cuda/flash_attn/kernels/flash_fwd_hdim256_bf16_sm80.cu.o.d -x cu -c /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/aten/src/ATen/native/transformers/cuda/flash_attn/kernels/flash_fwd_hdim256_bf16_sm80.cu -o caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/transformers/cuda/flash_attn/kernels/flash_fwd_hdim256_bf16_sm80.cu.o
[1620/1741] /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_aarch64_gcc13/external/cuda/12.9.0-bd75e525bda7b42f6cfd95d860bdb4ff/bin/nvcc -forward-unknown-to-host-compiler -DAT_BUILD_ARM_VEC256_WITH_SLEEF -DAT_PER_OPERATOR_HEADERS -DCAFFE2_PERF_WITH_SVE=1 -DFLASHATTENTION_DISABLE_ALIBI -DFMT_HEADER_ONLY=1 -DHAVE_MALLOC_USABLE_SIZE=1 -DHAVE_MMAP=1 -DHAVE_SHM_OPEN=1 -DHAVE_SHM_UNLINK=1 -DMINIZ_DISABLE_ZIP_READER_CRC32_CHECKS -DONNXIFI_ENABLE_EXT=1 -DONNX_ML=1 -DONNX_NAMESPACE=onnx_torch -DPROTOBUF_USE_DLLS -DTORCH_CUDA_BUILD_MAIN_LIB -DTORCH_CUDA_USE_NVTX3 -DUSE_CUDA -DUSE_EXTERNAL_MZCRC -DUSE_FLASH_ATTENTION -DUSE_MEM_EFF_ATTENTION -D_FILE_OFFSET_BITS=64 -Dtorch_cuda_EXPORTS -DTORCH_ASSERT_NO_OPERATORS -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/build/aten/src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/aten/src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/build -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0 -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/third_party/onnx -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/build/third_party/onnx -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/nlohmann -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/aten/src/THC -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/aten/src/ATen/cuda -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/aten/src/ATen/../../../third_party/cutlass/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/aten/src/ATen/../../../third_party/cutlass/tools/util/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/build/caffe2/aten/src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/aten/src/ATen/.. -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/c10/cuda/../.. -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/c10/.. -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/torch/csrc/api -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/torch/csrc/api/include -isystem /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_aarch64_gcc13/external/protobuf/3.21.9-75d10f9fa19400086a7e6b0af7776af5/include -isystem /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_aarch64_gcc13/external/OpenBLAS/0.3.27-b625e6b07d07859d9c5c91d31c55e065/include -isystem /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_aarch64_gcc13/external/eigen/3bb6a48d8c171cf20b5f8e48bfb4e424fbd4f79e-6dcfc9f2d1f3614b2e6856de400c8e93/include/eigen3 -isystem /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_aarch64_gcc13/external/cuda/12.9.0-bd75e525bda7b42f6cfd95d860bdb4ff/include -isystem /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/INTERFACE -isystem /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/third_party/nlohmann/include -isystem /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_aarch64_gcc13/external/fmt/10.2.1-df9b57de4d22becc29b7004da9cf4a4f/include -isystem /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_aarch64_gcc13/external/cudnn/9.9.0.52-7cc097f13f4a0ed946482d1e751acce9/include -isystem /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/cmake/../third_party/cudnn_frontend/include -DLIBCUDACXX_ENABLE_SIMPLIFIED_COMPLEX_OPERATIONS -D_GLIBCXX_USE_CXX11_ABI=1 -Xfatbin -compress-all -DONNX_NAMESPACE=onnx_torch -gencode arch=compute_60,code=sm_60 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_75,code=sm_75 -gencode arch=compute_80,code=sm_80 -gencode arch=compute_89,code=sm_89 -gencode arch=compute_90,code=sm_90 -gencode arch=compute_90,code=compute_90 -Xcudafe --diag_suppress=cc_clobber_ignored,--diag_suppress=field_without_dll_interface,--diag_suppress=base_class_has_different_dll_interface,--diag_suppress=dll_interface_conflict_none_assumed,--diag_suppress=dll_interface_conflict_dllexport_assumed,--diag_suppress=bad_friend_decl --expt-relaxed-constexpr --expt-extended-lambda  -Wno-deprecated-gpu-targets --expt-extended-lambda -DCUB_WRAPPED_NAMESPACE=at_cuda_detail -DCUDA_HAS_FP16=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -O3 -DNDEBUG -std=c++17 -Xcompiler=-fPIC -D__NEON__ -Xcompiler -Wall -Wextra -Wdeprecated -Wno-unused-parameter -Wno-missing-field-initializers -Wno-array-bounds -Wno-unknown-pragmas -Wno-strict-overflow -Wno-strict-aliasing -Wunused-function -Wunused-variable -Wunused-but-set-variable -Wno-maybe-uninitialized -MD -MT caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/transformers/cuda/flash_attn/kernels/flash_fwd_hdim256_fp16_sm80.cu.o -MF caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/transformers/cuda/flash_attn/kernels/flash_fwd_hdim256_fp16_sm80.cu.o.d -x cu -c /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/pytorch/2.6.0-cc8952f67199087c164a99af09496143/pytorch-2.6.0/aten/src/ATen/native/transformers/cuda/flash_attn/kernels/flash_fwd_hdim256_fp16_sm80.cu -o caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/transformers/cuda/flash_attn/kernels/flash_fwd_hdim256_fp16_sm80.cu.o
ninja: build stopped: subcommand failed.
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.vko7kb (%build)

RPM build errors:
Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.vko7kb (%build)

* The action "install-external+pytorch+2.6.0-cc8952f67199087c164a99af09496143" was not completed successfully because The following dependencies could not complete:

smuzaffar · 2025-06-01T19:07:04Z

+externals

cmsbuild · 2025-06-01T19:07:23Z

This pull request is fully signed and it will be integrated in one of the next IB/CMSSW_15_1_X/g13 IBs after it passes the integration tests. This pull request will now be reviewed by the release team before it's merged. @sextonkennedy, @antoniovilela, @mandrenguyen, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

cmsbuild · 2025-06-01T20:53:35Z

-1

Failed Tests: UnitTests RelVals
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-556a5f/46483/summary.html
COMMIT: 8263f39
CMSSW: CMSSW_15_1_X_2025-05-30-2300/el8_aarch64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/9901/46483/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-556a5f/46483/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-556a5f/46483/git-merge-result

Unit Tests

I found 2 errors in the following unit tests:

---> test testPhase2PixelNtuple had ERRORS
---> test TestDQMGUIUpload had ERRORS

RelVals

29634.75

----- Begin Fatal Exception 01-Jun-2025 22:29:28 CEST-----------------------
An exception of category 'RunTimeError.' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 1 stream: 0
   [1] Running path 'L1TrackTrigger_step'
   [2] Calling method for module trackerDTC::ProducerDTC/'ProducerDTC'
   [3] TTBV::checkU
Exception Message:
Value 1 does not fit into a -3b binary.
----- End Fatal Exception -------------------------------------------------

29834.999

----- Begin Fatal Exception 01-Jun-2025 22:34:11 CEST-----------------------
An exception of category 'RunTimeError.' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 1 stream: 0
   [1] Running path 'L1TrackTrigger_step'
   [2] Calling method for module trackerDTC::ProducerDTC/'ProducerDTC'
   [3] TTBV::checkU
Exception Message:
Value 1 does not fit into a -3b binary.
----- End Fatal Exception -------------------------------------------------

24834.911

----- Begin Fatal Exception 01-Jun-2025 22:37:54 CEST-----------------------
An exception of category 'RunTimeError.' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 1 stream: 0
   [1] Running path 'L1TrackTrigger_step'
   [2] Calling method for module trackerDTC::ProducerDTC/'ProducerDTC'
   [3] TTBV::checkU
Exception Message:
Value 1 does not fit into a -3b binary.
----- End Fatal Exception -------------------------------------------------

Expand to see more relval errors ...

[GCC13] Enable frame-pointer; sync latest changes from GCC14 branch

09e8c29

cmsbuild added tests-pending externals-pending pending-signatures orp-pending labels May 30, 2025

cmsbuild added tests-started tests-pending and removed tests-pending tests-started labels May 30, 2025

cmsbuild added tests-started and removed tests-pending labels May 30, 2025

cmsbuild added tests-rejected and removed tests-started labels May 30, 2025

glimpse: only set build flags for gcc14 and above

8263f39

cmsbuild added tests-pending and removed tests-rejected labels May 31, 2025

cmsbuild added tests-started and removed tests-pending labels May 31, 2025

cmsbuild added tests-rejected tests-started and removed tests-started tests-rejected labels Jun 1, 2025

smuzaffar merged commit 0e263a3 into IB/CMSSW_15_1_X/g13 Jun 1, 2025
14 of 17 checks passed

cmsbuild added externals-approved fully-signed and removed externals-pending pending-signatures labels Jun 1, 2025

cmsbuild added tests-rejected and removed tests-started labels Jun 1, 2025

cmsbuild mentioned this pull request Jun 2, 2025

[Do not merge] Testing debug root for ASAN/GCC13 #9902

Open

[GCC13] Enable frame-pointer; sync latest changes from GCC14 branch #9901

[GCC13] Enable frame-pointer; sync latest changes from GCC14 branch #9901

Uh oh!

Conversation

smuzaffar commented May 30, 2025

Uh oh!

smuzaffar commented May 30, 2025

Uh oh!

cmsbuild commented May 30, 2025

Uh oh!

cmsbuild commented May 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

smuzaffar commented May 30, 2025

Uh oh!

smuzaffar commented May 30, 2025

Uh oh!

cmsbuild commented May 30, 2025

External Build

Uh oh!

cmsbuild commented May 31, 2025

Uh oh!

smuzaffar commented May 31, 2025

Uh oh!

smuzaffar commented May 31, 2025

Uh oh!

cmsbuild commented May 31, 2025

Unit Tests

Comparison Summary

Uh oh!

smuzaffar commented May 31, 2025

Uh oh!

cmsbuild commented Jun 1, 2025

Unit Tests

Comparison Summary

Uh oh!

cmsbuild commented Jun 1, 2025

External Build

Uh oh!

smuzaffar commented Jun 1, 2025

Uh oh!

Uh oh!

cmsbuild commented Jun 1, 2025

Uh oh!

cmsbuild commented Jun 1, 2025

Unit Tests

RelVals

Uh oh!

Uh oh!

cmsbuild commented May 30, 2025 •

edited

Loading