ucx error while running HPCC application

[Shubham-Bokadiya]

1737742661.094513] [cn0244:260472:0] ib_mlx5_dv.c:430 UCX ERROR mlx5dv_devx_obj_create(QP) failed on mlx5_0, syndrome 0x2c4154: Remote I/O error
[cn0244:260472] pml_ucx.c:421 Error: ucp_ep_create(proc=47) failed: Input/output error
[cn0244:260472] pml_ucx.c:472 Error: Failed to resolve UCX endpoint for rank 19
[LOG_CAT_COMMPATTERNS] isend failed in comm_allreduce_pml at iterations 1

@keisukefukuda @yshestakov @pathscale @khamidouche @yuq

why such kind of error are comming any help ?

[yosefe]

What is the current FW version, is it possible upgrading to latest one?
Also, can try exporting UCX_IB_MLX5_DEVX_OBJECTS="" (empty string)

[yosefe]

Internal ref: https://redmine.mellanox.com/issues/3743237