ROCm Conv is not thread safe

[tianleiwu]

Describe the issue

The cache of conv algorithm finding result is not guarded by mutex like here the cache is clear but other thread might still use the cache.

The cache only uses x shape as key. If w shape changes, cache might lead to invalid algorithm (the cached algo is for another w shape) which causes MIOpen Error: No invoker was registered for convolution forward error. Another issue might be related to this: #19566

To reproduce

See the issue reported here:
#19567 (comment)

Urgency

No response

Platform

Linux

OS Version

ubuntu 20.04

ONNX Runtime Installation

Built from Source

ONNX Runtime Version or Commit ID

1.18.1

ONNX Runtime API

Python

Architecture

X64

Execution Provider

Other / Unknown

Execution Provider Library Version

ROCm 6.0

[github-actions]

This issue has been automatically marked as stale due to inactivity and will be closed in 30 days if no further activity occurs. If further support is needed, please provide an update and/or more details.

[tianleiwu]

There is mutex at ComputeInternal, which shall make the code thread-safe.

Status Conv<T, NHWC>::ComputeInternal(OpKernelContext* context) const {
  std::lock_guard<std::mutex> lock(s_.mutex)

So this issue is false alert.