Calculate the pass rate without changing pytest.skip to pytest.xfail in tests.

[anmyachev]

Such replacement in tests is done quite often. Obviously, such approach will not work when our part will be in-tree of Triton. What can be done about it?

@pbchekin @whitneywhtsang thoughts?

Part of #2030

[whitneywhtsang]

We probably want to change the title of this issue, we usually change pytest.skip to pytest.xfail instead of the other way around.
I think it is reasonable to add the code below to cuda specific unit tests upstream. WDYT?

if is_xpu():
  pytest.xfail

[anmyachev]

I think it is reasonable to add the code below to cuda specific unit tests upstream. WDYT?

I believe that they will not want to run tests for CUDA on HIP backend, since this will increase the CI time and there is also a chance that in some situations there may be a hanging or abortions, which will break their entire CI.

[anmyachev]

In theory, we know exactly how many tests we skip for our reasons, since all such cases are in the skipfiles. Maybe we can manually take this number into account when calculating the pass rate?

[whitneywhtsang]

I think it is reasonable to add the code below to cuda specific unit tests upstream. WDYT?

I believe that they will not want to run tests for CUDA on HIP backend, since this will increase the CI time and there is also a chance that in some situations there may be a hanging or abortions, which will break their entire CI.

True that xfail would execute the test case while skip doesn't.

[whitneywhtsang]

In theory, we know exactly how many tests we skip for our reasons, since all such cases are in the skipfiles. Maybe we can manually take this number into account when calculating the pass rate?

My concern is there may be new skip tests, and we accidentally forget to handle those. How can we prevent that case?