[CI] run_benchmark: treat runtime pytest.skip as skipped, not failed

[jhinpan]

Problem

test (linux-flydsl-*) has been red on main and on multiple open PRs (#785, #801, …). The test suites actually pass — the failure is entirely in the benchmark step.

scripts/run_benchmark.sh runs each kernel test as a plain script (python3 tests/kernels/test_*.py), not through pytest. When one of those files reaches a pytest.skip() / pytest.importorskip() at runtime, the skip raises an uncaught Skipped exception, the process exits non‑zero, and the op was counted as failed → the whole job (and CI) goes red.

Two independent triggers of the same flaw:

• mla (CDNA runners, and main itself): upstream aiter dropped/renamed hk_mla_decode_fwd → hk_mla_v32_decode_fwd. test_mla_decode.py guards the import with pytest.skip(allow_module_level=True) (added in [Fix]: mla_decode test aborts whole collection on aiter API drift #840), which correctly skips under pytest but hard‑fails when run as a script.
• layernorm (all single‑GPU runners incl. navi): test_layernorm.py's multi‑GPU test calls pytest.skip("needs >=2 GPUs").

So main and several PRs were failing for purely environmental reasons, not because of their own changes.

Fix

Add a small _fail_or_skip helper that downgrades a non‑zero benchmark exit to skipped when the log contains a Skipped: marker — mirroring the grep -q "Skipped:" handling this script already uses for hgemm/fp8. Route the plain failure branches through it: softmax, layernorm, rmsnorm, flash_attn, mla, gemm, gemm_async, hgemm, moe, moe_w4a16, RDNA WMMA.

• Genuine failures still count and still dump their log tail (unchanged behavior).
• The summary now reports a Skipped: count; the job still fails iff FAIL_COUNT > 0.
• The arch‑specific blocks (fp8_8wave_rowscale, gemm_fp4, moe_fp4, moe_a8w4) keep their existing bespoke skip detection.

Scope / non‑goals

• ✅ Unblocks the benchmark step on main, [2/5] autotune: add opt-in search and RMSNorm waves-per-EU tuning (#770) #785, [layernorm] Add backward pass for training (PR 3/3, #769) #801 (the mla + layernorm skips).
• ❌ Does not change the underlying aiter API drift — a follow‑up should either pin aiter or move test_mla_decode.py to hk_mla_v32_decode_fwd.
• ❌ Does not touch [3/5] autotune: add offline config artifacts (#770) #786's real unit‑test failures (tests/unit/test_autotune.py, empty caplog) — that's a separate code bug in that PR.

Test plan

• bash -n and sh -n pass (POSIX‑sh clean; script is #!/bin/sh).
• Unit‑tested the helper: a log with Skipped: → SKIP_COUNT, a log with AssertionError → FAIL_COUNT.
• CI: mla/layernorm should now show skip and the benchmark summary should end Failed: 0 → job green.

Made with Cursor

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.