fix: add per-test-file timeout to prevent CI from hanging 12h

[mkunredd]

Summary

• Add a 30-minute timeout wrapper around each torchrun invocation in run_unit_tests.sh
• Exit code 124 (timeout) is caught and reported as a test failure
• TEST_TIMEOUT env var allows overriding the limit without changing the script

Root Cause

test_model_configs.py (684 tests × 8 torchrun processes) was hanging for the entire 12h CI job timeout after the MORI commit (26912eb), which added early MORI shmem bootstrap and a new mori_ep process group registration in pretrain_gpt.py and fused_a2a.py. This appears to cause a deadlock in distributed process group initialization when torchrun --nproc_per_node=8 spawns workers for this test file.

Without a per-file timeout, one hung torchrun blocks all subsequent test files and consumes the entire 12h CI budget:

• Run 29034209991: last log entry at 22:13:36, hit 12h timeout
• test_model_configs.py started at T+5h44m and hung for ~6h16m

What changes

# Before: torchrun can hang forever per test file
torchrun --standalone --nproc_per_node=$NUM_GPUS -m pytest ... $file

# After: killed after 30 min and reported as failure
timeout "$TEST_TIMEOUT" \
    torchrun --standalone --nproc_per_node=$NUM_GPUS -m pytest ... $file
rc=$?
if [[ $rc -eq 124 ]]; then
    echo "TIMEOUT: $file exceeded ${TEST_TIMEOUT}s — marking as failed."
fi

Note

The underlying deadlock in test_model_configs.py needs a separate investigation. This PR ensures CI fails fast and continues running remaining test files rather than hanging until the job-level timeout kills everything.

[matthiasdiener]

LGTM, thanks!