Run an one-off benchmark for gpt-oss

[huydhn]

On 8xH100 and 8xB200, here are the current numbers for the two models:

gpt-oss-20b

+-----------------+-------+-------+--------+
|                 | b200  | h100  | mi325x |
+-----------------+-------+-------+--------+
| gpqa (low)      | 0.568 | 0.580 | n/a    |
| gpqa (medium)   | 0.662 | 0.670 | n/a    |
| gpqa (high)     | 0.729 | 0.735 | n/a    |
+-----------------+-------+-------+--------+
| aime25 (low)    | 0.333 | 0.342 | n/a    |
| aime25 (medium) | 0.717 | 0.737 | n/a    |
| aime25 (high)   | 0.858 | 0.842 | n/a    |
+-----------------+-------+-------+--------+

gpt-oss-20b perf numbers

gpt-oss-120b

+-----------------+-------+-------+--------+
|                 | b200  | h100  | mi325x |
+-----------------+-------+-------+--------+
| gpqa (low)      | 0.653 | 0.654 | n/a    |
| gpqa (medium)   | 0.718 | 0.713 | n/a    |
| gpqa (high)     | 0.785 | 0.786 | n/a    |
+-----------------+-------+-------+--------+
| aime25 (low)    | 0.512 | 0.500 | n/a    |
| aime25 (medium) | 0.754 | 0.758 | n/a    |
| aime25 (high)   | 0.883 | 0.921 | n/a    |
+-----------------+-------+-------+--------+

gpt-oss-120b perf numbers

Overall, the accuracy scores are on the same par as https://docs.vllm.ai/projects/recipes/en/latest/OpenAI/GPT-OSS.html#accuracy-evaluation-panels for H100, but aime25 results on B200 looks like there are issues there.

Here are the links to download the raw results:

• gpqa accuracy https://github.com/pytorch/pytorch-integration-testing/actions/runs/16823595241
• aime25 accuracy https://github.com/pytorch/pytorch-integration-testing/actions/runs/16839593742

[yeqcharlotte]

what's this eval library?

[huydhn]

That's just gpt_oss.evals calling inside gpt_oss directory. I couldn't under stand why I need to do that on ROCm and don't have ssh access to the runners to check why (they are on AMD side). My ROCm devgpu doesn't have Docker to try this out locally