🐺🐦‍⬛ LLM Comparison/Test: Phi-4, Qwen2 VL 72B Instruct, Aya Expanse 32B in my updated MMLU-Pro CS benchmark

Introduction

I've updated my MMLU-Pro Computer Science LLM benchmark results with new data from recently tested models: three Phi-4 variants (Microsoft's official weights, plus Unsloth's fixed HF and GGUF versions), Qwen2 VL 72B Instruct, and Aya Expanse 32B.

While adding these new models, I streamlined the graph by removing all QwQ-32B-Preview variants except for the main model, which consistently showed superior performance. I also incorporated the results from a third evaluation run for Claude, gemini-1.5-pro-002 and Athene-V2-Chat, which shifted Athene's rank downward due to fluctuations in its scores.

Here's an additional visualization that represents each model as a 3D bar where the height shows the MMLU score (%), the depth represents the model size in billions of parameters, and for quantized models, the bar is split into a full-color front section proportional to the quantized size and a lighter-colored back section showing the memory savings compared to full precision (16-bit) models:

New Models Tested

1. Phi-4:

   • Unsloth's fixed Transformers implementation showed minimal performance differences in benchmarks, with the GGUF version achieving marginally higher accuracy. Further testing would be needed to determine whether this improvement represents a statistically significant trend or random variation.
   • Temperature settings had a notable impact on performance: At 0, responses were consistent but repetitive, while at 1, outputs became erratic and unpredictable.
   • German language performance improved substantially compared to previous versions. Though it's a small model, which occasionally results in overly literal translations, the overall quality of German outputs is good enough for most purposes.
   • Censorship can be completely circumvented through basic prompt engineering techniques.
   • Still undecided on how good Phi-4 is for general-purpose tasks in real use - but generally recommend you run a bigger, better model if you can.

2. Qwen2 VL 72B Instruct:

   • Given its relatively low scores and reliance on the older Qwen2 series rather than the superior 2.5 architecture, I look forward to the release of a Qwen2.5 VL 72B model.

3. Aya Expanse 32B:

   • While this model shows the lowest score on the graph, keep in mind that I only included models scoring above 50%. There are other, worse models that fell below this threshold and didn't make it onto the visualization at all.
   • Its main advantage is the support of 23 languages, making it a solid choice when you need those multilingual capabilities and have no better alternatives. Of course, if your target language is supported by a better model, use that instead.

About the Benchmark

The MMLU-Pro benchmark is a comprehensive evaluation of large language models across various categories, including computer science, mathematics, physics, chemistry, and more. It's designed to assess a model's ability to understand and apply knowledge across a wide range of subjects, providing a robust measure of general intelligence. While it is a multiple choice test, instead of 4 answer options like in its predecessor MMLU, there are now 10 options per question, which drastically reduces the probability of correct answers by chance. Additionally, the focus is increasingly on complex reasoning tasks rather than pure factual knowledge.

For my benchmarks, I currently limit myself to the Computer Science category with its 410 questions. This pragmatic decision is based on several factors: First, I place particular emphasis on responses from my usual work environment, since I frequently use these models in this context during my daily work. Second, with local models running on consumer hardware, there are practical constraints around computation time - a single run already takes several hours with larger models, and I generally conduct at least two runs to ensure consistency.

Unlike typical benchmarks that only report single scores, I conduct multiple test runs for each model to capture performance variability. This comprehensive approach delivers a more accurate and nuanced understanding of each model's true capabilities. By executing at least two benchmark runs per model, I establish a robust assessment of both performance levels and consistency. The results feature error bars that show standard deviation, illustrating how performance varies across different test runs.

The benchmarks for this study alone required over 103 hours of runtime. With additional categories or runs, the testing duration would have become so long with the available resources that the tested models would have been outdated by the time the study was completed. Therefore, establishing practical framework conditions and boundaries is essential to achieve meaningful results within a reasonable timeframe.

Detailed Results

Here's the complete table, including results from previous reports:

• Model: Model name (with relevant parameter and setting details)
• HF Main Model Name: Full name of the tested model as listed on Hugging Face
• HF Draft Model Name (speculative decoding): Draft model used for speculative decoding (if applicable)
• Size: Parameter count
• Format: Model format type (HF, EXL2, etc.)
• API: Service provider (TabbyAPI indicates local deployment)
• GPU: Graphics card used for this benchmark run
• GPU Mem: VRAM allocated to model and configuration
• Run: Benchmark run sequence number
• Duration: Total runtime of benchmark
• Total: Number of correct answers (determines ranking!)
• %: Percentage of correct answers
• Correct Random Guesses: When MMLU-Pro cannot definitively identify a model's answer choice, it defaults to random guessing and reports both the number of these random guesses and their accuracy (a high proportion of random guessing indicates problems with following the response format)
• Prompt tokens: Token count of input text
• tk/s: Tokens processed per second
• Completion tokens: Token count of generated response
• tk/s: Tokens generated per second

Wolfram Ravenwolf is a German AI Engineer and an internationally active consultant and renowned researcher who's particularly passionate about local language models. You can follow him on X and Bluesky, read his previous LLM tests and comparisons on HF and Reddit, check out his models on Hugging Face, tip him on Ko-fi, or book him for a consultation.