LLM benchmark rankings and evaluation setup

Three lines of research show that LLM benchmark rankings reflect evaluation methodology as much as model capability. Rohan paul found that varying only evaluation parameters, such as prompt format, option order, and scoring method, across 3,679 questions and 12 fixed models caused Gemma 4-31B's score to range from 31% to 89%, with four of twelve models reaching rank 1 under at least one valid configuration. The scoring method was the single biggest driver, with 95.7% of the average gap between neighboring models coming from questions whose answers change when the evaluation setup changes. Cameron Wolfe's overview of benchmark methods notes a parallel effect from data quality: removing incorrect samples from MMLU caused Llama-3.1-405B to rise from 16th to 1st in Virology and Qwen-2-72B-Instruct to drop from 1st to 8th in College Chemistry. As a partial remedy, the overview describes Fluid Benchmarking, which dynamically selects the most informative evaluation examples for a particular model. A study of 160 commonsense problems found that reasoning models and humans tend to fail on the same questions, and that chain-of-thought length as a difficulty signal becomes more reliable when averaged across multiple runs; for GPT-OSS-20B, the human-model difficulty correlation rose from 0.41 to 0.55 when averaged across multiple reasoning paths.