Every new AI model arrives with a table of scores. One model "beats" another on MMLU, another leads on SWE-bench, a third tops a leaderboard built from human votes. Those numbers drive headlines and buying decisions, but most of them come from a handful of public tests that each measure one narrow thing.
This guide explains what an AI benchmark is, how the best-known ones work, what their original research papers say they measure, and how to read a score without being misled by it.
What is an AI benchmark?
A benchmark is a fixed set of tasks with known correct answers, plus a rule for grading. Researchers run a model on every task, grade each answer the same way, and report one number, usually the percentage the model got right.
The idea is simple: if every model takes the same test under the same rules, you can compare them. A benchmark is closer to a standardized exam than to a job interview. It tells you how a model does on that exam. It does not tell you how the model will handle your own work.
Most benchmarks fall into three groups:
- Knowledge and reasoning tests. Multiple-choice or short-answer questions with one correct answer. MMLU, GPQA and Humanity's Last Exam are in this group.
- Task tests. The model must produce something that works, such as code that passes tests or a patch that fixes a real bug. HumanEval and SWE-bench are in this group.
- Preference tests. People compare two answers and pick the one they like more. Chatbot Arena (now called Arena) is the best-known example.
The benchmarks you see most often
MMLU: broad school and professional knowledge
MMLU, short for Measuring Massive Multitask Language Understanding, was published by Dan Hendrycks and colleagues in 2020. It is a multiple-choice test that covers 57 tasks, including elementary mathematics, US history, computer science and law. The authors wrote that a high score needs "extensive world knowledge and problem solving ability."
When the paper came out, most models scored near random chance. The largest GPT-3 model did almost 20 percentage points better than chance on average, and the authors said the best models still needed large gains on every one of the 57 tasks before they reached expert level.
MMLU became one of the most quoted numbers in model launches. It is also a good example of how benchmarks age, which we come back to below.
GPQA: questions that a search engine cannot answer
GPQA, a "Graduate-Level Google-Proof Q&A Benchmark," has 448 multiple-choice questions written by domain experts in biology, physics and chemistry. The goal was questions that are hard even with the internet open.
The paper gives clear human baselines. Experts who have or are pursuing PhDs in the relevant field reached 65% accuracy (74% after removing clear mistakes the experts spotted later). Highly skilled non-experts reached only 34%, even though they spent more than 30 minutes per question on average with unrestricted web access. The strongest GPT-4 based baseline in the paper reached 39%.
Those human numbers are useful context. A model score on GPQA means more when you can compare it with what a PhD expert and a careful non-expert achieved on the same questions.
Humanity's Last Exam: built because older tests got too easy
Humanity's Last Exam, or HLE, exists because popular tests stopped separating the best models. Its authors wrote in 2025 that LLMs "now achieve over 90% accuracy on popular benchmarks like MMLU," which limits what those tests can tell us about top models.
HLE has 2,500 questions across dozens of subjects, including mathematics, the humanities and the natural sciences. It is multi-modal, so some questions include images. Subject-matter experts around the world wrote the questions, and each one has a single verifiable answer that cannot be found quickly through an internet search. When the paper was released, leading models showed low accuracy on it and were poorly calibrated, meaning they were often confident when they were wrong.
HumanEval: does the code actually run?
HumanEval came out of the 2021 paper that introduced Codex, a code model. The paper says a separate production version of Codex powers GitHub Copilot. HumanEval does not grade code by how it looks. It measures "functional correctness": the model reads a function description, writes the function, and the code passes or fails a set of tests.
The paper also reports results with a metric you still see today, pass@k: a problem counts as solved if any of k tries passes the tests. Codex solved 28.8% of the problems with one try. When the researchers let it generate 100 samples per problem, it solved 70.2%. So when you see a HumanEval score, check how many attempts the model was allowed. One try and many tries are very different tests.
SWE-bench: real bugs in real projects
SWE-bench moved coding tests closer to real software work. It has 2,294 problems drawn from real GitHub issues and their pull requests across 12 popular Python repositories. The model gets a codebase and an issue description, and it must edit the code to fix the issue.
The authors note that these fixes often need changes across several functions, classes and files, plus long context and interaction with a running environment. That is much harder than writing one short function. In the original paper, the best model, Claude 2, solved only 1.96% of the issues.
When you read a SWE-bench score, check which version of the test was used and how the model was allowed to work, because the setup changes the score.
Chatbot Arena: which answer do people prefer?
Chatbot Arena works differently. There is no answer key. Users type a prompt, see answers from two models side by side, and vote for the better one. The platform turns many of these pairwise votes into a ranking with statistical methods.
At the time of the paper, the team had collected more than 240,000 votes. They reported that the crowdsourced questions were diverse and that crowd votes agreed well with votes from expert raters. The paper describes Chatbot Arena as one of the most referenced LLM leaderboards. The platform has since been renamed, first to LMArena and in January 2026 to Arena.
A preference ranking answers a different question from an exam score. It tells you which model people liked more on the prompts they chose to type. That is valuable for chat quality. It is a weaker signal for narrow skills such as math proofs or bug fixing.
Rankings like this have their own weak spots, which we cover in our opinion piece on why a leaderboard rank tells you less than you think. For the statistics behind pairwise rankings, see our technical guide to evaluation arenas and Bradley-Terry modeling.
Why benchmark scores can mislead
Saturation: when everyone gets an A
A benchmark is most useful when models score across a wide range. Once the top models all score near the ceiling, small differences mostly reflect noise and test quirks. The HLE authors built their test for this reason, citing scores above 90% on MMLU. When you see two models a point apart on a saturated test, treat them as roughly tied.
Errors in the answer key
Benchmarks are written by people, and people make mistakes. The paper Are We Done with MMLU? re-checked 5,700 MMLU questions across all 57 subjects. The authors estimate that 6.49% of MMLU questions contain errors, and they found errors in 57% of the questions they analysed in the Virology subset. They also report that the corrected version changes how models compare with the originally reported results.
So part of the gap between a 90% and a 92% score can come from bad questions, not better models.
Contamination: the model may have seen the test
Most famous benchmarks are public on the internet, and models are trained on very large web datasets. If test questions, or close copies, end up in the training data, a model can score well by memory rather than skill.
A 2024 study tested this on grade-school math. GSM8K is a set of about 8,500 grade-school math word problems that the GSM1k authors call the gold standard for elementary math reasoning. The researchers commissioned a new set, GSM1k, designed to match GSM8K in style and difficulty with problems that, per the authors, are guaranteed not to be in model training data. Some models scored up to 8% lower on the new questions, and several model families showed signs of systematic overfitting. The team also found a link between how likely a model was to reproduce GSM8K examples and how much it dropped on GSM1k, which suggests partial memorization.
The same paper adds an important balance: many models, especially frontier models, showed minimal signs of overfitting, and all models broadly showed generalization to new math problems. Contamination is a real risk, not proof that every score is fake.
Different setups, different numbers
The same model can get different scores on the same benchmark depending on how it was run. The HumanEval paper shows this directly: one attempt per problem gave 28.8%, while 100 attempts gave 70.2%. Other setup choices also matter, such as the prompt format, the number of examples shown before the question, whether the model can use tools, and how much time or compute it gets.
So before you compare two scores, check the footnotes to see whether both models were tested under the same settings. Who ran the test matters too: one open dataset of 611 matched pairs found lab-published scores 6.4% higher on average than independent runs of the same tests, as our post on lab versus independent benchmark scores explains.
How to read a benchmark score: a checklist
When a launch post or headline cites a score, ask these questions:
- What does the test measure? Knowledge, coding, real bug fixing and human preference are different skills. A lead on one does not imply a lead on another.
- Is the test saturated? If top models all score near the ceiling, a small gap means little.
- Who ran it? A score from the lab that built the model, an independent evaluator and a public leaderboard can differ. Prefer scores run under the same setup for every model.
- What were the settings? Look for the number of attempts, tools, prompt style and any special "thinking" or effort mode.
- Is there a human baseline? Tests like GPQA give expert and non-expert results, which make a model score easier to judge.
- Could the model have seen the test? Newer or private test sets are harder to memorize than old public ones.
- Does it match your use? The best test is a small set of your own real tasks. Public benchmarks help you pick a shortlist, not the final winner.
Why benchmarks still matter
None of this means benchmarks are useless. They give researchers a shared yardstick, they show when a new method really moves the frontier, and they expose weaknesses. MMLU revealed near-random scores on subjects such as morality and law. SWE-bench showed how far early models were from fixing real bugs.
The healthy way to use them is as evidence, not as a verdict. Read the paper behind the number, check the setup, and compare like with like. When a benchmark starts to saturate, researchers build a harder one. The SWE-bench and Humanity's Last Exam authors both name saturated older tests as a reason for their work.
Sources
- Measuring Massive Multitask Language Understanding (MMLU), Hendrycks et al.
- GPQA: A Graduate-Level Google-Proof Q&A Benchmark, Rein et al.
- Humanity's Last Exam
- Evaluating Large Language Models Trained on Code (HumanEval), Chen et al.
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, Jimenez et al.
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference, Chiang et al.
- LMArena is now Arena, Arena blog (January 28, 2026)
- Are We Done with MMLU?, Gema et al.
- Training Verifiers to Solve Math Word Problems (GSM8K)
- A Careful Examination of Large Language Model Performance on Grade School Arithmetic (GSM1k), Zhang et al.
