Why the Leaderboard Lies: Reading AI Benchmarks Without Getting Fooled

Overview

This is a short guide to reading other people’s benchmark scores without being misled by them. The “why now”: in June 2026 Epoch AI shipped a corrected v2 of FrontierMath — a research-maths benchmark that labs cite in their launch posts — after its own audit found and fixed errors in 42% of the problems. A benchmark the field had quoted for months turned out, by its maintainer’s own count, to be flawed in more than 40% of its questions, and you’d never see it from the headline score. That’s the mild failure. The structural ones run deeper: a contamination report across 15-plus models and six question-answering benchmarks found test questions leaking into training data from 1% up to 45%, and a held-out re-run of grade-school maths showed some model families scoring up to 13% lower on fresh questions than on the public set (Li et al., EMNLP 2024; the GSM1k study, May 2024). The published number and the number that predicts your results are rarely the same. By the end you’ll have a quick way to decide whether a leaderboard score says anything about your own work.

The content

The obvious read of a benchmark is that a higher score means a better model for you. That read fails for a specific, structural reason. A public benchmark is a fixed set of questions with public answers, and the moment a number becomes the thing everyone competes on, it stops measuring the thing it was built to measure. That’s the trap in one phrase: the published-vs-yours gap. Occasionally the benchmark is simply wrong — Epoch AI’s own June 2026 audit of FrontierMath corrected errors in 42% of its questions — but the more interesting failures are structural.

Three forces open that gap. The first is contamination — the test questions, or close paraphrases, end up in the training data, so the model is partly remembering rather than reasoning. The open-source contamination report found this running from trivial to nearly half the questions depending on the benchmark, and getting worse over time as more of the web is scraped (Li et al., 2024). The second is benchmark-as-target: when a score drives funding, headlines and procurement, labs optimise for it, and a metric under that much pressure drifts away from the capability it was meant to stand in for. The third is memorisation you can measure indirectly — when researchers rebuilt grade-school maths as a fresh, matched set (GSM1k), several model families scored materially lower on the new questions, and a model’s tendency to reproduce the original questions verbatim correlated with the size of its drop (GSM1k study, 2024).

Two honest caveats keep this from tipping into cynicism. The same GSM1k work found that frontier models showed little overfitting and still solved genuinely novel problems — a high score is evidence, just weaker evidence than it looks. And contamination doesn’t lift every score equally; the gains cluster on some benchmarks and barely move others. So the move isn’t to dismiss benchmarks. It’s to treat a leaderboard rank as a hypothesis about a model, not a verdict, and to ask what the score was actually measured on before you let it pick your tools.

Try it

Next time a vendor, a colleague or a launch post leans on a benchmark number, run it through these questions before you act on it. Paste the claim in and answer honestly.

I'm evaluating this benchmark claim before trusting it:
[paste the claim, e.g. "Model X scores 92% on <benchmark>, beating Model Y"]

Help me pressure-test it:
1. What exactly does this benchmark measure, and how close is that
   to the work I actually need done? (name my real task)
2. Is this benchmark public and old enough that its questions
   could be in training data (contamination risk)?
3. Is there a held-out, contamination-resistant, or recently-refreshed
   version of this test, and how does the model score on THAT?
4. What's the gap between the top model and the next few — is the
   lead real or noise?
5. What would a 10-example trial on MY OWN task tell me that this
   score can't?

Then give me a one-line verdict: does this score predict anything
about my use case, or not?

Where this won’t save you: if your task is genuinely niche — your jargon, your documents, your edge cases — no public benchmark, clean or not, can answer the question. At that point stop reading other people’s tests and run ten of your own real cases. That’s the cut-off line for this skill: it makes you a sharper reader of published scores, not a substitute for trying the model on your work.

Additional reading

• FrontierMath v2 changelog (Epoch AI) — in June 2026 the benchmark’s maintainer corrected or removed errors in 42% of the problems, a reminder that even a maintained leaderboard’s own numbers can be substantially off before contamination enters the picture.
• A Careful Examination of Large Language Model Performance on Grade School Arithmetic (GSM1k) — built a fresh, matched version of a maths benchmark and found accuracy drops of up to 13% for the most overfit model families, with memorisation correlating to the gap.
• An Open-Source Data Contamination Report for Large Language Models — measured contamination of 1% to 45% across six QA benchmarks and 15-plus models, rising over time.
• Benchmarking Large Language Models Under Data Contamination: A Survey from Static to Dynamic Evaluation — surveys why static public benchmarks decay and what dynamic, contamination-resistant evaluation looks like.

Editor’s note

I have a love/hate relationship with model benchmarks. On the one hand, I can’t deny that when a new frontier model is released and its performance improves by a few points across the suite of popular benchmarks, I get excited to see the measure of improvement. On the other hand, I am a firm believer that the model is only part of the story when it comes to what will actually work best for a given task, and the harness, work style, problem type, and more also factor in. At the time of publication, there is a lot of community discussion around GPT-5.6 Sol vs Claude Fable 5. Something that I’m seeing fairly often is the sentiment that “5.6 Sol is dumber, but I prefer it”. As a power-user of both models and several others, I can relate. I reach for the model that fits the task; not the one that scored highest on the benchmarks.