Benchmarks, Evaluation, Versions and Dates

50.0 What this chapter gives you#

1. You will be able to say exactly what a benchmark is, in five mechanical parts, and name which part is being argued about when two numbers disagree.
2. You will be able to describe the main benchmarks by name, year, size and scoring rule, and say for each one what it does not measure.
3. You will be able to explain why the same model, on the same benchmark, run by two honest teams, can legitimately produce different scores.
4. You will be able to compute pass@k by hand and explain why it is reported instead of a plain percentage.
5. You will be able to explain benchmark contamination, list five ways people try to detect it, and say honestly why detection is hard and disputed.
6. You will be able to read a model announcement and list the eight things it did not tell you.
7. You will be able to explain the difference between a training cutoff date and a release date, and why models are bad at reporting their own cutoff.
8. You will be able to explain why a product name and a model identifier are different things, and why pinning a dated identifier matters in production.
9. You will be able to build your own evaluation set of 30 to 100 examples with a scoring rule, and use it to choose and re-choose a model.
10. Throughout, you will be able to sort a claim into three bins: established fact, active research, and marketing claim.

A warning that applies to this whole chapter. The dates, item counts, paper titles and naming conventions here were checked in August 2026. The item counts and the naming conventions are stable. The leaderboard numbers are not. Any specific score in this chapter should be read as “roughly this, in mid-2026”, and any score you care about should be re-checked at the source.

50.1 What a benchmark is, mechanically#

PLAIN50.1.1 in simple words#

1. A benchmark is a test. That is all it is.
2. It has a fixed list of questions.
3. Somebody already knows the right answer to every question.
4. There is a rule for deciding if a given answer counts as right.
5. You give the questions to the model, collect its answers, apply the rule, and count.
6. The count, turned into a percentage, is the score.
7. There is nothing magic in it. It is a school test with a marking scheme.
8. It is not a measurement of intelligence, any more than a spelling test is.
9. It measures one thing: how many of these particular items, marked this particular way, this particular model got right on this particular day.
10. Everything difficult about benchmarks comes from the words “particular”.
11. Change the wording of the question and the score moves.
12. Change the marking rule and the score moves.
13. Change which items you use and the score moves.
14. None of that is cheating. It is just what happens.
15. So when two labs publish different numbers for the same model on the same test, the first assumption should not be dishonesty. It should be setup.

PLAIN50.1.2 a picture in your head#

1. Think of a driving test.
2. The test has a fixed route, a fixed set of manoeuvres, and an examiner with a clipboard listing faults.
3. Two candidates take the same test and get different results. Fine.
4. Now take the same candidate to two different test centres.
5. One centre uses a route with a busy roundabout. The other uses a quiet estate. The candidate may pass one and fail the other.
6. One examiner counts a small kerb touch as a minor fault. Another counts it as a major. Same driving, different result.
7. One centre lets you do a warm-up lap first. The other does not.
8. Now imagine the candidate has driven that exact route four hundred times, because it is published online.
9. That last one is contamination, and section 50.5 is about it.
10. And here is the biggest problem. Passing a driving test does not tell you whether somebody can drive to Manchester in fog at night.

Where this comparison breaks: a driving test has a legal standard behind it and an examiner who watched you. Most AI benchmarks have neither. Nobody watched the model. A script compared strings. Also, driving tests are not published in advance with all the answers. Almost every AI benchmark is.

PLAIN50.1.3 a worked example#

1. Here is one real item, in the style of GSM8K, a set of grade-school maths word problems released by OpenAI in 2021.

Question:
Natalia sold clips to 48 friends in April, and then she
sold half as many clips in May. How many clips did she
sell altogether in April and May?

Reference answer (the dataset marks the final number
with four hash characters):
  48 / 2 = 24
  48 + 24 = 72
  #### 72

2. The #### marker is part of the dataset format. The number after it is the answer the scorer will compare against.
3. Now here are two different things a model might produce.

Model A output:
"She sold 48 in April and 24 in May, so 72 clips."

Model B output:
"72"

4. Both are correct. A strict scorer that demands the whole reply to equal “72” marks model A wrong and model B right.
5. A scorer that pulls the last number out of the reply marks both right.
6. Nothing about the model changed. The marking rule changed.
7. That single choice can move a reported score by several points across a whole benchmark.
8. Now a nastier case. A model replies “72 clips”. A scorer that pulls the last number finds 72. Good.
9. A model replies “The answer is 48 + 24 = 72.” A scorer that pulls the last number still finds 72. Good.
10. A model replies “She sold 72, up from 48.” A scorer that pulls the last number finds 48 and marks it wrong. The model was right.
11. Real answer-extraction bugs look exactly like this.

PLAIN50.1.4 what is really happening inside#

1. Every benchmark run has exactly five moving parts. Learn these five and you can interrogate any published number.
2. Part one, the items. The fixed list of questions and known answers.
3. Part two, the prompt template. The wrapper of text that surrounds each item before it reaches the model. Instructions, examples, formatting.
4. Part three, the generation settings. How the model is asked to produce text: temperature, how many tokens it may write, whether it may think first.
5. Part four, the answer extraction and scoring function. The code that turns a blob of model text into “right” or “wrong”.
6. Part five, the aggregate. How the per-item results are combined into one number, and whether that number is one run or an average of several.
7. Here is the flow as a picture.

  items        prompt       model       extract      score
+--------+   +--------+  +--------+  +--------+  +--------+
| Q + A  |-->| wrap in|->| sample |->| pull   |->| count  |
| pairs  |   |template|  | text   |  | answer |  | right  |
+--------+   +--------+  +--------+  +--------+  +--------+
                              ^                       |
                              |                       v
                        temperature,           "71.4 per cent"
                        max tokens, k

8. Each of the five boxes is a choice made by whoever ran the evaluation.
9. Every one of those five choices changes the number that comes out.
10. Only the first box, the items, is usually published. The other four are often described in a sentence or not at all.
11. That is the single most important fact in this chapter.
12. The honest version: the sentence “this model scores 88 per cent on MMLU” is incomplete in the way “this car does 60” is incomplete. A benchmark score is not a property of a model. It is a property of a model plus a harness, measured on one day.

TECHNICAL50.1.5 the engineer’s version#

1. Formally, an evaluation is a tuple: a dataset D of items, a prompt function p that maps an item to a model input, a decoding configuration, an extraction function e that maps model output to a candidate answer, a metric m that compares candidate to reference, and an aggregation over D.
2. The score is (1/|D|) * sum over i of m(e(model(p(d_i))), a_i) for accuracy-style metrics. Everything contested lives in p, e and the decoder.
3. Two implementation families exist for multiple-choice items and they do not give the same answer.
4. Log-likelihood scoring: present the question, then score each candidate continuation by the model’s log probability, and pick the highest. The model never generates a token. Common for base models.
5. Generative scoring: present the question with lettered options, let the model emit text, then parse a letter out of it. Common for chat models.
6. On the same items and the same model these can differ by several points. Neither is wrong. They measure different things.
7. Length normalization is a further fork. Raw log probability favours short options, so harnesses often divide by token count or by character count, or normalize against the unconditional probability of the option.
8. The EleutherAI lm-evaluation-harness is the de facto open standard for this. It supports over 60 academic benchmarks with hundreds of subtask variants, and it powered the Hugging Face Open LLM Leaderboard. Its stated principle is that publicly available prompts make results reproducible and comparable between papers. This is a convention, not a standard.
9. Stanford CRFM’s HELM, first published in November 2022 as “Holistic Evaluation of Language Models”, takes the other approach: fix the prompts centrally, report many metrics per scenario rather than one, and publish every raw model output.
10. Terminology: a harness is the code that runs the loop. A scenario or task is one benchmark inside it. A run is one execution.
11. There is no ISO or IEEE specification for any of this. There is no conformance test. Everything in this section is convention.

WORDS50.1.6 remember these#

1. Benchmark — a fixed test with known answers — a dataset plus a prompting, extraction and scoring protocol.
2. Item — one question — one example in the evaluation dataset, with a reference answer.
3. Prompt template — the wrapper text around a question — the function mapping a dataset item to model input, including any few-shot exemplars.
4. Harness — the program that runs the test — the evaluation framework that implements prompting, decoding, extraction and aggregation.
5. Extraction — pulling the answer out of the reply — the parser mapping raw generation to a comparable candidate answer.
6. Metric — the marking rule — the function comparing candidate to reference, such as exact match or pass@k.
7. Aggregate — the final single number — the reduction over all items, often plain accuracy, sometimes a macro average over subjects.
8. Log-likelihood scoring — asking which option the model thinks most likely — ranking candidate continuations by model log probability without generating.

50.2 The major benchmarks, one at a time#

PLAIN50.2.1 in simple words#

1. There are perhaps twenty benchmark names you will see repeatedly.
2. They fall into a few families.
3. Knowledge quizzes: multiple-choice questions across many school and university subjects. MMLU is the famous one.
4. Maths: word problems and competition problems. GSM8K, MATH, AIME.
5. Code: write a function that passes hidden tests. HumanEval, MBPP.
6. Hard science: questions a PhD student would struggle with. GPQA.
7. Agentic work: give the model a real repository and a real bug report and see whether the tests pass afterwards. SWE-bench.
8. Reasoning puzzles: coloured grid puzzles with no text at all. ARC-AGI.
9. Common sense: pick the sensible ending to a sentence. HellaSwag.
10. Instruction following: did it obey the format you asked for? IFEval.
11. Chat quality: judged by a human or by another model. MT-Bench.
12. Each of these was built for a reason, at a time, by somebody.
13. Almost all of them are now much easier than when they were built.
14. That is not because they were badly made. It is because the models improved and because the tests leaked into training data.
15. The single most useful habit is to ask, for each benchmark: what would a model have to be able to do to score well, and is that what I need?

PLAIN50.2.2 a picture in your head#

1. Think of a hospital’s set of blood tests.
2. Each test measures one narrow thing. Iron. Glucose. White cell count.
3. No single test tells you whether a person is healthy.
4. A doctor orders several, reads them together, and knows which ones are unreliable in which circumstances.
5. A benchmark suite works the same way. MMLU is a bit like a general inflammation marker: broad, cheap, and non-specific.
6. SWE-bench is like a treadmill stress test: expensive, slow, closer to the real activity, and it can be gamed by training on the treadmill.
7. And here is the part that matters. Some blood tests go out of date. A test that was diagnostic in 1990 may be useless now because everyone takes a supplement that changes the reading.
8. Benchmarks go out of date the same way, and for a similar reason: everybody now optimizes for the reading.

Where this comparison breaks: blood tests have reference ranges established by regulators and reproducible physical chemistry behind them. AI benchmarks have neither. Two labs running the same blood test on the same sample should agree within a known tolerance. Two labs running the same AI benchmark on the same model have no agreed tolerance at all.

PLAIN50.2.3 a worked example#

1. Take one benchmark and follow it end to end: GPQA.
2. GPQA stands for Graduate-level Google-Proof Question Answering.
3. It was published on 20 November 2023 by David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael and Samuel Bowman.
4. It has 448 multiple-choice questions in biology, physics and chemistry.
5. Every question was written by a person with or studying for a PhD in that subject.
6. “Google-proof” is a design goal, not a guarantee. It means the writers tried hard to make the answer un-findable by search.
7. Here is the evidence they gave for that. Skilled non-experts, allowed unrestricted web access and spending on average over 30 minutes per question, reached 34 per cent accuracy.
8. Experts in the matching field reached 65 per cent, or 74 per cent after correcting for questions the experts later agreed were flawed.
9. Chance is 25 per cent, because there are four options.
10. The best model at publication, based on GPT-4, scored 39 per cent.
11. GPQA Diamond is a 198-question subset. It keeps only questions where the expert validators agreed on the answer and most non-experts got it wrong. It is the hardest and cleanest slice.
12. Diamond is what almost every model announcement reports. When you see “GPQA”, check whether it means the 448 or the 198.
13. What GPQA measures well: whether a model holds deep, specific technical knowledge that is not one search away.
14. What it does not measure: whether the model can do science. There is no experiment, no hypothesis, no error bar, no lab. It is a multiple-choice quiz with four options and a one-in-four floor.
15. And by 2026 the top scores on Diamond had risen far above the 65 per cent expert figure, which does not mean the models are better scientists than PhD holders. It means they are better at that quiz.

PLAIN50.2.4 what is really happening inside#

1. Here is each major benchmark in one block: what it is, when, how big, how scored, and its main blind spot.
2. MMLU, 2020. “Measuring Massive Multitask Language Understanding”, by Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song and Jacob Steinhardt, first posted 7 September 2020.
3. 57 subjects, from elementary mathematics to professional law. Four options per question. About 14,000 test items in the widely used release, plus a 1,531-item validation split and a 285-item development split used for few-shot examples.
4. Scored as plain accuracy. Chance is 25 per cent.
5. Blind spot: it is recall of facts in multiple-choice form. It cannot tell you whether a model can write, reason for ten steps, use a tool, or admit ignorance. It also contains a known number of wrong reference answers.
6. MMLU-Pro, 2024. From the TIGER-Lab group, first posted 3 June 2024. About 12,000 questions, ten options instead of four, 14 subject areas, harder items, and many of the broken MMLU items removed.
7. Scores dropped by 16 to 33 points relative to MMLU. Chance falls to 10 per cent. Sensitivity to prompt wording fell from about 4 to 5 points on MMLU to about 2 points on MMLU-Pro, across 24 prompt styles they tried.
8. GSM8K, 2021. From “Training Verifiers to Solve Math Word Problems” by Karl Cobbe and colleagues at OpenAI, 27 October 2021. 8.5 thousand grade-school word problems: 7,473 for training, 1,319 for testing.
9. Scored by exact match on the final number. Blind spot: the arithmetic is trivial and the reasoning is two to eight steps. It has been effectively solved since about 2024 and no longer separates strong models.
10. MATH, 2021. Hendrycks and colleagues, 5 March 2021. 12,500 competition problems with full worked solutions, split 7,500 train and 5,000 test. Answers are mathematical expressions, not just numbers, which makes marking much harder. A 500-item subset called MATH-500 is widely reported.
11. HumanEval, 2021. From “Evaluating Large Language Models Trained on Code” by Mark Chen, Jerry Tworek, Heewoo Jun and 51 others at OpenAI, 7 July 2021. Exactly 164 hand-written Python problems.
12. Each has a function signature, a docstring and hidden unit tests. Scored by pass@k: the chance that at least one of k samples passes all the tests. The original Codex model scored 28.8 per cent at pass@1 and 70.2 per cent at pass@100.
13. HumanEval blind spots, and there are many. 164 items is tiny, so one item is 0.6 points. The problems are self-contained functions of a few lines. There is no project, no dependency, no existing code to read, no debugging. The tests are shallow. It is Python only. And it is thoroughly leaked.
14. MBPP, 2021. “Program Synthesis with Large Language Models” by Jacob Austin, Augustus Odena and colleagues at Google, 16 August 2021. 974 entry-level Python problems, each with three assert-statement tests. A hand-cleaned subset of 427 problems is often used instead.
15. GPQA, 2023, covered above. 448 items, Diamond subset 198.
16. SWE-bench, 2023. “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?” by Carlos Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press and Karthik Narasimhan at Princeton, 10 October 2023.
17. 2,294 tasks built from real merged pull requests in 12 popular Python repositories. The model gets the repository at the parent commit and the issue text. It must produce a patch. The patch is applied and the project’s own tests are run.
18. Scored as resolve rate: the percentage of tasks where the hidden tests that the real pull request made pass, do pass, and nothing that passed before breaks. This is a much stronger signal than string matching.
19. In the original paper the best model tested, Claude 2, resolved 1.96 per cent. By 2026 the reported figures on the Verified subset were far above 70 per cent. That is a genuine capability jump and also a heavily optimized target.
20. SWE-bench Verified, 13 August 2024, from OpenAI. 93 experienced Python developers reviewed 1,699 randomly drawn original tasks. Tasks were dropped if the issue text was underspecified or the tests were unfair. 500 were kept. OpenAI reported that 68.3 per cent of sampled original tasks were filtered out for such problems.
21. That is an important admission in both directions. The original benchmark understated model ability. And a benchmark curated by one lab is now the number everyone quotes.
22. Other official variants: SWE-bench Lite (300 tasks, cheaper), SWE-bench Multimodal (517 tasks with visual elements), SWE-bench Multilingual (300 tasks from 42 repositories in 9 languages), and a Bash-only view of the 500 Verified tasks.
23. ARC-AGI. Created by Francois Chollet and introduced in his 2019 paper “On the Measure of Intelligence”. The tasks are small coloured grids. You see two or three input-output examples and must produce the output for a new input. No text. No knowledge required.
24. The design goal is to measure skill acquisition on unseen problems rather than stored knowledge. ARC-AGI-2, launched in 2025, has 120 public evaluation tasks and 120 held-back semi-private tasks, and every task was solved by at least two humans in under two attempts, calibrated on over 400 members of the public. ARC-AGI-3 followed.
25. The ARC Prize leaderboard shows cost per task alongside score and by default excludes systems costing more than 10,000 dollars for a run. That is unusual and honest, and more benchmarks should copy it.
26. HellaSwag, 2019. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi and Yejin Choi, 19 May 2019. Pick the sensible continuation of a short description from four options. About 10,000 items in the reported split.
27. The wrong options were produced by Adversarial Filtering: generate candidates by machine, keep the ones a discriminator model finds hard. Humans scored above 95 per cent; the best models of 2019 under 48 per cent. Today it is saturated and is mostly a sanity check on base models.
28. BIG-bench, 2022. “Beyond the Imitation Game”, posted 9 June 2022, with over 450 authors from over 132 institutions and 204 tasks. Deliberately weird, broad and long-tailed.
29. Every BIG-bench task file contains a canary string, a fixed random-looking identifier the authors ask people never to remove, so that anyone can test whether a model has seen the benchmark. That is a genuinely good idea, and section 50.5 explains what it can and cannot prove.
30. BIG-Bench Hard, 2022, is a 23-task subset where models at the time did worse than the average human rater. It is what people actually run, usually as “BBH”.
31. IFEval, 2023. Jeffrey Zhou and colleagues at Google, 14 November 2023. About 500 prompts carrying 25 kinds of machine-checkable instruction, such as “write more than 400 words” or “mention the keyword AI at least 3 times” or “reply in JSON” or “no capital letters”.
32. Its whole point is that no judgement is needed: a short Python function checks compliance. It measures obedience to format, not quality of content.
33. MT-Bench, 2023. From “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena” by Lianmin Zheng, Wei-Lin Chiang, Ying Sheng and colleagues, 9 June 2023. 80 multi-turn questions in 8 categories, two turns each.
34. A strong model grades the answers on a 1 to 10 scale. The paper reported over 80 per cent agreement between the model judge and human raters, which is about the level humans agree with each other.
35. The same paper named the three biases of a model judge that you must remember: position bias (it prefers whichever answer came first), verbosity bias (it prefers the longer answer) and self-enhancement bias (it prefers text in its own style).
36. AIME. Not built for AI at all. The American Invitational Mathematics Examination is a competition run by the Mathematical Association of America: 15 questions, 3 hours, every answer an integer from 000 to 999, held twice a year.
37. That integer answer format is exactly why AI labs adopted it: marking is trivial and unambiguous. Its weaknesses are that 15 items means one question is 6.7 points, and that each year’s paper is public within hours and in training data within months.
38. Humanity’s Last Exam, launched in 2025 by the Center for AI Safety and Scale AI, published in the journal Nature on 28 January 2026. 2,500 questions across more than 100 subjects, written by close to a thousand expert contributors from over 500 institutions in 50 countries.
39. It was built precisely because MMLU had saturated. A private held-back portion is kept to detect overfitting, and a rolling variant was released on 8 October 2025 to keep adding fresh items.
40. FrontierMath, from Epoch AI, is a set of unpublished research-level mathematics problems in difficulty tiers, plus a separate collection of genuinely unsolved open problems. As of 31 July 2026 that open set held 50 unsolved research problems, of which AI systems had contributed solutions to three.
41. Terminal-Bench, a Stanford collaboration, gives an agent a real terminal and real tasks in software engineering, machine learning, security and data science, scored by whether the task actually completed. Version 1.0 had 80 tasks, version 2.0 had 89, and later versions followed.

TECHNICAL50.2.5 the engineer’s version#

1. The comparison table, split into four-column pieces so it prints. All item counts verified August 2026 against the dataset cards and papers.

2. Scoring rules and chance floors.

3. The gap between what each measures and what it is claimed to measure. This column is the one people skip.

4. Established fact: the item counts, dates, authors and scoring rules above. These are checkable in the papers and dataset cards, and they do not move.
5. Active research: whether any of these benchmarks predicts real-world task performance. The honest state of the field is that correlation exists but is weak and task-dependent, and there is no accepted transfer function from a benchmark score to a deployment outcome.
6. Marketing claim: any sentence of the form “scores X on benchmark Y, therefore it is a PhD-level scientist” or “therefore it can replace a junior engineer”. The benchmark supports the first clause only.
7. Tooling. lm-eval --model hf --model_args pretrained=... --tasks mmlu --num_fewshot 5 runs MMLU under the EleutherAI harness. The SWE-bench project ships its own Docker-based runner, because reproducing per-repository environments is most of the difficulty. bigcode-evaluation-harness is the common runner for HumanEval and MBPP and executes generated code in a sandbox, which you should never skip.
8. Note on the honest version of “SWE-bench measures agentic ability”. The honest version: SWE-bench measures the combined ability of a model plus a scaffold plus a retrieval strategy plus a test-running loop. Two entries using the same model and different scaffolds can differ by more than 20 points. Section 50.4 returns to this.

WORDS50.2.6 remember these#

1. Saturation — everyone scores near the top so the test stops sorting — the point where score variance is dominated by noise and label errors rather than capability differences.
2. Diamond subset — the hard, clean slice of GPQA — the 198 questions where expert validators agreed and most non-experts failed.
3. Resolve rate — how often the fix actually worked — the fraction of SWE-bench instances where the fail-to-pass tests pass and pass-to-pass tests still do.
4. Adversarial filtering — keeping only wrong answers that fool machines — an iterative data-collection loop where a discriminator selects distractors.
5. Canary string — a marker planted so leaks can be spotted — a fixed unique identifier embedded in benchmark files to detect web-scrape contamination.
6. Chance floor — the score you get by guessing — the expected accuracy of a uniform random policy, 1/n for n options.
7. Held-back set — questions kept secret — a private evaluation split withheld from publication to measure overfitting to the public split.
8. Scaffold — the harness of code around a model in agent tasks — the controller implementing retrieval, tool calls, retries and stopping rules.

50.3 Actually running an evaluation#

PLAIN50.3.1 in simple words#

1. Suppose you have the questions and the answers. How do you run the test?
2. First you must decide what text to actually send.
3. You could send only the question. That is called zero-shot.
4. You could send a few solved examples first, then the question. That is called few-shot, and five examples is called five-shot.
5. Few-shot examples do two jobs. They remind the model of the subject, and more importantly they show it the exact answer format you want.
6. Most of the benefit is the format, not the reminder.
7. Next you decide whether to let the model think out loud before answering. That is chain-of-thought.
8. On maths, letting it think out loud changes the score enormously. Not by a little. By tens of points.
9. Next you decide how random the model is allowed to be. That is the temperature setting.
10. Zero means always take the most likely next word. Higher means sometimes take a less likely one.
11. If temperature is above zero, the same model on the same question can give different answers on different days.
12. So the same evaluation run twice gives two different scores.
13. That is why careful teams run the whole thing several times and report the average, and ideally the spread.
14. Then you must decide how to mark the reply, which is the hardest part.
15. For a single number, compare numbers. For a letter, find the letter. For an essay, you need a human or another model, and both are unreliable.

PLAIN50.3.2 a picture in your head#

1. Think of a school exam being marked by three different people.
2. The first marker demands the answer in the box, in pen, and nothing else. Anything in the margin is ignored.
3. The second marker reads the whole page and looks for the answer wherever it is. More forgiving, but sometimes picks up a crossed-out number.
4. The third marker is an experienced teacher who reads the reasoning and awards a judgement out of ten.
5. The first marker is exact match. The second is regex extraction. The third is a model judge.
6. Now imagine the exam paper itself. One version says “Answer the question.” Another says “Answer the question. Show your working. Put your final answer after the words FINAL ANSWER.”
7. The second paper produces higher marks from the same pupils, because it told them the format the marker wants.
8. That is the whole of prompt template design, in one sentence.

Where this comparison breaks: a pupil who knows the answer but writes it in the margin still knows the answer, and a good teacher notices. A benchmark harness does not notice anything. It runs a regular expression. If the format is wrong the mark is zero, and the report will say the model does not know maths.

PLAIN50.3.3 a worked example#

1. Let us compute pass@k properly, because it is the metric people most often misread.
2. pass@k is not “the score when you take k tries”. It is an estimate of the probability that at least one of k independently drawn samples is correct.
3. The naive way is to draw k samples and see if any pass. That is a very noisy estimate for large k.
4. The Codex paper of July 2021 uses a better estimator. Draw n samples, count c that pass, and compute:

pass@k = 1 - C(n - c, k) / C(n, k)

  where C(a, b) is "a choose b", the number of ways
  to pick b items from a, and n >= k.

5. In words: one minus the chance that all k of your picks come from the failing pile.
6. Worked example. Take one problem. Draw n = 10 samples. Suppose c = 2 pass.
7. pass@1 = 1 - C(8,1)/C(10,1) = 1 - 8/10 = 0.20.
8. pass@5 = 1 - C(8,5)/C(10,5) = 1 - 56/252 = 1 - 0.2222 = 0.7778.
9. pass@10 = 1 - C(8,10)/C(10,10). C(8,10) is 0 because you cannot pick 10 things from 8. So pass@10 = 1 - 0 = 1.00.
10. Read those three numbers. One try: 20 per cent. Five tries: 78 per cent. Ten tries: 100 per cent.
11. The model did not get better. You bought more lottery tickets.
12. This is why pass@1 and pass@100 are not comparable, and why a headline that does not say which k it used is not a result.
13. Now the temperature effect, with real published behaviour. High temperature lowers pass@1 and raises pass@100, because diversity costs you on a single draw and pays on many draws.
14. So a lab reporting pass@1 will tune temperature low, and a lab reporting pass@100 will tune it high, and both are being reasonable.

PLAIN50.3.4 what is really happening inside#

1. Take one MMLU item and watch it through the whole pipeline.
2. Zero-shot generative version. The harness builds this text:

The following is a multiple choice question about
high school biology.

Which organelle produces most of a cell's ATP?
A. Ribosome
B. Mitochondrion
C. Golgi apparatus
D. Lysosome
Answer:

3. The model generates text. Suppose it writes ” B. Mitochondrion”.
4. The extractor strips whitespace, takes the first letter that is A, B, C or D, and compares it to the stored key.
5. Now the log-likelihood version. The harness does not let the model write anything. Instead it asks the model, four times, “how likely is this exact continuation?”
6. It scores ” A”, ” B”, ” C” and ” D” as continuations, and takes the highest.
7. Same item, same model, two protocols, and sometimes two different answers.
8. A model that would have written a paragraph of hedging before saying B still assigns the highest single-token probability to ” B”, and scores right under the second protocol and possibly wrong under the first.
9. This is not a subtle effect. It is a common cause of a several-point gap between two published MMLU numbers.
10. Now the few-shot version. The harness prepends five solved questions from the 285-item development split before the real one.
11. The model now sees five examples of the pattern “question, options, Answer: B” and copies the pattern. Format failures collapse.
12. Now chain-of-thought. The template adds “Think step by step, then give your final answer as a single letter after the words FINAL ANSWER:”.
13. The model now emits a paragraph of reasoning and then the letter. The extractor looks after “FINAL ANSWER:” instead of at the start.
14. On knowledge questions this changes little. On maths it changes everything, because the model can hold intermediate results in the text it has already written rather than trying to do the whole calculation in one forward pass.
15. That last sentence is the honest mechanism. Chain-of-thought is not the model “thinking harder”. It is the model using its own output as working memory, one token at a time.

TECHNICAL50.3.5 the engineer’s version#

1. Prompt sensitivity is measurable and large. The MMLU-Pro authors evaluated 24 prompt styles and found score variation of about 4 to 5 percentage points on original MMLU and about 2 points on MMLU-Pro. That is the size of the gap labs argue over in announcements.
2. Few-shot conventions by benchmark, as used in most reported numbers. These are conventions, not standards, and they drift.

3. Chain-of-thought prompting comes from “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models” by Jason Wei and colleagues at Google, posted January 2022. On GSM8K with a 540-billion-parameter model it moved accuracy from roughly 18 per cent to roughly 57 per cent.
4. “Large Language Models are Zero-Shot Reasoners” by Takeshi Kojima and colleagues, May 2022, showed that the single sentence “Let’s think step by step” produced much of the same effect with no examples at all.
5. Established fact: chain-of-thought raises measured accuracy on multi-step arithmetic and symbolic tasks by a large margin. Active research: whether the written chain reflects the computation that actually determined the answer. There is published evidence that models sometimes produce a correct answer with an unfaithful chain, and that they can be steered to a wrong answer while the chain still reads plausibly.
6. Grading methods, in increasing order of flexibility and decreasing order of reliability.

7. For MATH-style answers, plain string comparison is hopeless: 1/2, 0.5 and \frac{1}{2} are the same answer. Serious harnesses parse into a computer algebra system and test symbolic equality, with a timeout.
8. Model-as-judge biases are documented in the MT-Bench paper of June 2023: position bias, verbosity bias, self-enhancement bias, and limited reasoning on maths. Mitigations in common use are swapping the order of the two answers and averaging, calibrating with a reference answer, and using chain-of-thought in the judge prompt.
9. Sampling and reproducibility. With temperature 0 and greedy decoding, output is nominally deterministic. In practice it often is not.
10. The honest version: floating-point reduction order on a GPU depends on how work is split across threads, that split can depend on the batch size, and batch composition on a shared serving endpoint depends on who else is sending requests. Mixture-of-experts routing can also depend on the batch. So the same prompt at temperature 0 can produce different tokens.
11. Therefore report mean and standard deviation over repeated runs. Common practice for small benchmarks such as AIME with 15 items is to run 16, 32 or 64 times and average, sometimes written “avg@32” or “mean@64”.
12. Distinguish that from majority voting, sometimes written “cons@64” or self-consistency, where you sample 64 answers and take the most common one. That is a different, stronger system, and it costs 64 times as much.
13. Distinguish both from best-of-n, where you sample n answers and pick the best using a verifier or the reference answer. If the reference answer is used to pick, the number is close to meaningless as a measure of what a user would get.
14. Note the 2026 wrinkle: some providers have begun removing sampling controls entirely. Anthropic’s documentation as of August 2026 lists temperature, top_p and top_k as deprecated for its newest models, returning an error if set to a non-default value. If you cannot set temperature, you cannot match another lab’s sampling configuration.

WORDS50.3.6 remember these#

1. Zero-shot — no examples given — the item is presented with instructions only, no in-context exemplars.
2. Few-shot — a few solved examples first — k in-context exemplars prepended, usually drawn from a fixed development split.
3. Chain-of-thought — letting it show its working — eliciting intermediate reasoning tokens before the final answer, which act as external working memory.
4. Temperature — how random the wording is — the divisor applied to logits before the softmax; 0 gives greedy decoding.
5. pass@k — chance that one of k tries works — the unbiased estimator 1 - C(n-c,k)/C(n,k) over n samples with c correct.
6. Self-consistency — sample many, take the most common answer — majority voting over sampled chains, reported as cons@n.
7. Best-of-n — sample many, keep the best — reranking n samples with a verifier or, improperly, with the reference answer.
8. LLM-as-judge — a model marks the answers — an evaluator model scoring or ranking outputs, subject to position, verbosity and self-enhancement bias.

50.4 Why two labs report different numbers#

PLAIN50.4.1 in simple words#

1. You will often see the same model, the same benchmark, and two different numbers, from two respectable sources.
2. The first thing to accept is that this is normal.
3. The second thing to accept is that it is usually not lying.
4. There are about eight ordinary causes, and you can usually work out which one applies.
5. Cause one: different wording of the prompt.
6. Cause two: a different number of worked examples given first.
7. Cause three: a different way of pulling the answer out of the reply.
8. Cause four: a different slice of the items. Someone ran 500 of the 2,294.
9. Cause five: different randomness settings, or one run against an average of many runs.
10. Cause six: one lab used a system prompt telling the model how to behave and the other did not.
11. Cause seven: one setup let the model use a calculator, a code runner or a web search, and the other did not.
12. Cause eight: a plain bug. These are common and usually found later.
13. Only after all eight are ruled out is it worth wondering about anything worse.
14. The practical response is one question: which harness did you use, and with what settings?
15. If the answer is “our internal one, details not published”, the number is not comparable to anybody else’s number. It may still be true.

PLAIN50.4.2 a picture in your head#

1. Imagine two labs measuring the fuel economy of the same car.
2. One drives a flat motorway at a steady 90 kilometres per hour with the air conditioning off and the tyres at maximum pressure.
3. The other drives a hilly city route in traffic with two passengers.
4. Both publish a figure in litres per hundred kilometres. The figures differ by 40 per cent. Neither lab lied.
5. This is exactly why car fuel economy eventually got a legally defined test cycle, with a written specification, run by accredited labs.
6. AI benchmarks have no such thing. There is no accredited lab, no test cycle written into law, and no penalty for using a favourable route.
7. The nearest equivalent is a shared open harness that everybody agrees to run with the same settings, and that is a voluntary convention.

Where this comparison breaks: fuel economy has a physical ground truth you can approach by measuring more carefully. There is no single true MMLU score for a model waiting to be discovered. The number genuinely depends on the protocol, so “measuring more carefully” does not converge on one value.

PLAIN50.4.3 a worked example#

1. Here is a concrete, realistic divergence, with the arithmetic.
2. Lab A reports 88.4 per cent on a knowledge benchmark. Lab B reports 84.1 per cent for the same model. The gap is 4.3 points.
3. Investigation finds four contributions.

4. Every one of those four choices is defensible on its own.
5. Together they fully account for the gap without anybody doing anything improper.
6. Notice the last row. Reporting the best of three runs rather than the mean is the one that should make you uncomfortable, because it is a choice made after seeing the results.
7. That is the boundary between a defensible protocol difference and a presentational thumb on the scale.

PLAIN50.4.4 what is really happening inside#

1. The causes, with how you detect each one. This is the table to keep.

2. On tool access, be specific, because it is the largest single effect and the least often disclosed.
3. A model that may run Python and check its arithmetic will beat the same model without that on any maths benchmark, often by a wide margin.
4. A model that may search the web will beat the same model without it on any knowledge benchmark, and on a “Google-proof” benchmark it should not help much, which is itself a useful test of the benchmark’s design.
5. On agentic benchmarks, the scaffold is most of the system. An entry on the SWE-bench leaderboard is a model plus a controller: how files are found, how many edit attempts are allowed, whether tests may be run and re-run, how the loop decides to stop.
6. Two teams using the identical model with different scaffolds have reported resolve rates tens of points apart. The leaderboard entry is the pair, not the model.
7. That is why SWE-bench leaderboard entries are named after systems, not models, and why a model announcement quoting a SWE-bench number should say what scaffold produced it.

TECHNICAL50.4.5 the engineer’s version#

1. Ask for the harness by name and version. In practice this means one of: EleutherAI lm-evaluation-harness (state the version and the task name, because task definitions change between versions), Stanford HELM, the bigcode-evaluation-harness for code, the official SWE-bench runner with its Docker images, or a lab’s internal harness.
2. Task definitions inside a harness are versioned and do change. The same task name in two harness versions can differ in prompt, in normalization and in which split is used. Pin the harness version in your own reports.
3. The Hugging Face Open LLM Leaderboard is the clearest example of why this matters. Its second version, launched in June 2024, replaced the original task set with IFEval, BBH, MATH Level 5, GPQA, MuSR and MMLU-Pro, run through the EleutherAI harness with fixed settings.
4. The reason given was that the original tasks had saturated and were contaminated. The consequence was that scores from before and after are not comparable at all, even for the same model.
5. That is the honest version of “the leaderboard changed”: every historical number became a different measurement, and the ranking of models moved.
6. Normalization choices worth naming explicitly, because they each move multiple-choice scores by measurable amounts: raw log probability, log probability divided by token count, log probability divided by character count, and probability normalized against the model’s unconditional probability of the option text.
7. A concrete reproducibility checklist to publish with any number you report:

model id + exact snapshot
harness name + version + task name
n-shot, and which split the shots came from
system prompt (verbatim, or "none")
temperature, top_p, max tokens, stop sequences
tools available (none / code / search / retrieval)
number of runs, and mean and standard deviation
answer extraction rule
date the run was executed

8. If a published number lacks four or more of those nine lines, treat it as an indication rather than a measurement.
9. Established fact: protocol differences of the sizes described here are documented and reproducible. Marketing claim: that any single reported number is “the” score for a model.

WORDS50.4.6 remember these#

1. Harness — the program that runs the benchmark — the evaluation framework; name and version both matter because task definitions change.
2. Subset — a slice of the items — a filtered evaluation split such as Lite, Verified or Diamond, not comparable to the full set.
3. System prompt — standing instructions given before the conversation — a privileged message setting persona, format and constraints, which materially changes benchmark scores.
4. Scaffold — the code around the model in an agent task — the controller loop for retrieval, tool invocation, retries and termination.
5. Tool access — whether the model may use a calculator or search — the set of external functions the model may call during evaluation.
6. Reproduction — someone else getting your number — an independent run under a published protocol, the only real check on a self-reported score.
7. Task version — which definition of the test was used — the versioned task implementation inside a harness, including prompt and normalization.

50.5 Contamination#

PLAIN50.5.1 in simple words#

1. Contamination means the test questions were in the training data.
2. If the model has already seen the question and the answer, getting it right proves nothing about ability.
3. It might be memory. It might be ability. From the score alone you cannot tell which.
4. This happens almost by accident, not by plotting.
5. Benchmarks are published so that people can use them. Published means on the web. On the web means scraped. Scraped means in the training data.
6. GSM8K, MMLU, HumanEval and the rest are all on public code and dataset hosting sites, quoted in thousands of blog posts, pasted into forum questions, translated, reformatted and discussed.
7. So a model trained on a large web crawl has almost certainly seen many of them, in some form, unless someone worked hard to remove them.
8. That removal work is called decontamination, and every serious lab now does some and describes it.
9. But decontamination is filtering by text matching, and text matching misses anything reworded.
10. The effect on scores is upward, and it is uneven. It inflates old benchmarks more than new ones, and easy items more than hard ones.
11. It is very hard to prove for any particular model, because you usually cannot see the training data.
12. So the debate is conducted with indirect evidence, and reasonable people disagree about what that evidence shows.
13. The honest position: assume some contamination is present in every public benchmark number you read, of unknown size, and prefer evidence from tests the model cannot have seen.

PLAIN50.5.2 a picture in your head#

1. Imagine a school where the past twenty years of exam papers, with the official answers, are freely available online.
2. A pupil who has read all of them will do better on this year’s paper if this year’s paper reuses old questions.
3. Nobody stole anything. The papers were published on purpose.
4. Now the exam board tries to fix it. They remove any question that appears word for word in an old paper.
5. But a question that was reworded, or translated into French and back, or changed from apples to oranges, slips through.
6. And the pupil who read every paper does better on those too, because the structure is familiar even when the words changed.
7. Now the awkward part. The pupil who read all the old papers is also genuinely better prepared. Some of what they gained is real learning.
8. Separating the memory from the learning, from the mark alone, is impossible.
9. That is exactly the position everyone is in with model benchmarks.

Where this comparison breaks: a pupil remembers roughly what they revised and can be asked. A model cannot reliably report what was in its training data, and the people who built it often cannot enumerate it either, because the corpus is billions of documents assembled by pipelines from many sources.

PLAIN50.5.3 a worked example#

1. The clearest public experiment on this is GSM1k.
2. In May 2024 a team at Scale AI, led by Hugh Zhang, published “A Careful Examination of Large Language Model Performance on Grade School Arithmetic”.
3. They commissioned a brand-new set of grade-school maths problems, written from scratch, never published, deliberately matched to GSM8K in style, number of solution steps, answer size and human solve rate.
4. Then they ran many models on both, and compared.
5. If a model’s ability were real, the two scores should be about equal, since the sets are matched by construction.
6. What they found: accuracy drops of up to about 8 percentage points on the new set, with several model families showing systematic overfitting across almost all their sizes.
7. They also found a positive relationship, Spearman r squared of about 0.36, between how likely a model was to generate GSM8K examples verbatim and how large its drop was.
8. That second finding is the important one. It links the drop to memorization rather than to some accident of the new problem set.
9. And the honest counterweight, which the same paper reports: the strongest frontier models showed minimal signs of overfitting, and every model generalized to some extent to the new problems.
10. So the correct summary is not “the benchmarks are fake”. It is “some models are inflated by several points, the size varies, and you cannot tell from the leaderboard which”.

PLAIN50.5.4 what is really happening inside#

1. Five detection methods, what each can show, and where each fails.
2. N-gram overlap. Take every run of n consecutive words from every benchmark item, and search the training corpus for the same run.
3. The GPT-3 paper of 2020 used 13-gram overlap for its decontamination checks. Other projects use 8-gram, or a match on several random 50-character substrings.
4. What it shows: verbatim copying. What it misses: anything reworded, translated, reformatted, or discussed rather than quoted.
5. Also it requires access to the training corpus, which outsiders do not have.
6. Canary strings. The BIG-bench authors put a fixed unique identifier in every task file and asked everybody never to remove it.
7. If a model can reproduce that identifier, it saw the file. That is close to proof of exposure.
8. What it misses: everything. A model can be trained on a copy of the data with the canary stripped, on a reworded version, or on a discussion of the items. Absence of the canary proves nothing at all.
9. Perplexity comparison. Perplexity is a measure of how surprised the model is by a piece of text. Lower means more familiar.
10. Compare the model’s perplexity on benchmark items against its perplexity on freshly written text of the same kind. If the benchmark is much less surprising, that is a signal.
11. What it misses: benchmark text is often cleaner and more formulaic than fresh text anyway, so some gap is expected. Setting the threshold is a judgement call, and this requires log-probability access that many commercial endpoints no longer expose.
12. Old versus new items. Build a benchmark that keeps adding fresh items after model release dates, and compare a model’s score on items published before its training cutoff against items published after.
13. A large gap is strong evidence. This is the design principle behind continuously refreshed benchmarks, and behind coding benchmarks that only score problems released after a stated date.
14. What it misses: newer items may simply be harder, or drawn from a different distribution, so you need the sets to be matched carefully, which is what GSM1k did.
15. The completion trick. Give the model the first part of a benchmark item and see whether it continues with the rest, word for word, including the incidental details it could not guess.
16. A variant gives the model the dataset name, the split and an index, and asks it to produce that item. Another shows the item with the options in a scrambled order and checks whether the model still reproduces the original ordering.
17. What it shows: near-verbatim memory of the item. What it misses: a model that learned the answer without being able to reproduce the wording, which is the common case after instruction tuning.
18. Notice the pattern. Every method can produce a positive finding that is hard to dismiss, and no method can produce a negative finding that means anything.
19. That asymmetry is why this argument never ends.

TECHNICAL50.5.5 the engineer’s version#

1. The methods and their properties, side by side.

2. Real disputes, described without alleging deliberate wrongdoing, because in none of these cases was deliberate cheating established.
3. In 2023 the paper “Rethinking Benchmark and Contamination for Language Models with Rephrased Samples” showed that a modestly sized model trained on rephrased versions of benchmark test sets could reach very high scores on those benchmarks while passing standard n-gram decontamination checks. The point was methodological: the standard filter does not work.
4. Also in 2023, a deliberately satirical paper by Rylan Schaeffer titled “Pretraining on the Test Set Is All You Need” trained a tiny model directly on benchmark test sets and reported near-perfect scores, to make the same point in the bluntest possible way.
5. Several strong small models released between 2023 and 2025 attracted public questions about whether their synthetic training data, which was generated by larger models, indirectly reproduced benchmark items. The teams involved published decontamination procedures in response. The disputes were about method and evidence, not about proven intent.
6. In late 2024 and early 2025 there was a public discussion about FrontierMath, an Epoch AI benchmark, after it emerged that OpenAI had funded its creation and had access to problems. Epoch AI published clarifications about what access existed and what hold-out arrangements were in place.
7. The general lesson from that episode is structural, not personal: if a lab funds or co-designs a benchmark it is later measured on, the benchmark needs a documented held-out portion and a stated access policy, or the number cannot carry weight regardless of anyone’s good faith.
8. Which is why held-out sets now exist as standard practice: ARC-AGI keeps a semi-private evaluation set, Humanity’s Last Exam keeps a private portion, and SWE-bench-style benchmarks increasingly add tasks dated after model releases.
9. Typical decontamination procedures now described in model cards and technical reports: exact-substring matching against known benchmark items, n-gram overlap filters at 8 to 13 grams, removal of documents containing canary strings, embedding-similarity screening for near-duplicates, and post-hoc measurement of the train-test gap.
10. Established fact: benchmark items appear in web crawls, and models can be induced to reproduce some of them verbatim. Established fact: matched fresh benchmarks show measurable score drops for some model families.
11. Active research: how to measure contamination without corpus access, how to correct scores for it, and how to build benchmarks that resist it by construction. There is no accepted standard method.
12. Marketing claim: “our model is decontaminated”. Decontamination is a process with a coverage rate, not a state. Ask what filter, at what n-gram length, against which benchmark list, and what the measured residual was.
13. Expert disagreement, stated plainly. One camp holds that contamination is now the dominant explanation for headline benchmark gains on older tests, and that only dated, held-out or private evaluations should be believed. The other camp holds that frontier models demonstrably solve novel problems that cannot be in any corpus, so contamination inflates absolute numbers but rarely reverses rankings. Both camps agree that public numbers on pre-2023 benchmarks should be discounted.

WORDS50.5.6 remember these#

1. Contamination — the test was in the training data — overlap between evaluation items and the pretraining or post-training corpus.
2. Decontamination — trying to remove it — filtering training documents that match benchmark items by substring, n-gram or embedding similarity.
3. N-gram — a run of n consecutive words — the unit of overlap matching, with 8-gram and 13-gram both in common use.
4. Canary string — a planted marker — a unique identifier embedded in benchmark files, whose reproduction by a model evidences exposure.
5. Perplexity — how surprised the model is by text — the exponential of the mean negative log likelihood per token.
6. Held-out set — questions nobody publishes — a private split retained by the benchmark authors to measure overfitting to the public split.
7. Overfitting to a benchmark — being good at the test, not the skill — a gap between performance on the benchmark and on matched unseen items.
8. Membership inference — guessing whether an example was trained on — a statistical test using token likelihoods, such as scoring the lowest-k token probabilities of a document.

50.6 The other reasons to be sceptical#

PLAIN50.6.1 in simple words#

1. Contamination is not the only problem. There are six more, and they are all ordinary and human.
2. Saturation. When everyone scores 92, 93 and 94 per cent, the test has stopped telling you anything.
3. At that point the remaining errors are often wrong answer keys in the benchmark itself, not model failures.
4. Optimizing for the test. Once a number matters, people work on the number. Some of that work improves the model. Some improves only the number.
5. Self-reported results. Nearly every headline benchmark figure in a model announcement was produced by the team that made the model.
6. That is not automatically wrong. It is simply unverified until someone else runs it.
7. Cherry-picked comparisons. Showing your new model against a competitor’s older version, or against the competitor’s cheap model rather than its best.
8. Best-of-n reporting. Letting the model try many times and reporting the best result, without saying so clearly.
9. Different tools and scaffolding. Your model with a code runner against their model without one, in the same bar chart.
10. Charts that start above zero. A bar chart whose vertical axis begins at 70 makes a two-point difference look enormous.
11. None of these require anybody to lie. Every one of them can be produced by ordinary optimism and ordinary pressure to look good.
12. Your defence is not cynicism. It is a checklist.

PLAIN50.6.2 a picture in your head#

1. Think of school league tables.
2. Once schools are ranked by exam results, the results become the goal.
3. Schools start teaching to the exam. Then they enter weaker pupils for easier qualifications. Then some stop entering certain pupils at all.
4. The tables keep rising. Whether education improved is a separate question, and a much harder one to answer.
5. That is Goodhart’s law happening in public, over years, with real children.
6. The measure was fine as a measure. It stopped being fine the moment it became the target.
7. And note what is not implied: the teachers were not villains. They responded to the incentive they were given.

Where this comparison breaks: exam boards are regulated, publish syllabuses, and can be audited. Nobody audits an AI lab’s benchmark run. Also, schools cannot see next year’s paper. Anyone training a model can see every public benchmark, in full, with answers, for free.

PLAIN50.6.3 a worked example#

1. Goodhart’s law, stated properly, because it is usually misquoted.
2. Charles Goodhart was a British economist at the Bank of England. In a 1975 paper on monetary policy he wrote, in substance: any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.
3. That is the original and it is narrower than the popular version. It is about a regularity used as a control target.
4. The popular short form is due to the anthropologist Marilyn Strathern, who in 1997 wrote: when a measure becomes a target, it ceases to be a good measure.
5. A closely related statement is Campbell’s law, from the psychologist Donald Campbell in 1979: the more a quantitative indicator is used for social decision-making, the more it will be subject to corruption pressures and the more it will distort the processes it monitors.
6. Applied here: MMLU was an excellent measure in 2021, when nobody was training against it.
7. By 2024 every major lab knew the exact score they needed, the exact format, and the exact subjects where points were cheapest.
8. The measure did not become false. It became uninformative, which in practice is worse, because it still looks like a measurement.
9. Now a chart example, with numbers. Two models score 91.2 and 89.4 on some benchmark.

Axis starting at zero:          Axis starting at 88:

100 |                           92 |  ####
    |  ####  ####                  |  ####
 50 |  ####  ####               90 |  ####   ####
    |  ####  ####                  |  ####   ####
  0 +-------------              88 +--------------
      A=91.2  B=89.4                 A=91.2  B=89.4

10. Same two numbers. On the left, two nearly identical bars. On the right, one bar looks roughly twice the other.
11. The difference is 1.8 points, which on a 14,000-item benchmark is about 250 items, and which is within the range that a prompt template change can produce on its own.

PLAIN50.6.4 what is really happening inside#

1. The red flag checklist. Each line is a question to ask of any announcement or chart.
2. Does the chart’s vertical axis start at zero? If not, mentally redraw it.
3. Are the compared models named with their exact versions and dates, or just with product names?
4. Is the competitor’s newest model shown, or a version from six months ago?
5. Is the competitor’s flagship shown, or its cheaper tier?
6. Who ran the competitor’s numbers? If the announcing lab did, were the competitor’s own published numbers different?
7. Does the footnote say pass@1, or does it say something else, or nothing?
8. Does the footnote mention majority voting, consensus, best-of-n, or a number of samples greater than one?
9. Did both models have the same tools? Search, code execution, retrieval, file access, a scaffold?
10. Was there a system prompt, and was it the same for both?
11. Is the benchmark one where the top scores are already above 90 per cent?
12. Is the improvement larger than the known prompt-sensitivity range for that benchmark, which is roughly 2 to 5 points on multiple-choice tests?
13. Is the benchmark new, and was it built or funded by the announcing lab?
14. Has any independent party reproduced the number, and how long ago?
15. If you cannot answer eight of these fourteen from the announcement itself, the announcement is a claim, not a result.

TECHNICAL50.6.5 the engineer’s version#

1. Saturation has a precise consequence: as mean accuracy approaches the label ceiling, the between-model variance shrinks toward the label-noise variance, and rank ordering becomes unstable across runs.
2. MMLU is known to contain a non-trivial number of incorrect or ambiguous reference answers. Independent audits published between 2023 and 2024 put the error rate in some subsets in the range of several per cent, with virology and a few professional subjects worst. Estimates vary by auditor and by criteria, so treat the exact figure as disputed.
3. The practical implication: a reported 92 per cent on MMLU may be close to the maximum achievable, and a difference between 92 and 93 may be noise plus disagreement about wrong keys.
4. This is exactly why MMLU-Pro, GPQA Diamond, Humanity’s Last Exam and FrontierMath were built. New benchmarks appear when old ones saturate, and the appearance of a new hard benchmark is itself evidence that the previous one stopped working.
5. A documented case of the “which version did you test” problem. On 5 April 2025 Meta released Llama 4 in two configurations, Scout and Maverick. A variant identified as Llama-4-Maverick-03-26-Experimental, described as optimized for conversationality, scored very highly on the LMArena leaderboard. It was not the model released to the public.
6. On 8 April 2025 the arena operators stated that Meta’s interpretation of their policy did not match what they expect from providers, that Meta should have made it clearer that the entry was a customized model, and that they would update their policies. Meta denied training on test sets.
7. Report that as what it is: a disclosure and naming failure with a policy response, not an established case of fraud. The transferable lesson is that a leaderboard entry identifies a specific artifact, and the artifact you can download may not be the artifact that was measured.
8. Statistical hygiene almost nobody applies. For a benchmark of N items and an observed accuracy p, the standard error is roughly sqrt(p(1-p)/N).

9. Read that table carefully. On GPQA Diamond, a 2-point difference is about one standard error, and one item is 0.5 points. On a single AIME paper, getting one more question right is 6.7 points.
10. So a headline of the form “scores 93.3 on AIME versus 86.7 for the competitor” describes one question out of fifteen.
11. That is before considering run-to-run sampling variance, which for reasoning models on small maths sets is frequently larger than the between-model gap.
12. Established fact: the standard-error arithmetic above. Marketing claim: any ranking asserted from a gap smaller than one standard error.

WORDS50.6.6 remember these#

1. Goodhart’s law — a measure used as a target stops measuring — an observed regularity collapses when pressure is applied to it for control purposes.
2. Campbell’s law — indicators used for decisions get corrupted — quantitative social indicators distort the processes they are meant to monitor.
3. Saturation — the test is too easy now — mean score near the ceiling, so between-model variance falls below label-noise variance.
4. Label noise — wrong answers in the answer key — the fraction of benchmark items whose reference answer is incorrect or ambiguous.
5. Self-reported — the maker measured it — a result produced by the model provider under an unpublished or partially published protocol.
6. Truncated axis — a chart that hides how small a gap is — a bar chart whose value axis does not begin at zero.
7. Standard error — how much the score would wobble by chance — approximately sqrt(p(1-p)/N) for accuracy p over N independent items.
8. Cherry-picking — showing the favourable comparison — selective choice of baselines, versions, subsets or runs after seeing results.

50.7 The alternatives, and their own weaknesses#

PLAIN50.7.1 in simple words#

1. If fixed quizzes are weak, what else is there? Five things, each with its own flaw.
2. Preference arenas. Real people type a question, get two anonymous answers, and vote for the better one. Ratings are computed from the votes.
3. Private held-out tests. The same idea as a benchmark, but the questions are never published, so they cannot leak.
4. Domain evaluations. A test built for one job: medical coding, legal citation, chip design, customer support in Hindi.
5. Agentic and end-to-end tests. Give the model a real task in a real environment and check whether the task got done.
6. Red-teaming. People deliberately attack the model to find failures that no fixed test would catch.
7. Every one of these is better than MMLU for something, and worse for something else.
8. The arena measures whether people like the answer, which is not the same as whether the answer is correct.
9. A friendlier, longer, better-formatted wrong answer beats a curt correct one, quite often.
10. Private tests cannot be independently checked, because they are private.
11. Domain tests are only as good as the person who wrote them, and they do not transfer to other domains.
12. Agentic tests are expensive, slow, and measure the scaffold as much as the model.
13. Red-teaming finds problems but cannot tell you that no problems remain.
14. There is no method without a weakness. The working answer is to use several and to know what each one is blind to.

PLAIN50.7.2 a picture in your head#

1. Think about how you would judge a restaurant.
2. A hygiene inspection is a fixed checklist with a pass mark. Objective, narrow, and it says nothing about whether the food is good.
3. Customer star ratings are a preference arena. They capture something real that no inspection captures, and they reward large portions, friendly staff and good lighting as much as cooking.
4. A critic’s blind visit is a private held-out evaluation. Harder to game, impossible for you to verify.
5. Asking a chef to cook one specific dish you care about is a domain evaluation. Most informative for you, useless for anyone who wants something else.
6. Watching the kitchen run a full Saturday service is the agentic test. Closest to reality, and by far the most expensive to arrange.

Where this comparison breaks: restaurant customers eat the food, so their ratings are grounded in the actual product. Arena voters usually cannot verify whether the answer they preferred was true. They are rating the presentation of a claim, often on a topic where they are not the expert.

PLAIN50.7.3 a worked example#

1. How a preference arena turns votes into a number.
2. Each vote is a pairwise comparison: model A beat model B on this prompt.
3. The rating system is the one from chess, invented by Arpad Elo, a Hungarian-American physicist, and adopted by the United States Chess Federation in 1960 and by FIDE in 1970.
4. The core formula gives the expected score of A against B from the gap in their ratings:

E(A) = 1 / (1 + 10 ^ ((R_B - R_A) / 400))

R = rating. A 400-point gap means the stronger side
is expected to win about 10 times out of 11.

5. Work three cases. A gap of 10 points gives an expected win rate of 51.4 per cent. A gap of 100 points gives 64.0 per cent. A gap of 200 points gives 76.0 per cent.
6. Read the first one again. A 10-point difference on a leaderboard means the higher model wins about 51 times in 100. That is close to a coin flip.
7. Published confidence intervals on such leaderboards are frequently plus or minus 3 to 10 points, so models separated by fewer than about 15 points are usually statistically tied even though they occupy different rank numbers.
8. As of August 2026, the largest public text arena listed a few hundred ranked models, with top scores clustered within a few points of each other and stated uncertainties of roughly plus or minus 3 to 10 points.
9. A ranking that shows positions 1 through 8 spanning about 15 points is, in statistical terms, showing one group of models, not eight ranks.
10. The honest version: an arena rating is not a quality score. It is a fitted parameter that predicts how often this model’s answer is preferred over another model’s answer, by these voters, on these prompts, in this period. Every clause in that sentence limits what the number can be used for.

PLAIN50.7.4 what is really happening inside#

1. What an arena measures well: how a broad population of users, asking the kinds of question they actually ask, reacts to an answer.
2. That is genuinely valuable and no fixed benchmark captures it. It picks up tone, refusal behaviour, formatting, and the ability to guess what the user meant.
3. What it measures badly, and the biases are documented, not theoretical.
4. Length bias. Longer answers win more often, controlling for content. Arena operators now publish a length-controlled ranking alongside the raw one specifically because of this.
5. Style bias. Headings, bullet lists and bold text raise win rates. The same content flattened into prose wins less.
6. Verification gap. A voter comparing two answers about a legal deadline usually does not know the deadline. They vote on confidence and clarity.
7. Prompt distribution. The votes come from whoever uses the site. That population skews technical and English-speaking and asks a lot of short questions, so the rating reflects that mixture and not yours.
8. Provider self-selection. Providers choose when to enter a model and which variant, and can test variants privately before public entry.
9. Red-teaming, briefly, because it is a different shape of thing. It is not scored on a scale. Its output is a list of found failures, and its value is in what it finds, not in a number.
10. The structural weakness of red-teaming is that it can only ever demonstrate presence, never absence. Not finding a jailbreak is not evidence that none exists.

TECHNICAL50.7.5 the engineer’s version#

1. Modern arenas do not run a live Elo update loop, because Elo is order-dependent: the same set of games in a different order gives different ratings. They fit a Bradley-Terry model by maximum likelihood over the whole vote history, then bootstrap for confidence intervals, then present the result on an Elo-like scale.
2. That is why you should read the interval, not the rank. Ranks on such boards are usually published as a range, precisely because ties are common.
3. Style-controlled ranking works by fitting the preference model with additional covariates for response length and markdown features, so that the reported strength coefficient is adjusted for those factors.
4. Naming instability is real here too. The board widely known as LMSYS Chatbot Arena became LMArena; as of August 2026 the domain lmarena.ai redirects to arena.ai. Cite what a leaderboard is, not just where it lives.
5. Evaluation methods against what they are good and bad at.

6. Cost figures matter and are rarely mentioned. A full SWE-bench Verified run with an agentic scaffold means 500 tasks, each with many model calls and repository builds. Runs costing hundreds to thousands of dollars and taking many hours are normal, which is exactly why the 300-item Lite subset exists.
7. This is also why the ARC Prize leaderboard reports cost per task and by default hides systems whose run cost exceeded 10,000 dollars. Any score can be bought with enough sampling; a score with a cost attached cannot.
8. Established fact: length and position biases in preference judging, and the order dependence of online Elo. Active research: how to control for style without controlling away real quality, and how to weight votes by voter expertise. Marketing claim: “ranked number one” when the interval overlaps five other entries.

WORDS50.7.6 remember these#

1. Preference arena — people vote between two anonymous answers — a pairwise human preference collection platform with a fitted rating model.
2. Elo — a chess rating scale reused for models — a rating system where the expected score is a logistic function of the rating difference over 400.
3. Bradley-Terry — the proper way to fit ratings from all votes at once — a maximum-likelihood pairwise comparison model, order independent.
4. Confidence interval — how much the rating could be wrong — the bootstrapped range around a fitted rating; overlapping intervals mean a tie.
5. Length bias — longer answers win — a systematic preference for verbose responses independent of content quality.
6. Style control — adjusting for formatting and length — fitting the preference model with covariates for length and markdown structure.
7. Red-teaming — attacking your own model on purpose — adversarial testing to discover failures; evidences presence, never absence.
8. Held-out evaluation — a private test set — evaluation items never published, used to detect overfitting to public benchmarks.

50.8 What “eval” means inside a company#

PLAIN50.8.1 in simple words#

1. In public, “eval” means a famous benchmark with a leaderboard.
2. Inside a company that builds a product on a model, it means something much smaller and much more useful.
3. It means a file of examples of the job the product actually does, with the answer you want for each.
4. Thirty to a hundred examples is normal. Not fourteen thousand.
5. They are hand-picked, mostly from real cases that went wrong.
6. They are private, because they contain real customer text and because they are the company’s actual measuring stick.
7. Every time anyone changes the prompt, the model, the retrieval setup or a parameter, the file is run again.
8. If the score drops, the change is rejected. That is the whole discipline.
9. In other words, it is a regression test suite, exactly like the automated tests you would write for ordinary software.
10. The score is not comparable to anyone else’s score, and does not need to be.
11. It only needs to answer one question: did this change make my product better or worse at my job?
12. Nobody can build this for you, because only you know what your job is.
13. If you build software on a model and you do not have one of these, you are guessing every time you change anything.

PLAIN50.8.2 a picture in your head#

1. A bakery buys flour. The supplier changes.
2. The baker does not read the flour industry’s national quality rankings.
3. The baker bakes their own three standard loaves with the new flour and compares them to the old ones.
4. Same recipe, same oven, same timings. Only the flour changed.
5. If the loaves are worse, the flour is rejected, whatever the rankings said.
6. The bakery’s test is small, private, unscientific and completely decisive, because it measures the thing the bakery sells.
7. That is an internal eval.

Where this comparison breaks: bread quality is judged by a person who has baked for twenty years, in one taste. Model output quality often needs a written rule, because you will run the test hundreds of times and cannot taste every loaf yourself. Writing that rule down is the hard part of the job.

PLAIN50.8.3 a worked example#

1. A practical recipe for building one. Follow it in order.
2. Step 1. Write down the job in one sentence. “Turn a customer email into a support ticket with a category, a priority and a two-line summary.”
3. Step 2. Collect 30 to 100 real inputs. Not invented ones. Real.
4. Step 3. Make sure the mix is deliberate, not random.

5. Step 4. For each input, write the answer you want. Where several answers are acceptable, write the rule instead of a single answer.
6. Step 5. Choose a scoring rule per item, from the cheapest kind that works.

exact       category must equal "billing"
contains    summary must mention the order number
regex       priority must match ^(P1|P2|P3)$
schema      output must parse as valid JSON
assert      code must pass these three tests
judge       a model grades 1-5 against a rubric
human       a person grades, for a 10-item subset

7. Step 6. Store it as a plain file. One row per example. Input, expected, rule, and a note about why it is in the set.
8. Step 7. Write a script that runs all items and prints one number plus the list of failures. The list of failures is the part you will actually read.
9. Step 8. Run it three times and note the spread before you trust any change smaller than that spread.
10. Step 9. Keep it in version control next to the code, and add a new row every time a real failure reaches a user. The set grows by itself.

PLAIN50.8.4 what is really happening inside#

1. Why 30 to 100 and not 10 or 10,000.
2. Below about 30, one item is more than 3 per cent, and the whole set can be passed by luck or by a prompt change that happens to suit those items.
3. Above a few hundred, hand-writing expected answers becomes a project, and people stop maintaining it. An unmaintained eval set is worse than none, because it produces confident wrong signals.
4. The trade-off is honest: 30 to 100 items cannot tell you a 2 per cent difference. It can reliably tell you a 15 per cent difference, which is the size of change that usually matters when swapping models or prompts.
5. Where model-as-judge belongs. Use it when the output is free text and no mechanical rule works, but pin it down: a fixed rubric with numbered criteria, a fixed judge model with a fixed dated identifier, and a fixed temperature.
6. And validate the judge once: have a person grade 20 items, compare with the judge, and record the agreement rate. If the judge agrees with your human less than about 80 per cent of the time, fix the rubric before trusting it.
7. Also record cost and latency per item on every run. A change that improves quality by 3 per cent and triples cost is a decision, not an improvement, and you cannot make that decision without both numbers.

TECHNICAL50.8.5 the engineer’s version#

1. Treat the eval set as a first-class artifact: versioned in git, with a schema, a fixed identifier per item, and a changelog. Never edit an item silently; add a new one and deprecate the old one, or every historical number becomes uninterpretable.
2. A workable row schema:

id            stable, never reused
input         verbatim, including whitespace
expected      answer or acceptance rule
rule          exact | contains | regex | schema
              | assert | judge | human
slice         typical | hard | edge | refusal
added_on      date
reason        one line, why this exists

3. Report three numbers, not one: pass rate, mean latency, mean cost per item. Report pass rate per slice as well as overall, because a model that fixes typical cases and breaks refusals is a regression, not a gain.
4. Guard against a subtle trap: if you iterate on prompts against the eval set many times, you overfit to it exactly as labs overfit to public benchmarks. Hold back 20 per cent of items as a set you run only at decision time.
5. Available tooling as of 2026 includes provider-hosted eval products, open frameworks such as OpenAI’s evals repository, promptfoo, deepeval, Inspect from the UK government’s AI institute (named the AI Safety Institute at launch in 2024 and renamed the AI Security Institute in February 2025), and LangSmith-style tracing platforms. Note that provider-hosted eval products get deprecated too: as of mid-2026 OpenAI had announced shutdown dates for several platform features. Prefer a format you own.
6. Statistical note. With 50 items and an observed pass rate of 0.80, one standard error is sqrt(0.8 * 0.2 / 50) = 0.0566, about 5.7 points. So a move from 80 to 84 per cent on 50 items is well inside noise. Do not ship on it.
7. Established fact: internal regression evals are standard practice at companies shipping model-backed products. Marketing claim: any vendor assertion that their public benchmark results predict your task performance.

WORDS50.8.6 remember these#

1. Internal eval — your own small private test — a curated regression suite of task-representative items with per-item acceptance rules.
2. Regression test — checking a change did not break things — re-running a fixed suite after every modification and comparing to the last result.
3. Slice — a named group of items — a labelled subset such as typical, hard, edge or refusal, reported separately.
4. Rubric — the written marking scheme for a judge — the numbered criteria a judge model or human applies to score free text.
5. Golden set — the agreed correct answers — the reference outputs, versioned and never silently edited.
6. Holdout — items you do not tune against — the reserved fraction run only at decision time to detect overfitting to your own set.
7. Acceptance rule — how an item is marked — the per-item comparison, from exact match through to human judgement.

50.9 Reading a model announcement critically#

PLAIN50.9.1 in simple words#

1. A model announcement is a marketing document that contains real data.
2. Both halves of that sentence are true and you need both.
3. The data is usually accurate as far as it goes. The selection is not neutral.
4. So read it the way you read a job application: assume everything stated is true, and pay most attention to what is missing.
5. Nine questions do almost all the work, and section 50.9.3 lists them.
6. If the announcement answers all nine, that is a good sign about the team, independent of the scores.

PLAIN50.9.2 a picture in your head#

1. Think of a used-car advertisement listing the good features honestly.
2. New tyres, full service history, one owner. All true.
3. It does not mention the mileage, and you notice that it does not.
4. You do not conclude the car is bad. You conclude you must ask.
5. The absence of a number is itself information, because the seller chose the list.

Where this comparison breaks: a car has one mileage figure and the seller knows it. A model has no single benchmark score to withhold, because the score depends on the protocol. So the missing item is usually the protocol, not a number, and the protocol is much easier to omit without anyone noticing.

PLAIN50.9.3 a worked example#

1. The nine questions, in order, applied to any announcement.
2. Which benchmarks are shown, and which of the usual ones are absent? An absent standard benchmark is the loudest signal in the document.
3. Which comparison models, at which exact versions and dates? “GPT-4” is not a version. “Claude” is not a version.
4. Who produced the competitor numbers: this lab, or the competitor’s own published figures, or an independent runner?
5. What was the prompting setup: shots, chain-of-thought, system prompt?
6. Did the model have tools? Search, code execution, retrieval, file access? And did the comparison models have the same ones?
7. Is the score pass@1 and a single run, or is it best-of-n, majority voting, or an average over many samples?
8. Does the chart’s axis start at zero, and is the gap bigger than the benchmark’s known prompt sensitivity?
9. Is the benchmark new, and who built or funded it?
10. Has an independent third party reproduced any of it, and when?

PLAIN50.9.4 what is really happening inside#

1. Two phrasings that should always slow you down.
2. “State of the art on X” with no date. State of the art is a claim about a moment, and the moment is usually the week before publication.
3. “Comparable to human experts”. Ask which humans, doing what, under what time limit, and with what resources. GPQA is the model case: its expert baseline was 65 per cent, measured under stated conditions, and quoting a model above that as “beyond PhD level” ignores that the humans were answering a four-option quiz outside a lab.
4. Also watch for the shift between a model claim and a system claim. A product with retrieval, tools and a scaffold is a system. Its score belongs to the system.
5. And watch the switch of subset: a headline referencing SWE-bench and a footnote saying Verified, or GPQA in the headline and Diamond in the footnote, are not errors, but they change what is being compared.

TECHNICAL50.9.5 the engineer’s version#

1. Where the answers usually live, in decreasing order of reliability: the system card or technical report appendix, the model card, the announcement footnotes, the blog post body, the chart itself.
2. Reading order should therefore be reversed from the page order: appendix first, chart last.
3. Independent reproduction sources worth checking before believing a headline: the benchmark’s own leaderboard, third-party evaluation organizations that run their own harnesses and publish protocols, and the open harness community’s issue trackers, where discrepancies get argued out publicly.
4. A note on speed. Reproductions typically appear days to weeks after a release. The gap between announcement and reproduction is precisely the window in which the announcement is unchecked.
5. Established fact: benchmark numbers in major technical reports are generally reproducible within a few points when the protocol is published. Marketing claim: leaderboard positions asserted from gaps inside the reproduction spread.

WORDS50.9.6 remember these#

1. Model card — a short standard description of a model — a document stating intended use, training data summary, evaluations and limitations.
2. System card — the same for a whole product — documentation covering the model plus scaffolding, tools, safety mitigations and evaluations.
3. Self-reported — measured by the maker — a result produced by the provider, pending independent reproduction.
4. Reproduction — someone else got the same number — an independent run under a published protocol.
5. State of the art — best known result so far — a time-indexed claim, valid only at a stated date on a stated protocol.
6. System versus model — the product versus the weights — the distinction between an end-to-end deployed pipeline and the underlying network.

50.10 Training cutoff versus release date#

PLAIN50.10.1 in simple words#

1. There are two dates and people constantly confuse them.
2. The training data cutoff is the last date of text the model learned from.
3. The release date is the day you could first use it.
4. The gap between them is typically several months, and has often been a year or more.
5. During that gap the world kept happening and the model did not see any of it.
6. So a model released in one year can be confidently describing the world of a year or more earlier.
7. It does not know it is out of date. It has no clock and no calendar.
8. Unless you tell it today’s date, it does not know today’s date.
9. And it is often wrong about its own cutoff, which surprises people most.

PLAIN50.10.2 a picture in your head#

1. Imagine someone who read every newspaper up to a certain Tuesday, then went into a sealed room, and is now answering your questions.
2. They are extremely well read up to that Tuesday.
3. They do not know today is a different day. Nobody told them.
4. Ask what happened last week and they will describe the last week they remember, with total confidence.
5. Worse, they read fewer papers in the final month than in earlier months, because the papers about that month had not been written yet.
6. So their sense of “recent” lands earlier than the actual Tuesday.

Where this comparison breaks: a person in a sealed room knows they are in a sealed room, and will say so. A model has no representation of its own situation, so it cannot volunteer the warning. That is the whole practical danger of the gap.

PLAIN50.10.3 a worked example#

1. Real published gaps between stated cutoff and release.

2. Read the first row. GPT-4 shipped about 18 months after its data ended.
3. Read rows three and four. Two models released on the same day, from the same family, with cutoffs nine months apart.
4. That alone should end any belief that a family name implies a shared cutoff.
5. Since about 2024 the gaps have narrowed, often to a few months, because pipelines got faster and freshness became a selling point. But the gap has never been zero and cannot be.

PLAIN50.10.4 what is really happening inside#

1. Why the gap exists at all, in order.

data collection and filtering      weeks to months
pretraining run                    weeks to months
post-training: SFT, RL, tuning     weeks to months
internal + external evaluation     weeks
safety testing and red-teaming     weeks to months
staged rollout and capacity build  weeks

2. Every stage needs the previous one finished, so the delays add rather than overlap.
3. Now why models are unreliable about their own cutoff. Three separate causes.
4. First, the cutoff is not stored anywhere in the weights. If the model states one, it is because someone wrote it into the post-training data or the system prompt, or because the model is guessing.
5. Second, the internet writes about events with a lag. An event in the final two months before cutoff has far fewer documents about it than an event two years earlier. So the model’s felt sense of “the present” sits earlier than its real cutoff. Researchers call this the effective cutoff, and it is commonly earlier than the stated one.
6. Third, training corpora contain many documents about earlier models with earlier cutoffs. A model asked its own cutoff can echo a date it read.
7. The practical rule that follows: never ask a model for its cutoff and act on the answer. Look it up in the provider’s documentation.
8. And always put the current date in the system prompt if the date matters.

TECHNICAL50.10.5 the engineer’s version#

1. Terminology precision. “Knowledge cutoff” usually refers to the pretraining corpus cutoff. Post-training data is frequently newer, and retrieval or tool results are newer still. A deployed system can therefore have three different effective dates at once.
2. The paper “Dated Data: Tracing Knowledge Cutoffs in Large Language Models”, published in 2024, documented that measured effective cutoffs for many open models are earlier than the reported ones, and attributed this partly to corpus construction: deduplication and re-crawling mean older versions of pages persist while newer content is thinner.
3. Mitigations, in the order you should reach for them:

4. Established fact: the cutoff-to-release gap, and its documented sizes. Established fact: models frequently misreport their own cutoff. Active research: measuring effective cutoffs per topic. Marketing claim: “always up to date”, which describes a retrieval pipeline, not a model.
5. Practical detection test you can run in one minute: ask the model about three events with known dates spread across the suspected boundary, and see where its knowledge stops. That measures the effective cutoff for that topic, which is what you actually care about.

WORDS50.10.6 remember these#

1. Training cutoff — the last date of learned text — the corpus collection boundary for pretraining data.
2. Effective cutoff — where the knowledge really stops — the empirically measured boundary, usually earlier than the stated cutoff.
3. Release date — when you could first use it — general availability date of a specific model version.
4. Staleness — the model living in the past — degradation of factual accuracy for events after the cutoff.
5. Retrieval augmentation — giving it fresh documents at question time — supplying externally fetched context so the answer is not limited to weights.
6. Post-training — the tuning after the big training run — supervised fine-tuning and reinforcement learning stages, often using newer data than pretraining.

50.11 Versions, snapshots and the moving-name problem#

PLAIN50.11.1 in simple words#

1. A product name and a model identifier are two different things.
2. The product name is what the marketing says. It is a brand.
3. The model identifier is the exact string your program sends to the API.
4. A product name can point at different weights over time, without notice.
5. A dated identifier points at one fixed set of weights that never changes.
6. So the same product name can behave differently next month, and your carefully tuned prompt can quietly stop working.
7. Nobody broke anything on purpose. The name moved.
8. The rule that follows is short: in production, use a dated identifier, and re-run your own eval set whenever you change it.

PLAIN50.11.2 a picture in your head#

1. Think of a coffee shop’s “house blend”.
2. It is on the menu every year under the same name.
3. The beans change with the harvest, the roast is adjusted, the supplier changes. The name does not.
4. If you love it, you cannot order last year’s. It no longer exists.
5. A dated identifier is like buying a specific labelled lot, roasted on a stated day, from a stated farm.
6. You can reorder exactly that, until the shop stops stocking it.

Where this comparison breaks: coffee changes gradually and you taste it immediately. A moving model name can change one narrow behaviour, such as how it formats JSON or when it refuses, while everything else stays identical. You will not notice by tasting. You notice because a downstream parser starts failing at two in the morning.

PLAIN50.11.3 a worked example#

1. Real naming conventions from the three largest providers, as observed in August 2026. These change, so treat the shapes as the lesson and the examples as dated illustrations.

2. Read the OpenAI row. gpt-5-2025-08-07 is a snapshot: those exact weights, fixed. gpt-5 on its own is an alias that may be repointed.
3. Read the Anthropic rows. Some identifiers carry an eight-digit date, some do not. The dated ones are the ones to pin.
4. Google’s documentation as of August 2026 describes four channels: stable, preview, latest and experimental. The latest alias is explicitly defined as pointing at the newest release for a model variation, with a stated notice period of about two weeks before a breaking change.
5. That is the clearest possible statement of the trade-off. An alias gives you automatic upgrades and about two weeks of warning. A pinned snapshot gives you stability and the obligation to migrate yourself.
6. Note also the older Google style using numeric suffixes such as -001 and -002, where the trailing number is the revision of the same model generation. Different shape, same purpose.

PLAIN50.11.4 what is really happening inside#

1. Why aliases exist at all, honestly stated from both sides.
2. From the provider’s side: most users want improvements without doing work, safety fixes must be able to ship immediately, and serving many pinned versions costs real hardware.
3. From your side: any change to weights is a change to behaviour, and you have tested against the old behaviour.
4. Those two positions cannot both be fully satisfied, which is why both options exist.
5. What actually changes when a name is repointed: refusal boundaries, verbosity and formatting habits, tool-calling and JSON reliability, tokenizer or context limits in larger changes, and latency and price.
6. What almost never changes silently: the API shape. Providers version that separately, precisely because it would break everyone at once.
7. The distinction is worth naming. The API contract is close to a standard, documented and versioned. Model behaviour under an alias is a convention, and one that assumes you will cope.
8. There is also a middle case people forget: a provider may change the serving stack, quantization or routing behind a fixed identifier. The weights are the same; the outputs may not be bit-identical. Ask about this if determinism matters to you.

TECHNICAL50.11.5 the engineer’s version#

1. The production rule, in three lines.

1. Pin the dated snapshot in config, never an alias.
2. Store the exact id with every logged output.
3. Re-run your eval set before changing the pin.

2. Logging the identifier with each output is the step people skip and later regret. Without it, you cannot answer “did quality drop when we moved?” because you cannot tell which requests used which weights.
3. Keep the identifier in configuration, not in code, so a migration is a config change plus an eval run, not a code release.
4. Where aliases are still right: prototypes, internal tools, and anything where a two-week notice period is acceptable and nobody has built a parser around the exact output shape.
5. Where snapshots are essential: anything with a downstream schema, anything regulated, anything with a tuned prompt, and anything you cannot re-test quickly.
6. Watch for parameter-level changes too, not just weight changes. As one 2026 example, Anthropic’s documentation lists temperature, top_p and top_k as deprecated for its newest models, with non-default values returning an error. Code that hard-codes a temperature will fail on migration even though the API version did not change.
7. Established fact: aliases are repointed and dated snapshots exist at all three major providers. Marketing claim: “drop-in upgrade”. Every upgrade is a behaviour change until your own eval set says otherwise.

WORDS50.11.6 remember these#

1. Product name — the brand on the website — a marketing label that may map to different artifacts over time.
2. Model identifier — the exact string you send — the API-level model name that selects a served artifact.
3. Snapshot — a frozen dated version — an immutable identifier bound to one set of weights and serving configuration.
4. Alias — a name that follows the newest release — a mutable pointer, such as a bare family name or a latest channel.
5. Pinning — fixing the version in config — specifying an immutable snapshot identifier so behaviour does not change under you.
6. Channel — the stability tier — stable, preview, latest or experimental, each with different change and notice policies.

50.12 Deprecation and retirement#

PLAIN50.12.1 in simple words#

1. Every model you build on will eventually be switched off.
2. Not “might”. Will. Plan for it from the first day.
3. There are three stages and they mean different things.
4. Legacy means still working, no longer recommended, no further updates.
5. Deprecated means a shutdown date has been announced.
6. Retired means the identifier no longer works and your requests fail.
7. Providers give notice, but the notice period varies a great deal by provider and by model tier.
8. The single thing that makes a forced migration safe is your own eval set, because it is the only way to know whether the replacement is good enough for your job.

PLAIN50.12.2 a picture in your head#

1. Think of a bus route being withdrawn.
2. First, notices appear at the stops. That is deprecation.
3. Then a date is given, and an alternative route is suggested.
4. On the date, the bus stops coming. That is retirement.
5. If you never tried the alternative route before the date, you find out on the morning you needed it whether it gets you to work on time.
6. Trying the alternative early, with your actual journey, is the whole of migration strategy.

Where this comparison breaks: a bus route is either running or not, and you find out at the stop. A retired model identifier can fail in a way your code does not handle, at the worst possible moment, in production, returning an error your retry logic loops on. Test the failure path too.

PLAIN50.12.3 a worked example#

1. Real, checkable policies and dates, as published in August 2026.

2. And real retirement events, which show the policies being applied.

3. Read the third row against the first table. That snapshot was released in August 2025 and retired in August 2026. One year of life.
4. Read the fourth row. Announced on 11 June 2026 for shutdown on 11 December 2026: exactly the stated six months.
5. That is the realistic planning number. A pinned snapshot at a major provider has historically lasted roughly one to two years, with a few months of warning at the end.
6. Note also that whole product features get retired, not only models. In 2026 OpenAI announced shutdown dates for several platform features including its hosted evaluation product. Do not build your only eval harness inside a vendor’s product.

PLAIN50.12.4 what is really happening inside#

1. Why retirement happens: serving old weights occupies accelerators that could run new ones, old models lack current safety mitigations, and every supported version multiplies the testing burden.
2. Where to find the schedule: every major provider publishes a deprecations page listing model identifier, deprecation date, retirement date and the recommended replacement. Read it quarterly. Subscribe to the changelog.
3. The migration procedure, in order.

1. Read the deprecation notice; note both dates.
2. Identify the suggested replacement id.
3. Run your eval set on old and new, same prompts.
4. Read the failure list, not just the score.
5. Fix prompts for the new model's habits.
6. Re-run. Compare cost and latency too.
7. Ship behind a flag; keep the old id available.
8. Watch production metrics; then remove the flag.
9. Delete the old id from config before the date.

4. Step 3 is the one that makes the rest possible. Without an eval set, steps 4 to 8 are guesswork and step 9 is a leap of faith.
5. Expect prompt changes to be needed. A newer model in the same family often needs a shorter prompt, because instructions written to work around an older model’s weaknesses can actively hurt a newer one.
6. Budget the work. For a small production system with a good eval set, a migration is typically days. Without one, it is weeks and the outcome is unknown.

TECHNICAL50.12.5 the engineer’s version#

1. Operational controls worth having in place before you need them: the model identifier in configuration, a per-request log of the identifier used, a feature flag to switch identifiers without deploying, and an alert on the specific API error returned for a retired model.
2. Consider a fallback chain in code: primary snapshot, then a named replacement, then a degraded path. Test the fallback deliberately, because an untested fallback is not a fallback.
3. If continuity matters more than anything else, the alternative is running open-weight models yourself. Then nobody can retire your model, and the cost is that you own the serving, the security patching and the hardware. That is a genuine trade, not a free option.
4. Established fact: the notice periods and retirement dates above, as published in August 2026. They are policies, not contracts, and they change; check the current pages rather than trusting this table.
5. The honest version of “the provider will look after you”: the provider will give you a date and a suggested replacement. Whether the replacement does your job is entirely your problem, and only your eval set can answer it.

WORDS50.12.6 remember these#

1. Deprecated — announced for removal — still functional, with a published retirement date and a recommended replacement.
2. Retired — switched off — the identifier no longer serves requests and calls return an error.
3. Legacy — still working, no longer recommended — a supported but frozen version receiving no updates.
4. Notice period — how long you get to move — the interval between deprecation announcement and retirement.
5. Migration — moving to the replacement — re-pinning the identifier and re-validating behaviour against your own evaluation set.
6. Fallback chain — what to use when the first choice fails — an ordered list of model identifiers with a degraded final option.

50.13 What you should actually do#

PLAIN50.13.1 in simple words#

1. Here is the whole procedure, in order, for choosing a model for a job.
2. One. Write down the job in one sentence, and what a good answer looks like.
3. Two. Build the eval set from section 50.8. Thirty to a hundred real examples with a marking rule.
4. Three. Shortlist two or three candidate models. Use public benchmarks only for this step, only to shortlist, and never to decide.
5. Four. Run your eval set on all of them, with identical prompts.
6. Five. Read every failure. The failures tell you more than the score.
7. Six. Measure cost and latency at the same time, on the same runs.
8. Seven. Decide using all three numbers together, not quality alone.
9. Eight. Pin the dated identifier you chose.
10. Nine. Put the failures you saw into the eval set as new items.
11. Ten. Re-check every three months, and whenever a deprecation notice arrives.
12. That is it. It is not sophisticated. It is just done consistently.

PLAIN50.13.2 a picture in your head#

1. This is how you would hire someone, and you already know how to do that.
2. You do not hire from a league table of universities.
3. You shortlist from credentials, then you set a work sample task that resembles the actual job.
4. You watch how they handle the hard parts, not just whether they finish.
5. You consider salary and availability alongside skill.
6. And you review after a few months, because circumstances change.

Where this comparison breaks: a person learns your job over time and asks questions when confused. A pinned model snapshot is exactly as good on day 300 as on day 1, and never tells you it is uncertain unless you build that in.

PLAIN50.13.3 a worked example#

1. What to measure, and what a reasonable target looks like.

2. A worked cost calculation, using round illustrative prices rather than any provider’s current ones, since prices change monthly.
3. Suppose a task uses 4,000 input tokens and 800 output tokens, and the price is 3 dollars per million input and 15 dollars per million output.
4. Input cost: 4,000 / 1,000,000 * 3 = 0.012 dollars.
5. Output cost: 800 / 1,000,000 * 15 = 0.012 dollars.
6. Total 0.024 dollars per task. At 50,000 tasks a month that is 1,200 dollars.
7. Now suppose a model scoring 4 points higher costs three times as much. That is 3,600 dollars a month for 4 points on a 50-item eval set, where one standard error is about 5.7 points.
8. Written that way the decision is obvious, and it was invisible while you were looking at a leaderboard.

PLAIN50.13.4 what is really happening inside#

1. Why “try two or three” and not one or ten.
2. One tells you nothing comparative. Ten is a research project you will not finish, and the differences between the middle candidates will be inside your noise floor anyway.
3. Two or three, chosen to be genuinely different, is the sweet spot: for example one frontier model, one cheaper model from the same provider, and one from a different provider or an open-weight model you host.
4. Include a cheaper model deliberately. The most common real finding is that the cheap model is good enough for 80 per cent of traffic, and a routing rule sends the rest to the expensive one.
5. Why re-check periodically: models change under aliases, new versions appear, prices fall, and your traffic mix drifts as users learn what your product can do.
6. Why the failure list beats the score: a 90 per cent pass rate where the 10 per cent are harmless is fine, and a 95 per cent pass rate where the 5 per cent are confident wrong answers about money is not.

TECHNICAL50.13.5 the engineer’s version#

1. Measure latency properly and separately. Time to first token dominates perceived responsiveness in streaming interfaces; output tokens per second dominates total completion time for long outputs. Report both, at the median and at the 95th percentile, not as a mean.
2. Measure from where your users are. A model served in another region can add more latency in transit than the difference between two models.
3. Control the comparison. Same prompts, same temperature where settable, same context, same tools, same time of day. Provider load varies through the day and will otherwise contaminate your latency numbers.
4. Compute your noise floor first: run one model three times and record the spread. Any difference between models smaller than that spread is not a result.
5. Keep the artifacts: raw outputs, identifiers, timestamps and prices. When something changes in three months, the old raw outputs are the only way to prove what changed.
6. Established fact: the arithmetic and the method here. Active research: automated evaluation set generation and automatic prompt optimization, both of which are useful and neither of which removes the need for a human to define what a good answer is.

WORDS50.13.6 remember these#

1. Shortlist — the two or three you will actually test — candidate models chosen from public evidence before task-specific measurement.
2. Noise floor — the wobble between identical runs — the run-to-run standard deviation below which differences are meaningless.
3. Time to first token — how long before words appear — the latency from request to first streamed token, distinct from total completion time.
4. Unit economics — what one task costs — the token cost per completed task at your actual input and output lengths.
5. Routing — sending easy work to a cheap model — a policy that selects a model per request based on difficulty, cost or latency budget.
6. Regression — the new thing is worse at something — a drop on any slice of your evaluation set after a change.

50.98 Common wrong ideas#

1. Wrong: benchmark scores measure intelligence. Right: they measure performance on a fixed list of items under one protocol. Intelligence is not defined well enough to be scored, and no benchmark claims to.
2. Wrong: a higher MMLU means it is better for my task. Right: MMLU is four-option factual recall and is saturated. It predicts almost nothing about summarizing your emails or fixing your code. Only your own eval set answers that.
3. Wrong: the leaderboard is objective. Right: every leaderboard embeds choices about prompts, extraction, subsets and sampling, and preference leaderboards additionally embed the taste and the question mix of whoever voted.
4. Wrong: the model knows its own training cutoff. Right: the cutoff is not stored in the weights. Any date it gives is repeated from post-training data or guessed, and the felt cutoff is usually earlier than the real one.
5. Wrong: the same model name always means the same model. Right: undated product names are aliases and get repointed. Only a dated snapshot identifier is fixed, and even that is eventually retired.
6. Wrong: if two labs disagree, one is cheating. Right: prompt template, few-shot count, extraction, subset, sampling, system prompt and tool access routinely explain gaps of several points, with nobody at fault.
7. Wrong: contamination means the scores are fake. Right: contamination inflates scores by an unknown and uneven amount. Matched fresh benchmarks show drops of up to about 8 points for some model families and near zero for others.
8. Wrong: a 2-point lead is a meaningful lead. Right: on GPQA Diamond one standard error is about 2 points and one item is 0.5 points. On a single AIME paper, one question is 6.7 points.
9. Wrong: pass@k is just the score. Right: pass@k rises with k by construction. pass@1 and pass@100 are different quantities and cannot be compared.
10. Wrong: benchmarks are useless, so ignore them. Right: they are useful for shortlisting, for tracking the field over years, and for spotting obvious weakness. They are not useful for choosing between two close models for your specific job.

50.99 Chapter summary in 20 lines#

1. A benchmark is a fixed set of items with known answers plus a rule for scoring a response. Nothing more.
2. Every run has five parts: items, prompt template, generation settings, answer extraction, and aggregation.
3. Only the items are usually published, so a score belongs to a model plus a harness, not to a model.
4. MMLU (2020, 57 subjects, about 14,000 test items) and HellaSwag (2019) are saturated and now mostly sanity checks.
5. GSM8K (2021, 1,319 test items) is solved. MATH (2021), AIME and FrontierMath carry the maths signal now.
6. HumanEval (2021, 164 problems, pass@k) is tiny, isolated and leaked. MBPP (2021, 974 problems) has the same shape.
7. GPQA (2023, 448 items; Diamond 198) is hard four-option science recall, not the ability to do science.
8. SWE-bench (2023, 2,294 tasks; Verified 500) scores whether real repository tests pass, and measures the scaffold as much as the model.
9. ARC-AGI keeps a held-back set and reports cost per task, which more benchmarks should copy.
10. Few-shot examples mainly fix output format. Chain-of-thought uses the model’s own output as working memory and transforms maths scores.
11. pass@k = 1 - C(n-c,k)/C(n,k). With n=10 and c=2, pass@1 is 0.20 and pass@5 is 0.78. The model did not improve; you bought more tries.
12. Two labs disagree because of prompts, shots, extraction, subsets, sampling, system prompts, tools, or bugs. Ask which harness and version.
13. Contamination is near-inevitable when training on the web. Detection by n-gram overlap, canaries, perplexity, old-versus-new splits and completion tricks can prove presence but never absence.
14. Saturation, Goodhart’s law, self-reported numbers, cherry-picked baselines, best-of-n reporting, tool asymmetry and truncated axes are the seven ordinary ways a number misleads without anyone lying.
15. Preference arenas measure real user taste well and truth badly, with documented length and style bias; a 10-point gap is a 51 per cent win rate.
16. Inside a company, an eval is a private regression suite of 30 to 100 real examples with per-item marking rules, run on every change.
17. Training cutoff and release date are different, historically separated by months to over a year, and models are unreliable about their own cutoff.
18. Product names are mutable aliases; dated snapshots are fixed. Pin the snapshot in production and re-run your evals when you change it.
19. Every model is eventually retired. Typical notice in 2026 was at least six months at OpenAI, at least 60 days at Anthropic, about two weeks for preview channels.
20. Shortlist with public benchmarks, decide with your own eval set, measure cost and latency alongside quality, and re-check every few months.