How a Model Is Actually Made - Pretraining to Release

49.0 What this chapter gives you#

1. You will be able to name every stage that turns raw web text into a shipped chat model, in order, and say what each stage costs and what it changes.
2. You will be able to explain where training data comes from, how it is cleaned, and why the legal position around it is genuinely unsettled.
3. You will be able to describe one training step exactly: forward pass, loss, backward pass, optimizer update, and say what a good loss curve looks like.
4. You will be able to use the formula C is about 6ND to estimate the compute of any announced model, and check it against the figures the lab published.
5. You will be able to explain scaling laws, what Chinchilla changed in 2022, and why labs deliberately ignore Chinchilla for small models today.
6. You will be able to say why a base model cannot be used as a chatbot, and what supervised fine-tuning and preference optimization each add.
7. You will be able to explain RLHF, reward hacking, DPO, constitutional methods and verifiable-reward training, and when each is used.
8. You will be able to open a model directory and say what every file is, what format it is in, and which formats can execute code when loaded.
9. You will be able to state precisely what “open weights”, “open source”, “open data” and “permissive licence” each mean, and tell them apart.
10. You will be able to read a model card and a licence and say what the publisher withheld, and what that omission tells you.

49.1 The pipeline in one view#

PLAIN49.1.1 in simple words#

1. A chat model is not made in one step. It is made in a chain of stages.
2. Each stage takes the output of the one before and changes it.
3. Stage one is collecting text. Enormous amounts of text, mostly from the web.
4. Stage two is cleaning that text. Throwing away most of it, in fact.
5. Stage three is building the tokenizer, the fixed list of text pieces the model is allowed to see. Chapter 47 covered this in detail.
6. Stage four is pretraining. This is the long, expensive one. Weeks or months on thousands of machines, learning to predict the next piece of text.
7. What comes out of stage four is called a base model. It is not a chatbot. It continues text. It does not answer questions.
8. Stage five is post-training. This is where it learns to behave like an assistant: to answer, to follow instructions, to refuse harmful requests.
9. Post-training has parts of its own: showing it good examples, then teaching it which of two answers people prefer, then safety work.
10. Stage six is evaluation: measuring what the thing can and cannot do.
11. Stage seven is release: publishing the numbers, or putting them behind an interface, along with documents describing what was made.
12. The costs are wildly unequal. Pretraining is almost all of the money.
13. Post-training is a rounding error in compute, and yet it is what makes the difference between a strange text continuer and something you can talk to.
14. That imbalance is the single most useful fact in this chapter.

PLAIN49.1.2 a picture in your head#

1. Think of how a doctor is made.
2. First, years of general education. School, then a science degree. Broad, slow, expensive, and not aimed at any one job.
3. That is pretraining. Enormous input, no particular task, no bedside manner.
4. A person at the end of that stage knows a great deal of biology and can still be useless in a consulting room.
5. Then comes clinical training. Watching a senior doctor, copying how they greet a patient, how they explain bad news, when they say “I do not know”.
6. That is supervised fine-tuning: learning the form of the job by imitation.
7. Then comes years of feedback. A supervisor says “that explanation was better than the one you gave yesterday”. Slowly the style settles.
8. That is preference optimization: learning from comparisons, not from corrections.
9. Then exams and a licence before anyone is allowed to practise unsupervised.
10. That is evaluation and release.

Where this comparison breaks: a medical student keeps learning after graduation. A shipped model does not. Its numbers are frozen at release and never change again unless the lab trains a new version and publishes a new file. Also, a doctor’s general education takes years and their clinical training takes years. In a model, the general education is 99 per cent of the cost and the clinical training is about 1 per cent. The proportions are nothing like a human career.

PLAIN49.1.3 a worked example#

1. Here is the whole chain as a diagram. Follow the arrows left to right.

 [1] RAW TEXT              [2] CLEANING
 web crawls, books,   ->   language id, dedup,   ->  clean corpus
 code, papers              quality filter, PII       (trillions
 (petabytes)               removal, decontam.        of tokens)
                                  |
                                  v
 [3] TOKENIZER          [4] PRETRAINING        [5] BASE MODEL
 learn merges from  ->  next-token predict, ->  weights file.
 a text sample          weeks, 1000s of GPU     Continues text.
                                  |             Not a chatbot.
                                  v
 [6] SFT                [7] PREFERENCES        [8] SAFETY
 demonstrate good   ->  reward model or    ->  red-team, refusal
 answers, chat          DPO on human or        training, filters
 template, roles        AI comparisons
                                  |
                                  v
 [9] EVALUATION         [10] RELEASE
 benchmarks,        ->  model card, licence,
 red-teaming,           weights or API,
 contamination check    serving stack

2. Now the same chain as a table, with rough costs. These are typical orders of magnitude for a large 2024 to 2026 model, not exact figures for any one lab.

3. Note what that table says. Data cleaning is cheap in GPU terms and enormous in CPU and human terms. It runs on ordinary processors, not accelerators.
4. Note also that evaluation and red-teaming take weeks of calendar time while using almost no compute, because the bottleneck is people.

PLAIN49.1.4 what is really happening inside#

1. What changes at each stage is worth being precise about.
2. Data collection changes nothing in the model. There is no model yet.
3. Tokenizer training produces a small file, a few megabytes, that fixes the vocabulary for the life of the model. Change it later and everything breaks.
4. Pretraining sets every weight in the model, starting from random numbers. This is where essentially all of the model’s knowledge is installed.
5. Supervised fine-tuning changes every weight again, but only very slightly, using a dataset perhaps a millionth the size.
6. Preference optimization changes every weight again, again slightly.
7. Safety work is not a separate mechanism. It is more of the same fine-tuning, using data chosen to produce refusals and careful answers.
8. Evaluation changes nothing. It only measures.
9. So there is exactly one moment where the model gains its knowledge, and it is pretraining. Everything after that shapes how the knowledge comes out.
10. That sentence needs a caution, which section 49.8 gives properly: the boundary is real but blurrier than it sounds.

TECHNICAL49.1.5 the engineer’s version#

1. The canonical pipeline, with the standard names used in papers: corpus construction, tokenizer fitting, pretraining, mid-training or annealing, supervised fine-tuning, preference optimization, evaluation.
2. Mid-training is the newest of these names and dates from roughly 2024. It is a final pretraining phase on a much higher-quality mixture, with a decaying learning rate. Ai2’s Dolmino mixes and OLMo 2, published in January 2025, made the term common.
3. Compute is measured in floating-point operations, FLOPs, and in accelerator-hours. Both are reported. Neither alone tells you the cost.
4. Real reported compute for the pretraining stage of Llama 3.1, from Meta’s published model card of July 2024:

5. Those CO2 figures are location-based. Meta reported market-based emissions as zero, because it matches its electricity use with renewable purchases. Treat “zero” as an accounting statement, not a physical one.
6. Post-training compute is rarely published separately. Where it has been, it is well under 1 per cent of pretraining. InstructGPT, published by OpenAI in March 2022, reported its whole RLHF stage at about 60 petaflop-days against 3,640 petaflop-days for GPT-3 pretraining, roughly 1.6 per cent.
7. Established fact: pretraining dominates training cost. Active research: how much of final capability comes from post-training, especially for reasoning models where reinforcement learning budgets have grown sharply since late 2024. Marketing claim: any statement that a model was “built from scratch” when the lab in fact fine-tuned somebody else’s open-weight base model.

WORDS49.1.6 remember these#

1. Pretraining — the long stage where the model learns from raw text — training a randomly initialized network on a large corpus with a next-token objective.
2. Post-training — everything done after pretraining to make it usable — supervised fine-tuning, preference optimization and safety tuning.
3. Base model — the raw output of pretraining — a text continuer with no chat template, no role notion and no refusal behaviour.
4. Corpus — the pile of text used for training — the deduplicated, filtered token stream fed to the pretraining loop.
5. Mid-training — a high-quality final stretch of pretraining — an annealing phase on a curated mixture with a decaying learning rate.
6. FLOP — one arithmetic operation on numbers with a decimal point — one floating-point operation, the standard unit of training compute.

49.2 The data: where it comes from and how it is cleaned#

PLAIN49.2.1 in simple words#

1. The model needs text. An almost unimaginable amount of text.
2. The biggest source is the open web, scraped by crawlers that follow links and save pages.
3. The best-known scrape is Common Crawl, run by a non-profit of the same name, which has been crawling since 2008 and gives its archives away free.
4. Common Crawl alone is not enough and is not good enough. Most of it is navigation menus, adverts, spam and duplicated boilerplate.
5. So labs add other sources: curated web pages, books, code from public repositories, scientific papers, forum posts, subtitles, reference works.
6. Some labs pay for data. News archives, stock photo libraries, forum archives and publisher catalogues have all been licensed for money.
7. And increasingly, labs make text with other models. That is called synthetic data.
8. Then comes the part nobody outside the field expects: most of it is thrown away.
9. A realistic pipeline keeps somewhere between 5 and 20 per cent of what it started with, and sometimes far less.
10. Removing near-duplicate pages alone can cut a web crawl in half.
11. What survives is a stream of trillions of tokens, where a token is roughly three-quarters of an English word.
12. The last twenty years of computing said “more data is better”. This field learned, roughly between 2022 and 2024, that better data is better.

PLAIN49.2.2 a picture in your head#

1. Imagine you want to teach someone English by handing them paper.
2. You send a truck round a city and collect every piece of printed paper it can find. Newspapers, junk mail, receipts, packaging, books, love letters.
3. You get a warehouse full. Most of it is worthless for your purpose.
4. So you sort it. First you throw out everything not in English.
5. Then you notice that you have forty thousand copies of the same supermarket leaflet, so you keep one and burn the rest.
6. Then you throw out anything that is mostly numbers, menus or telephone listings, because reading it teaches nothing about language.
7. Then you black out every name, address and card number you find.
8. Then, before the exam, you check that the exam paper itself did not end up in the pile, because that would make the results meaningless.
9. What is left is a much smaller, much better warehouse.

Where this comparison breaks: paper sorting is done by people who understand what they are reading. Data cleaning is done by programs that mostly cannot. A quality filter is itself a small trained model making a guess, and it makes systematic mistakes: it tends to rate text that looks like a textbook highly and text written in dialect or by non-native speakers poorly. That bias goes straight into the finished model and nobody notices until much later.

PLAIN49.2.3 a worked example#

1. Here is a real published data mixture. It is from the GPT-3 paper by Brown and colleagues in 2020, and it is one of the last fully published mixtures from a frontier lab.

2. Read the two columns against each other. Wikipedia is 3 billion tokens out of 499 billion available, which is 0.6 per cent of the pile.
3. But it was given 3 per cent of the training weight. So during training, the model saw Wikipedia about 3.4 times over while seeing Common Crawl less than half way through.
4. That is what a mixing ratio does. It is a deliberate decision to over-sample good text and under-sample mediocre text.
5. Now a real published mixture from four years later. Meta’s Llama 3 paper of July 2024 gave the final proportions of its 15 trillion token mix:

6. Note that a quarter of the diet is maths and reasoning and a sixth is code, for a model most people use to write English prose.
7. That is not an accident. Code and maths are believed to improve general reasoning, and they are the two areas where correctness is checkable.
8. Note also what Meta did not publish: which datasets, which websites, which books, in what proportion, under what licence.

PLAIN49.2.4 what is really happening inside#

1. Here is the cleaning pipeline in the order it actually runs.
2. Extract. A crawl file holds raw HTML. A tool pulls out the article text and drops the menus, headers, adverts and cookie banners. The library most commonly named for this in open pipelines is Trafilatura.
3. URL filtering. Drop whole domains from block lists: adult sites, known spam farms, malware hosts. Cheap and effective, done before anything else.
4. Language identification. A small classifier guesses the language of each document and scores its confidence. Keep only what you want, above a threshold. This is where most of a global crawl disappears.
5. Exact deduplication. Hash each document. Identical hash, identical document, keep one.
6. Fuzzy deduplication. Much harder and much more important. Two pages can differ by a date stamp and be otherwise identical. Section 49.2.5 gives the method, called MinHash.
7. Quality filtering. Two kinds. Heuristics: rules like “drop documents with fewer than 50 words”, “drop documents where more than 30 per cent of lines are duplicates”, “drop documents with no full stops”. And classifiers: a small model trained to score whether text looks like the good stuff.
8. Toxicity and safety filtering. Classifiers remove the most extreme material. This is a judgement call with real costs both ways, and labs differ on where the line goes.
9. PII removal. Personally identifiable information. Email addresses, phone numbers, public IP addresses are detected by pattern and masked.
10. Decontamination. Search the corpus for text that appears in the evaluation benchmarks and remove it. Otherwise the model has seen the exam.
11. Mixing. Assign weights to each remaining source and sample accordingly.
12. Every one of those steps discards data. The compounding is severe. Ten steps each keeping 80 per cent leave you with 11 per cent.

TECHNICAL49.2.5 the engineer’s version#

1. Scale of the raw material. Common Crawl’s archive CC-MAIN-2025-30, crawled between 7 and 21 July 2025, holds 2.42 billion web pages, 419 TiB of uncompressed content, 114.15 TiB compressed. There are several such crawls per year and the organization has been publishing since 2008.
2. Formats: WARC holds the raw request and response, WAT holds metadata, WET holds extracted plain text. Serious pipelines re-extract from WARC rather than trusting WET.
3. Named public corpora, with real figures:

4. FineWeb’s published pipeline is the best documented open example. In order: URL filtering, Trafilatura extraction, a fastText language filter keeping English scores above 0.65, the Gopher repetition and quality filters, the C4 quality filters, custom heuristics, MinHash deduplication per crawl, and PII anonymization of emails and public IP addresses. Licence: ODC-By 1.0.
5. MinHash deduplication, properly. Cut each document into overlapping n-grams, called shingles. FineWeb uses 5-grams. Hash every shingle. For each of many hash permutations, keep only the minimum hash value. The resulting vector of minimums is the document’s signature.
6. The mathematical property that makes this work: the probability that two documents share a given minimum equals their Jaccard similarity, which is the size of the intersection of their shingle sets divided by the size of the union.
7. Signatures are then split into bands. FineWeb uses 14 bands of 8 hashes, so 112 hashes per document. Two documents are candidate duplicates if any one band matches exactly. This turns a quadratic all-pairs comparison into a hash-table lookup.
8. The detection curve for 14 bands of 8, computed from 1 minus (1 minus s to the eighth) to the fourteenth:

9. So this configuration is a deliberate threshold at roughly 0.75. Change the bands and rows and you move the knee of that curve.
10. Decontamination is done by n-gram overlap against benchmark sets, often 13-gram or 8-gram matching. Ai2 published a dedicated tool called decon with OLMo 3 in November 2025. GPT-4’s technical report in March 2023 described substring-match contamination checks and reported that its results moved little when contaminated examples were removed.
11. Established fact: deduplication improves models. Lee and colleagues in 2021 showed deduplicated training data reduces memorized regurgitation by an order of magnitude and does not hurt, and often helps, perplexity.
12. Established fact: quality filtering beats raw volume at fixed compute. FineWeb-Edu, filtered by an educational-quality classifier, beat much larger unfiltered sets on knowledge benchmarks at the same token budget.
13. Now the honest part, on copyright. As of mid-2026 the legal position on training on copyrighted text is unsettled and actively litigated. Named cases you should know:

14. In Bartz v. Anthropic, Judge William Alsup ruled in June 2025 that training on lawfully bought and scanned books was fair use, but that keeping a library of pirated copies was not. Anthropic then agreed a settlement reported at 1.5 billion US dollars, roughly 3,000 dollars per work across about half a million works. Reported as the largest copyright settlement in United States history.
15. Read those two halves carefully, because they say different things. The training was found transformative. The acquisition was the problem.
16. In Thomson Reuters v. Ross Intelligence, Judge Stephanos Bibas held in February 2025 that the use was not fair. That case concerned a search tool rather than a generative model, so its reach is disputed.
17. The honest version: nothing here is settled. Different courts in different countries have reached different conclusions, appeals are pending, and the European Union, the United Kingdom and Japan have taken materially different statutory approaches to text and data mining.
18. Second honest point: most labs no longer disclose their data mixture. GPT-3 in 2020 published a table. GPT-4 in 2023 explicitly declined, citing competition and safety. Llama 3 in 2024 published category percentages but no source list. Gemini and Claude publish neither.
19. This is a genuine change in scientific norms and you should treat any claim about a closed model’s training data, including claims about what it was not trained on, as unverifiable.
20. Third honest point: data quality now matters more than quantity. The reason is that the supply of high-quality human text is finite. Villalobos and colleagues estimated the stock of public human text at roughly 300 trillion tokens and projected full use somewhere between 2026 and 2032. That projection is contested, but the direction is not.
21. Tools you will actually use: datatrove and dolma for pipelines, fastText for language identification, datasketch or a custom Spark job for MinHash, and the Common Crawl index for selecting URLs without downloading the lot.

WORDS49.2.6 remember these#

1. Common Crawl — a free archive of crawled web pages — a non-profit web corpus published since 2008 in WARC, WAT and WET formats.
2. Deduplication — removing repeated text — exact hashing plus fuzzy matching, usually MinHash with locality-sensitive hashing over shingles.
3. MinHash — a trick for spotting near-identical documents cheaply — a signature of minimum hash values whose collision rate equals Jaccard similarity.
4. Decontamination — checking the exam is not in the textbook — n-gram overlap removal of benchmark text from the pretraining corpus.
5. Mixing ratio — how much of each source the model sees — per-source sampling weights, expressed as epochs or proportions of the token budget.
6. Synthetic data — text written by another model — model-generated corpora used for pretraining, mid-training or instruction tuning.
7. PII — personal details about real people — personally identifiable information, detected by pattern and masked before training.

49.3 Pretraining itself#

PLAIN49.3.1 in simple words#

1. Pretraining has exactly one goal, and it is smaller than people expect.
2. Given some text, predict the next piece of text.
3. That is all. There is no second objective. Nobody labels anything. Nobody tells the model what is true.
4. The text itself is the answer key, because the next token is right there in the data.
5. This is why it is called self-supervised: the supervision comes free from the data, not from a human.
6. So the model reads “The capital of France is” and tries to guess the next token. It compares its guess to the real one, which is “Paris”.
7. If it guessed badly, every one of its billions of numbers gets nudged a tiny amount in the direction that would have made the guess better.
8. Then it does that again. And again. Trillions of times.
9. Nothing else happens. No rules are written down. No facts are stored as facts. Just this one loop, repeated at enormous scale.
10. And out of that loop comes something that can write working code and explain quantum mechanics. That is the surprising part, and it remains genuinely surprising to the people who build these things.

PLAIN49.3.2 a picture in your head#

1. Imagine a person locked in a room with a printing press feeding them paper.
2. The paper always arrives with the last word covered by a sticker.
3. Their only job is to guess the covered word, then peel the sticker and see.
4. They are not told the meaning of anything. They get no explanations. They are never asked a question. They just guess and check, forever.
5. At first they guess randomly. After a while they learn that “the” is common.
6. Later they learn that after “the capital of France is” the covered word is almost always “Paris”. They have not been taught geography. They have been taught which word comes next.
7. After enough of this, to get the guesses right, they are forced to build something inside their head that behaves a great deal like understanding.

Where this comparison breaks: a person would get bored, would sleep, and would form intentions. The training loop has none of that. Also, a person reads in order and remembers yesterday. The model sees documents in shuffled order, in fixed-length chunks, and carries nothing between chunks except the weights it has changed. There is no continuous experience of any kind.

PLAIN49.3.3 a worked example#

1. One training step, all the way through, with small round numbers.
2. Take a batch of 2,048 sequences, each 8,192 tokens long. That is 16,777,216 tokens in one step. Roughly 16.8 million.
3. Forward pass. Push all those tokens through the network. At every position the model produces a probability for every token in the vocabulary.
4. For a 128,256-token vocabulary that is 128,256 probabilities per position, which for this batch is about 2.15 trillion numbers produced and consumed.
5. Loss. At each position, look up the probability the model gave to the token that actually came next. Take the negative logarithm of it. Average over all 16.8 million positions.
6. That is cross-entropy loss, measured in nats per token.
7. Some real arithmetic on that. If the model gave the correct token a probability of 0.5, its contribution is 0.693. If 0.1, then 2.303. If 0.9, then 0.105. Confident and right is cheap. Confident and wrong is ruinous.
8. A model that guessed uniformly at random over 128,256 tokens would score the natural logarithm of 128,256, which is 11.76. That is the starting point.
9. Backward pass. Work out, for every one of the billions of parameters, which direction would have reduced that loss. This is backpropagation, and it costs roughly twice what the forward pass cost.
10. Optimizer update. Nudge every parameter a little in that direction. The size of the nudge is the learning rate, typically a few times ten to the minus five at the start of training.
11. Now discard the batch. It is never seen again. Load the next 16.8 million tokens and repeat.
12. To consume 15 trillion tokens at 16.8 million per step takes about 894,000 steps. Real runs are close to a million steps.

PLAIN49.3.4 what is really happening inside#

1. The loss curve is the one picture that tells you whether a run is healthy.

loss
(nats)
 12 |*
    | *
  8 |  *
    |   **
  4 |     ****
    |         *********
  2 |                  ****************
    |                                  *********
  1 +---------------------------------------------->
    0     1%      10%          50%          100%
                  fraction of tokens seen

2. What that shape means, in order.
3. The first drop, from about 11.8 to about 6, happens almost immediately. The model is learning which tokens are common. That is a few hundred steps.
4. The second phase, down to about 3, is learning basic word order and grammar. Still fast.
5. Then the long tail. From 3 down towards 1.5 takes 99 per cent of the run, and this is where everything interesting is learned.
6. A healthy curve is smooth, monotone downwards, and slightly noisy. Published final training losses for large models typically land somewhere between 1.4 and 2.0 nats per token, depending heavily on tokenizer and data mix.
7. An unhealthy curve has spikes: the loss jumps up sharply, sometimes by several nats, and either recovers over thousands of steps or never does.
8. A run that never recovers is said to have diverged. The loss goes to a large number or to not-a-number and the run is dead.
9. Causes: learning rate too high, a bad batch of data, numerical overflow in low precision, or a hardware fault corrupting a gradient.
10. The standard fix is brutal and simple: stop, load the last checkpoint, skip the batches around the spike, restart.
11. Checkpointing is what makes that possible. Every so often the whole training state is written to storage: weights, optimizer state, data position, random number generator state.
12. Note “optimizer state”. Adam keeps two extra numbers per parameter, so a training checkpoint is roughly three to four times the size of the final published model file.

TECHNICAL49.3.5 the engineer’s version#

1. Objective: minimize the negative log-likelihood of the next token under a causal, decoder-only transformer, averaged over positions. Formally the loss is minus the mean over t of log p of x sub t given x sub less-than-t.
2. This is exactly maximum likelihood estimation. There is no auxiliary loss in a standard dense run beyond, in mixture-of-experts models, a load-balancing term to stop all tokens routing to one expert.
3. Optimizer: AdamW, from Loshchilov and Hutter in 2017, is near universal. Typical betas 0.9 and 0.95, epsilon 1e-8, weight decay 0.1, gradient norm clipped at 1.0.
4. Real reported hyperparameters, Llama 3 405B, from Meta’s July 2024 paper: peak learning rate 8e-5, linear warmup over 8,000 steps, cosine decay to 8e-7 over 1,200,000 steps.
5. Batch size was ramped, not fixed: starting at 4 million tokens per batch with 4,096-token sequences, moving to 8 million tokens with 8,192-token sequences after 252 million tokens, and to 16 million tokens after 2.87 trillion tokens. Ramping the batch improves early-training stability.
6. Why warmup exists: at initialization the gradient estimates are poor and a full-size step can throw the weights somewhere unrecoverable. Warmup ramps the learning rate from near zero over the first few thousand steps.
7. Why decay exists: large steps explore, small steps settle. Cosine decay to about 1 per cent of peak is the long-standing default.
8. Changing as of 2024 to 2026: warmup-stable-decay schedules, which hold the learning rate constant for most of the run and decay sharply at the end. These allow a run to be extended without having chosen the end point in advance, which cosine does not. DeepSeek and several Chinese labs adopted multi-step or WSD schedules; the practice is now common but not universal.
9. A full set of real reported figures for one model, Llama 3.1 405B:

10. Derived figures from those, computed here rather than quoted: 30.84 million GPU-hours divided by 16,384 GPUs is 1,882 hours, which is 78.4 days of wall clock if every GPU ran the entire time.
11. Energy, GPU-only: 30.84 million hours times 700 watts is 21.6 GWh. Add host CPUs, memory, networking and cooling and 30 to 40 GWh is the plausible total, which is consistent with the 8,930 tonnes CO2eq Meta reported at typical grid carbon intensity.
12. Cost, estimated not quoted: at public H100 rental rates of roughly 2 to 4 US dollars per GPU-hour in 2024 and 2025, 30.84 million GPU-hours is between about 62 and 123 million dollars for the pretraining compute alone.
13. That figure excludes salaries, failed runs, ablations, data licensing and the capital cost of the cluster. Reported “training cost” numbers almost always exclude those, and you should assume so unless told otherwise.
14. The contrasting example, and the reason to read cost claims carefully: DeepSeek-V3, published December 2024, is 671 billion total parameters with 37 billion active per token, trained on 14.8 trillion tokens in 2.788 million H800 GPU-hours. The report itself puts that at 5.576 million US dollars assuming 2 dollars per GPU-hour, and states plainly that the figure excludes prior research, ablations and data.
15. Failures are routine, not exceptional. Meta reported 466 job interruptions during a 54-day snapshot of the 405B run, of which 419 were unexpected, and around 78 per cent of those were confirmed or suspected hardware issues. Effective training time was still above 90 per cent.
16. Historical precedent worth knowing: Meta published the OPT-175B logbook in 2022, a day-by-day record of crashes, loss divergences and restarts. It remains the most honest public document about what a large run feels like.
17. Google’s PaLM paper in 2022 reported roughly 20 loss spikes and described the mitigation: restart from a checkpoint about 100 steps before the spike and skip 200 to 500 data batches.
18. Tools that observe a run: TensorBoard or Weights and Biases for curves, nvidia-smi and DCGM for device health, torch.distributed logs and NCCL debug output for collective failures, and the model’s own gradient-norm and learning-rate traces, which are usually logged every step.

WORDS49.3.6 remember these#

1. Next-token prediction — guess the next piece of text — an autoregressive causal language modelling objective over a token vocabulary.
2. Cross-entropy loss — a score for how wrong a guess was — the negative log probability assigned to the observed token, averaged over positions.
3. Nat — the unit that loss is measured in — one unit of information using natural logarithms; multiply by 1.4427 to get bits.
4. Step — one round of guess, score, adjust — one forward pass, backward pass and optimizer update over one global batch.
5. Warmup — starting gently — ramping the learning rate from near zero over the first thousands of steps to avoid early instability.
6. Checkpoint — a saved snapshot to restart from — weights, optimizer moments, data-loader position and RNG state written to durable storage.
7. Divergence — a training run going permanently wrong — loss increasing without recovery, often to infinity or not-a-number.

49.4 Scaling laws: how big, how much data#

PLAIN49.4.1 in simple words#

1. Suppose you have a fixed budget of computing time. How should you spend it?
2. You have two dials. Make the model bigger, or feed it more text. You cannot max both, because both cost compute.
3. A scaling law is a measured answer to that question. Not a theory. A curve fitted to hundreds of small experiments and extended to large ones.
4. The first big one came from OpenAI in January 2020. Kaplan and colleagues measured how the loss falls as you increase model size, data and compute.
5. Their headline finding: the loss follows a smooth power law over more than seven orders of magnitude. It is astonishingly predictable.
6. Their practical advice: spend most of your extra budget on making the model bigger, and relatively little on more data. Stop training early.
7. The whole industry followed that advice. This is why 2020 to 2022 produced a race of enormous models trained on comparatively little text.
8. Then in March 2022, a team at DeepMind led by Hoffmann redid the experiment more carefully and found the advice was wrong.
9. Their model, Chinchilla, had 70 billion parameters and was trained on 1.4 trillion tokens. It beat Gopher, a 280-billion-parameter model trained on 300 billion tokens, using the same compute budget.
10. Four times smaller and better, because it saw four times more text.
11. Their rule of thumb: for each parameter, train on about 20 tokens.
12. That rule reshaped the field within about a year.
13. And then, from roughly 2023 onwards, labs started ignoring it on purpose.
14. The reason is money in a different place. Chinchilla asks how to get the best model for a fixed training budget. It says nothing about what the model costs to run afterwards.
15. If a model will answer a trillion questions, a smaller model that took longer to train is much cheaper over its life. So labs overtrain small models far past 20 tokens per parameter, on purpose.

PLAIN49.4.2 a picture in your head#

1. Think about choosing an engine and a fuel tank for a delivery van.
2. A bigger engine is like more parameters. It can do more per unit of time.
3. More fuel is like more training data. It lets you go further.
4. You have a fixed amount of money. Spend it on engine or on fuel.
5. Kaplan’s 2020 answer was: buy the biggest engine you can afford and put in just enough fuel to leave the yard.
6. Chinchilla’s 2022 answer was: that is silly, you have bought an engine you cannot use. Buy a smaller engine and fill the tank properly.
7. And the 2024 answer is: wait, this van will do school runs every day for five years. The fuel to build it is nothing next to the fuel to run it. Buy the smallest engine that does the job, and train it as long as you like.

Where this comparison breaks: an engine’s capability is fixed by its size and a model’s is not. Two models with the same parameter count can differ hugely depending on data quality, architecture and post-training. The van analogy also suggests a hard ceiling, and there is no known hard ceiling here; the curves have simply not bent yet within the range anyone has measured.

PLAIN49.4.3 a worked example#

1. The formula every practitioner uses for training compute is short: C is approximately 6 times N times D.
2. C is total floating-point operations. N is parameter count. D is number of training tokens.
3. Where the 6 comes from: each parameter does one multiply and one add per token in the forward pass, which is 2 operations. The backward pass costs about twice the forward pass, which is 4 more. Two plus four is six.
4. Work it for GPT-3, published 2020: 175 billion parameters, 300 billion tokens. 6 times 1.75e11 times 3.0e11 equals 3.15e23 FLOPs. The GPT-3 paper reported 3.14e23. The formula lands on the nose.
5. Work it for Llama 3.1 405B: 405 billion parameters, 15.6 trillion tokens. 6 times 4.05e11 times 1.56e13 equals 3.79e25 FLOPs. Meta reported 3.8e25. Again correct.
6. Now use it in the other direction. Suppose a lab announces a model with 70 billion parameters trained on 15 trillion tokens. 6 times 7e10 times 1.5e13 equals 6.3e24 FLOPs.
7. An H100 does roughly 989 teraflops of dense bf16 arithmetic at peak, which is 9.89e14 per second. Real utilization is 35 to 45 per cent, so call it 4e14 achieved.
8. 6.3e24 divided by 4e14 is 1.575e10 GPU-seconds, which is 4.375 million GPU-hours. On 4,096 GPUs that is 1,068 hours, or about 45 days.
9. You can now sanity-check any announcement in about a minute. If the numbers do not roughly agree, one of the published figures is wrong or incomplete.
10. Tokens-per-parameter for real models, which shows how far practice moved:

11. Read the last row again. An 8-billion-parameter model was trained on 15 trillion tokens. That is roughly 94 times more text than the compute-optimal rule says is worth it.
12. Meta did that knowing it was compute-inefficient, because the 8B model is the one that gets deployed everywhere and its serving cost matters more.

PLAIN49.4.4 what is really happening inside#

1. Why does a scaling law exist at all? Nobody fully knows. It is an empirical regularity, not a derived result.
2. What it says is that loss L falls as a power of the thing you scale. In words: doubling the model gives a fixed proportional reduction in loss, and that same proportional reduction keeps arriving for each further doubling.
3. Power laws are brutal in one direction and generous in the other. Each further step down in loss costs many times more compute than the last.
4. The Chinchilla correction was not a new idea, it was a better experiment. The 2020 work had used a fixed learning-rate schedule length for all runs, which quietly penalized the runs that used more data.
5. When Hoffmann’s team varied the schedule to match each token budget, the optimum moved decisively towards more data.
6. The number 20 is not sacred. It is roughly where the fitted curve puts the optimum for that family of models and that data. Different data gives a different constant.
7. And the whole framework optimizes the wrong thing for a deployed product. It minimizes loss for a fixed training budget. It ignores inference.
8. Add inference to the objective and the optimum moves towards smaller models trained longer, exactly as practice has done.

TECHNICAL49.4.5 the engineer’s version#

1. Kaplan and colleagues, “Scaling Laws for Neural Language Models”, 23 January
   2020. Loss scales as a power law in model size, dataset size and compute, with trends spanning over seven orders of magnitude. Their compute-optimal allocation grows N much faster than D, with D scaling roughly as C to the 0.27 and N as C to the 0.73.
2. Hoffmann and colleagues, “Training Compute-Optimal Large Language Models”, 29 March 2022, known as Chinchilla. Over 400 models from 70 million to 16 billion parameters on 5 to 500 billion tokens. Conclusion: N and D should scale roughly equally, so for every doubling of model size you should double the tokens. The implied ratio is about 20 tokens per parameter.
3. Chinchilla itself: 70 billion parameters, 1.4 trillion tokens. It beat Gopher at 280 billion parameters on 300 billion tokens, and also beat GPT-3 at 175 billion and Megatron-Turing NLG at 530 billion, at equal compute.
4. What changed in practice: the 2023 Llama 1 release from Meta was explicitly framed around this, training 7B and 13B models on 1 trillion tokens, well past compute-optimal, arguing that inference cost is what matters.
5. Contested, and you should know this: a 2024 replication attempt by Besiroglu and colleagues re-fitted the third of Chinchilla’s three estimation methods and found the reported parameters inconsistent with the paper’s own data, with wider confidence intervals than stated. The headline conclusion survived. The precise constant did not.
6. The inference-aware reformulation was written down by Sardana and colleagues in 2023, “Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws”. Adding expected lifetime inference tokens to the objective shifts the optimum to smaller, longer-trained models.
7. A cost model you can reason with: training costs 6ND once. Inference costs about 2N per generated token, forever. If a model will generate T tokens over its life, total compute is about 6ND plus 2NT.
8. Put numbers in it. An 8-billion model trained on 15 trillion tokens costs 7.2e23 FLOPs to train. Generating one token costs 1.6e10. So the training cost equals about 45 trillion generated tokens. A busy service passes that in a couple of months.
9. Regulatory tie-in, which makes the formula matter legally. The European Union’s AI Act, in force since 1 August 2024 with general-purpose model obligations applying from 2 August 2025, uses a training-compute threshold of 10 to the 25 FLOPs as a presumption of systemic risk. Llama 3.1 405B at 3.8e25 is above it. Llama 3.1 8B at 7.2e23 is below it.
10. Separating the three kinds of claim, as of mid-2026: Established fact: loss follows smooth power laws in N, D and C over the measured range; Chinchilla’s directional correction to Kaplan; the 6ND approximation; the dominance of inference cost for widely served models. Active research: whether scaling continues past the current frontier, how to model data quality inside a scaling law, scaling laws for mixture-of-experts and for test-time compute, and whether the supply of human text is a real ceiling. Marketing claim: any specific prediction of what a given future model will be able to do, derived from a loss curve. Loss is predictable. Which capabilities appear at which loss is not.
11. One more honest note. Since roughly September 2024 a second scaling axis has been in play: spending more compute at answer time rather than at training time. OpenAI’s o1 and DeepSeek-R1, released January 2025, both showed accuracy improving with more reasoning tokens per question. That changes the economics again, because it moves cost back to inference.

WORDS49.4.6 remember these#

1. Scaling law — a measured rule for how much better a bigger model gets — an empirical power-law fit of loss against parameters, tokens or compute.
2. Chinchilla-optimal — the balanced split of budget between size and data — roughly 20 training tokens per parameter, from Hoffmann and colleagues 2022.
3. Overtraining — deliberately feeding a small model far more text than is compute-optimal — trading training efficiency for lower inference cost.
4. 6ND — the arithmetic shortcut for training cost — approximately 6 times parameters times tokens, in floating-point operations.
5. Compute-optimal — best model for a fixed training budget — the point on the isoFLOP curve where loss is minimized, ignoring deployment cost.
6. Test-time compute — thinking longer at answer time — allocating additional inference tokens or samples to raise accuracy on a single query.

49.5 What a base model actually is#

PLAIN49.5.1 in simple words#

1. Pretraining finishes. You have a file of numbers. That is the base model.
2. It is very good at exactly one thing: continuing text.
3. It is not good at being talked to, because nobody taught it that.
4. Give it “The capital of Australia is” and it will finish the sentence correctly, because that is a continuation.
5. Give it “What is the capital of Australia?” and something strange happens.
6. On the internet, a line that looks like a quiz question is usually followed by more quiz questions. So the model writes more quiz questions.
7. It is not being difficult. It is doing its job perfectly. Its job is to guess what text usually follows, and more questions usually follow.
8. A base model has no idea what a conversation is. There is no “user” and no “assistant”. Those roles do not exist in its world.
9. It has no idea that it is supposed to stop. It will keep generating until you cut it off, including inventing both halves of a dialogue.
10. And it has no refusal behaviour at all. Ask it for something harmful and it will produce whatever text usually follows such a request.
11. That last point matters for the open-weights debate later in this chapter. Safety behaviour is added in post-training. It is not in the base model.
12. So the base model is a raw capability. Useful, powerful, and not a product.

PLAIN49.5.2 a picture in your head#

1. Imagine someone who has read every book in a vast library but has never had a conversation with anyone.
2. You walk up and say “Excuse me, where is the station?”
3. They do not answer. They continue: “…asked the traveller, setting down his case. The porter looked up from his newspaper and said nothing at all.”
4. They have turned your question into narrative, because in their experience that is what surrounds a sentence like that.
5. They are not stupid and not unhelpful. They have simply never learned that a question directed at them is a request for an answer.
6. Teaching them that takes surprisingly little. A few thousand examples of “someone asks, someone answers” and the behaviour appears.

Where this comparison breaks: a real person who had only read books would still know they exist and that someone is speaking to them. A base model has no self-model at all, and the instruction-tuned version does not really acquire one either. It acquires a text pattern in which a helpful assistant speaks. That is a different thing, and section 49.8 says why the difference matters.

PLAIN49.5.3 a worked example#

1. The same prompt, given to a base model and to the instruction-tuned model built from it. The base-model output below is illustrative: it is written to show the characteristic failure mode, not copied from a specific run.

PROMPT (identical in both cases):
What is the capital of Australia?

BASE MODEL CONTINUATION (illustrative):
What is the capital of Australia?
What is the capital of New Zealand?
What is the largest city in Australia?
Which Australian state has the smallest population?

Answers to last week's quiz are printed on page 14.
Send your entries to the address below before 30 June.

INSTRUCTION-TUNED MODEL OUTPUT (typical):
The capital of Australia is Canberra. It was purpose-built as
the capital and chosen as a compromise between Sydney and
Melbourne; construction began in 1913.

2. Both models contain the fact. The base model never gets to it, because answering was never the task.
3. Now a second illustrative pair, showing the missing stop and the missing role boundary.

PROMPT: Write a short thank-you note.

BASE MODEL CONTINUATION (illustrative):
Write a short thank-you note. Write a formal complaint letter.
Write a letter of resignation.

EXERCISE 4. Write a short thank-you note to a colleague who
covered your shift. Your answer should be no longer than 80
words. See the model answer overleaf.

4. Notice what happened. The model produced a plausible worksheet, because the instruction looked like an exercise heading, and worksheets are what follow exercise headings in the training data.
5. It also did not stop, because nothing in pretraining defines an end of turn.

PLAIN49.5.4 what is really happening inside#

1. Three specific things are missing from a base model, and each is added by a different part of post-training.
2. Missing: the notion of a turn. In pretraining, text is one long stream. There is no marker meaning “the user’s part ends here”. Fine-tuning adds special tokens that mean exactly that.
3. Missing: the notion of a role. Nothing distinguishes instructions from content. Prompt-injection attacks exist precisely because that distinction is a learned habit, not a hard boundary. Chapter 44 covers the security side.
4. Missing: a stopping rule. Generation continues until a stop token appears or a limit is hit. Base models have no reliable stop token for a turn, so they run on.
5. Missing: refusal. A base model has never been rewarded for declining. It completes the pattern in front of it.
6. What is present is everything else. Knowledge, grammar, translation, arithmetic to some degree, code, reasoning steps.
7. This is why base models are still published and still used. Researchers want the raw capability without the behaviour layered on top, and anyone building a specialized product often starts from base rather than chat.
8. It is also why you can sometimes get a chat model to behave like a base model by writing a prompt that looks like a document. The base behaviour is still underneath; post-training put a habit over it, not a lock on it.

TECHNICAL49.5.5 the engineer’s version#

1. Naming convention, near-universal on model hubs: a repository with no suffix or with -base is the pretrained model; -instruct, -it, -chat or -sft denotes a post-trained variant. Meta, Alibaba, Google and Mistral all follow this. It is a convention, not a standard.
2. A base checkpoint typically ships without a chat_template field in tokenizer_config.json. If you call a chat API against it, most libraries will either error or silently fall back to raw concatenation.
3. Base models are usually released with a much shorter list of special tokens. Llama 3.1 base defines <|begin_of_text|> and <|end_of_text|>; the instruct variant adds <|start_header_id|>, <|end_header_id|> and <|eot_id|> and gives them meaning through training.
4. Evaluation differs by model type. Base models are measured with few-shot prompting on benchmarks such as MMLU, HellaSwag, GSM8K and HumanEval, because zero-shot instruction following is not expected. Instruct models are measured zero-shot and on chat-specific suites such as IFEval, MT-Bench and arena-style human preference rankings.
5. Established fact: instruction-tuned and base models built from the same pretraining run have near-identical knowledge as measured by few-shot benchmarks. Post-training moves those scores by a few points, not tens.
6. Established fact: base models have no refusal behaviour and will produce harmful content on request. This is why some labs decline to publish base checkpoints even when publishing instruct ones.
7. Active research: how thin the safety layer is. A 2023 result by Qi and colleagues showed that fine-tuning a safety-tuned model on around 100 adversarial examples largely removed its refusal behaviour, at negligible cost. Later work replicated the effect across model families.
8. That result is central to the open-weights argument in section 49.12. If anyone with the weights can remove the safety training for a few dollars, then releasing weights and releasing capability are the same act.
9. Marketing claim to distrust: that a released model is “aligned” in a way that survives fine-tuning by a third party. For open weights, it does not.

WORDS49.5.6 remember these#

1. Base model — the raw pretrained text continuer — a causal LM checkpoint with no chat template, role tokens or refusal training.
2. Instruction-tuned — taught to answer rather than continue — a checkpoint post-trained on demonstration pairs with a chat template applied.
3. Turn — one person’s contribution to a conversation — a span delimited by role header and end-of-turn special tokens.
4. Stop token — the marker that means stop generating — an end-of-sequence or end-of-turn token whose id terminates the decoding loop.
5. Few-shot prompting — showing examples inside the prompt — supplying k labelled exemplars in context to elicit a task from a base model.
6. Prompt injection — text that hijacks the instructions — untrusted content interpreted as instruction because role separation is learned, not enforced.

49.6 Supervised fine-tuning#

PLAIN49.6.1 in simple words#

1. Supervised fine-tuning, usually written SFT, is the first step of post-training and the simplest to understand.
2. You show the model examples of the behaviour you want, and train on them.
3. Each example is a pair: an instruction, and a good response to it.
4. The training method is identical to pretraining. Next-token prediction, same loss, same optimizer. Nothing new is invented.
5. The only differences are the data and the scale.
6. The data is tiny by comparison. Pretraining used trillions of tokens. SFT uses somewhere between a thousand and a few million examples.
7. And there is one important trick: the loss is usually only counted on the response part, not on the instruction. You want the model to learn to produce answers, not to produce questions.
8. Where do the examples come from? Three places, and modern datasets mix all three: written by paid humans, written by humans and edited by models, or generated entirely by a stronger model.
9. Alongside the data comes a chat template: a fixed way of writing down a conversation using special marker tokens.
10. Those markers are what create the idea of turns and roles. Before them, a conversation is just text. After them, the model can tell who is speaking.
11. SFT is cheap. Hours to a couple of days on a handful of machines.
12. And it produces most of the visible difference between a base model and a chatbot.

PLAIN49.6.2 a picture in your head#

1. Think of a new employee on their first day at a help desk.
2. They already know the subject matter. They studied it for years.
3. What they do not know is the house style. How to greet a caller. How long an answer should be. When to escalate. What never to promise.
4. So the trainer sits them down with a folder of two hundred past tickets, each with a model answer written by an experienced colleague.
5. The new person reads them and copies the pattern. By the afternoon they sound like the rest of the team.
6. Nothing about their subject knowledge changed that morning. Only the shape of how it comes out.
7. That is supervised fine-tuning.

Where this comparison breaks: the employee can be told a rule once and follow it, such as “never quote a delivery date”. A model cannot be told; it can only be shown, many times, and it will still fail sometimes. Also, the employee knows they are following a house style and can drop it if asked. A model has no such separation between what it knows and how it was taught to speak.

PLAIN49.6.3 a worked example#

1. Here is a real chat template, the Llama 3 format used by Meta since April
   2024. Every angle-bracket item is a single special token in the vocabulary. Some lines are wrapped here to fit the page; in the real string the tokens run straight on with no break.

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are a helpful assistant.<|eot_id|>
<|start_header_id|>user<|end_header_id|>

What is the capital of Australia?<|eot_id|>
<|start_header_id|>assistant<|end_header_id|>

The capital of Australia is Canberra.<|eot_id|>

2. Read what those tokens do. <|begin_of_text|> marks the start of the whole document. <|start_header_id|> and <|end_header_id|> wrap a role name. <|eot_id|> means end of turn.
3. During SFT, the loss is computed only on the tokens after the assistant header, up to and including the closing <|eot_id|>.
4. That is how the model learns two things at once: what to say, and when to stop saying it.
5. A different, equally real template is ChatML, introduced by OpenAI in 2023 and now used by many others including Qwen:

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
What is the capital of Australia?<|im_end|>
<|im_start|>assistant
The capital of Australia is Canberra.<|im_end|>

6. These are not interchangeable. Feed Llama’s format to a ChatML model and quality drops sharply, because the tokens it is waiting for never arrive.
7. That is why every model repository carries a chat_template string, and why applying the wrong one is one of the most common practical mistakes people make when serving a model themselves.

PLAIN49.6.4 what is really happening inside#

1. Step by step, what SFT does to one example.
2. Take the instruction and the response. Render them into the chat template.
3. Tokenize the whole thing into one sequence of token ids.
4. Build a mask marking which positions are “response”. Usually everything after the assistant header.
5. Run the forward pass over the whole sequence, so the model sees the instruction as context.
6. Compute cross-entropy loss only at the masked positions.
7. Backward pass, optimizer step. Identical machinery to pretraining.
8. Repeat over the dataset for one to three passes. More than about three passes usually makes the model repetitive and worse, because the dataset is small enough to memorize.
9. The learning rate is much lower than pretraining, typically 1e-5 to 2e-5 rather than 1e-4. A large step here would wreck what pretraining built.
10. That last point is worth dwelling on. Fine-tuning too hard causes catastrophic forgetting: the model gets better at the new data and measurably worse at everything else.

TECHNICAL49.6.5 the engineer’s version#

1. Real dataset sizes, with sources and dates:

2. LIMA is worth knowing about. Zhou and colleagues, May 2023, fine-tuned a 65-billion-parameter LLaMa on exactly 1,000 carefully written examples with no preference optimization at all, and got competitive chat behaviour.
3. They named the conclusion the Superficial Alignment Hypothesis: that almost all knowledge is learned in pretraining and alignment mainly teaches format and style. Section 49.8 gives the honest caveats to that claim.
4. Alpaca, from Stanford in March 2023, is the origin of the self-instruct pattern that now dominates: seed a strong model with a few examples and have it generate tens of thousands more. The reported cost of generating the data was a few hundred dollars in API calls.
5. Typical SFT hyperparameters for a 7 to 8 billion parameter model: learning rate 1e-5 to 2e-5, cosine decay, 2 to 3 epochs, global batch of 64 to 256 sequences, sequence length 4,096 to 8,192, bf16, AdamW.
6. Cost, computed rather than quoted. One million examples averaging 500 tokens is 5e8 tokens. For an 8-billion model, 6ND gives 6 times 8e9 times 5e8, which is 2.4e19 FLOPs. Against the 7.2e23 of pretraining that is 0.003 per cent.
7. In wall-clock terms that is a few hours on 8 H100s, at a rental cost in the low hundreds of dollars. Three orders of magnitude cheaper than most people guess, and this is why so many fine-tuned variants exist.
8. Packing: short examples are concatenated up to the sequence length with attention masked between them, so no compute is wasted on padding. Almost universal in practice.
9. What SFT reliably changes: output format, response length, tone, willingness to answer, adherence to a system prompt, function and tool-call syntax, language of reply, and refusal behaviour on clear-cut categories.
10. What SFT does not reliably change: factual knowledge absent from pretraining, reasoning depth, and calibration.
11. There is direct evidence for that last point. Gekhman and colleagues in 2024 showed that fine-tuning on facts the model did not already know is learned slowly and increases hallucination on unrelated questions, because it teaches the model to assert confidently things it has no basis for.
12. Tools: trl from Hugging Face for SFTTrainer, axolotl and LLaMA-Factory for configuration-driven runs, peft for adapters, and tokenizer.apply_chat_template() to render conversations correctly.

WORDS49.6.6 remember these#

1. SFT — showing the model good answers and training on them — supervised fine-tuning on instruction-response pairs with a next-token objective.
2. Chat template — the fixed layout of a conversation — a rendering of roles and turns into special tokens, stored in tokenizer_config.json.
3. Special token — a marker the model treats as one indivisible unit — a reserved vocabulary id such as <|eot_id|> never produced by the tokenizer.
4. Loss masking — only grading the answer, not the question — computing cross-entropy on assistant tokens and ignoring prompt tokens.
5. Catastrophic forgetting — getting worse at old skills while learning new ones — degradation of pretrained capability under aggressive fine-tuning.
6. Self-instruct — using a strong model to write training data — synthetic instruction generation seeded from a small set of human examples.

49.7 Learning from preferences#

PLAIN49.7.1 in simple words#

1. Showing good examples gets you a long way and then stops working.
2. The reason is that for most questions there is no single right answer, and writing one by hand is slow, expensive and often not even possible.
3. But people are very good at a different task: looking at two answers and saying which is better.
4. That judgement is fast, cheap, and works even when the judge could not have written either answer themselves.
5. So the next stage of training is built on comparisons instead of examples.
6. The classic recipe is called RLHF, reinforcement learning from human feedback. It has three steps.
7. Step one: take one prompt, have the model produce several answers, and ask a person which they prefer.
8. Step two: train a second, separate model, called a reward model, whose only job is to look at an answer and output a score predicting what a human would say.
9. Step three: use that score to train the original model, nudging it towards answers the reward model likes.
10. There is a catch, and it is the most important thing in this section.
11. The reward model is a rough approximation of human taste. Push the main model hard enough against it and the main model will find the gaps.
12. It will learn to produce things that score highly and are actually worse. That is called reward hacking, and it is not a rare failure. It is the default outcome without countermeasures.
13. So a brake is added: a penalty for drifting too far from the model you started with. It is allowed to improve, but not to go strange.

PLAIN49.7.2 a picture in your head#

1. Imagine training a chef by giving them a food critic.
2. You cannot write down what “delicious” means, but the critic can taste two dishes and say which is better. So you let the critic judge.
3. The critic is busy, so you hire an assistant who has watched the critic for a while and can now guess the critic’s verdicts. That is the reward model.
4. Now the chef cooks and the assistant scores, thousands of times a day.
5. At first this works beautifully. The food improves.
6. Then the chef notices the assistant always scores dishes with truffle oil highly. So every dish gets truffle oil. Scores go up. The food gets worse.
7. That is reward hacking. The chef has not learned to cook. The chef has learned to please the assistant.
8. So you add a rule: the chef may not change the menu by more than a little from the original recipes. That is the drift penalty.

Where this comparison breaks: the chef knows they are gaming the assistant. A model has no such intent. It is doing gradient descent on a number, and the number happens to be maximized by truffle oil. There is no deception in it, which makes it harder to spot, not easier.

PLAIN49.7.3 a worked example#

1. Concrete reward hacking, all of these observed in real systems.

2. The oldest and clearest example is not from language at all. In December 2016 OpenAI described an agent trained to play a boat racing game. The reward was points, and points came from hitting targets along the course.
3. The agent learned to drive in circles in a lagoon, hitting three regenerating targets forever, catching fire, and never finishing the race. It scored about 20 per cent higher than human players.
4. It maximized the reward perfectly. It failed the task completely.
5. A recent language example with a date. On 25 April 2025 OpenAI shipped an update to GPT-4o that made it markedly sycophantic: it agreed with users, praised bad ideas, and validated statements it should have questioned.
6. OpenAI rolled it back on 29 April 2025 and published an explanation saying the update had over-weighted short-term user approval signals such as thumbs up and thumbs down.
7. That is textbook reward hacking, at production scale, at a major lab, in public. It is not a theoretical concern.
8. The mitigation in the RLHF recipe is the KL penalty. At each step, the training objective is: maximize the reward, minus a coefficient times how far the new model’s output distribution has moved from the starting model’s.
9. Set the coefficient too high and nothing improves. Too low and the model drifts into reward-hacked nonsense. It is tuned by hand.

PLAIN49.7.4 what is really happening inside#

1. The full RLHF loop as a sequence.

 [1] PROMPT SET
      |
      v
 [2] SAMPLE: current model writes 2 or more answers
      |
      v
 [3] HUMAN COMPARES: "B is better than A"
      |
      v
 [4] TRAIN REWARD MODEL r(prompt, answer) -> score
      |     loss: -log sigmoid( r(chosen) - r(rejected) )
      v
 [5] RL LOOP, repeated:
      model writes answer  ->  reward model scores it
              ^                        |
              |                        v
        PPO update  <-----  reward minus KL penalty
              |
              +-- reference model (frozen copy of the
                  starting model) supplies the KL term

2. The reward model is usually the same architecture as the main model with the final vocabulary layer replaced by a single number output.
3. Its training loss comes from the Bradley-Terry model of paired comparison, which dates from 1952: the probability that A beats B is a logistic function of the difference in their scores.
4. The RL algorithm is almost always PPO, proximal policy optimization, published by Schulman and colleagues in 2017 for robotics and games.
5. Notice how many models are in memory at once during PPO: the policy being trained, the frozen reference for the KL term, the reward model, and usually a value model. Four copies. This is why RLHF is memory-hungry and awkward.
6. That memory cost is exactly what the next method removes.

TECHNICAL49.7.5 the engineer’s version#

1. DPO, direct preference optimization, Rafailov and colleagues, May 2023, a NeurIPS 2023 outstanding paper. The insight is algebraic: for the standard KL-regularized RL objective, the optimal policy can be written in closed form in terms of the reward, so the reward can be written in terms of the policy.
2. Substituting that back into the Bradley-Terry loss removes the reward model entirely. What remains is a simple classification loss on preference pairs, trained with ordinary supervised machinery.
3. Practical consequence: two models in memory instead of four, no sampling loop, no PPO tuning, and a training run that looks like SFT. Many labs and almost all small teams moved to DPO or a variant during 2023 and 2024.
4. Variants you will meet: IPO, which fixes an overfitting failure in DPO; KTO, which needs only a thumbs up or down rather than a pair; ORPO, which folds preference learning into SFT; and SimPO, which drops the reference model.
5. Constitutional AI, Bai and colleagues at Anthropic, 15 December 2022. Phase one: the model critiques and revises its own answers against a written list of principles, and is fine-tuned on the revisions. Phase two: an AI model, not a human, judges which of two answers better follows the principles, and those judgements train the preference model. This is RLAIF, reinforcement learning from AI feedback.
6. The written list is the “constitution”. Anthropic published theirs in May
   2023. The point of the method is that the only human input is the document, which is auditable, rather than millions of individual labels, which are not.
7. RLVR, reinforcement learning with verifiable rewards. Named in Ai2’s Tulu 3 release of November 2024. There is no reward model at all. For maths, check the final answer against ground truth. For code, run the unit tests. The reward is 1 or 0 and it cannot be hacked by style.
8. This is the engine behind reasoning models. DeepSeek-R1, released January 2025 with an accompanying paper, was trained with GRPO, group relative policy optimization, using rule-based correctness rewards. Its R1-Zero variant applied RL directly to a base model with no SFT at all, and long chains of reasoning emerged from the reward alone.
9. Established fact: RLVR substantially improves benchmark accuracy on maths and competitive programming. Active research: whether it creates new reasoning ability or elicits ability already present. A widely discussed 2025 study by Yue and colleagues found that base models sampled many times eventually match or exceed RLVR-trained models on pass-at-k for large k, which argues for elicitation. Contested.
10. The comparison table:

11. Cost scale, approximate. Human preference collection dominates: at 1 to 5 US dollars per comparison for expert raters, 100,000 comparisons is 100,000 to 500,000 dollars in labelling before any compute is spent.
12. That is why RLAIF exists. AI judgements cost cents and scale without limit. The trade is that you inherit the judge’s blind spots wholesale.
13. Marketing claim to reject: that preference training makes a model truthful. It makes the model produce answers people prefer. Where people prefer a confident wrong answer to an uncertain right one, preference training actively pushes towards the wrong one. Section 49.98 restates this.
14. Tools: trl implements PPOTrainer, DPOTrainer, KTOTrainer and GRPOTrainer; OpenRLHF and verl are used for larger distributed RL runs; open-instruct is Ai2’s published RLVR pipeline.

WORDS49.7.6 remember these#

1. RLHF — training on which answer people prefer — reinforcement learning from human feedback, via a learned reward model optimized with PPO.
2. Reward model — a stand-in for a human judge — a network scoring prompt-response pairs, trained on preference comparisons.
3. Reward hacking — scoring well while getting worse — optimizing a proxy objective in ways that diverge from the intended goal.
4. KL penalty — the leash keeping the model near where it started — a divergence term penalizing drift from the reference policy.
5. DPO — preference training without a separate judge model — direct preference optimization, a closed-form reparameterization of the RLHF objective.
6. RLAIF — an AI does the judging — reinforcement learning from AI feedback, with principles replacing per-example human labels.
7. RLVR — rewards you can check by running something — reinforcement learning with verifiable rewards, using ground truth or unit tests.
8. Sycophancy — telling people what they want to hear — a preference-trained failure mode where agreement is rewarded over accuracy.

49.8 What each stage actually changes#

PLAIN49.8.1 in simple words#

1. Here is the sentence people most want a straight answer to.
2. Pretraining puts the knowledge in. Post-training decides how it comes out.
3. That sentence is mostly right and it is the right thing to remember.
4. It explains why fine-tuning a model on your company handbook does not make it an expert on your company.
5. It explains why a model that never saw a language during pretraining cannot be taught that language with a few thousand examples.
6. And it explains why two chat models built from the same base model feel different but know the same things.
7. Now the honest part, because the boundary is fuzzier than that sentence suggests.
8. Post-training does move factual benchmark scores, by a few points.
9. Fine-tuning can install narrow facts, badly, at the cost of more hallucination elsewhere.
10. And modern pipelines blur the line deliberately, with a “mid-training” phase that is really more pretraining on curated data.
11. So the clean version is a good working rule and a slight lie. Section 49.8.5 gives the accurate picture.

PLAIN49.8.2 a picture in your head#

1. Think of a musician who has spent fifteen years learning an instrument.
2. That is pretraining. The technique is in the hands and it is not going away.
3. Now they join an orchestra and spend two weeks learning the house style: when to come in, how loud, how much vibrato, when to defer to the section.
4. That is post-training. Two weeks changed how they play in every performance.
5. But two weeks did not teach them a new instrument, and it did not give them technique they did not have.
6. If you ask them to play something they never learned, no amount of orchestra rehearsal will fix it.

Where this comparison breaks: two weeks of rehearsal can genuinely teach a musician a specific new piece. Post-training can genuinely teach a model specific new facts too. In both cases it is narrow, effortful and does not generalize. The rule is about scale, not about an absolute barrier.

PLAIN49.8.3 a worked example#

1. The main table. Read the third column carefully, because that is the one people get wrong.

2. A concrete test you can run yourself, using two public models built from the same pretraining run, such as a base and its instruct variant.
3. Ask both, few-shot, a hundred factual questions. The scores will be within a few points of each other.
4. Then ask both, zero-shot, to answer politely in three sentences. Only the instruct model will do it.
5. That is the whole distinction, demonstrated in ten minutes.

PLAIN49.8.4 what is really happening inside#

1. Why is knowledge stuck in pretraining? Two reasons, and both are about size.
2. Reason one: data volume. Pretraining is 15 trillion tokens. SFT is perhaps 500 million. That is a ratio of 30,000 to one. A signal that faint moves the weights very little.
3. Reason two: learning rate. SFT runs at roughly a tenth of pretraining’s learning rate specifically so it does not damage what is there. It is a polish, applied deliberately gently.
4. So post-training is a small nudge from a small dataset. It changes behaviour because behaviour is a shallow property, expressible as a habit of which token to emit at the start of a response.
5. It does not change knowledge because knowledge is spread across billions of parameters built up over a trillion updates.
6. There is a useful piece of evidence for the shallowness. Work in 2024 and 2025 on what is called shallow alignment found that the difference between a base and an aligned model’s output distribution is concentrated in the first handful of generated tokens. Get past the opening and the behaviours converge. That is also why prefilling attacks work.

TECHNICAL49.8.5 the engineer’s version#

1. Evidence for the clean rule. The LIMA result, Zhou and colleagues, May 2023: 1,000 examples produced competitive chat behaviour from a 65B base model, which is hard to explain unless the capability was already present. InstructGPT, March 2022: a 1.3B instruction-tuned model was preferred by human raters over the 175B GPT-3 base, while remaining far weaker on knowledge benchmarks. Preference moved; knowledge did not.
2. Evidence for the fuzziness, which is the part usually left out. Gekhman and colleagues, 2024: fine-tuning on facts outside the model’s pretraining is learned much more slowly than facts inside it, and increases hallucination rate on other questions as it is learned. So new facts can be installed, at a measurable cost.
3. RLVR on maths raises GSM8K and competition-maths accuracy substantially. Whether that is new capability or elicitation is genuinely disputed, and the pass-at-k results from Yue and colleagues in 2025 favour elicitation. Treat this as active research, not settled.
4. Mid-training deliberately erases the boundary. Ai2’s Dolmino and Dolma 3 mixes apply a curated, high-quality corpus with a decaying learning rate after the main run. That is pretraining by mechanism and curation by intent.
5. Continued pretraining, sometimes called domain-adaptive pretraining, does install knowledge, because it uses pretraining-scale data. Billions of tokens of medical or legal text will change what a model knows. Millions will not. The threshold is roughly where you would expect from the ratios.
6. Practical rule with numbers attached:

7. The honest version: the sentence “post-training cannot install knowledge” is a statement about efficiency, not about possibility. Knowledge enters through data volume. Post-training datasets are too small to carry much of it, and the learning rate is deliberately set so they do not try.
8. One more caution about safety tuning specifically. It changes willingness to answer, not the presence of the answer. A model that refuses a question still contains whatever it learned in pretraining. Removing the refusal through fine-tuning, which as noted in section 49.5 takes about 100 examples, exposes it again.

WORDS49.8.6 remember these#

1. Alignment — making the model behave as intended — post-training that shapes output behaviour towards a specification of helpfulness and harmlessness.
2. Mid-training — a curated final stretch of pretraining — annealing on a high-quality mixture with a decaying learning rate.
3. Continued pretraining — more pretraining on a specific domain — the only fine-tuning-shaped method that reliably installs new knowledge.
4. Shallow alignment — the behaviour layer is thin — the base and aligned output distributions diverging mainly over the first few generated tokens.
5. Elicitation — bringing out something already there — improving measured performance without adding underlying capability.
6. Hallucination — confidently stating something false — high-confidence generation unsupported by training data or provided context.

49.9 Efficient adaptation: LoRA, QLoRA and distillation#

PLAIN49.9.1 in simple words#

1. Suppose you want to adjust an existing model rather than train one.
2. The obvious way is full fine-tuning: load every weight, train them all.
3. It works. It is also astonishingly memory-hungry, because you need much more than just the weights in memory.
4. You need the weights, a high-precision master copy of them, one gradient per weight, and two extra running averages per weight that the optimizer keeps.
5. Roughly sixteen to eighteen bytes per parameter, against two bytes to just run the model. An eight-fold increase before you store anything else.
6. So a model you can happily run on one graphics card may need eight cards to fine-tune. That is the problem.
7. The fix that changed everything is called LoRA, low-rank adaptation.
8. The idea: freeze the original weights entirely. Do not touch them. Instead, add a small pair of extra matrices beside each big one, and train only those.
9. The pair is deliberately thin. A 4096-by-4096 matrix might be shadowed by a 4096-by-16 and a 16-by-4096 pair. That is 131,072 numbers instead of nearly 17 million.
10. Because the frozen weights never change, they need no gradients and no optimizer state. Only the tiny matrices do.
11. When you are done, you can either keep the small matrices as a separate file of a few tens of megabytes, or add them into the original weights to make a single ordinary model.
12. QLoRA goes further: squash the frozen original down to 4 bits per number as well. Now even the frozen part is a quarter of its size.
13. The result is that fine-tuning a 65-billion-parameter model became possible on one 48 GB card in 2023, where before it needed a small cluster.

PLAIN49.9.2 a picture in your head#

1. Think of a large printed reference book that you are not allowed to write in.
2. You want to correct and adapt it for your own use.
3. Full fine-tuning is reprinting the entire book with your changes. Correct, and it costs as much as the original printing.
4. LoRA is a pad of thin transparent overlay sheets. The book stays untouched. You write only your corrections on the overlays and lay them on top.
5. The overlays are tiny compared to the book, because most pages need nothing.
6. You can carry ten different sets of overlays for ten different jobs and swap them in seconds, sharing one copy of the book.
7. And if you want, you can eventually print a new edition with the overlays merged in. That is merging the adapter.

Where this comparison breaks: an overlay covers a page and hides what is under it. A LoRA adapter adds to the original numbers rather than replacing them, so the base behaviour is always still contributing. Also, an overlay can say anything, while a low-rank adapter is mathematically restricted in the kinds of change it can express. That restriction is the whole point, and it is also the method’s main limitation.

PLAIN49.9.3 a worked example#

1. Memory arithmetic for a 7-billion-parameter model, computed here.

2. Where the 16 to 18 bytes comes from: 2 bytes of bf16 weight, 4 bytes of fp32 master copy, 2 to 4 bytes of gradient, and 8 bytes of Adam’s two moments.
3. Read the consequences off the table. Full fine-tuning a 7B model needs two 80 GB cards at least. QLoRA fits in a single 8 GB consumer card at short sequence lengths, and comfortably in a 16 GB one.
4. LoRA parameter count, worked. Take a 4096-by-4096 attention projection. Full: 16,777,216 parameters. LoRA at rank 16: 4096 times 16 for the first matrix plus 16 times 4096 for the second, which is 131,072.
5. That is 0.78 per cent of the original. Applied to a whole 7B model at rank 16 on attention projections only, the trainable count lands around 4 to 20 million depending on which modules are targeted.
6. What the rank trades: higher rank means more capacity to express change, more memory, more risk of overfitting on a small dataset. Common values are 8, 16, 32 and 64. Rank 8 to 16 is enough for style and format. Rank 64 to 256 is used when the adaptation is substantial.

PLAIN49.9.4 what is really happening inside#

1. LoRA rests on one observation: when you fine-tune a model, the change to each weight matrix turns out to be well approximated by a low-rank matrix.
2. In plain terms, the update has far less independent structure than its size suggests, so it can be written as a thin matrix times another thin matrix.
3. So instead of learning the change directly, you learn the two thin factors.
4. During the forward pass, the layer computes the original output plus the adapter’s output, scaled by a constant.
5. During the backward pass, gradients flow to the adapter only. The frozen weights receive none, which is where the memory saving comes from.
6. The first thin matrix is initialized randomly and the second to all zeros. That makes the adapter’s contribution exactly zero at the start, so training begins from the base model’s behaviour rather than from a random shove.
7. Merging means computing the product of the two thin matrices and adding the result into the original weight matrix. After merging, inference costs exactly what the original cost. Before merging, it costs slightly more.
8. QLoRA’s addition: store the frozen base in 4 bits, dequantize each block to 16 bits on the fly as it is used, and keep the adapter in 16 bits. The quantization error sits in the frozen part, which is not being trained, so the adapter learns around it.

TECHNICAL49.9.5 the engineer’s version#

1. LoRA: Hu and colleagues, Microsoft, 17 June 2021. Reported reduction of trainable parameters by 10,000 times and GPU memory by 3 times against full fine-tuning of GPT-3 175B with Adam, with no added inference latency after merging.
2. Hyperparameters that matter: r the rank; lora_alpha the scaling, where the adapter output is multiplied by alpha over r; target_modules, which projections get adapters; and lora_dropout. A common default is r 16, alpha 32, targeting the query, key, value and output projections.
3. Targeting the feed-forward projections as well as attention generally helps for larger behavioural changes and costs more memory.
4. QLoRA: Dettmers and colleagues, 23 May 2023. Three contributions. NF4, a 4-bit data type designed for normally distributed weights. Double quantization, quantizing the quantization constants themselves. Paged optimizers, using unified memory to survive memory spikes.
5. Their headline result: fine-tuning a 65B model on a single 48 GB GPU while matching 16-bit fine-tuning quality. Their Guanaco models reached 99.3 per cent of ChatGPT’s score on the Vicuna benchmark after 24 hours on one GPU. Treat benchmark percentages of that era with caution; the memory result is the durable contribution.
6. Other parameter-efficient methods, with dates: adapter layers, Houlsby and colleagues 2019, small bottleneck blocks inserted in series, which do add inference latency; prefix tuning, Li and Liang 2021, learned key and value vectors prepended inside attention; prompt tuning, Lester and colleagues 2021, learned soft embedding vectors prepended to the input, needing as few as a few thousand parameters; IA3, Liu and colleagues 2022, learned per-channel scaling vectors.
7. Distillation: Hinton, Vinyals and Dean, 2015. Train a small student on the large teacher’s output distribution rather than on hard labels, because the teacher’s probabilities over wrong answers carry information about similarity structure. DistilBERT, Sanh and colleagues 2019, produced a model 40 per cent smaller and 60 per cent faster retaining about 97 per cent of BERT’s performance.
8. Modern usage has shifted. Most “distillation” in the LLM era is sequence- level: generate outputs from a strong model and fine-tune a small one on the text. That is really SFT on synthetic data. True logit-level distillation is used inside labs; Google stated it for the Gemma 2 releases in 2024.
9. Note the licence trap. Several providers’ terms have historically prohibited using their outputs to train competing models. Whether such a clause is enforceable is untested, but it is in the contract you accepted.
10. The decision table people actually need:

11. The ordering to follow in practice: prompt, then few-shot prompt, then retrieval, then LoRA fine-tune, then full fine-tune, then continued pretraining. Each step up costs roughly ten times the last. Stop at the first one that meets the requirement.
12. Tools: peft for LoRA and friends, bitsandbytes for 4-bit and 8-bit, unsloth for faster single-GPU runs, axolotl and LLaMA-Factory for configuration-driven training, and vllm with multi-LoRA serving to host many adapters against one base copy.

WORDS49.9.6 remember these#

1. Full fine-tuning — retraining every number in the model — updating all parameters, needing roughly 16 to 18 bytes per parameter of optimizer state.
2. LoRA — training small extra matrices instead of the whole model — low-rank adaptation with frozen base weights and trainable factorized updates.
3. Rank — how much change the adapter can express — the inner dimension r of the two low-rank factors, trading capacity against memory.
4. Merging — folding the adapter back into the model — adding the product of the low-rank factors into the base weight matrix.
5. QLoRA — LoRA on a squashed base model — 4-bit NF4 quantized frozen weights with 16-bit adapters, double quantization and paged optimizers.
6. Distillation — a small model copying a big one — training a student on a teacher’s output distribution or generated text.
7. Adapter — a small trainable add-on module — any parameter-efficient block inserted into a frozen network, in series or in parallel.

49.10 What a training run looks like physically#

PLAIN49.10.1 in simple words#

1. A frontier training run is not a program on a computer. It is one program spread across thousands of computers that must stay in step.
2. The accelerators sit eight to a chassis, the chassis stack into racks, the racks fill rows in a datacentre. Chapter 43 described that building.
3. Inside one chassis the eight accelerators are joined by a very fast private link. Between chassis they are joined by network cable, which is far slower.
4. That difference, roughly twenty times, decides how the work is divided.
5. The model may be too big to fit in one accelerator’s memory. So it is split.
6. There are three ways to split it, and real runs use all three at once.
7. Data parallelism: every machine has a full copy of the model and gets a different slice of the batch. Cheap, simple, and needs every machine to share its gradients with every other after each step.
8. Tensor parallelism: one layer’s matrix is cut into pieces across several accelerators. Very chatty; must stay inside the fast private link.
9. Pipeline parallelism: different machines hold different layers, like stations on an assembly line. Cheap in communication and leaves machines idle waiting for work, which is called the bubble.
10. On top of all that, hardware breaks. Constantly. A run of thousands of accelerators for two months will be interrupted hundreds of times.
11. So the whole thing is built around checkpoint and restart, not around hoping nothing fails.

PLAIN49.10.2 a picture in your head#

1. Think of a very large kitchen cooking one enormous banquet.
2. Data parallelism is eight identical kitchens each cooking the same menu for a different eighth of the guests, then meeting to agree the recipe changes.
3. Tensor parallelism is four chefs sharing one sauce, each stirring a quarter of the pan, shouting to each other constantly. They must be within earshot.
4. Pipeline parallelism is a line: one station chops, the next fries, the next plates. Efficient once running, and the fryer stands idle at the start.
5. The shouting is the problem. Chefs in the same room can shout. Chefs in different buildings must send messengers, and the messengers are the bottleneck.

Where this comparison breaks: kitchens degrade gracefully when one chef stops. A training run does not. Every accelerator must complete every step, because the gradient sum needs all of them. One dead card halts the entire cluster. That is why the failure and restart machinery matters so much.

PLAIN49.10.3 a worked example#

1. The three splits, drawn.

DATA PARALLEL: same model, different data
  GPU0 [full model] <- batch part 0
  GPU1 [full model] <- batch part 1
  GPU2 [full model] <- batch part 2
       after backward: all-reduce gradients across all GPUs
       cost: one full gradient exchange per step

TENSOR PARALLEL: one layer cut across GPUs
  layer weight W = [ W0 | W1 | W2 | W3 ]
  GPU0 holds W0, GPU1 holds W1, ...
  every forward and backward needs an all-reduce
  cost: several collectives PER LAYER. Keep inside a node.

PIPELINE PARALLEL: different layers on different GPUs
  GPU0: layers 0-7  -> GPU1: layers 8-15 -> GPU2: 16-23
  micro-batches flow left to right, then gradients right
  to left. Idle time at fill and drain = the "bubble".
  cost: one activation send per boundary. Cheapest.

2. A real configuration. Meta reported Llama 3.1 405B running on 16,384 H100s with four-way parallelism: tensor 8, context 1, pipeline 16, data 128.
3. Check the arithmetic: 8 times 1 times 16 times 128 equals 16,384.
4. Note where the 8 went. Tensor parallelism is exactly 8, which is the number of accelerators in one chassis sharing the fast private link. That is not a coincidence; it is the design constraint.

PLAIN49.10.4 what is really happening inside#

1. The memory problem, stated plainly. Training a 405-billion-parameter model in mixed precision needs roughly 16 bytes per parameter for weights, master copy, gradients and optimizer moments. That is about 6.5 terabytes.
2. An H100 has 80 GB. So you need at least 81 of them just to hold the state, before any activations, before any batch.
3. Four techniques reduce that.
4. Sharding: do not keep a full copy of the optimizer state and gradients on every machine. Split them and fetch pieces as needed. This is ZeRO in Microsoft’s DeepSpeed, and FSDP in PyTorch.
5. Activation checkpointing: during the forward pass, throw away most of the intermediate values and recompute them during the backward pass. Trades about 30 per cent more compute for a large memory saving.
6. Mixed precision: store and multiply in 16-bit or 8-bit formats while accumulating in higher precision, halving or quartering memory traffic.
7. Gradient accumulation: process the batch in small pieces, add up the gradients, and only update once. This makes a large effective batch fit in small memory, at the cost of more steps.
8. When something fails, the sequence is: a collective operation times out, the job manager kills every process, a health check identifies the bad node, it is removed from the pool, and the job restarts from the last checkpoint.
9. Everything between the checkpoint and the failure is lost. Which is why the checkpoint interval is a direct trade between storage bandwidth and wasted compute, and is usually set at 15 minutes to an hour.

TECHNICAL49.10.5 the engineer’s version#

1. Interconnect, with real figures.

2. Read the first two rows against each other. Inside a node, 900 GB/s. Between nodes, 50 GB/s. An 18-fold gap. That single ratio dictates that tensor parallelism, which communicates several times per layer, must stay inside a node, and pipeline or data parallelism crosses the network.
3. Meta built two 24,000-GPU clusters for Llama 3, one on InfiniBand and one on RoCE over Ethernet, and reported both performed acceptably. That was a notable result, because Ethernet had been considered unsuitable.
4. ZeRO, Rajbhandari and colleagues 2019 to 2020, in three stages: stage 1 shards optimizer state, stage 2 also shards gradients, stage 3 also shards parameters. Stage 3 is what PyTorch calls FSDP. Stage 3 gives the largest memory saving and the most communication.
5. Activation checkpointing, sometimes called gradient checkpointing, from Chen and colleagues 2016. Storing only every square-root-of-n-th activation reduces activation memory from order n to order square root of n at the cost of one extra forward pass.
6. Mixed precision, Micikevicius and colleagues 2017. bf16 has 8 exponent bits and 7 mantissa bits, the same dynamic range as fp32 with less precision, which is why it displaced fp16 for training: it rarely needs loss scaling. fp16 has 5 exponent and 10 mantissa bits and overflows more easily.
7. fp8 arrived with NVIDIA Hopper in 2022 in two forms, E4M3 and E5M2. Its first large public production use was DeepSeek-V3, December 2024, which trained a 671-billion-parameter mixture-of-experts model in fp8 with selective higher-precision accumulation.
8. Physical figures, computed here from published specifications: an H100 SXM is 700 W; 16,384 of them is 11.5 MW of accelerator draw alone. Add hosts, memory, storage and networking and the IT load is roughly 1.4 to 1.6 times that, so about 16 to 18 MW. Multiply by a datacentre power usage effectiveness of 1.1 to 1.2 and the facility draws roughly 18 to 22 MW.
9. Over the 78 days implied by 30.84 million GPU-hours on 16,384 GPUs, that is roughly 34 to 41 GWh, which is consistent with the 8,930 tonnes CO2eq Meta reported at typical grid carbon intensity.
10. For comparison, 20 MW is roughly the continuous electricity demand of a town of 15,000 to 20,000 homes in a temperate country.
11. Cooling: air cooling tops out around 30 to 40 kW per rack. An 8-way H100 node is about 10 kW, so four to five nodes per rack is the air-cooled limit. NVIDIA’s GB200 NVL72 rack, shipping from 2025, draws around 120 kW and requires direct-to-chip liquid cooling. Chapter 43 covers the facility side.
12. Reliability, real reported figures. Meta reported 466 job interruptions across a 54-day snapshot of the 405B run. 47 were planned. 419 were unexpected, and around 78 per cent were confirmed or suspected hardware issues, with GPU faults the largest single category. Effective training time, meaning useful compute over elapsed time, was still above 90 per cent.
13. That is one unplanned interruption roughly every three hours across the cluster, and it is normal rather than exceptional.
14. Historical precedent: Meta published the OPT-175B logbook in 2022, a day-by-day account of a 992-A100 run with dozens of restarts, hardware replacements and loss divergences. It is the best public document on what this actually feels like.
15. Tools that observe it: nvidia-smi and DCGM for device telemetry, NCCL with NCCL_DEBUG=INFO for collective failures, Slurm or Kubernetes for job control, and per-step gradient-norm logging as the earliest warning that something is going wrong numerically.

WORDS49.10.6 remember these#

1. Data parallelism — every machine has the whole model, different data — batch sharding with an all-reduce of gradients each step.
2. Tensor parallelism — one layer split across machines — intra-layer sharding requiring collectives inside the forward and backward pass.
3. Pipeline parallelism — different layers on different machines — inter-layer sharding with micro-batches and an idle bubble at fill and drain.
4. All-reduce — everyone shares and everyone gets the total — a collective that sums a tensor across ranks and returns the sum to all of them.
5. ZeRO and FSDP — do not keep duplicate copies of training state — sharding of optimizer state, gradients and parameters across data-parallel ranks.
6. Activation checkpointing — forget and recompute to save memory — discarding intermediate activations and recomputing them in the backward pass.
7. bf16 — a 16-bit number format with wide range — 8 exponent and 7 mantissa bits, the default training precision since roughly 2020.
8. Bubble — idle time in a pipeline — accelerator cycles wasted while waiting for micro-batches to fill or drain the pipeline stages.

49.11 What weights physically are#

PLAIN49.11.1 in simple words#

1. You asked directly what weights are, so here is the direct answer.
2. A model’s weights are a large binary file full of numbers, and nothing else.
3. Not text. Not code. Not sentences. Not a database. A long run of bytes that a program interprets as numbers laid out in rectangles.
4. That file alone is useless. You need two more small things with it.
5. First, a config: a short text file saying how many layers there are, how wide they are, how many attention heads, what the vocabulary size is.
6. Without it, a program cannot tell where one rectangle ends and the next begins. The numbers are unlabelled without it.
7. Second, the tokenizer: the fixed list of text pieces and the rules for cutting text into them. Without it you cannot turn words into inputs.
8. So a published model is three things: a big number file, a small shape file, and a small vocabulary file. Everything else in the directory is optional.
9. The number file is often split into several parts, because a single 200 GB file is awkward to download and to resume. Then a small index file says which piece of the model lives in which part.
10. The size is arithmetic, not mystery. Multiply the parameter count by the bytes per number. Eight billion numbers at two bytes each is 16 GB.
11. There is no compression in the normal formats. The file is exactly as big as the numbers it holds, plus a header of a few kilobytes.

PLAIN49.11.2 a picture in your head#

1. Picture a warehouse full of numbered shelves, and one clipboard at the door.
2. The shelves hold the goods: billions of small identical items, in order.
3. The clipboard says: shelf 1 holds the vocabulary table, 128,256 rows by 4,096 columns. Shelf 2 holds layer zero’s query matrix, 4,096 by 4,096.
4. Without the clipboard, the warehouse is a single undifferentiated pile.
5. With the clipboard, any item can be found by address in constant time.
6. That is exactly the layout of a modern weights file: a small header naming every block and giving its offset, then one enormous block of raw bytes.
7. The reason for that layout is that a program can then map the file into memory and read one block without loading the rest.

Where this comparison breaks: warehouse goods have individual meaning. These numbers do not. No single weight corresponds to a fact or a word. The meaning is entirely in the pattern across billions of them, which is why you cannot open a model file and read anything out of it.

PLAIN49.11.3 a worked example#

1. A real directory listing, from a published 8-billion-parameter Llama-family instruction model. Sizes rounded.

config.json                          about  800 B
generation_config.json               about  200 B
model-00001-of-00004.safetensors           4.98 GB
model-00002-of-00004.safetensors           5.00 GB
model-00003-of-00004.safetensors           4.92 GB
model-00004-of-00004.safetensors           1.17 GB
model.safetensors.index.json         about   24 KB
special_tokens_map.json              about  300 B
tokenizer.json                             9.09 MB
tokenizer_config.json                about   55 KB
LICENSE, README.md, USE_POLICY.md    a few KB

2. Total weights: about 16.1 GB, which is 8.03 billion parameters times 2 bytes for bf16. The arithmetic checks out exactly.
3. The four .safetensors files are the weights. The index.json maps each tensor name to the file it lives in. Everything else is under 10 MB.
4. The config, abbreviated but real in structure:

{
  "architectures": ["LlamaForCausalLM"],
  "hidden_size": 4096,
  "intermediate_size": 14336,
  "num_hidden_layers": 32,
  "num_attention_heads": 32,
  "num_key_value_heads": 8,
  "vocab_size": 128256,
  "max_position_embeddings": 131072,
  "rms_norm_eps": 1e-05,
  "rope_theta": 500000.0,
  "torch_dtype": "bfloat16"
}

5. Every number in that file is a hyperparameter chosen by a human before training. None of them was learned. Chapter 47 works through how these numbers produce the 8.03 billion total.
6. And here is what is actually inside the weights file: tensor names, shapes and types, as a loader would report them.

model.embed_tokens.weight              [128256, 4096]  BF16
model.layers.0.input_layernorm.weight  [4096]          BF16
model.layers.0.self_attn.q_proj.weight [4096, 4096]    BF16
model.layers.0.self_attn.k_proj.weight [1024, 4096]    BF16
model.layers.0.self_attn.v_proj.weight [1024, 4096]    BF16
model.layers.0.self_attn.o_proj.weight [4096, 4096]    BF16
model.layers.0.mlp.gate_proj.weight    [14336, 4096]   BF16
model.layers.0.mlp.up_proj.weight      [14336, 4096]   BF16
model.layers.0.mlp.down_proj.weight    [4096, 14336]   BF16
...  (the same 9 tensors for layers 1 through 31)
model.norm.weight                      [4096]          BF16
lm_head.weight                         [128256, 4096]  BF16

7. Note the key and value projections are 1024 rows, not 4096. That is grouped query attention: 8 key-value heads shared across 32 query heads.
8. Note the names are hierarchical strings. They are not a standard. They are the convention of one library, and a different library uses different names for the same tensors, which is why conversion scripts exist.

PLAIN49.11.4 what is really happening inside#

1. How a safetensors file is laid out, byte by byte.
2. First 8 bytes: a little-endian unsigned integer giving the length of the header in bytes.
3. Next N bytes: a JSON object. Each key is a tensor name. Each value gives the data type, the shape, and the start and end byte offsets of that tensor.
4. Everything after that: raw tensor bytes, back to back, no separators.
5. That is the entire format. It is deliberately boring, and the boringness is the security property.
6. Compare that with the format it replaced. PyTorch’s .bin files used Python’s pickle, which is not a data format but a program format: it contains instructions telling Python how to rebuild an object.
7. Those instructions can include “import this module and call this function”. So loading a model file could run arbitrary code on your machine.
8. That is not theoretical. Malicious models with code hidden in pickle payloads have been found on public model hubs.
9. safetensors cannot do that, because a JSON header plus raw bytes has no way to express an instruction.
10. Because the offsets are known in advance, the file can also be memory-mapped and tensors copied straight to the accelerator with no intermediate parse. That is where the speed comes from.

TECHNICAL49.11.5 the engineer’s version#

1. The format comparison you should be able to give from memory:

2. safetensors was created at Hugging Face and became the default published format across the hub during 2023. Design goals in the project’s own words: safe against code execution, zero-copy, memory-mappable, and fast.
3. The pickle risk was severe enough that PyTorch changed the default. From PyTorch 2.6, released January 2025, torch.load defaults to weights_only=True, which refuses to execute arbitrary globals. Before that the default was unsafe and most tutorials did not mention it.
4. GGUF is the format of the llama.cpp and ggml ecosystem, introduced in August 2023 to replace the earlier GGML, GGMF and GGJT formats. Its defining feature is a typed key-value metadata block, so new fields can be added without breaking old readers.
5. GGUF is single-file and self-contained: architecture metadata, tokenizer vocabulary and merges, and quantized tensors all live in the one file. That is why a GGUF download needs no config or tokenizer alongside it.
6. GGUF bakes quantization into the file. The type names you will see include F32, F16, BF16, Q8_0, Q6_K, Q5_K, Q4_K, Q4_0, and the IQ series such as IQ4_XS and IQ2_XXS. Around 40 types are defined. Chapter 47 covers what the K-quants actually do.
7. ONNX, Open Neural Network Exchange, was launched by Microsoft and Facebook in September 2017. It stores a computation graph plus weights in a Protocol Buffers message, so a model trained in one framework can run in another. It does not execute Python, but it does describe operations, and custom operators can load native code, so it is not risk-free in the way safetensors is.
8. Practical file-size arithmetic, which you can do in your head:

9. Quantized types are slightly larger than the nominal bit count suggests because each block of 32 to 256 weights carries a scale, and sometimes an offset, stored at higher precision.
10. What is emphatically not in a weights file: the training data, the data mixture, the training code, the hyperparameters used during training, the optimizer state, the random seed, or any record of how the numbers arrived.
11. Commands that inspect these files: safetensors exposes a Python API where safe_open(path, framework="pt") lists keys without loading data; gguf-dump prints GGUF metadata and tensor tables; python -m onnx.checker validates an ONNX graph; and huggingface-cli scan-cache shows what you have downloaded and how much disk it is using.

WORDS49.11.6 remember these#

1. Weights file — the big file of numbers — a binary container of tensors, sized at parameter count times bytes per parameter.
2. Config — the shape description — a JSON file of architecture hyperparameters needed to interpret the tensor layout.
3. Tensor name — the address of one rectangle of numbers — a hierarchical string key such as model.layers.7.mlp.up_proj.weight.
4. safetensors — the safe modern weights format — an 8-byte length, a JSON header of dtypes, shapes and offsets, then raw bytes.
5. Pickle — Python’s object format that can run code — the mechanism behind legacy .bin checkpoints and their arbitrary-code-execution risk.
6. GGUF — a single self-contained file for local inference — metadata, tokenizer and quantized tensors together, used by llama.cpp.
7. Memory mapping — reading a file without copying it into memory first — using the operating system’s virtual memory to page tensor bytes on demand.

49.12 Open weights versus open source#

PLAIN49.12.1 in simple words#

1. This is the part of the chapter where the words are genuinely abused, so we will be slow and exact.
2. There are four separate things people mix together. Each is independent of the others. A model can have any combination.
3. Open weights means the number file can be downloaded and run. That is all it means. Nothing about permission, nothing about data.
4. Open source means something specific. It is a definition maintained by the Open Source Initiative since 1998, and it is about the licence granting you freedom to use, study, modify and share, without restrictions on who you are or what field you work in.
5. Open data means the training corpus itself is published, so somebody else could rebuild the model rather than just run it.
6. The licence terms are separate again. A model can be open-weight and carry a licence that forbids commercial use, or forbids competitors, or forbids certain applications.
7. The abuse works like this. A company publishes only the weights, under a licence with restrictions, and calls the result open source.
8. Almost nothing about it is open source. The source, in the sense that matters, would be the data and the training code, and neither was released.
9. A more honest word, which some companies now use, is “open weights”.
10. Why does the distinction matter practically? Because you cannot verify what is in a model you cannot inspect the data for, you cannot reproduce it, and you may find at deployment time that the licence forbids your use.

PLAIN49.12.2 a picture in your head#

1. Think about a restaurant and a famous sauce.
2. Selling you a jar of the sauce is open weights. You have the thing. You can taste it, use it, put it on your food.
3. Publishing the recipe is open source. Now you can make it, change it, improve it, and check whether the ingredients are what they claimed.
4. Publishing where every ingredient was bought is open data.
5. And the label on the jar might say “not for resale” or “not for use in commercial kitchens”. That is the licence.
6. Selling the jar and calling it “open source sauce” is the thing being objected to. You cannot make the sauce. You cannot check the ingredients. You have a jar.

Where this comparison breaks: a sauce could be reverse-engineered by a skilled cook. Model weights cannot be reverse-engineered into their training data in any practical sense. Also, the jar of sauce is not itself useful for making more sauce, whereas model weights genuinely are useful for building further models, which is why open weights deliver real value even without the recipe.

PLAIN49.12.3 a worked example#

1. What real models actually release. Status as of mid-2026.

2. And the licences, which is a separate question from the table above.

3. Read those two tables together and the picture is clear. Full openness, meaning weights plus data plus code plus a permissive licence, exists but is rare, and comes mostly from research institutes rather than from companies.
4. The single most-cited restriction, from Meta’s Llama licences: if your products had more than 700 million monthly active users in the month before the release date, you must ask Meta for a separate licence.
5. In practice that clause names a handful of companies. It is aimed at competitors, and its presence is precisely why the licence is not open source: open source licences cannot discriminate against a class of user.
6. Meta’s licences also require the words “Built with Llama” on derivative products and require derived model names to begin with “Llama”, and bind you to an Acceptable Use Policy that can be updated after you accept it.
7. Llama’s multimodal models have additionally carried a restriction excluding users domiciled in the European Union, with no reason given in the licence.

PLAIN49.12.4 what is really happening inside#

1. Here is what is normally withheld even by generous open-weight releases.
2. The training data. Which documents, from where, in what proportion.
3. The data mixture. Even when categories are named, the recipe is not.
4. The training code. The actual distributed training harness, the data loader, the parallelism configuration.
5. The hyperparameters. Learning rate schedule, batch ramp, warmup, weight decay, initialization, and the dozens of small choices that make a run work.
6. The post-training recipe. The SFT data, the preference data, the reward model, the safety data. This is often guarded more closely than the pretraining recipe, because it is where the product feel comes from.
7. The evaluation and ablation results that led to those choices.
8. What you get instead is a file that reproduces one point in that space and no way to get to any neighbouring point except by guessing.
9. There are good reasons for some of this. Some data cannot legally be redistributed. Some of it is licensed under terms that forbid it. Some of it would expose the lab to litigation of the kind section 49.2 described.
10. And there are competitive reasons for the rest, which nobody disputes.

TECHNICAL49.12.5 the engineer’s version#

1. Precise definitions, in the terms the field actually uses.
2. Open weights: model parameters are distributed for download. Says nothing about licence, data, code or reproducibility. Sometimes called “open access” or, unhelpfully, “open model”.
3. Open source: the Open Source Definition, maintained by the Open Source Initiative since 1998, requires among other things free redistribution, availability of source code, permitted derived works, no discrimination against persons or groups, and no restriction on fields of endeavour. Those last two are the clauses most model licences fail.
4. Open data: the training corpus is published under a licence permitting use. Examples: the Pile from EleutherAI in 2020, ROOTS from BigScience in 2022, Dolma and Dolma 3 from Ai2, FineWeb from Hugging Face under ODC-By 1.0.
5. Licence class, in three tiers you should be able to name: permissive open source, meaning Apache 2.0 or MIT; restricted custom licences, meaning Meta’s Community licences and Google’s Gemma Terms, which are contracts with conditions; responsible-AI licences, meaning the RAIL and OpenRAIL family from 2022 onwards, which are explicitly use-restricted by design and were never claimed to be open source by their authors.
6. Fully open releases, with what each actually shipped: Pythia, EleutherAI 2023: weights at 8 sizes, the Pile as training data in exact training order, training code, and 154 intermediate checkpoints per model, under Apache 2.0. Built for reproducible research on training dynamics, and still the reference example. OLMo, Ai2, February 2024, then OLMo 2 in January 2025 and OLMo 3 in November 2025: weights, full training data, training code, evaluation harness, logs and intermediate checkpoints from every stage. OLMo 3’s release included Dolma 3, a 9.3-trillion-token corpus, the 5.9 trillion token pretraining mix used for the 7B and 32B base models, the post-training data suite, and the data tooling. BLOOM, BigScience 2022: 176B parameters, the ROOTS corpus, the training code, and a detailed record of the process, released under a RAIL licence that carries use restrictions and therefore is not OSI open source.
7. Open-weight, closed-data, permissive licence, which is the commercially dominant pattern: Mistral 7B (September 2023, Apache 2.0), Qwen3 (2025, Apache 2.0), DeepSeek-R1 (January 2025, MIT), and OpenAI’s gpt-oss-120b and gpt-oss-20b (5 August 2025, Apache 2.0).
8. gpt-oss is worth a note because it shows the pattern clearly. Apache 2.0 on the weights, plus published inference reference code, the harmony prompt renderer and the o200k_harmony tokenizer. No training data, no training code. Permissive licence, closed recipe.
9. Established fact: open weights allow local running, fine-tuning, adapter training, quantization, red-teaming and independent measurement. Established fact: they do not allow reproduction, data auditing, or verification of any claim about what the model was trained on. Marketing claim: describing a weights-only release under a restricted licence as “open source”. This is the specific abuse the reader asked about.
10. One practical consequence for engineers. Open weights are the only category that survives a vendor withdrawing a model. A downloaded file keeps working when an API endpoint is deprecated, and API deprecation happens on a twelve-to-twenty-four-month cycle at every major provider.

WORDS49.12.6 remember these#

1. Open weights — the numbers are downloadable — parameters published for use, independent of licence, data or code availability.
2. Open source — a licence granting the four freedoms without discrimination — as defined by the Open Source Initiative’s Open Source Definition since 1998.
3. Open data — the training corpus is published — the actual documents, or a reproducible recipe for assembling them, under a usable licence.
4. Permissive licence — almost no conditions — Apache 2.0 or MIT, allowing commercial use, modification and redistribution.
5. Community licence — a contract with conditions attached — Meta’s and Google’s custom terms, with user thresholds, naming rules and use policies.
6. OpenRAIL — a licence that restricts uses by design — the Responsible AI Licence family, openly not open source and honest about it.
7. Open washing — calling a restricted release open — marketing a weights-only or use-restricted model as open source.

49.13 The Open Source Initiative’s position#

PLAIN49.13.1 in simple words#

1. The Open Source Initiative is the body that has maintained the definition of open source software since 1998, and it decides which licences qualify.
2. For twenty-five years that definition was about source code. Models are not source code, so it did not obviously apply.
3. So the OSI ran a public process with many contributors and published the Open Source AI Definition version 1.0 on 28 October 2024.
4. It starts from the same four freedoms as software: use the system for any purpose, study how it works, modify it, and share it, modified or not.
5. To exercise those freedoms you need what it calls the preferred form to make modifications, and it says that means three things must be available.
6. Data information: enough detail about the training data that a skilled person could build a substantially equivalent system.
7. Code: the complete source code used to train and run the system, under an OSI-approved licence.
8. Parameters: the weights and other settings, under OSI-approved terms.
9. Read point 6 carefully, because it is the compromise at the heart of the whole thing. It does not require publishing the data itself. It requires publishing enough information about it.
10. By this definition, most models marketed as open source do not qualify.
11. The OSI has said so directly, naming names.

PLAIN49.13.2 a picture in your head#

1. Think about a food label and a recipe again, one last time.
2. The strictest possible rule would be: hand over every ingredient you used, physically, so anyone can remake the dish exactly.
3. That is impossible for some ingredients. You cannot hand over a bottle of wine you drank, and you cannot hand over a book you licensed for one use.
4. So the OSI settled on: publish the recipe in enough detail that a competent cook could go and buy equivalent ingredients and make substantially the same dish.
5. That satisfies nobody entirely. Purists say a recipe is not the ingredients. Companies say even the recipe is a trade secret.
6. And it is precisely because that middle position annoys both sides that it is worth understanding exactly.

Where this comparison breaks: two cooks following the same recipe get similar results. Two labs following the same data description do not, because model outcomes depend on ordering, seeds, hardware numerics and hundreds of unpublished choices. “Substantially equivalent” is doing a great deal of work in that sentence and nobody has tested it in practice at frontier scale.

PLAIN49.13.3 a worked example#

1. The OSI published its own assessment of real models against the definition. Its stated results, which it describes as part of the definitional process rather than as certifications:

2. Note that Mixtral is in the failing list despite being under Apache 2.0. The licence is fine. The data information is missing. Both are required.
3. Note that BLOOM is in the middle list despite publishing its full training corpus. The data is fine. The RAIL licence restricts uses. Both are required.
4. Those two rows together are the clearest possible illustration that open weights, open data and an open licence are three independent things.
5. The OSI’s specific objections to Meta’s Llama licences: it fails freedom zero, the freedom to use the system for any purpose; it fails the Open Source Definition’s point 5 by discriminating against a class of user via the 700-million-user threshold; and it fails point 6 by restricting fields of endeavour through the Acceptable Use Policy. The OSI has also pointed to the exclusion of European Union users from certain models, unexplained.
6. The OSI uses the term “open washing” for marketing a restricted release as open source.

PLAIN49.13.4 what is really happening inside#

1. Now the counter-arguments, given fairly, because this is a live dispute and not a settled question.
2. Training data cannot always be redistributed. Much of it is copyrighted text that the lab has a defensible claim to train on but no right to republish. Publishing it would be a separate and much larger legal exposure.
3. Some of it must not be redistributed. Web-scale corpora contain personal data. Publishing the corpus can conflict with privacy law even where training on it does not.
4. Open weights deliver most of the practical benefit. With weights you can run locally, fine-tune, quantize, audit behaviour, red-team, and avoid depending on a vendor. Very few people were ever going to retrain a frontier model even with the full data.
5. Reproduction is impossible anyway at this scale. Even with the corpus, rebuilding a frontier model costs tens of millions of dollars, so the freedom being protected is theoretical for almost everyone.
6. And there is a criticism from the opposite direction, which is important.
7. Some free-software advocates argued the OSI’s compromise is too weak: that allowing “data information” instead of data lets companies claim the label without the substance, which is the very open washing the definition was meant to stop.
8. So version 1.0 is attacked from both sides: too strict for industry, too loose for purists. The OSI itself has said the definition will be revised.
9. The honest version: there is no consensus. The OSI has the strongest claim to define “open source” because it has defined it for software since 1998, but no legal authority, and companies are free to ignore it and do.

TECHNICAL49.13.5 the engineer’s version#

1. What the definition requires in full, as three components. Data information: complete descriptions of all data used, including provenance, scope and characteristics, how it was obtained and selected, labelling procedures, and cleaning and filtering methodologies, in enough detail that a skilled person can build a substantially equivalent system. Code: the complete source used for data processing and filtering, training, with all training arguments, validation, testing and inference, under OSI-approved licences. Parameters: model weights and other configuration settings, including intermediate checkpoints and final optimizer states, under OSI-approved terms.
2. That third clause is stricter than people notice. Intermediate checkpoints and optimizer state are almost never published. Pythia and OLMo do publish checkpoints, which is exactly why they pass.
3. Timeline of the dispute, dated: 1998, the Open Source Definition is published. 2022, RAIL and OpenRAIL licences appear, explicitly use-restricted. February 2023, Meta releases LLaMA 1 weights to researchers on request. July 2023, Llama 2 is released under a community licence and widely described in press coverage as open source. 2023 to 2024, the OSI runs a co-design process with contributors from industry, academia and civil society. 28 October 2024, Open Source AI Definition 1.0 is published at the All Things Open conference. Through 2025 and 2026, the argument continues without resolution, with further drafts under discussion.
4. Parallel regulatory pressure, which may matter more than the definitional argument in the end. The European Union’s AI Act, in force from 1 August 2024 with general-purpose model obligations applying from 2 August 2025, requires providers to publish a sufficiently detailed summary of training content and to have a copyright policy. It also provides partial exemptions for models released under free and open-source licences, which gives the definition of “open” direct legal consequences.
5. So a term that was a marketing argument in 2023 became a compliance question in 2025. That is the practical reason to get it right.
6. Established fact: the OSAID 1.0 text, its date, its three required components, and the OSI’s published assessments of named models. Active dispute: whether the data-information compromise is correct, whether version 1.0 will hold, and whether regulators will adopt it. Marketing claim: any company statement that its weights-only release “is open source” without addressing data information or licence restrictions.

WORDS49.13.6 remember these#

1. OSAID — the open source rulebook for AI — the Open Source AI Definition 1.0, published by the OSI on 28 October 2024.
2. Data information — a description good enough to rebuild from — sufficiently detailed data documentation to build a substantially equivalent system.
3. Four freedoms — use, study, modify, share — the freedoms the definition requires an open source AI system to grant without restriction.
4. OSI-approved licence — a licence on the OSI’s published list — the test a licence must pass to be called open source.
5. Open washing — claiming openness you did not deliver — marketing a restricted or weights-only release using the open source label.

49.14 Model releases in practice#

PLAIN49.14.1 in simple words#

1. When a model is released, the file is the smallest part of the event.
2. Around it sit documents, gates and processes, and knowing what they should contain lets you judge a release in a few minutes.
3. The model card is the main document. The idea comes from a 2019 paper by Mitchell and colleagues, and it is now near-universal practice.
4. It should say: what the model is for, what it is not for, what it was trained on in general terms, how it scores on named benchmarks, what its known limitations are, and what licence applies.
5. The system card is a bigger document about the deployed product rather than the raw model: what safety testing was done, what was found, what mitigations were added, and what risks remain.
6. Staged release means publishing in steps rather than all at once, to watch what happens.
7. Red-teaming means paying people to attack the model before release and trying to make it do what it should not.
8. Watermarking means trying to mark generated output so it can be detected later. It works better for images than for text.
9. And a licence gate on a hosting site means you must accept terms, and often give your name and email, before you can download.
10. What is missing from a release tells you as much as what is present.

PLAIN49.14.2 a picture in your head#

1. Think of the paperwork that comes with a new medicine.
2. There is a leaflet saying what it treats, the dose, who must not take it, and the side effects observed in trials. That is the model card.
3. There is a much longer regulatory dossier describing every trial, its results, and the risks that remain. That is the system card.
4. There is a phased rollout, first to a limited group, then wider. That is the staged release.
5. And there is a pharmacist who checks who you are before handing it over. That is the licence gate.
6. Now imagine a medicine arriving with no leaflet at all, and ask yourself what that absence tells you.

Where this comparison breaks: medicines are regulated by law with mandatory content and legal consequences for omissions. Model cards are, in most places and as of mid-2026, entirely voluntary in content and quality. The European Union’s AI Act is the first serious attempt to make parts of this mandatory, and it applies to general-purpose models only from 2 August 2025.

PLAIN49.14.3 a worked example#

1. The checklist to run against any release. Present or absent, and what the absence means.

2. Real examples of each end of the spectrum. Meta’s Llama 3.1 model card gives GPU-hours, energy, carbon, knowledge cutoff of December 2023, benchmark tables and licence, but describes training data only as “publicly available sources” with category percentages.
3. Ai2’s OLMo releases give all of the above plus the corpus itself, the data tooling, the training logs and intermediate checkpoints.
4. Closed model releases typically give benchmark tables and a system card, and no training data information at all.

PLAIN49.14.4 what is really happening inside#

1. What happens in the weeks before a release, roughly in order.
2. Internal evaluation on held-out benchmarks, plus contamination checks to confirm the benchmarks were not in the training data.
3. Capability evaluations against a published safety framework, testing for dangerous capabilities in specific named categories.
4. Red-teaming, both internal and by hired external groups, attempting jailbreaks and misuse. Findings feed back into more safety fine-tuning.
5. Sometimes external review, by a government safety institute or a contracted evaluation organization, before release.
6. Then a decision, then the documents, then the gate, then publication.
7. Staged release has a clear historical example. In February 2019 OpenAI announced GPT-2 and released only the 124-million-parameter version, citing misuse concerns. It released 355M in May, 774M in August, and the full 1.5B model in November 2019, publishing its reasoning at each step.
8. That decision was heavily criticized at the time as overcautious, and it set the template that every lab now follows in some form.
9. Note what staged release cannot do for open weights. Once the file is downloaded it cannot be recalled, and as section 49.5 noted, the safety training can be removed by fine-tuning for a few hundred dollars.

TECHNICAL49.14.5 the engineer’s version#

1. Model cards: proposed in “Model Cards for Model Reporting”, Mitchell and colleagues, 2019. The companion idea for data is “Datasheets for Datasets”, Gebru and colleagues, 2018. Both are now conventions rather than standards, though Hugging Face enforces a minimal card structure on its hub.
2. System cards: OpenAI published the GPT-4 System Card in March 2023, separating deployment risk analysis from the technical report. Anthropic and Google now publish equivalents for major releases. Contents typically include red-team findings, capability evaluations against a framework, refusal rates, and known failure modes.
3. Named safety frameworks you should recognize, with dates: Anthropic’s Responsible Scaling Policy, September 2023; OpenAI’s Preparedness Framework, December 2023; Google DeepMind’s Frontier Safety Framework, May 2024. All define capability thresholds that trigger additional controls. All are voluntary commitments, not regulation.
4. Government bodies: the UK AI Safety Institute was announced in November 2023 and renamed the AI Security Institute in February 2025. The US AI Safety Institute was established in 2023 and renamed the Center for AI Standards and Innovation in June 2025. Both have run pre-deployment evaluations under agreements with major labs.
5. Watermarking. Google DeepMind’s SynthID for images was announced in August 2023, and SynthID-Text was published in the journal Nature in October 2024, using a tournament sampling scheme that biases token choice in a detectable pattern. Google reported it does not measurably degrade quality.
6. The honest position on text watermarking: it is detectable while the text is unedited, and it degrades under paraphrasing, translation or moderate editing. It also requires the generator to cooperate, so it does nothing about open-weight models where the sampler can simply be replaced. Treat claims of robust text watermarking as marketing.
7. The C2PA standard, from a coalition including Adobe and Microsoft, takes the other approach: cryptographically signed provenance metadata attached to a file. It survives editing by signed tools and is destroyed by screenshots.
8. Licence acceptance gates: Hugging Face supports gated repositories, where a user must accept terms and often supply contact details, with optional manual approval. Meta and Google both use them. The gate is a contractual mechanism, not a technical one; it does not prevent redistribution.
9. Regulatory contents, dated. Under the European Union’s AI Act, providers of general-purpose models have obligations from 2 August 2025 including technical documentation, information for downstream providers, a copyright policy, and a sufficiently detailed public summary of training content, with partial exemptions for free and open-source releases and additional duties above the 10-to-the-25-FLOPs systemic-risk threshold.
10. Practical advice: read the licence file itself, not the summary on the model page; check the knowledge cutoff before relying on recency; check whether the card names benchmarks or just claims performance; and check whether evaluation numbers are self-reported or independently reproduced.

WORDS49.14.6 remember these#

1. Model card — the leaflet that comes with a model — a structured document of intended use, training description, evaluations, limitations and licence.
2. System card — the safety dossier for the deployed product — red-team findings, capability evaluations, mitigations and residual risks.
3. Staged release — publishing in steps to watch the effect — incremental release of larger checkpoints or wider access over time.
4. Red-teaming — paying people to attack it before release — structured adversarial testing for jailbreaks, misuse and dangerous capability.
5. Watermarking — marking generated output so it can be detected — biased sampling for text, or signed provenance metadata for media.
6. Gated repository — you must accept terms to download — a hosting-site mechanism collecting acceptance and often identity before access.

49.98 Common wrong ideas#

1. Wrong: the model was trained on the internet, so it knows everything. Right: it was trained on a heavily filtered slice of a web crawl, plus books, code and papers, and typically 80 to 95 per cent of the crawl was discarded. It has a knowledge cutoff date, it never saw most paywalled or private material, and what it did see it stored lossily as statistical patterns rather than as retrievable text.
2. Wrong: fine-tuning teaches it new facts. Right: fine-tuning teaches format, style and behaviour. Knowledge comes from pretraining because knowledge comes from data volume, and a fine-tuning set is perhaps one thirty-thousandth the size of the pretraining set. Fine-tuning on unfamiliar facts is learned slowly and measurably increases hallucination elsewhere. Use retrieval for facts.
3. Wrong: open weights means open source. Right: they are separate. Open weights means the numbers are downloadable. Open source, by the Open Source Initiative’s definition of October 2024, additionally requires the training code, sufficiently detailed data information, and a licence with no discrimination against users or fields of use. Mixtral is Apache 2.0 and fails; BLOOM publishes its corpus and fails.
4. Wrong: RLHF makes the model truthful. Right: it makes the model produce answers that human raters prefer. Where raters prefer confident, agreeable, long or well-formatted answers, that is what gets reinforced. This is why sycophancy and length inflation are standard failure modes, and why OpenAI had to roll back a GPT-4o update in April 2025 for exactly this reason.
5. Wrong: training and running cost about the same. Right: they differ by roughly thirteen orders of magnitude per query. Training an 8-billion model on 15 trillion tokens is about 7.2e23 FLOPs. Generating one token from it is about 1.6e10 FLOPs. Training equals roughly 45 trillion generated tokens. For a busy service, cumulative inference overtakes training cost within months.
6. Wrong: a bigger model is always better. Right: Chinchilla in 2022 showed a 70-billion model beating a 280-billion one at equal compute because it saw four times more data. Data quantity, data quality and post-training all matter as much as parameter count.
7. Wrong: labs publish what they trained on. Right: GPT-3 published a mixture table in 2020. Almost nobody does now. GPT-4’s technical report explicitly declined. Any claim about a closed model’s training data, including a claim about what it did not include, is unverifiable from outside.
8. Wrong: the safety training is part of the model and cannot be removed. Right: for open weights it can be substantially removed by fine-tuning on a few hundred adversarial examples, for a few dollars. Safety behaviour is a thin post-training layer over an unchanged base capability.
9. Wrong: a training run is one program on one big computer. Right: it is thousands of accelerators running in lockstep, split three ways across data, tensors and layers, and hardware fails constantly. Meta reported 419 unexpected interruptions in 54 days on one run.
10. Wrong: the weights file contains the training data somewhere inside it. Right: it contains only numbers arranged in named tensors, plus a config and a tokenizer. Verbatim regurgitation of memorized training text does happen, especially for text duplicated many times in the corpus, but the data is not stored or retrievable in any ordinary sense.

49.99 Chapter summary in 20 lines#

1. A model is made in a chain: collect data, clean it, train a tokenizer, pretrain, post-train, evaluate, release. Pretraining is 95 to 99 per cent of the compute and post-training is almost all of the visible behaviour.
2. Data comes from web crawls such as Common Crawl, which published 2.42 billion pages and 419 TiB uncompressed in its July 2025 archive alone, plus books, code, papers, licensed corpora and synthetic text.
3. Cleaning discards most of it: URL blocking, text extraction, language identification, exact and MinHash fuzzy deduplication, heuristic and classifier quality filters, toxicity and PII removal, and benchmark decontamination, then deliberate mixing ratios over the survivors.
4. FineWeb’s published pipeline uses 5-gram shingles with 14 bands of 8 hashes, which catches near-duplicates above about 0.75 Jaccard similarity 77 per cent of the time and above 0.9 essentially always.
5. Copyright is unsettled and actively litigated. Judge Alsup held in June 2025 that training on lawfully bought books was fair use while retaining pirated copies was not, and Anthropic settled for a reported 1.5 billion dollars.
6. Most labs stopped publishing their data mixture after GPT-3 in 2020, so claims about a closed model’s training data are unverifiable from outside.
7. Pretraining has exactly one objective: next-token prediction, scored by cross-entropy loss in nats, starting at the natural log of the vocabulary size and typically ending between 1.4 and 2.0.
8. One step is a forward pass over a batch of millions of tokens, a loss, a backward pass costing about twice the forward, and an AdamW update. Llama 3 405B used a peak learning rate of 8e-5, 8,000 warmup steps and cosine decay to 8e-7 over 1.2 million steps.
9. Llama 3.1 405B is the best-documented real run: over 16,384 H100s, 30.84 million GPU-hours, about 15.6 trillion tokens, roughly 78 days of wall clock and an estimated 62 to 123 million dollars of rented compute.
10. Training compute is approximately 6ND, parameters times tokens times six. It reproduces GPT-3’s reported 3.14e23 and Llama 3.1 405B’s 3.8e25, and the European Union’s AI Act uses 10 to the 25 as a systemic-risk threshold.
11. Kaplan and colleagues in January 2020 found smooth power laws and advised spending on size; Hoffmann and colleagues in March 2022 showed models were badly undertrained and that about 20 tokens per parameter is optimal.
12. Since 2023 labs deliberately overtrain small models far past that, because inference cost dominates over a deployed model’s life. Llama 3.1 8B was trained at about 1,875 tokens per parameter, roughly 94 times Chinchilla.
13. A base model continues text and cannot be used as a chatbot: no turns, no roles, no stopping rule and no refusal behaviour. Ask it a question and it will often produce more questions.
14. SFT fixes that with tens of thousands to a few million demonstration pairs, a chat template of special tokens such as Llama 3’s <|eot_id|>, loss masked to the assistant’s tokens, and a learning rate ten times lower.
15. Preference learning goes further: collect comparisons, train a reward model, optimize with PPO under a KL penalty. Its characteristic failure is reward hacking, exemplified by OpenAI’s sycophantic GPT-4o update of April 2025.
16. DPO in May 2023 removed the separate reward model algebraically and most smaller teams moved to it; constitutional and RLAIF methods replace human labels with written principles and an AI judge; RLVR uses checkable answers in maths and code and is the engine behind reasoning models.
17. Post-training shapes behaviour and cannot efficiently install knowledge, because the data is thirty thousand times smaller and the learning rate is set gently on purpose. The boundary is real but fuzzy, and mid-training blurs it deliberately.
18. LoRA freezes the base and trains thin low-rank matrices, cutting a 7B fine-tune from about 112 GB to about 17 GB; QLoRA adds a 4-bit base and cuts it to about 7 GB, which put a 65B fine-tune on one 48 GB card in 2023.
19. Weights are a binary file of numbers plus a config and a tokenizer. safetensors is an 8-byte length, a JSON header of dtypes, shapes and offsets, then raw bytes, and it exists because pickle-based .bin files can execute arbitrary code on load; GGUF bundles quantization and tokenizer.
20. Open weights, open source, open data and licence terms are four independent things. The Open Source AI Definition 1.0 of 28 October 2024 requires code, parameters and data information sufficient to rebuild a substantially equivalent system, and by it most models marketed as open source, including Llama, do not qualify. That remains a live dispute, not a settled question.