You Will Understand 10 Years of LLMs

My new video, In 19 Minutes, You'll Understand 10 Years of LLMs, follows one question through every era of language models: "What is the capital of France?"

This post is the developer version. It goes through every chapter of the video with the actual terms and numbers. Where the video shows a model's output, we ran an open model ourselves, so you get the code to reproduce it.

pip install torch transformers accelerate 

All outputs below are what we got on an RTX 4070 Ti. Yours may differ slightly with hardware and precision.

TL;DR timeline

0. The quiz page (Experiment 1)

Before chatbots, a language model's only job was to continue text. Ask GPT-2 XL (1.5B parameters, 2019):

from transformers import AutoTokenizer, AutoModelForCausalLM import torch name = "openai-community/gpt2-xl" tok = AutoTokenizer.from_pretrained(name) model = AutoModelForCausalLM.from_pretrained(name).eval() ids = tok("What is the capital of France?", return_tensors="pt").input_ids out = model.generate(ids, max_new_tokens=60, do_sample=False, pad_token_id=tok.eos_token_id) print(tok.decode(out[0][ids.shape[1]:])) 

The capital of France is Paris. What is the capital of the United States? The capital of the United States is Washington, D.C. What is the capital of the United Kingdom? The capital of the United Kingdom is London. What 

It's not answering; it's writing a quiz page. We'll come back to why.

1. Before 2017: the road to attention

Shannon's game (1948, 1951)

Claude Shannon asked how predictable English is. Letters don't come at random: counted in Pride and Prejudice, after a T comes an H 29% of the time; after a Q, a U 99.85% of the time.

In 1948 he generated text that follows these odds with a book: pick a letter, open another page, find that letter, copy the one after it, repeat. Letter pairs give almost-words; word pairs give "The head and in frontal attack on an English writer that the character of this point…" Not English, but closer. The more context, the closer.

In 1951 he made it a game with people: guess the next letter. On one passage they were right on the first try 79 times out of 102. He estimated that a letter of English carries about 1 bit of new information, against about 4.75 bits if letters were random.

That's the objective LLMs still train on: predict the next token.

Words become vectors (2013)

A word ID (king = #4812) says nothing about meaning. word2vec (Mikolov et al., 2013) gives each word a vector of a few hundred numbers (300 in the classic setup), learned by predicting a word's neighbours over billions of words. Words used in similar contexts end up close, and the space has structure: king − man + woman ≈ queen (the nearest word, excluding the three inputs).

The limit: one vector per word. The "bank" of a river and the bank with your money get the same vector.

RNN, LSTM, seq2seq (1997–2014)

A recurrent neural network reads one token at a time and updates a hidden state, a running memory. Two problems: the state is overwritten at every step, so early words fade; and in training, the error signal has to flow backwards through every step and shrinks on the way (vanishing gradients).

The LSTM (Hochreiter & Schmidhuber, 1997) adds a cell state, a memory lane that runs straight through every step, with gates that decide what to write, what to read and (since 2000) what to forget.

By 2014, sequence to sequence translated with two of them: an encoder packs the English sentence into one fixed vector (8,000 numbers in Google's 4-layer version), and a decoder unpacks it into French, one word at a time. On basic encoder-decoders, quality dropped as sentences got longer: one fixed vector is a bottleneck.

Attention (2014)

Bahdanau, Cho and Bengio kept a vector for every source word, and at each output step let the decoder compute a weighted mix of all of them. To write "zone", it weighs "Area" most. Long sentences stopped falling apart.

One problem remained: everything was still sequential. Step 50 needs the state from step 49, while GPUs are built to do thousands of operations at once.

2. 2017: the Transformer

"Attention Is All You Need" (Vaswani et al., Google, June 2017; author order random, the paper says so) kept attention and removed recurrence. Every token attends to every other token, all at once: self-attention.

What attention does (Experiment 2)

Google's example: "The animal didn't cross the street because it was too tired" (it = the animal) vs "…too wide" (it = the street). Each token produces a query; every token has a key and a value. Query-key matches (dot products, scaled, then softmax) give each token a share, and "it" takes in a weighted mix of the values.

You can watch this in BERT (12 layers × 12 heads = 144 heads). We searched for the head whose focus flips most between the two sentences: layer 7, head 11 (index 6 and 10 in code):

from transformers import AutoModel btok = AutoTokenizer.from_pretrained("bert-base-uncased") bert = AutoModel.from_pretrained("bert-base-uncased", output_attentions=True).eval() for end in ["tired", "wide"]: s = f"The animal didn't cross the street because it was too {end}." enc = btok(s, return_tensors="pt") toks = btok.convert_ids_to_tokens(enc.input_ids[0]) with torch.no_grad(): att = bert(**enc).attentions[6][0, 10] # layer 7, head 11 it = toks.index("it") print(end, {w: round(att[it, toks.index(w)].item(), 3) for w in ["animal", "street"]}) 

tired {'animal': 0.284, 'street': 0.067} wide {'animal': 0.012, 'street': 0.59} 

We picked this head because it flips the most; other heads learn other links (the previous word, the next word, sentence boundaries). Running many heads side by side is multi-head attention.

The whole machine (Experiment 3)

Text is cut into tokens, pieces from a fixed vocabulary (byte-pair encoding; 50,257 pieces for GPT-2):

gtok = AutoTokenizer.from_pretrained("openai-community/gpt2") for text in ["The capital of France is Paris.", "unbelievably", "strawberry", " strawberry"]: print(repr(text), [gtok.decode([i]) for i in gtok(text).input_ids]) 

'The capital of France is Paris.' ['The', ' capital', ' of', ' France', ' is', ' Paris', '.'] 'unbelievably' ['un', 'bel', 'iev', 'ably'] 'strawberry' ['st', 'raw', 'berry'] ' strawberry' [' strawberry'] 

Then:

• each token becomes an embedding vector (word2vec's idea, learned with the model) plus a positional encoding, since attention alone has no notion of order;
• each layer has an attention block (tokens share information) and a feed-forward block (each token processed on its own);
• the 2017 model stacks 6 encoder layers (read the English) and 6 decoder layers (write the French);
• the output is a probability for every possible next token.

Why it won: parallel training

In training, the whole target sentence is known, so the model learns from every position at once and the GPUs are fully used. The big model trained in 3.5 days on one machine with 8 P100 GPUs and set records (BLEU 28.4 English–German, 41.8 English–French) at a fraction of the previous training cost.

Writing stayed sequential: the next token doesn't exist until it's chosen.

3. 2018–2019: reading vs writing

The blueprint split in two.

• BERT (Google, 2018): encoder only. Training game: mask a word and predict it from both sides. Great at understanding, can't generate.
• GPT (OpenAI): decoder only, with causal masking. Training game: Shannon's, predict the next token from the left.

Next-token prediction needs no labels: any text is a lesson. That's pretraining, and it turns the internet into training data.

• GPT-1 (2018): about 117M parameters, 12 layers, 7,000 unpublished books.
• GPT-2 (2019): same objective, 1.5B parameters, 48 layers, 8 million web pages. OpenAI initially held back the full model citing "concerns about malicious applications".

Experiment 1 now makes sense: GPT-2 continues documents, and on the web a question is often followed by its answer, then another question. It could also do untrained tasks by prompt alone: add TL;DR: after an article and it writes a rough summary.

4. 2020–2022: scale

Scaling laws (Kaplan et al., OpenAI, January 2020): train many models from tiny to large, and on log-log axes, loss falls along straight lines as parameters, data and compute grow. Not a law of nature, but a pattern you can plan around.

GPT-3 (2020): nearly the same architecture as GPT-2, but 175B parameters, 96 layers, trained on 300B tokens. It showed in-context learning: give a few examples in the prompt and it continues the pattern.

Experiment 4: in-context learning, already in GPT-2

def p_next(prompt, word=" Paris"): ids = tok(prompt, return_tensors="pt").input_ids with torch.no_grad(): probs = torch.softmax(model(ids).logits[0, -1].float(), -1) return probs[tok.encode(word)[0]].item() zero = "Q: What is the capital of France?\nA:" few = ("Q: What is the capital of Germany?\nA: Berlin\n" "Q: What is the capital of Italy?\nA: Rome\n" "Q: What is the capital of France?\nA:") print(f"zero-shot: {p_next(zero):.2f} few-shot: {p_next(few):.2f}") # GPT-2 XL from experiment 1 

zero-shot: 0.57 few-shot: 0.99 

Chinchilla (DeepMind, 2022) showed the balance was off: models were too big for their data. Chinchilla (70B parameters, 1.4T tokens) beat Gopher (280B, 300B tokens) almost everywhere. Data started to matter as much as size.

5. 2022: teaching it to help

Scale didn't fix behaviour. OpenAI's own example: prompt GPT-3 with "Explain the moon landing to a 6 year old in a few sentences" and it may continue with "Explain the theory of gravity to a 6 year old…"

InstructGPT (January 2022) added a second training stage:

1. Supervised fine-tuning: people write good answers; the model imitates them.
2. Reward model: the model writes several answers, people rank them, and a second model learns to predict the ranking.
3. Reinforcement learning: the language model is optimized against the reward model.

That's RLHF. People preferred a 1.3B InstructGPT model to the 175B GPT-3.

Experiment 5: the two stages in one small model

name = "Qwen/Qwen2.5-0.5B-Instruct" ctok = AutoTokenizer.from_pretrained(name) chat_model = AutoModelForCausalLM.from_pretrained(name).eval() q = "What is the capital of France?" chat = ctok.apply_chat_template([{"role": "user", "content": q}], add_generation_prompt=True, return_tensors="pt") out = chat_model.generate(chat, max_new_tokens=80, do_sample=False) print("CHAT:", ctok.decode(out[0][chat.shape[1]:], skip_special_tokens=True)) raw = ctok(q, return_tensors="pt").input_ids out = chat_model.generate(raw, max_new_tokens=60, do_sample=False) print("RAW:", ctok.decode(out[0][raw.shape[1]:])) 

CHAT: The capital of France is Paris. RAW: The capital of France is Paris. To verify this, I will use a simple Python code snippet to confirm ... 

With the chat template, it answers and stops. Without it, it rambles. Knowledge comes from pretraining; manners come from the second stage.

On November 30, 2022, OpenAI released ChatGPT, trained this way. A million users in five days, and serving these models at scale became the next problem.

6. 2023–2024: bigger, cheaper, longer

GPT-4 (March 2023): its report "contains no further details about the architecture (including model size)". The frontier went closed; open models like Meta's LLaMA (2023) show what changed inside the 2017 block:

• RoPE: positions as rotations applied inside attention, instead of added vectors;
• RMSNorm and SwiGLU: leaner normalization and feed-forward blocks;
• FlashAttention: the same attention, computed to fit the GPU's fastest memory.

Two harder problems:

Memory: the KV cache. During generation, each new token attends to all previous ones, so the model caches every past token's keys and values in every layer. The cache grows with context length and, for long conversations, can outgrow the model's own weights. Fixes: grouped-query attention (groups of heads share one set of keys/values) and DeepSeek's multi-head latent attention (store a compressed summary). Smaller caches are part of how context windows went from a few thousand tokens to about a million.

Compute: mixture of experts. Every token normally goes through every parameter. A MoE layer splits the feed-forward block into many experts and a router sends each token to just a few. DeepSeek-V3 (December 2024): 671B parameters stored, 37B used per token.

7. 2024–2025: thinking

In December 2024, Ilya Sutskever said at NeurIPS: "Pre-training as we know it will unquestionably end" and "We have but one internet." Epoch AI estimates models will have used up the public human text somewhere between 2026 and 2032.

Gains moved to inference time. Kojima et al. (2022) found that adding "Let's think step by step" took one maths benchmark (MultiArith) from 18% to 79%. A model spends roughly the same compute per token, so writing out steps buys more forward passes, and each step is easier than the whole jump.

• o1 (OpenAI, September 2024): trained with reinforcement learning to reason before answering; accuracy rose with thinking time. On the AIME 2024 maths competition: GPT-4o 12%, o1 74%.
• DeepSeek-R1 (January 2025), recipe published: reinforcement learning with verifiable rewards (maths with known answers, code with tests), rewarding only correct outcomes. R1-Zero's responses grew from about 500 to 9,500 tokens over training, and it started checking its own work: "Wait, wait. Wait. That's an aha moment I can flag here."

Experiment 6: a small model that thinks

DeepSeek also released distilled students of R1. The 1.5B one, with the model card's sampling settings:

name = "deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B" rtok = AutoTokenizer.from_pretrained(name) rmodel = AutoModelForCausalLM.from_pretrained(name, torch_dtype=torch.bfloat16).eval() q = "How many r letters are in the word strawberry?" chat = rtok.apply_chat_template([{"role": "user", "content": q}], add_generation_prompt=True, return_tensors="pt") out = rmodel.generate(chat, max_new_tokens=900, do_sample=True, temperature=0.6, top_p=0.95) print(rtok.decode(out[0][chat.shape[1]:])) 

On the capital question, it thought for 255 to 659 tokens and answered "Paris" four times out of five. On strawberry, across five runs: 3, 2, 3, 3, and once 8. Remember experiment 3: " strawberry" is one token, so the model never sees the letters. Thinking helps; it doesn't guarantee. Many models now decide for themselves whether to think, and for how long.

8. Now: senses and agents

Images and audio can be tokenized too, so the same architecture handles them: GPT-4o (2024) took text, images and audio in one network.

Agents run the model in a loop: plan, call a tool (search, code execution, file edits), read the result, repeat. METR measures the length of tasks (in human working time) that agents complete with 50% success, mostly on software tasks: GPT-2 a few seconds, GPT-4 (2023) about four minutes, early 2025 about an hour, and by April 2026 around 17 hours, beyond what their test suite measures reliably.

The architecture keeps moving too. Full attention costs more as context grows, so many new open models keep it in only some layers and use cheaper, recurrent-style layers in between, the RNN idea made trainable in parallel. And in September 2026, DeepSeek described a new model as a "causal encoder-decoder". Nine years later, the encoder is back.

9. 2017 vs today

Almost every part has been replaced. Two things remain: attention, in some layers at least, and the objective: predict the next token, Shannon's game from 1948.

▶ The full story, animated (19 min): In 19 Minutes, You will understand 10 years of LLMs

▶ What happens inside one model when it answers: What Actually Happens When You Press Enter on ChatGPT

Sources: Shannon (1948, 1951); Mikolov et al. (2013); Hochreiter & Schmidhuber (1997); Sutskever, Vinyals & Le (2014); Bahdanau, Cho & Bengio (2014); Vaswani et al. (2017); Devlin et al. (2018); Radford et al. (2018, 2019); Kaplan et al. (2020); Brown et al. (2020); Hoffmann et al. (2022); Ouyang et al. (2022); Kojima et al. (2022); OpenAI GPT-4 Technical Report (2023) and o1 (2024); Touvron et al. (2023); DeepSeek-AI, DeepSeek-V3 (2024) and DeepSeek-R1 (2025); METR; Epoch AI.