Originally published on wrightfunctions.com, where the demos are live.
You type a question into a chatbot and press Enter. A beat later, words start streaming out, a few at a time, like someone typing fast. It feels like one action, but it's a small pipeline, and each stage explains something you've probably noticed: why the first word takes a moment, why the rest flow quickly, and why the same question gets a different answer twice.
Text becomes numbers
A model can't read letters. Your message is first chopped into tokens, common chunks of text learned from data with byte-pair encoding. As a rule of thumb for English, one token is about four characters, or three-quarters of a word. Each token becomes an ID number, and each ID is swapped for an embedding, a long list of numbers standing for its meaning.
▶ Interactive demo: try it on the original post
Every word looks back at the words before it
Now the transformer does its work. Its main step is attention, which the 2017 paper Attention Is All You Need made the whole design: each token looks back at all the tokens before it and weighs which ones matter. In "we sat on the river bank", bank pays attention to river and settles on the right meaning. A model repeats this through dozens of layers, each refining every token's meaning a little further.
The output of all that is a single thing: a score for every token in the vocabulary, saying how likely each one is to come next. That's all a language model ever produces: one next token.
The next word is a dice roll
The model doesn't simply take the top-scoring token. It rolls weighted dice, which is sampling. Temperature reshapes the dice first: low values make the favourite almost certain, high values give long shots a real chance. OpenAI's API documents it as a number from 0 to 2, where "higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic."
▶ Interactive demo: try it on the original post
Many systems also let you trim the unlikely tail before rolling, a trick called nucleus sampling. And even at temperature 0, answers can differ run to run. In 2025, researchers at Thinking Machines traced that to the server: how many other users' requests are batched with yours changes the arithmetic very slightly, and one flipped token sends the rest of the answer somewhere new.
Then it does it again, and again
The chosen token is added to the text, and the whole thing runs again for the next one. That loop is why answers stream in word by word, and it splits serving into two very different phases.
First, prefill: the model reads your entire prompt in one parallel pass. That's heavy math on many tokens at once. It's most of the pause before anything appears, measured as time to first token. It's also why a long pasted document slows the start of a reply, even when the reply itself is short.
Then decode: one token per step, each step reading the entire model from memory to produce a single word. Researchers describe it as "a compute-intensive prompt computation" phase followed by "a memory-intensive token generation" phase, which is why GPUs spend so much of decoding waiting.
The cache that makes it cheap
What makes decode affordable is the KV cache. It keeps each token's attention data, so the model doesn't reprocess the whole conversation for every new word. It isn't small, either: for one 13-billion-weight model, the vLLM paper works out 800 KB per token, and up to 1.6 GB for a single long request. On the original post, you can hit Enter and watch it fill.
▶ Interactive demo: try it on the original post
That cache is one reason a model's context window has a limit, and one reason long conversations cost more. As of September 2026, Anthropic's larger Claude models take 1 million tokens, and every one of them needs a slot.
Making it faster
Almost every serving trick targets one of these phases. Batching lets many users share each read of the model during decode. Quantization makes the model smaller to read.
Speculative decoding lets a small model guess several tokens ahead and has the big one check them in one pass, for 2 to 3 times the speed on the paper's test model, with identical output. Some systems now run prefill and decode on different GPUs so the two stop getting in each other's way.
So what happens when you hit Enter?
Your text becomes tokens, tokens become numbers, and the transformer turns them into odds for one next token. A weighted dice roll picks it, and the whole loop runs again, token after token, until the answer is done. The pause before the first word is prefill; the steady stream after it is decode, one memory-bound step per token. Every stage is something someone has to serve, scale and pay for.
References & further reading
- What are tokens and how to count them? (OpenAI Help Center). For English, a token is about 4 characters, or three-quarters of a word.
- Attention Is All You Need (Vaswani et al. · Google · 2017). The transformer, "based solely on attention mechanisms". The design behind nearly every modern language model.
- Create chat completion (temperature) (OpenAI API reference). Temperature from 0 to 2; 0.8 is more random, 0.2 more focused and deterministic.
- The Curious Case of Neural Text Degeneration (Holtzman et al. · ICLR 2020). Nucleus (top-p) sampling, which cuts off the unreliable tail of the odds before rolling.
- Defeating Nondeterminism in LLM Inference (Horace He · Thinking Machines Lab · September 2025). Why temperature 0 isn't repeatable on real servers. Batch size changes with load, and most kernels aren't batch-invariant.
- Splitwise: Efficient generative LLM inference using phase splitting (Patel et al. · 2023). A compute-intensive prompt phase and a memory-intensive generation phase, and why they deserve different hardware.
- Efficient Memory Management for Large Language Model Serving with PagedAttention (Kwon et al. · SOSP 2023). 800 KB of KV cache per token for OPT-13B, up to 1.6 GB per request, and paging that cache like virtual memory.
- Models overview (Anthropic · docs · as of September 2026). 1M-token context windows on the larger Claude models.
- Fast Inference from Transformers via Speculative Decoding (Leviathan, Kalman & Matias · ICML 2023). A small model drafts, the big one verifies. 2–3× faster on T5-XXL with identical outputs.
- DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving (Zhong et al. · OSDI 2024). Prefill and decode on separate GPUs, tuned for time to first token and time per output token.
AI-assisted: I write these with help from AI tools.
