What Are Tokens, and Why Does AI Count Everything in Them?

AI models don't read words. They read tokens: small chunks of text that the model turns into numbers before it can do anything with them.

You see the word everywhere. API pricing is "per million tokens". Context windows are "200K tokens". Speed is "tokens per second". In the previous articles in this series, we saw that inference (the model answering your prompt) starts by splitting your text into tokens, and that decode (writing the answer) produces one token per loop.

This article looks at what a token actually is, how text gets split, and why the split affects your bill, your speed, and how much the model can remember.

What a token looks like

A token can be a whole word, part of a word, a space plus a word, or punctuation.

Here is our running example, "Why do cats purr?", split by o200k_base, the tokenizer used by OpenAI's GPT-4o models. I ran this with the tiktoken library:

"Why do cats purr?" → 6 tokens ["Why", " do", " cats", " p", "urr", "?"] [13903, 621, 28854, 275, 1057, 30] 

Three things to notice:

1. Spaces belong to the next word. The token is " do", not "do". A word at the start of a sentence and the same word mid-sentence can be different tokens.
2. Common words stay whole; rarer ones get split. "cats" is one token. "purr" is less common, so it becomes " p" + "urr".
3. Each token is just an ID. The model never sees the letters. It sees [13903, 621, 28854, ...] and looks up a learned vector (a list of numbers) for each ID.

A few more splits from the same tokenizer:

Every model family has its own tokenizer, so the same sentence gives different counts in GPT, Claude, Llama, or Gemini. Never assume one model's count applies to another.

How a tokenizer decides where to split

Most modern tokenizers learn their vocabulary from data using Byte Pair Encoding (BPE). The idea is simple: start with tiny pieces and keep merging the pairs that appear together most often.

A simplified version:

1. Start with every character as its own token: c a t s, p u r r.
2. Count every adjacent pair across a huge amount of text.
3. Merge the most frequent pair into a new token (say r + r → rr).
4. Repeat until the vocabulary reaches a target size.

After enough merges, frequent words like "cats" become single tokens, while rare words stay as a few pieces. The final vocabulary for o200k_base has about 200,000 entries.

Modern tokenizers run BPE on bytes, not characters. Any text, in any language or even emoji, can always be represented: worst case, it falls back to individual bytes. So there is no such thing as an "unknown word". There are only words that cost more tokens.

Why tokens matter

The token count decides three things you care about: cost, speed, and fairness across languages.

Cost. API providers charge per token, usually with separate prices for input (your prompt) and output (the answer). Output tokens typically cost several times more than input tokens, because each one needs its own decode step, while input tokens are processed together in parallel during prefill.

Speed. Decode produces one token per loop. A 600-token answer needs 600 loops. If a word takes 3 tokens instead of 1, writing it takes roughly 3 times as long.

Language. Tokenizers learn from their training data, and that data is mostly English. Other languages get fewer merges, so the same meaning costs more tokens. Here is the same question measured with o200k_base:

The Yoruba version costs 2.5 times as many tokens as the English one. Accented characters like ó, ò and ń often break words into small pieces (ol + ó + gb + ò). For a developer building for Nigerian users, that means higher API bills, slower answers, and less room in the context window for the same conversation.

The context window

The context window is the maximum number of tokens a model can handle in one request, counting both what you send and what it writes back.

Everything you send goes into that budget:

• The system prompt (the app's hidden instructions)
• The full chat history, resent on every turn
• Any documents, code, or search results you attach
• Tool definitions, if the model can call tools
• The answer the model is about to write

That last point catches people out. If a model has a 128K-token window and your prompt is 126K tokens, it only has about 2K tokens left to answer.

Larger windows also cost more to run. Every token in the context gets an entry in the KV cache (the stored attention data the model reuses during decode, covered in the inference article). A longer conversation means a bigger cache, more GPU memory per user, and slower decode steps.

When you hit the limit, the API either rejects the request or the app quietly drops or summarises older messages. That is why a long chat can suddenly "forget" something you said at the start.

Counting tokens yourself

Don't guess token counts in production; measure them. For OpenAI models in Python:

import tiktoken encoding = tiktoken.get_encoding("o200k_base") def count_tokens(text: str) -> int: return len(encoding.encode(text)) print(count_tokens("Why do cats purr?")) # 6 

Other providers expose their own counting endpoints or SDK methods, because their tokenizers differ. Use the one that matches the model you call.

A rough rule for English is about 4 characters, or three quarters of a word, per token. It is fine for a back-of-the-envelope estimate, and wrong for code, numbers, and most non-English text.

Wrapping up

Tokens are the unit everything in an AI model is measured in: what it reads, what it writes, what it costs, and what it can remember.

The tokenizer splits text into frequent chunks learned from data. Common English words become one token; rare words, code, and many other languages take several. More tokens mean higher cost, slower answers, and a fuller context window.

Next time a model forgets the start of a long chat, or an API bill comes in higher than expected, count the tokens. The answer is usually there.