What are tokens?

What is a token

A token is the smallest unit of text a language model reads or writes. It is not a word and not a character. It is a chunk of characters that the model's tokenizer has learned to treat as one piece, based on how often that chunk appeared in training data. Common words get their own token. Rare words get split into fragments.

Take the sentence The quick brown fox jumps over the lazy dog. OpenAI's o200k_base tokenizer splits it into 9 tokens:

The ·quick ·brown ·fox ·jumps ·over ·the ·lazy ·dog

Two things to notice. First, every word here is common enough to be a single token, so 9 words became 9 tokens. Second, the leading space is part of the token: ·quick (space included) is a different token than quick at the start of a line. Less common words split: unbelievably becomes three tokens in o200k_base (un, bel, ievably), and cl100k_base splits the same word differently (un, belie, vably).

These vocabularies come from byte pair encoding. Start with raw bytes, repeatedly merge the most frequent adjacent pair into a new token, and stop when the vocabulary hits its target size. The merges reflect the training corpus, which is why English packs tightly and everything underrepresented in that corpus does not.

The model never sees your text as letters. It sees a sequence of token IDs, predicts the next token ID, and the API decodes those IDs back to text. That is why token counts, not character counts, determine context window usage and cost.

Why the same text gives different counts

Each model family ships its own tokenizer with its own vocabulary, and vocabulary size changes how efficiently text packs into tokens. OpenAI's cl100k_base (GPT-4 era) has roughly 100k entries in its vocabulary. Its successor o200k_base (GPT-4o and later) has roughly 200k. Llama 3 uses a vocabulary of about 128k.

A larger vocabulary means more whole words and longer fragments exist as single tokens, so the same text usually needs fewer of them. The effect is small for plain English and large for everything else. The Hindi phrase नमस्ते दुनिया is 13 tokens in cl100k_base, 9 in Llama 3, and 5 in o200k_base. Same string, 2.6x spread. That spread is money: a Hindi-language chatbot on a cl100k-family model pays for more than twice the tokens per message compared to an o200k-family model at the same per-token price.

This is also why any tool that counts tokens with one tokenizer and quotes prices for every model is approximating. An accurate count requires running the tokenizer that matches the model you will call. Try it below: the pangram is 9 tokens in all three tokenizers, but paste in code or non-English text and watch the counts diverge.

How billing works

Providers meter two directions separately. Input tokens are everything you send: the system prompt, conversation history, retrieved documents, the user's message. Output tokens are everything the model generates. Prices are quoted per million tokens, and output is consistently more expensive, usually 2x to 6x the input rate, because generation costs more compute than reading.

A worked example. Say your app sends 2,000 input tokens and gets 500 output tokens back per request, at 10,000 requests a day. Over a 30-day month that is 600M input tokens and 150M output tokens.

Same workload, a 60x spread in monthly cost. Two structural points fall out of the math. Input volume usually dominates output volume (here 4:1), so a model's input price matters more than its headline output price for chat and RAG workloads. And because history gets resent on every turn, a long conversation re-bills its own past as input over and over. That is the mechanism behind most surprise bills.

Rules of thumb

When you need an estimate without running a tokenizer, these hold up well for English prose:

• 1 token is about 4 characters.
• 1 token is about 0.75 words, so 100 tokens is roughly 75 words.
• A single-spaced page of text runs around 500 tokens.
• Code tokenizes heavier than prose. Indentation, brackets, and identifier fragments add up; budget 20 to 50 percent more tokens than the character count suggests.
• Non-English text tokenizes heavier still, especially in non-Latin scripts. Depending on tokenizer and language, expect 1.5x to 3x the tokens of equivalent English.

Treat these as sizing tools, not billing tools. They are good enough to pick a context window or ballpark a budget, and off by enough to matter at invoice time. The same arithmetic works in reverse for context windows: a 128k-token window holds roughly 96,000 English words, and noticeably less if you are stuffing it with JSON or multilingual documents.

How to count exactly

For exact numbers, run the real tokenizer. Our token counter does this in your browser: cl100k_base, o200k_base, and Llama 3 run client-side, so your text never leaves your machine. That matters if you are counting prompts that contain customer data or unreleased code.

Not every provider publishes its tokenizer, so the counter labels each count with its provenance: whether the number comes from the model's own tokenizer or from a documented approximation ratio applied to a related one. You always know whether you are looking at an exact count or an estimate. The token visualizer shows the actual chunk boundaries, which is the fastest way to build intuition for why a prompt costs what it costs.

Common pitfalls

Chat template overhead. Counting the raw text undercounts the request. Chat APIs wrap every message in formatting tokens: role markers, message delimiters, the system prompt. Overhead is typically a few tokens per message plus whatever your system prompt weighs, and it recurs on every single call. A 20-token question inside a 400-token template costs 420 input tokens.

Reasoning tokens are billed as output. Reasoning models think in tokens you may never see, and those tokens land on the output line of your bill at the full output rate. A response that reads as 200 tokens can bill as several thousand once the hidden chain of thought is counted. If your costs on a reasoning model look detached from visible response length, this is why.

Cached input discounts. Most major providers now discount input tokens they have seen recently, typically by 50 to 90 percent, when the prefix of your prompt is identical between calls. Stable system prompts and shared document contexts qualify; anything after the first changed token does not. Structuring prompts so the static part comes first is the cheapest optimization most teams have not made.

The common thread: what you see in the response is not what you get billed for. Count with the right tokenizer, include the template, and read the usage field the API returns, because that number is the one the invoice uses.