What Tokens Actually Are (and Why Your Bill Depends on Them)

TL;DR

• A token is a chunk of text an LLM represents as a number. Encoding turns text into tokens; decoding turns tokens back into text.
• Input tokens include your conversation history, system prompt, and tool definitions, not just your latest message. Output tokens are what comes back. You're billed for both, usually at different rates.
• Tokenizers build their vocabulary by repeatedly merging common character patterns. Bigger vocabularies mean common words compress into fewer tokens.
• Rare or made-up words never earned a shortcut in that vocabulary, so they fragment into many small tokens.

Contents

• Encoding: turning text into numbers
• Decoding: turning numbers back into text
• The full round trip
• Where the vocabulary comes from
• Why vocabulary size matters
• Where it breaks: words the tokenizer has never seen
• Why this is worth knowing

Encoding: turning text into numbers

A token isn't a word, it's whatever chunk of text the tokenizer's trained vocabulary recognizes: a whole word, a word fragment, or a punctuation mark. The tokenizer splits your text into those chunks, then converts each one into the number the model actually operates on.

Decoding: turning numbers back into text

Decoding runs that process in reverse: each output number maps back to a token, and the tokens are joined into the string you read. Whitespace often rides along inside a token rather than standing on its own, which is part of why token counts never map cleanly onto word counts.

The full round trip

Encoding and decoding sit on either end of the model: your input is encoded into tokens, the model generates output tokens, and those get decoded back into the response you see.

The billing detail worth remembering: input tokens include your full conversation history, system prompt, and tool definitions, not just what you just typed, while output tokens are only the response. Providers usually price output tokens higher, so asking for shorter answers is a cheap, direct lever on cost.

Where the vocabulary comes from

Tokenizers are trained, not hand-built. Training starts from individual characters and repeatedly merges the pairs that appear together most often, so on a tiny corpus like "the cat sat on the mat," pairs like "th" and "at" merge first, and eventually whole common words like "the" collapse into one token.

Why vocabulary size matters

A bigger trained vocabulary means more of those merges already exist, so ordinary words compress into fewer tokens. The word "understanding" splits into 5 tokens at a 1K vocabulary, 3 tokens at 50K, and just 2 at 200K, purely because the larger vocabulary already had a shortcut for it.

Where it breaks: words the tokenizer has never seen

Rare or invented words never earned that shortcut. "Frabjous," from Lewis Carroll's poem, splits into a pile of small, awkward fragments instead of a clean token or two, since it never showed up often enough during training.

That's a good mental model for why LLMs stumble on invented product names or unusual jargon: the tokenizer is guessing at patterns it's never seen, and every extra fragment is a token you're paying for.

Why this is worth knowing

None of this is exotic, it's closer to the multiplication table of working with LLMs, but it's easy to skip when the API "just works." Seeing a prompt as a sequence of tokens instead of a block of text makes bloated system prompts, rambling outputs, and mangled rare words all make a lot more sense.

If you want to see it live, the Tiktokenizer playground lets you paste text and watch it get sliced into tokens in real time.