ModelRefs / Tokenization — Tutorial
Tokenization — Tutorial
How text is converted to numbers — the first step in every NLP pipeline. Covers Text is not tokens — the boundary matters, BPE and why it handles any language.
Overview
How text is converted to numbers — the first step in every NLP pipeline
Level: Intermediate. Estimated reading time: 20 minutes.
Text is not tokens — the boundary matters
LLMs don't process characters or words directly — they process tokens. Tokens are sub-word units: common words are one token ("the"), rare words split into multiple ("unbelievable" → ["un", "believ", "able"]), and punctuation and spaces are separate tokens.
The tokenizer is a fixed lookup table trained separately from the model using Byte-Pair Encoding (BPE) or WordPiece. Once fixed, it never changes for that model version. GPT-4 uses ~100k tokens; Claude uses a similar vocabulary.
BPE and why it handles any language
Byte-Pair Encoding starts with individual bytes (every byte 0-255). It iteratively merges the most frequent byte pair into a new token and repeats until the vocabulary reaches the target size (~32k–100k).
The result: common English words are 1-4 tokens. Rare words, code, and non-Latin scripts have more tokens per word. This also means token counting is not the same as word counting — a 1000-word essay might be 1200-1400 tokens.
Implication for cost and context windows: count tokens before sending to an API. tiktoken (OpenAI) and HuggingFace tokenizers let you count without making an API call.
Token budgets affect cost and quality
Every API call charges for input + output tokens. A 100k-token context window costs far more than a 1k context. Efficient prompting matters at scale.
Practical rules: • Structured output (JSON) uses fewer tokens than verbose prose instructions • Code is usually 1 token per ~4 characters • Non-English text uses more tokens per word than English • The special token [INST], <|im_start|>, etc. count against your budget • Always set a max_tokens limit to prevent runaway output costs
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Tokenization — Tutorial.