Byte-Pair Encoding (BPE) — AI Glossary

Reference details

Related terms

• SentencePiece
• Tiktoken
• Vocabulary Size
• Tokenizer

Example: The first merge, counted by hand

Corpus: “banana” ×3 and “bandana” ×4, split to characters. Count every adjacent pair. “an” appears twice in banana (2×3 = 6) and twice in bandana (2×4 = 8) — 14. “na” appears twice in banana (6) but once in bandana (4) — 10. “ba” appears once in each — 7. “an” wins, so merge rule #1 is a+n → “an”, and both words are rewritten (b·an·an·a, b·an·d·an·a) before the next round counts pairs again. Repeat until the vocabulary reaches its target size. That ordered list of merges is the tokenizer — which is why a vocabulary cannot be edited after training without invalidating the learned embeddings.

Commonly confused with

BPE is the merge algorithm, not the tokenizer product. tiktoken and Hugging Face tokenizers both implement BPE but over different pre-tokenization rules and byte handling, so the same text yields different token counts under each. “Which tokenizer” is a more precise question than “which algorithm” when you are costing a prompt.

When to use it

Reach for it when:

• Training a model from scratch on text where subword coverage matters more than morphological correctness
• You need deterministic, reversible tokenization with no out-of-vocabulary path
• Estimating cost or context fit — count with the target model's own vocabulary, not a proxy

Reach for something else when:

• Reasoning about non-Latin scripts by character count: one CJK character often costs several tokens
• Assuming token counts transfer between providers — they do not, even for identical text
• Domain vocabularies where whole-word terms fragment badly; an adapted vocabulary may be worth the retraining cost

Primary source

• Sennrich et al., Neural Machine Translation of Rare Words with Subword Units (arXiv)

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Byte-Pair Encoding (BPE) — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path