Subword tokenization — BPE, WordPiece, and how text becomes tokens

https://anirudhology.com/blog/demystifying-tokenization-part-1-why-tokenization-matters

Neural networks operate on numbers. Text is strings. Something has to map between them, and what it produces is the unit your model actually sees — the token.

The naive answer is to split on whitespace and assign each unique word an integer. That fails on real text within a paragraph: new words appear constantly (covid, chatgpt, kubernetes), different forms of the same word (run, ran, running) become unrelated entries, and languages without spaces (Chinese, Japanese, Thai) have nothing to split on. Every modern model uses something more sophisticated, and the family of solutions — BPE, WordPiece, SentencePiece, byte-level BPE — looks like one idea wearing four different hats.

This article walks through that family. We’ll see why word-level and character-level both break, what subword tokenization actually does, how the most common variants differ, and what the choice of tokenizer means for everything downstream — model size, cost per API call, cross-lingual fairness, even the kind of weird failures models exhibit. We’ll train a BPE tokenizer from scratch in about fifty lines of Python, then compare the result to four production tokenizers on the same input.

Why not just split on whitespace?

Word-level tokenization is the obvious starting point. Take a corpus, split on spaces, assign each unique word an integer, build a vocabulary. Three problems show up immediately.

Vocabulary size. English has somewhere between 200k and 1M words depending on how you count rare forms, proper nouns, and inflections. If each entry is embedded as a 768-dimensional vector (the bert-base size), a million-word vocab costs ~3GB just for the embedding table — most of the model’s parameters spent on words that appear once in the entire corpus. And that’s only the input side: a large vocabulary also makes the output layer expensive, because the final softmax normalizes over all $V$ entries on every step — an $O(V)$ cost that, in the word-level era, was painful enough to spawn workarounds like hierarchical and sampled softmax (see the word2vec article). Shrinking the vocabulary to subword units makes the plain softmax cheap enough that those tricks are no longer needed — arguably what retired hierarchical softmax for language models more than any competing approximation did.

Out-of-vocabulary words. Whatever cutoff you pick, words below it collapse into a single [UNK] token. The model can’t distinguish “tokenization” from “antidisestablishmentarianism” if both are unknown. Words coined after training are permanently invisible — a model trained in 2017 has no idea what covid is and never will.

Lost morphology. “Run”, “runs”, “running”, “ran” are obviously related. Word-level tokenization sees four unrelated entries and the model has to relearn the connection from scratch — separately for every verb. The same is true for prefixes (un-, re-, pre-), suffixes (-ation, -ing, -ed), and compounds (birthday, notebook, whiteboard).

Word-level works for closed-domain tasks — a spam classifier on English news with a fixed vocabulary — but not for general-purpose models trained on the open web.

The other extreme: characters

If word-level has too many tokens, why not go all the way down? A character-level vocabulary is tiny — ~100 entries covers ASCII plus common punctuation, ~5000 handles most of Unicode. Every word is in vocabulary by construction. Morphology falls out for free: run and running share four characters.

The problem is the opposite of word-level. Characters carry almost no meaning on their own — the token r means nothing without context. The model has to learn to bind characters into meaningful units before it can do anything else, and that takes a lot of layers and a lot of data. Sequences also get 4–5x longer (the average English word is about five characters), which directly multiplies attention compute (quadratic in sequence length).

Character models work, but they’re expensive. The dominant compromise is subword tokenization: a vocabulary of pieces between word and character. Common words get one token; rare words split into a handful of pieces; truly unknown words always decompose down to characters in the worst case. Vocabulary stays small (typically 30k–100k), every input is in vocabulary, and morphology is partially preserved because shared prefixes and suffixes tend to become their own tokens.

The interesting question is: which subword pieces?

BPE: merge frequent pairs

Byte-Pair Encoding (Sennrich et al., 2016) is the simplest subword scheme. The idea is older than NLP — BPE started as a text compression algorithm in 1994 — but applying it to neural-network vocabularies was new.

The training procedure has four steps:

1. Initialize with a character-level vocabulary. Every word in the corpus is split into individual characters, plus an end-of-word marker (we’ll use </w>).
2. Count how often every adjacent pair of symbols appears across the corpus.
3. Find the most frequent pair. Merge it: add a new symbol to the vocabulary representing the pair, and rewrite the corpus so all occurrences of the pair are replaced by that symbol.
4. Repeat steps 2–3 until you’ve performed a target number of merges (typically 30k–50k).

The output is a vocabulary plus an ordered list of merge rules. To tokenize new text at inference time, you split it into characters and apply the merge rules in their learned order.

A worked example

Take this tiny corpus, where each line is a word and the number is its frequency:

low      ×5
lower    ×2
newest   ×6
widest   ×3

After step 1 (character split with </w>):

l o w </w>      ×5
l o w e r </w>  ×2
n e w e s t </w> ×6
w i d e s t </w> ×3

Pair counts (each pair weighted by the frequency of the word it appears in):

Three pairs are tied at 9. BPE breaks ties in a fixed but arbitrary way — say, by the order pairs first appear. Merge e,s → es. The corpus becomes:

l o w </w>       ×5
l o w e r </w>   ×2
n e w es t </w>  ×6
w i d es t </w>  ×3

Now es,t appears 9 times — merge it. Then est,</w> appears 9 times — merge it. After three merges the vocabulary contains a recognizable morphological unit: est</w>, the suffix that “newest” and “widest” share.

A few more iterations and l,o merges to lo, then lo,w merges to low — another morphological unit emerging from frequency counting alone, with no linguistic supervision.

To tokenize a new word like lowest, apply the merges in order:

l o w e s t </w>            (initial split)
→ l o w es t </w>           (merge e,s)
→ l o w est </w>            (merge es,t)
→ l o w est</w>             (merge est,</w>)
→ lo w est</w>              (merge l,o)
→ low est</w>               (merge lo,w)

Two tokens out: low and est</w>. The word wasn’t in the training corpus, but its pieces were, and they reassemble into something the model can use. That’s the whole point of the algorithm.

What BPE actually does in practice

Real BPE adds practical details on top of the basic loop, none of which change the core idea:

• Normalization. Text is usually normalized before anything else — a canonical Unicode form, and sometimes case-folding or accent-stripping. Each choice is a trade-off: lowercasing shrinks the vocabulary but erases the US/us and CamelCase distinctions that matter for code; stripping accents tidies some European languages and destroys meaning in Vietnamese or Arabic.
• Pre-tokenization. Before the merge loop runs, the corpus is split on whitespace and punctuation so merges can’t accidentally cross word boundaries. Without this, the.cat would be a candidate merge target.
• Special tokens. Tokens like [CLS], [SEP], [MASK], <|endoftext|> are added to the vocabulary directly and never participate in merges.
• Frequency thresholds. Some implementations skip pairs below a minimum frequency to avoid memorizing one-off artifacts.
• Byte-level fallback (covered later) — operate on raw bytes instead of Unicode characters so nothing is ever truly out of vocabulary.

The basic loop is fifty lines of Python; production tokenizers wrap it in a few thousand lines of edge-case handling.

WordPiece: optimize likelihood instead of frequency

WordPiece (Schuster & Nakajima, 2012, popularized by BERT in 2018) keeps the same loop. The only change is step 3: instead of merging the most frequent pair, merge the pair whose merge would maximally increase the corpus’s likelihood under a unigram language model.

Concretely, for each candidate pair (x, y), the score is:

score(x, y) = freq(x, y) / ( freq(x) × freq(y) )

This is the pointwise mutual information of the pair. Pairs that occur together more than chance get high scores; pairs that just happen to be both frequent (like the followed by anything) score lower because the denominator is large.

The practical effect: BPE merges greedy frequency-driven pairs, which biases toward absorbing very common function words. WordPiece favors merges where the components are more meaningful together than apart, which tends to produce more morphologically coherent pieces.

WordPiece also marks continuation pieces explicitly. The first piece of a word stays as-is; subsequent pieces get a ## prefix. So tokenization becomes token and ##ization. This makes detokenization unambiguous (concatenate, removing the ## markers) and gives the model a positional cue: a ##-piece can never be a word’s first.

For most English inputs you can’t tell BPE and WordPiece apart by eye — the splits look similar. They diverge on rare words with strong morphological structure, where WordPiece is more likely to find the linguistic boundary, and on subwords that happen to be both very common (BPE absorbs them; WordPiece resists).

SentencePiece + Unigram: prune instead of merge

SentencePiece (Kudo & Richardson, 2018) is a different philosophy. BPE and WordPiece are bottom-up: start from characters, merge upward. SentencePiece’s most common variant — the Unigram language model — is top-down: start with a large candidate vocabulary, prune until you reach the target size.

Training procedure:

1. Generate a large initial vocabulary of frequent substrings (often via a fast pass of BPE, or simple n-gram counting). On the order of 1M candidates.
2. Estimate a unigram language model over the vocabulary using EM. Each token has a probability; any string can be segmented multiple ways, each with a different total probability.
3. For each token, compute how much the corpus log-likelihood drops if you remove it. Tokens whose removal barely hurts likelihood contribute little.
4. Remove the bottom 10–20% of tokens by likelihood contribution.
5. Repeat steps 2–4 until the vocabulary reaches the target size.

The advantage at inference time: a unigram model gives you a probabilistic segmentation. The same word can be tokenized multiple valid ways, and you can sample from the distribution during training (a regularization trick called subword regularization). BPE and WordPiece always produce the same tokenization for a given input.

SentencePiece is also language-agnostic. BPE and WordPiece typically pre-tokenize on whitespace, which fails for Chinese and Japanese. SentencePiece treats whitespace as a regular character (encoded as ▁, U+2581) and operates directly on raw Unicode. T5, ALBERT, XLNet, and the Llama family all use SentencePiece variants.

Byte-level BPE: characters that aren’t characters

GPT-2 introduced one more wrinkle. Instead of operating on Unicode characters, operate on bytes — the raw UTF-8 encoding.

The advantages:

• No OOV ever. Every input is a sequence of bytes, every byte is in the 256-entry base vocabulary, so any input — emoji, malformed Unicode, binary data — has a representation.
• Compact base vocabulary. Unicode has ~150k characters; UTF-8 represents all of them in 256 bytes. Starting from bytes keeps the base vocab small.
• Consistent across scripts. Emoji, Chinese, Arabic, Hindi all decompose to bytes the same way — one tokenizer covers everything.

The disadvantage: tokens can split mid-character. The character 中 is three bytes in UTF-8 (0xE4 0xB8 0xAD). A byte-level BPE tokenizer might learn one token for 0xE4 0xB8 and a separate token for 0xAD — neither piece is a valid character on its own. The model has to learn to recombine them, which works fine in practice but makes tokenization harder to inspect, and means token counts for non-Latin text can be very high.

One subtlety hides in the bytes: the same visible string can have more than one byte sequence. café might arrive as é (one code point, NFC) or as e followed by a combining accent (two code points, NFD) — different bytes, so a byte-level tokenizer treats them as different input and wastes vocabulary learning both spellings of the same word. This is why pipelines normalize to a canonical Unicode form (usually NFC) before tokenizing, even though byte-level BPE can technically represent whatever bytes it receives.

Byte-level BPE is what GPT-2, GPT-3, GPT-4, and most decoder-only models in the OpenAI lineage use, accessed through the tiktoken library. Llama uses SentencePiece. Claude uses a proprietary tokenizer in the same byte-level BPE family.

Try it

The widget below tokenizes the same input with four production tokenizers — bert-base-uncased (WordPiece), gpt2 (byte-level BPE), t5-small (SentencePiece + Unigram), and Llama-3 (SentencePiece + BPE). Pick a sentence, see how each tokenizer chops it.

A few things worth looking for:

• For common English words, all four tokenizers produce one token per word. The differences hide in rare words, technical jargon, code, and non-English text.
• WordPiece marks continuations with ##; SentencePiece marks word starts with ▁. The same information, opposite convention.
• Try a non-English sentence. WordPiece and the GPT-2 byte-level tokenizer fragment heavily; SentencePiece variants fragment less because they were trained on multilingual corpora.
• Try a JSON blob or a code snippet. GPT-style tokenizers were trained partly on code and tokenize structural punctuation ({, :, ()) more efficiently than the BERT tokenizer, which was trained mostly on prose.
• Try a word that was rare in 2018 — kubernetes, bitcoin, chatgpt. Older tokenizers fragment these into many pieces; newer ones often have them as single tokens.

What changes downstream

Tokenizer choice ripples through everything else.

Cost. API pricing is per token. The same prompt can be 30% longer or shorter depending on the tokenizer. For an English-only application this is a minor difference; for non-English text it can be 3–5x.

Cross-lingual fairness. Petrov et al., 2023 measured how many tokens different tokenizers use for the same content across 22 languages. English is consistently shortest; languages with non-Latin scripts (Burmese, Amharic, Khmer) use 5–15x more tokens for equivalent meaning. Speakers of those languages pay more, hit context limits sooner, and get slower responses — purely because of tokenizer choice.

Sequence length is a compression setting. It helps to see vocabulary size as a compression dial. A small vocabulary (say 8k) forces most words into several pieces — the vocabulary is highly compressed, but token sequences get longer and more of the model’s capacity goes to reassembling meaning from fragments. A large vocabulary (100k+) keeps sequences short but spends parameters on rare entries whose embeddings barely get trained. The 30k–100k range production models settle on is the empirical sweet spot, the same way JPEG quality settles between “tiny file, visible artifacts” and “pristine, enormous file.” The dial below lets you feel the tradeoff.

Vocabulary “knowledge”. A model’s vocabulary is a snapshot of the corpus it was trained on. Words coined after training don’t get their own token; they decompose into pieces. This is rarely a problem for understanding (the pieces still carry meaning), but it does mean parameters are spent on the distribution that existed at training time, not the one you’re using the model on.

Tokenizer leakage. Tokens are evidence about the training corpus. The presence of a username token like petertodd in early GPT-2 vocabularies tells you something about the relative weight of certain forums in WebText. Sometimes those tokens trigger unexpected behavior because the model rarely saw them in any other context — the famous “SolidGoldMagikarp” tokens that GPT-3.5 would refuse to repeat or would respond to with bizarre completions. The tokens existed (because the byte-level BPE training picked them up) but the model never learned good representations for them.

Character-level blindness. Because common words are single tokens, the model never sees their letters. Ask a model how many rs are in “strawberry” and it can get it wrong — not because it can’t count, but because strawberry arrives as one or two opaque tokens with no r visible inside them. The same blindness shows up in spelling, rhyming, anagrams, and reversing strings: any task that needs sub-token structure is fighting the tokenizer. It’s the main reason character- and byte-level tokenizers keep reappearing despite their cost — for those tasks, the atomic units have to be there.

Number handling. Numbers are a sharp case of the same problem. If 2024 is one token but 2025 splits into 20 and 25, the model can’t see the two years as one apart — the digit-level structure that makes arithmetic work is scrambled by frequency-driven merges, and “which is bigger, 9.11 or 9.9?” turns into a lookup over inconsistent pieces. This is why several recent tokenizers deliberately split numbers into single digits or fixed 3-digit chunks: they trade longer sequences for arithmetic the model can actually learn. Tokenizer choice bounds a capability, directly.

Lock-in. The tokenizer is chosen once, before pretraining, and then frozen for the life of the model. The embedding table and the output softmax are both indexed by token ID, so changing the vocabulary means retraining from scratch — you can’t swap tokenizers on a trained model the way you can swap a learning rate or a sampling temperature. A tokenizer that’s 20% less efficient than it could be is a 20% tax on every inference call, permanently, until the next full training run. Of all the decisions in the stack, it’s one of the few that’s effectively irreversible — worth an hour of thought before you commit.

Build it yourself

The script below trains a BPE tokenizer from scratch on a small corpus, then tokenizes a held-out word. The implementation is short because the algorithm is short — about fifty lines for the full training loop plus encoding.

from collections import Counter, defaultdict

def get_pair_counts(corpus):
    """Count adjacent symbol pairs across the corpus."""
    counts = Counter()
    for word, freq in corpus.items():
        symbols = word.split()
        for a, b in zip(symbols, symbols[1:]):
            counts[(a, b)] += freq
    return counts

def merge_pair(corpus, pair):
    """Replace every occurrence of the pair with the merged symbol."""
    a, b = pair
    merged = a + b
    new_corpus = {}
    for word, freq in corpus.items():
        symbols = word.split()
        out = []
        i = 0
        while i < len(symbols):
            if i < len(symbols) - 1 and (symbols[i], symbols[i + 1]) == pair:
                out.append(merged)
                i += 2
            else:
                out.append(symbols[i])
                i += 1
        new_corpus[" ".join(out)] = freq
    return new_corpus

def train_bpe(words, num_merges):
    """Train BPE on a {word: freq} dict. Returns the merge list."""
    corpus = {" ".join(list(w) + ["</w>"]): freq for w, freq in words.items()}
    merges = []
    for _ in range(num_merges):
        pairs = get_pair_counts(corpus)
        if not pairs:
            break
        best = max(pairs, key=pairs.get)
        corpus = merge_pair(corpus, best)
        merges.append(best)
    return merges

def encode(word, merges):
    """Tokenize a word by applying learned merges in order."""
    symbols = list(word) + ["</w>"]
    for a, b in merges:
        out = []
        i = 0
        while i < len(symbols):
            if i < len(symbols) - 1 and symbols[i] == a and symbols[i + 1] == b:
                out.append(a + b)
                i += 2
            else:
                out.append(symbols[i])
                i += 1
        symbols = out
    return symbols

# Train on the toy corpus from the worked example.
words = {"low": 5, "lower": 2, "newest": 6, "widest": 3}
merges = train_bpe(words, num_merges=10)
print(merges)
# [('e', 's'), ('es', 't'), ('est', '</w>'), ('l', 'o'), ('lo', 'w'), ...]

print(encode("lowest", merges))
# ['low', 'est</w>']

That’s the entire algorithm. Modern libraries — HuggingFace tokenizers, Google’s sentencepiece, OpenAI’s tiktoken — handle all the practical details (normalization, pre-tokenization, special tokens, fast Rust-backed inference) but the core loop is exactly this.

Is a tokenizer any good? Four numbers

Once you have a tokenizer, “good” isn’t a feeling — it’s measurable, on a sample of the text you actually intend to run through it. Four numbers do most of the work, and all four should be computed on held-out data from your real distribution, not the training corpus.

Compression ratio (bytes per token). Count the UTF-8 bytes in a held-out corpus, divide by the number of tokens the tokenizer produces. Around 4–6 bytes/token is healthy for English; higher means the tokenizer packs more text into each token, which means shorter sequences and lower cost. The number falls off a cliff on text the tokenizer wasn’t trained for — run the same measurement on your medical notes, your code, your target language, and compare.

Fertility (tokens per word). How many pieces does the average word break into? Close to 1 means most words are single tokens; climbing toward 2–3 means heavy fragmentation. In-domain English usually sits near 1.1–1.3; the same tokenizer on a specialized domain or a non-Latin script can double or triple that — the same content, more tokens, more money.

Single-token rate. The fraction of your common words that survive as exactly one token. This is the practical version of the OOV question: not “is the word representable” (byte-level guarantees yes) but “is it represented well.” If fewer than half of your common domain terms are single tokens, the vocabulary is mismatched to your data and you’re paying for it on every call.

Vocabulary allocation is zero-sum. These numbers all trace back to one fact: the vocabulary is a fixed budget, and every token spent on one fragment is a token denied to another. A tokenizer trained on 60% English text spends roughly 60% of its budget on English pieces — which is exactly why the cross-lingual disparity from earlier isn’t a bug but an accounting identity. Fixing it means changing what the tokenizer is trained on — oversampling the underrepresented language or domain — not tuning the algorithm.

Where this leads

You now have units. The next question is what vector represents each unit, and how that vector encodes meaning — an answer that evolved from “one fixed vector per token, learned from how words co-occur” (word2vec, GloVe) to “a vector that depends on the surrounding sentence” (BERT, GPT).

That’s a parallel thread, not the next step in this one. word2vec and GloVe are word-level: split on whitespace, keep the frequent words, drop the rest. The tokenization question barely exists there — each word is its own unit, and anything unknown is simply ignored. Subword tokenization matters for what came after: BERT and GPT embed pieces rather than words, precisely so that every input has a representation and rare words decompose into something the model can still use. So “how text becomes tokens” and “what vector each token gets” are two threads that run in parallel and converge at the contextual models — the BERT article is where they meet.