Tokenization: Why Subword Methods and What BPE Is Actually Doing

The way you split text into tokens is not a preprocessing detail — it determines the vocabulary size, how the model handles rare words and morphology, and how many tokens a sequence consumes (which directly affects cost and context length). BPE, WordPiece, and SentencePiece make different trade-offs. Understanding them explains why GPT-4 charges per token and why "tokenization bugs" are a real production failure mode.

Every language model operates on tokens, not raw text. The tokenizer converts a string of characters into a sequence of integers that the model can process. The choice of tokenization algorithm determines the vocabulary, the average sequence length, and the model's handling of rare words, numbers, and non-English text. It is not a detail — it is a design decision with cascading effects.

Character, word, and subword tokenization

Character tokenization: split every character into a separate token. Vocabulary is tiny (~256 for ASCII). Sequences are very long, which makes Transformer attention expensive. The model must learn morphology and spelling from scratch. Almost never used for large language models.

Word tokenization: split on whitespace and punctuation. Sequences are short. But the vocabulary is huge (every inflection of every word is a separate token), rare words (misspellings, proper nouns, scientific terms) become out-of-vocabulary tokens mapped to [UNK], and the model cannot share representations between related words ("run", "running", "runs").

Subword tokenization: the practical middle ground. Common words are single tokens; rare words are split into recognisable subword pieces. "tokenization" → ["token", "ization"]. The model can combine pieces it has seen in other contexts. Vocabulary is 32,000–100,000 tokens. This is what all modern LLMs use.

Byte-Pair Encoding: the algorithm

BPE (Sennrich et al., 2016) starts with a character-level vocabulary and iteratively merges the most frequent adjacent pair. Start: vocabulary = all individual characters. Count: find the most frequent pair (e.g., "t" + "h" appears 50,000 times). Merge: add "th" to vocabulary, replace all occurrences. Repeat until vocabulary reaches the target size (e.g., 50,000 merges for GPT-2).

The result: frequent subwords and whole words are single tokens; rare sequences are represented by their character-level components. The merge order is the vocabulary — tokenisation of new text replays the merges in order, greedily combining the longest matches.

GPT-2, GPT-3, GPT-4, and LLaMA use BPE. GPT-4's tokenizer (cl100k) has 100,256 tokens and was trained on a much larger and more multilingual corpus than GPT-2's, explaining better performance on non-English text.

WordPiece: BERT's variant

WordPiece (Schuster & Nakamura, 2012) is similar to BPE but uses a likelihood-based merge criterion instead of frequency: merge the pair that maximises the likelihood of the training corpus under the language model. Subword pieces beyond the first in a word are prefixed with "##" to mark continuation: "tokenization" → ["token", "##ization"]. BERT uses WordPiece with a 30,522-token vocabulary.

SentencePiece: language-agnostic tokenization

BPE and WordPiece assume whitespace separates words — a reasonable assumption for English but wrong for Chinese, Japanese, Thai, and other languages. SentencePiece (Kudo & Richardson, 2018) treats the input as a raw character stream with no whitespace pre-segmentation. It uses BPE or unigram language model to learn subword segmentation directly. T5, ALBERT, and XLM-R use SentencePiece. It is the standard for multilingual models.

Why tokenization is a production failure mode

Token counting determines cost (API pricing), context length (will the prompt fit?), and model behaviour. Real failure modes: a number like "1,000,000" tokenizes to 6 tokens in some vocabularies and 2 in others — arithmetic over numbers is harder when each digit is a separate token. Code with unusual indentation or symbols may consume far more tokens than expected. Non-English text tokenizes into more pieces per word than English — Russian or Chinese content is 2-4× more expensive per character. Prompt injection exploits can use unusual tokenization to bypass content filters. Knowing your tokenizer is operational hygiene for production LLM applications.

Try on Colab: use the tiktoken library (OpenAI) to tokenize a few edge cases: a number (1000000), a URL, a code snippet with indentation, and the same sentence in English, Spanish, and Chinese. Count tokens for each. Visualise the tokenization with tiktoken's visualiser. See how different languages encode with dramatically different efficiency — this is the root cause of multilingual LLM performance gaps.