Tokenization and Text Representations Inside a Model

Before processing a sentence, a model maps text to a finite vocabulary of IDs and looks up vectors. Network layers then produce context-dependent representations: the same ID can acquire different states in different sentences. This separates three questions often conflated in practice: how many tokens a character costs, whether a tokenizer can be replaced, and whether hidden states make useful search vectors.

Tokens encode text

A whole-word vocabulary struggles with new words, spelling variants, and compounds; individual characters can make sequences long. Sennrich et al.’s subword work represents rare words with reusable smaller pieces. A token may be a word, a fragment, punctuation, or a byte sequence. Its boundaries depend on the tokenizer.

BPE starts from small units and repeatedly merges the adjacent pair that occurs together most often into a new piece. Work one round by hand on a tiny corpus: low appears 7 times and log 3 times, both first split into single letters. Among adjacent letter pairs, l+o occurs in both words, $7+3=10$ times, more than any other, so it merges into lo. Counting again, lo+w occurs 7 times and lo+g 3 times, so lo+w merges into low.

The recorded order of merges is the tokenizer's rule set; a new word lows, merged in the same order, splits into low and s.

WordPiece uses its own vocabulary-learning and matching procedure. Unigram assigns probabilities to possible segmentations. All of them segment by statistics; none consults a dictionary to understand meaning. The Hugging Face algorithm documentation distinguishes them. SentencePiece is a tool that can train models such as BPE and Unigram directly on raw sentences, without requiring prior whitespace-based word segmentation; see its paper.

In an invented teaching vocabulary, 重新加载 (“reload”) is encoded as the two tokens 重新 and 加载; a different vocabulary could use four single characters.

How a real tokenizer splits Chinese starts from bytes: in UTF-8 a common Chinese character takes 3 bytes, and some rare characters take 4. A byte-level BPE merges the bytes of characters it saw often in training into 1 token, while a rarely seen character may be split into several byte pieces. So “how many tokens per Chinese character” has no fixed answer: count with the chosen model’s own tokenizer rather than a fixed character-to-token ratio.

By the same logic, full byte coverage lets a tokenizer encode unseen characters as byte pieces, but encodability does not establish language understanding. Rare characters, emoji, mixed languages, and long numbers can consume many tokens.

From text to token IDs to vectors

A tiny BPE tokenizer learns from the training text one merge at a time: it counts every pair of neighboring symbols and joins the most frequent pair into a new token. The two sentences below are encoded with the merges learned so far. Pick a token to follow its ID into the embedding table.

Training text

1 · Learn merges

Merge 14: ▁r + at → ▁rat, which occurred 2 times. Vocabulary: 282. No neighboring pair occurs twice anymore, so training stops after 14 merges.

▁the▁cat▁sat▁on▁the▁mat.

▁the▁rat▁sat▁on▁the▁hat.

▁the▁cat▁ate▁the▁rat▁on▁a▁mat.

No pair occurs twice.

Merges

14/14

2 · Encode two sentences

English 22 characters → 6 tokens

Chinese 6 characters → 19 tokens

Dashed tokens such as <E7> are single bytes. A character that never appeared in the training text is spelled out as its UTF-8 bytes. Add Chinese to the training text and watch the count fall.

3 · Token → ID → embedding row

▁cat has ID 275, so the model reads row 275 of the embedding matrix E (282 rows × 4 columns). The ID is only a row number.

#	Token	ID	Row of E
1	▁the	271	−0.4	0.86	−0.52	−0.76
2	▁cat	275	−0.12	0.09	0.14	−0.09
3	▁sat	277	0.45	−0.62	0.33	0.07
4	▁on	273	0.31	−0.03	0.67	−0.73
5	▁the	271	−0.4	0.86	−0.52	−0.76
6	▁mat	279	−0.76	−0.95	0.1	0.4

Illustrative: a toy BPE with byte fallback, trained only on the sentences shown, so its counts are not those of any real tokenizer. ▁ marks the start of a word. The embedding values are seeded random numbers standing in for trained weights.

From IDs to vectors

For vocabulary size $V$ and hidden width $d$, a trainable matrix $E\in\mathbb{R}^{V\times d}$ stores an input vector in row $i$ for ID $i$. IDs are indices: ID 20 does not carry twice the meaning of ID 10.

Keep the invented vocabulary above and suppose it contains BOS=0, 重新=1, and 加载=2, where BOS is a special token marking the start of a sequence. The input [BOS, 重新, 加载] has IDs [0,1,2], which select three rows. For batch size 2, padded length 5, and width 4:

Stage	Shape	Meaning of an element
Input IDs	[2,5]	An integer index
Embedding matrix	[V,4]	A trainable floating-point parameter
Looked-up input	[2,5,4]	An initial vector per token
Layer L output	[2,5,4]	A representation using permitted context

Lookup is equivalent to multiplying a one-hot vector by the matrix, without allocating a one-hot vector of length $V$. This plain Python example illustrates lookup only and trains nothing:

embedding = [[0.0, 0.1], [0.5, -0.2], [-0.1, 0.8]]
ids = [0, 1, 2, 1]
x = [embedding[i] for i in ids]
assert x[1] == x[3]  # The same ID has the same initial vector.

In the contextual layers, one ID no longer means one vector. Even if bank has the same ID in “my bank account” and “the river bank,” it acquires different representations because the visible context differs. Attention explains the operation that updates those representations.

Order and valid positions are inputs too

An unordered set of word vectors cannot distinguish “cat chases dog” from “dog chases cat.” Models need position information. The original Transformer adds positional vectors at the input; RoPE rotates queries and keys within attention. These enter at different places, so positional encoding sometimes adds a vector to the input and sometimes rewrites the queries and keys inside attention. See the Transformer architecture.

To batch sequences of lengths 3 and 5, the shorter can be padded to length 5. Padding occupies tensor positions but usually should neither supply valid attention content nor contribute supervised targets. An attention mask and a loss mask solve different problems: ignoring padding in the loss does not automatically prevent other positions from reading it. Libraries can separately handle causal masks, padding masks, and position IDs; follow the model’s input contract.

Chat templates add roles, message boundaries, and generation markers. Token comparisons must use the serialized input, not only the visible user sentence; that is what the provider counting endpoints count. Truncation removes evidence. Increasing the output budget cannot recover a condition already removed from the input.

Open full-size image

Read one column vertically: BERT adds three vectors of the same width. The token row identifies the piece, the A/B segment row identifies its sentence, and the bottom row identifies its position. The repeated separator token has the same token embedding but different segment and position embeddings. This is the BERT input construction; models using RoPE encode position elsewhere.

Zhang et al., Dive into Deep Learning, Figure 15.8.2. · CC BY-SA 4.0

Three meanings of embedding

A token embedding is an initial lookup vector. A contextual hidden state is the output at one position in one layer. A retrieval embedding is usually a sentence or document vector produced using a specified pooling procedure and training objective. They can share a dimension while doing different jobs.

Averaging [batch,sequence,d] into [batch,d] also yields a vector, but whether it works for search depends on the objective it was trained for; see embeddings, rerankers, and classifiers.

Count before you pay

Prices, rate limits, and context windows are all measured in tokens, so count a request with the provider’s own endpoint before sending it. As of 2026-10-29, the three main endpoints are:

• Anthropic’s count_tokens accepts the same inputs as message creation, including system prompts, tools, images, and PDFs. The documentation calls the count an estimate that can differ slightly from actual usage.
• OpenAI’s POST /v1/responses/input_tokens accepts the same input as the Responses API. The documentation notes that local tokenizers such as tiktoken work for plain text but have limitations; the endpoint’s count also includes formatting tokens such as message roles and boundaries, which tokenizing only the text locally misses.
• Gemini’s countTokens counts the input only. Its rule of thumb is that one token is about 4 characters, and 100 tokens are about 60–80 English words.

Counts also change between generations from one provider. Anthropic’s documentation states that Claude 4.7 and later models and Claude Mythos Preview use a newer tokenizer. The same input text produces approximately 30 percent more tokens, with the exact increase depending on the content; the documentation advises recounting against the model you plan to use rather than reusing counts from earlier models. A prompt that counted 10,000 tokens on an older model is about 13,000 on a newer one, and at the same unit price its input cost rises by about 30 percent as well.

With a local model, the tokenizer is at hand. Pick a few examples: equivalent Chinese and English sentences, code with and without spaces, an uncommon character, and one sentence as plain text and inside its chat template. Record IDs, length, special tokens, and decoded text. Some tokenizers normalize text, so decode(encode(text)) need not reproduce the original bytes. Model weights, tokenizer, special-token configuration, and chat template are one matched set. Replacing only the vocabulary changes what IDs mean while the learned embedding rows still represent the old pieces.

Continue with encoder, decoder, and encoder–decoder families, where a central distinction is which positions each token may read.

The Hugging Face BPE lesson builds a tokenizer step by step from word counts and pair frequencies. Work through a merge, then compare how the learned rules segment a new word. This makes the distinction between learning a vocabulary and applying an existing tokenizer concrete.

My reading