Tokenization and Text Representations Inside a Model

Before processing a sentence, a model maps text to a finite vocabulary of IDs and looks up vectors. Network layers then produce context-dependent representations: the same ID can acquire different states in different sentences. This separates three questions often conflated in practice: how many tokens a character costs, whether a tokenizer can be replaced, and whether hidden states make useful search vectors.

Tokens encode text

A whole-word vocabulary struggles with new words, spelling variants, and compounds; individual characters can make sequences long. Sennrich et al.’s subword work represents rare words with reusable smaller pieces. A token may be a word, a fragment, punctuation, or a byte sequence. Its boundaries depend on the tokenizer.

BPE learns rules that merge adjacent smaller units. WordPiece uses its own vocabulary-learning and matching procedure. Unigram assigns probabilities to possible segmentations. None is a dictionary-based understanding of meaning. The Hugging Face algorithm documentation distinguishes them. SentencePiece is a tool that can train models such as BPE and Unigram directly on raw sentences, without requiring prior whitespace-based word segmentation; see its paper.

In a deliberately invented vocabulary, 重新加载 might be represented by 重新 and 加载, or by four characters. This predicts no real model’s segmentation. Full byte coverage lets a tokenizer encode unseen characters as byte pieces, but encodability does not establish language understanding. Rare characters, emoji, mixed languages, and long numbers can consume many tokens. Count context with the actual model’s tokenizer rather than a fixed character-to-token ratio.

From IDs to vectors

For vocabulary size $V$ and hidden width $d$, a trainable matrix $E\in\mathbb{R}^{V\times d}$ stores an input vector in row $i$ for ID $i$. IDs are indices: ID 20 does not carry twice the meaning of ID 10.

Suppose a toy vocabulary has BOS=0, 重新=1, and 加载=2. IDs [0,1,2] select three rows. For batch size 2, padded length 5, and width 4:

Stage	Shape	Meaning of an element
Input IDs	[2,5]	An integer index
Embedding matrix	[V,4]	A trainable floating-point parameter
Looked-up input	[2,5,4]	An initial vector per token
Layer L output	[2,5,4]	A representation using permitted context

Lookup is equivalent to multiplying a one-hot vector by the matrix, without allocating that large vector. This plain Python example illustrates lookup only:

embedding = [[0.0, 0.1], [0.5, -0.2], [-0.1, 0.8]]
ids = [0, 1, 2, 1]
x = [embedding[i] for i in ids]
assert x[1] == x[3]  # The same ID has the same initial vector.

If bank has the same ID in “bank account” and “river bank,” its later representations can nevertheless differ because of context. Attention explains one operation that updates those representations.

Order and valid positions are inputs too

An unordered set of word vectors cannot distinguish “cat chases dog” from “dog chases cat.” Models need position information. The original Transformer adds positional vectors at the input; RoPE rotates queries and keys within attention. These enter at different places, so positional encoding is not always a number added to input vectors. See the Transformer architecture.

To batch sequences of lengths 3 and 5, the shorter can be padded to length 5. Padding occupies tensor positions but usually should neither supply valid attention content nor contribute supervised targets. An attention mask and a loss mask solve different problems: ignoring padding in the loss does not automatically prevent other positions from reading it. Libraries can separately handle causal masks, padding masks, and position IDs; follow the model’s input contract.

Chat templates add roles, message boundaries, and generation markers. Token comparisons must use the serialized input, not only the visible user sentence. Truncation removes evidence. Increasing the output budget cannot recover a condition already removed from the input.

Open full-size image

Read one column vertically: BERT adds three vectors of the same width. The token row identifies the piece, the A/B segment row identifies its sentence, and the bottom row identifies its position. The repeated separator token has the same token embedding but different segment and position embeddings. This is the BERT input construction; models using RoPE encode position elsewhere.

Zhang et al., Dive into Deep Learning, Figure 15.8.2. · CC BY-SA 4.0

Three meanings of embedding

A token embedding is an initial lookup vector. A contextual hidden state is the output at one position in one layer. A retrieval embedding is usually a sentence or document vector produced using a specified pooling procedure and training objective. Equal dimensions do not make them interchangeable.

Averaging [batch,sequence,d] into [batch,d] produces a vector, but search quality still depends on training, pooling, padding masks, normalization, and query/document formatting. Two 768-dimensional outputs need not share a meaningful index space. Embeddings, rerankers, and classifiers compares their tasks.

Inspect a real input pipeline

With an existing local model, inspect public examples: equivalent Chinese and English sentences, code with different whitespace, an uncommon character, and one sentence as plain text and inside its chat template. Record IDs, readable pieces, length, special tokens, and decoded text. This examines input representation; it is not a capability benchmark.

Check whether normalization changes the original, where truncation happens, whether special markers are duplicated, and whether valid positions remain consistent after padding. Some tokenizers normalize text, so decode(encode(text)) need not reproduce the original bytes. Keep model weights, tokenizer, special-token configuration, and chat template matched. Replacing only the vocabulary changes what IDs mean while the learned embedding rows still represent the old pieces.

Continue with encoder, decoder, and encoder–decoder families, where a central distinction is which positions each token may read.

The Hugging Face BPE lesson builds a tokenizer step by step from word counts and pair frequencies. Work through a merge, then compare how the learned rules segment a new word. This makes the distinction between learning a vocabulary and applying an existing tokenizer concrete.

My reading · Opened notes are remembered in this browser.