Transfer Learning, LoRA, and Distillation

A pretrained model provides parameters learned from another distribution. Adapting it can require fewer task examples than learning every useful feature from scratch, but the old representation may also omit the distinction you need. Decide what must change before choosing a training technique: the output head, the representation, or the deployment model itself.

Start with a frozen representation

Suppose a fictional document router chooses keep, summarize, or discard. A frozen text encoder maps each document to a vector $h\in\mathbb R^d$. A trainable head computes

$$z=Wh+b,\qquad p=\operatorname{softmax}(z).$$

With $d=768$ and three classes, the head has $3\times768+3=2{,}307$ parameters. If the encoder is truly frozen and preprocessing is deterministic, vectors can be cached; training the head then avoids repeatedly running the encoder. This is a concrete baseline for whether the existing representation separates your labels.

If the labels depend on an omitted phrase after input truncation, no output head can recover it. If the vector conflates “already saved” and “not yet saved,” changing the head may be insufficient. Inspect such failures before concluding that the dataset merely needs more epochs. The fine-tuning chapter in Dive into Deep Learning explains the distinction between reusing features and adapting pretrained parameters.

Full fine-tuning makes the backbone trainable as well. It offers more freedom but also requires more gradient and optimizer storage, and can overwrite useful earlier behavior. Compare a frozen-head baseline, partial unfreezing, and full adaptation on the same task split rather than assuming that more trainable parameters always help.

LoRA restricts the parameter update

For a linear layer $W\in\mathbb R^{d_{out}\times d_{in}}$, LoRA freezes the original weight and learns a low-rank update:

Open full-size image

Follow x along both branches. The blue weight matrix stays frozen; the orange branch projects through rank r and returns to the output dimension. Only A and B are trained. The zero initialization shown for B makes the added branch initially contribute zero.

Hu et al., LoRA (2021), Figure 1; cropped from the paper.

$$W'=W+sBA, \quad A\in\mathbb R^{r\times d_{in}}, \quad B\in\mathbb R^{d_{out}\times r}.$$

Here $r$ is the chosen rank and $s$ a scaling factor. The trainable matrix entries total $r(d_{in}+d_{out})$ instead of $d_{in}d_{out}$. For a $4096\times4096$ layer at rank 8, that is 65,536 rather than 16,777,216 entries, a factor of 256 for this layer alone. It is not a 256-fold reduction in total training memory: the base weights, activations, selected target layers, and any trainable heads still matter.

A tiny example shows what rank restricts. Let

$$A=\begin{bmatrix}1&-1\end{bmatrix},\qquad B=\begin{bmatrix}2\\1\end{bmatrix},\qquad BA=\begin{bmatrix}2&-2\\1&-1\end{bmatrix}.$$

With $s=1$, input $(3,1)$ receives an output correction $(4,2)$. Both output corrections are controlled by the same projected input difference. A rank-one update cannot express every possible $2\times2$ change. Several adapted layers and nonlinearities make the whole network more expressive than this single matrix example, but the rank remains a real constraint.

LoRA changes how weights are updated; it does not shrink the original architecture. Compatible adapters can be merged into suitable base weights for inference, subject to the runtime and numeric representation. Keeping many adapters separate may help serve different tasks, but loading one with the wrong base checkpoint or tokenizer is not a valid deployment.

Distillation changes who learns

Knowledge distillation trains a student using information from a teacher. For classification, a teacher can provide a distribution rather than only a winning label. A temperature $T$ softens its logits:

$$p_i^{(T)}=\frac{\exp(z_i/T)}{\sum_j\exp(z_j/T)}.$$

For invented logits $(2,0,-2)$, $T=1$ gives approximately $(0.867,0.117,0.016)$; $T=2$ gives $(0.665,0.245,0.090)$. The softer target reveals how the teacher ranks alternatives. A common training objective combines hard-label loss with a teacher–student distribution loss; record the temperature, mixing coefficient, and reduction convention because they affect gradient scale.

from math import exp

def softmax(logits, temperature):
    shifted = [(v - max(logits)) / temperature for v in logits]
    weights = [exp(v) for v in shifted]
    total = sum(weights)
    return [v / total for v in weights]

for temperature in (1.0, 2.0):
    print([round(v, 3) for v in softmax([2, 0, -2], temperature)])

A generative teacher can also create demonstrations or candidate explanations, which must be filtered or checked against the target task. Agreement with a teacher measures imitation, not truth: systematically wrong teacher labels transfer the same error. Evaluate the student against independently defined labels or outcomes.

These methods answer different questions

Method	What changes?	Main diagnostic
frozen encoder plus head	decision boundary over fixed features	do the features retain the needed information?
full fine-tuning	selected or all backbone weights	does adaptation transfer without unacceptable forgetting?
LoRA	low-rank updates in selected layers	are placement and rank sufficient for the task?
distillation	student behavior, often in a smaller architecture	does independent task quality survive compression?

They can be combined. A teacher can label examples for a student trained with LoRA. That does not make the labels correct, nor does using LoRA prove the student is smaller. Quantization is another axis: it changes numeric representation, whereas distillation changes the learned model and LoRA constrains updates.

Designing a useful adaptation experiment

Split by source document, user, or time when nearby examples share information. Freeze the test set before generating paraphrases; otherwise variants of one sentence can leak across the split and exaggerate success. Use a labeled validation set for rank, learning rate, stopping time, and class thresholds.

For the router, measure each class’s recall, the cost of wrongly discarding a useful document, latency at the intended input length, and how often uncertain cases are deferred. Compare against a simple rule and the frozen-head baseline. Include negations, near duplicates, new topics, and changed class proportions. These stress the distinction the router is actually supposed to learn.

Training on traces can be valuable only if the trace labels encode outcomes you want repeated. A historical action is not automatically a correct action. Correct recurring mistakes before replaying them as supervision, then calibrate decision probabilities on held-out data as described in probability calibration.

The PEFT quicktour turns LoRA into a small implementation exercise: create an adapter configuration, inspect the trainable parameter count, then save and reload the adapter. Compare those saved files with the base-model requirement to understand what an adapter checkpoint contains.

My reading · Opened notes are remembered in this browser.