Transfer Learning, LoRA, and Distillation

A pretrained model has already learned to turn text into useful vectors, so training a new classifier on top of it takes only a small set of labeled examples rather than the vast amount of data needed to learn reading from scratch. That works as long as the vectors still carry the distinction the new task needs. So before choosing a technique, decide which layer must change: the output head, the internal representation, or the deployed model itself.

Most problems do not need new weights

Fine-tuning changes a model's habits on a recurring kind of input; it is a poor way to add facts that change. So try the cheaper steps first: write the instruction more clearly and put a few examples in the prompt (few-shot); if what the model lacks is your documents or today's information, use retrieval. Fine-tuning comes next only when those steps stall below the bar on a fixed eval set and the failures are always the same behaviour (unstable format, a wrong label decision, the wrong tone) rather than missing knowledge.

Fine-tuning pays off in three situations. The first is one narrow task called thousands of times, where the long instructions and examples in the prompt are paid for on every call and still work only some of the time. The second is getting a small model to do what a large model does only with a long prompt; that is distillation, covered below. The third is a model that must run offline or on your own machine. Pretraining, supervised fine-tuning, and preference optimization explains which training stage fits which kind of problem.

Start with a frozen representation

Imagine you receive many web clippings every day and want a small model to decide, for each one, whether to keep it, summarize it, or throw it away. That model is a document router choosing among three labels: keep, summarize, and discard. The cheapest approach freezes a text encoder, which maps each clipping to a vector $h\in\mathbb R^d$, and trains only the output layer:

$$z=Wh+b,\qquad p=\operatorname{softmax}(z).$$

With $d=768$ and three classes, the head has $3\times768+3=2{,}307$ parameters. If the encoder's parameters and buffers are fixed, preprocessing is deterministic, and the encoder runs in evaluation mode so that dropout is off, vectors can be cached; training the head then avoids repeatedly running the encoder. This baseline answers one question directly: do the existing vectors already separate the three kinds of clipping?

Two failures cannot be fixed by training longer. If the sentence that decides the label was cut off by input truncation, no output head can recover it. If the encoder maps “this is almost the same as something I already saved” and “this is new” to nearly the same vector, no amount of tuning the head will separate them. Check for these two failures before deciding whether to add epochs. The fine-tuning chapter in Dive into Deep Learning explains the distinction between reusing features and adapting pretrained parameters.

Full fine-tuning makes the backbone trainable as well. It offers more freedom but also requires more gradient and optimizer storage, and can overwrite useful earlier behavior. More trainable parameters do not by themselves mean better results; compare a frozen encoder with a trained head, partial unfreezing, and full adaptation on the same data split, and see which one actually does better on the test set.

LoRA restricts the parameter update

For a linear layer $W\in\mathbb R^{d_{out}\times d_{in}}$, LoRA freezes the original weight and learns a low-rank update:

Open full-size image

Follow x along both branches. The blue weight matrix stays frozen; the orange branch projects through rank r and returns to the output dimension. Only A and B are trained. The zero initialization shown for B makes the added branch initially contribute zero.

Hu et al., LoRA (2021), Figure 1; cropped from the paper.

$$W'=W+sBA, \quad A\in\mathbb R^{r\times d_{in}}, \quad B\in\mathbb R^{d_{out}\times r}.$$

Here $r$ is the chosen rank and $s$ a scaling factor. The trainable matrix entries total $r(d_{in}+d_{out})$ instead of $d_{in}d_{out}$. For a $4096\times4096$ layer at rank 8, that is 65,536 rather than 16,777,216 entries, a factor of 256 for this layer alone. It is not a 256-fold reduction in total training memory: the base weights, activations, selected target layers, and any trainable heads still matter.

The smallest example already shows the limit: a rank-1 $2\times2$ update $BA$ reads the input along one direction and writes the output along one direction, so it cannot express an arbitrary $2\times2$ change. Several adapted layers and nonlinearities make the whole network far more expressive, but the rank remains a real constraint.

LoRA changes how weights are updated; it does not shrink the original architecture. Compatible adapters can be merged into suitable base weights for inference, subject to the runtime and numeric representation. Keeping many adapters separate may help serve different tasks, but loading one with the wrong base checkpoint or tokenizer is not a valid deployment.

Distillation changes who learns

Knowledge distillation trains a student using information from a teacher. For classification, a teacher can provide a distribution rather than only a winning label. A temperature $T$ softens its logits (how temperature reshapes a softmax is covered in softmax regression):

$$p_i^{(T)}=\frac{\exp(z_i/T)}{\sum_j\exp(z_j/T)}.$$

For invented logits $(2,0,-2)$, $T=1$ gives approximately $(0.867,0.117,0.016)$; $T=2$ gives $(0.665,0.245,0.090)$. The softer target reveals how the teacher ranks alternatives. A common training objective combines hard-label loss with a teacher–student distribution loss; record the temperature, mixing coefficient, and reduction convention because they affect gradient scale.

A generative teacher can also create demonstrations or candidate explanations, which must be filtered or checked against the target task. Agreement with a teacher measures imitation, not truth: systematically wrong teacher labels transfer the same error. Evaluate the student against independently defined labels or outcomes.

These methods answer different questions

Method	What changes?	Main diagnostic
frozen encoder plus head	decision boundary over fixed features	do the features retain the needed information?
full fine-tuning	all backbone weights	do earlier capabilities deteriorate as the new task improves?
LoRA	low-rank updates in selected layers	are placement and rank sufficient for the task?
distillation	student behavior, often in a smaller architecture	does independent task quality survive compression?

These methods can be combined: a teacher supplies examples, and a student learns them with LoRA. The student’s size is set by the architecture you choose, not by whether you use LoRA; and examples the teacher labeled wrongly are learned by the student just the same. Quantization is another axis: it changes numeric representation, whereas distillation changes the learned model and LoRA constrains updates.

Hosted fine-tuning has largely closed, so individuals run their own trainer

By late 2026 hosted fine-tuning from the two largest vendors had closed to individuals. OpenAI's deprecation notice, announced on 2026-05-07, closes it in three steps: from that day, organizations that had never run fine-tuning can no longer create training jobs; from 2026-07-02, neither can organizations with no inference on a fine-tuned model in the past 60 days; on 2027-01-06, active existing customers lose the ability to create new jobs too. Models already trained stay available for inference until their base model is deprecated. Its model optimization guide likewise says the platform is "no longer accessible to new users". Google's Gemini API tuning page (updated 2026-04-28) says there is no longer a model that supports fine-tuning in the Gemini API or AI Studio. The last one, Gemini 1.5 Flash-001, was deprecated in May 2025, and Google has no immediate plans to bring fine-tuning back.

So for an individual developer, fine-tuning now means downloading open weights and training an adapter on your own machine or a rented GPU. Three tools cover most setups:

• Mac (Apple silicon): MLX-LM trains with mlx_lm.lora --train and supports lora, dora and full; when --model points to a quantized model, training becomes QLoRA automatically. The data directory holds train.jsonl (and optionally valid.jsonl) in the chat, tools, completions or text format. If memory runs short, lower --batch-size, reduce --num-layers, or add --grad-checkpoint.
• NVIDIA and other GPUs: TRL's SFTTrainer applies the model's chat template automatically to conversational data; passing peft_config trains only LoRA, and adding a 4-bit quantization_config makes it QLoRA. The adapters themselves are managed by PEFT.
• Unsloth: supports LoRA, QLoRA, full fine-tuning, DPO and GRPO on NVIDIA, AMD, Intel and macOS. Its docs home page claims "2× faster with 70% less VRAM with no accuracy loss"; that is the vendor's own claim, and the page does not say what it is compared with.

QLoRA compresses the frozen base model to 4 bits, backpropagates through that 4-bit model, and updates only the LoRA adapters. The paper used it to fine-tune a 65B-parameter model on a single 48GB GPU. The 4 bits compress only the frozen weights; activation memory still grows with batch size and sequence length, and the training process lays out that memory budget.

A QLoRA adaptation in six steps

For the clipping router, the whole run looks like this:

1. Pick an open base model whose licence lets you use it. Read the licence first: whether commercial use is allowed, and whether you may publish modified weights. If you need to keep chat and tool-calling behaviour, start from the instruct version, not the base version.
2. Write the training data in that model's chat template. Each example is a list of messages, and the trainer applies the template. Hugging Face's chat templating docs note that with the wrong control tokens a model performs drastically worse. Compute the loss on the assistant replies only; loss masking explains why.
3. Train with QLoRA. On a Mac, load a quantized model in MLX-LM; on NVIDIA, use TRL or Unsloth. Before the real run, overfit a few dozen examples once to confirm the data and loss are wired correctly.
4. Compare before and after on the same fixed eval set. Fix that set before generating any paraphrases or augmented data, otherwise variants of one sentence land in both training and test; see data splits and leakage. For how to compare and which run settings to record, see evaluating a model on your own tasks.
5. Load the adapter in the runtime you actually use. MLX-LM loads it with --adapter-path, or merges it into the weights with mlx_lm.fuse; the llama.cpp server loads a GGUF adapter with --lora; in Python, use PEFT's PeftModel.from_pretrained.
6. Re-check tool calls and output format. If the training data has no tool calls, the model can lose the JSON and tool formats it used to follow. So rerun the exact-JSON and tool-call smoke tests from your eval; if tool use must survive, put conversations with tool calls into the training data (MLX-LM's tools format and TRL both support them).

Designing a useful adaptation experiment

The rules for splitting data and fixing the test set are in data splits and leakage. Use a labeled validation set for rank, learning rate, stopping time, and class thresholds.

For the router, measure each class’s recall, the cost of wrongly discarding a useful document, latency at the intended input length, and how often uncertain cases are deferred. Compare against a simple rule and the frozen-encoder baseline. Include negations, near duplicates, new topics, and changed class proportions. These stress the distinction the router is actually supposed to learn.

Training on traces can be valuable only if the trace labels encode outcomes you want repeated. A historical action is not automatically a correct action. Correct recurring mistakes before replaying them as supervision, then calibrate decision probabilities on held-out data as described in probability calibration.

The PEFT quicktour turns LoRA into a small implementation exercise: create an adapter configuration, run print_healthcheck() to see the trainable-parameter fraction and the adapter state, then save and reload the adapter. The save writes only two files, adapter_config.json and adapter_model.safetensors, and loading still needs the base model; comparing the two shows what an adapter checkpoint contains.

My reading