Pretraining, Supervised Fine-Tuning, and Preference Optimization

Architecture determines how a model processes information. Training determines which behavior its parameters acquire. Two models with the same Transformer architecture can behave differently because they saw different data, objectives, and update schedules. A useful first question is therefore: where does the target come from, and which outputs does the loss reward?

Pretraining learns from a broad source of examples

An autoregressive language model predicts the next token from earlier tokens. A sequence supplies its own targets: the text shifted by one position. Its negative log-likelihood is

$$L_{\mathrm{next}}=-\sum_t\log p_\theta(x_t\mid x_{<t}).$$

In an invented token sequence the / lamp / is / blue, predicting blue uses the prefix the / lamp / is. During training, many positions can be evaluated in parallel with a causal attention mask. At generation time, future generated tokens are unavailable, so the model extends the sequence one token at a time. Autoregressive generation develops that distinction.

An encoder can instead reconstruct masked tokens from surrounding context. BERT’s original paper describes masked-language-model pretraining and subsequent adaptation to downstream tasks. A bidirectional representation learned this way is useful for classification or extraction; it is not automatically an instruction-following assistant. See encoder and decoder families for the architectural distinction.

Self-supervision removes the need to label every example by hand, but a target and an error signal remain: the answer is the next token that actually appears in the text. What the model learns therefore depends on what it reads. Data sources, duplication, language balance, and document boundaries all change the result.

Supervised fine-tuning specifies a desired response

Supervised fine-tuning (SFT) trains on input–target examples. For a chat assistant, an input might be a request and the target an acceptable reply. For an encoder classifier, the target might be a class label. SFT is a training stage and data arrangement, not a requirement to use a chat architecture.

For conditional text generation, the objective commonly scores target response tokens given the prompt and previous target tokens. A loss mask can exclude prompt and padding positions. For example, if a training record contains 80 prompt tokens, 20 answer tokens, and 28 padding tokens, a response-only mean loss divides by the 20 scored tokens—not all 128 positions. Ignoring this distinction can quietly alter the relative weight of short and long answers.

Each SFT example teaches one particular decision, so whatever examples the training set lacks, the model lacks the matching decision. If the training set contains only duplicates as easy to spot as “identical to one already saved → discard”, the model will not learn what to do with an input that is half repeated and half new. So collect such boundary cases and annotator disagreements on purpose, and keep examples from the same source within the same data split. Transfer learning, LoRA, and distillation works this example through in full with a web-clipping router.

The InstructGPT paper provides a concrete pipeline: SFT on human demonstrations first, then people compare several responses to the same prompt, those comparisons train a reward model, and finally policy optimization raises the reward score. The last two steps use the preference data described in the next section.

Preferences compare alternatives

A demonstration says “produce this response.” A preference pair says “for this input, prefer A to B.” These convey different information. Two answers can both be factual while one follows the requested format more closely; two can both be wrong while one sounds more convincing.

In a reward-model pipeline, comparisons train a scorer, the reward model, and policy optimization then raises that score, while a constraint keeps the model from drifting too far from a reference policy (usually the model as it was before preference training).

The Direct Preference Optimization (DPO) paper shows that the reward can be rewritten so that the model being trained represents it directly. With that rewrite, the optimal solution of this objective, the one that includes “don’t drift too far from the original model,” can be written down in closed form. You can then skip the separately trained reward model and avoid having the model generate and score responses during training: pairs labelled “A wins, B loses” update the model through a simple classification loss. What DPO removes is the reward model and the sampling during training; the preference data still has to be collected.

Three imagined annotation pairs show what each pair teaches:

Input	Preferred response	Rejected response	What the pair teaches
Return exactly one label	keep	I would choose keep because…	output format
Resolve two conflicting dates	states the conflict and checks evidence	confidently picks one without support	handling uncertainty
Interpret a dangerous-looking code sample without executing it	explains the supplied code	refuses because of a keyword alone	task interpretation

If most of the data looks like the first row, the model learns to return a single label, and its factual reasoning does not improve as a result. So every preference label should correspond to a written evaluation rule, not just a vague “better.”

Open full-size image

Read left to right: demonstrations train the SFT policy, rankings train a reward model, and its scores guide PPO updates. This is the InstructGPT recipe, starting after pretraining. DPO, discussed above, uses a different preference-optimization procedure.

Ouyang et al., Training language models to follow instructions with human feedback (2022), Figure 2.

When a program can mark the answer, the reward can come from a checker

Preference data needs people to compare answers, but maths and coding problems can be marked by a program: if the final answer matches the reference or the code passes its tests, the reward is 1, otherwise 0. This is called a verifiable reward. Reinforcement learning with it does not ask the model to imitate anyone's worked solution; over many attempts, the model simply takes the paths that score more often.

GRPO (Group Relative Policy Optimization), introduced in the DeepSeekMath paper as a variant of PPO, is now a common way to run this step. It samples a group of answers to the same problem, marks each one, and uses the group's mean score as the baseline. Each answer's score minus the group mean, divided by the group's standard deviation, is its advantage: how much better it did than the rest of its group. PPO trains a separate value model to estimate that baseline; GRPO uses the group mean instead and saves that memory. The KL term that keeps the model close to a reference model is added directly to the loss.

Take a group of 4 answers: two are right and score 1, two are wrong and score 0, so the group mean is 0.5. The two right answers get a positive advantage, the two wrong ones a negative advantage of the same size, and the next update moves towards the way the right answers were written. If all 4 are right, or all 4 wrong, every score equals the group mean, every advantage is 0, and the problem contributes nothing to the update. So pick problems the base model sometimes solves: problems that are too easy or too hard give no signal.

The DeepSeek-R1 paper applied this route to a reasoning model. DeepSeek-R1-Zero skips SFT and runs reinforcement learning on the base model with rule-based marking. Maths problems require the final answer in a set format (for example inside a box) so it can be checked; a separate format reward requires the reasoning to sit inside designated tags. The paper reports that AIME 2024 pass@1 rose from 15.6% to 77.9% during training. It avoided neural reward models because they are open to reward hacking in large-scale reinforcement learning. R1-Zero's reasoning was hard to read and mixed English with Chinese. So the released DeepSeek-R1 starts with a small SFT "cold start" on long chain-of-thought data, then runs reinforcement learning, rejection sampling with SFT, and a final reinforcement-learning round aimed at helpfulness and harmlessness.

A checker rewards only what it checks. If the rule is "one point per citation added", the model learns to pile up citations whether or not they support the claim; reinforcement learning covers how rewards and proxy metrics get gamed. So write every condition you actually care about (correct answer, citations that support the claim, valid format) into the checker, and after training look beyond the score on held-out problems.

An individual can run this step on a small model: TRL's GRPOTrainer takes ordinary Python functions as reward functions, and Unsloth also lists GRPO among its training methods.

A numerical example of what improved loss means

Suppose the correct next-token probability rises from $0.2$ to $0.5$. Its contribution to natural-log loss falls from $-\log 0.2\approx1.609$ to $-\log0.5\approx0.693$. The objective only rewards the model for giving the token that actually appeared a higher probability; it never separately checks whether the whole sentence is true. The same goes for accurate citations and correct tool use: those tasks have different evaluation targets, and low text-prediction loss cannot establish them.

Similarly, increasing the relative probability of a preferred answer can improve a preference objective while teaching an accidental shortcut: longer answers win, confident wording wins, or one annotator’s style wins. Evaluate factuality, task success, format, and cost separately when these matter. Keep an untouched test set with prompt families or source documents separated from training, following data splitting and leakage.

Choosing a stage for a specific problem

If terminology from a domain is poorly represented, continued pretraining can expose the model to its text distribution. If it understands the input but uses the wrong output schema, supervised examples may address the gap more directly. If several valid responses differ in usefulness, preference data can teach that choice. Treat each as a hypothesis: compare it with a baseline that skips this training step, and invest only if it is clearly better. The three stages do not have to be run in order, one after another.

Suppose an assistant needs today’s document contents. Updating weights is a poor substitute for giving it current retrieved evidence. Suppose a small classifier consistently misroutes one recurring class: a focused labeled dataset may be more appropriate than adding a large generative model. The retrieval pipeline and transfer-learning note cover those options.

Training for a narrower objective can also make the model forget what it could do before. So besides the new task’s score, measure how much of the old capability survives, and pick a checkpoint from validation results. A label in a model’s name, such as base, instruct, or aligned, only says which training it went through; whether it works for your task is something you measure on your task.

My reading