Pretraining, Supervised Fine-Tuning, and Preference Optimization

Architecture determines how a model processes information. Training determines which behavior its parameters acquire. Two models with the same Transformer architecture can behave differently because they saw different data, objectives, and update schedules. A useful first question is therefore: where does the target come from, and which outputs does the loss reward?

Pretraining learns from a broad source of examples

An autoregressive language model predicts the next token from earlier tokens. A sequence supplies its own targets: the text shifted by one position. Its negative log-likelihood is

$$L_{\mathrm{next}}=-\sum_t\log p_\theta(x_t\mid x_{<t}).$$

In an invented token sequence the / lamp / is / blue, predicting blue uses the prefix the / lamp / is. During training, many positions can be evaluated in parallel with a causal attention mask. At generation time, future generated tokens are unavailable, so the model extends the sequence one token at a time. Autoregressive generation develops that distinction.

An encoder can instead reconstruct masked tokens from surrounding context. BERT’s original paper describes masked-language-model pretraining and subsequent adaptation to downstream tasks. A bidirectional representation learned this way is useful for classification or extraction; it is not automatically an instruction-following assistant. See encoder and decoder families for the architectural distinction.

Self-supervision reduces the need to label every example manually, but the objective still contains a target and an error signal. Dataset selection, duplication, language balance, and document boundaries shape what is learned. Low text-prediction loss does not by itself establish accurate citations or correct tool use, because those are different evaluation targets.

Supervised fine-tuning specifies a desired response

Supervised fine-tuning (SFT) trains on input–target examples. For a chat assistant, an input might be a request and the target an acceptable reply. For an encoder classifier, the target might be a class label. SFT is a training stage and data arrangement, not a requirement to use a chat architecture.

For conditional text generation, the objective commonly scores target response tokens given the prompt and previous target tokens. A loss mask can exclude prompt and padding positions. For example, if a packed training record contains 80 prompt tokens, 20 answer tokens, and 28 padding tokens, a response-only mean loss divides by the 20 scored tokens—not all 128 positions. Ignoring this distinction can quietly alter the relative weight of short and long answers.

Consider a fictional classification task with labels keep, summarize, and discard. An example teaching discard for “duplicate copy of an already saved note” supplies a particular decision rule. Repeating only easy duplicates will not teach how to handle an ambiguous or partially new note. Collect boundary cases and disagreements, and keep related examples within the same data split.

The InstructGPT paper provides a concrete pipeline using demonstrations, human comparisons, a reward model, and policy optimization. It establishes how that pipeline was constructed; it does not make that exact sequence compulsory for every contemporary model.

Preferences compare alternatives

A demonstration says “produce this response.” A preference pair says “for this input, prefer A to B.” These convey different information. Two answers can both be factual while one follows the requested format more closely; two can both be wrong while one sounds more convincing.

In a reward-model pipeline, comparisons train a scorer, and policy optimization seeks outputs with high scores, commonly with a constraint discouraging excessive drift from a reference policy. Direct Preference Optimization (DPO) instead fits the policy directly from preferred and rejected responses under its reference-policy formulation. It avoids a separate reward-model training and online policy-optimization loop in that formulation; collecting preferences and validating behavior are still necessary.

An invented pair makes the distinction concrete:

Input	Preferred response	Rejected response	What the pair teaches
Return exactly one label	keep	I would choose keep because…	output format
Resolve two conflicting dates	states the conflict and checks evidence	confidently picks one without support	handling uncertainty
Sort a dangerous-looking but inert code sample	explains the supplied code	refuses because of a keyword alone	task interpretation

These are example annotation intentions, not measured model outcomes. A dataset full of the first row could improve format adherence while leaving factual reasoning unchanged. Preference labels should therefore record a clear evaluation rule, not just a vague “better.”

Open full-size image

Read left to right: demonstrations train the SFT policy, rankings train a reward model, and its scores guide PPO updates. This is the InstructGPT recipe, starting after pretraining. DPO, discussed above, uses a different preference-optimization procedure.

Ouyang et al., Training language models to follow instructions with human feedback (2022), Figure 2.

A numerical example of what improved loss means

Suppose the correct next-token probability rises from $0.2$ to $0.5$. Its contribution to natural-log loss falls from $-\log 0.2\approx1.609$ to $-\log0.5\approx0.693$. The objective rewards placing more probability on the observed token. It has not separately checked whether the whole sentence is true.

Similarly, increasing the relative probability of a preferred answer can improve a preference objective while teaching an accidental shortcut: longer answers win, confident wording wins, or one annotator’s style wins. Evaluate factuality, task success, format, and cost separately when these matter. Keep an untouched test set with prompt families or source documents separated from training, following data splitting and leakage.

Choosing a stage for a specific problem

If terminology from a domain is poorly represented, continued pretraining can expose the model to its text distribution. If it understands the input but uses the wrong output schema, supervised examples may address the gap more directly. If several valid responses differ in usefulness, preference data can teach that choice. These are hypotheses to test against a baseline, not a mandatory sequence of expensive runs.

Suppose an assistant needs today’s document contents. Updating weights is a poor substitute for giving it current retrieved evidence. Suppose a small classifier consistently misroutes one recurring class: a focused labeled dataset may be more appropriate than adding a large generative model. The retrieval pipeline and transfer-learning note cover those options.

Training for a narrower objective can also damage earlier capabilities. Evaluate retained tasks, not just the new benchmark, and select a checkpoint using validation results. A model’s stage label—base, instruct, aligned—summarizes a training history; it does not replace direct evidence for the intended use.

My reading · Opened notes are remembered in this browser.