How LLMs Are Trained: Pretraining, Instruction Tuning and Alignment

Modern chat models are built in stages. Pretraining teaches the model language and world knowledge from huge text collections. Supervised instruction tuning teaches it to follow requests. Preference tuning then shapes which answers it prefers. Your own fine-tuning is usually one more stage added on top.

Stage 1: pretraining

The model reads very large amounts of text and learns one task: predict the next token. Given "The capital of France is", it learns to put high probability on " Paris". Doing this across trillions of tokens teaches grammar, facts, code syntax and many patterns of reasoning, because predicting text well requires modelling what the text is about.

The result is a base model. It is a powerful text continuer, but not an assistant. Ask it a question and it may continue with more questions, because that is what text on the web often looks like. Pretraining needs large clusters and is out of reach for most teams.

Stage 2: supervised fine-tuning

To make the base model useful, it is trained on curated pairs of instructions and good responses. The objective is still next-token prediction, but now only on the response tokens, and the data looks like conversations. After this stage the model answers questions and follows formats. Models released at this stage are often called instruct or chat models.

This is the same technique you will use. SFT on a few thousand examples is the workhorse of customization.

Stage 3: preference tuning and alignment

Supervised data shows one good answer. It cannot easily say "this answer is better than that one". Preference methods use comparisons. In classic RLHF, humans rank outputs, a reward model learns those rankings, and reinforcement learning optimizes the policy against the reward model. DPO is a simpler alternative that trains directly on chosen and rejected pairs without a separate reward model. More recent methods use rewards that can be checked automatically, such as passing unit tests or matching a math answer.

This stage pushes helpfulness, safety behaviour and style. Posts 61 to 68 cover it.

Where your work fits

raw text  --pretraining-->  base model
base model  --SFT on instructions-->  instruct model
instruct model  --preference tuning-->  aligned chat model
aligned chat model  --YOUR SFT (+ optional DPO)-->  your task model

You can also start one step earlier, from a base model, if you want full control of the chat format and have a lot of data. Post 6 helps you pick.

Seeing the next-token objective

The next code loads a small model, runs a sentence through it, and shows the loss that every stage minimizes. Set the model name through an environment variable.

import os
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

name = os.environ["BASE_MODEL"]
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name)

batch = tok("Fine-tuning adapts a pretrained model.", return_tensors="pt")
with torch.no_grad():
    out = model(**batch, labels=batch["input_ids"])
print("loss:", out.loss.item())

Passing labels equal to the inputs makes the library shift them internally, so each position predicts the next token. The printed value is the average negative log-likelihood. Training lowers it on your data.

Why this matters for you

• Your data should look like the stage you are continuing. Chat data for chat models, raw text for continued pretraining.
• A model that has already been aligned carries behaviours, including refusals, that your tuning may weaken or keep.
• Small learning rates matter because you are adjusting a finished model, not building one.
• Capabilities come mostly from stage 1 and large stage 2 and 3 efforts, so choose a base model with the skills you need.

With this map in mind, the next post helps you choose between starting from a base or an instruct checkpoint.

Compute and data at each stage

The stages differ enormously in scale. Pretraining uses trillions of tokens and weeks of cluster time. Instruction tuning typically uses thousands to millions of curated examples. Preference tuning uses far fewer, often tens of thousands of comparisons. Your own project sits at the small end: hundreds to tens of thousands of examples. This is why quality is the dominant factor. With so little data, each example has real influence on the final behaviour, and one badly written batch of examples can shift the model noticeably.

It also explains why hyperparameters differ. In pretraining, runs are long. In fine-tuning, you use small learning rates and a handful of epochs, because the model is already good and you only want to steer it. Too aggressive a run overwrites what pretraining built.

Common misconceptions

• "Fine-tuning retrains the whole model from scratch." No, it starts from existing weights and changes them slightly.
• "The base model is the raw, unfiltered truth." It is a text predictor trained on web-scale data with all its biases and errors.
• "Alignment is a single switch." It is a sequence of data and objectives, and later tuning can weaken earlier alignment, which is why post 93 covers safety after fine-tuning.