Why a pretrained model does not follow instructions
Pretraining optimizes the probability of the next token given all preceding text, over a corpus that contains questions, answers, code, dialogue, and lists in no particular arrangement. Nothing in that objective marks a question as something that should be answered. A prompt like "What is the capital of France?" is just a prefix, and the model completes it with whatever continuation is statistically most likely across the corpus, which may be another question, a quiz heading, or a list of unrelated capitals. The model has the knowledge but no reason to apply it in the form the user expects.
What changes during fine-tuning
Fine-tuning reuses the same architecture and the same loss. What changes is the data distribution and where the loss is applied. In a training example, the prompt tokens are fed in as context, and the loss is computed only over the response tokens. So the gradient rewards the model for raising the probability of the demonstrated response given that prompt, and it does not reward or penalize the model for the prompt itself. Repeating this over many instruction-response pairs shifts the conditional distribution toward answer-shaped continuations for instruction-shaped inputs.
A single training pair
Consider the pair: instruction "Summarize this paragraph in one sentence," followed by a paragraph, followed by a one-sentence summary written by a human. During fine-tuning, the model reads the instruction and paragraph as context, then is graded on how much probability it assigns to each token of the human summary. If the model instead assigns high probability to a continuation like "This paragraph discusses..." followed by a restatement, the loss is high and the gradient corrects it. After many such pairs, the model learns the shape of a summary response, not just the content of any one summary.
The ceiling of imitation
SFT can only reproduce what the demonstrations show. If every demonstrated response is polite and concise, the model tends to be polite and concise. But imitation gives no direct signal about which of two plausible responses is better when both are acceptable in format. That gap is what preference-based alignment addresses.