Why comparisons carry more information than demonstrations
A demonstration says "this response is acceptable." A comparison says "this response is better than that one." The second statement is a relative judgment, and relative judgments are easier for humans to make consistently and more informative for training. If you ask annotators to write the ideal response, you get one sample from a wide space of acceptable answers. If you ask them to choose between two candidates, you get a direct signal about the direction of improvement, even when neither candidate is perfect.
The reward model
The reward model is trained on the comparison data. For a pair where response A was preferred over response B for the same prompt, the training objective encourages the reward model to assign a higher score to A than to B. Once trained, the reward model acts as a stand-in for human judgment: it can score any new response quickly and cheaply, which is what makes it usable inside an optimization loop.
Optimizing the policy against the reward
With a reward model in place, the policy model generates responses to prompts, the reward model scores them, and a reinforcement learning algorithm updates the policy to increase the expected reward. A constraint keeps the policy from drifting too far from the supervised fine-tuned model, because an unconstrained policy will eventually exploit weaknesses in the reward model rather than genuinely improve. This is the stage commonly called RLHF: reinforcement learning from human feedback.
Why alignment is a separate stage
Pretraining, supervised fine-tuning, and preference alignment use different objectives, different data, and different starting points. Pretraining learns from raw text with no notion of a response. SFT learns from demonstrations with a response-shaped loss. Alignment learns from comparisons with a reward signal. Each stage assumes the previous one has already produced a model worth refining, which is why they are run in sequence rather than merged.