Direct Preference Optimization (DPO) — AI Glossary

Reference details

Related terms

• RLHF (Reinforcement Learning from Human Feedback)
• Reward Model
• AI Alignment

Example: What DPO removes

RLHF needs three stages: supervised fine-tune, train a separate reward model, then optimise the policy against it with PPO. DPO uses the same preference pairs but derives a loss the policy can be trained on directly, deleting the reward model and the RL loop. Fewer moving parts, fewer ways to diverge — which is why it became the default for open-weight alignment.

Commonly confused with

DPO is not a different kind of feedback. It consumes exactly the same chosen-versus-rejected preference data as RLHF; only the optimisation differs. If your preference data is biased, DPO reproduces the bias just as faithfully.

When to use it

Reach for it when:

• You have preference pairs and want alignment without RL infrastructure
• Stability and reproducibility matter more than the last increment of quality
• The team cannot maintain a reward model and a PPO loop

Reach for something else when:

• You need online exploration against a live reward signal
• Your preference data is thin or inconsistent — no method rescues that
• The objective is verifiable, where training against the verifier is stronger

Referenced by

This term is used by the following ModelRefs references:

• learn fine tuning
• become llm engineer

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Direct Preference Optimization (DPO) — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path