DPO vs RLHF: which should you use to align a model?

Use DPO first: it learns from the same preference labels as RLHF but drops the reward model and the online sampling loop, so it is cheaper, faster and easier to debug. On Nodus the whole DPO path runs on managed runtimes (nodus/sft, then nodus/dpo), while RLHF adds a nodus/reward-model stage and a PPO loop you run as your own Job. Move to RLHF with PPO when evaluations show you need online exploration, as on math and code, and you can afford the tuning.

What does each pipeline actually involve?

Both pipelines start from the same two ingredients: a supervised fine-tuned (SFT) model and a set of preference pairs, each a prompt with a chosen and a rejected answer. They differ in what happens next.

RLHF, as in InstructGPT, trains a reward model on the pairs, then runs Proximal Policy Optimization (PPO). PPO samples fresh answers from the policy, scores them with the reward model, and updates the policy. DPO (Direct Preference Optimization) replaces both steps with "only a simple classification loss" on the pairs, "eliminating the need for sampling from the LM during fine-tuning."

Runtimes are from the Nodus training guide, which lists nodus/reward-model as "a reward model for RLHF." The key point for your budget is the second row. RLHF does not need more human labels than DPO. It spends more on GPUs and on engineering.

How much longer does the RLHF loop take?

Two published runs put the PPO stage at roughly 1.3 to 8 times the GPU time of a DPO pass on the same data. The two runs differ in model size, hardware and reward model, so read the range as wide, not precise.

The AlpacaFarm paper trained LLaMA 7B on one 8 x A100 machine. PPO peaked in "less than 2 hours of compute time on the full machine", while one DPO epoch on half the machine "takes less than 3 hours". That is under 16 A100-hours for PPO against under 12 for DPO, a ratio of 16 / 12 = 1.33.

The Unpacking DPO and PPO paper from Ivison et al. reports much larger runs on 13B models and UltraFeedback, on TPUv3 pods of 128 or 256 cores. PPO with a 13B reward model "takes 3 days on a 256-size pod", the reward model about 10 hours, and DPO 9 hours. PPO alone is 72 / 9 = 8 times the DPO time, and the RLHF stages together are (72 + 10) / 9 = 9.1 times. The paper does not state DPO's pod size in that passage, so treat the ratio as wall-clock hours, not matched hardware.

At 13B scale, the RLHF stages took 82 hours against 9 for DPO on the same preference data.

What does each pipeline cost end to end on an 8B model?

On Nodus list prices, the GPU bill for aligning Qwen3-8B with QLoRA comes to about $2.02 for SFT plus DPO, and $2.63 to $5.70 for SFT, a reward model and PPO. We built this from the dry-run rule in the Nodus docs and the public dataset sizes of the Zephyr recipe, which used UltraChat for SFT and UltraFeedback for DPO.

The inputs are these. UltraChat 200k has 207,865 SFT training rows, and UltraFeedback binarized has 61,135 training pairs. The runtime defaults are batchSize: 4 and gradientAccumulation: 4, so one optimizer step covers 16 examples on one GPU. The Qwen3-8B QLoRA preset runs on 1 x L40S, RTX A6000 or A100 80GB, and the dry-run estimate uses "the slowest GPU type you allowed." That is the RTX A6000 at 1.3 s per step, listed at $0.333/hr on the Nodus pricing page.

• SFT, one epoch: 207,865 / 16 = 12,992 steps. 12,992 × 1.3 s = 16,890 s, or 4.69 h. 4.69 h × $0.333 = $1.56.
• DPO, one epoch: 61,135 / 16 = 3,821 steps. 3,821 × 1.3 s = 4,967 s, or 1.38 h. 1.38 h × $0.333 = $0.46.
• Reward model, one epoch: the same 3,821 steps at the same published step time, so $0.46.
• PPO: no managed runtime, so no published step time. At the 1.33x and 8x ratios above, it costs $0.46 × 1.33 = $0.61 to $0.46 × 8 = $3.68.

DPO pipeline: $1.56 + $0.46 = $2.02. RLHF pipeline: $1.56 + $0.46 + $0.61 = $2.63, up to $1.56 + $0.46 + $3.68 = $5.70. Prices checked November 13, 2026.

The PPO loop can nearly triple the GPU bill, yet both pipelines stay under $6 at this size.

Treat these as floors. The runtimes publish one step time per GPU type, re-measured nightly, not one per model, so an 8B step can run longer. Read the dry run for your own model before you trust the figure. The PPO range is an extrapolation from two papers, not a Nodus measurement.

Why do the preference labels cost more than the GPUs?

Human labels dominate the bill at almost any model size you would fine-tune on one GPU. AlpacaFarm paid annotators a median $21 an hour, "leading to a one-time $3000 cost" for its 10k-pair preference split. That is $3,000 / 10,000 = $0.30 per pair.

At that rate, a dataset the size of UltraFeedback would cost 61,135 × $0.30 = $18,340.50 to label by hand. AlpacaFarm's simulated annotator cost "$6" per 1,000 outputs, "50x cheaper than human annotation." At that 2023 rate the same pairs come to 61,135 × $0.006 = $366.81. UltraFeedback itself was labeled by a model, which is how Zephyr needed "no human annotation."

Human labels cost about 9,000 times the SFT plus DPO compute, and RLHF needs the same labels.

Since both pipelines need the same pairs, the label bill does not decide DPO vs RLHF. It does decide how many experiments you can afford, which favors the method that lets you reuse one dataset quickly.

Does RLHF with PPO give a better model than DPO?

Often slightly, when PPO is tuned well and the task rewards exploration. The DPO paper found it "matches or improves response quality in summarization and single-turn dialogue" against PPO-based RLHF. Later work with larger budgets found the opposite.

Xu et al., in Is DPO Superior to PPO for LLM Alignment?, report that "PPO is able to surpass other alignment methods in all cases." The gap was largest on code: on CodeContests their 34B PPO model reached 22.4% on the 10@1k test metric, against 0.0% for DPO and 3.2% for iterative DPO. They also show DPO "is significantly affected by the distribution shift between the model outputs and the preference dataset."

Ivison et al. measured smaller gaps. PPO beat DPO "by up to 2.5% in math and 1.2% in general domains", while better preference data gave "up to 8%" in instruction following and truthfulness. Their ranking puts data quality first and the algorithm second. Fix your pairs before you switch algorithms.

Is online or iterative DPO the middle ground?

Yes, and it is usually the next step after offline DPO. Online DPO samples new answers from the current model, labels them, and trains on them in rounds. That fixes the off-policy problem Xu et al. describe without adding a value model.

Guo et al., in Direct Language Model Alignment from Online AI Feedback, sample two responses per prompt each iteration and ask an LLM which is better. Their human evaluations show this "outperforms both offline DAP and RLHF methods." The RLHF Workflow report builds the same loop with a trained preference model as the labeler.

On Nodus you can assemble it from parts. Train a scorer with nodus/reward-model, generate and score new pairs in a Job, then train each round with nodus/dpo. A Pipeline chains Jobs and "the whole Pipeline shares one spending cap." Each round adds sampling and labeling cost, so budget per round.

When is RLHF with PPO the better choice?

Pick PPO when the model has to discover answers that are not in your dataset. Hard code and math, where Xu et al. and Ivison et al. both found the clearest PPO gains, fit that description. So does a reward model you trust more than any fixed set of pairs.

PPO also lets you train on extra prompts that have no labels at all, since the reward model scores the samples. Ivison et al. found prompts matched to the target task helped "in domain-specific settings" but had "limited to no effect" on overall performance.

The cost is real engineering. Xu et al. name advantage normalization, large batches and a moving-average reference model as the key factors for PPO. With a batch of 64 and none of those, their PPO dropped APPS introductory pass@5 from 38.6% to 18.0%. If nobody on your team can tune that loop, DPO's simplicity is worth more than PPO's ceiling.

For tasks scored by a program rather than a learned reward, our PPO vs DPO vs GRPO guide covers the GRPO route.

Which should you pick?

We use this rule, built from the papers and costs above:

1. Have preference pairs and a chat or tone goal: run SFT, then offline DPO on a held-out test set.
2. DPO results look off-distribution or stall: generate new pairs from your current model and run iterative DPO.
3. The task is hard code or math and you trust your reward model: try PPO, and budget 1.3 to 8 times the DPO stage's GPU time plus tuning runs.
4. Gains are small after any of these: improve the preference data before changing algorithms.

To start step 1 on Nodus, point a DPO run at the SFT model in your Volume. This is the example from the training guide:

spec: runtime: nodus/dpo model: {volume: {name: my-sft-model, revision: 3}} # a model you trained earlier, by Volume  data: volume: prefs format: JSONL columns: {prompt: prompt, chosen: chosen, rejected: rejected} parameters: {steps: 400} 

Price it before it runs with nodus apply -f dpo.yaml --dry-run -o yaml, which reports the cost to completion. If you are deciding between a QLoRA and a full run for any of these stages, our LoRA vs full fine-tuning guide has the memory and cost math.

FAQ

How many preference pairs do you need?

Published sets range widely. AlpacaFarm used 10k pairs, UltraFeedback binarized has 61,135 training pairs, and Anthropic's HH-RLHF has 160,800 training rows. Start with a few thousand clean pairs from your own domain and measure before you buy more.

Does a lower DPO loss mean a better model?

No. The Nodus training guide warns that "lower DPO loss does not establish better generated answers." Compare generated answers from the starting and trained models on an untouched test set.

Can you run PPO on Nodus today?

Yes, as your own training script in a Nodus Job, since there is no managed PPO runtime. You can train its reward model with nodus/reward-model and point PPO at the result.

Does DPO still need an SFT step first?

Usually. Xu et al. found DPO performed poorly when the starting model's outputs differed from the preference data, and extra SFT on that data helped. nodus/orpo is the exception, since it combines SFT and preference training in one pass.

Originally published on the Nodus blog.