Overview#

A supervised example says “given this prompt, produce this completion”. A contrastive pair says something weaker and often truer: that one response is better than another. Supervised fine-tuning cannot use the second statement, because it has no way to express “and not that”. A preference objective can.

Ster reads a preference from the pair set it already has. The positive side is the chosen response and the negative side the rejected one, so a set written for steering trains a preference without being rewritten, and no new file format exists.

The objective#

Direct preference optimization skips the reward model that reinforcement learning from human feedback would fit first, and optimizes the policy against the preference directly. For a pair, it forms the log-ratio of the policy to a frozen reference on each side, takes the difference, and pushes it up:

text

margin = (log pi(chosen)  - log pi_ref(chosen))
       - (log pi(rejected) - log pi_ref(rejected))

loss   = -log sigmoid(beta * margin)

beta sets how hard the reference pulls back. The reference term is what stops the trivial solution: without it, the policy could raise the chosen side's probability by becoming more confident about everything, and the objective could be satisfied without any preference being learned.

The quantity beta * (log pi - log pi_ref) is the implicit reward the derivation produces — DPO's central result is that optimizing the expression above is equivalent to fitting a reward model and optimizing against it. `ster tune dpo` reports it per side as mean_chosen_reward and mean_rejected_reward, and accuracy is the share of pairs where the chosen side earns the larger one.

How IPO differs#

IPO is the same machinery with the sigmoid replaced by a squared error against a fixed target, which is why supporting it costs one match arm rather than a second trainer:

text

loss = (margin - 1 / (2 * beta)) ** 2

The difference that matters is what the two objectives do when a pair is easy. The sigmoid is never satisfied — a larger margin is always a smaller loss — so DPO keeps pushing on pairs it has already got right, which is one route to overfitting a small preference set. IPO's target is a specific finite number, so a margin that reaches it stops attracting gradient.

IPO also scores length-normalized log-probabilities where DPO scores sums. That follows from the fixed target: a target a long sequence could reach by length alone is not a preference signal. --loss dpo is the default and --loss ipo selects the other; nothing else is accepted.

Why the reference is free#

The objective needs a frozen model's opinion of the same two sequences, and the obvious implementation loads the checkpoint twice. Ster does not have to. Runtime::load_trainable maps the base weights read-only and registers only the low-rank factors, and every adapter's B starts at zero — so the base weights are the model the policy was initialised as. Skipping the low-rank update at every projection for one pass reproduces the reference exactly, at the cost of one enum comparison per projection instead of a second multi-gigabyte checkpoint. This is the reason adapters are attached to the decoder rather than folded into the projection weights. See LoRA adapter.

It also makes the run self-checking. With an identity adapter the policy and the reference are the same model, so every margin is exactly zero and the first loss is exactly -log sigmoid(0), which is ln 2 — or exactly (1 / (2 * beta)) ** 2 under IPO, 25.0 at the default beta. Both are what the toy checkpoint reports. The policy pass runs the composed kernels and the reference pass the fused ones, so the two agreeing at step one is each checking the other. See differentiable pass.

Because a frozen model cannot move, the reference is scored once for the whole run rather than once per epoch.

Why a pair is scored whole#

A pair set carries two complete texts and no prompt field, so Ster scores each side end to end rather than splitting it. That is not an approximation of a prompt-conditioned objective. When the two sides share a leading prefix — which is exactly what `ster pairs synthesize` writes as Question: …\nAnswer: … — that prefix sits at the same positions in both sequences, so its log-probability is the same expression on both sides of the margin. It cancels out of the value and out of the gradient alike.