Policy optimization
Learning from text the model writes: sample a group of completions per prompt, normalize the advantage inside the group, and step the policy toward the ones that beat their own average, on a KL leash to the frozen base.
Overview#
Supervised and preference training both learn from text that already exists. Policy optimization learns from text the model produces: the policy writes, something scores what it wrote, and the gradient moves it toward the better attempts. There is no target sequence anywhere in it.
Ster implements group-relative policy optimization. `ster tune grpo` runs it.
The group is the baseline#
A policy gradient needs to know whether a reward was good, which normally means a second network — a value model — trained alongside the policy to predict the reward it should have expected. GRPO removes that network. It samples several completions for the same prompt and uses their mean as the baseline:
A_i = (r_i - mean(r_1..r_G)) / stdev(r_1..r_G)The advantage is then scale-free by construction: what matters is whether a completion beat the others for the same prompt, not what the reward's units were. What it costs is --group generations per prompt, and that is the dominant cost of the whole loop. Two is the smallest group with a baseline that is not the sample itself, and a smaller one is refused.
A group whose completions all scored the same has deviations of exactly zero, so its advantages are zero and it contributes only its KL term. That falls out of the arithmetic rather than being special-cased, and it is why there is no epsilon in the denominator: a floor there would turn “no signal” into “amplify the rounding”.
The objective#
Per token of each sampled completion, the objective is the importance ratio times the advantage, less a KL penalty, averaged over the completion's own tokens before the group mean so a long completion does not outvote a short one on length alone:
ratio = exp(log pi_theta(t) - log pi_old(t))
kl = exp(d) - d - 1, where d = log pi_ref(t) - log pi_theta(t)
loss = -mean( ratio * A - beta * kl )The ratio is exactly one in value, because with one gradient step per sampling round the sampling policy is the current policy. It is written out anyway rather than collapsed to log pi * A: the two carry the same gradient, but the ratio form needs no second forward pass to recover log pi_old, and it keeps the reported loss on the published scale of -A + beta * KL rather than an unbounded log-probability.
The KL uses the k3 estimator, exp(d) - d - 1, which is non-negative for every individual sample and unbiased for the divergence. The naive -d is neither: it is negative about half the time, which makes a per-step KL reading uninterpretable. The reference is the frozen base — the same weights with the adapters skipped — reached exactly as preference optimization reaches it.
Together those three make the first step's loss exactly zero, which is this objective's identity check: a fresh adapter is the reference so every KL term is zero, and the advantages are mean-centred so the policy term is zero however the rewards came out.
Where the reward comes from#
--reward names the source, and the two arms exist for different reasons.
--reward | What it scores | Why it exists |
|---|---|---|
length | The number of tokens the policy emitted. | A deterministic function with no model behind it, so the loop is runnable and checkable with no judge, no artifact and no download. If reward does not rise under it, the bug is in the loop. |
| a file path | A reward model applied to the prompt and completion together. | The real thing. Loaded frozen beside the policy — a judge is genuinely a different model from what it judges. |
The length reward counts tokens rather than characters because tokens are what the objective can move; a character count would reward whichever token happens to be spelled longest. It is also the clearest demonstration the loop works: on the toy checkpoint with a 16-token cap, mean reward rises 15.08, 15.33, 16.00 across three iterations while the spread collapses to zero as every completion reaches the cap, and mean KL rises the whole way as the policy leaves the reference.
There is no judge model and no LLM-as-critic anywhere in this loop. Ster's one hosted call is the optional pair writer behind `ster pairs synthesize --generator brama`, which produces training data and is not wired into any gradient.
Sampling#
Each iteration re-samples from the current policy, which is why the traversal is called an iteration rather than an epoch: it is not seeing the same data twice. --temperature must exceed zero, because argmax sampling would draw one identical completion per group and leave the baseline with nothing to compare. Every draw in the run gets its own seed advanced from --seed, so the run stays reproducible from one number while the group stays varied.
A completion whose first sampled token is end-of-sequence has nothing to take a gradient through, and dropping it would bias the baseline upward by removing the group's worst member — so the whole group is refused instead.