ster tune grpo

Invocation and purpose#

ster tune grpo --model <MODEL> --prompts <PROMPTS> --output <OUTPUT>
               [--revision <REVISION>] [--device <DEVICE>] [--reward <REWARD>]
               --group <GROUP> --iterations <ITERATIONS> --beta <BETA>
               --rank <RANK> --alpha <ALPHA> --learning-rate <LEARNING_RATE>
               --accumulation <ACCUMULATION> --max-new-tokens <MAX_NEW_TOKENS>
               --temperature <TEMPERATURE> --top-p <TOP_P>
               --max-sequence <MAX_SEQUENCE> --seed <SEED>
               [--targets <TARGETS>] [--layers <LAYERS>] [--warmup-steps <WARMUP_STEPS>]
               [--chat-template <auto|off>] [--precision <f32|f16|bf16>]

The only trainer in Ster that learns from text the model writes rather than text someone wrote for it. For each prompt it samples --group completions from the current policy, scores them, subtracts the group's own mean, and steps the policy toward the ones that beat it. See policy optimization for the objective.

--prompts reads {"prompts": ["…"]}, the same shape `ster extract` takes. Six prompts in the toy checkpoint's own vocabulary are checked in as `docs/examples/grpo-prompts.json`.

ster tune grpo --model ster-toy-model --prompts docs/examples/grpo-prompts.json \
  --output calm-grpo.safetensors --reward length --group 4 --iterations 3 \
  --beta 0.04 --rank 8 --alpha 16 --learning-rate 5e-3 --accumulation 1 \
  --max-new-tokens 16 --temperature 0.9 --top-p 0.95 --max-sequence 512 --seed 42

Those are the values the recorded run below trained with; when it was recorded they were the flags' defaults, which Ster no longer has. The run also predates the schedule decaying to zero — it decayed to a tenth of the learning rate — so the same command today moves the later steps less and reports slightly different numbers.

On the toy checkpoint that run reports 16 trainable tensors and 7168 trainable parameters over 18 optimizer steps, and a history of three iterations: mean reward 15.083333015441895, then 15.333333015441895, then 16.0, against a --max-new-tokens cap of 16; reward spread 1.5877132415771484, then 1.154700517654419, then exactly 0.0 as every completion reaches the cap; and mean KL rising 0.004853540100157261, 0.005867782048881054, 0.008418316021561623 as the policy leaves the reference.

The first_loss is exactly 0.0, and that is this objective's identity check. It requires three things to be right at once: a fresh adapter is the reference, so every KL term is zero; the k3 estimator is zero when the two distributions agree; and the advantages are mean-centred within the group, so the policy term is zero however the rewards came out.

Inputs and options#

--reward length is not a placeholder. It counts the tokens the policy actually emitted — a deterministic function with no model behind it — and it exists so the loop is runnable and checkable with no judge, no artifact and no download. If reward does not rise under it, the bug is in the loop. It counts tokens rather than characters because tokens are what the objective can move; a character count would reward whichever token happens to be spelled longest.

--reward <path> loads a reward artifact frozen beside the policy, and scores the whole sequence including the prompt, because a response is only good or bad relative to what it answers. That is genuinely a second model in memory: a judge is a different model from the thing it judges. The reward source is resolved before the policy is loaded, so a mismatched artifact is refused before an operator waits out a policy load to hear it.

Three details are decisions rather than defaults. A group whose completions all scored the same has advantages of exactly zero and contributes only its KL term — that falls out of the arithmetic rather than being special-cased, and it is why there is no epsilon in the denominator: a floor there would turn “no signal” into “amplify the rounding”. The importance ratio is exactly one in value, because with one gradient step per sampling round the sampling policy is the current policy; it is written as exp(logp - logp.detach()) anyway, which carries exactly the policy gradient, needs no second forward pass to recover the old log-probabilities, and keeps the reported loss on the published scale of -A + beta * KL. The KL is the k3 estimator, exp(d) - d - 1 for d = log pi_ref - log pi_theta, which is non-negative for every sample and unbiased for the divergence where the naive -d is neither.

Training loads no gateway and spends no quota. There is no judge model and no LLM-as-critic anywhere in the loop: the reward is a reward model you trained or a deterministic function of the completion.

Output and state#

Writes the same adapter pair every objective writes, with kind: "adapter". The checkpoint itself is untouched.

Stdout is a pretty JSON document with path and report. The report carries reward (the string length, or reward:<path>), prompts, trained_prompts, skipped_long, group, iterations, steps, beta, trainable_tensors, trainable_parameters, first_loss, final_loss, mean_reward, mean_kl, policy_loss, max_new_tokens, temperature, top_p, seed, rank, alpha, targets, layers, learning_rate, accumulation, chat_template, precision, and history.

history is one entry per iteration, in order, each carrying iteration, groups, completions, mean_reward, reward_spread, mean_kl, policy_loss, and mean_completion_tokens. The top-level mean_reward, mean_kl and policy_loss are the last entry's, because that is the iteration that describes the adapter that was written. The history is beside them because a single mean over a policy that moved the whole time hides exactly the trend the run exists to show.

training 16 adapter tensors (7168 parameters) at rank 8 across 4 layers
iteration 1/3 prompt 0 sampling 4 completions
iteration 1/3 step 1/18 loss 0.0000 reward 13.2500 kl 0.00000
iteration 1/3 mean reward 15.0833 spread 1.5877 mean kl 0.00485 policy loss 0.0002
iteration 3/3 mean reward 16.0000 spread 0.0000 mean kl 0.00842 policy loss 0.0003

Refusals#

• Every training and sampling number is the caller's: leaving out --group, --iterations, --beta, --rank, --alpha, --learning-rate, --accumulation, --max-new-tokens, --temperature, --top-p, --max-sequence or --seed is a usage error that names the flag, exit 2, before anything loads.
• group-relative policy optimization requires a group of at least two completions, because the group is the baseline.
• group-relative policy optimization requires a temperature above zero; argmax sampling would draw one identical completion per group.
• group-relative policy optimization requires a finite beta of zero or more.
• max_new_tokens must be greater than zero.
• The schedule is checked before the first gradient: group-relative policy optimization requires at least one iteration, group-relative policy optimization requires an accumulation of at least one prompt, and group-relative policy optimization requires a finite learning rate above zero.
• A reward source that is neither the keyword nor a file: reward source "nope" is neither the keyword length nor a file that exists.
• A generation adapter passed as a reward source: adapter artifact is a generation adapter, not a reward model.
• The prompt set is validated as ster extract validates it: prompt set contains no prompts.
• Skipping every prompt is every prompt is longer than the sequence limit, so there is nothing to train on.
• A completion whose first sampled token was end-of-sequence: a sampled completion was empty, so this group has nothing to score. The whole group is refused rather than the one draw dropped, because dropping it would bias the baseline upward by removing the group's worst member.