ster tune reward

Invocation and purpose#

ster tune reward --model <MODEL> --output <OUTPUT> [--pairs <PAIRS>]
                 --rank <RANK> --alpha <ALPHA> --epochs <EPOCHS>
                 --learning-rate <LEARNING_RATE> --accumulation <ACCUMULATION>
                 --max-sequence <MAX_SEQUENCE> --batch-size <BATCH_SIZE> --seed <SEED>
                 [--revision <REVISION>] [--device <DEVICE>] [--targets <TARGETS>]
                 [--layers <LAYERS>] [--warmup-steps <WARMUP_STEPS>]
                 [--chat-template <auto|off>] [--precision <f32|f16|bf16>]

Every other trainer in Ster produces a model that writes text. This one produces a model that judges it: one scalar per sequence, higher for the response an operator preferred. It exists because policy optimization needs a reward, and a reward that is a language model's own likelihood is circular. See reward model for what the artifact is and how it is gated.

--pairs takes the same contrastive pair set the other commands read, {"trait_name": "…", "pairs": [{"positive": "…", "negative": "…"}]}. The positive side is the response the head learns to score higher.

ster tune reward --model ster-toy-model --pairs docs/examples/pairs.json \
  --output calm.reward.safetensors --epochs 3 --accumulation 4 --learning-rate 5e-3

On the toy checkpoint that run reports 17 trainable tensors and 7232 trainable parameters — 7168 in the adapters and 64 in the head, which is one row of the hidden width — a first_loss of 0.6931471824645996, a final_loss of 0.48404383659362793, an accuracy of 1.0, and a mean_score_margin of 0.45127108693122864, with mean_chosen_score 0.11224722117185593 against mean_rejected_score -0.3390238583087921.

The first loss is exactly ln 2 for the same structural reason it is in `ster tune dpo`: the head is initialised to zeros, so every sequence scores exactly zero, so the Bradley-Terry probability is one half.

Inputs and options#

The objective is Bradley-Terry: the probability that the chosen response beats the rejected one is sigmoid(score_chosen - score_rejected), and the loss is the negative log of it. Two properties fall straight out of that difference rather than being chosen. The head has no bias, because a bias is added to both scores and cancels, so it would be a parameter with an identically zero gradient. And only differences are identified by the objective, so the scores in the report are meaningful against each other and against no external unit.

The head is one row of hidden_size weights applied to the last position's residual state after the final norm, which is the only position that has attended to the whole sequence. It is initialised to zeros rather than drawn: a single output row has no symmetry for a random draw to break, and the gradient at zero is the difference of the two residual states, so it learns on the first step.

The head and the adapters train in one VarMap under one optimizer, which is why the tensor count is one higher than an adapter-only run's. A reward pass computes no vocabulary projection at all — the head reads the residual stream and never looks at a token distribution, and on a real checkpoint that projection is the widest matmul in the pass.

Training loads no gateway, spends no quota, and writes nothing but the artifact pair.

Output and state#

Writes an artifact of a different kind. The safetensors file carries the adapter factors plus one extra tensor named reward.head, shaped [1, hidden_size], and the sidecar declares kind: "reward" rather than kind: "adapter". The two travel in one file because a head reads a residual stream the adapters shaped: pairing one with adapters it never saw would produce scores that mean nothing, and the artifact does not offer that as a possibility.

kind is #[serde(default)] and defaults to adapter, so every sidecar written before the field existed still loads and still means what it meant. The field is additive, which is why the schema version does not move.

Stdout is a pretty JSON document with path and report. The report carries pairs, trained_pairs, skipped_long, epochs, steps, trainable_tensors, trainable_parameters, first_loss, final_loss, mean_final_epoch_loss, accuracy, tied_pairs, mean_chosen_score, mean_rejected_score, mean_score_margin, rank, alpha, targets, layers, learning_rate, accumulation, batch, chat_template, and precision, with accuracy, tied_pairs and the scores measured over the final epoch.

One reading of that report is worth stating, because it looks alarming and is not. A run short enough that the head never moved reports accuracy 0.0 and mean_score_margin 0.0000. That is not a head that ranked every pair backwards; it is a head that has not moved. It starts at zeros, so every pair scores an exact tie, and a tie is not a strict win. tied_pairs is what tells the two apart from the document alone: a head that never moved ties every pair it trained on, so tied_pairs equals trained_pairs, while a head that learned the order backwards ties none of them and reports a negative mean_score_margin beside the same accuracy. It counts exact equality rather than closeness, because the tie it exists to name is two sides of one pair run through identical weights, not two scores a trained head happens to find similar. Read the tie count first and the accuracy second, and give the run more than one epoch before either number means anything.

Stderr names the trainable tensor count — as tensors, not adapter tensors, because the seventeenth one is a head — then one line per optimizer step and one per epoch. The epoch line carries the tie count beside the accuracy for that reason. On the run above the head had partly moved after one epoch: four of the eight pairs were still exact ties, so its 0.250 accuracy is two strict wins and four ties rather than two wins and six losses.

training 17 tensors (7232 parameters) at rank 8 across 4 layers
epoch 1/3 mean loss 0.6890 accuracy 0.250 ties 4/8 score margin 0.0094
epoch 3/3 mean loss 0.5001 accuracy 1.000 ties 0/8 score margin 0.4513

Refusals#

• Every training number is the caller's: leaving out --rank, --alpha, --epochs, --learning-rate, --accumulation, --max-sequence, --batch-size or --seed is a usage error that names the flag, exit 2, before anything loads.
• The pair set is validated as everywhere else: pair set {label} contains no pairs and pair set {label} contains an empty positive or negative prompt.
• The schedule is checked before the first gradient: reward modeling requires at least one epoch, reward modeling requires an accumulation of at least one pair, and reward modeling requires a finite learning rate above zero.
• max_sequence must be at least two tokens, so that one token can predict another.
• Skipping every pair is every pair is longer than the sequence limit, so there is nothing to train on.
• The artifact validates in both directions before it reaches disk: reward artifact is missing tensor reward.head, which is the head it exists to carry, and adapter artifact carries a reward.head tensor but declares kind adapter.
• The result is not a generation adapter, and every command that wants one says so: `ster generate --adapter`, `ster tune merge` and `ster tune evaluate` all refuse it with adapter artifact is a reward model, not a generation adapter.