ster tune dpo

Invocation and purpose#

ster tune dpo --model <MODEL> --output <OUTPUT> [--pairs <PAIRS>]
              --rank <RANK> --alpha <ALPHA> --beta <BETA> --epochs <EPOCHS>
              --learning-rate <LEARNING_RATE> --accumulation <ACCUMULATION>
              --max-sequence <MAX_SEQUENCE> --batch-size <BATCH_SIZE> --seed <SEED>
              [--revision <REVISION>] [--device <DEVICE>] [--targets <TARGETS>]
              [--layers <LAYERS>] [--loss <LOSS>] [--warmup-steps <WARMUP_STEPS>]
              [--chat-template <auto|off>] [--precision <f32|f16|bf16>]

Supervised fine-tuning can only say “produce this”. A preference set says “prefer this over that”, which is a different statement and needs a different objective: raise the policy's log-probability of the chosen side, lower it on the rejected side, both measured relative to a frozen reference so the policy cannot win by becoming more confident about everything at once. See preference optimization for the objective itself.

--pairs takes the contrastive pair set `ster vector train` already reads, {"trait_name": "…", "pairs": [{"positive": "…", "negative": "…"}]}. The positive side is the chosen response and the negative side the rejected one, so a set written for steering trains a preference without being rewritten. Eight pairs in the toy checkpoint's own vocabulary are checked in as `docs/examples/pairs.json`, so a first run needs no download.

ster tune dpo --model ster-toy-model --pairs docs/examples/pairs.json \
  --output calm-dpo.safetensors --epochs 3 --accumulation 4 --learning-rate 5e-3

On the four-layer, hidden-width-64 toy checkpoint that run reports 16 trainable tensors and 7168 trainable parameters over 6 optimizer steps, a first_loss of 0.6931473016738892, a final_loss of 0.6249638795852661, an accuracy of 1.0, and a mean_reward_margin of 0.12467505782842636, with mean_chosen_reward 0.0775139331817627 against mean_rejected_reward -0.047161124646663666.

The first loss is not approximately ln 2, it is exactly ln 2, and that is the cheapest available check that the run is wired correctly. A fresh adapter is the identity, so the policy and the reference are the same model, so every log-ratio is zero and -log sigmoid(0) is ln 2. The policy pass uses the composed kernels and the reference pass uses the fused ones; that they agree to F32 at step one is the two passes checking each other.

Inputs and options#

Two forward passes per pair — the policy and the frozen reference — with --batch-size pairs per forward, and --accumulation of those forwards folded into one optimizer step from their scaled losses. A pair is two rows, so a batch size of two is four sequences; a batch size of one is one row per forward, so the two sides of a pair go through as two separate passes, which is what keeps it byte-identical to every run recorded before batching existed. The KV cache is off, because the whole sequence goes through in one pass and one cache cannot hold rows that end in different places.

The reference is not a second checkpoint. It is the same mapped weights with the low-rank update skipped at every projection, which is exactly the model the adapters started as because B is zeros before the first step. It is also scored once for the whole run rather than once per epoch: a frozen model's log-probability of a fixed sequence is a constant, so recomputing it would be 2 × pairs × (epochs - 1) wasted forward passes.

A pair is scored whole rather than split into prompt and completion, because a pair set carries two complete texts and no prompt field. Nothing is lost when the two sides share a leading prefix — which is what `ster pairs synthesize` writes as Question: …\nAnswer: … — because that prefix sits at the same positions in both sequences, so its log-probability is the same expression on both sides of the margin and cancels out of the value and the gradient alike.

Training loads no gateway, spends no quota, and writes nothing but the adapter pair.

Output and state#

Writes the two files every objective writes: the adapter safetensors with the factors named layers.{layer}.{target}.a and layers.{layer}.{target}.b, and a JSON sidecar beside it carrying schema_version, product, kind (adapter here), model, model_revision, rank, alpha, targets, layers, hidden_size, and the run's report. The checkpoint itself is untouched.

Stdout is a pretty JSON document with path and report. The report carries loss (the string dpo or ipo), beta, pairs, trained_pairs, skipped_long, epochs, steps, trainable_tensors, trainable_parameters, first_loss, final_loss, mean_final_epoch_loss, accuracy, mean_reward_margin, mean_chosen_reward, mean_rejected_reward, rank, alpha, targets, layers, learning_rate, accumulation, batch, chat_template, and precision. The same report is recorded inside the sidecar, so a trained adapter always carries the run that produced it.

accuracy is the share of pairs whose chosen side earns the larger implicit reward, and the implicit reward is beta × (log pi - log pi_ref). Both it and the margins are measured over the final epoch, so they describe the adapter that was written rather than an average over a policy that was still moving.

Stderr gets one line naming the trainable tensor count, the parameter count, the rank and the layer count; one line as the reference pass starts and one per pair while it runs; one line per optimizer step; and one summary line per epoch.

training 16 adapter tensors (7168 parameters) at rank 8 across 4 layers
scoring 8 pairs under the frozen reference model
reference pair 8/8
epoch 1/3 step 1/6 pair 4/8 loss 0.6931 accuracy 0.000
epoch 1/3 mean loss 0.6867 accuracy 0.500 reward margin 0.0129
epoch 3/3 mean loss 0.6330 accuracy 1.000 reward margin 0.1247

Refusals#

• Every training number is the caller's: leaving out --rank, --alpha, --beta, --epochs, --learning-rate, --accumulation, --max-sequence, --batch-size or --seed is a usage error that names the flag, exit 2, before anything loads.
• --loss is parsed before anything loads: unknown preference loss "kto"; expected dpo or ipo.
• direct preference optimization requires a finite beta above zero.
• The pair set is validated as everywhere else: pair set {label} contains no pairs and pair set {label} contains an empty positive or negative prompt, where {label} is the file path when the set came from disk.
• The schedule is checked before the first gradient: direct preference optimization requires at least one epoch, direct preference optimization requires an accumulation of at least one pair, and direct preference optimization requires a finite learning rate above zero.
• max_sequence must be at least two tokens, so that one token can predict another.
• A side that encodes to a single token: pair 3 chosen side encodes to one token, so there is nothing to predict.
• Skipping a long pair is a progress line, not a refusal; skipping every pair is every pair is longer than the sequence limit, so there is nothing to train on.
• Model loading refuses through the shared path, including model architecture "plamo2" is unsupported by this Ster build; use a Hugging Face checkpoint whose model_type is one of <FAMILIES> (model resolution).
• The artifact is validated before it reaches disk, so a file Ster would refuse to read is never written.