ster tune bipo
Learn one steering vector by bi-directional preference optimization: +v prefers each pair's positive side, -v its negative side.
Invocation and purpose#
ster tune bipo --model <MODEL> [--pairs <PAIRS>] --output <OUTPUT> --layer <LAYER> --strength <S>
--beta <BETA> [--loss dpo|ipo] --epochs <N> --learning-rate <LR> --accumulation <N>
--warmup-steps <N> --max-sequence <TOKENS> --batch-size <N> --seed <SEED>
[--chat-template auto|off] [--precision f32|f16|bf16]
[--checkpoint <DIR> --checkpoint-every <STEPS>] [--resume <DIR>]Learns one steering vector instead of an adapter, by the bi-directional preference optimization of Cao et al. (arXiv 2406.00045). The policy is the frozen model with d · strength · v added to the residual stream at --layer; the reference is the same model with nothing added, scored once up front exactly as `ster tune dpo` scores it. Each forward draws d from {+, −}: with +v the loss asks the model to prefer each pair's positive side, with −v its negative side, which is the same --loss with --beta negated. One vector therefore learns to move the behaviour both ways, and `ster generate` --vector <artifact> --strength s turns it up, down or past zero afterwards. v starts at zero and is the only variable: no adapter is registered and no base weight moves.
ster model fixture create --from docs/examples/model/fixture.json --output .build/bipo-fixture
ster tune bipo --model .build/bipo-fixture --pairs docs/examples/pairs.json --output calm.bipo.json \
--layer 2 --strength 1 --beta 0.1 --epochs 2 --learning-rate 0.01 --accumulation 1 \
--warmup-steps 0 --max-sequence 256 --batch-size 4 --seed 7Inputs and options#
| Input | Required | Meaning |
|---|---|---|
--pairs <PAIRS> | no* | Pair set: the positive side is what +v makes the model prefer, the negative side what −v makes it prefer. Omit it to use the active imported set. |
--output <OUTPUT> | yes | Steering artifact JSON written. |
--layer, --strength, --beta | yes | The layer whose residual stream the vector is added to, the multiple added while training in each direction, and how hard the frozen reference pulls the policy back. |
--loss dpo|ipo | no | dpo (default) for the sigmoid loss, ipo for the squared error against 1/(2·beta) over length-normalized log-probabilities. |
--epochs, --learning-rate, --accumulation, --warmup-steps, --max-sequence, --batch-size, --seed | yes | Passes over the set, the AdamW learning rate, forwards per optimizer step, the warm-up, the token limit above which a pair is skipped rather than cut, pairs per forward (every pair of one forward shares its direction), and the seed. Ster assumes none of them. |
--chat-template auto|off, --precision f32|f16|bf16 | no | auto encodes both sides as the assistant turn of the model's own chat template when it publishes one; the precision is the frozen weights' (the vector and optimizer moments stay f32; bf16 needs --device metal). |
--checkpoint <DIR> --checkpoint-every <STEPS>, --resume <DIR> | no | As for `ster tune sft`: the checkpoint holds the vector with its two AdamW moments, and a resumed run replays the shuffle and every direction drawn, ending with the vector an uninterrupted run writes. A checkpoint of another run is refused naming every field that differs. |
Output and state#
The output is a steering artifact (method: "bipo", one layer vector whose train_margin and train_accuracy are the final epoch's mean reward margin and accuracy). The report records layer, strength, loss, beta, pairs, trained_pairs, skipped_long, epochs, steps, first_loss, final_loss, mean_final_epoch_loss, accuracy, mean_reward_margin, vector_norm, learning_rate, accumulation, batch, resumed_from, checkpoints, chat_template and precision. BiPO has no request API operation and no Ster Desktop screen; the CLI is its only surface.
Refusals#
- A layer outside the model:
layer <n> is outside the model's <layers> layers. - A beta, strength or learning rate that is not a finite number above zero:
bi-directional preference optimization requires a finite <name> above zero, not <value>; epochs, accumulation, batch size or sequence limit of zero:bi-directional preference optimization requires <name> of at least one. - Pairs are encoded and skipped as in `ster tune dpo`, and a run that takes no step is refused with
the run took no step, so it has no loss to report.