Ster/Docs

ster tune bipo

Learn one steering vector by bi-directional preference optimization: +v prefers each pair's positive side, -v its negative side.

Invocation and purpose#

text
ster tune bipo --model <MODEL> [--pairs <PAIRS>] --output <OUTPUT> --layer <LAYER> --strength <S>
               --beta <BETA> [--loss dpo|ipo] --epochs <N> --learning-rate <LR> --accumulation <N>
               --warmup-steps <N> --max-sequence <TOKENS> --batch-size <N> --seed <SEED>
               [--chat-template auto|off] [--precision f32|f16|bf16]
               [--checkpoint <DIR> --checkpoint-every <STEPS>] [--resume <DIR>]

Learns one steering vector instead of an adapter, by the bi-directional preference optimization of Cao et al. (arXiv 2406.00045). The policy is the frozen model with d · strength · v added to the residual stream at --layer; the reference is the same model with nothing added, scored once up front exactly as `ster tune dpo` scores it. Each forward draws d from {+, −}: with +v the loss asks the model to prefer each pair's positive side, with −v its negative side, which is the same --loss with --beta negated. One vector therefore learns to move the behaviour both ways, and `ster generate` --vector <artifact> --strength s turns it up, down or past zero afterwards. v starts at zero and is the only variable: no adapter is registered and no base weight moves.

bash
ster model fixture create --from docs/examples/model/fixture.json --output .build/bipo-fixture
ster tune bipo --model .build/bipo-fixture --pairs docs/examples/pairs.json --output calm.bipo.json \
  --layer 2 --strength 1 --beta 0.1 --epochs 2 --learning-rate 0.01 --accumulation 1 \
  --warmup-steps 0 --max-sequence 256 --batch-size 4 --seed 7

Inputs and options#

InputRequiredMeaning
--pairs <PAIRS>no*Pair set: the positive side is what +v makes the model prefer, the negative side what −v makes it prefer. Omit it to use the active imported set.
--output <OUTPUT>yesSteering artifact JSON written.
--layer, --strength, --betayesThe layer whose residual stream the vector is added to, the multiple added while training in each direction, and how hard the frozen reference pulls the policy back.
--loss dpo|iponodpo (default) for the sigmoid loss, ipo for the squared error against 1/(2·beta) over length-normalized log-probabilities.
--epochs, --learning-rate, --accumulation, --warmup-steps, --max-sequence, --batch-size, --seedyesPasses over the set, the AdamW learning rate, forwards per optimizer step, the warm-up, the token limit above which a pair is skipped rather than cut, pairs per forward (every pair of one forward shares its direction), and the seed. Ster assumes none of them.
--chat-template auto|off, --precision f32|f16|bf16noauto encodes both sides as the assistant turn of the model's own chat template when it publishes one; the precision is the frozen weights' (the vector and optimizer moments stay f32; bf16 needs --device metal).
--checkpoint <DIR> --checkpoint-every <STEPS>, --resume <DIR>noAs for `ster tune sft`: the checkpoint holds the vector with its two AdamW moments, and a resumed run replays the shuffle and every direction drawn, ending with the vector an uninterrupted run writes. A checkpoint of another run is refused naming every field that differs.

Output and state#

The output is a steering artifact (method: "bipo", one layer vector whose train_margin and train_accuracy are the final epoch's mean reward margin and accuracy). The report records layer, strength, loss, beta, pairs, trained_pairs, skipped_long, epochs, steps, first_loss, final_loss, mean_final_epoch_loss, accuracy, mean_reward_margin, vector_norm, learning_rate, accumulation, batch, resumed_from, checkpoints, chat_template and precision. BiPO has no request API operation and no Ster Desktop screen; the CLI is its only surface.

Refusals#

  • A layer outside the model: layer <n> is outside the model's <layers> layers.
  • A beta, strength or learning rate that is not a finite number above zero: bi-directional preference optimization requires a finite <name> above zero, not <value>; epochs, accumulation, batch size or sequence limit of zero: bi-directional preference optimization requires <name> of at least one.
  • Pairs are encoded and skipped as in `ster tune dpo`, and a run that takes no step is refused with the run took no step, so it has no loss to report.