Reproducibility

Overview#

A Ster run reproduces itself: the same command, at the same --seed, against the same checkpoint, writes the same numbers and the same adapter bytes, and --batch-size 1 reproduces every run recorded before batching existed byte for byte. --seed fixes a whole run — it seeds both the per-traversal shuffle and the draw that fills every A factor — and the draw is taken from Ster's own generator rather than Candle's initialiser, because the CPU device refuses to be seeded at all and an adapter nobody can reproduce is not an artifact. Ster assumes no seed or any other training number: every one is a required flag, so the command itself is the whole record of what the run was.

Changing --batch-size changes the run, for two independent reasons: a larger batch puts different examples in the same optimizer step, and the floating-point sums inside a batched forward associate in a different order. Neither is a defect and neither is a loss of reproducibility — every batch size reproduces itself. --precision changes the run for the obvious reason and reproduces itself the same way, and so does --chat-template: it changes the token sequence every read and every step sees. Both are recorded in what the run writes — a steering artifact carries them at its top level, an adapter sidecar in its train object — so reproducing a run does not depend on remembering which settings produced it.

What a batch of one means#

--batch-size counts rows per forward: examples for `sft` and `evaluate`, pairs for `dpo` and `reward`, where a pair is two rows. A batch of one is one row per forward whatever the unit, so at --batch-size 1 a preference pair's two sides go through the model as two separate one-row passes rather than one two-row pass. That is what makes the byte identity below hold, and it was worth getting right: folding the two sides into one pass at a batch of one changed the adapter bytes and broke the promise a batch of one exists for.

--accumulation keeps counting forwards, so one optimizer step sees up to batch-size * accumulation rows. A batched forward right-pads its rows and masks every padded key out of every real query, which is what makes stacking sequences of different lengths safe.

What batching costs and buys#

Measured on a four-layer, hidden-width-64 toy checkpoint with grouped-query attention (4 query heads, 2 key/value heads), RAYON_NUM_THREADS=1, minimum of three repetitions, and per-epoch cost taken as the nine-epoch run minus the one-epoch run divided by eight so that process start and model load cancel. CPU work on a toy checkpoint, not wall time on a loaded machine:

ster tune sft --model toy-model --examples examples64.json \
  --chat-template off --epochs 9 --accumulation 4 --learning-rate 0.001 \
  --seed 42 --batch-size 4 --output out.safetensors

Batch 8 does not beat batch 4, and that is real rather than noise: on a 64-wide model the padded width grows faster than the matmul saving. The figures are per-epoch user CPU seconds.

The numerical difference batching introduces is the reassociation, and it is small enough to state exactly. Scoring 64 examples with ster tune evaluate --chat-template off and no adapter, 63 of the 64 per-example losses are bitwise equal between --batch-size 1 and --batch-size 8; one differs, at index 63, 4.175486373901367 against 4.175485992431641 — the eighth significant digit. The corpus loss is 4.210041246069483 against 4.210041243771473, the tenth digit.

The identity check#

The unbatched path is verified rather than assumed, against the last release before batching, the rotary change and precision. Every objective run through the old binary and through the current one at --precision f32 --batch-size 1, on the toy checkpoint at a fixed seed:

• sft, dpo, reward and grpo write byte-identical adapter safetensors.
• tune evaluate returns all fourteen shared report fields equal, at a loss of 4.143039985939309 on both, with only the new precision provenance field added.
• generate is byte-identical over 24 tokens at temperature zero and seed 42.
• extract matches on 512 of 512 hidden floats.

That holds because every cast in the F32 path short-circuits to a handle clone rather than converting and rounding back.

These comparisons predate the learning-rate schedule decaying to zero. Until then the cosine decayed to a tenth of --learning-rate; a training run today at the same settings writes different adapter bytes from one recorded before the change whenever it had steps after its warmup, and reproduces itself exactly as before.