Precision

Overview#

Until 0.13 Ster mapped every checkpoint at F32 and offered nothing else, so a model published in bf16 cost twice its own file size to hold, and on Metal the half-precision matmul path the hardware has went unused. --precision f32|f16|bf16 names the dtype the frozen base weights are mapped at. It is on every command that maps a checkpoint — train, optimize, evaluate, generate, extract, pairs synthesize, and the five ster tune subcommands that load a model — and mirrored as precision on each matching /v1 endpoint. tune merge and tune inspect take none, because neither builds a decoder. It defaults to f32, so every run recorded before it existed is unchanged.

It names the base weights and nothing a run trains. Adapters, a reward run's scalar head, and both AdamW moments stay in F32 at every setting. That split is the whole of mixed precision rather than an implementation detail of it: a LoRA adapter is a small correction to a large weight, and an update below that weight's own ulp rounds to nothing in half — the adapter would train, the optimizer would report a falling loss, and the model would not move.

Why the split is nearly free#

Three properties of Candle 0.11 do the work, which is worth stating because it is why so little code implements this:

• AdamW::new builds each variable's two moments at that variable's own dtype, so F32 adapter variables give F32 optimizer state with nothing arranged.
• lora::Adapter::forward already casts each factor to the activation's dtype before its two matmuls, so an F32 factor meets a half-precision activation without a shape or dtype error.
• to_dtype is differentiable, and its backward is grad.to_dtype(arg.dtype()), so the gradient arrives back at an F32 variable in F32.

Nothing between the two dtypes had to be built. It only had to not be undone.

What stays F32 whatever the flag says#

Three places in the forward pass are held at single precision, because each is somewhere half precision is wrong rather than merely cheaper:

• Attention. Queries, keys and values are promoted before the score matmul; the mask fill and the softmax run in F32 and only the output is cast back.
• Rotary position embedding. The angles and the rotation are F32. Each forward computes the angles of exactly the positions it rotates from the rotation's frequencies, so nothing the cache holds grows with max_position_embeddings. A position is an absolute index rather than a weight, and a half mantissa there makes neighbouring late positions round to the same angle — a phase error, which reads as a slightly different sentence and never as a numerical fault. Measured: scoring two held-out examples of about 1860 tokens, against the same model's own F32 loss of 1.795495907465617, rotating in F32 costs 0.07% and rotating in half costs 1.1%, so the promotion removes roughly fifteen sixteenths of what half precision would otherwise take. At 128 tokens the two are indistinguishable, which is the honest limit of a short fixture rather than evidence the promotion is unnecessary.
• Every loss. A log-softmax adds tens of thousands of terms into one accumulator, and in half precision the smallest of them stop changing it.

The key-value cache and the residual stream still store half-width values, so the memory saving survives all three.

bf16 is refused on the CPU#

bf16 has no CPU matmul kernel in this Candle build; use --precision f16 for half precision on cpu, or --device metal for bf16

A fact about Candle rather than a policy. cpu_backend's matmul gates on F16 | F32 | F64 and returns unsupported dtype BF16 for op matmul for anything else, so a bf16 CPU run does not run slowly — it downloads and maps the whole checkpoint and dies at the first projection. The refusal fires at load, before a single weight is mapped. f16 on the CPU is real half arithmetic on an Apple-silicon class machine: gemm selects a native neonfp16 microkernel on aarch64 when the hardware reports the fp16 feature. On metal, all three dtypes have matmul kernels through MLX gemm. There is no --precision on ster generate, tune merge or tune inspect.

A steering vector survives the cast#

A steering artifact stores unit-normalized F32 vectors, and SteeringPlan::new casts each one to the model's dtype as it builds the plan, so a direction trained at one precision applies at another. What the cast costs is small and worth stating exactly: at f16 the mean relative error per component is 1.8e-4, the largest absolute error 2.9e-5, and the cosine similarity with the original is 1.0 to within F32's own rounding. Two to four components in 2048 fall below half precision's smallest normal and become subnormal; none flush to zero, and they are the near-zero components that carry none of the direction.

It still steers. Generating from TinyLlama-1.1B-Chat with the same artifact, strength, seed and greedy sampling, f32 and f16 produce character-identical text at strengths 1 and 2, over 32 and 96 tokens, on two prompts, steered and unsteered. At strength 4 both collapse to an immediate end-of-sequence, which is over-steering rather than a precision effect. An intervention is a scaled add into the residual stream, and both operands are in the model's dtype, so nothing in the apply path mixes widths.

What it does not survive is being fitted in one space and judged in another without anyone noticing, which is why a steering artifact records the precision it was fitted at. evaluate --precision had always said it should match the run that trained the artifact, and until the artifact carried the value nothing could check the claim. ster vector evaluate and ster generate now compare it and warn — warning: this direction was trained at precision f32 and this run maps the base weights at f16, so it is being read in a different space than it was fitted in — a warning rather than a refusal, because the cast above is small enough that reading across precisions is a reasonable thing to do on purpose. chat_template is recorded beside it and checked the same way, and there the difference is not small (chat template).

Measured#

TinyLlama/TinyLlama-1.1B-Chat-v1.0, revision fe8a4ea1ffedaf415f4da2f062534de366a451e6, CPU, scoring the eight checked-in examples with ster tune evaluate --max-sequence 128. Two runs of each, peak resident stable to 0.02% and user CPU to 4%, and independently reproduced on a second machine over three reps:

The 2.32 GiB saved is the checkpoint's own weight count at two bytes instead of four. The loss is 0.33% apart, which is what half precision cost in accuracy here, and each precision reproduces itself exactly. Wall clock is deliberately not quoted: the machine was shared, and identical f16 runs came in at 3.9 s and 10.8 s of wall time while their CPU time moved by 1%.

Training is where the flag decides whether a run happens at all. ster tune sft --model TinyLlama/TinyLlama-1.1B-Chat-v1.0 --examples examples.json --epochs 1 --max-sequence 128 --seed 42 trains 88 adapter tensors and 1126400 parameters across 22 layers, and asks for a peak memory footprint of 102.8 GB at f32 against 55.6 GB at f16 — a factor of 1.85. On a quiet machine both finish, at a final loss of 4.198415756225586 and 4.200512409210205 respectively. On a loaded one the f32 run was killed by the operating system partway through the first epoch, which is a signal rather than a Ster refusal or a Candle allocation failure, while f16 completed. The f32 run that did finish spent 105.6 s of system time against 76.9 s of user time with 2.4 million involuntary context switches, which is the shape of a machine compressing memory to stay alive.

Peak memory footprint is quoted for the training runs rather than peak resident, because resident moved between 23.9 and 34.6 GiB across machines while the footprint held: resident reflects what the operating system let the process keep, and the footprint reflects what it asked for. Most of that footprint is the autograd tape over 22 differentiable layers rather than the weights, and the tape is not what --precision controls — the way to shrink a tape is fewer differentiable layers, not a narrower dtype. What halving the base weights buys is the margin that decides whether a loaded machine finishes the run.

The default is byte-identical#

--precision f32 --batch-size 1 reproduces the pre-precision, pre-batching binary exactly, which is the check that proves the default path was not disturbed by either. Against the last release before both, on the toy checkpoint: sft, dpo, reward and grpo write byte-identical adapter safetensors; tune evaluate returns all fourteen shared report fields equal, at a loss of 4.143039985939309 on both, with only the new precision provenance field added; generate is byte-identical over 24 tokens at temperature zero; and extract matches on 512 of 512 hidden floats.

Every cast in the F32 path short-circuits to a handle clone, which is why that identity holds rather than merely rounding to the same numbers (reproducibility).