ster tune evaluate

Invocation and purpose#

ster tune evaluate --model <MODEL> --examples <EXAMPLES>
                   --max-sequence <MAX_SEQUENCE> --batch-size <BATCH_SIZE>
                   [--revision <REVISION>] [--device <DEVICE>]
                   [--adapter <ADAPTER>] [--chat-template <auto|off>]
                   [--precision <f32|f16|bf16>]

Every other command under ster tune changes weights. This one only measures them, and the absence of an optimizer is the point: the number is meaningful precisely because nothing about the run could have moved to produce it. It answers the question a training report structurally cannot — a training loss is measured on the data that produced the gradient, so it falls whether or not the model learned anything transferable.

--examples reads the same {"examples": [{"prompt": "…", "completion": "…"}]} that `ster tune sft` trains on, with an optional name. For a held-out measurement it should not be the set that was trained on.

ster tune evaluate --model ster-toy-model --examples docs/examples/examples.json \
  --max-sequence 512 --batch-size 1
ster tune evaluate --model ster-toy-model --examples docs/examples/examples.json \
  --max-sequence 512 --batch-size 1 --adapter calm.safetensors

On the toy checkpoint the bare run reports 8 examples, 81 completion tokens, a loss of 4.143039985939309 and a perplexity of 62.994031634237196, with mean_example_loss 4.142926827344027. After 40 epochs of supervised fine-tuning on that same set at a learning rate of 3e-3, the same evaluation with --adapter reports a loss of 3.788468490412206 and a perplexity of 44.18867305032514 — and that loss matches the trainer's own final loss of 3.7894 to three decimals, which is the differentiable cross-entropy path and the fused log-probability path agreeing.

Inputs and options#

--adapter attaches the adapter while the weights are mapped, exactly as `ster generate --adapter` does, so the score is the score of the model an operator would actually run rather than of an approximation of it.

The scored window is the completion only, the same window ster tune sft takes its gradient over: for a joined sequence of n tokens whose completion begins at boundary, the logits [boundary - 1, n - 1) against the targets ids[boundary..]. The pass takes the fused kernels, because no gradient is wanted and paying for the composed forms would buy an autograd tape that is discarded.

Nothing is written and nothing is trained. There is no optimizer, no VarMap, and no artifact.

Output and state#

Two aggregates are reported, because they answer different questions and reporting one would silently pick a side. loss is total negative log-likelihood over total completion tokens, so long examples count for more, and perplexity, its exponential, is the number comparable with corpus perplexity anywhere else. mean_example_loss is the mean of the per-example losses, weighing each example equally whatever its length, which is usually what an operator comparing two adapters on a curated set means.

Stdout is the report itself, pretty-printed — not wrapped in path and report, because nothing was written. It carries model, model_revision, adapter (null when none was attached, recorded either way so two reports are comparable), name, examples, evaluated, skipped_long, completion_tokens, loss, perplexity, mean_example_loss, mean_example_perplexity, chat_template, precision, and entries.

entries is one record per evaluated example carrying index, prompt, completion, completion_tokens, loss and perplexity, so finding the worst example is a sort rather than a second run.

Every number in this report is a double rather than a single. Perplexity is the exponential of a loss and overflows single precision at a loss of about 89, which a broken adapter can reach, and a measurement that reports infinity as a JSON null is worse than useless.

example 1/8 loss 4.1908 perplexity 66.073
8 examples, 81 completion tokens, loss 4.1430, perplexity 62.994

Refusals#

• Leaving out --max-sequence or --batch-size is a usage error that names the flag, exit 2, before anything loads.
• max_sequence must be at least two tokens, so that one token can predict another.
• A batch of zero: evaluation requires a batch of at least one example. Scoring runs no preflight, so this check sits in evaluate itself and ster request tune/evaluate makes it too.
• The example set is validated as ster tune sft validates it: example set {label} contains no examples and example set {label} contains an empty prompt or completion.
• Skipping every example is every example is longer than the sequence limit, so there is nothing to evaluate — note evaluate, where the trainers say train on.
• A reward artifact passed to --adapter: adapter artifact is a reward model, not a generation adapter.
• An adapter for another checkpoint: adapter was trained for model "…", current model is "…".