ster decisions benchmark
Measure a model, with or without an adapter and a calibration, on held-out labelled decisions: accuracy and negative log-likelihood overall, per question type, against a shuffled-state control, with latency and token counts.
Invocation and purpose#
ster decisions benchmark --model <MODEL> --examples <EXAMPLES> --output <OUTPUT> --ece-bins <BINS> [--adapter <ADAPTER>] [--calibration <CALIBRATION>] [--revision <REVISION>] [--device <DEVICE>] [--chat-template <auto|off>] [--precision <f32|f16|bf16>] [--permutations <PERMUTATIONS>]Reads every example the way `ster decide` would and scores its labels. Run it on the base model and again with the adapter `ster tune decide` wrote, on the held-out side of `ster decisions split`, and the two documents say what training bought. A calibration from `ster calibrate` supplies the temperature; without one the raw temperature is used.
Inputs and options#
| Input | Required | Meaning |
|---|---|---|
--model <MODEL> | yes | Hugging Face model id or local model directory. |
--examples <EXAMPLES> | yes | Held-out labelled decisions. |
--adapter <ADAPTER> | no | A LoRA adapter trained for this exact model, such as one from ster tune decide. |
--calibration <CALIBRATION> | no | A calibration written by ster calibrate for this exact model; its temperature is used. |
--permutations <PERMUTATIONS> | no | Option orders each question is shown in; 0, the default, is every cyclic shift. |
--revision, --device, --chat-template, --precision | no | As on `ster decide`. |
--ece-bins <BINS> | yes | Equal-width confidence bins the expected calibration error in metrics and control is measured over, recorded there as ece_bins. Ster assumes none, because an ECE is only comparable with one taken over as many bins. |
--output <OUTPUT> | yes | Where the benchmark document is written; it is also printed. |
Output and state#
The document names model, revision, adapter, chat_template, precision, calibration, permutations, examples and questions, then metrics over every labelled question at the temperature the run used, control — the same labels with each question judged against another example's state, absent below two examples — by_type with questions, accuracy and nll per question type, latency (total_seconds, mean_ms_per_example, mean_ms_per_question, model already loaded) and input_tokens across every example. Progress on stderr reads benchmarking example <i> of <n>.
Refusals#
- Leaving out
--ece-binsis a usage error naming the flag, exit 2;--ece-bins 0is refused asthe expected calibration error needs at least one bin. - The labelled-decision validation refusals of `ster calibrate`, before the checkpoint loads.
calibration <path> was fitted for model '<fitted>', not '<loaded>'and the other calibration refusals ofster decide --calibration.- The model-loading refusals shared by every model-loading command, and an adapter whose identity sidecar names another model.