ster calibrate

Invocation and purpose#

ster calibrate --model <MODEL> --examples <EXAMPLES> --output <OUTPUT> --ece-bins <BINS> [--revision <REVISION>] [--device <DEVICE>] [--chat-template <CHAT_TEMPLATE>] [--precision <PRECISION>] [--permutations <PERMUTATIONS>]

A raw language model is usually too sure. Temperature scaling is the smallest correction: one number the averaged log-scores of every question are divided by before the softmax, which sharpens or flattens every distribution at once and cannot change which option wins. ster calibrate fits it by minimizing the mean negative log-likelihood of the labels over the temperature's logarithm, with no window and no step count assumed: it starts at the raw temperature, doubles its stride outward until the likelihood stops improving, which brackets the minimum wherever it lies, and narrows the bracket by golden-section search until floating point cannot split it further. It writes the artifact to --output and prints it. `ster decide --calibration` applies it.

Inputs and options#

The labelled set. A choice is labelled with its option name, a score with its level index, a noul with true or false; not every question needs a label.

{
  "examples": [
    {
      "state": "I was charged twice for order 4471 and the second invoice is wrong. Fix this today or I am disputing it with my bank.",
      "questions": {
        "department": { "type": "choice", "instructions": "Which team should handle this?", "criteria": { "returns": "Exchanges, refunds, wrong or damaged items", "shipping": "Delivery status, delays, lost packages", "billing": "Charges, invoices, payment problems" } },
        "frustration": { "type": "score", "instructions": "How frustrated is the customer?", "criteria": ["Calm, just stating facts", "Frustrated but civil", "Very angry, strong language or threatening to leave"] },
        "is_urgent": { "type": "noul", "instructions": "Does this message convey urgency?" }
      },
      "answers": { "department": "billing", "frustration": 2, "is_urgent": true }
    }
  ]
}

Output and state#

The artifact written from the repository's tests/decide/fixtures/examples.json — four support messages, eleven labels — on HuggingFaceTB/SmolLM2-1.7B-Instruct, CPU, f32, in 48 seconds:

{
  "schema": "ster-calibration/1",
  "model": "HuggingFaceTB/SmolLM2-1.7B-Instruct",
  "revision": "31b70e2e869a7173562077fd711b654946d38674",
  "chat_template": "applied",
  "precision": "f32",
  "temperature": 0.8367,
  "examples": 4,
  "questions": 11,
  "before":  { "temperature": 1.0,    "nll": 0.6469, "ece": 0.3095, "accuracy": 0.5455 },
  "after":   { "temperature": 0.8367, "nll": 0.6415, "ece": 0.2943, "accuracy": 0.5455 },
  "control": { "temperature": 0.8367, "nll": 1.5052, "ece": 0.4362, "accuracy": 0.2727 }
}

• before and after are the same labels scored at temperature 1.0 and at the fitted one: mean negative log-likelihood of the correct option, expected calibration error over the stated confidence bins, and how often the winning option was the labelled one. Accuracy is identical in both, because scaling never changes the winner; the fit can only move nll and ece. This run predates --ece-bins: it measured over ten bins, and its blocks carry no ece_bins, which a calibration written today records.
• control is the same labels with every question judged against another example's state — each example's questions paired with the next example's state. A model that reads the state scores its labels better on the real pairing than on the shuffled one; a model answering from the options and the letters alone scores the same on both, and no temperature can make that honest. Here 0.55 against 0.27. The control needs two examples to exist and is absent below that.
• A temperature below one sharpens the distributions; above one flattens them. A fit that keeps improving until the temperature leaves the finite range — every label already won with certainty, or none ever can be — stops at the last finite temperature, which is worth reading as a warning that the labels and the model disagree badly.
• Eleven labels demonstrates the mechanism and is far too few to trust the number; fit on the decisions your workflow actually makes.
• The last progress line on stderr is fitted temperature 0.8367 over 11 labelled questions: nll 0.6469 -> 0.6415, ece 0.3095 -> 0.2943, accuracy 0.5455 against 0.2727 on shuffled states; before it, reading example <i> of <n> and one state-reading line per pass.

Refusals#

All exit 1 with Error: and the sentence, before the checkpoint loads; no artifact is written.

• Leaving out --ece-bins is a usage error naming the flag, exit 2; --ece-bins 0 is refused as the expected calibration error needs at least one bin.
• calibration needs at least one labelled example
• calibration needs at least one labelled question
• calibration example <i> is not a valid request, over the request refusal
• example <i> labels question '<id>', which it does not ask
• example <i> answers question '<id>' with <label>, which is not one of its options
• invalid calibration examples <path>, failed to read <path>
• Handed to ster decide --calibration: invalid calibration <path>; calibration <path> has schema "<schema>"; this Ster reads ster-calibration/1; calibration <path> has a temperature that is not a positive number; calibration <path> was fitted for model '<fitted>', not '<loaded>'.