ster decide

Invocation and purpose#

ster decide --model <MODEL> --request <REQUEST> [--revision <REVISION>] [--device <DEVICE>] [--chat-template <CHAT_TEMPLATE>] [--precision <PRECISION>] [--permutations <PERMUTATIONS>] [--calibration <CALIBRATION>] [--output <OUTPUT>] [--explain]

Reads a request document — one state and a map of typed questions — and answers every question from the model's next-token distribution over answer letters, without generating text. The state goes through the model once with the key-value cache on; each question rendering runs only its own suffix; every option is shown under every letter so the model's letter preference cancels. The mechanism is on the decision page. Prints the response document on stdout.

Inputs and options#

The request document#

{
  "state": "My running shoes arrived in the wrong size. Can I swap them for a size 10? I paid already and the invoice was fine.",
  "questions": {
    "department": {
      "type": "choice",
      "instructions": "Which team should handle this?",
      "criteria": {
        "returns": "Exchanges, refunds, wrong or damaged items",
        "shipping": "Delivery status, delays, lost packages",
        "billing": "Charges, invoices, payment problems"
      }
    },
    "frustration": {
      "type": "score",
      "instructions": "How frustrated is the customer?",
      "criteria": ["Calm, just stating facts", "Frustrated but civil", "Very angry, strong language or threatening to leave"]
    },
    "is_urgent": {
      "type": "noul",
      "instructions": "Does this message convey urgency?"
    }
  }
}

• state is a string, or any JSON value, which is rendered as JSON for the model to read.
• Question ids are yours; the model never sees them and the answer comes back under the same id. A model field is accepted and ignored, because the model is the checkpoint on the command line.
• A choice picks one option from a named set: criteria maps each option name to a description, or null when the name says enough. Options are shown in name order.
• A score places the state on an ordered scale: criteria is the levels in order, and level i is the ith entry.
• A noul is a yes/no judgement; criteria may carry "true" and "false" descriptions of what each side means.
• instructions and every description may be a string, an object or an array; anything that is not a string is rendered as JSON. Every option is answered with one letter, so a question carries at most 26 options or levels.

Output and state#

The response the request above produced on HuggingFaceTB/SmolLM2-1.7B-Instruct at revision 31b70e2e869a7173562077fd711b654946d38674, CPU, f32, in 8.1 seconds including the checkpoint load (macOS arm64, cargo build --release):

{
  "model": "HuggingFaceTB/SmolLM2-1.7B-Instruct",
  "revision": "31b70e2e869a7173562077fd711b654946d38674",
  "chat_template": "applied",
  "precision": "f32",
  "temperature": 1.0,
  "calibration": null,
  "permutations": 0,
  "answers": {
    "department": {
      "type": "choice",
      "choice": "returns",
      "probabilities": { "billing": 0.0998, "returns": 0.8411, "shipping": 0.0590 },
      "confidence": 0.5061
    },
    "frustration": {
      "type": "score",
      "score": 0.9432,
      "legend": { "0": "Calm, just stating facts", "1": "Frustrated but civil", "2": "Very angry, strong language or threatening to leave" },
      "probabilities": { "0": 0.2082, "1": 0.6404, "2": 0.1514 },
      "confidence": 0.1827
    },
    "is_urgent": { "type": "noul", "noul": 0.6120 }
  },
  "usage": { "input_tokens": 464, "forward_passes": 9, "output_tokens": 0 }
}

• choice is the option with the most probability; probabilities sums to one over the options.
• score is the probability-weighted level, Σ level × p(level), so it can land between levels; legend maps each level number back to its text.
• noul is the probability of yes. It carries no confidence: with two options the number already says how sure the model is.
• confidence on a choice or a score is 1 − entropy / ln(n): 1 when all the mass is on one option, 0 when it is spread evenly. It describes the distribution, not whether the answer is right.
• temperature is what the averaged log-scores were divided by — 1.0 raw, or the fitted value when --calibration was given, whose path is recorded in calibration.
• usage.input_tokens counts the tokens the model actually processed: the shared state once, then every suffix. forward_passes is the prefix pass plus one per rendering. output_tokens is always zero.
• Progress goes to stderr: the chat-template sentence, then reading the state once (84 tokens), then 8 question renderings (380 tokens between them).
• --output writes the same document to a file as well; stdout still carries it.

Reading an answer with --explain

--explain adds an explain block per question with the option texts in canonical order and, for each order the question was shown in, the letter every option wore and the probability the model put on it in that order alone. Read it when an answer looks flat: it says whether the content or the letter decided it. The same run with --explain, for the choice:

"department": {
  "options": ["billing: Charges, invoices, payment problems", "returns: Exchanges, refunds, wrong or damaged items", "shipping: Delivery status, delays, lost packages"],
  "orders": [
    { "letters": ["A", "B", "C"], "probabilities": [0.1112, 0.8733, 0.0155] },
    { "letters": ["C", "A", "B"], "probabilities": [0.0890, 0.7223, 0.1887] },
    { "letters": ["B", "C", "A"], "probabilities": [0.0903, 0.8467, 0.0631] }
  ]
}

returns wins in every order whatever letter it wears, which is what a reading of the content looks like. On TinyLlama/TinyLlama-1.1B-Chat-v1.0 the same request gives returns 0.78 in the order where it wears B and 0.11 and 0.08 in the other two; the averaged answer is 0.34 / 0.32 / 0.34 with confidence 0.0002 — the honest result for a model that answers B whatever the message says.

Refusals#

All exit 1 with Error: and the sentence, before the checkpoint loads. The runbook says what to do about each.

• a decide request has an empty state
• a decide request needs at least one question
• a decide request has a question with an empty id
• question '<id>' has empty instructions
• choice question '<id>' needs at least two options
• choice question '<id>' has <n> options; Ster labels at most 26
• choice question '<id>' has an option with an empty name "<name>"
• score question '<id>' needs at least two levels
• score question '<id>' has <n> levels; Ster labels at most 26
• score question '<id>' has an empty description at level <i>
• failed to read <path>, invalid decide request <path> or invalid decide request on standard input, with the reader's own cause beneath
• calibration <path> was fitted for model '<fitted>', not '<loaded>', and the other artifact refusals on `ster calibrate`
• After the load: question '<id>' renders to <n> tokens, past the <max> this model was built for, and this tokenizer has no single token for the answer label "<label>", so it cannot be read from one position.

Tests#

tests/decide/ in the repository drives the built binary against the real HuggingFaceTB/SmolLM2-1.7B-Instruct checkpoint: the request above is answered and its response read back from disk, every order is explained, a calibration is fitted and then read by a decision and refused by another model, every refusal above that needs no model is checked for its exact sentence, and ster request decide answers the body Ster Desktop writes to its stdin and refuses a malformed one with a single status-2 result. cargo test --release --test decide: 6 passed in 69 seconds on an M2 Max at ster 7661b9a1, when the desktop's route was still ster serve; the ster request story replaces that route's test in ster 4c1d5658.