Runbook
Ster fails closed: a CLI refusal exits non-zero and anyhow prints Error: plus its context chain. Under ster request, a refusal before the operation starts is one status-2 result; a failure after it starts is a final error log event and one status-1 result.
Overview#
Ster fails closed: a CLI refusal exits non-zero and anyhow prints Error: plus its context chain. Under ster request, a refusal before the operation starts is one status: 2 result and exit status 2; a failure after it starts becomes a final error: log event, one status: 1 result, and exit status 1. Match the stable source-authored sentence below; substituted values are shown as <PLACEHOLDERS>.
Model and checkpoint loading#
| Sentence | Meaning | Fix |
|---|---|---|
unknown device "<VALUE>"; expected cpu, metal, or cuda | --device/device is not one of the three lowercase names. | Use a supported name. |
this Ster binary was built without the metal feature | Metal was selected on a CPU-only build. | Rebuild/install with --features metal, then retry. |
this Ster binary was built without the cuda feature | CUDA was selected without CUDA support. | Rebuild/install with --features cuda on a supported host. |
failed to initialize Metal device | Feature exists but Candle could not open Metal device 0. | Check OS/device support and driver/runtime state; CPU is the explicit fallback. |
failed to initialize CUDA device | Feature exists but Candle could not open CUDA device 0. | Check CUDA device/driver/runtime compatibility; CPU is the explicit fallback. |
failed to list <DIR> | A local model directory could not be enumerated. | Check existence and read/execute permissions. |
model config is missing: <PATH> | Local config.json is absent or not a regular file. | Restore it at the model root. |
tokenizer is missing: <PATH> | Local tokenizer.json is absent or not a regular file. | Restore it at the model root. |
model directory contains no safetensors weights | No eligible top-level .safetensors file was found. | Add model weights; optimizer-named files do not count. |
failed to initialize Hugging Face Hub client | hf-hub could not build its client/cache/token configuration. | Check home/cache permissions and TLS/client setup, or use a local directory. |
failed to read model repository <MODEL> | Hub repository metadata lookup failed. | Check model id, revision, network, authentication, and gated access. |
model <MODEL> publishes no safetensors weights | Repository siblings contain no eligible weights. | Choose a Safetensors model or materialize a supported local directory. |
failed to download <WEIGHT_NAME> | A discovered shard could not be downloaded. | Check access/network/cache space; retry the same immutable revision. |
failed to read <CONFIG_PATH> | Resolved config could not be read. | Check file/cache integrity and permissions. |
invalid model config <CONFIG_PATH> | config.json is not JSON. | Replace it with the checkpoint's valid config. |
model architecture "<TYPE>" is unsupported by this Ster build; use a Hugging Face checkpoint whose model_type is one of <FAMILIES> | model_type is missing/non-string (shown as "") or not a family Ster's decoder implements; <FAMILIES> is the list on model resolution. | Use a checkpoint of one of those families. |
invalid <TYPE> config <CONFIG_PATH> | JSON passed the architecture gate but lacks a dimension the decoder needs (hidden, intermediate and vocabulary sizes, layer and head counts, rms_norm_eps), or carries a rope_scaling that is not Llama 3's. | Use the checkpoint's complete config; do not hand-delete required dimensions. |
<CONFIG_PATH> rotates <N> of <HEAD> components per head (partial_rotary_factor, rotary_pct or rotary_dim); the rotated width must be even, above zero and at most the head | partial_rotary_factor or rotary_pct times the head width truncates to an odd number or zero, or rotary_dim is odd, zero or wider than the head. | Use the checkpoint's own config; a rotated width like that is not one any published family uses. |
<CONFIG_PATH> declares longrope <KEY> that is not <N> numbers, one per rotated frequency | Phi-3's short_factor or long_factor is missing, holds a non-number, or has the wrong length for the rotated width. | Use the checkpoint's complete config. |
<CONFIG_PATH> declares rope_scaling <TYPE>; Ster implements llama3, linear, longrope, yarn and dynamic rotary scaling | The config scales rotary positions another way (ntk, su, …). | Use a checkpoint whose rope_scaling is absent, llama3, linear, longrope, yarn or dynamic. |
<CONFIG_PATH> declares dynamic rope_scaling with alpha but no head width to scale by | An alpha-scaled dynamic scaling in a config with neither head_dim nor hidden_size and num_attention_heads. | Use the checkpoint's complete config. |
<CONFIG_PATH> declares yarn rope_scaling with no factor | A yarn scaling without the factor that stretches the context. | Use the checkpoint's complete config. |
<CONFIG_PATH> declares kv_lora_rank without <KEY> | A latent-attention config missing qk_nope_head_dim, qk_rope_head_dim or v_head_dim. | Use the checkpoint's complete config. |
<CONFIG_PATH> declares no kv_lora_rank; Ster implements <TYPE> with its latent attention | A deepseek_v2, deepseek_v3 or minicpm3 config without the key-value bottleneck. | Use the checkpoint's complete config. |
<CONFIG_PATH> declares scoring_func "<NAME>"; Ster implements softmax and sigmoid expert scores | A DeepSeek router scored another way. | Use a checkpoint whose scoring_func is softmax or sigmoid. |
<CONFIG_PATH> declares topk_method "<NAME>"; Ster implements greedy, group_limited_greedy and noaux_tc | A DeepSeek router that chooses experts another way. | Use a checkpoint with one of those methods. |
<CONFIG_PATH> routes by <METHOD> without n_group and topk_group | Group-limited routing without its group counts. | Use the checkpoint's complete config. |
<CONFIG_PATH> splits <N> experts into <G> groups and keeps <K>; the groups must divide the experts evenly and at least as many must exist as are kept | n_group does not divide n_routed_experts, or topk_group exceeds it. | Use the checkpoint's complete config. |
<CONFIG_PATH> declares linear rope_scaling with no factor | A linear scaling without the factor that divides each angle. | Use the checkpoint's complete config. |
<CONFIG_PATH> lists <N> no_rope_layers for <L> layers | SmolLM3's per-layer rotary flags do not match the layer count. | Use the checkpoint's complete config. |
<CONFIG_PATH> marks layer <L> in no_rope_layers as <VALUE>; expected 0 or 1 | A no_rope_layers entry other than 0 (no rotation) or 1 (rotation). | Use the checkpoint's complete config. |
<CONFIG_PATH> declares hidden_act "<NAME>" for its <TYPE> feed-forward; Ster implements silu, gelu, gelu_pytorch_tanh, gelu_new, gelu_fast, relu and relu2 | The config names a feed-forward activation (hidden_act, hidden_activation or activation_function) the decoder has no implementation of. | Use the family's own config; a different activation is a different model. |
<CONFIG_PATH> declares scale_attn_by_inverse_layer_idx; Ster scales every layer's attention by the head width alone | A GPT-2 config that divides each layer's scores by its depth as well. | Use a GPT-2 checkpoint without it. |
<CONFIG_PATH> declares do_layer_norm_before false; Ster implements OPT's pre-norm blocks | An OPT config (opt-350m) that normalises after each residual add. | Use a pre-norm OPT checkpoint. |
<CONFIG_PATH> projects <W>-wide word embeddings into a <H>-wide model (word_embed_proj_dim); Ster implements OPT with embeddings as wide as the model | An OPT config whose embeddings pass through project_in and project_out. | Use an OPT checkpoint whose word_embed_proj_dim equals hidden_size. |
<CONFIG_PATH> declares apply_residual_connection_post_layernorm; Ster implements BLOOM with the residual taken before each norm | A BLOOM config that takes each residual after its norm. | Use a BLOOM checkpoint without it. |
<CONFIG_PATH> declares attn_config.<KEY> other than MPT's defaults; Ster implements MPT's plain multi-head attention with ALiBi's standard slopes | An MPT config with qk_ln, clip_qkv, softmax_scale, a non-multi-head attn_type, or an alibi_bias_max other than eight. | Use an MPT checkpoint with the default attention configuration. |
<CONFIG_PATH> uses ALiBi over <N> heads; Ster implements MPT's slopes for a power-of-two head count | An MPT config whose head count is not a power of two, where MPT's slopes differ from BLOOM's. | Use an MPT checkpoint with a power-of-two head count. |
<CONFIG_PATH> declares qk_layernorm; Ster implements Phi without query and key norms | A Phi-2-family config that normalises query and key. | Use a Phi checkpoint without qk_layernorm. |
this model has no single <TARGET> projection to adapt because <REASON>; choose adapter targets among <TARGETS> | An adapter spec or artifact names gate on a family whose feed-forward is plain (Phi-2, Starcoder2, Nemotron), any feed-forward target on a mixture of experts (Mixtral, Qwen MoE, OLMoE, GraniteMoE, DeepSeek), or the key, value or bottlenecked query of latent attention (DeepSeek, MiniCPM3). | Choose among the targets the sentence lists; it names exactly the ones this model has. |
this model's blocks are state-space mixers (Mamba) with no attention or feed-forward projection to adapt; Ster steers it but trains no adapters on it | A tune command or an adapter artifact names any target on a Mamba or Falcon-Mamba checkpoint. | Steer the model with a direction (ster vector train) instead. |
<CONFIG_PATH> declares a state-space model without intermediate_size (or expand), state_size and conv_kernel | A Mamba config missing a mixer dimension. | Use the checkpoint's complete config. |
<CONFIG_PATH> declares a Jamba model without mamba_expand, mamba_d_state and mamba_d_conv | A Jamba config missing a mixer dimension. | Use the checkpoint's complete config. |
<CONFIG_PATH> declares a Mamba-2 mixer without <HEADS_KEY> and <WIDTH_KEY> | A Mamba-2 or Bamba config missing the head count or head width. | Use the checkpoint's complete config. |
<CONFIG_PATH> splits <N> Mamba-2 heads into <G> groups; the groups must divide the heads evenly | n_groups (or mamba_n_groups) is zero or does not divide the head count. | Use the checkpoint's complete config. |
<CONFIG_PATH> declares a Mamba-2 model without state_size and conv_kernel | A Mamba-2 config missing a mixer dimension. | Use the checkpoint's complete config. |
<CONFIG_PATH> declares a Bamba model without mamba_d_state and mamba_d_conv | A Bamba config missing a mixer dimension. | Use the checkpoint's complete config. |
<CONFIG_PATH> declares an LFM2 model without conv_L_cache | An LFM2 config missing the convolution span. | Use the checkpoint's complete config. |
this model's blocks include short-convolution mixers (LFM2) with no attention projection to adapt on every layer; Ster steers it but trains no adapters on it | A tune command or adapter artifact names any target on an LFM2 checkpoint. | Steer the model with a direction (ster vector train) instead. |
<CONFIG_PATH> declares a mixture of experts without <COUNT_KEY>, num_experts_per_tok and <WIDTH_KEY> | A mixture-of-experts config missing the expert count, the experts per token, or the expert width. | Use the checkpoint's complete config. |
<CONFIG_PATH> routes each token to <K> of <N> experts; it must be at least one and at most all of them | num_experts_per_tok is zero or larger than the expert count. | Use the checkpoint's complete config. |
<CONFIG_PATH> normalises expert weights by their <P>-norm; Ster implements the 1-norm (moe_normalize_expert_weights 1) or none | A DBRX config whose moe_normalize_expert_weights is a power other than one. | Use a DBRX checkpoint that normalises by the 1-norm or not at all. |
<CONFIG_PATH> routes <K> experts per token; PhiMoE's SparseMixer chooses exactly 2 | A PhiMoE config whose num_experts_per_tok is not two. | Use a PhiMoE checkpoint that routes two experts per token. |
<CONFIG_PATH> states no router_jitter_noise; PhiMoE's SparseMixer needs it to bound the experts it weighs | A PhiMoE config without router_jitter_noise. | Use the checkpoint's published config, which states it. |
<CONFIG_PATH> lists <KEY> values that differ between layers or are not whole numbers; Ster builds every mixture-of-experts layer alike | A mixture-of-experts config whose per-layer expert count, experts per token or expert width varies between layers. | Use a checkpoint whose expert settings are the same on every layer. |
<CONFIG_PATH> shares attention across layers (use_cla) or compresses it (use_mla); Ster implements HunYuan's own attention on every layer | A HunYuan-MoE config with use_cla or use_mla set. | Use a HunYuan-MoE checkpoint with its own attention on every layer, such as Hunyuan-A13B. |
<CONFIG_PATH> adds its residual after the norm (apply_residual_connection_post_layernorm) or normalises the embeddings (embed_layernorm); Ster implements TeleChat2's pre-norm block without an embedding norm | A TeleChat config with either setting true. | Use a TeleChat2 or TeleChat2.5 checkpoint, which set both false. |
<CONFIG_PATH> declares a Granite 4.0 model without layer_types | A granitemoehybrid config that does not say which layers are Mamba-2 and which attention. | Use the checkpoint's published config, which lists layer_types. |
<CONFIG_PATH> names layer <N> <TYPE>; a Granite 4.0 layer is mamba or attention | A layer_types entry other than mamba or attention. | Use a Granite 4.0 checkpoint whose layers are Mamba-2 or attention. |
<CONFIG_PATH> declares position_embedding_type <TYPE>; Granite 4.0 rotates (rope) or does not (nope) | A Granite 4.0 config with another position scheme. | Use a checkpoint with rope or nope. |
<CONFIG_PATH> declares a Granite 4.0 model without shared_intermediate_size | A Granite 4.0 config that does not size its shared_mlp. | Use the checkpoint's published config. |
<CONFIG_PATH> declares a Nemotron-H model without hybrid_override_pattern | A nemotron_h config that does not say which layer is which. | Use the checkpoint's published config. |
<CONFIG_PATH> spells <N> layers in hybrid_override_pattern for <LAYERS> layers | The pattern has a letter count other than num_hidden_layers. | Use a config whose pattern has one letter per layer. |
<CONFIG_PATH> marks layer <N> <LETTER> in hybrid_override_pattern; Ster implements Nemotron-H's M (Mamba-2), * (attention), - (feed-forward) and E (mixture-of-experts) layers | A Nemotron-H pattern with another letter. | Use a Nemotron-H checkpoint whose layers are Mamba-2, attention, feed-forward or mixture-of-experts. |
<CONFIG_PATH> projects its experts' input into moe_latent_size; Ster implements Nemotron-H experts on the full hidden width | A Nemotron-H config whose experts work in a smaller latent width. | Use a Nemotron-H checkpoint without moe_latent_size. |
<CONFIG_PATH> normalises its output head (norm_head) or its softmax (norm_softmax); Ster implements Ling's plain lm_head | A Ling config with norm_head or norm_softmax set. | Use a Ling checkpoint that sets both false. |
<CONFIG_PATH> declares score_function <NAME>; Ster implements softmax and sigmoid expert scores | A Ling config scoring experts another way. | Use a Ling checkpoint with softmax or sigmoid scores. |
<CONFIG_PATH> declares use_mup without input_mult, output_mult and a nonzero mup_scale_factor | A Tele-FLM config that turns muP on without its multipliers. | Use the checkpoint's published config. |
<CONFIG_PATH> declares lightning-attention layers (attn_type_list 0) or shared experts (shared_intermediate_size); Ster implements MiniMax-M2 with full attention everywhere and routed experts only | A minimax_m2 config outside MiniMax-M2's published layout. | Use a MiniMax-M2 checkpoint (in BF16). |
<CONFIG_PATH> declares qk_norm_type <TYPE>; Ster implements per_layer and per_head | A MiniMax-M2 config with another query and key norm. | Use the checkpoint's published config. |
<CONFIG_PATH> lists kda_layers entry <ENTRY>, outside layers 1 to <LAYERS> | A Kimi-Linear kda_layers entry that names no layer (the list counts from one). | Use the checkpoint's published config. |
<CONFIG_PATH> declares linear_attn_config without <KEY> | A Kimi-Linear config whose linear_attn_config lacks kda_layers, num_heads, head_dim or short_conv_kernel_size. | Use the checkpoint's published config. |
<CONFIG_PATH> declares a Step3 model without share_q_dim | A step3_text config that does not size its query bottleneck. | Use the checkpoint's published config. |
<CONFIG_PATH> lists moe_layers_enum entry <ENTRY>, which names no layer below <LAYERS> | A Step3 moe_layers_enum entry that is not a layer index (counted from zero). | Use the checkpoint's published config. |
<CONFIG_PATH> declares attn_layer_indices, mamba_use_mlp false, mamba_norm_before_gate, or a projectors_bias unlike mamba_proj_bias; Ster implements Falcon-H1 with attention, a scan and a feed-forward on every layer, the gate before the norm, and one bias setting for the scan's projections | A Falcon-H1 config outside the published layout. | Use a published Falcon-H1 checkpoint. |
<CONFIG_PATH> sizes its scan at <WIDTH> but <H> heads of <D> make <H·D>; the inner width must be the heads' total | mamba_d_ssm (or mamba_expand times the hidden size) disagrees with mamba_n_heads times mamba_d_head. | Use the checkpoint's published config. |
<CONFIG_PATH> declares <KEY> that is not <N> numbers | Falcon-H1's ssm_multipliers is not five numbers or its mlp_multipliers not two. | Use the checkpoint's published config. |
<CONFIG_PATH> lists <N> layer_types for <LAYERS> layers | layer_types does not have one entry per layer. | Use the checkpoint's own config. |
<CONFIG_PATH> declares layer <L> as <KIND>; Ster implements sliding_attention and full_attention | A layer type other than sliding or full attention. | That layer kind is not implemented; use a checkpoint of a family on model resolution. |
<CONFIG_PATH> makes some layers sliding-window but declares no sliding_window | Layers are marked local, but the window width is missing. | Use the checkpoint's own config, which states sliding_window. |
<CONFIG_PATH> has <N> layers with per-layer attention settings; Ster tracks them per layer for at most 128 layers | A model deeper than the per-layer sets Ster keeps (sliding window, Gemma 3 pattern, SmolLM3's unrotated layers). | No supported checkpoint is that deep; this refusal exists so a deeper one is not loaded with the wrong per-layer settings. |
failed to load tokenizer <PATH>: <DETAIL> | tokenizer JSON or its model is invalid/incompatible. | Restore the tokenizer shipped with the checkpoint. |
failed to map <N> model weight files | Safetensors mapping/header/tensor loading failed. | Check every shard, permissions, completeness, disk/memory, and config/weight agreement. |
A model-looking string that is not an existing directory is treated as a Hub id. If an intended local path produces Hub errors, fix or create the directory. Hub config.json/tokenizer.json fetches and low-level Candle operations can also surface dependency-generated detail without an additional Ster sentence; retain the full Error: chain when reporting them.
Pair and prompt inputs#
| Sentence | Meaning | Fix |
|---|---|---|
failed to read pair set <PATH> | Pair file cannot be opened. | Correct the path and permissions. |
invalid pair set JSON in <PATH> | JSON does not deserialize to trait_name plus pairs[].positive/negative. | Fix syntax, field names, and string types. |
pair set <PATH> contains no pairs | pairs is empty. | Add at least one matched pair. |
pair set <PATH> contains an empty positive or negative prompt | A side is blank after trimming. | Fill both sides of every pair. |
the held-out fraction must be above zero and below one | ster vector optimize --holdout (or holdout) is 0, 1, or outside that range. | State a fraction between 0 and 1. |
a held-out fraction of <F> over <N> contrastive pairs leaves <FIT> pairs to fit and <HELD> to rank on; both need at least one | The stated fraction, rounded to whole pairs, leaves one side empty. | Add pairs or change the fraction; use train when selection is not needed. |
the expected calibration error needs at least one bin | --ece-bins 0 (or eceBins: 0) on calibrate or decisions benchmark. | State at least one bin. |
error: the following required arguments were not provided: <FLAG> | A training, sampling or judging number was left out; Ster assumes none. Exit 2. | State the named flag. |
request body is not valid JSON: missing field \<FIELD>\ at line <L> column <C> | A ster request body left out a required number. Status 2. | Send the named field. |
failed to read <PATH> | extract cannot read its prompt file. | Correct path/permissions. |
invalid prompt JSON in <PATH> | Extract input is not {"prompts":[strings...]}. | Fix syntax and shape. |
prompt set contains no prompts | Extract prompts is empty. | Add at least one prompt. |
prompt must not be empty | A generation, pair, or extract prompt is blank. | Supply non-whitespace text. |
failed to tokenize prompt: <DETAIL> | Tokenizer rejected the text. | Check tokenizer compatibility/input and retain the detail. |
tokenizer produced no tokens | Tokenization succeeded with an empty id sequence. | Supply text recognized by a valid tokenizer. |
prompt contains <N> tokens, model context allows fewer than <LIMIT> | Generation prompt already fills/exceeds context. | Shorten it; prompt length must be strictly below the limit. |
failed to decode generated tokens: <DETAIL> | Tokenizer could not decode the generated suffix. | Check tokenizer/model vocabulary agreement and preserve the detail. |
max_new_tokens must be greater than zero | Generation token budget is 0. | Set --max-new-tokens/maxNewTokens to at least 1. |
An immediate EOS is not an error: generation succeeds with an empty text line or {"text":""}.
Layer selection#
| Sentence | Meaning | Fix |
|---|---|---|
model has no layers | Loaded config reports zero decoder blocks. | Use a valid model config. |
invalid layer "<SEGMENT>" | A non-range segment is not an unsigned integer. | Use 5, a valid range, or all. |
invalid layer range "<SEGMENT>" | A range endpoint is not an unsigned integer. | Use syntax such as 8..16. |
layer range "<SEGMENT>" must have start < end | Range is empty or reversed. | Make the half-open end greater than start. |
no layers selected | The expression contains only commas/whitespace. | Select at least one index or all. |
at least one layer is required | A library call or artifact supplies an empty layer list. | Provide a non-empty list/artifact. |
layer <L> is outside the model's 0..<MAX> range | Selection/artifact layer is too large. | Select only 0-based decoder layers in range. |
Training and numerical routines#
These normally indicate bad direct library inputs; CLI capture creates consistent rows.
| Sentence | Meaning | Fix |
|---|---|---|
unknown training method "<VALUE>"; expected caa, pca, or logistic | Method is unsupported. | Use a canonical method or documented alias. |
training requires the same non-zero number of positive and negative examples | Row sides are empty or counts differ. | Supply matched non-empty rows. |
activation vectors are empty | Hidden width is zero. | Supply model-width rows. |
activation vectors must have one finite, consistent width | A row width differs or any value is NaN/infinite. | Normalize shape and remove non-finite inputs. |
training produced a zero or non-finite direction | Learned vector cannot be unit-normalized. | Improve non-degenerate contrastive data. |
direction width <D> does not match activation width <A> | Evaluation direction has wrong width. | Use a direction from the same model space. |
cosine similarity requires equal non-empty vectors | Direct similarity inputs differ in width or are empty. | Supply equal non-empty vectors. |
cosine similarity is undefined for a zero vector | Either similarity input has near-zero norm. | Use non-zero vectors. |
unknown chat template mode "<VALUE>"; expected auto or off | --chat-template / chatTemplate got a third value. | Use auto or off; there is no on, because a checkpoint with no template cannot be forced into one. |
this model's chat template does not parse: <ERROR> | The checkpoint publishes a chat_template MiniJinja cannot compile. | Fix the template in tokenizer_config.json, or run with --chat-template off. |
this model's chat template failed to render: <ERROR> | The template raised while rendering the turn, often its own raise_exception for a role order it forbids. | A pair set renders one lone assistant turn; use --chat-template off for a template that requires a user turn first. |
this model's chat template uses <TOKEN>, but its tokenizer config declares none | The template writes bos_token or eos_token and the config has no such token. | Add the token to tokenizer_config.json; rendering without it would silently drop the marker. |
this model's chat template does not render the assistant turn after the prompt, so the completion boundary cannot be located | The full conversation does not begin with the generation prompt, so the loss window cannot be taken by subtraction. | The template is not usable for supervised training; run with --chat-template off. |
this model's chat template rendered an empty assistant turn | The template dropped the completion entirely. | Check the assistant branch of the template. |
this model publishes several named chat templates and none of them is the default | The chat_template list form carries no entry named default. | Name one entry default, or run with --chat-template off. |
this model's chat template list has an entry with no template | A list entry has a name but no template. | Repair tokenizer_config.json. |
this model's chat_template is neither a template nor a list of them | chat_template is a JSON value of some other type. | Repair tokenizer_config.json. |
<SUBJECT> requires a batch of at least one <UNIT> | --batch-size / batchSize is zero — e.g. supervised fine-tuning requires a batch of at least one example, reward modeling requires a batch of at least one pair. | Fold at least one row into each forward pass. |
unknown precision "<VALUE>"; expected f32, f16, or bf16 | --precision / precision got an unsupported dtype. | Use f32, f16, or bf16. |
bf16 has no CPU matmul kernel in this Candle build; use --precision f16 for half precision on cpu, or --device metal for bf16 | bf16 was asked for on the cpu device. Candle's CPU matmul accepts F16, F32 and F64 only, so the run would map the whole checkpoint and then die at the first projection. | Use --precision f16 on cpu, or --device metal for bf16. Refused at load, before any weight is mapped. |
An interrupted training run loses everything#
Before starting a long run, know what an interruption costs. The product has exactly one adapter writer, lora::Artifact::save in src/lora.rs, and each of the four objectives reaches it once — after tune::sft, tune::dpo, tune::reward or tune::grpo has returned its finished report. The objectives themselves write nothing to disk at all.
- No periodic checkpoint and no per-epoch snapshot: there is no partial adapter to salvage.
- No signal handler:
Ctrl-C, a closed terminal, an OOM kill or a lost machine ends the process with nothing written. - No
--resumeflag on anyster tunesubcommand, and nothing on disk for one to read: the only way to continue is to start again from the base checkpoint.
A run killed in its sixth hour costs all six. Size runs to what you are willing to lose: several shorter runs over smaller inputs, each with its own --output, keep finished work; one long run keeps none until it returns.
Artifact I/O and validation#
| Sentence | Meaning | Fix |
|---|---|---|
failed to read steering artifact <PATH> | File cannot be opened. | Correct path/permissions. |
<PATH> is a LoRA adapter sidecar, not a steering artifact: it carries rank and targets where a steering artifact carries trait_name and vectors | A --vector (or /v1 vector) path names an adapter sidecar. The loader recognises the other product's document instead of letting serde escape as a missing field a hundred thousand columns in. | Pass the steering artifact train or optimize wrote. An adapter belongs on --adapter. |
invalid steering artifact JSON in <PATH> | JSON syntax or required field/type is wrong. | Regenerate or repair against schema 1. |
artifact schema <N> is unsupported; this Ster build reads schema 1 | Schema version differs. | Use a schema-1 artifact with this build; do not relabel incompatible data. |
artifact belongs to product "<PRODUCT>", not Ster | Product discriminator is not ster. | Supply a genuine Ster artifact. |
artifact has no steering vectors | Width is 0 or vector list empty. | Retrain/regenerate the artifact. |
layer <L> has vector width <W>, expected <H> | Artifact's own width invariant fails. | Restore untampered output or retrain. |
layer <L> contains a non-finite value | A vector contains NaN/infinity. | Retrain; do not apply it. |
failed to create <PARENT> | Artifact parent directory creation failed. | Fix path/permissions. |
failed to write steering artifact <PATH> | Artifact serialization reached a filesystem write failure. | Fix space, permissions, or destination. |
failed to inspect <PATH> | inspect context wrapper; the next cause is the actual I/O/schema refusal. | Fix the deepest cause. |
<PATH> is a steering artifact, not a LoRA adapter sidecar: it carries trait_name and vectors where an adapter sidecar carries rank and targets | An --adapter (or /v1 adapter) path names a steering artifact. The mirror of the refusal above, written as one: generate takes --vector and --adapter side by side, and an operator who crosses them gets the same sentence from whichever end they crossed. | Pass the adapter weights ster tune wrote. A direction belongs on --vector. |
artifact model "<A>" does not match runtime model "<B>" | evaluate requires exact model-string equality. | Load the artifact's exact model string or retrain. |
artifact was trained for model "<A>", current model is "<B>" | generate model-string gate failed. | Same: exact string match or retrain. |
artifact width <A> does not match model width <M> | Runtime hidden width differs. | Use the training checkpoint/artifact pair. |
layer <L> steering vector width <W> does not match model width <H> | Low-level plan width gate failed. | Supply model-width vectors. |
steering plan contains no vectors | Direct low-level plan construction got no entries. | Supply at least one vector. |
failed to apply steering at layer <L> | Candle rejected broadcast-add during the forward pass. | Treat as model/plan shape or device failure; retain the chained detail. |
Activation export may return failed to write <PATH> for its destination; fix parent permissions/space. Unlike artifact save, export's parent-directory creation may surface a raw OS error without a Ster wrapper.
Both cross-document sentences name the path the reader would recognise: the --vector refusal names the path the operator gave, and the --adapter refusal names the sidecar it derived from the weights — which for a .json path is the same file. Neither is a warning; a document of the wrong kind has no reading that could be right.
CLI parser errors and exit behavior#
Missing required arguments, invalid numeric values, unknown flags/subcommands, an unknown --device (invalid value 'gpu' for '--device <DEVICE>': unknown device "gpu"; expected cpu, metal, or cuda), --precision, --chat-template or train --method, and values outside clap's target type fail before Ster code runs, so a typo never costs a checkpoint load. Clap prints a lowercase error: usage message and exits 2. A known device the binary was not built for, and bf16 on cpu, are refused later, at load, with exit 1. In particular, attach a negative number: --strength=-2.0; --strength -2.0 is parsed as an unexpected flag (the captured build reported error: unexpected argument '-2' found).
After parsing, any anyhow::Result failure exits non-zero and begins with Error:. Progress already written to stderr is not rolled back, and output files are not transactional: if a filesystem fails during a write, inspect or remove the partial destination before retrying.
Desktop request refusals#
ster request <operation> refuses before anything runs with exactly one result event whose status is 2 and whose json is {"error": "<sentence>"}, and exits 2. No log event precedes a refusal (desktop request reference).
| Sentence | When | Fix |
|---|---|---|
unknown operation: <OPERATION> | The argument names no operation. | Use one of the operations ster request --help lists. An HTTP path such as /v1/train is not an operation; the operation is train. |
request body is not valid JSON: <DETAIL> | stdin did not deserialize into the operation's body: malformed JSON, a field of the wrong type, or an unknown field on workspace/import-pairs, which denies them. | Write one JSON document to stdin, or nothing for {}. The detail is serde's, with the line and column it stopped at. |
A body that cannot be read at all is not a refusal: Error: failed to read the request body from stdin goes to stderr and the process exits 1 without any event, because nothing on stdout was written yet.
Required request fields
These are pre-run validation sentences, each returned as a status-2 result. Within one operation the fields are checked in the order listed. The tune/* operations list theirs on the desktop request reference.
workspace import-pairs requires a source pathvector/train requires a modelvector/train requires a pairs filevector/train requires an output pathvector/optimize requires a modelvector/optimize requires a pairs filevector/optimize requires an output pathvector/evaluate requires a modelvector/evaluate requires a pairs filevector/evaluate requires a steering artifactgenerate requires a modelgenerate requires a promptextract requires a modelextract requires a prompt input fileextract requires an output pathvector/inspect requires a steering artifactpairs inspect requires a pairs filepairs save requires an output pathpairs save requires at least one pairpairs synthesize requires a modelon the local generator,pairs synthesize with the brama generator requires generatorModelon the Brama one, andunknown generator; expected local or bramafor any otherpairs synthesize requires a trait descriptionpairs synthesize requires an output pathpairs synthesize requires a pair count above zerodecide requires a modeldecide requires a request with a state and questions, then any request refusal from the decision refusals belowcalibrate requires a modelcalibrate requires an examples pathcalibrate requires an output path
A failure after the operation starts is not a refusal: its sentence arrives as the last log event, prefixed error: , the result has status: 1 and json.error, and the process exits 1. Gate success on the result's status or on the exit status. A successful request ends with exactly one status: 0 result and exits 0. A process that ends with no result event at all was killed or crashed, and its stderr says why; Ster Desktop reports that case as The run ended with status <N> before a result was received. followed by the stderr text.
Workspace import refusals#
status: rejected— source JSON is unreadable, has unknown fields, contains no pairs, contains an empty side, or--nameviolates the identifier rule. No pair file or manifest entry is committed.status: conflicting— the logical name or destination already belongs to different content. Choose another--name; Ster preserves the existing set.active Ster pair set is missing— an indexed file was removed outside Ster. Import it again. The manifest is not silently healed to another set.
Decision refusals#
ster decide and ster calibrate check their documents before a weight is mapped, so every sentence below costs milliseconds and exits 1. Under ster request decide, the request's own refusals — the a decide request …, question, option and level rows — are checked before anything runs and return a status-2 result. The rest read a file or the tokenizer, so under ster request they arrive after the operation has started, as the last error: log event and a status-1 result. The command pages (decide, calibrate) say what each field is.
| Sentence | What to do |
|---|---|
a decide request has an empty state | Put the content to judge in state; a string or any JSON value. |
a decide request needs at least one question | Add a question under questions. |
a decide request has a question with an empty id | Give every question a non-blank id. |
question '<id>' has empty instructions | Write the question in instructions. |
choice question '<id>' needs at least two options | A choice with one option is not a choice; add options or ask a noul. |
choice question '<id>' has <n> options; Ster labels at most 26 | Every option is answered by one letter. Split the question or shortlist first. |
choice question '<id>' has an option with an empty name "<name>" | Name every option; the name is what the answer returns. |
score question '<id>' needs at least two levels | A scale needs two levels at least. |
score question '<id>' has <n> levels; Ster labels at most 26 | Fewer, better-described levels. |
score question '<id>' has an empty description at level <i> | Describe the level; the model reads only the descriptions. |
invalid decide request <path> / invalid decide request on standard input | The reader's cause is beneath it: a missing field, a wrong type, or malformed JSON. |
question '<id>' renders to <n> tokens, past the <max> this model was built for | Shorten the state or the options, or use a model with a longer context. |
this tokenizer has no single token for the answer label "<label>", so it cannot be read from one position | This checkpoint's vocabulary cannot spell that letter as one token; fewer options, or another checkpoint. |
calibration needs at least one labelled example | The examples array is empty. |
calibration needs at least one labelled question | No example carries an answers entry. |
calibration example <i> is not a valid request | The request refusal beneath it names the field. |
example <i> labels question '<id>', which it does not ask | The answers key must be one of that example's question ids. |
example <i> answers question '<id>' with <label>, which is not one of its options | A choice is labelled with an option name, a score with a level index, a noul with true or false. |
invalid calibration <path> / invalid calibration examples <path> | The document is not the schema the page describes; the cause is beneath. |
calibration <path> has schema "<schema>"; this Ster reads ster-calibration/1 | Refit with this Ster. |
calibration <path> has a temperature that is not a positive number | The artifact was edited by hand; refit. |
calibration <path> was fitted for model '<fitted>', not '<loaded>' | A temperature belongs to one checkpoint. Refit on the model you are deciding with. |