Ster
Ster/Docs

ster generate

Generate text from a local model of a supported family, optionally adding any number of validated steering artifacts, each at its own strength, at every generation step.

Invocation and purpose#

text
ster generate --model <MODEL> --prompt <PROMPT> --max-new-tokens <COUNT> --temperature <TEMPERATURE> --seed <SEED> [--vector <VECTOR> --strength <STRENGTH>]... [--system <FILE>] [--revision <REVISION>] [--device <DEVICE>] [--adapter <ADAPTER>] [--top-p <TOP_P>] [--chat-template <auto|off>] [--precision <f32|f16|bf16>]

Runs autoregressive generation with a KV cache. Without --vector it is an unsteered baseline; with artifacts it adds, after each represented transformer layer on every forward step, the sum of every vector times the --strength in the same place — several traits steered at once, as wisent's multi-steer did. A direction and its negation at equal strengths cancel to the unsteered answer, and one vector named twice at a strength steers as once at twice that strength. --adapter is the other way to change what the model says: a trained LoRA adapter is attached, frozen, while the weights are mapped, so every token is generated through the adapted projections. The two compose — an adapter changes the projections, a vector is added to the residual stream — and neither writes anything back to the checkpoint. Through ster request generate the pairs travel as steering: [{"vector": "…", "strength": …}], each strength required.

Inputs and options#

InputRequiredMeaning
--model <MODEL>yesHugging Face model id or local model directory.
--prompt <PROMPT>yesNon-empty input text shorter than the model context.
--system <FILE>noFile whose text is the system turn the prompt is answered under. The two are rendered as a system turn and a user turn by the model's own chat template, so a model trained under a system prompt is asked the way it was trained. It is read before any weight is mapped.
--revision <REVISION>noImmutable Hugging Face revision; Hub resolution defaults to main.
--device <DEVICE>nocpu (default), metal, or cuda.
--vector <VECTOR>no, repeatableValidated Ster steering artifact for the same model identity and hidden width; repeat it to steer several traits at once.
--adapter <ADAPTER>noValidated LoRA adapter artifact trained for this exact model. Name the safetensors path; the JSON sidecar beside it must be present too.
--strength <STRENGTH>one per --vectorLinear multiplier on the --vector in the same place; attach negatives as --strength=-2.0. It exists only with a vector: Ster assumes none, and --strength without --vector is a usage error.
--max-new-tokens <COUNT>yesMaximum generated tokens; must be greater than zero.
--temperature <TEMPERATURE>yes0 selects deterministic argmax; positive values sample.
--top-p <TOP_P>noOptional nucleus threshold used only with positive temperature; without it sampling draws from the whole distribution.
--seed <SEED>yesSampling seed; irrelevant to argmax.
--chat-template <auto|off>noauto (default) or off. auto encodes the prompt through the model's own chat template when the checkpoint publishes one, and as raw text when it does not, saying which on one progress line and recording it in the report as chat_template = applied, absent, or off. off is the raw-text encoding every release before 0.13 used. Any other value is refused with `unknown chat template mode "<VALUE>"; expected auto or off` (chat template).
--precision <f32|f16|bf16>nof32 by default. The dtype the base weights are mapped at: f32, f16, or bf16. bf16 is refused on cpu at load with `bf16 has no CPU matmul kernel in this Candle build; use --precision f16 for half precision on cpu, or --device metal for bf16` (precision). A steering vector is cast to it on the way in and survives as the same direction.

Output and state#

Prints generated text only, excluding the prompt and skipping special tokens. It writes no files. Generation stops at the token limit, an EOS token, or model context capacity; an immediate EOS can produce an empty line.

Stderr carries the run's chat-template decision, and, when --vector names a direction whose recorded precision or chat_template differs from this run, a warning before the first token is drawn: warning: this direction was trained with chat template applied and this run encodes off, so the number below describes a format it was not trained in, or the precision equivalent. It is a warning and not a refusal, but for generation it is the sharper of the two — a direction fitted in one format and added during a decode in another is measured in one space and steers another (chat template).

Refusals#

  • Leaving out --max-new-tokens, --temperature or --seed is a usage error that names the flag, exit 2, before anything loads. A --vector without its own --strength is refused as every --vector needs its own --strength: got <V> vector(s) and <S> strength(s); Ster assumes no steering scale before a weight is mapped, and --strength without --vector is a usage error. Strengths that are all zero are refused as every steering strength is zero (<STRENGTHS>), so the vectors would steer nothing, and a non-finite one as steering strength <S> is not finite.
  • Zero output length is refused exactly as max_new_tokens must be greater than zero; blank input is refused as prompt must not be empty.
  • A prompt already at context capacity is refused as prompt contains <COUNT> tokens, model context allows fewer than <LIMIT>.
  • A steering artifact for another model or width is refused as artifact was trained for model <ARTIFACT_MODEL>, current model is <MODEL> or artifact width <WIDTH> does not match model width <WIDTH>.
  • --system is refused as cannot read the system turn <FILE> when the file cannot be read, as the system turn <FILE> is empty when it holds only whitespace, and as a conversation of several turns needs the model's chat template, and this run's chat template is <absent|off> when the checkpoint publishes no template or --chat-template off was given.
  • An adapter is gated the same way, at load time rather than halfway through a decode: adapter was trained for model "<ADAPTER_MODEL>", current model is "<MODEL>" and adapter width <WIDTH> does not match model width <WIDTH>. A path that is not there is refused as failed to read adapter <PATH>, and weights whose sidecar is absent as adapter <PATH> has no sidecar at <SIDECAR>; the pair is written together and must travel together. Every other adapter validation refusal is listed under LoRA adapter.
  • Artifact validation also refuses unsupported schema/product, no vectors, inconsistent widths, non-finite values, or vector layers outside the model.
  • Crossing the two flags is refused by name rather than by serde. --adapter handed a steering artifact refuses as direction.json is a steering artifact, not a LoRA adapter sidecar: it carries trait_name and vectors where an adapter sidecar carries rank and targets, and --vector handed an adapter sidecar refuses as direction.json is a LoRA adapter sidecar, not a steering artifact: it carries rank and targets where a steering artifact carries trait_name and vectors. The two flags sit side by side here, so an operator who crosses them gets the same sentence from whichever end they crossed.