ster vector evaluate
Measure pair-ordering accuracy and mean projection margin for every direction in an existing steering artifact.
Invocation and purpose#
ster vector evaluate --model <MODEL> --vector <VECTOR> [--pairs <PAIRS>] [--chat-template <auto|off>] [--precision <f32|f16|bf16>] [--revision <REVISION>] [--device <DEVICE>]Loads a model, pair set, and steering artifact; re-captures activations at exactly the artifact's layers; and measures how often each positive prompt projects above its paired negative prompt plus the mean projection gap.
Inputs and options#
| Input | Required | Meaning |
|---|---|---|
--model <MODEL> | yes | Hugging Face model id or local model directory; its string identity must equal the artifact model. |
--pairs <PAIRS> | no* | Non-empty JSON contrastive pair set to score. Omit it to use the active pair set accepted by ster workspace import-pairs; without either source Ster refuses before loading a model. |
--vector <VECTOR> | yes | Existing schema-1 Ster steering artifact. |
--revision <REVISION> | no | Immutable Hugging Face revision; Hub resolution defaults to main. |
--device <DEVICE> | no | cpu (default), metal, or cuda. |
--chat-template <auto|off> | no | auto by default. auto reads every pair through the model's own chat template when the checkpoint publishes one, off reads it as raw text (chat template). It should match the run that trained the artifact for the same reason --precision should, and because the artifact now records what that run did, a mismatch is said out loud rather than left to memory. |
--precision <f32|f16|bf16> | no | f32 by default. The dtype the base weights are mapped at: f32, f16, or bf16. bf16 is refused on cpu at load with `bf16 has no CPU matmul kernel in this Candle build; use --precision f16 for half precision on cpu, or --device metal for bf16` (precision). It should match the run that trained the artifact, or the score measures the direction in a space it was not fitted in. |
Output and state#
Makes no durable state change. Stdout is pretty-printed JSON with model, trait_name, method, pair_count, chat_template — the format this run encoded in, not the one the artifact was fitted in — and one layer, accuracy, and margin record per artifact vector. Activation-capture progress goes to stderr.
Before the first pair is read, stderr also carries a warning when the artifact was produced in a different space than this run reads in: warning: this direction was trained with chat template absent and this run encodes off, so the number below describes a format it was not trained in, and the same shape for precision, warning: this direction was trained at precision f32 and this run maps the base weights at f16, so it is being read in a different space than it was fitted in. Both are warnings and not refusals, because measuring how far a direction transfers out of the space it was fitted in is a real question; only an unnoticed mismatch is a defect. An artifact written before those fields existed records neither and is warned about neither.
Refusals#
- A model identity mismatch is refused exactly as
artifact model <ARTIFACT_MODEL> does not match runtime model <RUNTIME_MODEL>. - Artifact validation refuses an unsupported schema, another product, no vectors, a wrong vector width, or a non-finite value.
- Missing, invalid, empty, or partly blank pair files are refused by the pair loader.
- Artifact layers outside the model's layer range and unsupported model/device selections are refused before scoring.