Evaluation

Overview#

Evaluation asks whether a trained direction orders each matched contrastive pair correctly at one layer. It is the same calculation used while training, while selecting a method and layer, and by ster vector evaluate; no separate classifier or generation judge is involved.

The two measures#

For positive activation $p$, negative activation $n$, and direction $d$, Ster computes the pair margin

$$m=(p\cdot d)-(n\cdot d).$$

A pair is correct only when $m>0$; a zero margin is not correct.

• accuracy is correct_pair_count / pair_count, in [0, 1];
• margin is the arithmetic mean of all pair margins and may be negative.

Directions are unit-normalized before scoring, so the margin is in activation projection units and is comparable across training methods for the same model and layer. It is not a probability, confidence interval, or generation-quality score.

Three places it appears#

1. `ster vector train` scores every selected layer on the same pairs used to train it. The values become vectors[].train_accuracy and vectors[].train_margin in the artifact. These are training scores, not held-out evidence.
2. `ster vector optimize` splits pairs in file order: the first floor(4n/5) pairs train each candidate and the remainder score it. The highest holdout accuracy wins; exact accuracy ties use the higher margin. Every candidate's holdout_accuracy and holdout_margin are published in the result beside the winner, with holdout giving the two counts the split produced, so the choice can be checked rather than trusted. The winning method and layer are then retrained on all pairs, so the scores saved in the final artifact are again full-set training scores rather than the holdout scores that selected it.
3. `ster vector evaluate` captures a supplied pair set at every layer carried by an artifact and returns fresh scores without changing the artifact.

The executed toy evaluation was:

{
  "model": "toy-llama",
  "trait_name": "calm",
  "method": "caa",
  "pair_count": 8,
  "layers": [
    { "layer": 1, "accuracy": 0.875, "margin": 0.55983764 },
    { "layer": 2, "accuracy": 0.875, "margin": 0.76135504 }
  ]
}

A holdout of one pair scores every candidate 0 or 1, which separates almost nothing; optimize says so on stderr when the split leaves one, and a four-pair set — the documented minimum — always does. It is a fact about the input rather than a defect, so it is stated rather than refused (`ster vector optimize`).

Reading the result#

An accuracy near 0.5 means the pair ordering is not reliably separated at that layer; a high training score may still be memorization or a confound in the pair design. Keep a held-out pair file for the final evaluate, and compare unsteered and steered generation separately: projection ordering does not prove that generation changed in the intended way.

A negative margin is worth separating from a low one. A margin near zero means the direction barely orders the pairs; a margin below zero means it orders them backwards, so adding it during generation pushes toward the trait's opposite. One cause is measuring a direction in a format it was not fitted in — on TinyLlama-1.1B-Chat-v1.0 that inverts the deepest three of 22 layers (chat template) — and ster vector evaluate now warns before the first pair is read when the artifact records a chat_template or precision this run does not use.

Invariants and refusals#

evaluate_direction requires equal, non-empty positive and negative rows, one finite non-zero width, and a direction of that same width. Its source-authored refusals are:

• training requires the same non-zero number of positive and negative examples
• activation vectors are empty
• activation vectors must have one finite, consistent width
• direction width <D> does not match activation width <A>

The CLI additionally gates the pair set, artifact, model identity, and artifact layers before or during capture; see the runbook.