Overview#

This is the recorded offline Ster 0.13 session behind the quick start. It ran on CPU against the seeded toy Llama checkpoint produced by `examples/make-toy-model.py`. No network, GPU, or model service was used. Random weights make the generated text meaningless; the session demonstrates the real data path and gates.

Prepare the inputs#

bash

WORK=$(mktemp -d)
cd "$WORK"
python3 <ster-repo>/docs/examples/make-toy-model.py toy-llama
cp <ster-repo>/docs/examples/pairs.json .
cp <ster-repo>/docs/examples/prompts.json .
STER=<ster-repo>/target/release/ster

The executed pair file has eight matched calm/stormy pairs. The checkpoint has four layers (0..3) and hidden width 64.

Train two layers#

console

$ $STER train --model toy-llama --pairs pairs.json --layers 1..3 --method caa --output calm.ster.json
this model publishes no chat template, so prompts and completions are encoded as raw text
reading pair 1/8
reading pair 2/8
reading pair 3/8
reading pair 4/8
reading pair 5/8
reading pair 6/8
reading pair 7/8
reading pair 8/8
{
  "artifact": {
    "chat_template": "absent",
    "hidden_size": 64,
    "layers": [
      {
        "layer": 1,
        "norm": 1.0000000062843015,
        "train_accuracy": 0.875,
        "train_margin": 0.5598376393318176,
        "width": 64
      },
      {
        "layer": 2,
        "norm": 1.0000000878582571,
        "train_accuracy": 0.875,
        "train_margin": 0.7613550424575806,
        "width": 64
      }
    ],
    "metadata": {},
    "method": "caa",
    "model": "toy-llama",
    "model_revision": null,
    "precision": "f32",
    "product": "ster",
    "schema_version": 1,
    "trait_name": "calm"
  },
  "chat_template": "absent"
}

The template line and the eight reading pair lines were stderr; the JSON summary was stdout. The toy checkpoint publishes no chat template, so auto reports absent and encodes raw text — the same ids off would produce, recorded under a different word because the two mean different things about the run. The compact calm.ster.json includes both full 64-value vectors, not just the summary.

Search method and layer#

console

$ $STER optimize --model toy-llama --pairs pairs.json --layers all --output calm.best.ster.json
this model publishes no chat template, so prompts and completions are encoded as raw text
reading pair 1/8
reading pair 2/8
reading pair 3/8
reading pair 4/8
reading pair 5/8
reading pair 6/8
reading pair 7/8
reading pair 8/8
fitting each candidate on 6 pairs and ranking on a 2-pair holdout

The command evaluated caa, pca, and logistic on every one of the four layers using the ordered 80/20 holdout — twelve candidates, fitted on six pairs and ranked on two — then retrained the winner on all eight pairs. The recorded output artifact selected CAA at layer 3:

json

{
  "model": "toy-llama",
  "precision": "f32",
  "chat_template": "absent",
  "trait_name": "calm",
  "method": "caa",
  "hidden_size": 64,
  "vectors": [
    { "layer": 3, "train_margin": 0.92032135, "train_accuracy": 0.875 }
  ],
  "metadata": { "selection":
    "chosen over 12 candidates on a 2-pair holdout, then refitted on all 8 pairs" }
}

That excerpt omits schema/product/revision and the 64 values; it reports exact fields read from calm.best.ster.json. The saved scores are the retrained full-set scores, not the holdout scores used for selection.

Stdout carries the evidence for that choice. Three of the twelve candidate rows, exact:

json

{ "selection": {
    "holdout": { "fit_pairs": 6, "holdout_pairs": 2 },
    "candidates": [
      { "layer": 2, "method": "caa",
        "holdout_accuracy": 1.0, "holdout_margin": 0.7814546823501587,
        "selected": false },
      { "layer": 2, "method": "pca",
        "holdout_accuracy": 0.5, "holdout_margin": -0.0392594039440155,
        "selected": false },
      { "layer": 3, "method": "caa",
        "holdout_accuracy": 1.0, "holdout_margin": 0.9493659734725952,
        "selected": true } ] } }

Eleven of the twelve candidates reach holdout accuracy 1.0 on two pairs, which is exactly the weakness of a small holdout: the ranking is decided on margin almost throughout, and layer 3 CAA wins it at 0.9493659734725952. The one candidate that does not is PCA at layer 2, at 0.5 with a negative margin — it ordered one of the two holdout pairs backwards.

For comparison, separately executed layer-2 artifacts scored CAA 0.875 / 0.76135504, PCA 0.625 / 0.23738393, and logistic 1.0 / 0.73598194 as training accuracy/margin. Method quality is layer-dependent; a probe's perfect training score is not proof it generalizes.

Evaluate the two-layer artifact#

console

$ $STER evaluate --model toy-llama --pairs pairs.json --vector calm.ster.json
{
  "chat_template": "absent",
  "layers": [
    { "layer": 1, "accuracy": 0.875, "margin": 0.5598376393318176 },
    { "layer": 2, "accuracy": 0.875, "margin": 0.7613550424575806 }
  ],
  "method": "caa",
  "model": "toy-llama",
  "pair_count": 8,
  "trait_name": "calm"
}

This deliberately reuses the training set to show mechanics. Use a different held-out pair file for evidence. chat_template is what this run encoded in; it matches the absent the artifact recorded, so no provenance warning was printed. Running the same command with --chat-template off prints one — warning: this direction was trained with chat template absent and this run encodes off, so the number below describes a format it was not trained in — and returns the same numbers, because a checkpoint with no template encodes both ways identically.

Compare generation#

console

$ $STER generate --model toy-llama --prompt "describe the evening lake ." --max-new-tokens 12
loud sea hard hard , white , white , white loud loud

$ $STER generate --model toy-llama --vector calm.ster.json --strength 1.0 --prompt "describe the evening lake ." --max-new-tokens 12
white white white howls howls howls drifts , white white white white

Both runs used deterministic argmax. The changed text proves the real intervention reached logits. It says nothing semantic about a random model. The same experiment at strength 2.0 collapsed to one repeated token; larger positive values reached EOS immediately. Negative values need attached CLI syntax, for example --strength=-1.0.

Export raw activations#

console

$ $STER extract --model toy-llama --input prompts.json --layers 0,3 --output activations.json
extracting prompt 1/2
extracting prompt 2/2
activations.json

The recorded file has schema 1, product ster, model toy-llama, width 64, two records, and two full vectors per record under string keys "0" and "3". No Ster command consumes this activation artifact; it is the external analysis boundary.

Inspect and trip a gate#

bash

$STER inspect calm.best.ster.json

inspect pretty-printed the artifact summary — identity, precision, chat_template, metadata, and layer, width, norm, train_accuracy and train_margin for the one selected layer — without loading a model, and without the 64 vector values. A copy whose product was changed to other was refused:

text

Error: failed to inspect tampered.json

Caused by:
    artifact belongs to product "other", not Ster

The same product gate applies before evaluation or intervention. Other operator-facing failures and fixes are in the runbook.

The document-kind gate is the other one worth tripping. $STER evaluate --vector calm.lora.json, pointed at an adapter sidecar rather than a direction, refused with calm.lora.json is a LoRA adapter sidecar, not a steering artifact: it carries rank and targets where a steering artifact carries trait_name and vectors, and $STER generate --adapter calm.ster.json refused with the mirror sentence, calm.ster.json is a steering artifact, not a LoRA adapter sidecar: it carries trait_name and vectors where an adapter sidecar carries rank and targets.