Overview#

A contrastive pair is one positive prompt and one negative prompt that differ in the trait being studied; a pair set is a named list of them. Pairs are the only supervision Ster ever takes: directions are learned from the difference between the two sides' activations, and evaluation asks whether each positive out-projects its own negative.

Shape#

One JSON document (PairSet in src/artifact.rs):

json

{
  "trait_name": "calm",
  "pairs": [
    {
      "positive": "the sea is calm and quiet tonight .",
      "negative": "the storm waves crash loud against the rocks ."
    }
  ]
}

trait_name — optional (defaults to ""); copied verbatim into every artifact trained from the set. Name the trait, not the experiment.
pairs[] — objects with exactly positive and negative strings (ContrastivePair). Order matters twice: pairs are matched positionally during training, and ster vector optimize holds out the last --holdout fraction in file order — shuffle before saving if the file is sorted by anything meaningful.

Lifecycle#

ster vector train, ster vector optimize, and ster vector evaluate all take --pairs and load the set the same way. Each prompt of each pair is run through the model independently and its final-token activation captured per selected layer (reading pair 3/8 progress on stderr — one pair is two forward passes). Both sides go through whatever --chat-template selects, each as a lone user turn under auto, so the direction is fitted in the space it will be applied in (chat template). Training never modifies the set it read.

A set is also an output. ster pairs add appends one pair and creates the file and its missing parent directories when they do not exist yet; ster pairs remove drops one pair by its zero-based index; ster pairs synthesize writes a whole set generated by a model — the local one with --generator local, or a hosted one through Brama with --generator brama (synthetic pairs). All three write the document back as pretty-printed JSON with a trailing newline, and all three validate the set before touching disk, so a write that would leave a file the loader refuses is itself refused and the file on disk stays as it was — removing the last remaining pair is the case that bites.

ster pairs inspect [--pairs <PAIRS>] is the read-only counterpart: it audits a set — duplicates, refusal-reading sides, length balance, diversity — and prints a report without resolving or loading a model. For a desktop app the same code paths are the pairs/inspect, pairs/save, and pairs/synthesize operations of ster request (desktop request reference).

A set can also arrive from another product, and exactly one is implemented: Preferences, Wisent's pairwise voting tool. preferences export --arena <ARENA> --category <CATEGORY> --format ster-pairs writes a document --pairs accepts unchanged — every choice in a text category is already a contrastive pair — and ster-pairs is the only format its exporter will emit, refusing anything else with format "<VALUE>" is not supported; use ster-pairs. Nothing else in the product suite produces pair sets: every other set is one you wrote by hand, one ster pairs add built, or one ster pairs synthesize generated.

Invariants and refusals#

Checked at load (PairSet::load), before any model work:

The file must exist and be valid JSON: failed to read pair set <path> / invalid pair set JSON in <path>.
At least one pair: pair set <path> contains no pairs.
No side may be empty or whitespace-only: pair set <path> contains an empty positive or negative prompt.

Checked later, per prompt, by the runtime: every prompt must tokenize to at least one token (prompt must not be empty, tokenizer produced no tokens) and fit the model context. ster vector optimize additionally refuses sets smaller than four: optimization requires at least four contrastive pairs.

Writing pairs that work#

The learned direction is exactly the regularity that separates the two columns. Keep everything except the trait matched — topic, length, format — or the direction will encode the confound instead of the trait. The README's example trait (truthful) contrasts evidence-hedged answers with confident fabrications over the same questions; this corpus's executed toy trait (calm) contrasts calm and stormy sentences over the same scenery. Training accuracy near 0.5 means the pairs do not separate at that layer — see evaluation.

`ster pairs inspect` measures that advice instead of leaving it to judgement:

Duplicates. duplicate_count counts the pairs that repeat an earlier one — identical after normalization, or within the stated --dedupe-bits SimHash Hamming distance of it. Paraphrases weight one phrasing heavily and make the training margin look better than it is. First occurrence wins, so the pair flagged is the later of the two.
Refusals. refusal_count counts the pairs where either side reads as a refusal at or above the stated --refusal-threshold, with the scoring family and the matched snippet on the entry. A side that declines to answer differs from its partner along the refusal axis rather than the trait axis.
Length. length_ratio is the longer side over the shorter side in characters, and unbalanced_count counts the pairs above the stated --unbalanced-ratio, echoed as unbalanced_ratio — the length confound named above, made countable.
Diversity. unique_unigrams, unique_bigrams, avg_jaccard, mean_simhash_hamming, and min_simhash_hamming over the positive sides. Lower avg_jaccard and higher SimHash distances mean a varied set rather than one template with the nouns swapped.

None of it loads a model, so a set can be checked before a single forward pass is spent on it.

Importing existing sets#

A canonical import is JSON with optional trait_name and a non-empty pairs array; each element has exactly non-empty positive and negative strings. Unknown fields are rejected rather than silently dropped. The import validates the whole set before mutation, derives a stable logical name unless --name is supplied, and hashes normalized canonical content. The same content is unchanged and reactivated; the same name with different content is conflicting; malformed content is rejected; existing data is never overwritten.