Chat template

Overview#

Every tokenizer path in Ster before 0.13 treated text as a bare completion prompt. That is exactly right for a base model and wrong for every instruct checkpoint published in the last two years. Those models were post-trained on a conversation wrapped in special markers, and the exact wrapping travels with the weights: a Jinja2 template in tokenizer_config.json under chat_template, or, on repositories published since mid-2025, in a standalone chat_template.jinja beside it. The standalone file wins when both exist, which is the precedence transformers itself uses.

Fine-tuning such a model on untemplated prompts and completions teaches it a format it will never be prompted in, and generating from it without the template is what produces the rambling continuations that make people conclude a checkpoint is broken. Neither failure raises anything: the loss falls, the run reports numbers, and the adapter is wrong. --chat-template exists so that the shape a run trains in is a decision the operator makes and the report records, rather than an accident of which tokenizer path the text happened to take.

auto, off, and why there is no on#

--chat-template on the CLI and chatTemplate in a `ster request` body take two values and default to auto:

• auto applies the checkpoint's own template when it publishes one, and encodes raw text when it does not.
• off encodes raw text always — the behaviour of every Ster release before 0.13, byte for byte, and what a base model wants.

There is no on. A checkpoint that publishes no template has no conversation format to apply, and an operator who asked for on could only ever be handed a refusal; guessing a format on their behalf would be the silent-wrong-data failure this feature exists to remove. So auto says which of the two things it did, in one sentence on the progress stream, once per run:

• applying the model's own chat template to every prompt and completion
• this model publishes no chat template, so prompts and completions are encoded as raw text
• chat template off, so prompts and completions are encoded as raw text

The same decision lands in the run's report as chat_template, one of applied, absent, or off. That report is folded into the adapter artifact's sidecar, so an adapter carries the shape it was trained in and an operator months later does not have to guess which encoding produced it. A steering artifact is itself a JSON document and carries the same word at its top level, in the same three spellings, so one comparison reads either kind. unknown chat template mode "<VALUE>"; expected auto or off refuses anything else.

ster vector evaluate and ster generate read that record and warn when it disagrees with the run doing the reading — warning: this direction was trained with chat template absent and this run encodes off, so the number below describes a format it was not trained in — the same helper the tune half uses on an adapter. A warning and not a refusal: reading a direction in a space it was not fitted in is a real question to ask, and a refusal would make the experiment impossible rather than merely deliberate. Only an unnoticed mismatch is a defect.

Where the completion boundary comes from#

A supervised example is a prompt and a completion, and the loss must cover the completion alone. Untemplated, the boundary is simply how many tokens the prompt made. Templated, it cannot be: the prompt is wrapped in markers, and a boundary measured on the raw prompt is short by their length, so the model would be scored on producing its own header.

Ster therefore renders twice and subtracts. The first render is the message list [{role: user}] with the generation prompt appended — the conversation cut off exactly where the assistant begins to speak. The second is [{role: user}, {role: assistant}] with no generation prompt — the whole conversation. The completion half is the difference between the two strings, which is the completion plus whatever the template put after it, normally the turn's end marker. That marker belongs inside the loss window: the model has to learn to stop.

Neither half is tokenized with the tokenizer's own special tokens. The rendered string already spells every marker the model expects, so asking the tokenizer to add its begin-of-sequence again would place a second one mid-sequence — a token the model is asked to predict and never sees at inference. This is what apply_chat_template(tokenize=True) does in transformers too.

A worked example, on the toy checkpoint with the fixture template from `examples/chat-template/tokenizer_config.json` copied beside it. The prompt is describe the evening lake . and the completion the lake surface is like a mirror .:

prompt render : <s>question : describe the evening lake . answer :
full render   : <s>question : describe the evening lake . answer : the lake surface is like a mirror . </s>
completion    :  the lake surface is like a mirror . </s>

ids      [1, 52, 65, 53, 3, 54, 55, 63, 51, 65, 3, 55, 56, 5, 58, 48, 57, 63, 2]
boundary 10   total 19

same example with --chat-template off
ids      [53, 3, 54, 55, 63, 3, 55, 56, 5, 58, 48, 57, 63]
boundary 5    total 13

Both are observable from outside: ster tune evaluate reports completion_tokens of [9, 9, 9, 9, 10, 9, 9, 8] for the eight fixture examples under applied against [8, 8, 8, 8, 9, 8, 8, 7] under off — one more token each, the </s> — and sweeping --max-sequence puts the all-pass threshold at 19 templated against 13 raw.

Preference pairs render as one assistant turn#

A contrastive pair carries no prompt: both sides are complete responses, which is why they go through a different encoder than a supervised example. Under a template each side is rendered as a lone assistant turn, so preference optimization and a reward model compare exactly the shape supervised fine-tuning trains, wearing the markers inference will put around it. The markers are common to both sides, so they cancel in the log-ratio the preference loss takes; what changes is that the sequence being scored is one the model has actually seen before.

A template that forbids a conversation opening on the assistant — several enforce strict user/assistant alternation — raises rather than being worked around, and the run stops with this model's chat template failed to render: <ERROR> carrying the template author's own message. Inventing a user turn to satisfy it would put words into the operator's data that the operator did not write; --chat-template off is the answer when that is what is wanted.

Rendering, and what is refused#

Rendering uses MiniJinja, a pure-Rust Jinja2 engine, and it is a dependency rather than a hand-written subset on purpose. Real templates use loops, tests, filters and exceptions, and a renderer that understands most of that fails silently into plausible-looking, wrongly-marked training data — strictly worse than an error, because nothing about the resulting run looks wrong. On top of stock Jinja, Ster adds only what Hugging Face's own renderer adds: the raise_exception and strftime_now globals, the Python string and mapping methods (strip, split, startswith, replace, items, and their neighbours) that templates call as methods rather than filters, and the {% generation %} … {% endgeneration %} block that marks assistant tokens for Transformers' return_assistant_tokens_mask and otherwise renders its body unchanged (Arcee Trinity's template uses it). Anything outside that raises, naming the template's own line. A template that does not compile is refused only by a run under --chat-template auto; under off the checkpoint loads and runs on raw text whatever its template says.

The refusals are all fail-closed and all raised while the operator is still waiting on the load, not halfway through an epoch:

• this model's chat template does not parse: <ERROR>
• this model's chat template failed to render: <ERROR>
• this model's chat template uses <TOKEN>, but its tokenizer config declares none
• this model's chat template does not render the assistant turn after the prompt, so the completion boundary cannot be located
• this model's chat template rendered an empty assistant turn
• this model publishes several named chat templates and none of them is the default
• this model's chat template list has an entry with no template
• this model's chat_template is neither a template nor a list of them

The third is the one worth explaining. A template that writes bos_token into its output and is handed no value for it renders a sequence with no begin-of-sequence marker at all — every position shifted by one, under a loss that still looks entirely reasonable. That is precisely the failure mode this feature exists to prevent, so a missing token is a refusal rather than an empty string.

What fitting out of format costs#

A direction is a displacement between two points in the residual stream, and where those points sit depends on the markers around the text that produced them. That is the argument for putting the flag on a training command, and it is the kind of argument that should be measured rather than asserted.

Measured on TinyLlama-1.1B-Chat-v1.0 at f32 over all 22 layers, with a four-pair set for the trait calm and measured, never alarmed. One direction was fitted with --chat-template off and another with auto, and each was then evaluated under both settings — four runs, each reporting accuracy and margin at every layer:

Read it in two halves. Matched — the first two rows — the direction separates all four pairs at every one of the 22 layers, whichever format it was fitted in, so this is not a quality setting; the two matched margins are not comparable to each other either, because they are projections measured in different spaces. Crossed, off fitted and auto evaluated falls below accuracy 1.0 at 12 of the 22 layers, and the margin collapses by roughly two orders of magnitude: at layer 21 the matched off direction scores +20.6697 and the same direction read under auto scores -0.0486.

The sign is the part worth pausing on. In the deepest layers the crossed margin goes negative — layers 19, 20 and 21, in both crossed directions — which means the direction points the wrong way there. A negative margin is not a weak direction, it is an inverted one: added during generation it pushes toward the trait's opposite. Nothing about the artifact or the run used to say so, which is why the default is auto on the steering half too, and why the artifact now records which setting fitted it.

Where it applies#

The steering half took the flag last, and later than it should have. Runtime::activations — the one read behind train, optimize, evaluate and extract — now encodes through the template like every other tokenizer path, so a direction is fitted in the space it will be applied in. ster tune merge, ster tune inspect and ster vector inspect read no prose at all and take no flag; pairs synthesize --generator brama accepts it and ignores it, because the gateway is already a chat API. Artifacts written before this existed are unaffected: they record chat_template as null and nothing treats that as a mismatch.

The toy checkpoint publishes no chat template, which makes it the test for the fallback rather than a gap in the fixtures: under auto it reports absent and produces ids identical to off. Note that absent and off are still different words in the record, and ster vector evaluate warns across them, because they mean different things about the run even where they mean the same tokens. To exercise the applied path offline, copy `examples/chat-template/tokenizer_config.json` beside a copy of the toy model — it is a template written in the toy's own vocabulary — and train on `examples/chat-examples.json`, whose prompts and completions are bare so that the markers come from the template rather than from the data.