ster pairs import
Write a pair set from a published benchmark export: TruthfulQA, Do-Not-Answer, LiveCodeBench, a BiPO behaviour CSV, multiple-choice rows or preference rows.
Invocation and purpose#
ster pairs import --benchmark truthfulqa|dna|livecodebench|bipo|choices|preference --source <FILE> --output <FILE>
[--examples <FILE>] [--count <N>] --seed <SEED> [--trait <NAME>]
[--question <POINTER> --choices <POINTER> --answer <POINTER>
--answer-form index|label|text [--labels <POINTER>]]Every row becomes one pair written as `ster pairs synthesize` writes one, Question: … then Answer: … on both sides. No model is loaded.
What each benchmark becomes#
- `truthfulqa` (
TruthfulQA_en.csv): the row's Best Answer against one of its Incorrect Answers, picked with--seed. - `dna` (Do-Not-Answer,
DnA_en.csv): questions only; each is paired as "I am happy to help!" (positive) against "I am sorry, I cannot comply with this request" (negative). - `livecodebench` (
problems.json): each problem'squestion_contentwith the good and bad example code in thequestion_examples.jsonthat--examplesnames; without--examplesthe import refuses. - `bipo`: the BiPO paper's behaviour CSVs (
question,matching,not_matching, such as itspower-seeking/train.csv), the matching answer (positive) against the other (negative) — the setster tune bipolearns a vector from. - `choices`: multiple-choice rows of any dataset (ARC, HellaSwag, MMLU and the like) as JSON Lines or one JSON list.
--question,--choicesand--answerare JSON pointers to the row's parts, and--answer-formsays how the answer names its choice:index(from zero),label(one of the labels--labelspoints to, aligned with the choices) ortext(the choice itself). The correct choice is the positive side, against one of the others drawn with--seed. - `preference`:
prompt,chosenandrejectedrows, as TRL's DPO trainer and the Hugging Face preference datasets publish them, as JSON Lines or one JSON list. The chosen answer is the positive side and the rejected one the negative. A side in the conversational form (a list of{"role", "content"}messages) is its last message's content, and with nopromptthe messages before it are the question.
Inputs and options#
| Input | Required | Meaning |
|---|---|---|
--benchmark <NAME> | yes | Which export --source is: truthfulqa, dna, livecodebench, bipo, choices or preference. |
--source <FILE> | yes | The export file. |
--examples <FILE> | with livecodebench | The question_examples.json holding each question's good and bad example code; refused with any other benchmark. |
--question, --choices, --answer, --answer-form, --labels | with choices | Where each part of a row sits (JSON pointers starting with /) and how the answer names its choice; --labels only with --answer-form label. Refused with any other benchmark. |
--count <N> | no | Keep this many pairs, drawn with --seed; every pair when omitted. Zero is refused. |
--seed <SEED> | yes | Seed for the drawn pairs, TruthfulQA's pick among incorrect answers and the other choice of a multiple-choice row; Ster assumes none. |
--trait <NAME> | no | The trait_name written on the set; the benchmark name when omitted. |
--output <FILE> | yes | Pair-set JSON the pairs are written to. |
Output and state#
Writes the set as pretty-printed JSON with a trailing newline. Stdout is a pretty JSON document with output, trait_name, and report. A row that cannot become a pair is listed under report.skipped with its row (its record number, its line in JSON Lines, its position from one in a list, or problem <n> for a LiveCodeBench problem without a question_id), its reason, naming the column or pointer and what it held, and its cause of ster.pairs.import.rows.
Refusals#
- Before the export is read (exit 2):
--count 0;--exampleswith any benchmark butlivecodebench; the row flags with any benchmark butchoices(--question, --choices, --answer, --answer-form and --labels apply only to --benchmark choices);choiceswithout them (--benchmark choices needs --question, --choices, --answer and --answer-form: where each part of a row sits) or without one of them; a pointer that does not start with/;--answer-form labelwithout--labels, and--labelswith any other form. - Rows skipped under
ster.pairs.import.rows:question-blank,answer-blank,incorrect-none,problem-id-missing,problem-content-blank,example-missing,example-code-missing; forchoiceschoice-question-missing,choices-not-list,choice-not-text,choices-few,answer-missing,answer-not-index,answer-index-outside,labels-not-list,label-not-text,labels-misaligned,answer-label-not-text,answer-label-unknown,answer-text-not-textandanswer-text-unknown; forpreferencepreference-side-missing. - An export no row of which becomes a pair is refused with its first skipped row's cause and fix, followed by how many rows were skipped under each cause. An unreadable file, a CSV that is not the benchmark's (a missing column, a record whose field count differs from its header, a quoted field never closed), a file that is neither a JSON list nor JSON Lines (naming the line that is not JSON), an export that holds no row (
no-pairs), or a--countabove the pairs the export yields refuses the import and writes nothing.