Ster/Docs

ster pairs import

Write a pair set from a published benchmark export: TruthfulQA, Do-Not-Answer, LiveCodeBench, a BiPO behaviour CSV, multiple-choice rows or preference rows.

Invocation and purpose#

text
ster pairs import --benchmark truthfulqa|dna|livecodebench|bipo|choices|preference --source <FILE> --output <FILE>
                  [--examples <FILE>] [--count <N>] --seed <SEED> [--trait <NAME>]
                  [--question <POINTER> --choices <POINTER> --answer <POINTER>
                   --answer-form index|label|text [--labels <POINTER>]]

Every row becomes one pair written as `ster pairs synthesize` writes one, Question: … then Answer: … on both sides. No model is loaded.

What each benchmark becomes#

  • `truthfulqa` (TruthfulQA_en.csv): the row's Best Answer against one of its Incorrect Answers, picked with --seed.
  • `dna` (Do-Not-Answer, DnA_en.csv): questions only; each is paired as "I am happy to help!" (positive) against "I am sorry, I cannot comply with this request" (negative).
  • `livecodebench` (problems.json): each problem's question_content with the good and bad example code in the question_examples.json that --examples names; without --examples the import refuses.
  • `bipo`: the BiPO paper's behaviour CSVs (question,matching,not_matching, such as its power-seeking/train.csv), the matching answer (positive) against the other (negative) — the set ster tune bipo learns a vector from.
  • `choices`: multiple-choice rows of any dataset (ARC, HellaSwag, MMLU and the like) as JSON Lines or one JSON list. --question, --choices and --answer are JSON pointers to the row's parts, and --answer-form says how the answer names its choice: index (from zero), label (one of the labels --labels points to, aligned with the choices) or text (the choice itself). The correct choice is the positive side, against one of the others drawn with --seed.
  • `preference`: prompt, chosen and rejected rows, as TRL's DPO trainer and the Hugging Face preference datasets publish them, as JSON Lines or one JSON list. The chosen answer is the positive side and the rejected one the negative. A side in the conversational form (a list of {"role", "content"} messages) is its last message's content, and with no prompt the messages before it are the question.

Inputs and options#

InputRequiredMeaning
--benchmark <NAME>yesWhich export --source is: truthfulqa, dna, livecodebench, bipo, choices or preference.
--source <FILE>yesThe export file.
--examples <FILE>with livecodebenchThe question_examples.json holding each question's good and bad example code; refused with any other benchmark.
--question, --choices, --answer, --answer-form, --labelswith choicesWhere each part of a row sits (JSON pointers starting with /) and how the answer names its choice; --labels only with --answer-form label. Refused with any other benchmark.
--count <N>noKeep this many pairs, drawn with --seed; every pair when omitted. Zero is refused.
--seed <SEED>yesSeed for the drawn pairs, TruthfulQA's pick among incorrect answers and the other choice of a multiple-choice row; Ster assumes none.
--trait <NAME>noThe trait_name written on the set; the benchmark name when omitted.
--output <FILE>yesPair-set JSON the pairs are written to.

Output and state#

Writes the set as pretty-printed JSON with a trailing newline. Stdout is a pretty JSON document with output, trait_name, and report. A row that cannot become a pair is listed under report.skipped with its row (its record number, its line in JSON Lines, its position from one in a list, or problem <n> for a LiveCodeBench problem without a question_id), its reason, naming the column or pointer and what it held, and its cause of ster.pairs.import.rows.

Refusals#

  • Before the export is read (exit 2): --count 0; --examples with any benchmark but livecodebench; the row flags with any benchmark but choices (--question, --choices, --answer, --answer-form and --labels apply only to --benchmark choices); choices without them (--benchmark choices needs --question, --choices, --answer and --answer-form: where each part of a row sits) or without one of them; a pointer that does not start with /; --answer-form label without --labels, and --labels with any other form.
  • Rows skipped under ster.pairs.import.rows: question-blank, answer-blank, incorrect-none, problem-id-missing, problem-content-blank, example-missing, example-code-missing; for choices choice-question-missing, choices-not-list, choice-not-text, choices-few, answer-missing, answer-not-index, answer-index-outside, labels-not-list, label-not-text, labels-misaligned, answer-label-not-text, answer-label-unknown, answer-text-not-text and answer-text-unknown; for preference preference-side-missing.
  • An export no row of which becomes a pair is refused with its first skipped row's cause and fix, followed by how many rows were skipped under each cause. An unreadable file, a CSV that is not the benchmark's (a missing column, a record whose field count differs from its header, a quoted field never closed), a file that is neither a JSON list nor JSON Lines (naming the line that is not JSON), an export that holds no row (no-pairs), or a --count above the pairs the export yields refuses the import and writes nothing.