Skip to content
Dataset By Humans
LLM training data

SFT data vs preference data: what each one needs

Supervised fine-tuning teaches a model what to say; preference data teaches it what is better. What goes into each dataset, how big it should be and who should write it.

By Dataset By HumansPublished 2 min read

In this article

Short answer: SFT (supervised fine-tuning) data is a set of prompts with one ideal response each; it teaches a model what to say. Preference data pairs each prompt with two or more responses ranked by a person; it teaches the model which answer is better, and feeds methods such as RLHF and DPO. Most teams need a few thousand excellent SFT examples first, then preference data to polish behaviour.

How the two datasets differ

SFT dataPreference data
RecordPrompt + one ideal responsePrompt + chosen and rejected responses (or a ranking)
TeachesFormat, knowledge, style, task skillsJudgement: helpfulness, accuracy, safety, tone
Used bySupervised fine-tuningReward models (RLHF), DPO and similar methods
Who writesDomain experts write full answersRaters compare answers against guidelines
Typical sizeHundreds to tens of thousandsThousands to hundreds of thousands of comparisons
Main riskWeak or generated answers set a low ceilingInconsistent raters teach noise

What a good SFT record looks like

{
  "messages": [
    {"role": "user", "content": "Máy móc giá 120 triệu, dùng 5 năm, không có giá trị thu hồi. Khấu hao đường thẳng mỗi năm là bao nhiêu?"},
    {"role": "assistant", "content": "Mỗi năm khấu hao 24 triệu đồng. Theo phương pháp đường thẳng, khấu hao năm = (nguyên giá − giá trị thu hồi) / số năm sử dụng = (120 − 0) / 5 = 24 triệu..."}
  ],
  "domain": "accounting",
  "author_role": "certified accountant",
  "reviewed": true
}

The response is complete, correct, written in the voice you want, and checked by a second expert. Quality beats volume: the LIMA paper (Zhou et al., 2023) showed strong results from only about 1,000 carefully curated examples.

What a good preference record looks like

{
  "prompt": "Summarise this refund policy for a customer in two sentences.",
  "chosen": "You can return unused items within 30 days for a full refund...",
  "rejected": "Our refund policy is designed to ensure customer satisfaction and...",
  "reason": "Chosen answers the question directly and states the 30-day limit; rejected is vague.",
  "rater_role": "support team lead"
}

Good preference data has a short written reason, clear guidelines for what "better" means, and a sample of pairs rated twice so you can measure agreement between raters.

In what order should you collect them?

  1. SFT first. Teach the model your domain, format and tone with expert-written answers.
  2. Evaluate. Find where the fine-tuned model still fails.
  3. Preference data on the failures. Have raters compare the model's own answers on those prompts; that targets real weaknesses.
  4. Hold out a test set that never enters training, so you can tell whether each round helped.

Should people or models write the data?

Generating SFT answers with a larger model is fast, but your model inherits its errors and style, and the provider's terms of use may restrict it. AI-assisted rating is useful for scale, but guidelines and spot checks still need people. For domains where accuracy matters (law, tax, medicine, math) and for languages other than English, human-written data is usually the difference between a demo and a product.

We write both kinds with vetted professionals: see our human-authored SFT dataset creation service, and for chat assistants, conversational AI datasets.

sft datasetpreference datadporlhfhuman authored sft dataset