In this article
Short answer: SFT (supervised fine-tuning) data is a set of prompts with one ideal response each; it teaches a model what to say. Preference data pairs each prompt with two or more responses ranked by a person; it teaches the model which answer is better, and feeds methods such as RLHF and DPO. Most teams need a few thousand excellent SFT examples first, then preference data to polish behaviour.
How the two datasets differ
| SFT data | Preference data | |
|---|---|---|
| Record | Prompt + one ideal response | Prompt + chosen and rejected responses (or a ranking) |
| Teaches | Format, knowledge, style, task skills | Judgement: helpfulness, accuracy, safety, tone |
| Used by | Supervised fine-tuning | Reward models (RLHF), DPO and similar methods |
| Who writes | Domain experts write full answers | Raters compare answers against guidelines |
| Typical size | Hundreds to tens of thousands | Thousands to hundreds of thousands of comparisons |
| Main risk | Weak or generated answers set a low ceiling | Inconsistent raters teach noise |
What a good SFT record looks like
{
"messages": [
{"role": "user", "content": "Máy móc giá 120 triệu, dùng 5 năm, không có giá trị thu hồi. Khấu hao đường thẳng mỗi năm là bao nhiêu?"},
{"role": "assistant", "content": "Mỗi năm khấu hao 24 triệu đồng. Theo phương pháp đường thẳng, khấu hao năm = (nguyên giá − giá trị thu hồi) / số năm sử dụng = (120 − 0) / 5 = 24 triệu..."}
],
"domain": "accounting",
"author_role": "certified accountant",
"reviewed": true
}
The response is complete, correct, written in the voice you want, and checked by a second expert. Quality beats volume: the LIMA paper (Zhou et al., 2023) showed strong results from only about 1,000 carefully curated examples.
What a good preference record looks like
{
"prompt": "Summarise this refund policy for a customer in two sentences.",
"chosen": "You can return unused items within 30 days for a full refund...",
"rejected": "Our refund policy is designed to ensure customer satisfaction and...",
"reason": "Chosen answers the question directly and states the 30-day limit; rejected is vague.",
"rater_role": "support team lead"
}
Good preference data has a short written reason, clear guidelines for what "better" means, and a sample of pairs rated twice so you can measure agreement between raters.
In what order should you collect them?
- SFT first. Teach the model your domain, format and tone with expert-written answers.
- Evaluate. Find where the fine-tuned model still fails.
- Preference data on the failures. Have raters compare the model's own answers on those prompts; that targets real weaknesses.
- Hold out a test set that never enters training, so you can tell whether each round helped.
Should people or models write the data?
Generating SFT answers with a larger model is fast, but your model inherits its errors and style, and the provider's terms of use may restrict it. AI-assisted rating is useful for scale, but guidelines and spot checks still need people. For domains where accuracy matters (law, tax, medicine, math) and for languages other than English, human-written data is usually the difference between a demo and a product.
We write both kinds with vetted professionals: see our human-authored SFT dataset creation service, and for chat assistants, conversational AI datasets.
sft datasetpreference datadporlhfhuman authored sft dataset