Skip to content
Dataset By Humans
LLM training data

How to build a task suite to evaluate AI agents

A step-by-step method for writing agent tasks with start states, success checks and rubrics, plus the mistakes that make agent benchmarks meaningless.

By Dataset By HumansPublished 2 min read

In this article

Short answer: an agent task suite is a set of realistic multi-step tasks, each with a fixed starting state, a goal written the way a user would ask, and an automatic check of the end state that says whether the agent succeeded. Build it in this order: choose the skills to measure, freeze the environment, write tasks, write the checks, have a person solve every task, then keep the suite private.

Why you need your own suite

Public agent benchmarks such as WebArena, SWE-bench, GAIA and τ-bench are useful for comparing models, but they have two limits. They leak into training data over time, and they test someone else's tools and users. A private suite built around your product tells you whether your agent works, and catches regressions when you change a model or a prompt.

Anatomy of a good task

{
  "id": "task-0142",
  "goal": "Find the cheapest flight from Hanoi to Da Nang on 14 Nov and book it for Nguyen Van An.",
  "start_state": "demo-travel-site snapshot 2026-10-01, user logged in, no bookings",
  "allowed_tools": ["browser"],
  "constraints": ["Do not use saved payment methods"],
  "success_check": "bookings table has exactly one row: route HAN-DAD, date 2026-11-14, passenger 'Nguyen Van An', fare = min(fare)",
  "human_solution_steps": 9,
  "human_time_seconds": 140,
  "skills": ["search", "comparison", "form filling"],
  "difficulty": "medium"
}

Step 1: Decide what to measure

List the skills your users depend on: searching, comparing, filling forms, editing files, calling APIs, recovering from errors, refusing unsafe requests. Aim for coverage across skills and three difficulty levels rather than many near-identical tasks.

Step 2: Freeze the environment

Agents must start from the same state every run. Use a snapshot of a demo site, a seeded database or a container image. Live websites change and make results impossible to compare.

Step 3: Write goals like real users

Write the request the way a person would type it, including the vagueness and typos. Avoid giving away the steps; the agent should plan them.

Step 4: Check outcomes, not paths

Score the end state (a row in a database, a file's contents, an email sent) rather than the exact sequence of clicks. There is usually more than one correct path. For open-ended outputs such as a written summary, add a rubric with clear points a grader can apply.

Step 5: Have a person solve every task

A second person completes each task from the start state, noting steps and time. If they cannot finish it, or finish it in a way the check rejects, fix the task. This single step removes most broken benchmarks.

Step 6: Keep it private and versioned

Never publish the suite. Version it, record which model and prompt scored what, and add new tasks whenever you find a failure in production.

Metrics worth tracking

  • Success rate per skill and difficulty
  • Consistency: success across repeated runs of the same task (agents are not deterministic)
  • Cost and steps per successful task
  • Unsafe actions attempted, even in tasks the agent completed

Common mistakes

  • Checks that can be gamed (for example, only checking that a confirmation page appeared)
  • Tasks that depend on today's date or live prices
  • Too many easy tasks, so scores saturate quickly
  • Using the same tasks for training and evaluation

If you would rather not write hundreds of tasks yourself, we build task suites for AI agents with QA engineers, each solved by a second person before delivery.

task suitetask data for ai agentsagent evaluationagent benchmark