Skip to content
Dataset By Humans

Task data and test suites for AI agents

Human-written multi-step tasks for AI agents with success checks, expected end states and grading rubrics. Built for agent training, evaluation and regression testing.

Free samples made to your spec, reply within 48 hours.

In short

AI agent task data is a set of realistic, multi-step tasks, each with a clear goal, the starting state and an automatic or rubric-based check that says whether the agent succeeded. We write task suites by hand, test that every task is solvable by a person, and keep them unpublished so they stay out of model training data.

What is in a task suite

  • A natural-language goal written the way a real user would ask
  • Starting state: files, accounts, web pages or tool descriptions
  • Expected end state or a success check you can automate
  • A grading rubric for partial credit and open-ended outputs
  • Difficulty, skill and tool labels (browse, code, spreadsheet, email, API)
  • A human reference run proving the task can be completed

Why public agent benchmarks are not enough

Public benchmarks leak into training data and saturate quickly. They also rarely reflect your product: your tools, your users' language, your edge cases. A private suite written for your agent measures what it really does and catches regressions before your users do.

Use cases

  • Held-out evaluation of browser, computer-use and tool-calling agents
  • Regression tests run on every model or prompt change
  • Training trajectories: human demonstrations of how to complete each task
  • Red-teaming tasks that check the agent refuses unsafe actions

Matching datasets

Start from a catalog dataset with free samples, or ask us to adapt one to your spec.

Tell us what you are building

Describe the data you need in two minutes. Within 48 hours we send samples made to your spec, a fixed price and a delivery date. You pay only after you approve the samples.

Request samples

Frequently asked questions

Do you provide the environment the agent runs in?

We write tasks against the environment you use (your app, a sandbox or public websites) and document the setup. We can also deliver tasks for standard tool sets described in plain JSON.

Can you record human demonstrations?

Yes. For training, people can complete each task and we deliver the step-by-step trajectory alongside the task.

How do you keep a task suite out of training data?

We never publish it, it is licensed under terms that forbid training the evaluated model on it, and an Exclusive license keeps it with you alone.

Last updated .