Skip to content
Dataset By Humans
Computer vision & video

First-person video datasets for robotics and embodied AI

What egocentric video is, the public datasets teams start with, where they fall short, and how to specify a first-person video collection of your own.

By Dataset By HumansPublished 2 min read

In this article

Short answer: first-person (egocentric) video is footage filmed from a camera worn on a person's head or chest, showing their hands and the objects they handle. It is the closest thing to a robot's point of view, so robotics and embodied-AI teams use it to learn manipulation and recognise activities. Public sets such as Ego4D and EPIC-KITCHENS are the usual starting point; production teams typically add their own collections for the tasks, homes and regions they care about.

Why first-person video matters for robots

Third-person video shows what happened; first-person video shows how it was done: where the hands go, which tool is picked up, what the person looks at before acting. That makes it valuable for:

  • imitation learning and pre-training manipulation policies
  • action recognition and step segmentation in long tasks
  • video-language models that answer "what should I do next?"
  • assistive technology that follows daily-living activities

Public datasets teams start with

DatasetWhat it isNotes
Ego4DAbout 3,670 hours of daily-life video from hundreds of camera wearers in several countriesBroad activities; access requires agreeing to its license
EPIC-KITCHENS-100About 100 hours of kitchen activity in 45 kitchens, densely annotatedKitchens only; check terms for commercial use
Ego-Exo4DSkilled activities filmed from first- and third-person views at onceGood for skill and pose research

Read each license carefully. Several large egocentric datasets were released primarily for research, and terms can differ between the video and the annotations.

Where public data falls short

  • Coverage: most footage comes from a limited set of countries, so kitchens, tools and habits elsewhere are under-represented.
  • Task fit: your robot may need many repetitions of a handful of tasks, not one example of thousands.
  • Annotation: labels may not match your action vocabulary.
  • Licensing: research terms rarely cover a commercial robot.

How to specify your own collection

  1. Task list: the exact tasks, how many repetitions, and acceptable variations.
  2. Camera setup: head or chest mount, resolution, frame rate and field of view. Wide-angle at 30–60 fps is common.
  3. Settings: number of homes, kitchen types, lighting, clutter levels.
  4. Participants: adults, paid, with signed consent covering AI training.
  5. Annotation: step segments with start and end times and a description per step; optionally hand-object labels.
  6. Privacy: faces of other people, documents and screens blurred; audio removed unless consented.
  7. Delivery: clips, metadata (device, fps, duration, setting), a datasheet and the license.

Start with a pilot of a few hours across several homes, check that the footage and labels work in your pipeline, then scale.

Our first-person video

We film first-person household task video in Southeast Asian homes, with step-level descriptions, and street and traffic video for perception. See video training data for what every delivery includes.

first person video training dataegocentric video datasetvideo training datarobotics