The data pyramid

Every source of robot training data trades quantity against fidelity — how close it is to the real robot doing the real task in the real world. The entire field can be read as an argument about how much of the expensive top layer you can avoid buying by getting cleverer about the cheap bottom layers.

The bottom layer contains no information about what a robot should do — it's just pixels of the world. The top layer is the only place where the observations, the physics, and the actions all come from the actual system being deployed. Everything in between borrows scale from below without losing too much truth from above.

Parts Two through Four now follow the pyramid tier by tier. Part Two covers the top: teleoperation and a robot's own deployment experience — both a real robot doing the real task, just collected at different moments. Part Three covers the middle: human video, a real body, but the wrong one. Part Four covers the bottom: simulation and widely available video, neither one a direct capture of anything — one imagined by a physics engine, the other never made with a robot in mind. Part Five pulls all four back together against the pyramid.

Driving a real robot

A human drives a real robot through a task while every sensor stream and motor command is recorded, synchronised in time. This is the gold standard: the only source where observations, physics, sensor noise and actions all come from the system that will actually be deployed. The technique is called teleoperation.

How a collection pipeline actually runs

What goes wrong

• Only successes get recorded. Operators naturally complete the task, so the model never sees a mistake or a recovery. This feeds directly into compounding error: small drift pushes the robot into situations the training data never covered, where its next guess is worse, and so on.
• Coverage is narrow by construction. You collect what you thought to script. The wet mug, the sticking drawer, the long tail of reality — precisely what's missing.
• Operator style leaks in. Two people solving the same task differently produces contradictory examples that confuse the model.

That's the pipeline. Here's the hardware that actually does the driving — four different ways to do it.

Four ways to drive one

Every rig solves the same problem — turning a human's motion into a command a robot can replay later. What separates them is how much translation sits in between. The closer a rig is to being a literal copy of the robot, the cleaner the recorded action; the further away, the cheaper and faster collection becomes.

Every rig in the last section captures data before deployment — a human driving, on purpose, to build a training set. This section covers what the robot generates entirely on its own, once it's already out working.

Learning on the job

Data the robot generates itself once it's out doing real work: its own successes, failures, near-misses, and every moment a human had to take over. That last one — an intervention — is unusually valuable, because it marks the exact point where the robot's judgment broke down.

Fleet scale is the point. Any single robot samples only a narrow slice of what deployment actually looks like, whereas a fleet spans different tasks, rooms, objects and instructions, producing edge cases no individual machine would ever encounter. The important recent shift is from harvesting demonstrations to running proper reinforcement learning across both the stored and the live data with one unified objective. In plain terms: instead of only imitating recorded successes, the policy is trained by trial and error — try an action, get a signal for how good the outcome was, adjust to make good outcomes more likely. That lets it learn something from a failure too, rather than discarding it as unusable, which is exactly what imitation on successes-only data can't do.

What goes wrong

• Nothing keeps score. There's no points counter in a kitchen. Detecting success automatically from video is unreliable, and having a human label it defeats the purpose.
• Safety caps exploration. The random flailing that makes trial-and-error work in simulation breaks things and hurts people in a real room.
• Variety runs out. Improvement stalls once the diversity of objects and environments plateaus — a robot deployed in one warehouse gets very good at that warehouse and learns nothing else.

Watching people instead

Video of people doing things, ideally filmed from their own point of view. Effectively unlimited supply, with real-world physics and visual variety thrown in free. The catch: no action labels exist anywhere in the footage, and the body doing the task is the wrong shape. That second problem has a name — the embodiment gap, the mismatch between a human hand with five soft fast fingers and a robot gripper with two rigid slow ones.

Why this isn't just teleoperation with a cheaper rig

The distinction is worth being precise about, because the hardware can be identical. A headset used for teleoperation is steering a real robot live, and what gets recorded is the robot's own resulting joint state. A headset used for this category is just a well-built camera strapped to someone's head — no robot in the room, no motor command produced anywhere.

A middle ground: handheld capture tools

Between passive video and a full teleoperation rig sits a handheld capture tool — imagine tongs with a camera mounted on them — that a person carries around and uses to do the task themselves, no robot involved. It's still human data, not robot data, so the embodiment gap doesn't go away. But the tool itself records something a camera alone can't: how far the gripper opened and closed, which is closer to an action label than raw video gets. It still needs a conversion step to translate a human hand's grip onto a robot's — just a smaller one.

Industrialising it

Capture like that doesn't stay a research setup — it scales into an actual company. The clearest example of this category run as an actual operation rather than a research convenience is Shift, built by the German embodied-AI lab MicroAGI. Cleaners wear camera headsets inside customers' homes, and the resulting footage is licensed to labs training household robots. Operators record everyday household and professional tasks for pay in New York and parts of Europe, with a viral free-cleaning offer in New York serving as both marketing and collection funnel. Several robotics labs are already buying data through the platform.

Figure has since pushed the same idea further with Index, an app that pays people to film their own everyday tasks by the minute, and can send a paid contributor to do a customer's chores on camera. By October 2026 it reported 28.5 million uploaded videos from 111 countries, and $35 million paid to contributors.

Encord runs it as a data business for other people's robots. It started out selling software for labelling images and video, and moved into producing training data once its customers began teaching robots manipulation end to end. It now collects in three ways: workers in factories around the world wear cameras on the job; trained operators at its warehouse in San Leandro, California, record specific skills such as pouring coffee or plugging in cables; and the same team drives leader-follower robot arms. Its argument is the one this whole guide makes: physical data can't be scraped, so someone has to manufacture it.

Encord is also adding a sense the camera lacks. Most of the muscles that move your fingers sit in your forearm, and their electrical activity can be read through the skin at the wrist, a technique called electromyography (EMG). In October 2026 it released EMG-1, the largest open dataset pairing wrist EMG with synchronised first-person video: 307 hours of real work recorded with wristbands on both wrists alongside head and wrist cameras, depth and motion sensors, plus a 10-hour lab set with motion capture to check accuracy against. The muscle signal keeps flowing when a hand disappears behind an object, which is exactly when cameras lose track of the fingers. In Encord's lab tests, a model combining the two, built by its wristband partner Wearable Devices, placed fingertips about a quarter more accurately than cameras alone, and held up best during the moments the cameras lost sight of the hand.

What still breaks

• The camera won't hold still. A human head moves constantly. Separating "the scene changed" from "I turned my head" is a genuine modelling problem, and it gets worse the more natural the footage.
• No sense of force. Video shows you geometry, never how hard someone squeezed. Muscle signals like EMG-1's are an early attempt to fill that gap, but EMG doesn't measure force directly; whether it can reliably show when a grip starts and how hard it squeezes is still being studied.
• Privacy is an operating cost. Recording inside real homes raises exactly the issues you'd expect, and consent and access controls become as much a part of the product as the sensors.

Practising in a fake world

Instead of a human or robot producing data in the real world, a computer program that pretends to be the real world produces it. You describe a table, a cup and an arm, and it calculates thousands of times per second what would happen if the arm moved this way or that. Free and instant — but fake, because the program has to guess how friction works and how objects bounce. That gap has a name: the sim-to-real gap.

Building the world

Every simulator starts from a description: the robot's body (its joints, motors and sensors), the scene around it, and ranges for the physics nobody can pin down exactly, such as friction, object mass and how hard a random push can be. In MuJoCo, for example, that description is an XML file. It can come from three places:

• Written by hand. An engineer models the robot and a few objects. Precise, but slow to scale beyond a handful of scenes.
• Generated. Software assembles scenes from parts. One collection spans 365 household tasks across 2,500 generated kitchen scenes.
• Scanned from reality. Film a real room and software builds a realistic 3D copy of it, a digital twin. The enabling technique is 3D Gaussian splatting — representing a scanned scene as millions of fuzzy coloured blobs that render photorealistically and fast — so high-fidelity environments can come straight from camera captures instead of an artist's hand. That's the thesis behind World Labs' acquisition of the simulation company SceniX: an engine that turns one physical task into many controllable, reusable worlds.

Two ways to train inside it

Once the world exists, a robot can learn in it in one of two ways, and either works in any of the worlds above.

Copy: multiply a few demonstrations.

The kitchen collection above works this way: over 2,200 hours of interaction data, about a quarter of it from human demonstrations and the rest generated. The shift is from "manually collect every recording" to "collect a small seed, then generate variations." The open question is whether those variations preserve the physical details that actually matter — contact, stability, friction, deformation, failure modes — or are simply a thousand visually different copies of the same physics mistake.

Practise: learn by doing. Skip demonstrations entirely and let the robot practise directly inside the simulator through trial and error, guided by a reward signal — the same reinforcement learning idea from Part Two, but far easier here, because a simulator can compute an exact reward on demand instead of guessing at one. A simple, concrete version: give the robot nothing but a high-level command like "walk forward," and let it learn the actual walking policy itself through repeated simulated attempts. A companion piece walks through exactly this case.

What goes wrong

• Contact and squishiness. Rigid blocks on rigid tables simulate well. Cloth, cable, liquid, and the moment a gripper's fingers actually start applying pressure do not.
• Generated video can lie about physics. A video-generating model will happily render an object floating or a hand passing through a table — visually seamless, physically impossible. A policy trained on it learns the mistake as fact. (How labs try to measure this is covered in World models, from the ground up.)
• Even the newest models struggle here. The first successful transfer of a world-action model trained purely in simulation to real manipulation reached 35% average success on lifting, drawer-opening and pick-and-place tasks — a genuine first, and a long way from reliable.

Simulation fakes the world; this technique fakes the label instead — starting from video that's completely real, just never meant to teach a robot anything.

Inventing the missing label

Human video, from two parts ago, was at least captured on purpose — someone strapped on a headset specifically to generate training data. There's a far larger pool that wasn't: home movies, cooking videos, sports footage, security-camera feeds — an internet's worth of people using their hands, none of it recorded with a robot in mind, and none of it carrying any record of what movements were made. The fix isn't to recover the real action — it's to invent a substitute and let the model define it. The technique is called a latent action model; "latent" just means learned but never directly labelled. (Demystifying AI models in robotics gives a shorter introduction to the same idea.)

The trick

The decoder never sees the answer directly — the only route for that information is through the code, and the code is far too small to carry a whole image. That size limit is the entire mechanism. The encoder is forced to throw away everything that stayed the same (the countertop, the wall, the lighting) and keep only what changed. In footage of someone doing a task, what changed is almost always the motion.

Once trained, the decoder is discarded. What's left — the encoder — is a tool that can watch any video and label what movement occurred, which is exactly what unlocks all that footage as training material in the first place.

What these discovered movements look like

Nobody tells the system what categories to find, but recognisable ones emerge anyway. On video game footage, the codes have naturally sorted themselves into things like move-left, move-right, jump, and do-nothing. On robot footage, they tend to separate into hand-opening, hand-closing, moving in a rough direction, wrist-turning, and staying still. They're coarse — think about thirty broad categories of motion, not precise measurements.

The hard part: translating into real movement

The practical fix works around most of this by splitting training into two stages.

Two stages

The coarseness of the code is exactly what makes this work across different bodies. It never tried to capture five-fingered dexterity, so it's abstract enough to be true of both a hand and a two-finger gripper. The grounding step fills in the body-specific detail.

Where the encoding itself goes wrong

The compression bottleneck captures any change, not just the one you care about. A camera pan, a flickering light, someone walking through the background — all of it competes for room in an already tiny code. This is the dominant failure mode: the representation entangles motion with appearance rather than separating them. Most current research targets exactly this, by masking out backgrounds, using the task description to steer attention toward relevant motion, or explicitly factoring out camera movement before the code is computed.

The latent action model trick narrows the embodiment gap computationally, after the fact. There's a second way to close it: build the robot's body to need less translation in the first place.

Building a body to fit the footage

The embodiment gap is why the trick above is needed at all: a five-fingered hand doing delicate manipulation doesn't map cleanly onto a two-finger parallel gripper, no matter how good the translation is. One response is computational — invent a code, ground it, accept the loss. A growing number of companies are betting on a different response: shrink the gap in hardware, by building the robot's body to already resemble the one in the footage.

Two arms and a human-shaped hand

Figure, 1X, Tesla and others have converged on a similar template: two arms roughly human in proportion, and — increasingly — hands with five independently articulated fingers rather than a simple two- or three-finger gripper. None of that is necessary for a robot confined to a fixed cell doing one task, where a purpose-built gripper usually wins outright. It matters specifically for a robot meant to learn from, and eventually work inside, spaces built around human bodies: human tools, human doorknobs, human stairs — and, most relevantly here, an internet's worth of video of human hands doing things.

Read against the pyramid, this is a bet that paying for hardware compatibility beats paying for top-tier data collection: match the body to video that already exists at effectively unlimited scale, and trade a smaller translation problem for a larger collection problem you never have to solve.

Every source across the last three parts has a distinct failure mode. That's exactly why no lab relies on just one.

Why nobody picks just one

No lab uses a single category. Mixing is a first-class problem rather than an implementation detail, because the sources fail in complementary ways: simulation supplies the robot actions that human video lacks, while human video supplies the real-world appearance that simulation struggles to render. Training on several at once, rather than one then the other, is called co-training.

The open questions here are unglamorous but decisive. What mixture ratio? Should sources be weighted by quality, by volume, or by how closely they resemble the intended deployment? Does the answer change as models get bigger? Right now these are tuned by hand, lab by lab, and mostly unpublished — there is no equivalent yet of the well-understood data-mixture literature that language models have.

The four, side by side

Read across any row and the same pattern holds: fidelity and scale trade against each other, tier by tier. Every technique in this guide is an attempt to buy back scale without giving up too much fidelity.

That table compares what each source costs to collect. Every source then pays a second cost before a model can learn from it: getting the data ready to train on.

Getting data ready for training

Raw recordings aren't training data. Whatever the source, they go through roughly the same three phases, and the middle one is what the industry calls annotation.

Preparing and converting are engineering: mostly automatic, and much the same whatever the source. Annotating is where the judgement lives. Models now draft most of the physical labels, such as hand poses and object positions, and people do what machines still get wrong: deciding where a task starts and stops, writing the instruction a robot would actually be given, and checking the edge cases. Timing runs through all three phases, because a label attached to the wrong instant is worse than no label at all.

How much annotation each source needs

recorded with the data partly recorded, the rest worked out afterwards added afterwards, by people or models

Among real sources, the pattern follows the pyramid: the higher the tier, the more labels arrive free at the moment of recording, and the lower the tier, the more have to be manufactured afterwards. Teleoperation gets the hardest label, the action, for nothing; human video has to work it out from hand motion; widely available video has to invent it. Simulation is the outlier. Every label is free and exact because the world itself was made up, so its cost is realism, not labelling.

That's why annotation has become a business in its own right. Encord, which began as labelling software, estimates that densely annotated task data is worth about 100 times as much as low-quality footage, at about 20 times the cost to produce. The value is in the labels, not the pixels.

Those are the costs on paper. Here's how five real companies are actually weighting them.

Who's betting on what

No two labs weight these four the same way, and the differences track their business models more than their research taste.

Glossary

Policy

The trained model that turns what a robot senses into what it should do — the thing all this training data ultimately exists to teach.

Reinforcement learning

Training a policy by trial and error rather than by copying demonstrations: try an action, get a signal for how good the outcome was, adjust to make good outcomes more likely. Needs a reward signal, which is easy to define in simulation and hard to define in the real world.

Action label

The recorded motor command at a given instant — what the robot was told to do, not merely what was seen.

Teleoperation

A human remotely driving a real robot while data is recorded.

Leader-follower

A cheap replica arm the human moves by hand; the real robot mirrors it, producing exact action labels for free.

Compounding error

Small mistakes push the robot into situations it never saw in training, where its next guess is worse, causing further drift.

Embodiment gap

The mismatch between a human body and a robot body that stops human demonstrations from being directly executable.

EMG (electromyography)

Reading the electrical activity of muscles through the skin. A wristband can pick up the forearm muscles that move the fingers, even when the hand is out of the camera's view.

Latent action model

A technique that invents a code for "what changed between two frames," used as a stand-in action label when no real one exists.

Sim-to-real gap

The accumulated difference between simulated and real physics, rendering and sensor behaviour.

3D Gaussian splatting

Representing a scanned real scene as millions of fuzzy coloured blobs — the current standard trick for making simulation look real.

Digital twin

A simulated replica of one specific real environment.

Intervention

A human taking over mid-task. Valuable because it marks the exact moment the policy failed.

Co-training

Training on several data sources at once rather than one after another, so each keeps informing the others.

Annotation

Adding labels to recorded data: where each task starts and stops, what is happening in words, where hands and objects are, and whether it worked. Models draft much of it; people check and correct.