The data pyramid

Every source of robot training data trades quantity against fidelity — how close it is to the real robot doing the real task in the real world. The entire field can be read as an argument about how much of the expensive top layer you can avoid buying by getting cleverer about the cheap bottom layers.

The bottom layer contains no information about what a robot should do — it's just pixels of the world. The top layer is the only place where the observations, the physics, and the actions all come from the actual system being deployed. Everything in between borrows scale from below without losing too much truth from above.

Parts Two through Four now follow the pyramid tier by tier. Part Two covers the top: teleoperation and a robot's own deployment experience — both a real robot doing the real task, just collected at different moments. Part Three covers the middle: human video, a real body, but the wrong one. Part Four covers the bottom: simulation and widely available video, neither one a direct capture of anything — one imagined by a physics engine, the other never made with a robot in mind. Part Five pulls all four back together against the pyramid.

Driving a real robot

A human drives a real robot through a task while every sensor stream and motor command is recorded, synchronised in time. This is the gold standard: the only source where observations, physics, sensor noise and actions all come from the system that will actually be deployed. The technique is called teleoperation.

How a collection pipeline actually runs

What goes wrong

• Only successes get recorded. Operators naturally complete the task, so the model never sees a mistake or a recovery. This feeds directly into compounding error: small drift pushes the robot into situations the training data never covered, where its next guess is worse, and so on.
• Coverage is narrow by construction. You collect what you thought to script. The wet mug, the sticking drawer, the long tail of reality — precisely what's missing.
• Operator style leaks in. Two people solving the same task differently produces contradictory examples that confuse the model.

That's the pipeline. Here's the hardware that actually does the driving — four different ways to do it.

Four ways to drive one

Every rig solves the same problem — turning a human's motion into a command a robot can replay later. What separates them is how much translation sits in between. The closer a rig is to being a literal copy of the robot, the cleaner the recorded action; the further away, the cheaper and faster collection becomes.

Every rig in the last section captures data before deployment — a human driving, on purpose, to build a training set. This section covers what the robot generates entirely on its own, once it's already out working.

Learning on the job

Data the robot generates itself once it's out doing real work: its own successes, failures, near-misses, and every moment a human had to take over. That last one — an intervention — is unusually valuable, because it marks the exact point where the robot's judgment broke down.

Fleet scale is the point. Any single robot samples only a narrow slice of what deployment actually looks like, whereas a fleet spans different tasks, rooms, objects and instructions, producing edge cases no individual machine would ever encounter. The important recent shift is from harvesting demonstrations to running proper reinforcement learning across both the stored and the live data with one unified objective. In plain terms: instead of only imitating recorded successes, the policy is trained by trial and error — try an action, get a signal for how good the outcome was, adjust to make good outcomes more likely. That lets it learn something from a failure too, rather than discarding it as unusable, which is exactly what imitation on successes-only data can't do.

What goes wrong

• Nothing keeps score. There's no points counter in a kitchen. Detecting success automatically from video is unreliable, and having a human label it defeats the purpose.
• Safety caps exploration. The random flailing that makes trial-and-error work in simulation breaks things and hurts people in a real room.
• Variety runs out. Improvement stalls once the diversity of objects and environments plateaus — a robot deployed in one warehouse gets very good at that warehouse and learns nothing else.

Watching people instead

Video of people doing things, ideally filmed from their own point of view. Effectively unlimited supply, with real-world physics and visual variety thrown in free. The catch: no action labels exist anywhere in the footage, and the body doing the task is the wrong shape. That second problem has a name — the embodiment gap, the mismatch between a human hand with five soft fast fingers and a robot gripper with two rigid slow ones.

Why this isn't just teleoperation with a cheaper rig

The distinction is worth being precise about, because the hardware can be identical. A headset used for teleoperation is steering a real robot live, and what gets recorded is the robot's own resulting joint state. A headset used for this category is just a well-built camera strapped to someone's head — no robot in the room, no motor command produced anywhere.

A middle ground: handheld capture tools

Between passive video and a full teleoperation rig sits a handheld capture tool — imagine tongs with a camera mounted on them — that a person carries around and uses to do the task themselves, no robot involved. It's still human data, not robot data, so the embodiment gap doesn't go away. But the tool itself records something a camera alone can't: how far the gripper opened and closed, which is closer to an action label than raw video gets. It still needs a conversion step to translate a human hand's grip onto a robot's — just a smaller one.

Industrialising it

Capture like that doesn't stay a research setup — it scales into an actual company. The clearest example of this category run as an actual operation rather than a research convenience is Shift, built by the German embodied-AI lab MicroAGI. Cleaners wear camera headsets inside customers' homes, and the resulting footage is licensed to labs training household robots. Operators record everyday household and professional tasks for pay across fifteen countries, with a viral free-cleaning offer in New York serving as both marketing and collection funnel. Several robotics labs are already buying data through the platform.

What still breaks

• The camera won't hold still. A human head moves constantly. Separating "the scene changed" from "I turned my head" is a genuine modelling problem, and it gets worse the more natural the footage.
• No sense of force. Video shows you geometry, never how hard someone squeezed.
• Privacy is an operating cost. Recording inside real homes raises exactly the issues you'd expect, and consent and access controls become as much a part of the product as the sensors.

Practising in a fake world

Instead of a human or robot producing data in the real world, a computer program that pretends to be the real world produces it. You describe a table, a cup and an arm, and it calculates thousands of times per second what would happen if the arm moved this way or that. Free and instant — but fake, because the program has to guess how friction works and how objects bounce. That gap has a name: the sim-to-real gap.

Trick one — start real, then multiply

One such collection runs 365 household tasks across 2,500 kitchen scenes with over 2,000 hours of interaction data, mixing real and generated demonstrations. The shift is from "manually collect every recording" to "collect a small seed, then generate variations." The open question is whether those variations preserve the physical details that actually matter — contact, stability, friction, deformation, failure modes — or are simply a thousand visually different copies of the same physics mistake.

Trick two — scan reality, then simulate it

The enabling technique is 3D Gaussian splatting — representing a scanned scene as millions of fuzzy coloured blobs that render photorealistically and fast. It means high-fidelity environments can be built directly from camera captures rather than hand-modelled by an artist, which is the thesis behind World Labs' acquisition of the simulation company SceniX: an engine that turns one physical task into many controllable, reusable worlds.

Trick three — skip demonstrations, learn by doing

Both tricks above still start from recorded demonstrations and multiply them. A third approach skips demonstrations entirely: let the robot practice directly inside the simulator through trial and error, guided by a reward signal — the same reinforcement learning idea from Part Two, but far easier here, because a simulator can compute an exact reward on demand instead of guessing at one. A simple, concrete version: give the robot nothing but a high-level command like "walk forward," and let it learn the actual walking policy itself through repeated simulated attempts. A companion piece walks through exactly this case.

What goes wrong

• Contact and squishiness. Rigid blocks on rigid tables simulate well. Cloth, cable, liquid, and the moment a gripper's fingers actually start applying pressure do not.
• Generated video can lie about physics. A video-generating model will happily render an object floating or a hand passing through a table — visually seamless, physically impossible. A policy trained on it learns the mistake as fact.
• Even the newest models struggle here. The first successful transfer of a world-action model trained purely in simulation to real manipulation reached 35% average success across four tasks — a genuine first, and a long way from reliable.

Simulation fakes the world; this technique fakes the label instead — starting from video that's completely real, just never meant to teach a robot anything.

Inventing the missing label

Human video, from two parts ago, was at least captured on purpose — someone strapped on a headset specifically to generate training data. There's a far larger pool that wasn't: home movies, cooking videos, sports footage, security-camera feeds — an internet's worth of people using their hands, none of it recorded with a robot in mind, and none of it carrying any record of what movements were made. The fix isn't to recover the real action — it's to invent a substitute and let the model define it. The technique is called a latent action model; "latent" just means learned but never directly labelled.

The trick

The decoder never sees the answer directly — the only route for that information is through the code, and the code is far too small to carry a whole image. That size limit is the entire mechanism. The encoder is forced to throw away everything that stayed the same (the countertop, the wall, the lighting) and keep only what changed. In footage of someone doing a task, what changed is almost always the motion.

Once trained, the decoder is discarded. What's left — the encoder — is a tool that can watch any video and label what movement occurred, which is exactly what unlocks all that footage as training material in the first place.

What these discovered movements look like

Nobody tells the system what categories to find, but recognisable ones emerge anyway. On video game footage, the codes have naturally sorted themselves into things like move-left, move-right, jump, and do-nothing. On robot footage, they tend to separate into hand-opening, hand-closing, moving in a rough direction, wrist-turning, and staying still. They're coarse — think about thirty broad categories of motion, not precise measurements.

The hard part: translating into real movement

The practical fix works around most of this by splitting training into two stages.

Two stages

The coarseness of the code is exactly what makes this work across different bodies. It never tried to capture five-fingered dexterity, so it's abstract enough to be true of both a hand and a two-finger gripper. The grounding step fills in the body-specific detail.

Where the encoding itself goes wrong

The compression bottleneck captures any change, not just the one you care about. A camera pan, a flickering light, someone walking through the background — all of it competes for room in an already tiny code. This is the dominant failure mode: the representation entangles motion with appearance rather than separating them. Most current research targets exactly this, by masking out backgrounds, using the task description to steer attention toward relevant motion, or explicitly factoring out camera movement before the code is computed.

The latent action model trick narrows the embodiment gap computationally, after the fact. There's a second way to close it: build the robot's body to need less translation in the first place.

Building a body to fit the footage

The embodiment gap is why the trick above is needed at all: a five-fingered hand doing delicate manipulation doesn't map cleanly onto a two-finger parallel gripper, no matter how good the translation is. One response is computational — invent a code, ground it, accept the loss. A growing number of companies are betting on a different response: shrink the gap in hardware, by building the robot's body to already resemble the one in the footage.

Two arms and a human-shaped hand

Figure, 1X, Tesla and others have converged on a similar template: two arms roughly human in proportion, and — increasingly — hands with five independently articulated fingers rather than a simple two- or three-finger gripper. None of that is necessary for a robot confined to a fixed cell doing one task, where a purpose-built gripper usually wins outright. It matters specifically for a robot meant to learn from, and eventually work inside, spaces built around human bodies: human tools, human doorknobs, human stairs — and, most relevantly here, an internet's worth of video of human hands doing things.

Read against the pyramid, this is a bet that paying for hardware compatibility beats paying for top-tier data collection: match the body to video that already exists at effectively unlimited scale, and trade a smaller translation problem for a larger collection problem you never have to solve.

Every source across the last three parts has a distinct failure mode. That's exactly why no lab relies on just one.

Why nobody picks just one

No lab uses a single category. Mixing is a first-class problem rather than an implementation detail, because the sources fail in complementary ways: simulation supplies the robot actions that human video lacks, while human video supplies the real-world appearance that simulation struggles to render. Training on several at once, rather than one then the other, is called co-training.

The open questions here are unglamorous but decisive. What mixture ratio? Should sources be weighted by quality, by volume, or by how closely they resemble the intended deployment? Does the answer change as models get bigger? Right now these are tuned by hand, lab by lab, and mostly unpublished — there is no equivalent yet of the well-understood data-mixture literature that language models have.

The four, side by side

Read across any row and the same pattern holds: fidelity and scale trade against each other, tier by tier. Every technique in this guide is an attempt to buy back scale without giving up too much fidelity.

That's the abstract trade-off. Here's how five real companies are actually weighting it.

Who's betting on what

No two labs weight these four the same way, and the differences track their business models more than their research taste.

Glossary

Policy

The trained model that turns what a robot senses into what it should do — the thing all this training data ultimately exists to teach.

Reinforcement learning

Training a policy by trial and error rather than by copying demonstrations: try an action, get a signal for how good the outcome was, adjust to make good outcomes more likely. Needs a reward signal, which is easy to define in simulation and hard to define in the real world.

Action label

The recorded motor command at a given instant — what the robot was told to do, not merely what was seen.

Teleoperation

A human remotely driving a real robot while data is recorded.

Leader-follower

A cheap replica arm the human moves by hand; the real robot mirrors it, producing exact action labels for free.

Compounding error

Small mistakes push the robot into situations it never saw in training, where its next guess is worse, causing further drift.

Embodiment gap

The mismatch between a human body and a robot body that stops human demonstrations from being directly executable.

Latent action model

A technique that invents a code for "what changed between two frames," used as a stand-in action label when no real one exists.

Sim-to-real gap

The accumulated difference between simulated and real physics, rendering and sensor behaviour.

3D Gaussian splatting

Representing a scanned real scene as millions of fuzzy coloured blobs — the current standard trick for making simulation look real.

Digital twin

A simulated replica of one specific real environment.

Intervention

A human taking over mid-task. Valuable because it marks the exact moment the policy failed.

Co-training

Training on several data sources at once rather than one after another, so each keeps informing the others.