Two very different jobs that share a name

People often use one label — VLA, short for "vision-language-action" — for two things that are actually doing opposite jobs. The difference comes down to a simple question: what does the model produce?

So why does everyone mix them up? Partly because both take an "action" as an input — a button press for the game, a movement command for the robot. Partly because they're built from very similar machinery underneath. But mostly because researchers are now deliberately combining them into single systems that do both jobs at once. There's even a name for these hybrids: World Action Models. So the blurring isn't purely a mistake — it's where the field is heading.

That's the outside view of a VLA — what it takes in, what it puts out. Here's what's actually happening between those two points.

What's actually inside a robot's brain

A VLA works as a loop that repeats many times per second:

The three inputs

• What it sees — one or more camera feeds, often including a camera on the wrist so it can see what its hand is doing.
• What it's been told — a plain-English instruction like "put the red block in the bin."
• Where its own body is — the current angle of every joint. This matters more than it sounds: you can't decide how to move your arm without knowing where your arm currently is.

Crucially, nobody builds this understanding from scratch. Researchers start with an existing AI model that already understands images and language from being trained on the internet, then teach it about robot movement on top. That inherited knowledge is why a robot can be told "pick up the fruit" and reach for a banana it's never seen before.

Planning a burst, not a step

Rather than deciding one tiny movement at a time, most robots plan a short sequence — roughly the next second of motion — and carry it out before checking back in. This turns out to be essential, for the reason in the next section.

Planning a burst rather than a single step, from the section above, is a workaround. Here's the problem it's working around.

The mismatch: slow thinking, fast hands

This is the constraint that shapes robot design more than almost anything else, and it's rarely mentioned in headlines.

The "thinking" part of a robot's brain is big and slow. It can form a fresh understanding of a scene maybe two to five times per second. But hands doing careful work — anything involving touch, grip, or balance — need updates ten to fifty times per second to stay steady. There's a gap, and it's a problem.

The fix most systems now use is a two-speed design: a slower part works out the overall plan a few times a second, while a smaller, much faster part handles the split-second adjustments in between. This mirrors how people work — you consciously decide "I'll pick up that mug," but the fine muscle coordination that actually does it runs faster and more automatically, without you thinking through every millimetre.

That faster part is where the term "policy," in its narrow, original sense, actually lives: a small, fast model — sometimes little more than straightforward maths — whose only job is turning the slow part's rough intention into a real stream of joint-by-joint motor commands, tens of times a second. It doesn't understand language or images at all; it just executes.

When a world model helps a robot

Every job an imagination does for a robot happens at one of two moments: while the robot is training, or in the instant it's deciding what to do.

Training-time uses are the practical win happening today; runtime, in-the-moment decision-making is the harder, less-solved half.

One of the training-time uses above — making up practice material — assumed there was footage to learn from in the first place. Here's where that footage can come from when nobody demonstrated the task on a robot at all.

Learning movement from video nobody labelled

Here's the puzzle. There's an enormous amount of video of people doing things with their hands. It would be extraordinarily useful for training robots — except none of it records what movements were made. You see the outcome, not the instruction.

The solution is a clever trick called a latent action model. "Latent" here just means hidden — it works out the hidden action that must have happened between one frame and the next.

The trick: feed two consecutive frames into an encoder that compresses "whatever changed" into a tiny code. A separate decoder gets only the first frame plus that code — never the real second frame — and has to predict it. Compare the guess to reality, adjust both, repeat millions of times. Because the code is too small to carry a whole image, the encoder is forced to throw away everything that stayed the same and keep only the motion.

Predicting pictures vs. predicting ideas

This is the deepest live disagreement in the field, and it's surprisingly easy to grasp.

Most world models predict what the world will look like — actual images. An alternative approach, championed by Yann LeCun and called JEPA, argues this is fundamentally wasteful. Instead of predicting pictures, it predicts a compressed summary of what's happening — the gist rather than the image.

This maps onto the three-way split from Part One: predicting pictures is the renderer approach; predicting the gist is closer to how a planner reasons. (The simulator camp — modelling physics directly rather than predicting either pixels or gists — is a third option this particular debate mostly sets aside.)

An example: a ball rolling behind a box

A red ball rolls across a table and disappears behind a box for a moment.

The claim is that this is exactly what a robot needs. To decide what to do, a robot doesn't need a perfect video of the future — it needs to know "if I push here, the cup tips over and spills." That's a fact about consequences, not appearances. It's also cheaper to work out and less easily confused by irrelevant visual changes like a shadow moving.

That's the abstract argument. Here's what it looks like as two companies, and roughly two billion dollars, betting opposite ways.

Two companies betting opposite ways

The two most prominent startups in this space disagree about precisely the question above — and both have raised enormous sums to prove their side.

Tellingly, Nvidia has invested in both — opposite philosophies, same cheque. Which is a fair signal that this genuinely isn't settled yet.

Who's building what

Robot brains (VLAs)

World models

Practice environments

These are the virtual worlds where robots do most of their learning:

• MuJoCo — the academic default, free and open. Especially good at simulating touch and contact, which makes it the standard for manipulation research. Nearly every open robot model works with it out of the box.
• Isaac Lab / Isaac Sim — NVIDIA's industrial-grade option. Much faster on their hardware, and the usual choice for training walking robots or generating simulated footage at scale.
• Genesis — a newer entrant that can also simulate soft materials and fluids, not just rigid objects.
• Gazebo, PyBullet, Drake, Webots — established general-purpose tools, still widely used.

Shared training data

Because collecting robot data is so expensive, the field pools it. The main collections:

• Open X-Embodiment — the big combined dataset spanning many different robot types. The usual starting point.
• DROID — one robot recorded across a huge variety of settings, which makes it especially useful for testing whether a robot copes with unfamiliar surroundings.
• RH20T — 110,000+ recordings across 7 robots and 147 tasks, strong on tasks involving touch and contact.
• RoboCasa — kitchen scenarios, and probably the closest thing the field has to a standard benchmark environment.
• LeRobot (Hugging Face) — not a dataset but a shared format and toolkit, an attempt to get everyone's robot data working together. Hundreds of community datasets use it, though quality varies enormously.

Glossary