Two very different jobs that share a name

People often use one label — VLA, short for "vision-language-action" — for two things that are actually doing opposite jobs. The difference comes down to a simple question: what does the model produce?

So why does everyone mix them up? Partly because both take an "action" as an input — a button press for the game, a movement command for the robot. Partly because they're built from very similar machinery underneath. But mostly because researchers are now deliberately combining them into single systems that do both jobs at once. There's even a name for these hybrids: World Action Models. So the blurring isn't purely a mistake — it's where the field is heading.

What's actually inside a robot's brain

A VLA works as a loop that repeats many times per second:

The three inputs

• What it sees — one or more camera feeds, often including a camera on the wrist so it can see what its hand is doing.
• What it's been told — a plain-English instruction like "put the red block in the bin."
• Where its own body is — the current angle of every joint. This matters more than it sounds: you can't decide how to move your arm without knowing where your arm currently is.

Crucially, nobody builds this understanding from scratch. Researchers start with an existing AI model that already understands images and language from being trained on the internet, then teach it about robot movement on top. That inherited knowledge is why a robot can be told "pick up the fruit" and reach for a banana it's never seen before.

Turning understanding into movement: two approaches

Planning a burst, not a step

Rather than deciding one tiny movement at a time, most robots plan a short sequence — roughly the next second of motion — and carry it out before checking back in. This turns out to be essential, for the reason in the next section.

The mismatch: slow thinking, fast hands

This is the constraint that shapes robot design more than almost anything else, and it's rarely mentioned in headlines.

The "thinking" part of a robot's brain is big and slow. It can form a fresh understanding of a scene maybe two to five times per second. But hands doing careful work — anything involving touch, grip, or balance — need updates ten to fifty times per second to stay steady. There's a gap, and it's a problem.

The fix most systems now use is a two-speed design: a slower part works out the overall plan a few times a second, while a smaller, much faster part handles the split-second adjustments in between. This mirrors how people work — you consciously decide "I'll pick up that mug," but the fine muscle coordination that actually does it runs faster and more automatically, without you thinking through every millimetre.

Four ways a world model helps a robot

These are genuinely different jobs, even though the same underlying model might do more than one of them.

Numbers 2 and 4 are largely about making the imagination trustworthy enough that 1 and 3 can be relied on. Right now, number 1 is the practical win happening today; 3 and 4 are the harder, less-solved prize.

Telling the similar ones apart

Two pairs above sound alike. Both distinctions matter, and both come down to when the imagining happens and what it decides.

"Think before acting" vs. "check the work"

"Make up practice material" vs. "check the work"

These get used together so routinely they can look like one step. But making things up is about quantity — more situations than anyone could demonstrate by hand. Checking is about quality, and it will judge anything you hand it, no matter where it came from: made-up footage, real recordings, or the robot's own attempts during practice. It's a separate job that happens to be useful downstream of the first one.

You can skip the checking — early systems did. It's just risky, because generated footage sometimes shows things that are physically impossible, and a robot that learns from impossible examples learns nonsense.

Learning movement from video nobody labelled

Here's the puzzle. There's an enormous amount of video of people doing things with their hands. It would be extraordinarily useful for training robots — except none of it records what movements were made. You see the outcome, not the instruction.

The solution is a clever trick called a latent action model. "Latent" here just means hidden — it works out the hidden action that must have happened between one frame and the next.

How it works, step by step

Take two consecutive frames from unlabelled video: a robot hand 10cm from a cup, then the same hand 5cm from the cup. Nobody recorded what caused that.

Once trained, the second part is discarded. The first part is now a tool that can watch any video and label what movement occurred — which unlocks all that unlabelled human footage as robot training material.

What these discovered movements look like

Nobody tells the system what categories to find, but recognisable ones emerge anyway. On video game footage, the codes have naturally sorted themselves into things like move-left, move-right, jump, and do-nothing. On robot footage, they tend to separate into hand-opening, hand-closing, moving in a rough direction, wrist-turning, and staying still. They're coarse — think about thirty broad categories of motion, not precise measurements.

Predicting pictures vs. predicting ideas

This is the deepest live disagreement in the field, and it's surprisingly easy to grasp.

Most world models predict what the world will look like — actual images. An alternative approach, championed by Yann LeCun and called JEPA, argues this is fundamentally wasteful. Instead of predicting pictures, it predicts a compressed summary of what's happening — the gist rather than the image.

An example: a ball rolling behind a box

A red ball rolls across a table and disappears behind a box for a moment.

The claim is that this is exactly what a robot needs. To decide what to do, a robot doesn't need a perfect video of the future — it needs to know "if I push here, the cup tips over and spills." That's a fact about consequences, not appearances. It's also cheaper to work out and less easily confused by irrelevant visual changes like a shadow moving.

Two companies betting opposite ways

The two most prominent startups in this space disagree about precisely the question above — and both have raised enormous sums to prove their side.

Tellingly, Nvidia has invested in both — opposite philosophies, same cheque. Which is a fair signal that this genuinely isn't settled yet.

Who's building what

Robot brains (VLAs)

World models

Practice environments

These are the virtual worlds where robots do most of their learning:

• MuJoCo — the academic default, free and open. Especially good at simulating touch and contact, which makes it the standard for manipulation research. Nearly every open robot model works with it out of the box.
• Isaac Lab / Isaac Sim — NVIDIA's industrial-grade option. Much faster on their hardware, and the usual choice for training walking robots or generating simulated footage at scale.
• Genesis — a newer entrant that can also simulate soft materials and fluids, not just rigid objects.
• Gazebo, PyBullet, Drake, Webots — established general-purpose tools, still widely used.

Shared training data

Because collecting robot data is so expensive, the field pools it. The main collections:

• Open X-Embodiment — the big combined dataset spanning many different robot types. The usual starting point.
• DROID — one robot recorded across a huge variety of settings, which makes it especially useful for testing whether a robot copes with unfamiliar surroundings.
• RH20T — 110,000+ recordings across 7 robots and 147 tasks, strong on tasks involving touch and contact.
• RoboCasa — kitchen scenarios, and probably the closest thing the field has to a standard benchmark environment.
• LeRobot (Hugging Face) — not a dataset but a shared format and toolkit, an attempt to get everyone's robot data working together. Hundreds of community datasets use it, though quality varies enormously.

Where things actually stand

A recent example: Gemini Robotics 2

Google DeepMind released three models in July 2026, which map neatly onto the ideas in this guide:

Read against everything above, this is a "brain and hands" arrangement — a thinking layer on top of a doing layer — rather than a robot with an imagination attached. Nothing published suggests any of the three predicts future video or mentally rehearses actions. Which is interesting: one of the best-resourced labs in the world has, at least for now, not bet on the world-model approach.

Google's own reporting notes that fine finger control remains inconsistent, and that its safety testing doesn't replace properly certified safety systems.

The honest scorecard

Glossary