What a simulator actually is

A physics engine does one narrow thing: given a description of bodies, joints, and forces, it computes where everything will be a few milliseconds later. It knows nothing about robots, learning, or goals. Those are layers built on top.

MuJoCo — "Multi-Joint dynamics with Contact" — is the standard choice in robot learning research. It was built for robotics, biomechanics and animation, aiming for both physical accuracy and computational efficiency, with particular attention to contact dynamics. DeepMind acquired it in 2021 and open-sourced it in 2022.

The property that matters is speed. A simulator can run a walking robot far faster than real time. If learning to walk takes a million attempts, that is decades of real-world falling and a few hours of simulation. Everything in this guide follows from that one fact.

A simulator is not a learned world model

This distinction is worth settling immediately, because both are casually called "world models." A physics engine contains hand-written equations of motion. A world model in the machine-learning sense is a neural network that inferred how things move by watching data. Both predict what happens next; only one is guaranteed to be physically consistent.

For everything that follows, the simulator is the world model in the functional sense — it is the thing that predicts consequences so a policy can practise safely.

The simulator provides a world. The next question is what is being trained inside it.

What a trained policy is

A policy is a function answering one question, repeatedly: given what I sense right now, what should my motors do next? Input: the robot's situation. Output: motor commands. A trained policy is one where that function is a neural network whose weights were tuned through millions of simulated attempts.

The word comes from its everyday sense — a standing rule for how to respond. "If I feel myself tipping left, push harder with the left leg." A trained policy is millions of such rules, learned rather than written, compressed into a network.

A policy is not remote control

This is the most common misunderstanding, and worth clearing before anything else. When you press a key and a simulated robot walks, you are not driving its joints. A biped might have fifteen servos updating fifty times a second — far beyond human input bandwidth. Your keypress sets a command ("go forward at 0.2 m/s"); the policy works out which leg lifts when, how much to lean, and how to correct a wobble.

Training and running are entirely separate phases. Once trained, the network is frozen — usually exported to a portable format — and does not learn while operating. That command idea returns in §2.3, where it becomes a concrete part of the policy's input.

A simulator and a policy are not enough on their own — something has to connect them, and something has to do the learning.

The three-layer stack

Tutorials often blur these together, which makes the whole thing harder than it needs to be. There are three distinct layers, usually three separate software packages.

The middle layer is the conceptually important one, and it is where nearly all your design work happens. It decides what the policy can see, what it can do, what counts as good, and when to stop. Part 2 is entirely about that layer.

GPU physics

Classic MuJoCo runs on the CPU, stepping one robot at a time. MuJoCo Warp reimplements it on NVIDIA's Warp framework so physics runs on the GPU, stepping thousands of robots simultaneously as one batched operation. Physics and neural network then both live on the GPU, so observations never have to be copied back and forth — often the real bottleneck in older setups.

This is why GPU-parallel training frameworks require an NVIDIA card — not as an optional accelerator, but because the physics engine itself only runs there. Why running thousands of robots matters so much is the subject of §3.1.

Before breaking these pieces apart in detail, here is how they fit together in a single pass.

The whole loop at a glance

Everything in Parts 2 and 3 is an expansion of this diagram. It is worth reading once now and returning to as a map.

┌─ Every timestep (e.g. 50 times per second) ──────────────────┐
│                                                              │
│  COMMAND  "walk forward at 0.2 m/s"      (§2.3)          │
│      +                                                       │
│  SENSORS  joint angles, tilt, velocities   (§2.2)          │
│      ↓  concatenated into one observation vector             │
│  POLICY   neural network                   (§1.2)          │
│      ↓                                                       │
│  ACTION   motor targets                                       │
│      ↓                                                       │
│  PHYSICS  simulator advances 20 ms         (§1.1)          │
│      ↓                                                       │
│  REWARD   a score for this moment          (§2.4)          │
│  DONE?    did it fall or time out?         (§2.5)          │
│                                                              │
└──────────────────────────────────────────────────────────────┘
        ↓  repeat across 4,096 robots at once          (§3.1)
        ↓  collect ~24 timesteps of this               (§3.3)
   LEARNING ALGORITHM nudges the network's weights     (§3.2)
        ↓
   repeat thousands of times → a walking gait

Two things are easy to miss on a first read, and both get their own section later. First, the command is part of the observation — the policy is told what to do through the same channel it senses through. Second, the weight update happens every 24 timesteps, not at the end of a run, which is why §3.3 spends time separating those two rhythms.

Six configuration elements

You do not write the training loop yourself. Modern frameworks use a manager-based API (an approach from Isaac Lab): you fill in six lists describing what the robot sees, does, and is scored on, and the framework runs the loop for you. Each list is one of the elements below.

Events are deferred to Part 4 because they exist almost entirely to serve sim-to-real, and they make more sense once that problem has been stated. A seventh element, curriculum, is sometimes used to widen ranges progressively as training improves.

Why lists instead of code

Writing the loop by hand works for one task, but a project with a dozen similar tasks ends up with a dozen near-identical copies of it. Filling in lists means a new task starts from an existing one and changes two entries.

The benefit that matters most comes later: because each reward is a named entry in a list, the framework can log every one separately. That per-term breakdown is what makes debugging possible at all (§5.4, §5.5) — a single combined score would tell you almost nothing.

Taking those elements in the order they appear in the loop — first, what goes in.

Observations and actions

The observation is everything the policy sees at a timestep, concatenated into one vector. It splits into two very different kinds of information: sensing, and instruction.

Proprioception — sensing your own body

Joint angles, joint velocities, body orientation (which way is up), angular velocity (am I tipping), and often the previous action taken. Note what is typically absent: no camera, no map, no absolute position in the world. A locomotion policy is essentially blind, feeling only itself.

This is deliberate. State-based observations are dramatically faster to learn from than rendered images, and far easier to transfer to real hardware, since a real robot does not magically know an object's coordinates but does know its own joint angles. Learning from pixels is the harder, more general path — worth attempting only after the state-based version works.

Actions

The action is the policy's output — usually target positions or torques, one per joint. The action config defines how those raw numbers are interpreted, typically scaled and offset from a neutral "home" pose. You will rarely change this.

A common refinement is action chunking: predicting a short sequence of future actions in one forward pass rather than a single next action. This matters when the network is slower than the control loop needs to be.

The second half of the observation is not sensing at all. It is how the robot is told what to do — the concrete form of the "command" idea introduced in §1.2.

The command space

The command space is the slice of the observation carrying instructions rather than sensor readings. It is what makes one network steerable across a range of behaviors instead of doing one fixed thing.

The standard locomotion command is a twist: three numbers for forward velocity, lateral velocity, and turning rate. Crucially it is expressed in the robot's body frame — "forward" means whichever way it is currently facing, not a fixed direction on the floor.

twist = [vx, vy, wz]
          │   │   └── yaw rate, rad/s, positive = turn left
          │   └────── lateral, m/s, positive = strafe left
          └────────── forward, m/s, positive = forward

The last row matters: these are simultaneous, not sequential. You do not walk then turn — all three numbers arrive at once and the policy produces a gait satisfying all of them.

Where commands come from during training

This is the part most easily misunderstood. During training, nobody drives the robot. The command manager samples randomly: each parallel environment draws its own command from a configured range, and resamples to a new value every few seconds of simulated time.

That resampling is what teaches transitions — accelerating, stopping, reversing while already moving. With one fixed command per episode, the policy would only practise steady-state walking and would stumble the first time a user changed direction.

At deployment the sampling stops and a keypress writes into the same slots. The policy cannot tell the difference — it sees the same numbers in the same positions. That is precisely why it responds sensibly to a human: during training it saw thousands of random variations of exactly the input now being typed. This is the mechanical answer to §1.2's "a policy is not remote control."

What you configure

• Ranges per dimension — the envelope of what the policy will be able to do. Too wide and it learns none of it well; too narrow and the deployed robot cannot be asked for much. A real design decision, not a formality.
• Resampling interval — how often the command changes mid-episode.
• Standing probability — an explicit fraction of environments should receive a zero command, or "stand still" is under-represented and the robot marches in place when the user releases the key.
• Curriculum — optionally start narrow and widen as the policy improves.

The shared observation contract

A useful architectural convention: keep the observation layout identical across every policy in a project, with unused command slots zero-padded rather than dropped. The cost is a few wasted dimensions per policy. The benefit is that a runtime can hot-swap between policies mid-motion — walk, recover, trick — without reshaping anything, because they all accept the same vector. §6.2 shows a real implementation of this.

Translating a user input into a command

1. Stay inside the trained range. Commanding a value the policy never practised will not error — it will just behave badly.
2. Ramp, don't jump. Smoothing a keypress over a few hundred milliseconds beats an instantaneous step input.
3. Have a real zero. Apply a deadzone so near-zero snaps to exact zero — the value standing behavior was trained against.
4. Scale, don't switch. Map analog input linearly onto the trained range; for a keyboard, pick one sensible value per key, usually mid-to-upper range rather than the extreme.
5. Don't assume symmetry. Backward and lateral ranges are usually narrower than forward.

Note also that a twist is a rate ("keep moving at this speed") while a pose command is an absolute target ("put the head here"). Rate commands naturally return to zero on key release; pose commands should hold their last value.

The policy now has an input and an output. What remains is telling it which outputs were any good.

Rewards — defining "good" in code

The reward is a number computed every timestep, almost always a weighted sum of eight to twelve terms. Writing it is writing a definition of success, and it is where most experiments succeed or fail.

For locomotion the terms fall into three groups.

Note how group 1 depends directly on §2.3: the reward is defined relative to the command. That is the mechanism letting one network cover a continuous range of speeds rather than one fixed gait.

Two terms deserving special attention

Action rate penalises how much the motor targets change between consecutive timesteps. It is the single most important term for smoothness. Without it you get high-frequency jitter that looks terrible and destroys real hardware.

Feet air time rewards each foot for spending a decent stretch airborne per step. This is the classic fix for shuffling: without it, a policy often discovers it can slide both feet along the ground, satisfy the velocity term, and never actually step.

Dense versus sparse — why locomotion is the easy case

Every term above is computable right now, this timestep: am I at the right speed, am I upright, did my foot slide. That is a dense reward, and it is why locomotion is tractable and why it is the standard teaching example.

Manipulation is much harder precisely because "did you grasp the cup" is sparse — one binary event that random exploration will essentially never stumble into. You must then invent dense intermediate rewards (approach the object, make contact, move it toward the goal), and designing those well has no clean recipe. Keep this distinction in mind; it explains a specific failure mode in §5.4.

One element of the loop remains: deciding when a run is over.

Terminations

Typically: body tilt past a threshold, or the torso touching the ground. Note that this is a different mechanism from a reward penalty, not merely a bigger one.

• A reward penalty is a number subtracted at that timestep. The episode continues.
• A termination ends the episode, which costs the policy all future reward it would have earned.

That is why falling is handled by termination rather than a large negative number. It is the stronger signal, and it also stops the robot from lying on the floor accumulating whatever rewards technically still apply.

That completes the per-timestep loop from §1.4. Part 3 turns to what happens across timesteps — how all this collected experience becomes an improved network.

Thousands of robots, one brain

§1.3 introduced GPU physics as a way to run thousands of environments at once. Here is why that matters — and it is not mainly about speed.

A typical run uses something like 4,096 environments simultaneously. The crucial detail: they all share one neural network. They are not 4,096 independent students — they are 4,096 simultaneous experiments run by a single student.

1. One shared policy network exists.
2. 4,096 robots spawn, each in a slightly different situation —
   different pose, friction, mass, random shoves.  (see §4.2)
3. Each runs a short stretch, all consulting the SAME network.
4. Each records: what I saw, what I did, what reward I got.
5. All records pool into one batch.
6. ONE weight update is computed from that pooled batch.
7. Updated weights apply to all 4,096.
8. Repeat, thousands of times.

Why breadth beats depth

With one robot, each update rests on a single robot's recent experience, which is enormously noisy. Did that gait work because it is good, or because that robot happened to start on flat ground with lucky footing? You cannot tell from one sample.

With 4,096, that same gait was just tried across 4,096 different starting conditions and disturbances at once. If it scored well on average, it is actually good. The noise averages out and the algorithm can take a confident step instead of a timid one.

Step 6 above — computing the weight update — is the algorithm's job. Almost always the same one.

PPO — the learning algorithm

Proximal Policy Optimization adjusts weights after each batch. The basic loop of any policy-gradient method is: try things, see what scored well, make those more likely. PPO's contribution is the word proximal — "stay close by."

The problem it solves: if a policy stumbles onto something slightly better and the algorithm updates aggressively toward it, the policy can lurch so far that it lands somewhere worse — and now all the experience just collected is useless, because it came from the old policy. Training collapses, sometimes irrecoverably.

PPO adds a leash. On each update it measures how much the new policy differs from the old, and clips the update if the change is too large.

PPO is not the cleverest algorithm available; it is the one that is hard to break. It tolerates imperfect hyperparameters and messy reward functions, which is exactly what you have in practice. That is why it is the default in robotics.

SAC is the main alternative — more sample-efficient, better when each environment step is expensive. For massively-parallel simulation where steps are cheap, PPO's stability wins.

PPO updates weights "after each batch." Defining that batch requires separating two rhythms that are easy to conflate.

Episodes versus rollout slices

Episode (~1,000 timesteps, ~20 s)
├── slice (24 steps) → weight update
├── slice (24 steps) → weight update
├── slice (24 steps) → weight update
└── ... ~40 slices; the episode continues across all of them

What updates, and when

Weights update every slice — globally, synchronized, roughly 40 times within a single episode. The policy stepping at timestep 500 is not the same policy that stepped at timestep 100 of that same episode.

Episode ends reset one environment — asynchronously, at unpredictable moments, with no effect on weights and no effect on the other environments.

Resets do not wait for slice boundaries

If a robot falls at step 11 of a 24-step slice, it respawns immediately at step 12. Waiting would waste enormous compute — early in training, when most robots fall within a second, most environments would sit idle.

The consequence: a single environment's 24 records can span two or even three episodes. That is normal, not an edge case.

slice:  |─────────────── 24 steps ───────────────|
env 0:  ────────────────────────────────────────   (mid-episode throughout)
env 1:  ──────────✗↺────────────────────────────   (fell at 11, reset at 12)
env 2:  ────✗↺──────────────────✗↺───────────────   (fell twice this slice)
env 3:  ──────────────────────────────⏱↺─────────   (timed out, reset)

Stepping is synchronized; episodes are not. The buffer is fundamentally a flat list of timesteps, and episode structure is encoded entirely in per-timestep done flags rather than in how the data is chunked. The obvious objection — how can half a second of data evaluate an action whose consequences unfold over twenty? — is the subject of the next section.

Answering that objection requires one more network.

Bootstrapping and the critic

Two things make short slices workable. First, the rewards are dense (§2.4): every one of those 24 moments carries a real score computed on the spot. Nothing waits for an outcome. Second, and more importantly, bootstrapping.

The critic

PPO trains a second network alongside the policy. The critic (or value function) has one job: look at a state and predict how much total reward is expected from here onward.

So when a slice ends mid-stride, the calculation does not pretend the world ended:

value of this moment = rewards actually observed in the slice
                     + critic's estimate of everything after

You observe a little, then estimate the rest. The name comes from "pulling yourself up by your bootstraps" — the circularity is the point. The critic's estimates train the policy, and the critic is trained using its own future estimates as targets. It works because real observed rewards are mixed in at every step, and that truth propagates backward over thousands of iterations.

The payoff, concretely

A robot lunges forward. Within the slice it moves fast, so the velocity reward is good — naively the action looks great. But the slice ends with the robot tilted 40 degrees. The critic looks at that state and says "expected future: very low," because it has seen thousands of similar states end in a fall. That low estimate drags the total down and correctly marks the lunge as bad. The consequence is captured — estimated rather than observed.

Where terminated versus truncated finally matters

This is the calculation §2.5 pointed forward to:

• Terminated (fell): do not bootstrap. Future reward is genuinely zero. This is what makes falling costly.
• Truncated (timeout): do bootstrap. The robot was fine; cutting off the future would wrongly punish a good state.
• Slice boundary (neither): bootstrap. The episode continues.

Advantage calculation runs backward through the slice, which is how a fall at step 11 makes a reckless action at step 9 look bad. A done flag acts as a wall: the chain stops there, so a new episode's rewards never leak backward into the old one's evaluation. This is why a slice spanning two episodes (§3.3) is harmless.

There is a dial (commonly called GAE λ) controlling how much to lean on observed rewards versus the critic. Near 1: trust observations — accurate but noisy. Near 0: trust the critic — stable but biased. Typical values sit around 0.95.

With PPO and the critic both in play, the data each timestep must carry is now fully determined.

What each timestep record contains

The intuitive guess — observation, action, reward — is right but incomplete.

That last field is the heart of PPO's leash. It is also what lets PPO reuse the same batch for several gradient passes: after the first pass the policy has moved, but the stored old values still permit a valid clipped ratio.

You do not need to store the resulting next observation separately — the buffer is sequential, so it is simply the next row.

In practice this is not a list of tuples but a set of parallel tensors, all shaped [slice_length, num_envs, ...], which is what makes the update a few large tensor operations rather than a loop over ~100,000 records. Advantage is not stored at collection — it is computed after the slice closes, working backward through rewards, values, and done flags.

At this point the simulated picture is complete: a policy can be trained to walk. Part 4 addresses why that is not yet the same as a robot that walks.

The sim-to-real gap

A policy that walks flawlessly in simulation often collapses on real hardware. The reason is simple: the simulator is wrong. Not badly wrong, but wrong in a thousand small ways — friction is a little different, motors respond a little slower, sensors are noisy, gears have play, the battery sags under load.

The trouble is that reinforcement learning is an exceptionally good exploiter. If your simulation has one exact friction coefficient, the policy will happily learn a gait that depends on that precise value, because doing so scores marginally better than a robust one. It has no incentive to be general — only to score.

Two mitigations dominate, and they attack different halves of the problem. Domain randomization (§4.2) prevents the policy from over-fitting to any single physics configuration. Actuator modelling (§4.3) attacks the largest single source of error, particularly for small robots.

The first mitigation is also the sixth environment element deferred from §2.1.

Domain randomization — and the Events element

Events are the config element that fires at reset or on a schedule, and their main use is domain randomization: varying the physics from environment to environment so no two of the 4,096 robots live in quite the same world.

Commonly randomized: ground friction, body masses, motor strength, command delay, battery voltage, and random pushes applied mid-stride.

This connects directly back to §3.1. The reason breadth across environments improves the learning signal is the same reason it improves robustness: the policy is rewarded for what works on average across conditions, not for what works in one lucky configuration.

There is a cost. Wider randomization makes learning harder and slower, and can lower peak performance — a policy hedging against many possible frictions is less optimal for any single one. Practitioners often disable randomization while debugging a reward function and re-enable it before a real run.

Randomizing an inaccurate model only goes so far. For small robots, one component dominates the remaining error.

Actuator models and backlash

The default way to simulate a motor is as an idealised controller: you ask for a joint angle, and the joint moves there smoothly, with predictable force. Real motors are not like this at all — and at small scales, that difference is most of the sim-to-real gap.

What a real servo actually does

A hobby servo like the Dynamixel XL330 (widely used in small robots) has behaviors an idealised model omits entirely:

• Voltage control law — the motor is driven by voltage, not by a direct force command, so available torque depends on the battery.
• Back-EMF — a spinning motor generates its own opposing voltage, so it gets weaker the faster it turns.
• Friction, in several flavours: constant resistance (Coulomb), a sticky extra resistance right at the point of starting to move (Stribeck), and friction that grows with load.
• Command delay — the time between issuing an instruction and the motor acting on it.
• Voltage sag — the battery drooping under heavy current draw, weakening every motor at once, exactly when the robot needs them most.

BAM — a library of realistic motor models

BAM ("Better Actuator Models," from the Rhoban robotics team) is an open-source project that fits these behaviors to real servos by measuring them on a test bench, then packages the result as a simulation model. Its models are named by complexity — M1 being the simplest and M6 one of the most detailed, incorporating the voltage law, back-EMF, and the full friction picture above.

So "using the BAM M6 model for the XL330" means: instead of pretending each joint is a perfect position controller, the simulation reproduces the measured electrical and frictional behavior of that specific real servo.

Backlash — the wobble in the gears

Backlash is the small amount of looseness in a set of gears. The teeth cannot fit together perfectly tightly, so the motor turns a tiny bit — around one degree in a hobby servo — before the joint it is driving actually moves.

To simulate it, you add an extra unpowered joint next to each servo that is free to wobble within that one degree. Now the simulated robot has the same looseness as the real one.

A common pattern is to train two versions of each behavior: a clean one, which trains faster and is good for iterating on rewards, and a backlash one, which is what you actually deploy. §6.2 shows this in practice.

Hardware and software

Hardware

• CPU-only path: a modern laptop is genuinely fine for classic control tasks. Physics runs on CPU; the networks involved are tiny by LLM standards. 16 GB RAM is comfortable.
• GPU required for two things: massively-parallel frameworks built on GPU physics (§1.3 — the physics engine itself only runs there), and any vision-based policy learning from rendered images.
• Cloud alternative: most frameworks now offer a flag to submit training to a hosted job service, removing the GPU requirement at the cost of credits and slower iteration.

Software

Install the deep-learning framework first and pick the variant matching your hardware — CPU-only, CUDA for NVIDIA, ROCm for AMD. Getting this wrong causes hard-to-diagnose errors later.

Use an isolated environment: either a traditional package manager, or a modern project runner like uv, which combines dependency management and command running.

Choosing a simulator

Simulator choice matters less than it seems at the start. MuJoCo is the path of least resistance; move to Isaac Lab when you specifically need massive GPU-parallel locomotion training or photorealistic rendering. Note that frameworks are converging — mjlab reimplements Isaac Lab's manager-based API (§2.1) on MuJoCo Warp physics, so those skills transfer in both directions.

A suggested first month

1. See the loop work. Train a standard benchmark walker. The goal is confirming the install and internalising §1.4, not the result.
2. Break it deliberately. Remove the energy penalty and watch it get twitchy. Remove the upright term and watch it discover the face-dive exploit. This teaches reward design faster than any tutorial.
3. Build your own environment. Take a real robot model, write a task with your own reward. Everything after this is a variation.
4. Switch to pixels. Everything gets dramatically slower, and you will understand immediately why the field cares so much about pretrained visual representations.

With a working setup, the recurring task is defining new behaviors. Here is what that actually requires.

Designing a new behavior

Four decisions define an experiment, and they matter far more than which algorithm you pick.

1. The command (§2.3) — what can be asked, and over what range.
2. The reward (§2.4) — the terms and their weights. Most of your time goes here.
3. The initial state distribution — what pose robots spawn in. Non-obvious and critical: a stand-up behavior must spawn robots lying down; a walking behavior must not.
4. The termination conditions (§2.5) — usually inherited, unless failure means something different for your task.

A fifth input is sometimes needed: the body model itself. A task where the robot must lie on the ground needs full-body collision geometry that a walking-only model may strip out for speed.

When to make a new policy versus extend an existing one

A frequent early mistake is creating a separate policy per behavior. The right test is reward compatibility:

• Same policy: forward, backward, turning, and strafing all share one reward — "track the commanded twist." They differ only in command values, so they belong to one network. Creating separate policies here throws away the entire point of a command space.
• Separate policies: walking and standing-up-from-face-down have irreconcilable rewards. A policy rewarded for staying upright can never learn to stand up, because lying on the ground is a failure state it has been trained to avoid entirely.

Split by reward incompatibility, not by behavior count. Multiple policies are then switched at runtime via the shared observation contract from §2.3.

Most runs fail. The skill is not avoiding that but diagnosing it quickly.

Five ways training fails

The instinct when a run fails is to start adjusting weights. Resist it. First read the per-term reward breakdown (§5.5) and identify which failure this is, because they need different fixes.

What to change next, in order

1. Reward weights — cheapest and most common fix. Change one at a time.
2. Add or remove a term — when there is a loophole to close.
3. Reward shaping — convert sparse to dense.
4. Command range — narrow it; widen later once it learns.
5. Initial state distribution — spawn closer to success. Enormously effective for acrobatic behaviors.
6. Terminations — too strict and it never survives to learn; too loose and it accumulates reward while sprawled.
7. Disable domain randomization temporarily (§4.2) — if it cannot learn with clean physics, randomization is not the problem. Re-enable before any real run.
8. Training length — sometimes it just needs more steps.
9. PPO hyperparameters — last. Rarely the actual issue.
10. Reference motion — for acrobatic behaviors exploration will never find, supply a trajectory to imitate rather than hoping for discovery.

Discipline that pays off: one change per run, or you learn nothing about either. Kill runs fast — most should die within five minutes, because if episode length is not climbing early it will not. And expect 10–20 attempts for a new behavior; that is normal, not a sign you are doing it wrong.

All of that diagnosis depends on being able to see what is happening.

Monitoring and outputs

You do not observe individual episodes. With thousands of environments producing episodes continuously, that is neither possible nor useful. You watch aggregate statistics logged per iteration, usually in an experiment tracker.

Weights & Biases (W&B) is the common choice: training logs metrics as it goes and renders them as live charts in a browser, storing each run's config so you can later see exactly which weights produced a good result. Its most valuable feature is overlaying multiple runs on shared axes — which is how you tell whether a change helped. TensorBoard is the local, no-account alternative.

This is the payoff promised back in §2.1: because each reward is a named entry in a list, the framework logs each one separately. When a policy stands still, you see it directly — tracking reward near zero, penalties near zero — which immediately says the tracking weight is too low. You would never diagnose that from a total.

What a training run produces

• Checkpoints — network weights saved periodically. Raw, not deployable.
• Logs — metrics and config.
• An exported policy — produced by a separate export step, typically to ONNX, a portable format a lightweight runtime can execute without the training framework.

The repository

Microduck is an ~800 g, ~25 cm bipedal robot from Pollen Robotics (acquired by Hugging Face), with 15 Dynamixel XL330 servos and a 50 Hz control loop. Its training code — pollen-robotics/microduck_rl, Apache 2.0 — is an unusually complete public example of everything above.

The stack maps exactly onto the three layers from §1.3:

microduck_rl          ← the repo you clone; task definitions
  └── mjlab           ← training framework
        ├── manager-based API   (§2.1, from Isaac Lab)
        ├── MuJoCo Warp         (§1.3, physics, 4,096 envs on GPU)
        └── rsl_rl              (§3.2, PPO implementation)

Policies train at 50 Hz, export to ONNX, and are deployed on the real robot by a separate runtime. Roughly one to two hours on a CUDA GPU produces a usable gait at 4,096 parallel environments.

Project structure

src/mjlab_microduck/
├── robot/
│   ├── microduck/                 # MJCF exports, scenes, add_backlash.py
│   └── microduck_constants.py     # robot cfgs, HOME frame, actuator cfg
├── actuator/friction_dr_bam.py    # BAM model + friction DR + backlash (§4.3)
├── tasks/
│   ├── __init__.py                # task registration
│   ├── mdp.py                     # reward/event/observation FUNCTIONS
│   ├── backlash.py                # make_backlash_variant() wrapper
│   └── microduck_*_env_cfg.py     # one cfg module per task family
├── train_cli.py                   # `train` entry point (+ --hf-jobs)
└── hf_jobs.py                     # Hugging Face Jobs submission

The task registry

Thirteen registered tasks (uv run list-envs prints the live list). The main one is Mjlab-Velocity-{Flat,Rough}-MicroDuck — walking with velocity commands plus head-pose commands. Others include VelStand (walking + fall recovery in one policy), StandUp, SitStand, GroundPick (touch the ground with the mouth tip), BallKick, Roulade (a forward roll over the head), and six roller-skate variants.

That list is §5.3's reward-compatibility rule in practice: thirteen tasks because thirteen irreconcilable reward functions, not thirteen behaviors.

Concepts mapped to code

The observation contract, concretely

§2.3's "shared observation contract" is implemented here as a 61-dimensional actor observation: 48 proprioception + 13 command, where the command splits as twist(3), head_pose(4), body_pose(6). Environments not using a command slot zero-pad it rather than dropping it — which is exactly what allows the runtime to hot-swap walk / recover / trick policies mid-motion.

So "walk forward at 0.2 m/s" is literally:

twist     = [0.20, 0.0, 0.0]        ← the command (§2.3)
head_pose = [0, 0, 0, 0]            ← unused by this task → zeros
body_pose = [0, 0, 0, 0, 0, 0]      ← unused by this task → zeros
                                     + 48 proprioception = 61 in
                                     → 15 servo targets out, 50×/sec

Body models differ per task

A concrete instance of §5.3's fifth input:

Part 4, made specific

This repo is a direct implementation of §4.3. All tasks use the BAM M6 actuator model for the Dynamixel XL330 — voltage control law, back-EMF, and Coulomb/Stribeck/load-dependent friction — rather than an ideal PD controller, with per-environment randomization on battery voltage, voltage sag under load, command delay, and friction magnitude.

The backlash implementation is the textbook version of the encoder-placement subtlety flagged in §4.3. Every main task has a Backlash twin — insert -Backlash before MicroDuck in the task id — training against ±1° of gear play (2° total) in series with each of the 14 servo joints. Each servo gets an unactuated passive_<joint>_backlash hinge, and because the real encoder sits on the output side of the play, both the firmware PD emulation and the joint_pos/joint_vel observations read through the backlash. Observation and action dimensions are unchanged, so export and runtime need no modification.

Three conventions worth knowing

1. Unactuated joints (roller wheels, backlash hinges) are all named passive_*; actuators, joint observations and pose rewards select servo joints with ^(?!passive_).*.
2. Joint layout, 14 servos: 0–4 left leg (hip_yaw, hip_roll, hip_pitch, knee, ankle), 5–8 neck/head, 9–13 right leg.
3. The exporter bakes the observation normalizer into the ONNX graph — §5.5's failure mode, made concrete.

Running it

# once, on your machine
curl -LsSf https://astral.sh/uv/install.sh | sh

git clone https://github.com/pollen-robotics/microduck_rl
cd microduck_rl                    # uv run only works from inside here (§5.1)
# On ARM (DGX Spark / Jetson): export UV_HTTP_TIMEOUT=600 for the first sync

uv run list-envs                                    # the live task registry
uv run --with pytest pytest tests/                  # CPU-only, verifies setup

# sanity-check the environment BEFORE burning GPU hours
uv run play Mjlab-Velocity-Flat-MicroDuck --agent zero
uv run play Mjlab-Velocity-Flat-MicroDuck --agent random

# train (needs a CUDA GPU — MuJoCo Warp is GPU-only, §1.3)
uv run train Mjlab-Velocity-Flat-MicroDuck --env.scene.num-envs 4096
# ...or run it on Hugging Face Jobs instead (§5.1)
uv run train Mjlab-Velocity-Flat-MicroDuck --hf-jobs

# watch a trained policy
uv run play Mjlab-Velocity-Flat-MicroDuck --wandb-run-path <entity/project/run_id>

# export for deployment (§5.5 — never skip this step)
uv run scripts/export.py Mjlab-Velocity-Flat-MicroDuck --wandb-run-path <...>

# drive the exported policy in CPU MuJoCo — no GPU needed
uv run scripts/infer_policy.py --walking output.onnx

Resuming uses --agent.run-name resume --agent.load-checkpoint model_29999.pt --agent.resume True.

The zero/random agents

The --agent zero and --agent random options are a genuinely useful habit and a good instance of §5.4's "kill fast" discipline. Zero actions confirm the robot spawns correctly and does not explode on load; random actions confirm terminations fire and rewards produce sane numbers. Catching a broken config here costs seconds instead of an hour.

Deployment and hot-swapping

scripts/infer_policy.py runs on CPU MuJoCo, so policies can be tested on a laptop. It also rehearses exactly what the real runtime does — loading several policies behind the shared 61-dim contract (§6.2) and switching between them:

uv run scripts/infer_policy.py --walking walk.onnx --standing stand.onnx \
    --sitstand sitstand.onnx --roulade roulade.onnx --new-cmd-obs

It is keyboard-driven — velocity commands plus G ground pick, Y sit/stand, R roulade, K/L kicks — with --debug, --save-csv, and --record for sim2real comparisons. The scene*.xml files wrap the robots with a floor and STAND/SIT/FOLD keyframes specifically for this script.

Adding a new behavior

Take "walk sideways" as a worked example, following §5.3's four decisions.

1. Write or reuse reward functions in tasks/mdp.py — here, lateral velocity tracking plus a heading penalty so the robot does not simply turn 90° and walk forward.
2. Copy the nearest env cfg and adjust: swap the tracking term, set the command range on the lateral twist dimension, keep the quality terms (upright, energy, action rate, foot slip, air time).
3. Register it in tasks/__init__.py.
4. Sanity-check with --agent random, then train 30 minutes and look.
5. Iterate — expect 10–20 rounds, most killed within five minutes.

Failures to expect, cross-referenced

All four are reward-design bugs, not code bugs. Training runs cleanly and produces a policy doing something you did not intend — which is why §5.4 insists on watching the rendered behavior rather than only the curves.

A better first experiment

Rather than a new behavior, take the existing walking task and change one reward weight. Multiply the energy penalty by five and watch it refuse to move. Zero out the action-rate penalty and watch it jitter. Widen the forward velocity range and see whether low-speed performance degrades as capacity spreads across a wider command envelope (§2.3).

That calibrates your intuitions for what each term does — which is exactly what you will need when a novel behavior misbehaves ambiguously.