Sim knowledge · Ground control
Everything to understand before you touch the terminal — so the hands-on part feels easy, not scary. Read it on your phone, tap through the quizzes, tick the boxes.
Why we teach robots in a fake world first, and the two main ways they learn.
A robot arm that picks things up looks simple when a human does it. But a robot has to figure out, thousands of times a second, exactly how much to turn each joint. Writing those numbers by hand is nearly impossible for real tasks. So instead, we let the robot learn — from examples, or from trial and error.
Learning means failing a lot. And failing on a real $5,000 robot arm is slow, expensive, and sometimes dangerous. So we do the failing in a simulator first: a fake physics world inside the computer where the arm can crash a million times for free.
flowchart LR
subgraph REAL["Real robot"]
R1["slow: 1x speed"]
R2["breaks / wears out"]
R3["needs a human watching"]
end
subgraph SIM["Simulator"]
S1["fast: many x speed"]
S2["reset for free, forever"]
S3["run 1000 in parallel"]
end
REAL -->|"but 100% real"| TRUTH["the real world"]
SIM -->|"but only a close guess"| TRUTH
Key idea
sim-to-real gap
A simulator's physics is a close guess, not perfect. Friction, weight, and timing are slightly off. So a robot that works great in sim can stumble in real life. Shrinking this gap is one of the biggest jobs in robotics (with tricks like "domain randomization" and fine-tuning on a little real data).
This is exactly why the ALOHA project (which we use) ships both a simulator and real low-cost hardware — so you can practice in sim, then cross the gap to a real arm.
Almost all robot learning is one of these two (or a mix):
Different as they sound, both run on the same heartbeat — a loop we'll meet in Chapter 2.
Words you'll keep seeing
policy — the robot's "brain": a function that looks at what it sees and decides what to do. agent — the thing that acts (its brain is the policy). environment — the world it acts in (the arm + objects + physics).
Why do we train robots in a simulator first, instead of on the real arm?
Answer: b. Learning = lots of trial and error. In sim you can reset for free, run faster than real time, and never break anything. Reality is actually more accurate than sim (that's the whole "sim-to-real gap" problem).
A robot trained perfectly in simulation sometimes fails on the real arm. Why?
Answer: c. That is the sim-to-real gap. The simulator approximates friction, weight, and timing, so a policy tuned to the fake physics can misbehave on the true physics.
✓ Chapter 1 checklist
One loop runs under every robot-learning program. Learn it once, and the code stops looking mysterious.
Here is the single most important picture in this whole course. Everything — random arms, trained arms, our "pat" motion — is this loop repeating.
flowchart LR A["Agent (policy)
the brain"] -->|"action:
what to do"| E["Environment
the world"] E -->|"observation:
what it now sees
(+ reward)"| A
Read it as a conversation that repeats forever:
Key idea
a "step"
One trip around the loop is called a step. A whole attempt from start to finish (many steps) is an episode. Training a robot means running millions of steps and slowly improving the policy.
This is why, later, you'll see code shaped like while ...: action = ...; env.step(action). The while loop is the heartbeat. In our first "hello" script the arm sits still — because there's no loop, no step. In the live script the arm moves — because there's a loop calling step over and over.
Remember this
Movement = running the loop. No loop, no motion. If you ever see a frozen sim, the first question is: "is anything calling step?"
The loop itself just runs the robot. Learning is a slower process wrapped around it: after many steps/episodes, we nudge the policy's numbers so its actions get better. Our course stops at running the loop with a simple hand-written policy (a sine wave). Training a smart policy is the next mission.
In the loop, what does the environment give back to the agent after an action?
Answer: b. The environment returns the next observation (what the world looks like now) plus a reward in RL. The agent needs that next observation to decide its next action — that's why the loop connects.
A simulation window opens but the arm never moves. What's the most likely cause?
Answer: a. Movement comes from stepping the physics. No loop calling step → the world never advances → a frozen arm.
✓ Chapter 2 checklist
One standard remote control fits thousands of simulators. Learn its five buttons and their return values.
What is it
Gymnasium was “OpenAI Gym”
A standard set of commands for talking to any learning environment. Think of it as a universal remote: once you know its buttons, you can drive almost any simulator — robots, games, control problems — the same way. (It used to be called "Gym"; the maintained version is "Gymnasium". Same idea.)
Almost everything you do to an environment is one of five methods. A method is just "a thing an object can do" — you write it as object.method().
Pay attention to the return value (what you get back). The return value tells you what the method is for.
flowchart TD M["gym.make(id)
→ an env object"] --> R["env.reset(seed)
→ (observation, info)"] R --> S["env.step(action)
→ (obs, reward, terminated, truncated, info)"] S --> D{"episode over?
(terminated or truncated)"} D -->|no| S D -->|yes, go again| R S -.optional.-> V["env.render()
→ picture (or a window)"] D -->|all done| C["env.close()
→ nothing"]
env = gym.make("gym_aloha/AlohaTransferCube-v0") builds the environment and hands you back an env object to control.
Clear up a common confusion
The env is NOT a "space". The env is the whole world object. It has parts, like env.action_space (the rulebook of valid actions). You reach a part with a dot: env.action_space means "the action_space of env". Beginners often think env = the space; it doesn't.
obs, info = env.reset(seed=0) puts the arm and cube back to a starting pose. It returns two things: the first observation (so the agent can see the start) and info (extra debug details).
Why this return, not another?
Notice reset gives an observation but no reward. Why? Because reward is the result of an action, and reset hasn't taken any action yet — the attempt just began. This one asymmetry teaches you that "reward" always answers "how good was that action?"
This is the busy one. env.step(action) applies your action, moves physics one tick, and returns five things:
obs, reward, terminated, truncated, info = env.step(action)
obs — the new observation (what the world looks like now).reward — a score for that action (used in RL; imitation learning often ignores it).terminated — did the task end naturally? (success, or a real failure).truncated — was it cut off for an outside reason? (usually "ran out of time / max steps").info — extra debug details.The classic mix-up
terminated ≠ truncated
terminated = the episode ended by the rules of the task (you succeeded, or you failed for real). truncated = the episode was stopped from the outside, most often a time limit — not a real ending. They're split apart because a learning algorithm must treat "I finished" and "I ran out of time" differently. (Old Gym lumped both into one done value; that caused subtle bugs, so Gymnasium separated them.)
That's why real loops end each turn with: if terminated or truncated: env.reset() — "if the attempt is over for any reason, start a fresh one."
env.render() draws the current scene. In rgb_array mode it returns a picture as numbers (an array you can save). In human mode it opens a window and returns nothing — the window is the output. (Much more on this in the rendering chapter.)
env.close() shuts the window and frees resources. It returns nothing — because it's an action, not an information request. "What a method returns (or doesn't) tells you its job": the three that hand back data are information-givers; close just does a chore.
The word "Gym" is literally "gymnasium". But because the whole point is reset-and-repeat practice, a training dojo fits the five methods even better:
make → enter the dojo (pick which training room) reset → start a fresh session (back to the mat) step → do one move; the coach tells you the result render → look in the mirror / film it close → leave, tidy up
Why does env.reset() return an observation but no reward?
Answer: b. The attempt just started — no action has happened — so there's nothing to score. Reset gives you the starting observation so the agent can choose its first action.
Your robot hits the max number of steps before finishing the task. Which flag becomes true?
Answer: c. Hitting a step/time limit is an outside cutoff → truncated. terminated is for a real ending (success or genuine failure). The split lets learning code tell "I finished" from "time's up".
env.close() returns nothing. What does that tell you about it?
Answer: a. Methods that return data are information-givers (reset, step, render). close just does a job — shut the window, free memory — so it has nothing to hand back.
✓ Chapter 3 checklist
What an "action space" really is, why every command is a number between −1 and 1, and the tiny bit of math behind it.
In Chapter 3 you met env.action_space. Let's open it up — this is where a lot of beginners feel lost, and it's actually simple.
What is it
a space
A space is just a rulebook of valid values. action_space = "which commands are allowed?" observation_space = "what shape does what I see come in?" It's not the arm, and not the data — it's the rules for the data.
flowchart TD SP["a space
(rulebook of valid values)"] --> B["Box
continuous numbers in a range
e.g. any real number in [-1, 1]"] SP --> D["Discrete(n)
one whole number from 0..n-1
e.g. up/down/left/right = Discrete(4)"] B --> AL["ALOHA action:
Box(-1, 1, shape=(14,))"]
ALOHA's action space is written Box(-1.0, 1.0, (14,), float32). Read it as:
Plain words
vector = an ordered list of numbers, like [0.1, -0.4, …]. dimension = how many numbers are in it. So a 14-D action is a list of 14 numbers, and its .shape is (14,). Each number drives one thing on the robot.
Two handy tools every space has: space.sample() gives a random valid value (great for a "do anything" test agent), and space.shape tells you the shape (here (14,); an image observation might be (480, 640, 3)).
Real robot joints have messy ranges: one joint might turn from −2.6 to +2.6 radians, a gripper might open 0 to 4 centimeters. Feeding those raw, wildly different ranges into a neural network makes learning shaky. So we put every joint on the same −1…1 ruler. This is called normalization.
Formula · map a real value from [a, b] into [−1, 1]
Check it: if x = a (the low end) → −1. If x = b (the high end) → +1. If x is the middle → 0. Nice and tidy.
And the inverse · turn a −1…1 command back into a real value
The simulator uses this inverse to turn your −1…1 action back into a real joint angle.
Worked example
A joint's real range is a = −2.6, b = +2.6 radians. You want to command the middle (straight, 0 rad).
So "straight" is 0 in the −1…1 world. Full one way is −1, full the other way is +1. That's the whole idea behind our sine-wave "pat": sin gently rides between −1 and +1, so both arms sweep smoothly.
Two words worth 30 seconds
radians — a way to measure angles where a full turn is 2π ≈ 6.28 (instead of 360°). Half a turn = π ≈ 3.14, a quarter = π/2 ≈ 1.57. Robots use radians because the math of rotation is cleanest that way. Convert with radians = degrees × π / 180.
Subtle but important
The action space bounds are −1…1 (normalized), but the numbers those map to are real joint targets in radians. So "−1…1" is the clean outer language; "radians" is what the physics actually uses underneath. The env does the translation for you.
Why are all 14 action numbers squeezed into the same −1…1 range?
Answer: c. Raw ranges differ wildly (radians vs centimeters). Putting them on one −1…1 ruler (normalization) keeps the learning stable and fair across joints. The simulator converts back to real units internally.
A robot that only chooses up / down / left / right would use which space?
Answer: b. Four fixed buttons = a discrete choice → Discrete(4). Box is for continuous dials (like joint angles that can be anything in a range).
Using the formula, a joint's range is [0, 10]. What is the value x = 10 in the −1…1 world?
Answer: a. xnorm = 2·(10−0)/(10−0) − 1 = 2·1 − 1 = +1. The top of the real range always maps to +1, the bottom to −1, the middle to 0.
✓ Chapter 4 checklist
What ALOHA is, why it has two arms, and how it differs from “ACT”.
What is it
ALOHA “A Low-cost Open-source Hardware system for bimanual teleoperation”
A two-arm ("bimanual") robot designed to be cheap and open, built at Stanford and introduced in a 2023 paper. The whole point: make good robots affordable so more people can do this. gym-aloha (by Hugging Face) is the simulation of it, running on MuJoCo — that's what we use, no hardware required.
ALOHA records human demonstrations with a clever trick. There are leader arms (small ones the human moves by hand) and follower arms (which copy the leader in real time and actually do the task). While the human puppeteers the leaders, the system records every motion — that recording becomes the demonstration data for imitation learning.
flowchart LR H["human hand"] --> L["leader arms
(you move these)"] L --> F["follower arms
(copy the motion,
do the real task)"] F --> DATA["recorded demo data
obs → action pairs"] DATA --> POL["train a policy
(imitation learning)"]
Don't mix these up
ALOHA = the hardware/robot. ACT (Action Chunking Transformer) = a learning algorithm introduced in the same paper. ALOHA is the body; ACT is one possible brain. You'll see both names together, but they're different things.
One arm picking up a block is tricky. Two arms cooperating — one handing an object to the other in mid-air — needs coordination: both must be in the right place at the right time. That's exactly the TransferCube task we simulate: the right arm grabs a red cube and passes it to the left arm. The other task, Insertion, has the arms fit a peg into a socket together.
This is why the action is 14 numbers, from Chapter 4: two arms × (6 joints + 1 gripper) = 14. And why the picture from our very first script showed two grippers facing each other.
What the robot sees & when it "wins"
In sim, the observation is usually a top-down camera image plus the 14 joint positions (called proprioception — the robot "feeling" its own pose). The task gives a small reward as it gets closer; a specific top reward means full success (cube transferred), which also flips terminated to true.
What's the difference between ALOHA and ACT?
Answer: b. Body vs brain. ALOHA is the low-cost bimanual hardware (and its sim); ACT is one algorithm you can train to control it. Same paper, different roles.
Why does ALOHA use a leader–follower teleoperation setup?
Answer: c. The human moves the small leader arms; the followers copy and do the task; every motion is recorded as demo data for imitation learning. Teleoperation = turning human skill into training data.
✓ Chapter 5 checklist
The thing that actually computes gravity, joints, and collisions — and its two key objects, model and data.
What is it
MuJoCo “Multi-Joint dynamics with Contact”
A physics engine: software that computes how bodies move under forces, gravity, joints, and collisions. It does this by stepping time forward in tiny slices — given the current state and your commands, it calculates the next state a few milliseconds later. Made by DeepMind, free and open since 2022.
MuJoCo reads the robot's design from an MJCF file — an XML text file describing bodies, joints, shapes, cameras, and lights. Think of it as the architectural drawing the engine builds the world from. You won't write one in this course; just know that's where the arm's shape comes from.
flowchart LR MJCF["MJCF file
(XML blueprint)"] --> MODEL["MjModel
the constant blueprint
(masses, joints, shapes)
never changes"] MODEL --> DATA["MjData
the live state
(qpos, qvel, contacts)
changes every step"] ACT["your action → ctrl"] --> DATA DATA -->|"mj_step advances time"| DATA
The distinction in one line
MjModel = what doesn't change (the design: how heavy, how long, how many joints). MjData = what does change every moment (where the joints are right now, how fast they're moving). One model, and a data that keeps updating.
The most important thing inside MjData:
qpos — "generalized positions": the current angle of every joint. When the arm moves, qpos changes. (This is exactly the value our live viewer watches to redraw.)qvel — "generalized velocities": how fast each joint is moving.ctrl — the control input: this is where your action enters the physics. Your 14 numbers become the targets the engine drives the joints toward.Nice-to-know (won't trip you up)
The list of positions (qpos) and the list of velocities (qvel) aren't always the same length — some joint types need more numbers to describe position than speed. Don't assume len(qpos) == len(qvel). (You rarely touch this directly as a beginner.)
MuJoCo advances in a fixed timestep, often about 0.002 seconds (2 ms). Each "physics step" moves the world forward by that slice. Smaller slices = more accurate but slower. (A policy usually decides less often than the physics ticks, so several physics steps run per action.)
When two shapes touch, MuJoCo detects a contact and computes the push-back forces so things don't pass through each other. Contacts are the hard, expensive part of physics — and the reason a gripper can actually "hold" a cube.
When the arm moves during simulation, which changes — MjModel or MjData?
Answer: b. MjModel is the unchanging design. Motion = MjData.qpos changing over time. That's why a live viewer just needs to re-read data each frame.
How does your 14-number action actually reach the physics?
Answer: a. Actions enter through the control input ctrl. The engine then computes forces to move the joints toward those targets over the next timestep(s).
Why use a small timestep like 2 ms instead of, say, 1 second?
Answer: c. Physics is integrated slice by slice. Tiny slices track fast motion and contacts accurately; huge slices skip over collisions and blow up. The cost is speed — more slices per second of sim.
✓ Chapter 6 checklist
Why the real physics is buried three layers deep, and what env.unwrapped peels away.
Remember this line from the live script? env.unwrapped._env.physics.model.ptr. It looks scary. By the end of this chapter it will look obvious — it's just peeling an onion.
What is it
a wrapper wrap = to cover
A wrapper is a layer placed around something to make it nicer to use — like a phone case around a phone. It adds convenience without changing what's inside. In our stack, the real physics engine is wrapped, and then that wrapper is wrapped again.
flowchart TD G["Gymnasium env
(what gym.make gives you)
+ small helper wrappers"] --> A["gym-aloha env
(the ALOHA task)"] A --> D["dm_control
(DeepMind's MuJoCo wrapper)
reached via ._env"] D --> P["Physics
reached via .physics"] P --> M["the real MuJoCo
.model.ptr / .data.ptr"]
So the scary line reads, left to right:
env.unwrapped — take off Gymnasium's outer helper wrappers, exposing the ALOHA env inside.._env — inside that, reach the dm_control environment..physics — inside that, reach the Physics object..model.ptr / .data.ptr — finally, the real MjModel and MjData from Chapter 6.It's a dot-path down the onion. Each dot means "the thing inside."
Why wrap at all? (a real trade-off)
Raw MuJoCo is powerful but low-level and fiddly. dm_control adds "tasks, resets, observations." gym-aloha adds "the Gymnasium API you already know." Each layer buys convenience — but the price is that the real physics is buried, so when you need it you must dig with .unwrapped._env.physics. Convenience vs direct access: a classic software trade-off.
Gymnasium often adds small wrappers automatically. Two you should know because they explain earlier mysteries:
truncated = True (from Chapter 3!). The "time limit" isn't magic; it's a wrapper counting steps.reset() before step(). If you forget, it raises a clear error instead of a confusing one.Heads-up (nuance)
The exact path ._env is a private detail of gym-aloha — the underscore means "internal, may change." It works today, but it's the kind of thing that can break between versions. Public code usually avoids reaching into privates; we do it here only because we truly need the raw physics for the live viewer.
What does env.unwrapped do?
Answer: b. Wrappers stack around the core env; .unwrapped gives you the innermost env so you can reach things (like the physics) the wrappers hide.
Which wrapper is responsible for truncated becoming true?
Answer: a. truncated = "cut off from outside", and the outside cutoff is usually a TimeLimit wrapper counting steps. That ties Chapter 3's terminated/truncated to a concrete cause.
Why is reaching env.unwrapped._env.physics considered a bit fragile?
Answer: c. The leading underscore signals "internal". It works now, but library updates could rename it. Fine for our hands-on need; just don't be surprised if a future version differs.
✓ Chapter 7 checklist
What an image really is, the two ways to draw a sim, and why training almost always skips the window.
Our first script gave a frame of shape (480, 640, 3). That's not jargon — it's literally the picture's size:
Each of those numbers is 0–255 (a uint8 — an 8-bit unsigned integer). (0,0,0) is black, (255,0,0) is pure red (that's your cube!), (255,255,255) is white.
How big is one frame?
That's why images are "heavy" and why a neural network that reads images needs real computing power. A whole video is this, many times per second.
flowchart TD SIM["the simulation state
(MjData)"] --> OFF["OFFSCREEN
render_mode = rgb_array
→ returns a numbers array
→ save to file / feed a network"] SIM --> ON["ONSCREEN
a viewer window
→ you watch live
→ returns nothing to save"] OFF --> USE1["training, recording,
servers with no screen"] ON --> USE2["watching, debugging"]
Why training uses offscreen (headless)
Training runs the loop millions of times, often on servers that have no monitor at all ("headless"). Opening a window would be slow and impossible there. So training grabs the picture as numbers (rgb_array) — fast, and it works with no screen. Watching a window is only for humans debugging.
The fact that surprises people
gym-aloha only supports the offscreen (rgb_array) way. It has no built-in live window. So when we wanted a live window, we couldn't ask gym-aloha for one — we had to go below it and open a raw MuJoCo viewer ourselves. That single fact explains the whole mjpython adventure in the next chapter.
sleep(0.02)A moving picture is just still frames shown quickly. fps = frames per second. Film is ~24 fps; our sim targets 50 fps.
Where 0.02 comes from
That's exactly the time.sleep(0.02) in the live loop — it paces the loop to about 50 fps so motion looks smooth instead of flashing past.
To turn 3D shapes into pixels, the computer opens a connection to its graphics hardware — an OpenGL context. Key point for later: on macOS, the offscreen path uses a method (CGL) that needs no window, so it's calm and safe. The onscreen path needs a real window — and windows on macOS have a strict rule that causes the crash we'll meet next.
Preview of a pitfall (Ch 13 has the fix)
Saving .mp4 secretly needs a video tool called ffmpeg. If it's missing you'll get "No ffmpeg". And you may see a harmless warning about "macro_block_size 16" — that's just the video codec wanting width/height divisible by 16 (640×480 already is).
A frame has shape (480, 640, 3). What is the 3?
Answer: b. Height × Width × 3 (RGB). Each pixel is 3 numbers (0–255). The red cube is roughly (255, 0, 0).
Why does training almost always use offscreen (rgb_array) instead of a live window?
Answer: a. Training = millions of steps, often headless. Grabbing pixels as numbers is fast and needs no monitor. Live windows are just for human watching/debugging.
You want a live window of gym-aloha. What's the catch?
Answer: c. gym-aloha has no built-in window. To watch live you go below it to mujoco.viewer — which is why the next chapter's mjpython story exists.
✓ Chapter 8 checklist
The famous macOS crash, what a "thread" and "main thread" are, and why a special launcher exists.
What is it
a thread thread = a line of work
A program can do more than one thing at once by running several threads — separate lines of work happening in parallel. When a program starts, it has one thread already: the main thread. Extra threads can be created for background jobs.
Apple's rule: only the main thread is allowed to create or touch a window. A window object is called an NSWindow. If any other (background) thread tries to make a window, macOS refuses — loudly — by crashing the whole program with:
*** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'NSWindow should only be instantiated on the main thread!'
flowchart TD
subgraph MAIN["MAIN thread"]
UI["allowed: make windows (NSWindow) ✓"]
end
subgraph BG["BACKGROUND thread"]
WORK["allowed: run your script, do math ✓"]
BADWIN["NOT allowed: make a window ✗ → CRASH"]
end
mjpython existsA live MuJoCo viewer must show a window (main thread), but it also has to run your loop (which you'd normally run on your thread). To satisfy Apple's rule, MuJoCo ships a special launcher: mjpython. It arranges things so the window stays on the main thread and your script runs on a background thread — cooperating safely.
That's why plain python live_sim.py fails on a Mac with:
RuntimeError: launch_passive requires that the Python script be run under mjpython on macOS
Plain python has no such arrangement, so the viewer can't safely make its window. Use mjpython live_sim.py and it works.
What actually bit us (the real bug)
We did use mjpython, but still crashed. Why? Our first live script also set render_mode="rgb_array". That made the environment try to set up its own graphics on the background thread — on top of the viewer's window on the main thread. Two things fighting over graphics across threads → the NSWindow crash. The fix was to remove render_mode, leaving the viewer as the single owner of drawing. One drawer, no fight.
The clean mental rule
For a live window: run with mjpython, and let one thing own the graphics (the viewer). Don't also ask the env to render. For saving video: no window at all, so plain python is fine (that's the offscreen path from Chapter 8).
What is the macOS rule behind the NSWindow crash?
Answer: b. macOS requires all window (NSWindow) work on the main thread. A background thread making a window → instant crash.
What does mjpython do that plain python doesn't?
Answer: a. It arranges the threads so the viewer's window sits on the main thread (obeying Apple's rule) while your loop runs elsewhere. Plain python doesn't, so launch_passive refuses on macOS.
We ran with mjpython but still crashed. What fixed it?
Answer: c. With render_mode set, the env also tried to draw (on a background thread), fighting the viewer's window → crash. One owner of graphics (the viewer) → no fight → no crash.
✓ Chapter 9 checklist
seed=0Computer "random" isn't really random — and that's the feature that makes bugs findable.
What is it
pseudo-random pseudo = fake
Computers can't make true randomness. Instead they run a formula that spits out numbers that look random. The formula starts from one number called the seed. Same seed in → exact same sequence of "random" numbers out. Every time.
flowchart LR S0["seed = 0"] --> F["the RNG formula"] --> SEQ0["0.71, 0.13, 0.94, ...
(always this exact list)"] S1["seed = 7"] --> F2["the RNG formula"] --> SEQ1["0.42, 0.88, 0.05, ...
(a different fixed list)"]
That's why env.reset(seed=0) gives the same starting scene every run — the cube lands in the same spot, because the "random" placement grew from seed 0.
Beginners assume "surely it defaults to a fixed start." The opposite is true: if you give no seed, the environment picks a fresh, different one each run. And that default makes sense —
Why "random by default" is the right choice
A robot that only ever sees one starting position would just memorize that one case (this is called overfitting). To become genuinely skilled, it must practice on many different starts. So the sensible default is variety (random), and you only pin a seed on purpose — for debugging or fair comparisons.
Gotcha you'll meet later
There isn't just one source of randomness. Python's random, NumPy, PyTorch, and the environment each have their own RNG. Seeding only one leaves the others wandering. For a truly repeatable run you seed them all (and even then, GPU math can vary slightly). One seed is rarely enough.
Why does reset(seed=0) give the same starting scene every time?
Answer: c. Pseudo-randomness is deterministic given the seed. Seed 0 always produces the same sequence, so the same "random" cube position.
Why is the default (no seed) a different start each run?
Answer: a. One start → memorization (overfitting). Many starts → real skill. So "varied by default, pinned on purpose" is the sensible design.
You set reset(seed=0) but results still vary run to run. Likely reason?
Answer: b. Randomness comes from several libraries. Seeding only the env leaves the rest free. Full reproducibility means seeding them all (and GPU math may still wobble a little).
✓ Chapter 10 checklist
What conda actually does, why python suddenly works, and what an editable install is.
Key idea
an environment is a real folder
A conda "environment" isn't a metaphor — it's a folder on disk (like .../envs/lerobot/) containing its own python, its own installed packages, its own tools. Each project gets its own box, so their packages never fight.
flowchart TD E[".../envs/lerobot/"] --> B["bin/
python, pip, mjpython"] E --> L["lib/
installed packages
(torch, gym-aloha, ...)"] ACT["conda activate lerobot"] --> PATH["puts bin/ at the FRONT of PATH"]
bin/ to the front of the search path.PATH is the list of folders your Mac searches, front to back, when you type a command. When you type python, it uses the first python it finds along PATH.
flowchart TD
T["you type: python"] --> P1[".../envs/lerobot/bin ← activate put this first ✓ found!"]
P1 --> P2["/opt/homebrew/bin"]
P2 --> P3["/usr/bin (system python3)"]
This is why, after conda activate lerobot, plain python suddenly works and points to 3.12: the box's python is now first in line. Nothing was deleted or overwritten — conda deactivate puts it all back.
ffmpeg, or even different Python versions). Heavier, but great for scientific/robotics stacks.pip install -e .What the -e does
A normal install copies a package into your env. An editable install (-e) instead leaves the code where it is and drops a pointer ("the source lives over there") onto Python's import path. So when you edit the source, the change is live immediately — no reinstall. That's perfect for developing.
The catch (and why folder order mattered)
Because it's a pointer to a path, if you move or rename the folder after installing, the pointer breaks and import lerobot fails. That's exactly why, in the hands-on course, we fix the folder layout before installing — not after.
Extras, one more time
pip install -e ".[pusht,aloha]" — the [pusht,aloha] are extras: named optional add-on groups the package defines. You opt into the pieces you need (here, the two simulators) instead of installing everything. The quotes matter on a Mac — zsh treats [ ] as special without them.
What does conda activate actually change?
Answer: b. Activation just reorders the search path. Your Mac finds the env's python first. Deactivate reverses it — nothing is destroyed.
Why does moving the repo folder after an editable install break import lerobot?
Answer: c. -e records where the source lives. Move the source and that recorded path is wrong. Fix the layout before installing.
Which tool can also install non-Python things like ffmpeg?
Answer: a. pip and venv are Python-only. conda manages environments and can install system-level, non-Python dependencies — handy for robotics/scientific stacks.
✓ Chapter 11 checklist
What a policy is, how a neural network learns one, and the words (ACT, diffusion, loss…) you'll see next.
Key idea
policy = a function: observation → action
A policy is the robot's decision rule: given what it sees (obs), it outputs what to do (action). Everything we've run has been a policy — just simple ones.
flowchart LR O["observation
(camera + joint angles)"] --> POL["policy"] --> A["action
(14 numbers)"] POL -.simplest.-> R["random: ignore obs"] POL -.a plan.-> S["rule: sin(step) — our pat"] POL -.the goal.-> N["neural network: actually looks at obs"]
Our sine wave is a policy that ignores what it sees — it just sweeps on a timer. A real policy watches the cube and reacts. To get that, we don't write rules by hand (impossible for real tasks) — we learn the function from data. That learned function is a neural network.
Neural network in one breath
A neural network is a big adjustable function made of layers of multiply-add-and-bend steps, with millions of tunable numbers called weights. "Learning" = nudging those weights until the function's outputs match the examples. Its data lives in tensors (multi-dimensional number arrays — vectors and grids, generalized).
flowchart LR D["demo data
(obs → action pairs)"] --> PRED["network guesses an action"] PRED --> LOSS["loss:
how wrong was the guess?"] LOSS --> GRAD["gradient descent:
nudge weights to reduce loss"] GRAD --> PRED
The vocabulary, one line each
training = adjusting the weights from data · inference = using the finished network to act (no more changes) · loss = a number for "how wrong" · gradient descent = step the weights downhill to shrink the loss · learning rate = how big each step is (too big → unstable, too small → slow) · epoch = one full pass over the data · batch = a small chunk processed at once.
Names you'll meet in the next mission
ACT (Action Chunking Transformer) — an imitation policy that predicts a chunk of future actions at once (smoother, fewer compounding mistakes). Diffusion Policy — an imitation policy that "denoises" random noise into an action sequence; great when there are many valid ways to do a task. Dataset / Hugging Face Hub — where demo data and pretrained policies are shared and downloaded. Fine-tuning — taking a trained model and training it a little more on your own data (also a way to cross the sim-to-real gap). Checkpoint — a saved snapshot of the weights.
So the arc of this whole course: you'll first drive the arm with a rule policy (the sine "pat"). The next mission swaps in a learned policy — a neural network trained by imitation — and the arm starts doing real tasks, watching the cube instead of sweeping blindly.
Why is our sine-wave "pat" not a "real robot brain"?
Answer: c. A real policy maps obs → action (it reacts to what it sees). The sine wave outputs the same sweep no matter where the cube is. No looking = no real decision-making.
In one line, what is "loss"?
Answer: a. Loss quantifies error. Gradient descent nudges the weights to make loss smaller — that is learning.
Imitation learning differs from reinforcement learning because it…
Answer: b. Imitation = copy demos (no reward needed). RL = trial and error guided by a reward. ALOHA/ACT is an imitation-learning story.
✓ Chapter 12 checklist
Skim these now; they'll feel familiar the day they happen. Each is a tap-to-reveal symptom → cause → fix. You won't hit them all — but you'll hit some.
The golden path (prevents most of the list below)
If you follow this order once, you dodge the majority of these problems:
# one-time toolchain xcode-select --install # compiler + git brew install git-lfs ; git lfs install # environment conda config --set auto_activate_base false conda create -y -n lerobot python=3.12 conda activate lerobot # check: echo $CONDA_PREFIX conda install -c conda-forge ffmpeg # code + install (note the quotes!) cd ~/workspace && git clone https://github.com/huggingface/lerobot.git cd lerobot python -m pip install -U pip python -m pip install -e ".[pusht,aloha]" # sanity checks which python ; python -c "import lerobot; print('ok')" python -c "import torch; print(torch.backends.mps.is_available())"
conda: command not found right after installingconda init, so nothing put conda on your PATH.source ~/miniforge3/etc/profile.d/conda.sh (works now), then conda init zsh and exec zsh.conda init ran but new terminals still can't find conda~/.zprofile or a dotfile manager short-circuits before the conda block in ~/.zshrc.grep "conda initialize" ~/.zshrc; if present but ignored, source ~/miniforge3/etc/profile.d/conda.sh always works as a fallback.Run 'conda init' before 'conda activate'activate is a shell function, not the binary.source ~/miniforge3/etc/profile.d/conda.sh then activate; make it stick with conda init zsh && exec zsh. Never use the old source activate.(base) and you install into the wrong placeconda config --set auto_activate_base false, open a new terminal, and always conda activate lerobot.conda config --set solver libmamba (recent miniforge already defaults to it). Clear a bloated cache with conda clean --all.incompatible architecture (have 'x86_64', need 'arm64') / everything is slowconda info | grep platform (want osx-arm64) and uname -m (want arm64). If wrong, reinstall the Apple-Silicon miniforge (Miniforge3-MacOSX-arm64.sh).conda install and pip installconda env remove -n lerobot).CondaHTTPError / 403 fetching packagesconda config --add channels conda-forge ; conda config --set channel_priority strict.python: command not found (but python3 works)python. Outside an activated env there's only python3.python exists and points to the env's 3.12. Don't globally alias python=python3 (it hides "you forgot to activate").pip install succeeds but import failspip and python resolve to different installs (classic multi-Python PATH mixup).which python && which pip should agree. Safest habit: use python -m pip install ... so pip always matches the running python.which python still points to Homebrew/pyenvecho $CONDA_PREFIX (should end in /envs/lerobot). Keep the conda init block last; for this course, avoid installing pyenv.~/Library/Python/... / PEP 668 errorpython3 fell through to Apple's system Python.which python3 starts with your env path, not /usr/bin.brew install git-lfs ; git lfs install ; then in the repo git lfs pull. Install git-lfs before cloning next time.lerobot/lerobot/ (repo inside repo)git clone while already inside a folder of the same name.cd ~/workspace && git clone …. If nested, move the inner one up or delete and re-clone. Check pwd first.mkdir -p ~/workspace && cd ~/workspace before cloning. From inside a repo, git rev-parse --show-toplevel prints its root.git pops "command line developer tools" dialogxcode-select --install. Good first step anyway — it provides the compiler for later builds.command '/usr/bin/clang' failed / invalid active developer pathxcode-select --install; if still broken, sudo xcode-select --reset, then retry.Could not build wheels for mujocopython -m pip install -U pip setuptools wheel; use Python 3.12 (best wheel coverage); if needed pip install "mujoco>=3.0" first, then the extras.Can not perform a '--user' install--user inside a conda/virtual env — incompatible.--user: just pip install -e ".[pusht,aloha]".error: externally-managed-environment--break-system-packages on a dev Mac. This error usually means "you forgot to activate."ResolutionImpossible / pip backtracks through many versionspython -m pip install -U pip first. If a specific conflict is named, read which two packages disagree.WARNING: lerobot does not provide the extra 'X' → nothing installs".[pusht,aloha]". Full list is in pyproject.toml under [project.optional-dependencies].zsh: no matches found: .[pusht,aloha][ ] as a filename glob and tries to expand it.pip install -e ".[pusht,aloha]". (A Mac/zsh-specific trap.)does not appear to be a Python project on -e .. must contain pyproject.toml).cd ~/workspace/lerobot, confirm ls pyproject.toml, then install.ReadTimeoutError / downloads stall (torch is huge)pip install --timeout 120 …; if a cache got corrupted, pip cache purge then retry.import lerobot failspython -m pip show lerobot to confirm where it's installed; run imports from a directory other than the repo root.uv, tutorials say pip — confusionuv-based; conda+pip is a valid alternative but ignores uv.lock.pip install -e is fine — just don't paste uv sync/uv run into the same env.A module compiled using NumPy 1.x cannot run in NumPy 2.xpip install "numpy<2".Could not find a version that satisfies… on Python 3.13+conda create -n lerobot python=3.12. Best wheel coverage.requires a different Python: 3.11 not in '>=3.12'python=3.12. Don't rely on the system's older python3.Torch not compiled with CUDA enabled / "how do I use my GPU?"pip install torch (no CUDA index URL). Check torch.backends.mps.is_available(). Never copy the --index-url .../cu121 line from Linux tutorials.operator 'aten::…' is not implemented for the MPS deviceexport PYTORCH_ENABLE_MPS_FALLBACK=1 before running (unsupported ops fall back to CPU).libsvtav1 loading datasetsconda install -c conda-forge ffmpeg (its build includes the right codecs).cv2 import errors / segfaultspip uninstall opencv-python -y then conda install -c conda-forge "opencv=4.10.0"." or — instead of straight " / --.ModuleNotFoundError (the #1 mistake)conda activate lerobot; verify the (lerobot) prompt and echo $CONDA_PREFIX.pwd then cd ~/workspace/lerobot. Prefer absolute paths when in doubt."…" cannot be opened / "…" is damagedxattr -d com.apple.quarantine /path/to/file. Better: install tools via Terminal/Homebrew to avoid it.exec zsh.which python in each terminal.brew: command not found after installing Homebrew/opt/homebrew and isn't auto-added to PATH.~/.zprofile: eval "$(/opt/homebrew/bin/brew shellenv)", then exec zsh.launch_passive requires … mjpython on macOSmjpython arranges that for a live viewer.mjpython script.py (not python). Under uv: uv run mjpython script.py. (See Chapter 9.)python, or the loop exits immediately.mjpython; keep the loop alive with with mujoco.viewer.launch_passive(...) as v: while v.is_running(): ….NSWindow should only be instantiated on the main thread!python, or the env also renders (render_mode) while a live viewer is up.mjpython AND let only the viewer own graphics — drop render_mode when you open a live viewer. (Our exact bug; Chapter 9.)mjpython: command not foundmujoco isn't installed in the active env, or the env isn't active.python -c "import mujoco, shutil; print(shutil.which('mjpython'))".mjpython missing specifically under uvpip install --force-reinstall mujoco), ensure a recent 3.x.MUJOCO_GL?glfw=window, egl=GPU-offscreen, osmesa=CPU-offscreen).rgb_array, leave it unset — macOS offscreen uses CGL and just works. Setting egl/osmesa on a Mac usually breaks things.mujoco.FatalError: gladLoadGL errorMUJOCO_GL=egl python … (GPU) or MUJOCO_GL=osmesa (CPU). On macOS: unset MUJOCO_GL and use rgb_array.Cannot initialize a headless EGL displayMUJOCO_GL=osmesa (install libosmesa6-dev on Linux). Not a Mac issue.reset()/step(), so nothing was simulated yet.reset(), then step() at least once, then render(). Verify with print(frame.mean()) (should be > 0).MUJOCO_GL in code did nothingos.environ["MUJOCO_GL"]=… at the very top, before import mujoco / gym_aloha / dm_control — or export it in the shell.GLEW/libGL errorsmujoco-py binding, or broken Linux GL drivers.mujoco package (gym-aloha uses it). If mujoco-py appears anywhere, that's the red flag.NameNotFound: Environment 'AlohaTransferCube' doesn't existimport gym_aloha (registration happens on import), or used the wrong id.import gym_aloha before gym.make, and use the full id "gym_aloha/AlohaTransferCube-v0" (or .../AlohaInsertion-v0).MjModel/MjData but only have the envphysics = env.unwrapped._env.physics; then physics.model.ptr / physics.data.ptr. (Chapter 7. Note ._env is private and version-dependent.)top camera in its render path.physics.render(height=480, width=640, camera_id="angle") (use a camera name from the task's XML).end_effector_* task variants failtransfer_cube / insertion (the two published env ids).ImportError/AttributeError between mujoco / dm_control / gymnasiummujoco>=3.0,<3.9).~/.mujocopip install mujoco. No ~/.mujoco, no LD_LIBRARY_PATH.No ffmpeg exe could be found when writing .mp4pip install imageio-ffmpeg (bundles a binary) or conda install -c conda-forge ffmpeg. Verify: python -c "import imageio_ffmpeg; print(imageio_ffmpeg.get_ffmpeg_exe())".input image is not divisible by macro_block_size=16 warningmacro_block_size=1 to mimsave.imageio.mimsave("out.mp4", frames, fps=env.unwrapped.metadata["render_fps"]) (that's 50 for gym-aloha).ValueError about array shape / greenish videouint8 H×W×3 RGB arrays; you passed floats or an alpha channel.frame.dtype == uint8, shape (H,W,3). Convert floats: (frame*255).astype(np.uint8).imageio.mimsave("out.gif", frames, fps=…). For quality/size, prefer mp4 + imageio-ffmpeg.too many values to unpack on reset()reset() returns a 2-tuple (obs, info) (old gym returned just obs).obs, info = env.reset(seed=0).not enough values to unpack (expected 4, got 5) on step()done was split into terminated + truncated.obs, reward, terminated, truncated, info = env.step(action); then done = terminated or truncated.'…' object has no attribute 'seed'env.seed(...) was removed; seeding moved into reset.obs, info = env.reset(seed=42).action not within the bounds of the action space[-1, 1].action = np.clip(action, env.action_space.low, env.action_space.high).Box validation fails with a Python list actionBox spaces expect a numpy array of the right dtype, not a plain list.action = np.asarray(my_list, dtype=np.float32). Check with env.action_space.contains(action).env.render() returns None / warns about render_moderender().gym.make(id, render_mode="rgb_array"), then frame = env.render().import gymimport gym; gym-aloha registers against gymnasium.import gymnasium as gym everywhere.Cannot call env.step() before calling env.reset()OrderEnforcing wrapper (Chapter 7).env.reset() first.mps if it helps, keep sim on CPU, .numpy() actions before step; benchmark both. Guard with torch.backends.mps.is_available().random, torch) — seeding one leaves the rest free.random.seed(s); np.random.seed(s); torch.manual_seed(s); env.reset(seed=s); env.action_space.seed(s). (Chapter 10.)env.close() when done; reuse one env across episodes instead of recreating.Two decision aids to memorize
"Do I need mjpython?" Only for a live interactive window. Saving video / rgb_array → plain python is fine.
"Do I need MUJOCO_GL?" On a Mac with rgb_array: no (leave unset). On headless Linux: yes — egl (GPU) or osmesa (CPU).
You just want to save an mp4 of the arm. Do you need mjpython?
Answer: b. mjpython is only for a live window. No window = no main-thread rule = plain python works.
zsh: no matches found: .[pusht,aloha]. The fix?
Answer: a. zsh tries to glob [ ]. Quotes stop it. A trap almost every Mac beginner hits once.
✓ Chapter 13 checklist
A fast A–Z of the terms in this course. Come back whenever a word slips.
action — what the policy sends the robot each step (ALOHA: 14 numbers in −1…1).
action space — the rulebook of valid actions; ALOHA's is Box(-1,1,(14,)).
agent — the thing that acts; its decision-maker is the policy.
ALOHA — a low-cost two-arm (bimanual) robot from Stanford; gym-aloha is its simulator.
ACT — Action Chunking Transformer: an imitation-learning policy that predicts a chunk of actions at once. (An algorithm, not the robot.)
backronym — a name where the letters were fit to a word afterward (ALOHA).
behavioral cloning — simplest imitation learning: copy the human's action for each observation.
bimanual — two-armed; needs coordination, which makes tasks harder.
Box — a continuous space: any real number within low…high (e.g. −1…1).
checkpoint — a saved snapshot of a network's weights.
conda — a tool that manages isolated environments and packages (incl. non-Python ones).
contact — when two shapes touch; MuJoCo computes forces so they don't pass through.
ctrl — MuJoCo's control input; where your action enters the physics.
Discrete(n) — a space of fixed choices 0…n−1 (e.g. four buttons).
dm_control — DeepMind's wrapper around MuJoCo; gym-aloha sits on top of it.
editable install (-e) — install that points at your source so edits are live; breaks if you move the folder.
env — the environment object from gym.make: the whole world (arm + objects + physics).
episode — one attempt from reset to end (many steps).
extras — optional dependency groups, e.g. [pusht,aloha].
fine-tuning — training a pretrained model a bit more on your own data.
fps — frames per second; gym-aloha targets 50 (→ sleep(0.02)).
frame — one still picture; here a (480,640,3) array of RGB numbers.
gradient descent — nudging weights downhill to shrink the loss.
gripper — the arm's hand that opens/closes to hold objects.
Gymnasium — the standard API for environments (the maintained successor to OpenAI Gym).
headless — no screen; render to numbers, not a window (how servers train).
imageio — Python library to read/write images and video.
imitation learning — learn by copying human demonstrations (no reward needed).
inference — using a trained network to act (no more weight changes).
learning rate — how big each gradient-descent step is.
loss — a number for how wrong a prediction is; training shrinks it.
main thread — a program's first line of work; macOS only lets it make windows.
MJCF — MuJoCo's XML file describing the robot/scene.
MjModel / MjData — the constant blueprint vs the changing live state (qpos, qvel).
mjpython — MuJoCo's macOS launcher that keeps the viewer window on the main thread.
MPS — Apple's Metal GPU backend for PyTorch (Macs have no CUDA).
MuJoCo — the physics engine (Multi-Joint dynamics with Contact).
neural network — a big adjustable function with learnable weights.
normalize — rescale values to a common range (here −1…1) for stable learning.
observation — what the agent sees each step (camera image + joint positions).
OpenGL context — the connection to graphics hardware needed to draw pixels.
overfitting — memorizing the training cases instead of learning the general skill.
PATH — the ordered list of folders the shell searches for a command.
policy — the robot's brain: a function observation → action.
pseudo-random — "random" numbers made by a formula from a seed (repeatable).
qpos / qvel — MuJoCo's current joint positions / velocities.
radians — angle unit where a full turn is 2π (robots use these).
reinforcement learning — learn by trial and error to maximize reward.
render / render_mode — draw the scene; rgb_array (pixels) or a window.
reset — start a fresh episode; returns (obs, info).
reward — a score for an action (RL); ALOHA's top reward means success.
seed — the starting number for pseudo-randomness; fixes the "random" sequence.
sim-to-real gap — the mismatch between simulator physics and reality.
step — one loop turn: apply action, advance physics, return the 5-tuple.
tensor — a multi-dimensional array of numbers (deep learning's basic data).
terminated — the episode ended by the task's own rules (success/failure).
truncated — the episode was cut off from outside (usually a time limit).
teleoperation — a human remotely driving the robot to record demonstrations.
unwrapped — env.unwrapped: peel off wrappers to reach the core env.
viewer — MuJoCo's live window (launch_passive shows our steps).
wrapper — a layer around something that adds convenience (gym → dm_control → MuJoCo).
zsh — the default macOS shell; conda activate relies on its init.
Twelve questions across the whole course. No pressure — guess, tap, and let the explanations close any gaps.
Why train robots in simulation first?
b — learning needs many failures; sim makes them free and safe. Reality is actually the accurate one (the sim-to-real gap).
What makes the arm move?
a — motion = stepping the physics in a loop. No loop, no motion.
env.reset() returns…
c — no action has happened, so no reward. Reset gives the starting observation.
Ran out of steps before finishing. Which is true?
b — a time/step limit is an external cutoff → truncated. terminated is for a real ending.
Why is every action number in −1…1?
a — real ranges differ wildly; one −1…1 ruler stabilizes learning. The sim converts back to radians.
ALOHA vs ACT?
c — body vs brain. Same paper, different roles.
During simulation, which changes each step?
b — the blueprint (MjModel) is fixed; the live state (MjData) changes.
Why does a live viewer need mjpython on macOS?
a — the window must live on the main thread; mjpython sets that up while your script runs elsewhere.
Live viewer crashed with NSWindow even under mjpython. Fix?
c — two things fighting over graphics across threads → crash. One owner (the viewer) fixes it.
reset(seed=0) with no other seeding still varies. Why?
b — randomness comes from several libraries; seed them all for full reproducibility.
Why does moving the repo after pip install -e break imports?
a — -e records the source path; move it and the pointer dangles. Fix layout before installing.
What turns our sine-wave "pat" into a real robot brain?
c — the sine wave ignores what it sees. A trained policy watches the world and reacts. That's the next mission.
✓ You made it
🎓 → 🦾
You're ready to build.
Now do it for real: the hands-on course walks you through setup and makes the arm give you a pat, on video. Open “Pat me on the back” and go.