Why teach a robot to walk instead of programming it?

Walking is harder than it looks

You learned to walk before you could talk, so it feels easy. For a robot it is not. Four legs, eight motors, a body that wants to tip over, a floor that is sometimes slippery and sometimes sticky, and motors that are slow and not very precise. Every 20 milliseconds the robot has to decide where all eight joints should go next.

There are two big ways to make those decisions:

1. Write the rules yourself. You decide how each leg moves: lift, swing forward, put down, push back. You tune the numbers until it works.
2. Let the robot learn the rules. You describe what “good walking” means as a score, and an algorithm searches for a controller that gets a high score, by trying things millions of times.

This course is about the second way, done honestly: what it is good at, what it costs, and where it can fail.

The robot we build on

The course uses the four-legged robot from the Quadruped Walking Robot (Sesame-based) guide: eight MG90S hobby servos, a LOLIN S2 Mini board (an ESP32-S2) running MicroPython, and an MPU6500 IMU that measures tilt and turning. Each leg has two servos:

Servo	Names	Job
hip	R1, R2, L1, L2	swings the leg forward and back
leg	R4, R3, L3, L4	lifts the foot and puts it down

R1 and L1 are the front hips, R2 and L2 the rear ones. You do not need the robot for the first eight lessons: everything until lesson 9 happens on your computer. But everything we build is shaped to fit this robot: its joint order, its angle limits, its 50 Hz control loop.

The hand-written gait it already has

The guide’s firmware already walks. Its gait.py describes a trot: diagonal legs move together, and each leg follows a fixed cycle.

$$\text{swing:}\quad \text{hip} = -A\cos(\pi u),\quad \text{lift} = H\sin(\pi u) \qquad \text{stance:}\quad \text{hip} = A\cos(\pi u),\quad \text{lift} = 0$$

with a stride $A$ = 25°, a lift $H$ = 40° and a period of 0.8 s. That is about fifteen lines of Python, and it works. So why learn anything?

	Hand-written gait	Learned policy
How you make it	think, write formulas, tune numbers	design a reward, train for hours
What it reacts to	the clock (and the IMU in balance mode)	everything you give it: tilt, turning, its last move, a command
On a new surface	you retune by hand	it may cope, if training included such surfaces
When it fails	you can read the code and see why	it is a table of thousands of numbers: hard to explain
Risk	predictable, smooth	can make sudden, strange moves
Cost	your time	computer time, and a lot of your time debugging rewards

Training a dog: the idea of reinforcement learning

Think about how you teach a dog to sit. You cannot explain “bend your back legs and lower your hips”. Instead:

1. The dog is in some situation: standing in front of you, you holding a treat.
2. It does something: sits, jumps, barks, walks away.
3. You react: a treat when it sits, nothing otherwise.
4. Repeat many times. The dog slowly learns which actions in which situations lead to treats.

The dog never sees the rule. It discovers it by trying, from the treats alone. That is reinforcement learning (RL): learning what to do from a reward signal, by trial and error.

Two details of dog training matter a lot for robots too:

• Timing. A treat ten minutes later teaches nothing. Rewards are clearest when they come right after the action that earned them. When they come later, the learner has to work out which of its many earlier actions deserved them. That is called the credit assignment problem, and lesson 2 shows how RL deals with it.
• You get what you reward, not what you mean. Reward the dog for being quiet, and you may get a dog that sleeps all day. Robots are worse: they find every loophole in your score. Lesson 6 shows several such loopholes for a walking robot.

The same idea, precisely

Now without the dog. A reinforcement learning problem has an agent (the controller) and an environment (the robot’s body and the world). Time goes in steps $t = 0, 1, 2, \dots$. On every step:

1. The agent receives an observation $s_t$ of the state: a list of numbers describing the situation.
2. It chooses an action $a_t$: another list of numbers.
3. The environment moves on one step and returns a reward $r_t$ (one number) and the next observation $s_{t+1}$.

A run from a start until the end is an episode. An episode ends when the robot falls over (a termination), or when a time limit is reached (a truncation). Then everything is reset and a new episode starts.

The agent’s goal is not the biggest reward on the next step, but the biggest return: the sum of rewards over the episode,

$$G = r_0 + r_1 + r_2 + \dots + r_{T-1}$$

(lesson 2 adds a discount factor that makes later rewards count a little less). A big return means walking well for a long time, not a lucky lurch forward followed by a fall.

For our quadruped, the four words mean this:

RL word	For the quadruped	Example numbers
state / observation	tilt and turning rate from the IMU, the eight joint targets and how fast they change, the last action, the commanded speed, a clock	33 numbers
action	how far to move each of the eight joints from a standing pose	8 numbers between −1 and 1
reward	forward speed (up to the command), minus penalties for wasted energy, jerky moves, tilting, falling	one number per step, 50 per second
episode	one attempt from standing until it falls or 10 seconds pass	up to 500 steps

What is a policy?

The thing that RL produces is a policy: a rule that maps an observation to an action,

$$a_t = \pi(s_t)$$

$\pi$ is just the traditional letter for it. A policy can be anything that turns numbers into numbers: a table, a formula, or a small program. The hand-written trot is a policy too: its observation is the clock, its action is eight angles.

For walking, the observation has 33 numbers, and they can take any values. No table can list them all. So the policy is a neural network: a function with a few thousand adjustable numbers inside (called weights). Training adjusts the weights until the network’s actions get high returns. Ours is small on purpose: 33 inputs, two layers of 64 units, 8 outputs, about 6,900 numbers in total. That is small enough to run on the S2 Mini itself, as lesson 9 shows. Lesson 3 explains how such a network computes, with nothing but multiplications, additions and one squashing function.

During training, the policy is random on purpose: it picks actions from a bell curve around its best guess, so that it keeps trying slightly different things. When we run it on the robot, we switch the randomness off and always take the best guess.

Why we train in a simulator

A simulator is a program that computes how the robot would move: gravity, contacts with the floor, friction, the servos pushing on the joints. We will use MuJoCo, a free physics engine that is widely used for robot learning. Three reasons to train in it:

1. It does not break. The policy will fall over thousands of times while it learns. Real servos would strip their gears, the frame would crack, and you would spend the week picking up the robot.
2. It is fast and parallel. The computer can simulate many robots at once, faster than real time. Training needs millions of steps; on the real robot that would be days of non-stop walking.
3. It is measurable. In the simulator you know everything exactly: the true speed, the true tilt, the force on every foot. That makes rewards easy to compute and bugs easier to find.

The price is the reality gap: the simulator is never exactly the real robot. Real MG90S servos are slow, have a small dead zone, and get weaker when the battery runs down; the real floor is not a perfect plane; the IMU is noisy. A policy can learn to exploit details of the simulator that do not exist in reality. Lesson 9 is all about narrowing that gap, lesson 10 about the IMU’s part of it, and lesson 11 about testing on the robot carefully.

Safety from day one

A learned policy is a black box that can make sudden moves. Servos are strong enough to pinch fingers, and a robot that lurches can walk off a table. From lesson 11 on, every program that drives the real robot has joint limits it cannot exceed, a speed limit, a stop button, a tilt check and a time limit, and the first tests happen with the robot on a stand, legs in the air. The power switch on the battery is your emergency stop: know where it is before you start.

What the course builds

By the end you will have:

• a MuJoCo model of the quadruped, with the same joint names, order and limits as the real servos;
• the hand-written trot running in simulation, measured as a baseline;
• a Gymnasium environment that turns the model into an RL problem, with observations the real robot can actually measure;
• a PPO training script (with Stable-Baselines3) and the plots to see whether it learns;
• a domain-randomised version of the environment for a better chance on the real robot;
• the trained network exported to a small file and a forward pass in plain MicroPython that runs on the S2 Mini;
• the IMU chain from chip to observation, with its signs, mounting and filter checked against the simulator;
• a safe test procedure on the robot, logging, and a fair comparison with the hand-written trot;
• policies that start from the trot: a cloned copy fine-tuned with PPO, and a small learned correction on top of it;
• a heat index for every servo, a reward term against overloading one leg, and a plan for measuring current and temperature;
• speed and turning commands, trained with a curriculum.

Check yourself

What the full course covers

The full course has fourteen lessons, and you also get the whole course as a PDF by email:

1. Why teach a robot to walk instead of programming it? (this chapter)
2. Reinforcement learning on a tiny problem: returns, values and Q-learning
3. Just enough neural network: a policy is a function with knobs
4. A robot in a box: simulating the quadruped in MuJoCo
5. Legs, angles and a baseline: the hand-written trot in simulation
6. What the robot feels, does and wants: observations, actions and rewards
7. Training a walking policy with PPO
8. Is it learning? Plots, reward debugging and fair comparisons
9. Mind the gap: domain randomisation and a policy small enough for the ESP32-S2
10. Which way is down? The IMU on the real robot
11. On the robot: safe deployment, logging and the first steps
12. Start from the trot: behaviour cloning and residual policies
13. Warm servos: energy, current and heat
14. Steering, curriculum and your final project