- Why teach a robot to walk instead of programming it?Hand-written gaits work, until the floor changes. This chapter explains the reinforcement learning loop with a dog-training analogy and then precisely, what a policy is, and why we train in simulation first.
- Reinforcement learning on a tiny problem: returns, values and Q-learningBefore eight servos, a 4 by 4 floor with two holes. Discounted returns, value functions and Q-learning, worked out by hand and then in 50 lines of Python.
- Just enough neural network: a policy is a function with knobsWhat happens inside the 33-to-8 network that will drive the robot: neurons, layers, tanh, the forward pass by hand, counting parameters, and gradients. Then a tiny network learns the hand-written trot in plain numpy.
- A robot in a box: simulating the quadruped in MuJoCoInstall MuJoCo and Gymnasium, write a complete MJCF model of an 8-servo quadruped with the real robot's joint names, order and limits, step it, and understand the difference between the physics timestep and the control rate.
- Legs, angles and a baseline: the hand-written trot in simulationForward kinematics of one leg, checked against MuJoCo; from hip swing to step length to an expected speed; then the guide's trot running in the simulator, measured, as the baseline every learned policy must beat.
- What the robot feels, does and wants: observations, actions and rewardsDesign an observation the real robot can measure, an action that cannot jerk the servos, and a reward that asks for walking and not for a loophole. Then wrap it all in a Gymnasium environment and test it with random actions.
- Training a walking policy with PPOHow PPO learns: policy gradients, advantages and the clipped update, with light maths. Then training with Stable-Baselines3: parallel environments, normalisation, the numbers in the log, saving, and what to do when nothing is learned.
- Is it learning? Plots, reward debugging and fair comparisonsA rising return is not proof of walking. Plot the learning curve and every reward term, watch the policy, measure it with fixed commands and evaluation seeds, run the hand-written trot through the same environment, and catch reward hacking before it reaches the robot.
- Mind the gap: domain randomisation and a policy small enough for the ESP32-S2Why a policy that walks in MuJoCo may stumble on hobby servos, how randomising friction, mass, servo strength, delay and sensor noise makes it tougher, and how to export the network to a 27 KB file and run it with plain loops in MicroPython, checked number by number against the original.
- Which way is down? The IMU on the real robotSix of the policy's 33 numbers come from the IMU. Follow them from the chip to the observation: units, sign conventions checked against the simulator, how the chip is mounted, calibration while still, the complementary filter and its time constant, what footsteps do to it, and a blind test that shows how much the policy actually uses them.
- On the robot: safe deployment, logging and the first stepsRun the learned policy on the quadruped in four careful stages: checks on the bench, legs in the air on a stand, short runs on the floor, then a fair comparison with the hand-written trot. With limits, a stop button, a tilt check, a time limit and a log of every step.
- Start from the trot: behaviour cloning and residual policiesTraining from scratch found small, fast steps that hobby servos may not follow. Two ways to start from the gait that already walks instead: copy it into the policy network and let PPO improve it, or keep the trot running and learn only a small correction on top. Both trained, measured on random robots and compared.
- Warm servos: energy, current and heatWhere a servo's electrical power goes, why holding still can heat it while lesson 6's energy term sees nothing, a heat index for every servo in the simulator, a policy that leans hard on one leg, a reward term against it, and how to measure current and temperature on the robot safely.
- Steering, curriculum and your final projectGive the policy a second command, turning, and train it with a curriculum that starts easy and gets harder only when the robot is ready. Then choose a final project, and look back at the whole path from a grid world to a robot on the floor.