Reinforcement learning tutorials usually stop at the simulator. The reward curve climbs, the agent balances the pole, everyone moves on. Shawn Hymel’s “Train a Balance Bot With Reinforcement Learning,” published October 26 in Make: Volume 99, keeps going past that point — training a policy entirely in simulation and then flashing it to an M5Stack Bala2-Fire, a real two-wheeled robot that has to balance in the actual world, with actual sensor noise and actual motor slop the simulator never promised would behave.
The toolchain is entirely open and named explicitly: training happens in Google Colab or a local Jupyter notebook, using the MuJoCo physics engine with an MJCF model of the robot, wrapped in a custom Gymnasium environment (balance_bot_env.py). The policy itself trains with CleanRL’s ppo_continuous_action.py, modified into ppo_trainer.py — Proximal Policy Optimization, the same algorithm family behind a lot of recent sim-to-real robotics work, chosen here because it tolerates the kind of noisy, continuous control problem a balancing robot actually is. Separate actor and critic networks, two hidden layers of 32 nodes each, train across 8 parallel environments for 500,000 timesteps, watched live through TensorBoard.
The reward function is where the actual design work lives. It’s not just “stay upright” — it’s a weighted sum: an alive bonus of 1.0, a pitch penalty of 0.5, a speed penalty of 20.0 (the heaviest term, discouraging frantic overcorrection), plus smaller penalties on action magnitude, position drift and yaw. Get those weights wrong and the robot either oscillates wildly trying to minimize pitch, or drifts off indefinitely because nothing is punishing it for wandering. An episode resets the moment the robot tips past 30 degrees.
The part that actually makes sim-to-real work is a three-phase curriculum, introduced gradually rather than all at once: first, learn to balance at all, in a clean simulation. Second, control domain randomization — inject sensor noise, up to a millisecond of action delay, motor noise, varied motor torque. Third, physical domain randomization — random nudges, varied part mass, varied wheel-ground friction. Skip straight to a clean sim and train, and the policy that comes out is often brittle: perfect in MuJoCo, and immediately knocked over by the first real-world wobble a clean simulation never modeled. The randomization phases are what close that gap.
Deployment strips the trained actor network down (the critic is discarded entirely — it only ever existed to help training, not to run the robot), exports it to ONNX, inspectable in Netron, and converts it to a C header (actor.h) that gets flashed onto the ESP32 via the Arduino IDE with the M5Unified library. A 5ms control loop reads the IMU and both motor encoders, runs the tiny network, and maps its continuous output to PWM motor commands — no retraining, no fine-tuning on the physical unit, just the policy the simulator already learned.
Full source, including the MJCF model, the Gymnasium environment and the training/export scripts, is on GitHub. The project is explicitly framed as something that “should take only a few hours” for someone comfortable with Python — not a research artifact, a tutorial with a working repo behind it.