LEARNING TO QWOP
Highlighted keys show controller inputs. Press Q, W, O or P to take control.
Control a simulated runner with Q, W, O and P. Compare a scripted controller with three stages of neural network training: behavior cloning, DAgger and PPO. PPO runs 100 m in 18.6 seconds with a galloping gait. A fifth model, trained to alternate its legs and keep its knees off the ground, takes 20.0 seconds.
How it was trained
Training uses a custom rigid-body simulator in Python. This page uses a matching JavaScript version. Each neural network has 77 inputs and two hidden layers with tanh activations. It selects one of nine key combinations 30 times per second.
PPO reached 21.4 seconds per 100 m after 300 million steps on a laptop CPU. Another 4.4 billion steps on a GPU reduced the time to 18.6 seconds. At roughly 130 million steps, PPO initialized from DAgger reached 22.1 seconds, compared with 38.0 seconds from random weights. This comparison used one run per setup. Penalizing knee contact produced an upright gait at about one-third the speed.
The 18.6-second PPO gait is a gallop: the right leg leads 78% of the time and a knee touches the ground on 8% of steps. The alternating-stride model pays most of the progress reward per stride, and a stride counts only when a foot lands ahead of the other foot and the two feet take turns. Knee contact ends the episode. It runs 100 m in 20.0 seconds with no knee contact, and each leg leads about half the time.
Statistics use 2,048 runs per controller with randomized starting poses and a 100-second limit. Training code: qwop/train.