As a comparison, we try the more straightforward approach: training one policy directly from scratch with varying desired speed at each episode, but without any distillation. We test it on the same desired speed profile. Although this policy does learn to change the stepping frequency and the swing ratio, it fails to learn a distinctive walking gait at the lower speed, and keeps trotting throughout the episode (Fig. REF ). As a result, it exhibits lower energy efficiency than our distilled policy (Fig. REF ). We hypothesize that this is because the RL agent gets stuck at a local optimum. As noted in [1]}, gait transition usually occurs abruptly at discrete velocities, and the optimal gaits can be quite different (e.g. walking vs trotting). Therefore, the optimization landscape is often jaggy and full of local optima. Learning distinctive gaits and transitioning between them requires the agent to conduct a more global search in the controller space, which is challenging for the state-of-the-art RL algorithms. We use distillation from expert policies to overcome the this challenge.
