The reinforcement learning paradigm allows, in principle, for complex behaviours to be learned directly from simple reward signals. In practice, however, it is common to carefully hand-design the reward function to encourage a particular solution, or to derive it from demonstration data. In this paper explore how a rich environment can help to promote the learning of complex behavior. Specifically, we train agents in diverse environmental contexts, and find that this encourages the emergence of robust behaviours that perform well across a suite of tasks. We demonstrate this principle for locomotion -- behaviours that are known for their sensitivity to the choice of reward. We train several simulated bodies on a diverse set of challenging terrains and obstacles, using a simple reward function based on forward progress. Using a novel scalable variant of policy gradient reinforcement learning, our agents learn to run, jump, crouch and turn as required by the environment without explicit reward-based guidance. A visual depiction of highlights of the learned behavior can be viewed following this https URL .
Basic locomotion behaviors learned end-to-end via RL have been demonstrated, for instance, by @cite_12 @cite_7 @cite_10 @cite_0 or guided policy search @cite_16 . Locomotion in the context of higher-level tasks has been considered in @cite_4 . Terrain-adaptive locomotion with RL has been demonstrated by @cite_6 , but they still impose considerable structure on their solution. Impressive results were recently achieved with learned locomotion controllers for a 3D humanoid body @cite_25 , but these rely on a domain-specific structure and human motion capture data to bootstrap the movement skills for navigating flat terrains.
@cite_4 @cite_7 @cite_6 @cite_0 @cite_16 @cite_10 @cite_25 @cite_12
2739330054 2592891024 2107822562 2119468631
Learning physics-based locomotion skills is a difficult problem, leading to solutions that typically exploit prior knowledge of various forms. In this paper we aim to learn a variety of environment-aware locomotion skills with a limited amount of prior knowledge. We adopt a two-level hierarchical control framework. First, low-level controllers are learned that operate at a fine timescale and which achieve robust walking gaits that satisfy stepping-target and style objectives. Second, high-level controllers are then learned which plan at the timescale of steps by invoking desired step targets for the low-level controller. The high-level controller makes decisions directly based on high-dimensional inputs, including terrain maps or other suitable representations of the surroundings. Both levels of the control policy are trained using deep reinforcement learning. Results are demonstrated on a simulated 3D biped. Low-level controllers are learned for a variety of motion styles and demonstrate robustness with respect to force-based disturbances, terrain variations, and style interpolation. High-level controllers are demonstrated that are capable of following trails through terrains, dribbling a soccer ball towards a target location, and navigating through static or dynamic obstacles. This work introduces a novel hybrid control architecture for a hexapod platform (Weaver), making it capable of autonomously navigating in uneven terrain. The main contribution stems from the use of vision-based exteroceptive terrain perception to adapt the robot's locomotion parameters. Avoiding computationally expensive path planning for the individual foot tips, the adaptation controller enables the robot to reactively adapt to the surface structure it is moving on. The virtual stiffness, which mainly characterizes the behavior of the legs' impedance controller is adapted according to visually perceived terrain properties. To further improve locomotion, the frequency and height of the robot's stride are similarly adapted. Furthermore, novel methods for terrain characterization and a keyframe based visual-inertial odometry algorithm are combined to generate a spatial map of terrain characteristics. Localization via odometry also allows for autonomous missions on variable terrain by incorporating global navigation and terrain adaptation into one control architecture. Autonomous runs on a testbed with variable terrain types illustrate that adaptive stride and impedance behavior decreases the cost of transport by 30 compared to a non-adaptive approach and simultaneously increases body stability (up to 88 on even terrain and by 54 on uneven terrain). Weaver is able to freely explore outdoor environments as it is completely free of external tethers, as shown in the experiments. Navigating through large virtual environments using a head-mounted display (HMD) is difficult due to the spatial limitations of the tracking system. We conducted two experiments to examine methods of exploring large virtual spaces with an HMD under translation conditions different than normal walking. Experiment 1 compares locomotion in the virtual environment using two different motor actions to translate the subject. The study contrasts user learning and orientation of two different translational gains of bipedal locomotion (not scaled and scaled by ten) with joystick locomotion, where rotation in both locomotion interfaces is accomplished by physically turning. Experiment 2 looks further at the effects of increasing the translational gain of bipedal locomotion in a virtual environment. A subject's spatial learning and orientation were evaluated in three gain conditions where each physical step was: not scaled, scaled by two, or scaled by ten (1:1, 2:1, 10:1, respectively). A sub-study of this experiment compared the performance of people who played video games against people who did not. To compare and evaluate locomotion interfaces for users who are (virtually) moving on foot in VEs, we performed a study to characterize task behavior and task performance with different visual and locomotion interfaces. In both a computer-generated environment and a corresponding real environment, study participants walked to targets on walls and stopped as close to them as they could without making contact. In each of five experimental conditions participants used a combination of one of three locomotion interfaces (really walking, walking-in-place, and joystick flying), and one of three visual conditions (head-mounted display, unrestricted natural vision, or field-of-view-restricted natural vision). We identified metrics and collected data that captured task performance and the underlying kinematics of the task. Our results show: 1) Over 95 of the variance in simple motion paths is captured in three critical values: peak velocity; when, in the course of a motion, the peak velocity occurs; and peak deceleration. 2) Correlations of those critical value data for the conditions taken pairwise suggest a coarse ordering of locomotion interfaces by "naturalness." 3) Task performance varies with interface condition, but correlations of that value for conditions taken pairwise do not cluster by naturalness. 4) The perceptual variable, r (also known as the time-to-contact) calculated at the point of peak deceleration has higher correlation with task performance than r calculated at peak velocity.