We aim to build complex humanoid agents that integrate perception, motor control, and memory. In this work, we partly factor this problem into low-level motor control from proprioception and high-level coordination of the low-level skills informed by vision. We develop an architecture capable of surprisingly flexible, task-directed motor control of a relatively high-DoF humanoid body by combining pre-training of low-level motor controllers with a high-level, task-focused controller that switches among low-level sub-policies. The resulting system is able to control a physically-simulated humanoid body to solve tasks that require coupling visual perception from an unstabilized egocentric RGB camera during locomotion in the environment. Supplementary video link:  https://youtu.be/fBoir7PNxPk In reinforcement learning (RL), a major challenge is to simultaneously cope with high-dimensional input and high-dimensional action spaces. As techniques have matured, it is now possible to train high-dimensional vision-based policies from scratch to generate a range of interesting behaviors ranging from game-playing to navigation BID17 BID32 BID41 . Likewise, for controlling bodies with a large number of degrees of freedom (DoFs), in simulation, reinforcement learning methods are beginning to surpass optimal control techniques. Here, we try to synthesize this progress and tackle high-dimensional input and output at the same time. We evaluate the feasibility of full-body visuomotor control by comparing several strategies for humanoid control from vision.Both to simplify the engineering of a visuomotor system and to reduce the complexity of taskdirected exploration, we construct modular agents in which a high-level system possessing egocentric vision and memory is coupled to a low-level, reactive motor control system. We build on recent advances in imitation learning to make flexible low-level motor controllers for high-DoF humanoids. The motor skills embodied by the low-level controllers are coordinated and sequenced by the high-level system, which is trained to maximize sparse task reward.Our approach is inspired by themes from neuroscience as well as ideas developed and made concrete algorithmically in the animation and robotics literatures. In motor neuroscience, studies of spinal reflexes in animals ranging from frogs to cats have led to the view that locomotion and reaching are highly prestructured, enabling subcortical structures such as the basal ganglia to coordinate a motor repertoire; and cortical systems with access to visual input can send low complexity signals to motor systems in order to evoke elaborate movements BID7 BID1 BID9 .The study of "movement primitives" for robotics descends from the work of BID16 . Subsequent research has focused on innovations for learning or constructing primitives for control of movments BID15 BID20 ), deploying and sequencing them to solve tasks BID36 BID19 BID22 , and increasing the complexity of the control inputs to the primitives BID31 . Particularly relevant to our cause is the work of BID21 in which primitives were coupled by reinforcement learning to external perceptual inputs.Research in the animation literature has also sought to produce physically simulated characters capable of distinct movements that can be flexibly sequenced. This ambition can be traced to the virtual stuntman BID6 a) and has been advanced markedly in the work of Liu BID27 . Further recent work has relied on reinforcement learning to schedule control policies known as "control fragments", each one able to carry out only a specialized short movement segment BID24 . In work to date , such control fragments have yet to be coupled to visual input as we will pursue here. From the perspective of the RL literature BID38 , motor primitives and control fragments may be considered specialized instantiations of "option" sub-policies.Our work aims to contribute to this multi-disciplinary literature by demonstrating concretely how control-fragment-like low-level movements can be coupled to and controlled by a vision and memory-based high-level controller to solve tasks. Furthermore, we demonstrate the scalability of the approach to greater number of control fragments than previous works. Taken together, we demonstrate progress towards the goal of integrated agents with vision, memory, and motor control. In this work we explored the problem of learning to reuse motor skills to solve whole body humanoid tasks from egocentric camera observations. We compared a range of approaches for reusing lowlevel motor skills that were obtained from motion capture data, including variations related to those presented in BID24 BID34 . To date, there is limited learning-based work on humanoids in simulation reusing motor skills to solve new tasks, and much of what does exist is in the animation literature. A technical contribution of the present work was to move past hand-designed observation features (as used in BID34 ) towards a more ecological observation setting: using a front-facing camera is more similar to the kinds of observations a real-world, embodied agent would have. We also show that hierarchical motor skill reuse allowed us to solve tasks that we could not with a flat policy. For the walls and go-to-target tasks, learning from scratch was slower and produced less robust behavior. For the forage tasks, learning from scratch failed completely. Finally, the heterogeneous forage is an example of task that integrates memory and perception.There are some other very clear continuities between what we present here and previous work. For learning low-level tracking policies from motion capture data, we employed a manually specified similarity measure against motion capture reference trajectories, consistent with previous work BID26 BID34 . Additionally, the low-level policies were time-indexed: they operated over only a certain temporal duration and received time or phase as input. Considerably less research has focused on learning imitation policies either without a pre-specified scoring function or without time-indexing (but see e.g. ). Compared to previous work using control fragments BID24 , our low-level controllers were built without a sampling-based planner and were parameterized as neural networks rather than linear-feedback policies.We also want to make clear that the graph-transition and steerable structured low-level control approaches require significant manual curation and design: motion capture clips must be segmented by hand, possibly manipulated by blending/smoothing clips from the end of one clip to the beginning of another. This labor intensive process requires considerable skill as an animator; in some sense this almost treats humanoid control as a computer-aided animation problem, whereas we aim to treat humanoid motor control as an automated and data-driven machine learning problem. We acknowledge that relative to previous work aimed at graphics and animation, our controllers are less graceful. Each approach involving motion capture data can suffer from distinct artifacts, especially without detailed manual editing -the hand-designed controllers have artifacts at transitions due to imprecise kinematic blending but are smooth within a behavior, whereas the control fragments have a lesser but consistent level of jitter throughout due to frequent switching. Methods to automatically (i.e. without human labor) reduce movement artifacts when dealing with large movement repertoires would be interesting to pursue.Moreover, we wish to emphasize that due to the human-intensive components of training structured low-level controllers, fully objective algorithm comparison with previous work can be somewhat difficult. This will remain an issue so long as human editing is a significant component of the dominant solutions. Here, we focused on building movement behaviors with minimal curation, at scale, that can be recruited to solve tasks. Specifically, we presented two methods that do not require curation and can re-use low-level skills with cold-switching. Additionally, these methods can scale to a large number of different behaviors without further intervention.We view this work as an important step toward the flexible use of motor skills in an integrated visuomotor agent that is able to cope with tasks that pose simultaneous perceptual, memory, and motor challenges to the agent. Future work will necessarily involve refining the naturalness of the motor skills to enable more general environment interactions and to subserve more complicated, compositional tasks.
Solve tasks involving vision-guided humanoid locomotion, reusing locomotion behavior from motion capture data.
{"src": "scitldr-AIC_train", "paper_id": "BJfYvo09Y7"}