We introduce Lyceum, a high-performance computational ecosystem for robotlearning.    Lyceum is built on top of the Julia programming language and theMuJoCo physics simulator, combining the ease-of-use of a high-level program-ming  language  with  the  performance  of  native  C.  Lyceum  is  up  to  10-20Xfaster  compared  to  other  popular  abstractions  like  OpenAI’sGymand  Deep-Mind’sdm-control.   This substantially reduces training time for various re-inforcement learning algorithms;  and is also fast enough to support real-timemodel  predictive  control  with  physics  simulators.    Lyceum  has  a  straightfor-ward API and supports parallel computation across multiple cores or machines. The code base,  tutorials,  and demonstration videos can be found at: https://sites.google.com/view/lyceum-anon. Progress in deep learning and artificial intelligence has exploded in recent years, due in large part to growing computational infrastructure. The advent of massively parallel GPU computing, combined with powerful automatic-differentiation tools like TensorFlow (Abadi et al., 2016) and PyTorch (Paszke et al., 2017b) , have lead to new classes of algorithms by enabling what was once computational intractable. These tools, alongside fast and accurate physics simulators like MuJoCo (Todorov et al., 2012) and associated frameworks like OpenAI's Gym (Brockman et al., 2016 ) and DeepMind's dm_control (Tassa et al., 2018) , have similarly transformed various aspects of robotic control like Reinforcement Learning (RL), Model-Predictive Control (MPC), and motion planning. These platforms enable researchers to give their ideas computational form, share results with collaborators, and deploy their successes on real systems. From these advances, simulation to real-world (sim2real) transfer has emerged as a promising paradigm for robotic control. A growing body of recent work suggests that robust control policies trained in simulation can successfully transfer to the real world (Lowrey et al., 2018a; OpenAI, 2018; Rajeswaran et al., 2016; Sadeghi & Levine, 2016; Tobin et al., 2017) . Despite these advances, many algorithms are computationally inefficient and have been unable to scale to complex problem domains. Training control policies with state-of-the-art RL algorithms often takes hours to days of compute time. For example, OpenAI's extremely impressive Dactyl work (OpenAI, 2018) required 50 hours of training time across 6144 CPU cores and 8 powerful NVIDIA V100 GPUs. Such computational budgets are available only to a select few labs. Furthermore, such experiments are seldom run only once in deep learning and especially in deep RL. Indeed, RL algorithms are notoriously sensitive to choices of hyper-parameters and require reward shaping (Henderson et al., 2017; Rajeswaran et al., 2017; 2018; Mania et al., 2018) . Thus, many iterations of the learning process may be required, with humans in the loop, to improve reward and hyperparameters, before deploying solutions in real world. This computational bottleneck often leads to a scarcity of hardware results, relative to the number of papers that propose new algorithms on highly simplified and well tuned benchmark tasks (Brockman et al., 2016; Tassa et al., 2018) . Exploring avenues to reduce experiment turn around time is thus crucial to scaling up to harder tasks and making resource-intensive algorithms and environments accessible to research labs without massive cloud computing budgets. In a similar vein, computational considerations have also limited progress in model-based control algorithms. For real-time model predictive control, the computational restrictions manifest as the requirement to compute controls in bounded time with limited local resources. As we will show, existing frameworks such as Gym and dm_control, while providing a convenient abstraction in Python, are too slow to meet this real-time computation requirement. As a result, most planning algorithms are run offline and deployed in open-loop mode on hardware. This is unfortunate, since it does not take feedback into account which is well known to be critical for stochastic control. Our Contributions: Our goal in this work is to overcome the aforementioned computational restrictions to enable faster training of policies with RL algorithms, facilitate real-time MPC with a detailed physics simulator, and ultimately enable researchers to engage more complex robotic tasks. To this end, we develop Lyceum, a computational ecosystem that uses the Julia programming language and the MuJoCo physics engine. Lyceum ships with the main OpenAI gym continuous control tasks, along with other environments representative of challenges in robotics. Julia's unique features allow us to wrap MuJoCo with zero-cost abstractions, providing the flexibility of a high-level programming language to enable easy creation of environments, tasks, and algorithms while retaining the performance of native C/C++. This allows RL and MPC algorithms implemented in Lyceum to be 10-100X faster compared to Gym and dm_control. We hope that this speedup will enable RL researchers to scale up to harder problems without increased computational costs, as well as enable real-time MPC that looks ahead through a simulator. We intruced Lyceum, a new computational ecosystem for robot learning in Julia that provides the rapid prototyping and ease-of-use benefits of a high-level programming language, yet retaining the performance of a low-level language like C. We demonstrated that this ecosystem can obtain 10-20X speedups compared to existing ecosystems like OpenAI gym and dm_control. We also demonstrated that this speed up enables faster experimental times for RL algorithms, as well as real-time model predictive control. In the future, we hope to port over algorithmic infrastructures like OpenAI's baselines (Dhariwal et al., 2017) as well as environments like hand manipulation suite (Rajeswaran et al., 2018) and DoorGym (Urakami et al., 2019) .
A high performance robotics simulation and algorithm development framework.
{"src": "scitldr-AIC_train", "paper_id": "SyxytxBFDr"}