Ahstract- We present an algorithm for rapidly learning neural network policies for robotics systems. The algorithm follows the model-based reinforcement learning paradigm and improves upon existing algorithms: PILeO and a sample-based version of PILeo with neural network dynamics (Deep-PILeO). To improve convergence, we propose a model-based algorithm that uses fixed random numbers and clips gradients during optimization. We propose training a neural network dynamics model using variational dropout with truncated Log-Normal noise. These improvements enable data-efficient synthesis of complex neural network policies. We test our approach on a variety of benchmark tasks, demonstrating data-efficiency that is competitive with that of PILeO, while being able to optimize complex neural network controllers. Finally, we assess the performance of the algorithm for learning motor controllers for a six legged autonomous underwater vehicle. This demonstrates the potential of the algorithm for scaling up the dimensionality and dataset sizes, in more complex tasks.
Dynamics models have long been a core element in the modeling and control of robotic systems. Trajectory optimization approaches @cite_24 @cite_22 @cite_20 can produce highly effective controllers for complex robotic systems when precise analytical models are available. For complex and stochastic systems such as swimming robots, classical models are less reliable. In these cases, either performing online system identification @cite_7 or learning complete dynamics models from data has proven to be effective, and can be integrated tightly with model-based control schemes @cite_28 @cite_13 @cite_1 @cite_18 . Multiple works have applied Deep RL methods to learn various continuous control tasks @cite_29 @cite_6 , including full-body control of humanoid characters @cite_12 . These methods do not assume a known reward function, estimating the value of each action from experience. Along with their model-free nature, this results in lower data efficiency compared with the methods we consider here, but there are ongoing efforts to connect model-based and model-free approaches @cite_5 .
@cite_18 @cite_22 @cite_7 @cite_28 @cite_29 @cite_1 @cite_6 @cite_24 @cite_5 @cite_13 @cite_12 @cite_20
2763676071 2890276699 2079868754 2001095967
In this work, we present an approach to deep visuomotor control using structured deep dynamics models. Our deep dynamics model, a variant of SE3-Nets, learns a low-dimensional pose embedding for visuomotor control via an encoder-decoder structure. Unlike prior work, our dynamics model is structured: given an input scene, our network explicitly learns to segment salient parts and predict their pose-embedding along with their motion modeled as a change in the pose space due to the applied actions. We train our model using a pair of point clouds separated by an action and show that given supervision only in the form of point-wise data associations between the frames our network is able to learn a meaningful segmentation of the scene along with consistent poses. We further show that our model can be used for closed-loop control directly in the learned low-dimensional pose space, where the actions are computed by minimizing error in the pose space using gradient-based methods, similar to traditional model-based control. We present results on controlling a Baxter robot from raw depth data in simulation and in the real world and compare against two baseline deep networks. Our method runs in real-time, achieves good prediction of scene dynamics and outperforms the baseline methods on multiple control runs. Video results can be found at: this https URL Transferring solutions found by trajectory optimization to robotic hardware remains a challenging task. When the optimization fully exploits the provided model to perform dynamic tasks, the presence of unmodeled dynamics renders the motion infeasible on the real system. Model errors cannot be only a result of model simplifications, but also naturally arise when deploying the robot in unstructured and nondeterministic environments. Predominantly, compliant contacts and actuator dynamics lead to bandwidth limitations. While classical control methods provide tools to synthesize controllers that are robust to a class of model errors, such a notion is missing in modern trajectory optimization, which is solved in the time domain. We propose frequency-shaped cost functions to achieve robust solutions in the context of optimal control for legged robots. Through simulation and hardware experiments we show that motion plans can be made compatible with bandwidth limits set by actuators and contact dynamics. The smoothness of the model predictive solutions can be continuously tuned without compromising the feasibility of the problem. Experiments with the quadrupedal robot ANYmal, which is driven by highly compliant series elastic actuators, showed significantly improved tracking performance of the planned motion, torque, and force trajectories and enabled the machine to walk robustly on terrain with unmodeled compliance. Accurate dynamic models can be very difficult to compute analytically for complex robots; moreover, using a precomputed fixed model does not allow to cope with unexpected changes in the system. An interesting alternative solution is to learn such models from data, and keep them up-to-date through online adaptation. In this paper we consider the problem of learning the robot inverse dynamic model under dynamically varying contexts: the robot learns incrementally and autonomously the model under different conditions, represented by the manipulation of objects of different weights, that change the dynamics of the system. The inverse dynamic mapping is modeled as a multi-valued function, in which different outputs for the same input query are related to different dynamic contexts (i.e. different manipulated objects). The mapping is estimated using IMLE, a recent online learning algorithm for multi-valued regression, and used for Computed Torque control. No information is given about the context switch during either learning or control, nor any assumption is made about the kind of variation in the dynamics imposed by a new contexts. Experimental results with the iCub humanoid robot are provided. Many Stochastic Optimal Control (SOC) approaches rely on samples to either obtain an estimate of the value function or a linearisation of the underlying system model. However, these approaches typically neglect the fact that the accuracy of the policy update depends on the closeness of the resulting trajectory distribution to these samples. The greedy operator does not consider such closeness constraint to the samples. Hence, the greedy operator can lead to oscillations or even instabilities in the policy updates. Such undesired behaviour is likely to result in an inferior performance of the estimated policy. We reuse inspiration from the reinforcement learning community and relax the greedy operator used in SOC with an information theoretic bound that limits the ‘distance’ of two subsequent trajectory distributions in a policy update. The introduced bound ensures a smooth and stable policy update. Our method is also well suited for model-based reinforcement learning, where we estimate the system dynamics model from data. As this model is likely to be inaccurate, it might be dangerous to exploit the model greedily. Instead, our bound ensures that we generate new data in the vicinity of the current data, such that we can improve our estimate of the system dynamics model. We show that our approach outperforms several state of the art approaches on challenging simulated robot control tasks.