The use of robotics in controlled environments has flourished over the last several decades and training robots to perform tasks using control strategies developed from dynamical models of their hardware have proven very effective. However, in many real-world settings, the uncertainties of the environment, the safety requirements and generalized capabilities that are expected of robots make rigid industrial robots unsuitable. This created great research interest into developing control strategies for flexible robot hardware for which building dynamical models are challenging. In this paper, inspired by the success of deep reinforcement learning (DRL) in other areas, we systematically study the efficacy of policy search methods using DRL in training flexible robots. Our results indicate that DRL is successfully able to learn efficient and robust policies for complex tasks at various degrees of flexibility. We also note that DRL using Deep Deterministic Policy Gradients can be sensitive to the choice of sensors and adding more informative sensors does not necessarily make the task easier to learn.
Robot controller design is dominated by building precise mathematical models of its dynamics. It is not always practical to build a general model of a robot's dynamics that is invariant to the various real-world factors ranging from noise to changes in the environment, motor backlash, motor torque output, or the focus of this paper, link flexibility. In such cases, reinforcement learning and policy search algorithms that can learn from a robot's experience have been shown to be successful @cite_20 @cite_0 for tasks such as object manipulation @cite_23 @cite_15 @cite_2 , locomotion @cite_3 @cite_30 @cite_28 @cite_1 and flight @cite_16 . However, most of this work involves using a model-free component to approximate features of the robot or the world that cannot be modeled while still using model-based controllers for other parts of the system @cite_11 @cite_9
@cite_30 @cite_28 @cite_9 @cite_1 @cite_3 @cite_0 @cite_23 @cite_2 @cite_15 @cite_16 @cite_20 @cite_11
2130105540 2964201544 2124980883 2132586633
Many control problems in the robotics field can be cast as partially observed Markovian decision problems (POMDPs), an optimal control formalism. Finding optimal solutions to such problems in general, however is known to be intractable. It has often been observed that in practice, simple structured controllers suffice for good sub-optimal control, and recent research in the artificial intelligence community has focused on policy search methods as techniques for finding sub-optimal controllers when such structured controllers do exist. Traditional model-based reinforcement learning algorithms make a certainty equivalence assumption on their learned models and calculate optimal policies for a maximum-likelihood Markovian model. We consider algorithms that evaluate and synthesize controllers under distributions of Markovian models. Previous work has demonstrated that algorithms that maximize mean reward with respect to model uncertainty leads to safer and more robust controllers. We consider briefly other performance criterion that emphasize robustness and exploration in the search for controllers, and note the relation with experiment design and active learning. To validate the power of the approach on a robotic application we demonstrate the presented learning control algorithm by flying an autonomous helicopter. We show that the controller learned is robust and delivers good performance in this real-world domain. Recent successes in reinforcement learning have lead to the development of complex controllers for realworld robots. As these robots are deployed in safety-critical applications and interact with humans, it becomes critical to ensure safety in order to avoid causing harm. A first step in this direction is to test the controllers in simulation. To be able to do this, we need to capture what we mean by safety and then efficiently search the space of all behaviors to see if they are safe. In this paper, we present an active-testing framework based on Bayesian Optimization. We specify safety constraints using logic and exploit structure in the problem in order to test the system for adversarial counter examples that violate the safety specifications. These specifications are defined as complex boolean combinations of smooth functions on the trajectories and, unlike reward functions in reinforcement learning, are expressive and impose hard constraints on the system. In our framework, we exploit regularity assumptions on individual functions in form of a Gaussian Process (GP) prior. We combine these into a coherent optimization framework using problem structure. The resulting algorithm is able to provably verify complex safety specifications or alternatively find counter examples. Experimental results show that the proposed method is able to find adversarial examples quickly. This paper presents the design of a variable structure-like tracking controller for a mobile robot system. The controller provides robustness with regard to bounded disturbances in the kinematic model. Through the use of a dynamic oscillator and a Lyapunov-based stability analysis, we demonstrate that the position and orientation tracking errors exponentially converge to a neighborhood about zero that can be made arbitrarily small (i.e., the controller ensures that the tracking error is globally uniformly ultimately bounded (GUUB)). In addition, we illustrate how the proposed tracking controller can also be utilized to achieve GUUB regulation to an arbitrary desired setpoint. An extension is also provided that illustrates how a smooth, time-varying control law can be utilized to achieve setpoint regulation despite parametric uncertainty in the kinematic model. Simulation results are presented to demonstrate the performance of the proposed controllers. @S0022-0434 00!00504-9# Service robots will play an increasing and more important role in the society in the next years. One of the main challenges is to endow robots with enough autonomy to operate on real environments. To reach that goal, the design of controllers to solve simple tasks must be automatized. Engineers look for learning algorithms that are general, robust, require low expertise knowledge, and generate controllers that can run on the real robot without any tuning stage. In this paper, a framework to learn behaviors (controllers) in mobile robotics, fulfilling the previous requirements, has been used. The framework is based on two modules: dataset generation and a data-driven evolutionary-based learning algorithm to obtain fuzzy controllers. Nevertheless, the design of a fuzzy controller still requires the selection of the type of learning algorithm, and also to choose the value of some design parameters. In this paper we present an exhaustive study on a set of evolutionary-based data-driven learning algorithms, for learning fuzzy controllers in mobile robotics, that cover a wide range of the accuracy interpretability trade-off. The study has also evaluated the influence of the values of all the design parameters over accuracy and interpretability. The objective is to analyze the performance of the different algorithms for the design of behaviors in mobile robotics, and to extract some general rules that can help in the process to design new behaviors. The analysis comprises two different behaviors (wall-following and moving object following) and more than 450 tests, both in simulation and on a Pioneer II AT robot. Results have shown very good performances in complex and realistic conditions for the different combinations of algorithms and parameters.