Reinforcement learning in robotics: A survey <s> GPR <s> In this paper, we present an approach that applies the reinforcement learning principle to the problem of learning height control policies for aerial blimps. In contrast to previous approaches, our method does not require sophisticated hand- tuned models, but rather learns the policy online, which makes the system easily adaptable to changing conditions. The blimp we apply our approach to is a small-scale vehicle equipped with an ultrasound sensor that measures its elevation relative to the ground. The major problem in the context of learning control policies lies in the high-dimensional state-action space that needs to be explored in order to identify the values of all state-action pairs. In this paper, we propose a solution to learning continuous control policies based on the Gaussian process model. In practical experiments carried out on a real robot we demonstrate that the system is able to learn a policy online within a few minutes only. <s> BIB001 </s> Reinforcement learning in robotics: A survey <s> GPR <s> When children learn to grasp a new object, they often know several possible grasping points from observing a parent's demonstration and subsequently learn better grasps by trial and error. From a machine learning point of view, this process is an active learning approach. In this paper, we present a new robot learning framework for reproducing this ability in robot grasping. For doing so, we chose a straightforward approach: first, the robot observes a few good grasps by demonstration and learns a value function for these grasps using Gaussian process regression. Subsequently, it chooses grasps which are optimal with respect to this value function using a mean-shift optimization approach, and tries them out on the real system. Upon every completed trial, the value function is updated, and in the following trials it is more likely to choose even better grasping points. This method exhibits fast learning due to the data-efficiency of the Gaussian process regression framework and the fact that the mean-shift method provides maxima of this cost function. Experiments were repeatedly carried out successfully on a real robot system. After less than sixty trials, our system has adapted its grasping policy to consistently exhibit successful grasps. <s> BIB002 </s> Reinforcement learning in robotics: A survey <s> GPR <s> While today's industrial robots achieve high performance in a wide range of tasks, the programming and adaptation of such robots to new tasks is still time-consuming and far from convenient. This shortcoming is a major obstacle for the deployment of robots in small to medium-size companies and also for service applications. Robot learning by imitation has been identified as an important paradigm to tackle this practicability issue. However, such approaches are only applicable in controlled static scenarios. To perform less constrained tasks, the robot needs the ability to generalize its skills and to adapt to changing conditions. It is thus a natural idea to formulate the problem as learning from experience and to incorporate demonstrations by an expert in the learning process. We present a new framework for learning of motion skills that seamlessly integrates demonstration learning from an expert teacher and further optimization of the demonstrated skill by own experience. Our method employs Gaussian Process regression to generalize a measure of performance to new skills that are close to already learned skills in the state-action space. We evaluate our approach for the task of grasping an object. <s> BIB003 </s> Reinforcement learning in robotics: A survey <s> GPR <s> Grasping an object is a task that inherently needs to be treated in a hybrid fashion. The system must decide both where and how to grasp the object. While selecting where to grasp requires learning about the object as a whole, the execution only needs to reactively adapt to the context close to the grasp's location. We propose a hierarchical controller that reflects the structure of these two sub-problems, and attempts to learn solutions that work for both. A hybrid architecture is employed by the controller to make use of various machine learning methods that can cope with the large amount of uncertainty inherent to the task. The controller's upper level selects where to grasp the object using a reinforcement learner, while the lower level comprises an imitation learner and a vision-based reactive controller to determine appropriate grasping motions. The resulting system is able to quickly learn good grasps of a novel object in an unstructured environment, by executing smooth reaching motions and preshaping the hand depending on the object's geometry. The system was evaluated both in simulation and on a real robot. <s> BIB004
Parametrized global or local models need to pre-specify, which requires a trade-off between representational accuracy and the number of parameters. A nonparametric function approximator such as GPR could be employed instead, but potentially at the cost of a higher computational complexity. GPR has the added advantage of providing a notion of uncertainty about the approximation quality for a query point. Hovering with an autonomous blimp BIB001 has been achieved by approximation the state-action value function with a GPR. Similarly, another paper shows that grasping can be learned using GPR BIB003 by additionally taking into account the uncertainty to guide the exploration. Grasping locations can be learned by approximating the rewards with a GPR, and trying candidates with predicted high rewards BIB002 , resulting in an active learning approach. High reward uncertainty allows intelligent exploration in reward-based grasping BIB004 in a bandit setting.