Reinforcement learning in robotics: A survey <s> Gaussian mixture models and radial basis function models <s> Skill acquisition is a difficult , yet important problem in robot performance. The authors focus on two skills, namely robotic assembly and balancing and on two classic tasks to develop these skills via learning: the peg-in hole insertion task, and the ball balancing task. A stochastic real-valued (SRV) reinforcement learning algorithm is described and used for learning control and the authors show how it can be used with nonlinear multilayer ANNs. In the peg-in-hole insertion task the SRV network successfully learns to insert to insert a peg into a hole with extremely low clearance, in spite of high sensor noise. In the ball balancing task the SRV network successfully learns to balance the ball with minimal feedback. > <s> BIB001 </s> Reinforcement learning in robotics: A survey <s> Gaussian mixture models and radial basis function models <s> A tabula rasa learning technique has worked well in well defined grid like problems (Barto et al., 1993). Nevertheless, it has severe limitations when applied in complex domains. In order to build a true learning system in complex domains, we have to begin to integrate a considerable amount of bias with the learner that has the ability to adapt a priori knowledge. This bias can assume a variety of forms. In this paper, in addition to reflex rules (Millan, 1996), symbolic knowledge about the environment is embedded into the learning system (Kaelbling, 1993). The incorporation of such knowledge aids the learner to identify and split key states rapidly. The learner is tested on a B21 robot for a goal reaching task. Experimental results show that after few trials the robot has indeed learned to unfold its path and to consistently follow the shortest path to the goal. <s> BIB002 </s> Reinforcement learning in robotics: A survey <s> Gaussian mixture models and radial basis function models <s> In this paper, we present our design and experiments of a planar biped robot ("RunBot") under pure reflexive neuronal control. The goal of this study is to combine neuronal mechanisms with biomechanics to obtain very fast speed and the on-line learning of circuit parameters. Our controller is built with biologically inspired sensor- and motor-neuron models, including local reflexes and not employing any kind of position or trajectory-tracking control algorithm. Instead, this reflexive controller allows RunBot to exploit its own natural dynamics during critical stages of its walking gait cycle. To our knowledge, this is the first time that dynamic biped walking is achieved using only a pure reflexive controller. In addition, this structure allows using a policy gradient reinforcement learning algorithm to tune the parameters of the reflexive controller in real-time during walking. This way RunBot can reach a relative speed of 3.5 leg-lengths per second after a few minutes of online learning, which is faster than that of any other biped robot, and is also comparable to the fastest relative speed of human walking. In addition, the stability domain of stable walking is quite large supporting this design strategy. <s> BIB003 </s> Reinforcement learning in robotics: A survey <s> Gaussian mixture models and radial basis function models <s> The goal of developing algorithms for programming robots by demonstration is to create an easy way of programming robots that can be accomplished by everyone. When a demonstrator teaches a task to a robot, he/she shows some ways of fulfilling the task, but not all the possibilities. The robot must then be able to reproduce the task even when unexpected perturbations occur. In this case, it has to learn a new solution. In this paper, we describe a system to teach to the robot constrained reaching tasks. Our system is based on a dynamical system generator modulated with a learned speed trajectory and combined with a reinforcement learning module to allow the robot to adapt the trajectory when facing a new situation, such as avoiding obstacles. <s> BIB004 </s> Reinforcement learning in robotics: A survey <s> Gaussian mixture models and radial basis function models <s> In this paper we describe a learning framework for a central pattern generator (CPG)-based biped locomotion controller using a policy gradient method. Our goals in this study are to achieve CPG-based biped walking with a 3D hardware humanoid and to develop an efficient learning algorithm with CPG by reducing the dimensionality of the state space used for learning. We demonstrate that an appropriate feedback controller can be acquired within a few thousand trials by numerical simulations and the controller obtained in numerical simulation achieves stable walking with a physical robot in the real world. Numerical simulations and hardware experiments evaluate the walking velocity and stability. The results suggest that the learning algorithm is capable of adapting to environmental changes. Furthermore, we present an online learning scheme with an initial policy for a hardware robot to improve the controller within 200 iterations. <s> BIB005 </s> Reinforcement learning in robotics: A survey <s> Gaussian mixture models and radial basis function models <s> We consider policy search for reinforcement learning: learning policy parameters, for some fixed policy class, that optimize performance of a system. In this paper, we propose a novel policy gradient method based on an approximation we call the Signed Derivative; the approximation is based on the intuition that it is often very easy to guess the direction in which control inputs affect future state variables, even if we do not have an accurate model of the system. The resulting algorithm is very simple, requires no model of the environment, and we show that it can outperform standard stochastic estimators of the gradient; indeed we show that Signed Derivative algorithm can in fact perform as well as the true (model-based) policy gradient, but without knowledge of the model. We evaluate the algorithm’s performance on both a simulated task and two realworld tasks — driving an RC car along a specified trajectory, and jumping onto obstacles with an quadruped robot — and in all cases achieve good performance after very little training. <s> BIB006 </s> Reinforcement learning in robotics: A survey <s> Gaussian mixture models and radial basis function models <s> Reaching is a fundamental skill for a robot. The purpose of robot reaching is to bring a robot hand to an object without any obstacle collision. Conventional handcrafted methods were complicated to implement reaching skill. Thus, this paper proposes a method using primitives acquired from human demonstrations to learn collision-free reaching skill. End-effector and joint trajectories of primitives are encoded by Gaussian Mixture Model (GMM). The way to avoid an obstacle for a primitive uses reinforcement learning to adjust the part of its end-effector trajectory near the obstacle by adapting the associative Gaussian parameters. By doing this, the movement pattern of robot reaching is goal-directed, collision-free, and similar to human-like reaching. In this paper, we validated the proposed method on an Staibli TX-40 industrial robot. The results showed that the TX-40 robot was able to perform human-like skillful reaching for an object placed in an untrained location of a table. <s> BIB007
When more general policies with a strong state dependence are needed, general function approximators based on radial basis functions, also called Gaussian kernels, become reasonable choices. While learning with fixed basis function centers and widths often works well in practice, estimating them is challenging. These centers and widths can also be estimated from data prior to the reinforcement learning process. This approach has been used to generalize a open-loop reaching movement BIB004 BIB007 and to learn the closed-loop cart-pole swing-up task . Globally linear models were employed in a closed-loop block stacking task . BIB006 . (Reprinted with permission from Zico Kolter.) 4.3.5 . Neural networks These are another general function approximation used to represent policies. Neural oscillators with sensor feedback have been used to learn rhythmic movements where open-and closed-loop information were combined, such as gaits for a two-legged robot BIB003 BIB005 . Similarly, a peg-in-hole (see Figure 1b ), a ball-balancing task BIB001 , and a navigation task BIB002 have been learned with closed-loop neural networks as policy function approximators.