Reinforcement learning in robotics: A survey <s> Appropriate policy representation <s> Many control problems take place in continuous state-action spaces, e.g., as in manipulator robotics, where the control objective is often defined as finding a desired trajectory that reaches a particular goal state. While reinforcement learning offers a theoretical framework to learn such control policies from scratch, its applicability to higher dimensional continuous state-action spaces remains rather limited to date. Instead of learning from scratch, in this paper we suggest to learn a desired complex control policy by transforming an existing simple canonical control policy. For this purpose, we represent canonical policies in terms of differential equations with well-defined attractor properties. By nonlinearly transforming the canonical attractor dynamics using techniques from nonparametric regression, almost arbitrary new nonlinear policies can be generated without losing the stability properties of the canonical system. We demonstrate our techniques in the context of learning a set of movement skills for a humanoid robot from demonstrations of a human teacher. Policies are acquired rapidly, and, due to the properties of well formulated differential equations, can be re-used and modified on-line under dynamic changes of the environment. The linear parameterization of nonparametric regression moreover lends itself to recognize and classify previously learned movement skills. Evaluations in simulations and on an actual 30 degree-of-freedom humanoid robot exemplify the feasibility and robustness of our approach. <s> BIB001 </s> Reinforcement learning in robotics: A survey <s> Appropriate policy representation <s> In the past, computational motor control has been approached from at least two major frameworks: the dynamic systems approach and the viewpoint of optimal control. The dynamic system approach emphasizes motor control as a process of self-organization between an animal and its environment. Nonlinear differential equations that can model entrainment and synchronization behavior are among the most favorable tools of dynamic systems modelers. In contrast, optimal control approaches view motor control as the evolutionary or development result of a nervous system that tries to optimize rather general organizational principles, e.g., energy consumption or accurate task achievement. Optimal control theory is usually employed to develop appropriate theories. Interestingly, there is rather little interaction between dynamic systems and optimal control modelers as the two approaches follow rather different philosophies and are often viewed as diametrically opposing. In this paper, we develop a computational approach to motor control that offers a unifying modeling framework for both dynamic systems and optimal control approaches. In discussions of several behavioral experiments and some theoretical and robotics studies, we demonstrate how our computational ideas allow both the representation of self-organizing processes and the optimization of movement based on reward criteria. Our modeling framework is rather simple and general, and opens opportunities to revisit many previous modeling results from this novel unifying view. <s> BIB002 </s> Reinforcement learning in robotics: A survey <s> Appropriate policy representation <s> Robots in a human environment need to be compliant. This compliance requires that a preplanned movement can be adapted to an obstacle that may be moving or appearing unexpectedly. Here, we present a general framework for movement generation and mid-flight adaptation to obstacles. For robust motion generation, Ijspeert et al developed the framework of dynamic movement primitives which represent a demonstrated movement with a set of differential equations. These equations allow adding a perturbing force without sacrificing stability of the desired movement. We extend this framework such that arbitrary movements in end-effector space can be represented - which was not possible before. Furthermore, we include obstacle avoidance by adding to the equations of motion a repellent force - a gradient of a potential field centered around the obstacle. In addition, this article compares different potential fields and shows how to avoid obstacle-link collisions within this framework. We demonstrate the abilities of our approach in simulations and with an anthropomorphic robot arm. <s> BIB003 </s> Reinforcement learning in robotics: A survey <s> Appropriate policy representation <s> Policy Gradient methods are model-free reinforcement learning algorithms which in recent years have been successfully applied to many real-world problems. Typically, Likelihood Ratio (LR) methods are used to estimate the gradient, but they suffer from high variance due to random exploration at every time step of each training episode. Our solution to this problem is to introduce a state-dependent exploration function (SDE) which during an episode returns the same action for any given state. This results in less variance per episode and faster convergence. SDE also finds solutions overlooked by other methods, and even improves upon state-of-the-art gradient estimators such as Natural Actor-Critic. We systematically derive SDE and apply it to several illustrative toy problems and a challenging robotics simulation task, where SDE greatly outperforms random exploration. <s> BIB004
The policy is represented by dynamical system motor primitives BIB001 BIB002 . The global movement is encoded as a point attractor linear dynamical system with an additional local transformation that allows a very parsimonious representation of the policy. This framework ensures the stability of the movement and allows the representation of arbitrarily shaped movements through the primitive's policy parameters. These parameters can be estimated straightforwardly by locally weighted regression. Note that the dynamical systems motor primitives ensure the stability of the movement generation but cannot guarantee the stability of the movement execution. These primitives can be modified through their meta-parameters in order to adapt to the final goal position, the movement amplitude, or the duration of the movement. The resulting movement can start from arbitrary positions and velocities and go to arbitrary final positions while maintaining the overall shape of the trajectory. While the original formulation of BIB001 for discrete dynamical systems motor primitives used a secondorder system to represent the phase z of the movement, this formulation has proven to be unnecessarily complicated in practice. Since then, it has been simplified and, in BIB002 , it was shown that a single first-order system sufficesż = −τ α z z. ( 7 ) This canonical system has the time constant τ = 1/T where T is the duration of the motor primitive, a parameter α z which is chosen such that z ≈ 0 at T to ensure that the influence of the transformation function, shown in Equation (9), vanishes. Subsequently, the internal state x of a second system is chosen such that positions q of all DoFs are given by q = x 1 , the velocitiesq byq = τ x 2 =ẋ 1 and the accelerationsq byq = τẋ 2 . Under these assumptions, the learned dynamics of Ijspeert motor primitives can be expressed in the following forṁ This set of differential equations has the same time constant τ as the canonical system, parameters α x , β x set such that the system is critically damped, a goal parameter g, a transformation function f and an amplitude matrix A = diag( a 1 , a 2 , . . . , a n ), with the amplitude modifier a = [a 1 , a 2 , . . . , a n ]. BIB002 used a = g − x 0 1 with the initial position x 0 1 , which ensures linear scaling. Alternative choices are possibly better suited for specific tasks, see, e.g., BIB003 . The transformation function f (z) alters the output of the first system, in Equation (7), so that the second system, in Equation (8), can represent complex nonlinear patterns and it is given by Here, w i contains the ith adjustable parameter of all DoFs, N is the number of parameters per DoF, and ψ i ( z) are the corresponding weighting functions BIB002 . Normalized Gaussian kernels are used as weighting functions given by These weighting functions localize the interaction in phase space using the centers c i and widths h i . Note that the DoFs are usually all modeled as independent in Equation (8). All DoFs are synchronous as the dynamical systems for all DoFs start at the same time, have the same duration, and the shape of the movement is generated using the transformation f (z) in Equation (9). This transformation function is learned as a function of the shared canonical system in Equation (7). This policy can be seen as a parameterization of a mean policy in the formā = θ T μ( s, t), which is linear in parameters. Thus, it is straightforward to include prior knowledge from a demonstration using supervised learning by locally weighted regression. This policy is augmented by an additive exploration ( s, t) noise term to make policy-search methods possible. As a result, the explorative policy can be given in the form a = θ T μ( s, t) + ( μ( s, t) ). Some policy-search approaches have previously used state-independent, white Gaussian exploration, i.e . ( μ( s, t) ) ∼ N ( 0, ). However, such unstructured exploration at every step has several disadvantages, notably: (i) it causes a large variance which grows with the number of time-steps; (ii) it perturbs actions too frequently, thus, "washing out" their effects; and (iii) it can damage the system that is executing the trajectory. Alternatively, one could generate a form of structured, state-dependent exploration BIB004 ( μ( s, t)  ij are meta-parameters of the exploration that can also be optimized. This argument results in the policy a = (θ + ε t ) T μ( s, t) corresponding to the distribution a ∼ π ( a t |s t , t) = N ( a|θ T μ( s, t) , μ( s, t) Tˆ ( s, t) ). Instead of directly exploring in action space, this type of policy explores in parameter space.