Reinforcement learning in robotics: A survey <s> Iterative learning control <s> This article proposes a betterment process for the operation of a mechanical robot in a sense that it betters the next operation of a robot by using the previous operation's data. The process has an iterative learning structure such that the (k + 1)th input to joint actuators consists of the kth input plus an error increment composed of the derivative difference between the kth motion trajectory and the given desired motion trajectory. The convergence of the process to the desired motion trajectory is assured under some reasonable conditions. Numerical results by computer simulation are presented to show the effectiveness of the proposed learning scheme. <s> BIB001 </s> Reinforcement learning in robotics: A survey <s> Iterative learning control <s> Model-Based Control of a Robot Manipulator presents the first integrated treatment of many of the most important recent developments in using detailed dynamic models of robots to improve their control. The authors' work on automatic identification of kinematic and dynamic parameters, feedforward position control, stability in force control, and trajectory learning has significant implications for improving performance in future robot systems. All of the main ideas discussed in this book have been validated by experiments on a direct-drive robot arm.The book addresses the issues of building accurate robot models and of applying them for high performance control. It first describes how three sets of models - the kinematic model of the links and the inertial models of the links and of rigid-body loads - can be obtained automatically using experimental data. These models are then incorporated into position control, single trajectory learning, and force control. The MIT Serial Link Direct Drive Arm, on which these models were developed and applied to control, is one of the few manipulators currently suitable for testing such concepts.Contents: Introduction. Direct Drive Arms. Kinematic Calibration. Estimation of Load Inertial Parameters. Estimation of Link Inertial Parameters. Feedforward and Computed Torque Control. Model-Based Robot Learning. Dynamic Stability Issues in Force Control. Kinematic Stability Issues in Force Control. Conclusion.Chae An is Research Staff Member, IBM T.J. Watson Research Center, Christopher Atkeson is an Assistant Professor and John Hollerbach is an Associate Professor in the MIT Department of Brain and Cognitive Sciences and the MIT Artificial Intelligence Laboratory. Model-Based Control of a Robot Manipulator is included in the Artificial Intelligence Series edited by Patrick Winston and Michael Brady. <s> BIB002 </s> Reinforcement learning in robotics: A survey <s> Iterative learning control <s> In supervised motor learning, where the desired movement pattern is given in task-oriented coordinates, one of the most essential and difficult problems is how to convert the error signal calculated in the task space into that of the motor command space. We proposed the feedback-error-learning approach where the feedback motor command is used as an error signal to train a neural network which then generates a feedforward motor command. We successfully applied this approach to learning control in joint space of PUMA, an industrial manipulator, trajectory control in visual coordinates even when PUMA was given an extra degree of freedom, and Softarm with muscle-like actuators (manufactured by Bridgestone Co.). I mathematically show that the feedback-error-learning approach can be considered a Newton-like method in functional space. Mathematical proof of the convergence of the feedback-error-learning control scheme is provided both for trajectory stability and learning stability. The feedback-error-learning neural network was originally proposed as a model for the lateral cerebellum and the parvocellular part of the red nucleus. It will be shown that functions of other parts of the cerebellum, such as the vermis or the flocculus, can also be understood from the feedback-error-learning scheme. <s> BIB003 </s> Reinforcement learning in robotics: A survey <s> Iterative learning control <s> An adaptive iterative learning control (ILC) algorithm based on an estimation procedure using a Kalman filter and an optimization of a quadratic criterion is presented. It is shown that by taking the measurement disturbance into consideration the resulting ILC filters become iteration-varying. Results from experiments on an industrial robot show that the algorithm is successful also in an application. <s> BIB004 </s> Reinforcement learning in robotics: A survey <s> Iterative learning control <s> A combination of model-based and iterative learning control (ILC) is proposed as a method to achieve high-quality motion control of direct-drive robots in repetitive motion tasks. We include both model-based and learning components in the total control law, as their individual properties influence the performance of motion control. The model-based part of the controller compensates much of the nonlinear and coupled robot dynamics. A new procedure for estimating the parameters of the rigid body model, implemented in this part of the controller, is used. This procedure is based on a batch-adaptive control algorithm that estimates the model parameters online. Information about the dynamics not covered by the rigid body model, due to flexibilities, is acquired experimentally, by identification. The models of the flexibilities are used in the design of the iterative learning controllers for the individual joints. Use of the models facilitates quantitative prediction of performance improvement via ILC. The effectiveness of the combination of the model-based and the iterative learning controllers is demonstrated in experiments on a spatial serial direct-drive robot with revolute joints. <s> BIB005 </s> Reinforcement learning in robotics: A survey <s> Iterative learning control <s> In the model-based policy search approach to reinforcement learning (RL), policies are found using a model (or "simulator") of the Markov decision process. However, for high-dimensional continuous-state tasks, it can be extremely difficult to build an accurate model, and thus often the algorithm returns a policy that works in simulation but not in real-life. The other extreme, model-free RL, tends to require infeasibly large numbers of real-life trials. In this paper, we present a hybrid algorithm that requires only an approximate model, and only a small number of real-life trials. The key idea is to successively "ground" the policy evaluations using real-life trials, but to rely on the approximate model to suggest local changes. Our theoretical results show that this algorithm achieves near-optimal performance in the real system, even when the model is only approximate. Empirical results also demonstrate that---when given only a crude model and a small number of real-life trials---our algorithm can obtain near-optimal performance in the real system. <s> BIB006 </s> Reinforcement learning in robotics: A survey <s> Iterative learning control <s> Synchronization is routinely required to coordinate the actions of the various sub-systems involved in process applications. This is commonly achieved through direct mechanical coupling, involving gears, drive belts and cams. Apart from the additional cost incurred, these components are subject to wear, constrain the layout of the plant and may have limited accuracy. It is shown in this paper that a mechanical linkage between two sub-systems may be replaced by instead implementing a control scheme comprising an iterative learning controller together with a supervisory control loop. To illustrate the approach, two types of iterative learning controller are first implemented on a gantry robot test facility to confirm the high levels of tracking accuracy that may be achieved. The supervisory control loop is then added to synchronize the ‘pick and place’ action of the robot with a conveyor system moving at constant velocity. Experimental results are provided to confirm both the accurate tracking performance produced by the iterative learning controller, and the high level of synchronization achieved by the overall scheme. <s> BIB007
A powerful idea that has been developed in multiple forms in both the reinforcement learning and control communities is the use of crude, approximate models to determine gradients, e.g. for an update step. The resulting new policy is then evaluated in the real world and the model is updated. This approach is known as iterative learning control BIB001 . A similar preceding idea was employed to minimize trajectory tracking errors BIB002 and is loosely related to feedback error learning BIB003 . More recently, variations on the iterative learning control has been employed to learn robot control BIB004 BIB005 , steering a RC car with a general analysis of approximate models for policy search in BIB006 , a pick and place task BIB007 , and an impressive application of tying knots with a surgical robot at superhuman speeds (van den Berg et al., 2010).