Reinforcement learning in robotics: A survey <s> Physics-inspired features <s> Model-Based Control of a Robot Manipulator presents the first integrated treatment of many of the most important recent developments in using detailed dynamic models of robots to improve their control. The authors' work on automatic identification of kinematic and dynamic parameters, feedforward position control, stability in force control, and trajectory learning has significant implications for improving performance in future robot systems. All of the main ideas discussed in this book have been validated by experiments on a direct-drive robot arm.The book addresses the issues of building accurate robot models and of applying them for high performance control. It first describes how three sets of models - the kinematic model of the links and the inertial models of the links and of rigid-body loads - can be obtained automatically using experimental data. These models are then incorporated into position control, single trajectory learning, and force control. The MIT Serial Link Direct Drive Arm, on which these models were developed and applied to control, is one of the few manipulators currently suitable for testing such concepts.Contents: Introduction. Direct Drive Arms. Kinematic Calibration. Estimation of Load Inertial Parameters. Estimation of Link Inertial Parameters. Feedforward and Computed Torque Control. Model-Based Robot Learning. Dynamic Stability Issues in Force Control. Kinematic Stability Issues in Force Control. Conclusion.Chae An is Research Staff Member, IBM T.J. Watson Research Center, Christopher Atkeson is an Assistant Professor and John Hollerbach is an Associate Professor in the MIT Department of Brain and Cognitive Sciences and the MIT Artificial Intelligence Laboratory. Model-Based Control of a Robot Manipulator is included in the Artificial Intelligence Series edited by Patrick Winston and Michael Brady. <s> BIB001 </s> Reinforcement learning in robotics: A survey <s> Physics-inspired features <s> By now it is widely accepted that learning a task from scratch, i.e., without any prior knowledge, is a daunting undertaking. Humans, however, rarely attempt to learn from scratch. They extract initial biases as well as strategies how to approach a learning problem from instructions and/or demonstrations of other humans. For teaming control, this paper investigates how learning from demonstration can be applied in the context of reinforcement learning. We consider priming the Q-function, the value function, the policy, and the model of the task dynamics as possible areas where demonstrations can speed up learning. In general nonlinear learning problems, only model-based reinforcement learning shows significant speed-up after a demonstration, while in the special case of linear quadratic regulator (LQR) problems, all methods profit from the demonstration. In an implementation of pole balancing on a complex anthropomorphic robot arm, we demonstrate that, when facing the complexities of real signal processing, model-based reinforcement learning offers the most robustness for LQR problems. Using the suggested methods, the robot learns pole balancing in just a single trial after a 30 second long demonstration of the human instructor. <s> BIB002 </s> Reinforcement learning in robotics: A survey <s> Physics-inspired features <s> One of the basic problems of machine learning is deciding how to act in an uncertain world. For example, if I want my robot to bring me a cup of coffee, it must be able to compute the correct sequence of electrical impulses to send to its motors to navigate from the coffee pot to my office. In fact, since the results of its actions are not completely predictable, it is not enough just to compute the correct sequence; instead the robot must sense and correct for deviations from its intended path. ::: In order for any machine learner to act reasonably in an uncertain environment, it must solve problems like the above one quickly and reliably. Unfortunately, the world is often so complicated that it is difficult or impossible to find the optimal sequence of actions to achieve a given goal. So, in order to scale our learners up to real-world problems, we usually must settle for approximate solutions. ::: One representation for a learner's environment and goals is a Markov decision process or MDP. MDPs allow us to represent actions that have probabilistic outcomes, and to plan for complicated, temporally-extended goals. An MDP consists of a set of states that the environment can be in, together with rules for how the environment can change state and for what the learner is supposed to do. ::: One way to approach a large MDP is to try to compute an approximation to its optimal state evaluation function, the function which tells us how much reward the learner can be expected to achieve if the world is in a particular state. If the approximation is good enough, we can use a shallow search to find a good action from most states. Researchers have tried many different ways to approximate evaluation functions. This thesis aims for a middle ground, between algorithms that don't scale well because they use an impoverished representation for the evaluation function and algorithms that we can't analyze because they use too complicated a representation. <s> BIB003
If good hand-crafted features are known, value-function approximation can be accomplished using a linear combination of features. However, good features are well known in robotics only for a few problems, such as features for local stabilization BIB002 and features describing rigid-body dynamics BIB001 ). Stabilizing a system at an unstable equilibrium point is the most well-known example, where a second-order Taylor expansion of the state together with a linear value-function approximator often suffice as features in the proximity of the equilibrium point. For example, BIB002 showed that such features suffice for learning how to stabilize a pole on the end-effector of a robot when within ±15-30 • of the equilibrium angle. For sufficient features, linear function approximation is likely to yield good results in an on-policy setting. Nevertheless, it is straightforward to show that impoverished value-function representations (e.g. omitting the cross-terms in quadratic expansion in Schaal's setup) will make it impossible for the robot to learn this behavior. Similarly, it is well known that linear value-function approximation is unstable in the offpolicy case   BIB003 .