Reinforcement learning in robotics: A survey <s> Local models <s> By now it is widely accepted that learning a task from scratch, i.e., without any prior knowledge, is a daunting undertaking. Humans, however, rarely attempt to learn from scratch. They extract initial biases as well as strategies how to approach a learning problem from instructions and/or demonstrations of other humans. For teaming control, this paper investigates how learning from demonstration can be applied in the context of reinforcement learning. We consider priming the Q-function, the value function, the policy, and the model of the task dynamics as possible areas where demonstrations can speed up learning. In general nonlinear learning problems, only model-based reinforcement learning shows significant speed-up after a demonstration, while in the special case of linear quadratic regulator (LQR) problems, all methods profit from the demonstration. In an implementation of pole balancing on a complex anthropomorphic robot arm, we demonstrate that, when facing the complexities of real signal processing, model-based reinforcement learning offers the most robustness for LQR problems. Using the suggested methods, the robot learns pole balancing in just a single trial after a 30 second long demonstration of the human instructor. <s> BIB001 </s> Reinforcement learning in robotics: A survey <s> Local models <s> Robots are now being used in complex, unstructured environments, performing ever more sophisticated tasks. As the task and environmental complexity increases, the need for effective learning on such systems is becoming more and more apparent. Robot programmers often find it difficult to articulate their knowledge of how to perform a given task in a form suitable for robots to use. Even when they can, the limitations of robot sensors and actuators might render their intuitions less effective. Also, it is often not possible to anticipate (and code for) all environments in which the robot might find itself having to perform a certain task. Therefore, it seems useful to have the robot be able to learn to act, in the hope of overcoming these two difficulties. One promising approach to learning on real robots that is attracting considerable interest at the moment is reinforcement learning. The programmer must supply a reward function which maps states of the world onto a scalar reward, essentially saying how good or bad it is to be in a given state. Once given this function, the robot can, in theory, learn an appropriate action policy to maximize some measure of reward over time. Designing such a function seems more intuitive than writing code to perform the task, since we can do so in more general terms. However, there are some significant problems when we attempt to use this strategy on a real, physical robot, including (but not limited to) the following. <s> BIB002 </s> Reinforcement learning in robotics: A survey <s> Local models <s> This paper describes the rise of task primitives in robot learning from observation. A framework is developed that uses observed data to initially learn a task and the agent then goes on to increase its performance through repeated task performance (learning from practice). Data that is collected while the human performs a task is parsed into small parts of the task called primitives. Modules are created for each primitive that encode the movements required during the performance of the primitive, and when and where the primitives are performed. The feasibility of this method is currently being tested with agents that learn to play a virtual and an actual air hockey game. <s> BIB003
Local models can be seen as an extension of generalization among neighboring cells to generalizing among neighboring data points. Locally weighted regression creates particularly efficient function approximation in the context of robotics both in supervised and reinforcement learning. Here, regression errors are weighted down by proximity to query point to train local models. The predictions of these local models are combined using the same weighting functions. Using local models for valuefunction approximation has allowed learning a navigation task with obstacle avoidance BIB002 , a pole swing-up task BIB001 , and an air hockey task BIB003 .