Reinforcement learning in robotics: A survey <s> Value function methods with learned models <s> This paper proposes a method that acquires robots' behaviors based on the estimation of the state vectors. In order to acquire the cooperative behaviors in multi-robot environments, each learning robot estimates the local predictive model between the learner and the other objects separately. Based on the local predictive models, the robots learn the desired behaviors using reinforcement learning. The proposed method is applied to a soccer playing situation, where a rolling ball and other moving robots are well modeled and the learner's behaviors are successfully acquired by the method. Computer simulations and real experiments are shown and a discussion is given. <s> BIB001 </s> Reinforcement learning in robotics: A survey <s> Value function methods with learned models <s> This paper describes quasi-online reinforcement learning: while a robot is exploring its environment, in the background a probabilistic model of the environment is built on the fly as new experiences arrive; the policy is trained concurrently based on this model using an anytime algorithm. Prioritized sweeping, directed exploration, and transformed reward functions provide additional speed-ups. The robot quickly learns goal-directed policies from scratch, requiring few interactions with the environment and making efficient use of available computation time. From an outside perspective it learns the behavior online and in real time. We describe comparisons with standard methods and show the individual utility of each of the proposed techniques <s> BIB002 </s> Reinforcement learning in robotics: A survey <s> Value function methods with learned models <s> In the paper we evaluate two learning methods applied to the ball-in-a-cup game. The first approach is based on imitation learning. The captured trajectory was encoded with Dynamic motion primitives (DMP). The DMP approach allows simple adaptation of the demonstrated trajectory to the robot dynamics. In the second approach, we use reinforcement learning, which allows learning without any previous knowledge of the system or the environment. In contrast to the majority of the previous attempts, we used SASRA learning algorithm. Experimental results for both cases were performed on Mitsubishi PA10 robot arm. <s> BIB003
Obviously, mental rehearsal can be used straightforwardly with value-function methods by simply pretending that the simulated roll-outs were generated from the real system. Learning in simulation while the computer is idle and employing directed exploration allows Q-learning to learn a navigation task from scratch in 20 minutes BIB002 . Two robots taking turns in learning a simplified soccer task were also able to profit from mental rehearsal BIB001 . BIB003 used a value function learned in simulation to initialize the real robot learning. However, it is clear that model-based methods that use the model for creating more direct experience should potentially perform better.