Reinforcement learning in robotics: A survey <s> Reinforcement learning by policy search <s> We discuss Hinton's (1989) relative payoff procedure (RPP), a static reinforcement learning algorithm whose foundation is not stochastic gradient ascent. We show circumstances under which applying the RPP is guaranteed to increase the mean return, even though it can make large changes in the values of the parameters. The proof is based on a mapping between the RPP and a form of the expectation-maximization procedure of Dempster, Laird, and Rubin (1977). <s> BIB001 </s> Reinforcement learning in robotics: A survey <s> Reinforcement learning by policy search <s> Many motor skills in humanoid robotics can be learned using parametrized motor primitives as done in imitation learning. However, most interesting motor learning problems are high-dimensional reinforcement learning problems often beyond the reach of current methods. In this paper, we extend previous work on policy learning from the immediate reward case to episodic reinforcement learning. We show that this results in a general, common framework also connected to policy gradient methods and yielding a novel algorithm for policy learning that is particularly well-suited for dynamic motor primitives. The resulting algorithm is an EM-inspired algorithm applicable to complex motor learning tasks. We compare this algorithm to several well-known parametrized policy search methods and show that it outperforms them. We apply it in the context of motor learning and show that it can learn a complex Ball-in-a-Cup task using a real Barrett WAM™ robot arm. <s> BIB002
Policy-search methods are better suited for a scenario such as this, where the task is episodic, local optimization is sufficient (thanks to the initial demonstration), and highdimensional, continuous states and actions need to be taken into account. A single update step in a gradient-based method usually requires as many episodes as parameters to be optimized. Since the expected number of parameters was in the hundreds, a different approach had to be taken because gradient based methods are impractical in this scenario. Furthermore, the step-size parameter for gradientbased methods often is a crucial parameter that needs to be tuned to achieve good performance. Instead, an EMinspired algorithm was employed that requires significantly less samples and has no learning rate. BIB002 have derived a framework of reward weighted imitation. Based on BIB001 they consider the return of an episode as an improper probability distribution. A lower bound of the logarithm of the expected return is maximized. Depending on the strategy of optimizing this lower bound and the exploration strategy, the framework yields several well-known policysearch algorithms as well as the novel Policy learning by Weighting Exploration with the Returns (PoWER) algorithm. PoWER is an EM-inspired algorithm that employs state-dependent exploration (as discussed in Section 7.2). The update rule is given by μ( s, t) μ( s, t) T ( μ( s, t) Tˆ μ( s, t) ) −1 . Intuitively, this update can be seen as a reward-weighted imitation, (or recombination) of previously seen episodes. Depending on its effect on the state-action value function, the exploration of each episode is incorporated more or less strongly into the updated policy. To reduce the number of trials in this on-policy scenario, the trials are reused through importance sampling  . To avoid the fragility that sometimes results from importance sampling in reinforcement learning, samples with very small importance weights were discarded. In essence, this algorithm performs a local search around the policy learned from demonstration and prior knowledge.