Reinforcement learning in robotics: A survey <s> Value-function approaches versus policy search <s> A straightforward approach to the curse of dimensionality in reinforcement learning and dynamic programming is to replace the lookup table with a generalizing function approximator such as a neural net. Although this has been successful in the domain of backgammon, there is no guarantee of convergence. In this paper, we show that the combination of dynamic programming and function approximation is not robust, and in even very benign cases, may produce an entirely wrong policy. We then introduce Grow-Support, a new algorithm which is safe from divergence yet can still reap the benefits of successful generalization. <s> BIB001 </s> Reinforcement learning in robotics: A survey <s> Value-function approaches versus policy search <s> Policy gradient methods for reinforcement learning avoid some of the undesirable properties of the value function approaches, such as policy degradation (Baxter and Bartlett, 2001). However, the variance of the performance gradient estimates obtained from the simulation is sometimes excessive. In this paper, we consider variance reduction methods that were developed for Monte Carlo estimates of integrals. We study two commonly used policy gradient techniques, the baseline and actor-critic methods, from this perspective. Both can be interpreted as additive control variate variance reduction methods. We consider the expected average reward performance measure, and we focus on the GPOMDP algorithm for estimating performance gradients in partially observable Markov decision processes controlled by stochastic reactive policies. We give bounds for the estimation error of the gradient estimates for both baseline and actor-critic algorithms, in terms of the sample size and mixing properties of the controlled system. For the baseline technique, we compute the optimal baseline, and show that the popular approach of using the average reward to define the baseline can be suboptimal. For actor-critic algorithms, we show that using the true value function as the critic can be suboptimal. We also discuss algorithms for estimating the optimal baseline and approximate value function. <s> BIB002 </s> Reinforcement learning in robotics: A survey <s> Value-function approaches versus policy search <s> Reinforcement learning (RL) methods provide a means for solving optimal control problems when accurate models are unavailable. For many such problems, however, RL alone is impractical and the associated learning problem must be structured somehow to take advantage of prior knowledge. In this paper we examine the use of such knowledge in the form of a stable controller that generates control inputs in parallel with an RL system. The controller acts as a supervisor that not only teaches the RL system about favorable control actions but also protects the learning system from risky behavior. We demonstrate the approach with a simulated robotic arm and a real seven-DOF manipulator. <s> BIB003
Some methods attempt to find a value function or policy which eventually can be employed without significant further computation, whereas others (e.g. the roll-out methods) perform the same amount of computation each time. If a complete optimal value function is known, a globally optimal solution follows simply by greedily choosing actions to optimize it. However, value-function-based approaches have thus far been difficult to translate into high-dimensional robotics as they require function approximation for the value function. Most theoretical guarantees no longer hold for this approximation and even finding the optimal action can be a hard problem due to the brittleness of the approximation and the cost of optimization. For high-dimensional actions, it can be as hard to compute an improved policy for all states in policy search as finding a single optimal action on-policy for one state by searching the state-action value function. In principle, a value function requires total coverage of the state-space and the largest local error determines the quality of the resulting policy. A particularly significant problem is the error propagation in value functions. A small change in the policy may cause a large change in the value function, which again causes a large change in the policy. While this may lead more quickly to good, possibly globally optimal solutions, such learning processes often prove unstable under function approximation BIB001  and are considerably more dangerous when applied to real systems where overly large policy deviations may lead to dangerous decisions. In contrast, policy-search methods usually only consider the current policy and its neighborhood in order to gradually improve performance. The result is that usually only local optima, and not the global one, can be found. However, these methods work well in conjunction with continuous features. Local coverage and local errors results into improved scalability in robotics. Policy-search methods are sometimes called actor-only methods; value-function methods are sometimes called critic-only methods. The idea of a critic is to first observe and estimate the performance of choosing controls on the system (i.e. the value function), then derive a policy based on the gained knowledge. In contrast, the actor directly tries to deduce the optimal policy. A set of algorithms called actor-critic methods attempt to incorporate the advantages of each: a policy is explicitly maintained, as is a valuefunction for the current policy. The value function (i.e. the critic) is not employed for action selection. Instead, it observes the performance of the actor and decides when the policy needs to be updated and which action should be preferred. The resulting update step features the local convergence properties of policy gradient algorithms while reducing update variance BIB002 . There is a trade-off between the benefit of reducing the variance of the updates and having to learn a value function as the samples required to estimate the value function could also be employed to obtain better gradient estimates for the update step. BIB003 proposed an actor-critic method that additionally features a supervisor in the form of a stable policy.