Safe and Efficient Off Policy Reinforcement Learning
In this work, we take a fresh look at some old and new algorithms for off policy, return based reinforcement learning. Expressing these in a common form, we derive a novel algorithm, Retrace(\lambda$ with three desired properties: (1) it has low variance; (2) it safely uses samples collected from any behaviour policy, whatever its degree of "off policyness" and (3) it is efficient as it makes the best use of samples collected from near on policy behaviour policies. We analyze the contractive nature of the related operator under both off policy policy evaluation and control settings and derive online sample based algorithms. We believe this is the first return based off policy control algorithm converging a.s. to $Q^ without the GLIE assumption (Greedy in the Limit with Infinite Exploration) As a corollary, we prove the convergence of Watkins' Q(\lambda$ which was an open problem since 1989. We illustrate the benefits of Retrace(\lambda$ on a standard suite of Atari 2600 games.
NIPS