We investigate the combination of actor-critic reinforcement learning algorithms with uniform large-scale experience replay and propose solutions for two challenges: (a) efficient actor-critic learning with experience replay (b) stability of very off-policy learning. We employ those insights to accelerate hyper-parameter sweeps in which all participating agents run concurrently and share their experience via a common replay module.

 To this end we analyze the bias-variance tradeoffs in V-trace, a form of importance sampling for actor-critic methods. Based on our analysis, we then argue for mixing experience sampled from replay with on-policy experience, and propose a new trust region scheme that scales effectively to data distributions where V-trace becomes unstable.

 We provide extensive empirical validation of the proposed solution. We further show the benefits of this setup by demonstrating state-of-the-art data efficiency on Atari among agents trained up until 200M environment frames. Value-based and actor-critic policy gradient methods are the two leading techniques of constructing general and scalable reinforcement learning agents (Sutton et al., 2018) . Both have been combined with non-linear function approximation (Tesauro, 1995; Williams, 1992) , and have achieved remarkable successes on multiple challenging domains; yet, these algorithms still require large amounts of data to determine good policies for any new environment. To improve data efficiency, experience replay agents store experience in a memory buffer (replay) (Lin, 1992) , and reuse it multiple times to perform reinforcement learning updates (Riedmiller, 2005) . Experience replay allows to generalize prioritized sweeping (Moore & Atkeson, 1993) to the non-tabular setting (Schaul et al., 2015) , and can also be used to simplify exploration by including expert (e.g., human) trajectories (Hester et al., 2017) . Overall, experience replay can be very effective at reducing the number of interactions with the environment otherwise required by deep reinforcement learning algorithms (Schaul et al., 2015) . Replay is often combined with the value-based Q-learning (Mnih et al., 2015) , as it is an off-policy algorithm by construction, and can perform well even if the sampling distribution from replay is not aligned with the latest agent's policy. Combining experience replay with actor-critic algorithms can be harder due to their on-policy nature. Hence, most established actor-critic algorithms with replay such as (Wang et al., 2017; Gruslys et al., 2018; Haarnoja et al., 2018) employ and maintain Q-functions to learn from the replayed off-policy experience. In this paper, we demonstrate that off-policy actor-critic learning with experience replay can be achieved without surrogate Q-function approximators using V-trace by employing the following approaches: a) off-policy replay experience needs to be mixed with a proportion of on-policy experience. We show experimentally ( Figure 2 ) and theoretically that the V-trace policy gradient is otherwise not guaranteed to converge to a locally optimal solution. b) a trust region scheme (Conn et al., 2000; Schulman et al., 2015; can mitigate bias and enable efficient learning in a strongly off-policy regime, where distinct agents share experience through a commonly shared replay module. Sharing experience permits the agents to benefit from parallel exploration (Kretchmar, 2002) (Figures 1 and 3 ). Our paper is structured as follows: In Section 2 we revisit pure importance sampling for actor-critic agents (Degris et al., 2012 ) and V-trace, which is notable for allowing to trade off bias and variance in its estimates. We recall that variance reduction is necessary (Figure 4 left) but is biased in V-trace. We derive proposition 2 stating that off-policy V-trace is not guaranteed to converge to a locally optimal solution -not even in an idealized scenario when provided with the optimal value function. Through theoretical analysis (Section 3) and experimental validation (Figure 2 ) we determine that mixing on-policy experience into experience replay alleviates the problem. Furthermore we propose a trust region scheme (Conn et al., 2000; Schulman et al., 2015; in Section 4 that enables efficient learning even in a strongly off-policy regime, where distinct agents share the experience replay module and learn from each others experience. We define the trust region in policy space and prove that the resulting estimator is correct (i.e. estimates an improved return). As a result, we present state-of-the-art data efficiency in Section 5 in terms of median human normalized performance across 57 Atari games (Bellemare et al., 2013) , as well as improved learning efficiency on DMLab30 (Beattie et al., 2016) (Table 1 ). Figure 1: Sharing experience between agents leads to more efficient hyper-parameter sweeps on 57 Atari games. Prior art results are presented as horizontal lines (with scores cited from Gruslys et al. (2018) , Hessel et al. (2017) and Mnih et al. (2013) ). Note that the only previous agent "R2D2" that achieved a score beyond 400% required more than 3,000 million environment steps (see Kapturowski et al. (2019) , page 14, Figure 9 ). We present the pointwise best agent from hyper-parameter sweeps with and without experience replay (shared and not shared). Each sweep contains 9 agents with different learning rate and entropy cost combinations. Replay experiment were repeated twice and ran for 50M steps. To report scores at 200M we ran the baseline and one shared experience replay agent for 200M steps. Table 1 : Comparison of state-of-the-art agents on 57 Atari games trained up until 200M environment steps (per game) and DMLab-30 trained until 10B steps (multi-task; all games combined). The first two rows are quoted from Xu et al. (2018) and Hessel et al. (2019) , the third is our implementation of a pixel control agent from Hessel et al. (2019) and the last two rows are our proposed LASER (LArge Scale Experience Replay) agent. All agents use hyper-parameter sweeps expect for the marked. V-trace importance sampling is a popular off-policy correction for actor-critic agents (Espeholt et al., 2018) . In this section we revisit how V-trace controls the (potentially infinite) variance that arises from naive importance sampling. We note that this comes at the cost of a biased estimate (see Proposition 1) and creates a failure mode (see Proposition 2) which makes the policy gradient biased. We discuss our solutions for said issues in Section 4. Figure 2: Left: Learning entirely off-policy from experience replay fails, while combining on-policy data with experience replay leads to improved data efficiency: We present sweeps on DMLab-30 with experience replays of 10M capacity. A ratio of 87.5% implies that there are 7 replayed transitions in the batch for each online transition. Furthermore we consider an agent identical to "LASER 87.5% replay" which however draws all samples from replay. Its batch thus does not contain any online data and we observe a significant performance decrease (see Proposition 2 and 3). The shading represents the point-wise best and worst replica among 3 repetitions. The solid line is the mean. Right: The effect of capacity in experience replay with 87.5% replay data per batch on sweeps on DMLab-30. Data-efficiency improves with larger capacity. Figure 3: Left: Naively sharing experience between distinct agents in a hyper-parameter sweep fails (green) and is worse than the no-replay baseline (blue). The proposed trust region estimator mitigates the issue (red). Right: Combining population based training with trust region estimation improves performance further. All replay experiments use a capacity of 10 million observations and 87.5% replay data per batch. We have presented LASER -an off-policy actor-critic agent which employs a large and shared experience replay to achieve data-efficiency. By sharing experience between concurrently running experiments in a hyper-parameter sweep it is able to take advantage of parallel exploration. As a result it achieves state-of-the-art data efficiency on 57 Atari games given 200M environment steps. Furthermore it achieves competitive results on both DMLab-30 and Atari under regular, not shared experience replay conditions. To facilitate this algorithm we have proposed two approaches: a) mixing replayed experience and on-policy data and b) a trust region scheme. We have shown theoretically and demonstrated through a series of experiments that they enable learning in strongly off-policy settings, which present a challenge for conventional importance sampling schemes.
We investigate and propose solutions for two challenges in reinforcement learning: (a) efficient actor-critic learning with experience replay (b) stability of very off-policy learning.
{"src": "scitldr-AIC_train", "paper_id": "HygaikBKvS"}