In this work we present a new agent architecture, called Reactor, which combines multiple algorithmic and architectural contributions to produce an agent with higher sample-efficiency than Prioritized Dueling DQN and Categorical DQN , while giving better run-time performance than A3C . Our first contribution is a new policy evaluation algorithm called Distributional Retrace, which brings multi-step off-policy updates to the distributional reinforcement learning setting. The same approach can be used to convert several classes of multi-step policy evaluation algorithms designed for expected value evaluation into distributional ones. Next, we introduce the β-leaveone-out policy gradient algorithm which improves the trade-off between variance and bias by using action values as a baseline. Our final algorithmic contribution is a new prioritized replay algorithm for sequences, which exploits the temporal locality of neighboring observations for more efficient replay prioritization. Using the Atari 2600 benchmarks, we show that each of these innovations contribute to both the sample efficiency and final agent performance. Finally, we demonstrate that Reactor reaches state-of-the-art performance after 200 million frames and less than a day of training. Model-free deep reinforcement learning has achieved several remarkable successes in domains ranging from super-human-level control in video games and the game of Go BID10, to continuous motor control tasks .Much of the recent work can be divided into two categories. First, those of which that, often building on the DQN framework, act -greedily according to an action-value function and train using minibatches of transitions sampled from an experience replay buffer BID10 BID13 BID5 BID0. These value-function agents benefit from improved sample complexity, but tend to suffer from long runtimes (e.g. DQN requires approximately a week to train on Atari). The second category are the actor-critic agents, which includes the asynchronous advantage actor-critic (A3C) algorithm, introduced by. These agents train on transitions collected by multiple actors running, and often training, in parallel (; BID12 . The deep actor-critic agents train on each trajectory only once, and thus tend to have worse sample complexity. However, their distributed nature allows significantly faster training in terms of wall-clock time. Still, not all existing algorithms can be put in the above two categories and various hybrid approaches do exist BID17 ; BID4 BID14 . We consider a Markov decision process (MDP) with state space X and finite action space A. A (stochastic) policy π(·|x) is a mapping from states x ∈ X to a probability distribution over actions. We consider a γ-discounted infinite-horizon criterion, with γ ∈ the discount factor, and define for policy π the action-value of a state-action pair (x, a) as DISPLAYFORM0 where ({x t} t≥0 ) is a trajectory generated by choosing a in x and following π thereafter, i.e., a t ∼ π(·|x t) (for t ≥ 1), and r t is the reward signal. The objective in reinforcement learning is to find an optimal policy π *, which maximises Q π (x, a). The optimal action-values are given by Q * (x, a) = max π Q π (x, a). The Deep Q-Network (DQN) framework, introduced by , popularised the current line of research into deep reinforcement learning by reaching human-level, and beyond, performance across 57 Atari 2600 games in the ALE. While DQN includes many specific components, the essence of the framework, much of which is shared by Neural Fitted Q-Learning , is to use of a deep convolutional neural network to approximate an action-value function, training this approximate action-value function using the Q-Learning algorithm BID15 and mini-batches of one-step transitions (x t, a t, r t, x t+1, γ t) drawn randomly from an experience replay buffer . Additionally, the next-state action-values are taken from a target network, which is updated to match the current network periodically. Thus, the temporal difference (TD) error for transition t used by these algorithms is given by δ t = r t + γ t max a ∈A Q(x t+1, a ;θ) − Q(x t, a t ; θ),where θ denotes the parameters of the network andθ are the parameters of the target network. Since this seminal work, we have seen numerous extensions and improvements that all share the same underlying framework. Double DQN BID10, attempts to correct for the over-estimation bias inherent in Q-Learning by changing the second term of to Q(x t+1, arg max a ∈A Q(x t+1, a ; θ);θ). The dueling architecture BID13, changes the network to estimate action-values using separate network heads V (x; θ) and A(x, a; θ) with Q(x, a; θ) = V (x; θ) + A(x, a; θ) − 1 |A| a A(x, a ; θ).Recently, BID6 introduced Rainbow, a value-based reinforcement learning agent combining many of these improvements into a single agent and demonstrating that they are largely complementary. Rainbow significantly out performs previous methods, but also inherits the poorer time-efficiency of the DQN framework. We include a detailed comparison between Reactor and Rainbow in the Appendix. In the remainder of the section we will describe in more depth other recent improvements to DQN. The experience replay buffer was first introduced by and later used in DQN . Typically, the replay buffer is essentially a first-in-first-out queue with new transitions gradually replacing older transitions. The agent would then sample a mini-batch uniformly at random from the replay buffer. Drawing inspiration from prioritized sweeping , prioritized experience replay replaces the uniform sampling with prioritized sampling proportional to the absolute TD error .Specifically, for a replay buffer of size N, prioritized experience replay samples transition t with probability P (t), and applies weighted importance-sampling with w t to correct for the prioritization bias, where DISPLAYFORM0 Prioritized DQN significantly increases both the sample-efficiency and final performance over DQN on the Atari 2600 benchmarks BID13. Retrace(λ) is a convergent off-policy multi-step algorithm extending the DQN agent . Assume that some trajectory {x 0, a 0, r 0, x 1, a 1, r 1, . . ., x t, a t, r t, . . .,} has been generated according to behaviour policy µ, i.e., a t ∼ µ(·|x t). Now, we aim to evaluate the value of a different target policy π, i.e. we want to estimate Q π. The Retrace algorithm will update our current estimate Q of Q π in the direction of DISPLAYFORM0 DISPLAYFORM1 is the temporal difference at time s under π, and DISPLAYFORM2 The Retrace algorithm comes with the theoretical guarantee that in finite state and action spaces, repeatedly updating our current estimate Q according to produces a sequence of Q functions which converges to Q π for a fixed π or to Q * if we consider a sequence of policies π which become increasingly greedy w.r.t. the Q estimates . Distributional reinforcement learning refers to a class of algorithms that directly estimate the distribution over returns, whose expectation gives the traditional value function BID2. Such approaches can be made tractable with a distributional Bellman equation, and the recently proposed algorithm C51 showed state-of-the-art performance in the Atari 2600 benchmarks. C51 parameterizes the distribution over returns with a mixture over Diracs centered on a uniform grid, DISPLAYFORM0 with hyperparameters v min, v max that bound the distribution support of size N. In this section we review the actor-critic framework for reinforcement learning algorithms and then discuss recent advances in actor-critic algorithms along with their various trade-offs. The asynchronous advantage actor-critic (A3C) algorithm , maintains a parameterized policy π(a|x; θ) and value function V (x; θ v), which are updated with DISPLAYFORM0 where, A(x t, a t ; DISPLAYFORM1 A3C uses M = 16 parallel CPU workers, each acting independently in the environment and applying the above updates asynchronously to a shared set of parameters. In contrast to the previously discussed value-based methods, A3C is an on-policy algorithm, and does not use a GPU nor a replay buffer. Proximal Policy Optimization (PPO) is a closely related actor-critic algorithm , which replaces the advantage with, DISPLAYFORM2 where ρ t is as defined in Section 2.1.2. Although both PPO and A3C run M parallel workers collecting trajectories independently in the environment, PPO collects these experiences to perform a single, synchronous, update in contrast with the asynchronous updates of A3C.Actor-Critic Experience Replay (ACER) extends the A3C framework with an experience replay buffer, Retrace algorithm for off-policy corrections, and the Truncated Importance Sampling Likelihood Ratio (TISLR) algorithm used for off-policy policy optimization BID14. The Reactor is a combination of four novel contributions on top of recent improvements to both deep value-based RL and policy-gradient algorithms. Each contribution moves Reactor towards our goal of achieving both sample and time efficiency. The Reactor architecture represents both a policy π(a|x) and action-value function Q(x, a). We use a policy gradient algorithm to train the actor π which makes use of our current estimate Q(x, a) of Q π (x, a). Let V π (x 0) be the value function at some initial state x 0, the policy gradient theorem says that DISPLAYFORM0, where ∇ refers to the gradient w.r.t. policy parameters BID9. We now consider several possible ways to estimate this gradient. To simplify notation, we drop the dependence on the state x for now and consider the problem of estimating the quantity DISPLAYFORM1 In the off-policy case, we consider estimating G using a single actionâ drawn from a (possibly different from π) behaviour distributionâ ∼ µ. Let us assume that for the chosen actionâ we have access to an unbiased estimate R(â) of Q π (â). Then, we can use likelihood ratio (LR) method combined with an importance sampling (IS) ratio (which we call ISLR) to build an unbiased estimate of G: DISPLAYFORM2 where V is a baseline that depends on the state but not on the chosen action. However this estimate suffers from high variance. A possible way for reducing variance is to estimate G directly from by using the return R(â) for the chosen actionâ and our current estimate Q of Q π for the other actions, which leads to the so-called leave-one-out (LOO) policy-gradient estimate: This estimate has low variance but may be biased if the estimated Q values differ from Q π. A better bias-variance tradeoff may be obtained by the more general β-LOO policy-gradient estimate: DISPLAYFORM3 DISPLAYFORM4 where β = β(µ, π,â) can be a function of both policies, π and µ, and the selected actionâ. Notice that when β = 1, reduces to, and when β = 1/µ(â), then iŝ DISPLAYFORM5 This estimate is unbiased and can be seen as a generalization ofĜ ISLR where instead of using a state-only dependent baseline, we use a state-and-action-dependent baseline (our current estimate Q) and add the correction term a ∇π(a)Q(a) to cancel the bias. Proposition 1 gives our analysis of the bias of G β-LOO, with a proof left to the Appendix. DISPLAYFORM6 Thus the bias is small when β(a) is close to 1/µ(a), or when the Q-estimates are close to the true Q π values, and unbiased regardless of the estimates if β(a) = 1/µ(a). The variance is low when β is small, therefore, in order to improve the bias-variance tradeoff we recommend using the β-LOO estimate with β defined as: β(â) = min c, 1 µ(â), for some constant c ≥ 1. This truncated 1/µ coefficient shares similarities with the truncated IS gradient estimate introduced in BID14 ) (which we call TISLR for truncated-ISLR): DISPLAYFORM7 The differences are: (i) we truncate 1/µ(â) = π(â)/µ(â) × 1/π(â) instead of truncating π(â)/µ(â), which provides an additional variance reduction due to the variance of the LR ∇ log π(â) = DISPLAYFORM8 (since this LR may be large when a low probability action is chosen), and (ii) we use our Q-baseline instead of a V baseline, reducing further the variance of the LR estimate. In off-policy learning it is very difficult to produce an unbiased sample R(â) of Q π (â) when following another policy µ. This would require using full importance sampling correction along the trajectory. Instead, we use the off-policy corrected return computed by the Retrace algorithm, which produces a (biased) estimate of Q π (â) but whose bias vanishes asymptotically .In Reactor, we consider predicting an approximation of the return distribution function from any state-action pair (x, a) in a similar way as in BID2. The original algorithm C51 described in that paper considered single-step Bellman updates only. Here we need to extend this idea to multi-step updates and handle the off-policy correction performed by the Retrace algorithm, as defined in. Next, we describe these two extensions. Multi-step distributional Bellman operator: First, we extend C51 to multi-step Bellman backups. We consider return-distributions from (x, a) of the form i q i (x, a)δ zi (where δ z denotes a Dirac in z)which are supported on a finite uniform grid DISPLAYFORM0 The coefficients q i (x, a) (discrete distribution) corresponds to the probabilities assigned to each atom z i of the grid. From an observed n-step sequence {x t, a t, r t, x t+1, . . ., x t+n}, generated by behavior policy µ (i.e, a s ∼ µ(·|x s) for t ≤ s < t + n), we build the n-step backed-up return-distribution from (x t, a t). The n-step distributional Bellman target, whose expectation is t+n−1 s=t γ s−t r s + γ n Q(x t+n, a), is given by: DISPLAYFORM1 Since this distribution is supported on the set of atoms {z n i}, which is not necessarily aligned with the grid {z i}, we do a projection step and minimize the KL-loss between the projected target and the current estimate, just as with C51 except with a different target distribution BID2.Distributional Retrace: Now, the Retrace algorithm defined in involves an off-policy correction which is not handled by the previous n-step distributional Bellman backup. The key to extending this distributional back-up to off-policy learning is to rewrite the Retrace algorithm as a linear combination of n-step Bellman backups, weighted by some coefficients α n,a. Indeed, notice that rewrites as DISPLAYFORM2 where α n,a = c t+1... c t+n−1 π(a|x t+n) − I{a = a t+n}c t+n. These coefficients depend on the degree of off-policy-ness (between µ and π) along the trajectory. We have that n≥1 a α n,a = n≥1 c t+1... c t+n−1 (1 − c t+n) = 1, but notice some coefficients may be negative. However, in expectation (over the behavior policy) they are non-negative. Indeed, DISPLAYFORM3 by definition of the c s coefficients. Thus in expectation (over the behavior policy), the Retrace update can be seen as a convex combination of n-step Bellman updates. Then, the distributional Retrace algorithm can be defined as backing up a mixture of n-step distributions. More precisely, we define the Retrace target distribution as: DISPLAYFORM4 where h zi (x) is a linear interpolation kernel, projecting onto the support {z i}: DISPLAYFORM5 We update the current probabilities q(x t, a t) by performing a gradient step on the KL-loss DISPLAYFORM6 Again, notice that some target "probabilities" q * i (x t, a t) may be negative for some sample trajectory, but in expectation they will be non-negative. Since the gradient of a KL-loss is linear w.r.t. its first argument, our update rule provides an unbiased estimate of the gradient of the KL between the expected (over the behavior policy) Retrace target distribution and the current predicted distribution. Remark: The same method can be applied to other algorithms (such as TB(λ) and importance sampling ) in order to derive distributional versions of other off-policy multi-step RL algorithms. Prioritized experience replay has been shown to boost both statistical efficiency and final performance of deep RL agents . However, as originally defined prioritized replay does not handle sequences of transitions and weights all unsampled transitions identically. In this section we present an alternative initialization strategy, called lazy initialization, and argue that it better encodes prior information about temporal difference errors. We then briefly describe our computationally efficient prioritized sequence sampling algorithm, with full details left to the appendix. It is widely recognized that TD errors tend to be temporally correlated, indeed the need to break this temporal correlation has been one of the primary justifications for the use of experience replay . Our proposed algorithm begins with this fundamental assumption. Assumption 1. Temporal differences are temporally correlated, with correlation decaying on average with the time-difference between two transitions. Prioritized experience replay adds new transitions to the replay buffer with a constant priority, but given the above assumption we can devise a better method. Specifically, we propose to add experience to the buffer with no priority, inserting a priority only after the transition has been sampled and used for training. Also, instead of sampling transitions, we assign priorities to all (overlapping) sequences of length n. When sampling, sequences with an assigned priority are sampled proportionally to that priority. Sequences with no assigned priority are sampled proportionally to the average priority of assigned priority sequences within some local neighbourhood. Averages are weighted to compensate for sampling biases (i.e. more samples are made in areas of high estimated priorities, and in the absence of weighting this would lead to overestimation of unassigned priorities).The lazy initialization scheme starts with priorities p t corresponding to the sequences {x t, a t, . . ., x t+n} for which a priority was already assigned. Then it extrapolates a priority of all other sequences in the following way. Let us define a partition (I i) i of the states ordered by increasing time such that each cell I i contains exactly one state s i with already assigned priority. We define the estimated priorityp t to all other sequences asp t = si∈J(t) DISPLAYFORM0 is a collection of contiguous cells (I i) containing time t, and w i = |I i | is the length of the cell I i containing s i. For already defined priorities denotep t = p t. Cell sizes work as estimates of inverse local density and are used as importance weights for priority estimation.2 For the algorithm to be unbiased, partition (I i) i must not be a function of the assigned priorities. So far we have defined a class of algorithms all free to choose the partition (I i) and the collection of cells I(t), as long that they satisfy the above constraints. FIG6 in the Appendix illustrates the above description. Now, with probability we sample uniformly at random, and with probability 1 − we sample proportionally top t. We implemented an algorithm satisfying the above constraints and called it Contextual Priority Tree (CPT). It is based on AVL trees BID11 and can execute sampling, insertion, deletion and density evaluation in O(ln(n)) time. We describe CPT in detail in the Appendix in Section 6.3.We treated prioritization as purely a variance reduction technique. Importance-sampling weights were evaluated as in prioritized experience replay, with fixed β = 1 in. We used simple gradient magnitude estimates as priorities, corresponding to a mean absolute TD error along a sequence for Retrace, as defined in for the classical RL case, and total variation in the distributional Retrace case. In order to improve CPU utilization we decoupled acting from learning. This is an important aspect of our architecture: an acting thread receives observations, submits actions to the environment, and stores transitions in memory, while a learning thread re-samples sequences of experiences from memory and trains on them (Figure 2, left). We typically execute 4-6 acting steps per each learning step. We sample sequences of length n = 33 in batches of 4. A moving network is unrolled over frames 1-32 while the target network is unrolled over frames 2-33.We allow the agent to be distributed over multiple machines each containing action-learner pairs. Each worker downloads the newest network parameters before each learning step and sends delta-updates at the end of it. Both the network and target network are stored on a shared parameter server while each machine contains its own local replay memory. Training is done by downloading a shared network, evaluating local gradients and sending them to be applied on the shared network. While the agent can also be trained on a single machine, in this work we present of training obtained with either 10 or 20 actor-learner workers and one parameter server. In Figure 2 (right) we compare resources and runtimes of Reactor with related algorithms. In some domains, such as Atari, it is useful to base decisions on a short history of past observations. The two techniques generally used to achieve this are frame stacking and recurrent network architectures. We chose the latter over the former for reasons of implementation simplicity and computational efficiency. As the Retrace algorithm requires evaluating action-values over contiguous sequences of trajectories, using a recurrent architecture allowed each frame to be processed by the convolutional network only once, as opposed to n times times if n frame concatenations were used. The Reactor architecture uses a recurrent neural network which takes an observation x t as input and produces two outputs: categorical action-value distributions q i (x t, a) (i here is a bin identifier), and policy probabilities π(a|x t). We use an architecture inspired by the duelling network architecture BID13. We split action-value -distribution logits into state-value logits and advantage logits, which in turn are connected to the same LSTM network BID7.Final action-value logits are produced by summing state-and action-specific logits, as in BID13. Finally, a softmax layer on top for each action produces the distributions over discounted future returns. The policy head uses a softmax layer mixed with a fixed uniform distribution over actions, where this mixing ratio is a hyperparameter (, Section 5.1.3). Policy and Q-networks have separate LSTMs. Both LSTMs are connected to a shared linear layer which is connected to a shared convolutional neural network . The precise network specification is given in TAB6 in the Appendix. Gradients coming from the policy LSTM are blocked and only gradients originating from the Qnetwork LSTM are allowed to back-propagate into the convolutional neural network. We block gradients from the policy head for increased stability, as this avoids positive feedback loops between π and q i caused by shared representations. We used the Adam optimiser , with a learning rate of 5 × 10 −5 and zero momentum because asynchronous updates induce implicit momentum . Further discussion of hyperparameters and their optimization can be found in Appendix 6.1. We trained and evaluated Reactor on 57 Atari games BID1. FIG4 compares the performance of Reactor with different versions of Reactor each time leaving one of the algorithmic improvements out. We can see that each of the algorithmic improvements (Distributional retrace, beta-LOO and prioritized replay) contributed to the final . While prioritization was arguably the most important component, Beta-LOO clearly outperformed TISLR algorithm. Although distributional and non-distributional versions performed similarly in terms of median human normalized scores, distributional version of the algorithm generalized better when tested with random human starts (We evaluated Reactor with target update frequency T update = 1000, λ = 1.0 and β-LOO with β = 1 on 57 Atari games trained on 10 machines in parallel. We averaged scores over 200 episodes using 30 random human starts and noop starts TAB8 in the Appendix). We calculated mean and median human normalised scores across all games. We also ranked all algorithms (including random and human scores) for each game and evaluated mean rank of each algorithm across all 57 Atari games. We also evaluated mean Rank and Elo scores for each algorithm for both human and noop start settings. Please refer to Section 6.2 in the Appendix for more details. TAB2 with several other state-of-art algorithms across 57 Atari games for a fixed random seed across all games BID1. We compare Reactor against are: DQN , Double DQN BID10, DQN with prioritised experience replay BID13, dueling architecture and prioritised dueling BID13, ACER BID14, A3C , and Rainbow BID6. Each algorithm was exposed to 200 million frames of experience, or 500 million frames when followed by 500M, and the same pre-processing pipeline including 4 action repeats was used as in the original DQN paper .In TAB2, we see that Reactor exceeds the performance of all algorithms across all metrics, despite requiring under two days of training. With 500 million frames and four days training we see Reactor's performance continue to improve significantly. The difference in time-efficiency is especially apparent when comparing Reactor and Rainbow (see FIG4 . Additionally, unlike Rainbow, Reactor does not use Noisy Networks BID3, which was reported to have contributed to the performance gains. When evaluating under the no-op starts regime TAB4, Reactor out performs all methods except for Rainbow. This suggests that Rainbow is more sample-efficient when training and evaluation regimes match exactly, but may be overfitting to particular trajectories due to the significant drop in performance when evaluated on the random human starts. Regarding ACER, another Retrace-based actor-critic architecture, both classical and distributional versions of Reactor FIG4) exceeded the best reported median human normalized score of 1.9 with noop starts achieved in 500 million steps. In this work we presented a new off-policy agent based on Retrace actor-critic architecture and show that it achieves similar performance as the current state-of-the-art while giving significant real-time performance gains. We demonstrate the benefits of each of the suggested algorithmic improvements, including Distributional Retrace, beta-LOO policy gradient and contextual priority tree. DISPLAYFORM0 Proof. The bias ofĜ β-LOO is DISPLAYFORM1 As we believe that algorithms should be robust with respect to the choice of hyperparameters, we spent little effort on parameter optimization. In total, we explored three distinct values of learning rates and two values of ADAM momentum (the default and zero) and two values of T update on a subset of 7 Atari games without prioritization using non-distributional version of Reactor. We later used those values for all experiments. We did not optimize for batch sizes and sequence length or any prioritization hyperparamters. Commonly used mean and median human normalized scores have several disadvantages. A mean human normalized score implicitly puts more weight on games that computers are good and humans are bad at. Comparing algorithm by a mean human normalized score across 57 Atari games is almost equivalent to comparing algorithms on a small subset of games close to the median and thus dominating the signal. Typically a set of ten most score-generous games, namely Assault, Asterix, Breakout, Demon Attack, Double Dunk, Gopher, Pheonix, Stargunner, Up'n Down and Video Pinball can explain more than half of inter-algorithm variance. A median human normalized score has the opposite disadvantage by effectively discarding very easy and very hard games from the comparison. As typical median human normalized scores are within the range of 1-2.5, an algorithm which scores zero points on Montezuma's Revenge is evaluated equal to the one which scores 2500 points, as both performance levels are still below human performance making incremental improvements on hard games not being reflected in the overall evaluation. In order to address both problem, we also evaluated mean rank and Elo metrics for inter-algorithm comparison. Those metrics implicitly assign the same weight to each game, and as a is more sensitive of relative performance on very hard and easy games: swapping scores of two algorithms on any game would in the change of both mean rank and Elo metrics. We calculated separate mean rank and Elo scores for each algorithm using of test evaluations with 30 random noop-starts and 30 random human starts TAB8 ). All algorithms were ranked across each game separately, and a mean rank was evaluated across 57 Atari games. For Elo score evaluation algorithm, A was considered to win over algorithm B if it obtained more scores on a given Atari. We produced an empirical win-probability matrix by summing wins across all games and used this matrix to evaluate Elo scores. A ranking difference of 400 corresponds to the odds of winning of 10:1 under the Gaussian assumption. Contextual priority tree is one possible implementation of lazy prioritization FIG6 ). All sequence keys are put into a balanced binary search tree which maintains a temporal order. An AVL tree BID11 ) was chosen due to the ease of implementation and because it is on average more evenly balanced than a Red-Black Tree. Each tree node has up to two children (left and right) and contains currently stored key and a priority of the key which is either set or is unknown. Some trees may only have a single child subtree while Sampling is done by going from the root node up the tree by selecting one of the children (or the current key) stochastically proportional to orange proportions. Sampling terminates once the current (square) key is chosen. Figure 6: Example of a balanced priority tree. Dark blue nodes contain keys with known priorities, light blue nodes have at least one child with at least a single known priority, while ping nodes do not have any priority estimates. Nodes 1, 2 and 3 will obtain priority estimates equal to 2/3 of the priority of key 5 and 1/3 of the priority of node 4. This implies that estimated priorities of keys 1, 2 and 3 are implicitly defined by keys 4 and 6. Nodes 8, 9 and 11 are estimated to have the same priority as node 10.some may have none. In addition to this information, we were tracking other summary statistics at each node which was re-evaluated after each tree rotation. The summary statistics was evaluated by consuming previously evaluated summary statistics of both children and a priority of the key stored within the current node. In particular, we were tracking a total number of nodes within each subtree and mean-priority estimates updated according to rules shown in FIG7. The total number of nodes within each subtree was always known (c in FIG7), while mean priority estimates per key (m in FIG7) could either be known or unknown. If a mean priority of either one child subtree or a key stored within the current node is unknown then it can be estimated to by exploiting information coming from another sibling subtree or a priority stored within the parent node. Sampling was done by traversing the tree from the root node up while sampling either one of the children subtrees or the currently held key proportionally to the total estimated priority masses contained within. The rules used to evaluate proportions are shown in orange in FIG7. Similarly, probabilities of arbitrary keys can be queried by traversing the tree from the root node towards the child node of an interest while maintaining a product of probabilities at each branching point. Insertion, deletion, sampling and probability query operations can be done in O(ln(n)) time. The suggested algorithm has the desired property that it becomes a simple proportional sampling algorithm once all the priorities are known. While some key priorities are unknown, they are estimated by using nearby known key priorities (Figure 6).Each time when a new sequence key is added to the tree, it was set to have an unknown priority. Any priority was assigned only after the key got first sampled and the corresponding sequence got passed through the learner. When a priority of a key is set or updated, the key node is deliberately removed from and placed back to the tree in order to become a leaf-node. This helped to set priorities of nodes in the immediate vicinity more accurately by using the freshest information available. The value of = 0.01 is the minimum probability of choosing a random action and it is hard-coded into the policy network. FIG8 shows the overall network topology while TAB6 specifies network layer sizes. In this section we compare Reactor with the recently published Rainbow agent BID6. While ACER is the most closely related algorithmically, Rainbow is most closely related in terms of performance and thus a deeper understanding of the trade-offs between Rainbow and Reactor may benefit interested readers. There are many architectural and algorithmic differences between Rainbow and Reactor. We will therefore begin by highlighting where they agree. Both use a categorical action-value distribution critic BID2, factored into state and state-action logits BID13, DISPLAYFORM0 Both use prioritized replay, and finally, both perform n-step Bellman updates. Despite these similarities, Reactor and Rainbow are fundamentally different algorithms and are based upon different lines of research. While Rainbow uses Q-Learning and is based upon DQN , Reactor is an actor-critic algorithm most closely based upon A3C . Each inherits some design choices from their predecessors, and we have not performed an extensive ablation comparing these various differences. Instead, we will discuss four of the differences we believe are important but less obvious. First, the network structures are substantially different. Rainbow uses noisy linear layers and ReLU activations throughout the network, whereas Reactor uses standard linear layers and concatenated ReLU activations throughout. To overcome partial observability, Rainbow, inheriting this choice from DQN, uses frame stacking. On the other hand, Reactor, inheriting its choice from A3C, uses LSTMs after the convolutional layers of the network. It is also difficult to directly compare the number of parameters in each network because the use of noisy linear layers doubles the number of parameters, although half of these are used to control noise, while the LSTM units in Reactor require more parameters than a corresponding linear layer would. Second, both algorithms perform n-step updates, however, the Rainbow n-step update does not use any form of off-policy correction. Because of this, Rainbow is restricted to using only small values of n (e.g. n = 3) because larger values would make sequences more off-policy and hurt performance. By comparison, Reactor uses our proposed distributional Retrace algorithm for off-policy correction of n-step updates. This allows the use of larger values of n (e.g. n = 33) without loss of performance. Third, while both agents use prioritized replay buffers , they each store different information and prioritize using different algorithms. Rainbow stores a tuple containing the state x t−1, action a t−1, sum of n discounted rewards n−1 k=0 r t+k k−1 m=0 γ t+m, product of n discount factors n−1 k=0 γ t+k, and next-state n steps away x t+n−1. Tuples are prioritized based upon the last observed TD error, and inserted into replay with a maximum priority. Reactor stores length n sequences of tuples (x t−1, a t−1, r t, γ t) and also prioritizes based upon the observed TD error. However, when inserted into the buffer the priority is instead inferred based upon the known priorities of neighboring sequences. This priority inference was made efficient using the previously introduced contextual priority tree, and anecdotally we have seen it improve performance over a simple maximum priority approach. Finally, the two algorithms have different approaches to exploration. Rainbow, unlike DQN, does not use -greedy exploration, but instead replaces all linear layers with noisy linear layers which induce randomness throughout the network. This method, called Noisy Networks BID3, creates an adaptive exploration integrated into the agent's network. Reactor does not use noisy networks, but instead uses the same entropy cost method used by A3C and many others , which penalizes deterministic policies thus encouraging indifference between similarly valued actions. Because Rainbow can essentially learn not to explore, it may learn to become entirely greedy in the early parts of the episode, while still exploring in states not as frequently seen. In some sense, this is precisely what we want from an exploration technique, but it may also lead to highly deterministic trajectories in the early part of the episode and an increase in overfitting to those trajectories. We hypothesize that this may be the explanation for the significant difference in Rainbow's performance between evaluation under no-op and random human starts, and why Reactor does not show such a large difference.6.6 ATARI Table 4: Scores for each game evaluated with 30 random human starts. Reactor was evaluated by averaging scores over 200 episodes. All scores (except for Reactor) were taken from BID13, and BID6
Reactor combines multiple algorithmic and architectural contributions to produce an agent with higher sample-efficiency than Prioritized Dueling DQN while giving better run-time performance than A3C.