This paper presents GumbelClip, a set of modifications to the actor-critic algorithm, for off-policy reinforcement learning. GumbelClip uses the concepts of truncated importance sampling along with additive noise to produce a loss function enabling the use of off-policy samples. The modified algorithm achieves an increase in convergence speed and sample efficiency compared to on-policy algorithms and is competitive with existing off-policy policy gradient methods while being significantly simpler to implement. The effectiveness of GumbelClip is demonstrated against existing on-policy and off-policy actor-critic algorithms on a subset of the Atari domain. Recent advances in reinforcement learning (RL) have enabled the extension of long-standing methods to complex and large-scale tasks such as Atari , Go, and DOTA . The key driver has been the use of deep neural networks, a non-linear function approximator, with the combination usually referred to as Deep Reinforcement Learning (DRL) . However, deep learning-based methods are usually data-hungry, requiring millions of samples before the network converges to a stable solution. As such, DRL methods are usually trained in a simulated environment where an arbitrary amount of data can be generated. RL algorithms can be classified as either learning in an off-policy or on-policy setting. In the onpolicy setting, an agent learns directly from experience generated by its current policy. In contrast, the off-policy setting enables the agent to learn from experience generated by its current policy or/and other separate policies. An algorithm that learns in the off-policy setting has much greater sample efficiency as old experience from the current policy can be reused; it also enables off-policy algorithms to learn an optimal policy while executing an exploration-focused policy . The most famous off-policy method is Q-Learning which learns an actionvalue function, Q(s, a), that maps the value to a state s and action a pair. Deep Q-Learning (DQN), the marriage of Q-Learning with deep neural networks, was popularised by and used various modifications, such as experience replay, for stable convergence. Within DQN, experience replay is often motivated as a technique for reducing sample correlation. Unfortunately, all action-value methods, including Q-Learning, have two significant disadvantages. First, they learn deterministic policies, which cannot handle problems that require stochastic policies. Second, finding the greedy action with respect to the Q function is costly for large action spaces. To overcome these limitations, one could use policy gradient algorithms , such as actor-critic methods, which learn in an on-policy setting at the cost of sample efficiency. The ideal solution would be to combine the sample efficiency of off-policy algorithms with the desirable attributes of on-policy algorithms. Work along this line has been done by using importance sampling or by combining several techniques together, as in ACER . However, the ing methods are quite complex and require many modifications to existing algorithms. This paper, proposes a set of adjustments to A2C, a parallel on-policy actor-critic algorithm, enabling off-policy learning from stored trajectories. Therefore, our contributions are as follows: • GumbelClip, a fully off-policy actor-critic algorithm, the of a small set of simple adjustments, in under 10 lines of code (LOC) 1, to the A2C algorithm. • GumbelClip has increased sample efficiency and overall performance over on-policy actorcritic algorithms, such as A2C. • GumbelClip performs similarily to other off-policy actor-critic algorithms, such as ACER, while being significantly simpler to implement. The paper is organized as follows: Section 2 covers information, Section 3 describes the GumbelClip algorithm, Section 4 details the experiments along with , discussion, and ablations of our methodology. Section 5 discusses possible future work, and finally Section 6 provides concluding remarks. Consider an agent interacting with a Markov Decision Process (MDP) consisting of a set of states S, a set of actions A, a transition function P:, and a reward function r: S × A → R. Within this work, discrete actions and time steps are assumed. A policy is a probability distribution over actions conditioned on states, π: At each time step t, the agent observes the environment state s t ∈ S, chooses an action a t ∈ A from its policy π(a t |s t), and receives a reward r t from the environment. The goal of the agent is to maximize the discounted future return The discount factor γ ∈ trades off the importance of immediate and future rewards. Following from this, the value function of a policy π, in a discounted problem, is defined as the expected return To optimize the parameters θ of the stochastic policy π the policy gradient theorem is used, which provides an expression for the gradient of the discounted reward objectives with respect to parameter θ. Therefore, the parameters θ of the differentiable stochastic policy π θ (a t |s t) are updated as: where Ψ π (s t, a t), as shown by Schulman et al. (2015b), can be replaced with quantities such as: the total reward of the trajectory, the TD residual, or the state-action value function Q π (s t, a t). The choice of Ψ π affects the variance of the estimated gradient. This work uses the advantage function, which provides a relative measure of value for each action. The advantage function helps to reduce the variance of the gradient estimator while keeping the bias unchanged. The Gumbel-Softmax distribution (GSD) is a continuous distribution used to approximate samples from a categorical distribution. The GSD uses standard Gumbel noise to sample directly from the softmax distribution. On its own, the Gumbel distribution is typically used to model the maximum of a set of independent samples. Given categorical class probabilities π 1, π 2,..., π k the Gumbel-Softmax distribution is defined as: 1 Assuming code for replay memory is available. In practice, the policy gradient is estimated from a trajectory of samples generated by the on-policy stationary distribution π(a|s). This limits the efficiency of typical policy gradient methods, such as actor-critic, compared to methods like Deep Q-Learning which can learn off-policy. A common approach to using off-policy samples is a technique known as importance sampling (; ; ;). Given a trajectory of samples generated by some behaviour policy B(a|s), the policy gradient from Equation 1 is modified to be: where ρ is the known as the importance weight and is defined as a ratio between the current policy π(a|s) and the behaviour policy B(a|s): Unfortunately, the importance weighted gradient in Equation 3 suffers from high variance. To reduce variance, Wawrzyński proposed truncating each importance weight to the interval [0, c] where c is some constant. GumbelClip builds off the on-policy actor-critic algorithm A2C. To enable off-policy learning GumbelClip uses clipped importance sampling, policy forcing through Gumbel noise, and large batches sampled from a replay memory. Psuedo-code for GumbelClip is provided in Algorithm 1. We begin by defining a few quantities. In importance weighting, two policy classes exist: the current policy π(a|s; θ) and the behaviour policy B(a|s). Because replay memory is being used, the behaviour policy is simply the distribution over actions with an old parameter setting θ *: GumbelClip introduces a third policy, the forced policy F(a|s) using the Gumbel-Softmax distribution, which from adding noise (i) sampled from a standard Gumbel distribution to the normalized logits of the current policy: As the name implies, adding Gumbel noise has the effect of "forcing" the sampled policy distribution to be more categorical such that one action contains most of the probability mass. As Equation 6 is identical to the Gumbel-Softmax distribution, with temperature τ = 1, we can refer to Figure 1b) to understand the characteristics of the ing sampled distribution. Initialize parameters θ and θv. Initialize replay memory D with capacity N. repeat for i ∈ {0, · · ·, k} do Perform ai according to π(·|si; θ). Receive reward ri and new state si+1., 0, c end for end for Perform update of θ using dθ and θv using dθv. until Max iteration or time reached. Similarly to previous work, this study uses importance sampling ρ to weight the updates of the loss function. However, instead of using the current policy, π(·|s t), in the numerator, it is replaced with the forced policy F(a t |s t): The range of ρ t is clipped to [0, c] and this clipped importance weight is referred to asρ t. Clipping the upper bound prevents the product of many importance weights from exploding and reduces the variance (Wawrzyński, 2009). Putting this all together the update equation for GumbelClip is as follows: The gradient is estimated by uniformly sampling b trajectories of length k from a replay memory with size N. In addition, the network parameters are updated using the forced policy F(a|s). The advantage function A(s t, a t) = R is the bootstrapped k-step return for time t. As the forced policy in the numerator tends towards a categorical distribution, it became evident that the importance weights had the habit of clumping near the boundaries of the interval and often near 1. Following from Figure 2, we can see the distribution of the ratio between GumbelClip and a typical application of truncated importance sampling to A2C as might be suggested by Wawrzyński. The effect of clipping and Gumbel noise has an interesting effect on the way the policy is updated. From Figure 2 (b), we see that three modes exist, which roughly correspond to the cases of agreement and disagreement. The case of agreement corresponds to ratios and the mode near 1 while disagreements correspond to ratios and modes at 0 and c. More exactly, when the forced policy disagrees with the behaviour policy, say F(·|s) ≈ 1 and B(·|s) ≈ 0, the update the policy receives is at most clipped by the upper bound of our interval: c. On the other hand, when the situation is reversed, but still in disagreement, with F(·|s) ≈ 0 and B(·|s) ≈ 1, the policy has an importance weight of~0. Our experiments focus on the Atari domain as there exists a large amount of variety between environments and the states are represented as raw high-dimensional pixels. The gym software package by was used to conduct all the experiments. All experiments used the same algorithm, network architecture, and hyper-parameters to learn to play Atari games using only raw pixel observations. This study used the same input pre-processing and network architecture as. The network architecture consists of three convolutional layers as follows: 32 8 × 8 filters with stride 4, 64 4 × 4 filters with stride 2, and 32 3 × 3 filters with stride 1. The final convolutional layer feeds into a fully-connected layer with 512 units. All layers are followed by rectified non-linearity. Finally, the network outputs a softmax policy over actions and a state-value. The experimental set-up used 16 threads running on a GPU equipped machine. All experiments trained for 40 million frames. A replay memory of size N = 250000 was kept, an update was performed every k = 5 steps in the environment, and a clamping coefficient of c = 4 was used. The optimization procedure used RMSProp with a learning rate of 5e − 4, entropy regularization of 0.01, and a discount factor of γ = 0.99. We sample 64 trajectories of length 5 for each update. Learning begins after we have collected 10, 000 samples in the replay memory. We tuned the hyperparameters and developed GumbelClip on the FishingDerby environment only; the other environments can be considered "out of sample". All experiments used the same hyperparameter settings and network architecture. We use the best 3 of 4 seeds and report the mean value with 1 standard deviation as shaded areas on all graphs. As GumbelClip uses larger batchsizes we see an improvement in total run time per seed of 6-7 hours over A2C which takes 9-10 hours per seed. Due to limited computational resources, we were only able to evaluate GumbelClip on a subset of the environments and with a smaller replay memory size. Therefore, in this study, an effort was made to select environments that best showcase the performance of off-policy (ACER) and on-policy (A2C) Figure 3: Training performance across 8 Atari games. We see the performance of GumbelClip (shown in blue) against ACER (shown in green), an off-policy actor-critic algorithm, and the onpolicy algorithm A2C (shown in red). The graphs show the average performance over 3 seeds with 1 standard deviation shown as the shaded region. GumbelClip matches or exceeds the performance of the on-policy algorithm on all environments shown; while in all cases achieving improved convergence speed. It also shows a respectable performance when compared to the ACER algorithm on many of the environments. actor-critic methods. We note that the performance can be expected to improve with a larger replay memory, as seen with DQN and other methods using replay memory. Additionally, we focused the examination of our ablations on the Alien environment to reduce computational requirements. GumbelClip is based on a modified version of the open-source A2C implementation by. The present work was compared with an off-policy actor-critic algorithm, ACER , and an on-policy actor-critic algorithm, A2C, the synchronous version of A3C. Both baseline models, A2C and ACER, used the baselines package provided by OpenAI . ACER used a replay ratio of 4, trust region updates (a), a discount factor of γ = 0.99, entropy regularization of 0.01, updated every 20 steps, and a 50000 capacity replay memory per thread. To test the proposed methodology, the performance of GumbelClip on a subset of Atari environments was examined. In particular, the following environments were investigated: Alien, BeamRider, Boxing, FishingDerby, MsPacman, Qbert, Seaquest, and SpaceInvaders. We report the average reward every 1000 episodes over 40 million frames. As mentioned previously, environments were chosen where either the on-policy algorithms perform well or where there is a clear difference in performance between an off-policy and on-policy method. From Figure 3, we see the performance of GumbelClip, shown in blue, in comparison to the onpolicy ACER algorithm, shown in green, and the off-policy A2C algorithm. We see that the use of replay memory and learning with off-policy samples significantly improves the sample efficiency of GumbelClip over A2C. Across the board, we see that GumbelClip converges significantly faster than A2C while also exceeding A2C's performance. We achieve similar sample efficiency between the off-policy actor-critics, GumbelClip and ACER, across each environment. The additive noise and aggressive clamping seem to have both positive and negative effects. We see that GumbelClip sees a faster initial increase in performance across almost all environments, even outperforming ACER in some cases. However, clamping has been noted by to have the possibility of introducing bias. We hypothesize this might be the cause of poor performance on the Qbert environment, which while better than A2C, is much lower when compared to ACER. The benefit of GumbelClip comes from its simplicity, requiring a small number of easy changes to the A2C algorithm, while still providing better performance than other on-policy actor-critic algorithms. Figure 4: Variations in distributions used to sample additive noise. We compare the impact of noise drawn from the standard Gumbel distribution to the standard Normal distribution and Uniform distribution between. Here we examine the distribution used to draw the additive noise (i) for use in the forced policy F(a|s). We compare the performance of noise sampled from the standard Gumbel distribution to that of the standard Normal distribution and the Uniform distribution. The experiments only adjust the sampling distribution with no other parameter changes. We investigate the performance on the Alien Atari game over 40 million frames. From the in Figure 4, we see that the Gumbel and Normal distributions have the same initial rate of improvement but diverge roughly midway through training. The Normal distribution appears to degrade in performance before becoming stable at~1500. The uniform distribution, sampled between, has the same convergence characteristic as the other two distributions but instead of diverging away, similar to the Normal distribution, it continues upward before converging at~2000. From Figure 4, we see that the Gumbel distribution is the most performant. Figure 5: Stability of GumbelClip with extended training time. GumbelClip is trained for 150M frames,~4x longer, on the Boxing and FishingDerby environments. We see that GumbelClip experiences little to no oscillations in performance even with continued weight updates. It is natural to inquire on the stability of this method, as we rely on additive noise which could cause instability after an optimal policy has been reached. To this end, we evaluate the stability of GumbelClip by increasing the number of training iterations such that 150 million frames are seen. The Boxing and FishingDerby environments are used for this evaluation. The environments were chosen as the policy had achieved the highest score during training, a known ceiling, and any instability would cause a divergence to a sub-optimal policy. From the rather uninteresting graphs, shown in Figure 5, we see that GumbelClip can converge to and maintain a stable policy even with continued parameter updates. To overcome the noise added by the Gumbel distribution requires the network to output a near one-hot-encoded categorical distribution. In our final set of experiments, we performed ablations over the various components of GumbelClip to tease apart the cause of improvements in performance. The of the ablation are shown in Figure 6 (a) with the complete GumbelClip algorithm in blue and each stripped version of GumbelClip as a red curve. We start with a base version, which is the application of truncated importance sampling and replay memory to the off-policy algorithm A2C; this base version is shown in the leftmost panel in Figure 6 (a). From Figure 6: Ablations of GumbelClip. We gradually introduce each component of GumbelClip to a base version of an off-policy actor-critic A2C algorithm. a) The full GumbelClip algorithm is shown as the blue curve while the stripped versions are shown in red. From left to right we gradually add the components onto the base version until we arrive at the full GumbelClip algorithm, shown in the last pane. The lines in the last pane are identical with one graph is stylized. b) The table provides the percent deltas between either the stripped version to GumbelClip or the current model to the last. We measure the change between the last 100 episodes. lower than GumbelClip. The simple addition of a larger batchsize, from 16 sampled trajectories, to 64 as shown in the second panel in Figure 6 (a) causes an increase of +21.30% over the base version and narrows the difference to GumbelClip to −26.49%. Using an aggressive clamp of 4 instead of 10 on the importance sampling ratioρ improves performance by an additional +12.33%. Finally, the addition of noise sampled from the Gumbel distribution closes the gap with a final increase of +21.09%. The addition of noise changes Equation 8 from being in terms of π(a|s), in both the policy being optimized and the numerator inρ, to F(a|s). It is clearly shown that the large batchsize and additive noise contribute the most to the performance increase between stripped versions. Additionally, while difficult to quantify, we can see from the plots that the aggressive clamp and additive noise improve sample efficiency the most. Figure 7: Change in distributions overρ as components are added, shown as histograms. The xaxis corresponds to the magnitude ofρ and the y-axis is the number of occurrences. From left to right we see the distribution overρ when using the base version. As components are added we see three modes, at approximately 0, 1, and c, become more pronounced. The addition of Gumbel noise increases the smoothness between 0 → 1, creates a fatter tail from the mode 1 → c, and increases the density of all modes. From Figure 7, we see the effect that each addition to the base version has on the ratioρ and therefore the updates to the network. In the last two panes of Figure 7, we see three clear modes at 0, 1, and c. Addition of Gumbel noise increases the density across all modes and reduces the "noise" seen from 0 → 1. The modes, as discussed in Section 3, correspond to "agreement", at 1, and "disagreement" at {0, c}. As GumbelClip relies heavily on the Gumbel distribution to sample noise from, it is best suited to environments with discrete actions. Therefore, an interesting avenue for future work would be to find a suitable distribution to sample noise from that works with continuous actions. Additionally, further investigation into possible annealing schedules for a temperature hyperparameter, in Equation 6, or around the clamping constant c could yield interesting . In this paper we have presented GumbelClip, a set of adjustments to the on-policy A2C algorithm, enabling full off-policy learning from stored trajectories in a replay memory. Our approach relies on aggressive clipping of the importance weight, large batchsize, and additive noise sampled from the Gumbel distribution. We have empirically validated the use of each component in GumbelClip through ablations and shown the stability of the algorithm. Furthermore, we have shown that GumbelClip achieves superior performance and higher sample efficiency than A2C. GumbelClip nears the performance and sample efficiency of ACER on many of the tested environments. Our methodology requires minimal changes to the A2C algorithm, which in contrast to ACER, makes the implementation of GumbelClip straightforward.
With a set of modifications, under 10 LOC, to A2C you get an off-policy actor-critic that outperforms A2C and performs similarly to ACER. The modifications are large batchsizes, aggressive clamping, and policy "forcing" with gumbel noise.