This paper presents GumbelClip, a set of modifications to the actor-critic algorithm, for off-policy reinforcement learning. GumbelClip uses the concepts of truncated importance sampling along with additive noise to produce a loss function enabling the use of off-policy samples. The modified algorithm achieves an increase in convergence speed and sample efficiency compared to on-policy algorithms and is competitive with existing off-policy policy gradient methods while being significantly simpler to implement. The effectiveness of GumbelClip is demonstrated against existing on-policy and off-policy actor-critic algorithms on a subset of the Atari domain. Recent advances in reinforcement learning (RL) have enabled the extension of long-standing methods to complex and large-scale tasks such as Atari (Mnih et al., 2015) , Go , and DOTA (OpenAI, 2018) . The key driver has been the use of deep neural networks, a non-linear function approximator, with the combination usually referred to as Deep Reinforcement Learning (DRL) (LeCun et al., 2015; Mnih et al., 2015) . However, deep learning-based methods are usually data-hungry, requiring millions of samples before the network converges to a stable solution. As such, DRL methods are usually trained in a simulated environment where an arbitrary amount of data can be generated. RL algorithms can be classified as either learning in an off-policy or on-policy setting. In the onpolicy setting, an agent learns directly from experience generated by its current policy. In contrast, the off-policy setting enables the agent to learn from experience generated by its current policy or/and other separate policies. An algorithm that learns in the off-policy setting has much greater sample efficiency as old experience from the current policy can be reused; it also enables off-policy algorithms to learn an optimal policy while executing an exploration-focused policy (Sutton et al., 1998) . The most famous off-policy method is Q-Learning (Watkins & Dayan, 1992) which learns an actionvalue function, Q(s, a), that maps the value to a state s and action a pair. Deep Q-Learning (DQN), the marriage of Q-Learning with deep neural networks, was popularised by Mnih et al. (2015) and used various modifications, such as experience replay, for stable convergence. Within DQN, experience replay (Lin, 1992) is often motivated as a technique for reducing sample correlation. Unfortunately, all action-value methods, including Q-Learning, have two significant disadvantages. First, they learn deterministic policies, which cannot handle problems that require stochastic policies. Second, finding the greedy action with respect to the Q function is costly for large action spaces. To overcome these limitations, one could use policy gradient algorithms (Sutton et al., 2000) , such as actor-critic methods, which learn in an on-policy setting at the cost of sample efficiency. The ideal solution would be to combine the sample efficiency of off-policy algorithms with the desirable attributes of on-policy algorithms. Work along this line has been done by using importance sampling (Degris et al., 2012) or by combining several techniques together, as in ACER (Wang et al., 2016) . However, the resulting methods are quite complex and require many modifications to existing algorithms. This paper, proposes a set of adjustments to A2C , a parallel on-policy actor-critic algorithm, enabling off-policy learning from stored trajectories. Therefore, our contributions are as follows: • GumbelClip, a fully off-policy actor-critic algorithm, the result of a small set of simple adjustments, in under 10 lines of code (LOC) 1 , to the A2C algorithm. • GumbelClip has increased sample efficiency and overall performance over on-policy actorcritic algorithms, such as A2C. • GumbelClip performs similarily to other off-policy actor-critic algorithms, such as ACER, while being significantly simpler to implement. The paper is organized as follows: Section 2 covers background information, Section 3 describes the GumbelClip algorithm, Section 4 details the experiments along with results, discussion, and ablations of our methodology. Section 5 discusses possible future work, and finally Section 6 provides concluding remarks. In this paper we have presented GumbelClip, a set of adjustments to the on-policy A2C algorithm, enabling full off-policy learning from stored trajectories in a replay memory. Our approach relies on aggressive clipping of the importance weight, large batchsize, and additive noise sampled from the Gumbel distribution. We have empirically validated the use of each component in GumbelClip through ablations and shown the stability of the algorithm. Furthermore, we have shown that GumbelClip achieves superior performance and higher sample efficiency than A2C. GumbelClip nears the performance and sample efficiency of ACER on many of the tested environments. Our methodology requires minimal changes to the A2C algorithm, which in contrast to ACER, makes the implementation of GumbelClip straightforward.
With a set of modifications, under 10 LOC, to A2C you get an off-policy actor-critic that outperforms A2C and performs similarly to ACER. The modifications are large batchsizes, aggressive clamping, and policy "forcing" with gumbel noise.
{"src": "scitldr-AIC_train", "paper_id": "SJxngREtDB"}