Learning Architecture: We use a variance scaling initializer with truncated normal distribution [1]} with \(\textrm {scale}=2.0\)  to initialize the parameters of our policy and value function NNs that has been shown to work well with ReLU activations. The input to our NN comprises of stacked observations for the past few times steps (marked as frames in Table REF ), which is passed through three conv1d convolution layers with varying number of filters (marked as net arch in Table) and fixed kernel_size = 3, strides = (2, 1, 1), padding = (same, valid, valid) for each conv layer, followed by two MLP layers with (256, 128) units and ReLU activations (see Table REF  for more details). We use gradient clipping to restrict the norm of the gradient to be less than \(0.5\)  and normalize the advantage by subtracting the mean and dividing by the standard deviation once before training - we find that advantage normalisation by minibatch yields worse performance. We use a discount factor of \(\gamma =0.99\) , a learning rate of \(10^{-4}\) , and clipping parameter \(\epsilon =0.2\) .
