IMPALA agent: This agent is described by [1]}, and used population-based training as presented by [2]}. Pixel observations passed through a residual-network encoder and fully connected layer, and proprioceptive observations were concatenated to the input at every spatial location before each residual-network block. The resulting output was fed into the core of the network, an LSTM.Recurrent state was set to zero at episode boundaries, but not between trials, enabling the agents (in principle) to utilise knowledge of the chemistry accumulated over previous trials. Pixel control and kickstarting losses were used, as in the VMPO agent. See Appendix Table REF  for details.
