VMPO agent: As described in [1]} and [2]}, this agent centered on a gated transformer XL network. Image-frame observations were passed through a residual-network encoder and fully connected layer, with proprioceptive observations, previous action and reward then concatenated. In the symbolic task, observations were passed directly to the transformer core. Losses were included for policy and value heads, pixel control [3]}, kickstarting [4]}, [5]} and latent state prediction (see Section REF ) weighted by \(\beta _{\text{loss type}}\) . Where kickstarting was used (see below), the loss was \(KL(\pi _{\text{student}} \Vert  \pi _{\text{teacher}})\) 
and the weighting was set to 0 after 5e8 steps. Further details are presented in Appendix Table REF .
