For A3C and Rainbow we use the Atari 2600 domain [1]} and for IMPALA DeepMind Lab [2]}. In both cases, we treat the list of games/levels as a single learning problem by averaging across games in Atari and training the agent on all level in parallel in case of DeepMind Lab. In order to reduce the noise in the MC-hardness metric, caused by agents being unable to learn the task and behaving randomly, we filter out the levels in which the original trained agent performs poorly.
We apply the retraining procedures \(\mathbf {T_{1}}\) , \(\mathbf {T_{2}}\)  on all the models, and on A3C we apply additionally \(\mathbf {T_{3}}\)  and \(\mathbf {T_{4}}\) . All the hyperparameters are kept the same during the training and retraining procedures.
