For the training of the student policy we used hyperparameters similar to [1]}, except a 4 times smaller buffer size and mini-batch size (to limit the available compute). Buffer and mini-batch size for the teacher was chosen such that the time of each optimization iteration was similar for teacher and student policy. The student and teacher policy was trained on four and one A100 NVIDIA GPUs, respectively. The training time was 8 days for the Hide and Seek task and 55 hours for the maze tasks.
<TABLE>