The student policy architecture was very similar to [1]}: Agents' observations of episode time, their own locations and velocities, the location and velocities of other agents, and the location and velocities of boxes and ramps were each embedded in a 256-dimensional space. Objects of the same type (i.e. other agents, boxes or ramps) were embedded using the same weights with entity-centric observations [2]}, [3]}. The LIDAR-like observation (which measured the distance to the nearest objects using 30 evenly arrayed radial range sensors) was passed through a 1d-convolutional layer with 9 filters and kernel size 3 before being embedded together with agents' observations of their own locations and velocities. We then pooled over entities using mean-pooling. In the Hide and Seek environment, invisible agents or objects where masked out before mean-pooling. Further processing after the pooling layer consisted of a fully-connected layer, an LSTM (each of size 256), a layer norm and a factorial action head. Each value function consisted of a separate network whose architecture is identical to the policy network. Masks where not used for value functions (i.e. the value function observed both visible and invisible objects).
