Figure REF  shows a high-level representation of our proposed network. We use Resnet-50 [1]} to extract useful features from the stack of input images. Instead of adding or concatenating the speed and planner inputs we use a learnable soft-attention mechanism [2]} to attend to the extracted image features using the speed and planner inputs as represented by the input attention block in Figure REF . A learnable soft-attention mechanism allows for the low dimensional speed and planner inputs to have a large impact on the relatively higher-dimensional image feature vector. This attention mechanism is represented by a neural network layer and is jointly learned with the main network. The speed and planner inputs separately attend to the extracted feature vector and a union of the two attended features is taken to achieve a jointly attended vector. This attended observation feature is used to predict the visual affordances and action primitives. Lastly, in the Auxiliary Task Attention Block, a similar aforementioned double-attention mechanism is used to attend over the observation feature vector using the visual affordances and action primitives, which is then used to predict the final control predictions. The dashed-line vectors in Figure REF  represent the ground-truth positions from where the gradients are jointly back-propagated.
