We only conduct training on the training set (no data augmentation with visual genome dataset), and evaluation on the validation set. For MC dropout, we use the dropout rate of \(0.1\)  for Bernoulli dropout as in [1]} and the standard deviation parameter of \(1/3\)  for Gaussian dropout. For concrete dropout, we initialize the dropout rate at \(0.1\)  and set the Concrete temperature at \(0.1\)  [2]}. For hyperparameter tuning, we randomly hold out \(20\%\)  of the training set for validation. After tuning, we train on the whole training set and evaluate on the validation set. We initialize the weights with He-initialization preserving the magnitude of the variance of the weights in the forward pass [3]}. We initialize the biases in the way that the dropout rate is \(0.1\)  when the weights for contextual dropout are zeros. We also initialize our prior dropout rate at \(0.1\) . We use ReLU as the non-linear operator in contextual dropout. We use
\(\gamma =8\)  for layers with \(C_d^l > 8\) , otherwise \(\gamma = 1\) . We set \(\alpha \in \mathbb {R}^{d_V}\)  for residual layers.
