MLP: We consider an MLP with two hidden layers of size 300 and 100, respectively, and use ReLU activations. Dropout is applied to all three full-connected layers. We use MNIST as the benchmark. All models are trained for 200 epochs with batch size 128 and the Adam optimizer [1]} (\(\beta _1 = 0.9\) , \(\beta _2 = 0.999\) ). The learning rate is \(0.001\) . We compare contextual dropout with MC dropout [2]} and concrete dropout [3]}. For MC dropout, we use the hand-tuned dropout rate at \(0.2\) . For concrete dropout, we initialize the dropout rate at \(0.2\)  for Bernoulli dropout and the standard deviation parameter at 0.5 for Gaussian dropout. and set the Concrete temperature at \(0.1\)  [3]}. We initialize the weights in contextual dropout with He-initialization preserving the magnitude of the variance of the weights in the forward pass [5]}. We initialize the biases in the way that the dropout rate is \(0.2\)  when the weights for contextual dropout are zeros. We also initialize our prior dropout rate at \(0.2\) . For hyperparameter tuning, we
hold out
\(10,000\)  samples randomly selected from the training set for validation. We use the chosen hyperparameters to train on the full training set (\(60,000\)  samples) and evaluate on the testing set (\(10,000\)  samples).
We use Leaky ReLU [6]} with \(0.1\)  as the non-linear operator in contextual dropout. The reduction ratio \(\gamma \)  is set as 10, and sigmoid scaling factor \(t\)  as \(0.01\) . For Bayes by Backprop, we use \(-\log \sigma _1=0, -\log \sigma _2=6, \pi =0.2\)  (following the notation in the original paper). For evaluation, we set \(M=20\) .
