Choice of hyper-parameters in Contextual Dropout: Contextual dropout introduces two additional hyperparameters compared to regular dropout. One is the channel factor \(\gamma \)  for the encoder network. In our experiments, the results are not sensitive to the choice of the value of the channel factor \(\gamma \) . Any number from 8 to 16 would
give similar results, which is also observed in [1]}. The other is the sigmoid scaling factor \(t\)  that controls the learning rate of the encoder. We find that the performance is not that sensitive to its value and it is often beneficial to make it smaller than the learning rate of the decoder. In all experiments considered in the paper, which cover various noise levels and model sizes, we have simply fixed it at \(t=0.01\) .
