However, the quality of uncertainty estimation depends heavily on the dropout probabilities [1]}. To avoid grid-search over the dropout probabilities, [1]} and [3]} propose to automatically learn the dropout probabilities, which not only leads to a faster experiment cycle but also enables the model to have different dropout probabilities for each layer, bringing greater flexibility into uncertainty modeling. But, these methods still impose the restrictive assumption that dropout probabilities are global parameters shared across all data samples.
By contrast, we consider parameterizing dropout probabilities
as a function of input covariates,
treating them as data-dependent local variables.
Applying covariate-dependent dropouts
allows different data to have different distributions over the NN weights. This generalization
has the potential to greatly enhance the expressiveness of a Bayesian NN. However, learning covariate-dependent dropout rates is challenging. [4]} propose standout, where a binary belief network is laid over the original network, and develop a heuristic approximation to optimize free energy. But, as pointed out by [1]}, it is not scalable due to its need to significantly increase the model size.
