Consider a supervised learning problem with training data \(\mathcal {D}:=\lbrace x_i, y_i\rbrace _{i=1}^N\) , where we model the conditional probability \(p_{\theta }(y_i\,|\,x_i)\)  using a NN parameterized by \(\theta \) . Applying dropout to a NN often means element-wisely reweighing each layer with a data-specific Bernoulli/Gaussian distributed random mask \(z_i\) , which are \(iid\)  drawn from a prior \(p_{{\eta }}(z)\)  parameterized by \({\eta }\)  [1]}, [2]}. This implies dropout training can be viewed as approximate Bayesian inference [3]}. More specifically, one may view the learning objective of a supervised learning model with dropout as a log-marginal-likelihood: \(\log \int \prod _{i=1}^N p(y_i\,|\,x_i, z)p(z)d z\) . To maximize this often intractable log-marginal, it is common to resort to variational inference [4]}, [5]} that introduces a variational distribution \(q({z})\)  on the random mask \(z\)  and optimizes an evidence lower bound (ELBO):
\(\leavevmode \xbox {resizebox}{\XMLaddatt {width}{0.0pt}\mathcal {L}(\mathcal {D})=\textstyle \mathbb {E}_{q(z)}\left[ \log \frac{\prod _{i=1}^N p_{\theta }(y_i\,|\,x_i, z)p_{{\eta }}(z)}{q(z)}\right] = \big (\sum _{i=1}^N\mathbb {E}_{z_i\sim q(z)}\left[ \log p_{\theta }(y_i\,|\,x_i, z_i)\right]\big )-\mbox{KL}(q(z)||p_\eta (z)),}\) 
