In regular dropout, as shown in (REF ), while we make the dropout masks data specific during optimization, we keep their distributions the same.
This implies that while the NN weights can vary from data to data, their distribution is kept data invariant. In this paper, we propose contextual dropout, in which the distributions of dropout masks \(z_i\)  depend on covariates \(x_i\)  for each sample \((x_i,y_i)\) . Specifically, we define the variational distribution as \(q_{}(z_i\,|\,x_i)\) , where \(\)  denotes its NN parameters. In the framework of amortized variational Bayes [1]}, [2]}, we can view \(q_{}\)  as an inference network (encoder) trying to approximate the posterior \(p(z_i\,|\,y_i, x_i)\propto p(y_i\,|\,x_i,z_i)p(z_i)\) . Note as we have no access to \(y_i\)  during testing, we parameterize our encoder in a way that it depends on \(x_i\)  but not \(y_i\) .
From the optimization point of view, what we propose corresponds to the ELBO of \(\log \prod _{i=1}^N \int p(y_i\,|\,x_i,z_i) p(z_i) dz_i \)  given \(q_{}(z_i \,|\,x_i)\)  as the encoder, which can be expressed as
\(\small \textstyle \mathcal {L}(\mathcal {D}) = \sum _{i=1}^N\mathcal {L}(x_i,y_i),~\mathcal {L}(x_i,y_i) =\mathbb {E}_{z_i\sim q_{}(\cdot \,|\,x_i)}[\log p_{\theta }(y_i\,|\,x_i,z_i)] -\text{KL}(q_{}(z_i\,|\,x_i) || p_{{\eta }}(z_i)). \) 
