Data-dependent variational distribution: [1]} model attentions as latent-alignment variables and optimize a tighter lower bound (compared to hard attention) using a learned inference network.
To balance exploration and exploitation for contextual bandits problems,
[2]} introduce local variable uncertainty under the Thompson sampling framework.
However, their inference networks of are both independent of the decoder, which may considerably increase memory and computational cost for the considered applications. [3]} propose Bayesian attention modules with efficient parameter sharing between the encoder and decoder networks. Its scope is limited to attention units as [1]},
while we demonstrate the general applicability of contextual dropout to fully connected, convolutional, and attention layers in supervised learning models.
Conditional computation [5]}, [6]}, [7]}, [8]} tries to increase model capacity without a proportional increase in computation, where an independent gating network decides turning which part of a network active and which inactive for each example. In contextual dropout, the encoder works much like a gating network choosing the distribution of sub-networks for each sample. But the potential gain in model capacity is even larger, \(e.g.,\)  there are potentially \(\sim O((2^d)^L)\)  combinations of nodes for \(L\)  fully-connected layers, where \(d\)  is the order of the number of nodes for one layer. Generalization of dropout: DropConnect [9]} randomly drops the weights rather than the activations so as to generalize dropout. The dropout distributions for the weights, however, are still the same across different samples. Contextual dropout utilizes sample-dependent dropout probabilities, allowing different samples to have different dropout probabilities.
