Dropout is a regularization technique originally proposed by [1]} to alleviate the over-fitting problem of fully connected neural networks, which randomly drops neurons with fixed probability during network training. However, all neuron activations are retained while testing the model, and the final output will be scaled according to the dropout probability. In this way, the network can easily integrate several different small models to achieve good regularization. Inspired by Dropout, Dropconnect[2]}, Monte Carlo dropout[3]}, and many others were also introduced. This is only useful for the fully connected layer, but useless for the convolution layer. One of the reasons may be that the spatially adjacent pixels on the feature map are related to each other and share contextual information. After that, DeVries et al.[4]} designed Cutout to randomly drop out contiguous regions of input images, Tompson et al.[5]} also proposed SpatialDropout to randomly drops partial channels of a feature map, rather than dropping the independent random pixels. Besides, ADL[6]} was proposed to utilize the attention mechanism to erase the maximally activated part.
