Dropout is a common regularization technique utilized for fully-connected layers. It is implemented by dropping, or temporarily removing, random units in a neural network [1]}. This is comparable to adding noise to a model's hidden units. While dropout removes parameters at intermediate layers, the technique cutout randomly occludes sections of the input images [2]}. By doing so, the model is able to focus more on less prominent features of the image rather than important features, i.e. those with high activations in intermediate layers of the network. This practice improves robustness and yields higher accuracy levels. Another variation of regularization, mixup, applies data augmentation [3]}. It enlarges the training dataset by taking the linear interpolations of two examples and their labels from the training data. By including these constructed examples in training, the neural network becomes more robust against adversarial examples and lessens the memorization of corrupt labels. Overall, mixup improves the generalization errors of these models on popular datasets.
