Regularization.
Regularization refers to any technique that reduces overfitting in machine learning; see [1]}, [2]} and references therein, in particular for a discussion of implicit regularization, a topic that has received attention recently in the context of stochastic gradient optimization applied to neural network models.
Traditional regularization techniques such as ridge regression, weight decay and dropout do not make use of the training data to reduce the model capacity.
A powerful class of techniques is data augmentation, which constructs additional examples from the training set, e.g., by applying geometric transformations to the original data [3]}. A recently proposed technique is mixup [4]}, where the examples are created by taking convex combinations of pairs of inputs and their labels.
[5]} extends mixup to hidden representations in deep neural networks. Subsequent works by [6]}, [7]}, [8]}, [9]}, [10]}, [11]} introduce different variants and extensions of mixup. Regularization is also intimately connected to robustness [12]}, [13]}, [14]}, [15]}, [16]}. Adding to the list is NFM, a powerful regularization method that we propose to improve model robustness.
