One popular technique harnessed in previous work for boosting model performance is regularization, which has been widely studied in the deep learning domain. Several common regularization approaches are parameter norm penalty, early stopping, data augmentation and the addition of noise during training time. An example of this is mixup, a data augmentation method that adds convex combinations of pairs of training instances and their labels [1]}. By training neural networks on such combinations of inputs, mixup improves the generalization errors of cutting-edge network architectures, alleviates the memorization of corrupt labels, and increases the sensitivity to adversarial examples. Likewise, the cutout approach in [2]} randomly masks out contiguous sections of the input images to CNNs, thereby improving robustness and yielding higher accuracy level [2]}. Alternatively, the technique dropout drops, or temporarily removes, random units in a neural network at intermediate layers [4]}. However, previous works have emphasized the use of regularization for preventing overfitting and guaranteeing that the resulting models generalize well to unseen data. They seemed to overlook the computational complexity of such regularized models in the inference process. Particularly, few papers have tested whether regularization speeds up the calculations of CNNs or introduces computational overhead during inference time.
