L1 and L2 regularizers are critical tools in machine learning due to their ability to simplify solutions. However, imposing strong L1 or L2 regularization with gradient descent method easily fails, and this limits the generalization ability of the underlying neural networks. To understand this phenomenon, we investigate how and why training fails for strong regularization. Specifically, we examine how gradients change over time for different regularization strengths and provide an analysis why the gradients diminish so fast. We find that there exists a tolerance level of regularization strength, where the learning completely fails if the regularization strength goes beyond it. We propose a simple but novel method, Delayed Strong Regularization, in order to moderate the tolerance level. Experiment results show that our proposed approach indeed achieves strong regularization for both L1 and L2 regularizers and improves both accuracy and sparsity on public data sets. Our source code is published. Regularization has been very common for machine learning to prevent over-fitting and to obtain sparse solutions. Deep neural networks (DNNs), which have shown huge success in many tasks such as computer vision BID9 BID15 BID5 and speech recognition , often contain a number of parameters in multiple layers with non-linear activation functions, in order to gain enough expressive power. However, DNNs with many parameters are often prone to over-fitting, so the need for regularization has been emphasized. While new regularization techniques such as dropout BID16 and pruning BID2 have been proposed to solve the problem, the traditional regularization techniques using L1 or L2 norms have cooperated with them to further improve the performance significantly. L1 regularization, often called Lasso BID17 , obtains sparse solutions so that the required memory and power consumption are reduced while keeping reasonable accuracy. On the other hand, L2 regularization smooths the parameter distribution and reduces the magnitude of parameters, so the resulting solution is simple (i.e., less prone to over-fitting) and effective. Indeed, our empirical results show that applying strong L2 regularization to the deep neural networks that already has dropout layers can reduce the error rate by up to 24% on a public data set.Strong regularization is especially desired when the model contains too many parameters for the given amount of training data. This is often the case for deep learning tasks in practice because DNNs often contain millions of parameters while labeled training data set is limited and expensive. However, imposing strong L1 or L2 regularization on DNNs is difficult for gradient descent method due to the vanishing gradient problem. If we impose too strong regularization, the gradient from regularization becomes dominant, and DNNs stop learning. In this paper, we first study the interesting phenomenon that strong regularization fails in learning. We also provide an analysis why the gradients diminish so quickly that learning completely fails. Then, we propose a simple yet effective solution, Delayed Strong Regularization, which carries a time-dependent schedule of regularization strength. We find that we can overcome the failure in learning by waiting for the model to reach an "active learning" phase, where the gradients' magnitudes are significant, and then enforcing strong regularization. Delayed Strong Regularization enables us to obtain the superior performance that is otherwise hidden by learning failure in deep networks. The proposed approach is general and does not require any additional computation. The experiment results indicate that the proposed approach indeed achieves strong regularization, consistently yielding even higher accuracy and higher compression rate that could not be achieved. In this work, we studied the problem of achieving strong regularization for deep neural networks. Strong regularization with gradient descent algorithm easily fails for deep neural networks, but few work addressed this phenomenon in detail. We provided investigation and analysis of the phenomenon, and we found that there is a strict tolerance level of regularization strength. To avoid this problem, we proposed a novel but simple method: Delayed Strong Regularization. We performed experiments with fine tuning of regularization strength. Evaluation results show that (1) our model successfully achieves strong regularization on deep neural networks, verifying our hypothesis that the model will keep learning once it reaches an "active learning" phase, (2) with strong regularization, our model obtains higher accuracy and sparsity, (3) the number of hidden layers in neural networks affects the tolerance level, and (4) L1/L2 regularization is difficult to tune, but it can yield great performance boost when tuned well.There are limitations in this work. Our proposed method can be especially useful when strong regularization is desired. For example, deep learning projects that cannot afford a huge labeled data set can benefit from our method. However, strong regularization may not be necessary in some other cases where the large labeled data set is available or the networks do not contain many parameters. In addition, our experiments were not performed on a bigger data set such as ImageNet data set. We need to fine-tune the models with different regularization parameters, and we also need multiple training sessions of each model to obtain confidence interval. For example, the experiment results in FIG1 and 4 include 750 training sessions in total. This is something we cannot afford with ImageNet data set, which requires several weeks of training for EACH session (unless we have GPU clusters). Our approach cannot be applied to architectures containing normalization techniques for the reason in Section 2.2. We actually tried to intentionally exclude normalization part from Residual Networks BID5 ) and train the model to see if we can apply our method to non-normalized Residual Networks. However, we could not control the exploding gradients caused by the exclusion of normalization.Our work can be further extended in several ways. Since our model can achieve strong regularization, it will be interesting to see how the strongly regularized model performs if combined with pruning-related methods BID2 . We applied our approach to only L1 and L2 regularizers, but applying it to other regularizers such as group sparsity regularizers will be promising as they are often employed for DNNs to compress networks. Lastly, our proposed Delayed Strong Regularization is very simple, so one can easily extend it to more complicated methods. All these directions are left as our future work.
We investigate how and why strong L1/L2 regularization fails and propose a method than can achieve strong regularization.
{"src": "scitldr-AIC_train", "paper_id": "Bys_NzbC-"}