Revisiting lp-constrained Softmax Loss: A Comprehensive Study
Normalization is a vital process for any machine learning task as it controls
the properties of data and affects model performance at large. The impact of
particular forms of normalization, however, has so far been investigated in
limited domain-specific classification tasks and not in a general fashion.
Motivated by the lack of such a comprehensive study, in this paper we
investigate the performance of lp-constrained softmax loss classifiers across
different norm orders, magnitudes, and data dimensions in both proof-of-concept
classification problems and real-world popular image classification tasks.
Experimental results suggest collectively that lp-constrained softmax loss
classifiers not only can achieve more accurate classification results but, at
the same time, appear to be less prone to overfitting. The core findings hold
across the three popular deep learning architectures tested and eight datasets
examined, and suggest that lp normalization is a recommended data
representation practice for image classification in terms of performance and
convergence, and against overfitting.