Recently, a parametrized class of loss functions called @math -loss, @math , has been introduced for classification. This family, which includes the log-loss and the 0-1 loss as special cases, comes with compelling properties including an equivalent margin-based form which is classification-calibrated for all @math . We introduce a generalization of this family to the entire range of @math and establish how the parameter @math enables the practitioner to choose among a host of operating conditions that are important in modern machine learning tasks. We prove that smaller @math values are more conducive to faster optimization; in fact, @math -loss is convex for @math and quasi-convex for @math . Moreover, we establish bounds to quantify the degradation of the local-quasi-convexity of the optimization landscape as @math increases; we show that this directly translates to a computational slow down. On the other hand, our theoretical results also suggest that larger @math values lead to better generalization performance. This is a consequence of the ability of the @math -loss to limit the effect of less likely data as @math increases from 1, thereby facilitating robustness to outliers and noise in the training data. We provide strong evidence supporting this assertion with several experiments on benchmark datasets that establish the efficacy of @math -loss for @math in robustness to errors in the training data. Of equal interest is the fact that, for @math , our experiments show that the decreased robustness seems to counteract class imbalances in training data.
While the previous result establishes an equivalent margin-based form for @math -loss, our next result establishes some of its basic optimization characteristics. Finally, we conclude this section with another basic property of @math -loss that highlights its suitability for classification. For binary classification and margin-based losses, Bartlett in @cite_18 introduce as a means to compare the performance of a loss function relative to the 0-1 loss. A margin-based loss function @math is classification-calibrated if its minimum conditional risk given @math is attained by a @math such that @math , where @math is the true posterior. Building upon such a result in @cite_9 for @math -loss, the following proposition shows that @math is classification-calibrated for all @math . The proof of Proposition is given in Appendix .
@cite_9 @cite_18
2912269121 1982032418
We present @math -loss, @math , a tunable loss function for binary classification that bridges log-loss ( @math ) and @math - @math loss ( @math ). We prove that @math -loss has an equivalent margin-based form and is classification-calibrated, two desirable properties for a good surrogate loss function for the ideal yet intractable @math - @math loss. For logistic regression-based classification, we provide an upper bound on the difference between the empirical and expected risk for @math -loss by exploiting its Lipschitzianity along with recent results on the landscape features of empirical risk functions. Finally, we show that @math -loss with @math performs better than log-loss on MNIST for logistic regression. Problems of data classification can be studied in the framework of regularization theory as ill-posed problems. In this framework, loss functions play an important role in the application of regularization theory to classification. In this paper, we review some important convex loss functions, including hinge loss, square loss, modified square loss, exponential loss, logistic regression loss, as well as some non-convex loss functions, such as sigmoid loss, @math -loss, ramp loss, normalized sigmoid loss, and the loss function of 2 layer neural network. Based on the analysis of these loss functions, we propose a new differentiable non-convex loss function, called smoothed 0-1 loss function, which is a natural approximation of the 0-1 loss function. To compare the performance of different loss functions, we propose two binary classification algorithms for binary classification, one for convex loss functions, the other for non-convex loss functions. A set of experiments are launched on several binary data sets from the UCI repository. The results show that the proposed smoothed 0-1 loss function is robust, especially for those noisy data sets with many outliers.