Recently, a parametrized class of loss functions called @math -loss, @math , has been introduced for classification. This family, which includes the log-loss and the 0-1 loss as special cases, comes with compelling properties including an equivalent margin-based form which is classification-calibrated for all @math . We introduce a generalization of this family to the entire range of @math and establish how the parameter @math enables the practitioner to choose among a host of operating conditions that are important in modern machine learning tasks. We prove that smaller @math values are more conducive to faster optimization; in fact, @math -loss is convex for @math and quasi-convex for @math . Moreover, we establish bounds to quantify the degradation of the local-quasi-convexity of the optimization landscape as @math increases; we show that this directly translates to a computational slow down. On the other hand, our theoretical results also suggest that larger @math values lead to better generalization performance. This is a consequence of the ability of the @math -loss to limit the effect of less likely data as @math increases from 1, thereby facilitating robustness to outliers and noise in the training data. We provide strong evidence supporting this assertion with several experiments on benchmark datasets that establish the efficacy of @math -loss for @math in robustness to errors in the training data. Of equal interest is the fact that, for @math , our experiments show that the decreased robustness seems to counteract class imbalances in training data.
For binary classification, where @math , it is common to use classification functions of the form @math such that the classifier, for any given @math , outputs the hypothesis (hard decision) @math @cite_18 @cite_0 @cite_10 @cite_2 @cite_9 . A classification function corresponds to the certainty of an algorithm's prediction (e.g., SVM). Examples of loss functions that act on classification functions include logistic loss, hinge loss, and square loss.
@cite_18 @cite_9 @cite_0 @cite_2 @cite_10
2151532749 2912269121 1964840769 2101557761
Problems of data classification can be studied in the framework of regularization theory as ill-posed problems. In this framework, loss functions play an important role in the application of regularization theory to classification. In this paper, we review some important convex loss functions, including hinge loss, square loss, modified square loss, exponential loss, logistic regression loss, as well as some non-convex loss functions, such as sigmoid loss, @math -loss, ramp loss, normalized sigmoid loss, and the loss function of 2 layer neural network. Based on the analysis of these loss functions, we propose a new differentiable non-convex loss function, called smoothed 0-1 loss function, which is a natural approximation of the 0-1 loss function. To compare the performance of different loss functions, we propose two binary classification algorithms for binary classification, one for convex loss functions, the other for non-convex loss functions. A set of experiments are launched on several binary data sets from the UCI repository. The results show that the proposed smoothed 0-1 loss function is robust, especially for those noisy data sets with many outliers. We present @math -loss, @math , a tunable loss function for binary classification that bridges log-loss ( @math ) and @math - @math loss ( @math ). We prove that @math -loss has an equivalent margin-based form and is classification-calibrated, two desirable properties for a good surrogate loss function for the ideal yet intractable @math - @math loss. For logistic regression-based classification, we provide an upper bound on the difference between the empirical and expected risk for @math -loss by exploiting its Lipschitzianity along with recent results on the landscape features of empirical risk functions. Finally, we show that @math -loss with @math performs better than log-loss on MNIST for logistic regression. The goal of binary classification is to estimate a discriminant function y from observations of covariate vectors and corresponding binary labels. We consider an elaboration of this problem in which the covariates are not available directly but are transformed by a dimensionality-reducing quantizer Q. We present conditions on loss functions such that empirical risk minimization yields Bayes consistency when both the discriminant function and the quantizer are estimated. These conditions are stated in terms of a general correspondence between loss functions and a class of functionals known as Ali-Silvey or -divergence functionals. Whereas this correspondence was established by Blackwell [Proc. 2nd Berkeley Symp. Probab. Statist. 1 (1951) 93-102. Univ. California Press, Berkeley] for the 0-1 loss, we extend the correspondence to the broader class of surrogate loss functions that play a key role in the general theory of Bayes consistency for binary classification. Our result makes it possible to pick out the (strict) subset of surrogate loss functions that yield Bayes consistency for joint estimation of the discriminant function and the quantizer. We study losses for binary classification and class probability estimation and extend the understanding of them from margin losses to general composite losses which are the composition of a proper loss with a link function. We characterise when margin losses can be proper composite losses, explicitly show how to determine a symmetric loss in full from half of one of its partial losses, introduce an intrinsic parametrisation of composite binary losses and give a complete characterisation of the relationship between proper losses and "classification calibrated" losses. We also consider the question of the "best" surrogate binary loss. We introduce a precise notion of "best" and show there exist situations where two convex surrogate losses are incommensurable. We provide a complete explicit characterisation of the convexity of composite binary losses in terms of the link function and the weight function associated with the proper loss which make up the composite loss. This characterisation suggests new ways of "surrogate tuning" as well as providing an explicit characterisation of when Bregman divergences on the unit interval are convex in their second argument. Finally, in an appendix we present some new algorithm-independent results on the relationship between properness, convexity and robustness to misclassification noise for binary losses and show that all convex proper losses are non-robust to misclassification noise.