A Review on Quantification Learning <s> Quantification via Threshold Selection Policies <s> Receiver Operating Characteristics (ROC) graphs are a useful technique for organizing classifiers and visualizing their performance. ROC graphs are commonly used in medical decision making, and in recent years have been increasingly adopted in the machine learning and data mining research communities. Although ROC graphs are apparently simple, there are some common misconceptions and pitfalls when using them in practice. This article serves both as a tutorial introduction to ROC graphs and as a practical guide for using them in research. <s> BIB001 </s> A Review on Quantification Learning <s> Quantification via Threshold Selection Policies <s> This paper promotes a new task for supervised machine learning research: quantification - the pursuit of learning methods for accurately estimating the class distribution of a test set, with no concern for predictions on individual cases. A variant for cost quantification addresses the need to total up costs according to categories predicted by imperfect classifiers. These tasks cover a large and important family of applications that measure trends over time.The paper establishes a research methodology, and uses it to evaluate several proposed methods that involve selecting the classification threshold in a way that would spoil the accuracy of individual classifications. In empirical tests, Median Sweep methods show outstanding ability to estimate the class distribution, despite wide disparity in testing and training conditions. The paper addresses shifting class priors and costs, but not concept drift in general. <s> BIB002 </s> A Review on Quantification Learning <s> Quantification via Threshold Selection Policies <s> Many business applications track changes over time, for example, measuring the monthly prevalence of influenza incidents. In situations where a classifier is needed to identify the relevant incidents, imperfect classification accuracy can cause substantial bias in estimating class prevalence. The paper defines two research challenges for machine learning. The `quantification' task is to accurately estimate the number of positive cases (or class distribution) in a test set, using a training set that may have a substantially different distribution. The `cost quantification' variant estimates the total cost associated with the positive class, where each case is tagged with a cost attribute, such as the expense to resolve the case. Quantification has a very different utility model from traditional classification research. For both forms of quantification, the paper describes a variety of methods and evaluates them with a suitable methodology, revealing which methods give reliable estimates when training data is scarce, the testing class distribution differs widely from training, and the positive class is rare, e.g., 1% positives. These strengths can make quantification practical for business use, even where classification accuracy is poor. <s> BIB003
The AC quantifier produces accurate estimations for many tasks, but, according to several studies, its performance degrades, especially when the training data are highly imbalanced. BIB002 BIB003 proposes a group of methods based on calibrated threshold classifiers to overcome this issue: (i) a threshold classifier is learned (for instance using SVM) and (ii) the threshold is adjusted. The only difference between them is the strategy to select the threshold. These methods were mainly designed for binary quantification but can be applicable for any other quantification problem that employs an underlying threshold classifier, for instance, in cost quantification (Section 10). When the training data are imbalanced, usually because the positives examples are scarce, the performance of AC degrades severely BIB002 ). The problem is that, in such cases, the classifier tends to predict the majority (negative) class, reducing the number of false positives, but at the cost of a low tpr , see BIB001 . Mathematically, this implies a small denominator in Equation (32) that produces bigger corrections, being more vulnerable to small errors in the estimates of tpr and f pr. Formally, these methods are based on selecting a threshold that reduces the variance of the estimations for tpr and f pr. The idea is to select a threshold that increments the number of true positives but usually at the cost of incrementing the rate of false positives. Whenever tpr ≫ f pr the denominator in Equation (32) increases, resulting in an adjusting method that produces smaller corrections and is more resistant to small errors in the estimates for tpr and f pr. Following this idea, Forman proposes different threshold selection policies: -The Max method selects the threshold that maximizes tpr − f pr. This gives the biggest possible denominator in Equation (32) for the trained classifier, smoothing the posterior corrections. -The X method tries to obtain a f pr equal to 1 − tpr to avoid the tails of both curves. -The T50 method picks the threshold with tpr = 0.5, assuming that the positives are the minority class. The idea is again to avoid the tails of the tpr curve. -The Median Sweep (MS) method follows a kind of ensemble approach, computing the prevalence for all possible thresholds. MS returns the median of these values as the final prediction. Notice that this strategy requires to estimate the tpr and the f pr for such thresholds during training.