A Review on Quantification Learning <s> Precision Correction Methods <s> Many business applications track changes over time, for example, measuring the monthly prevalence of influenza incidents. In situations where a classifier is needed to identify the relevant incidents, imperfect classification accuracy can cause substantial bias in estimating class prevalence. The paper defines two research challenges for machine learning. The `quantification' task is to accurately estimate the number of positive cases (or class distribution) in a test set, using a training set that may have a substantially different distribution. The `cost quantification' variant estimates the total cost associated with the positive class, where each case is tagged with a cost attribute, such as the expense to resolve the case. Quantification has a very different utility model from traditional classification research. For both forms of quantification, the paper describes a variety of methods and evaluates them with a suitable methodology, revealing which methods give reliable estimates when training data is scarce, the testing class distribution differs widely from training, and the positive class is rare, e.g., 1% positives. These strengths can make quantification practical for business use, even where classification accuracy is poor. <s> BIB001 </s> A Review on Quantification Learning <s> Precision Correction Methods <s> The problem of estimating the class distribution (or prevalence) for a new unlabelled dataset (from a possibly different distribution) is a very common problem which has been addressed in one way or another in the past decades. This problem has been recently reconsidered as a new task in data mining, renamed quantification when the estimation is performed as an aggregation (and possible adjustment) of a single-instance supervised model (e.g., a classifier). However, the study of quantification has been limited to classification, while it is clear that this problem also appears, perhaps even more frequently, with other predictive problems, such as regression. In this case, the goal is to determine a distribution or an aggregated indicator of the output variable for a new unlabelled dataset. In this paper, we introduce a comprehensive new taxonomy of quantification tasks, distinguishing between the estimation of the whole distribution and the estimation of some indicators (summary statistics), for both classification and regression. This distinction is especially useful for regression, since predictions are numerical values that can be aggregated in many different ways, as in multi-dimensional hierarchical data warehouses. We focus on aggregative quantification for regression and see that the approaches borrowed from classification do not work. We present several techniques based on segmentation which are able to produce accurate estimations of the expected value and the distribution of the output variable. We show experimentally that these methods especially excel for the relevant scenarios where training and test distributions dramatically differ. <s> BIB002
The difference of this group of methods with respect to the previous one is that they focus on improving the classifier precision. The precision is the fraction of positive instances classified as positives by the classifier. Improving precision allows to obtain a better estimate of C + . The goal is to characterize and correct the bias of the precision and then to compute C + more accurately. The methods discussed here are based on the same idea used to design the AC method (Section 6.2). Three different approaches belong to this group: -Precision-Corrected Average*Quantifier (PCAQ). The straightforward approach for correcting the precision is to estimate it over a validation or test set. The problem is the distribution shift that occurs on cost quantification problems. The author propose to use a multi-step process to overcome this issue: (1) The first step is to estimate tpr and f pr of the classifier, using typically cross-validation over the training set, as the AC method does. (2) To estimate the proportion of positives,p, in the test set using a quantification algorithm, for instance, the AC method. (3) Finally, the precision, Pr , of the classifier over the test set is estimated by a scaling formula using tpr and f pr, similar to the AC correction, Using this adjusted estimate of the precision, it is possible to obtain the average cost C + 0 of the instances predicted as positives by the classifier: In this equation, the value of C + 0 is computed as The problem is that both the true average cost of the positives, C + , and of the negatives, C − , are unknowns. To compute them, especially C + , which is our goal, we can obtain a similar equation using the test set T : in which C all = x i ∈T c i . Solving Equations (73) and (74) to obtain the estimate of C + , we have The total cost is then computed using Equation (68). However, this method has the same problem for imbalanced datasets. This affects not only the estimation of tpr like it occurs in binary quantification but also the computation of C + 0 . The idea proposed is again the same than that for binary classification: to select a threshold that gives a worse precision (providing a greater number of predicted positives) but that it is more stable to characterize the bias of the precision. Any of the policies discussed in Section 6.5 may be applicable, but the author selected the X method in BIB001 . -Median Sweep of PCAQ (MS-PCAQ). The implementation of this method is quite straightforward, similar to that of the MS method for binary quantification. Rather than use just a single threshold to build a PCAQ model, the process can be repeated using a collection of thresholds, obtaining an ensemble of PCAQ models. The aggregation rule proposed is again the median. However, for this method, the author suggests three possible policies to discard some of the potential thresholds: (i) the number of predicted positives is lower than a predefined minimum (the value suggested is 30), (ii) the confidence interval for C + is too large, and (iii) the estimation for the precision Pr was computed using fewer than 30 instances predicted as positives. -Mixture Model Average*Quantifier (MMAQ). This algorithm is based on the Mixture Model explained in Section 8.3. MMAQ considers all possible thresholds for modeling the shape of C t curve, applying Equation (56) and solving for C + and C − using linear algebra, Finally, C + can be computed using basic linear regression over a set of data points, (x t , y t ). Most of the regression quantification methods are from BIB002 using the learning setting described in Section 2.3. These methods adapt some Forman's methods (e.g., CC, AC) and are based on some heuristics devised for regression quantification problems. They can be divided into two groups: (i) approaches based on the Regress & Sum method and (ii) methods based on discretization. The predictions obtained with these models can be: a single indicator (e.g., the mean) or the form of the distribution (e.g., shape, dispersion).