A Review on Quantification Learning <s> Adjusted Count <s> In an article in this issue, Webb and Ting criticize ROC analysis for its inability to handle certain changes in class distributions. They imply that the ability of ROC graphs to depict performance in the face of changing class distributions has been overstated. In this editorial response, we describe two general types of domains and argue that Webb and Ting's concerns apply primarily to only one of them. Furthermore, we show that there are interesting real-world domains of the second type, in which ROC analysis may be expected to hold in the face of changing class distributions. <s> BIB001 </s> A Review on Quantification Learning <s> Adjusted Count <s> Many business applications track changes over time, for example, measuring the monthly prevalence of influenza incidents. In situations where a classifier is needed to identify the relevant incidents, imperfect classification accuracy can cause substantial bias in estimating class prevalence. The paper defines two research challenges for machine learning. The `quantification' task is to accurately estimate the number of positive cases (or class distribution) in a test set, using a training set that may have a substantially different distribution. The `cost quantification' variant estimates the total cost associated with the positive class, where each case is tagged with a cost attribute, such as the expense to resolve the case. Quantification has a very different utility model from traditional classification research. For both forms of quantification, the paper describes a variety of methods and evaluates them with a suitable methodology, revealing which methods give reliable estimates when training data is scarce, the testing class distribution differs widely from training, and the positive class is rare, e.g., 1% positives. These strengths can make quantification practical for business use, even where classification accuracy is poor. <s> BIB002 </s> A Review on Quantification Learning <s> Adjusted Count <s> Verbal autopsy procedures are widely used for estimating cause-specific mortality in areas without medical death certification. Data on symptoms reported by caregivers along with the cause of death are collected from a medical facility, and the cause-of-death distribution is estimated in the population where only symptom data are available. Current approaches analyze only one cause at a time, involve assumptions judged difficult or impossible to satisfy, and require expensive, time-consuming, or unreliable physician reviews, expert algorithms, or parametric statistical models. By generalizing current approaches to analyze multiple causes, we show how most of the difficult assumptions underlying existing methods can be dropped. These generalizations also make physician review, expert algorithms and parametric statistical assumptions unnecessary. With theoretical results, and empirical analyses in data from China and Tanzania, we illustrate the accuracy of this approach. While no method of analyzing verbal autopsy data, including the more computationally intensive approach offered here, can give accurate estimates in all circumstances, the procedure offered is conceptually simpler, less expensive, more general, as or more replicable, and easier to use in practice than existing approaches. We also show how our focus on estimating aggregate proportions, which are the quantities of primary interest in verbal autopsy studies, may also greatly reduce the assumptions necessary for, and thus improve the performance of, many individual classifiers in this and other areas. As a companion to this paper, we also offer easy-to-use software that implements the methods discussed herein. <s> BIB003 </s> A Review on Quantification Learning <s> Adjusted Count <s> Quantification is the name given to a novel machine learning task which deals with correctly estimating the number of elements of one class in a set of examples. The output of a quantifier is a real value, since training instances are the same as a classification problem, a natural approach is to train a classifier and to derive a quantifier from it. Some previous works have shown that just classifying the instances and counting the examples belonging to the class of interest classify count typically yields bad quantifiers, especially when the class distribution may vary between training and test. Hence, adjusted versions of classify count have been developed by using modified thresholds. However, previous works have explicitly discarded (without a deep analysis) any possible approach based on the probability estimations of the classifier. In this paper, we present a method based on averaging the probability estimations of a classifier with a very simple scaling that does perform reasonably well, showing that probability estimators for quantification capture a richer view of the problem than methods based on a threshold. <s> BIB004 </s> A Review on Quantification Learning <s> Adjusted Count <s> We introduce Fisher consistency in the sense of unbiasedness as a desirable property for estimators of class prior probabilities. Lack of Fisher consistency could be used as a criterion to dismiss estimators that are unlikely to deliver precise estimates in test datasets under prior probability and more general dataset shift. The usefulness of this unbiasedness concept is demonstrated with three examples of classifiers used for quantification: Adjusted Classify & Count, EM-algorithm and CDE-Iterate. We find that Adjusted Classify & Count and EM-algorithm are Fisher consistent. A counter-example shows that CDE-Iterate is not Fisher consistent and, therefore, cannot be trusted to deliver reliable estimates of class probabilities. <s> BIB005
Maybe one of the most interesting and relevant quantification methods is the Adjusted Count (AC) approach proposed by  BIB002 . The main interest lies in the fact that the AC method is a theoretically perfect method when its learning assumptions are fulfilled. It is based on correcting the estimate provided by the CC approach, depending on the characteristics of the underlying classifier. An AC model is composed by two elements: a classifier (like the CC method) but also an estimation of its tpr and f pr, obtained using, for instance, cross-validation or a separated validation set. In the prediction phase, AC counts the number of examples in the testing sample predicted as positives by the classifier (following the CC approach described before) getting a first estimate,p 0 , that it is afterwards adjusted applying the following formula: This expression is obtained by solving for the true prevalence, p, in Equation (30).p represents the adjusted prevalence of the positives in the test set, withp 0 being the estimate of CC. Sometimes, this expression leads to an infeasible value ofp that needs to be clipped into the range [0, 1] in a final step. Theoretically, AC returns perfect predictions, independently of the classifier's accuracy, whenever (1) P (x |y) is constant and (2) the estimations of tpr and f pr are perfect. Notice that when P (x |y) does not vary, it ensures that tpr and f pr are independent of a change in the distribution of the classes BIB001 . BIB005 proves that AC is Fisher consistent under such a perfect scenario. Unfortunately, it is rare to fulfill both conditions in real-world applications: P (x |y) shows, at least, small variations and it is difficult to obtain perfect estimations for tpr and f pr in some domains, because only small samples are available and/or they are highly imbalanced. But even in these cases, the performance of the AC method is usually better than that of the CC approach. Actually, the correction procedure of the AC method has a long history in epidemiology   . This same idea was already presented in the 1970s for the estimation of class proportion using screening tests. Many clinical tests are not 100% accurate, and, thus, their results are corrected taking into account the sensitivity (sens = tpr ) and the specificity (spec = 1 − f pr) of the test observed in the past. The prevalence when the clinical test is applied to a given population is corrected using the following expression: 6.3 Multi-Class AC The AC method can be easily extended to multi-class quantification BIB003  . Given a multi-class classifier h and a test set T , the probability of predicting that a random example x belongs to class i can be written as in which P (h(x ) = i |y = j ) is the probability that h predicts class i when the example actually belongs to class j and P T ( j ) represents the actual prevalence of class j in T . The key element is P (h(x ) = i |y = j ), twhich can be estimated from the training set using, for instance, crossvalidation. This same expression can be written for all classes resulting in a system of l equations with l unknowns, the values of P T ( j ). Actually, l − 1 unknowns, because l j=1 P T ( j ) = 1. The solution of this system gives us the predicted prevalence for each class, [p 1 , . . . ,p l ]. BIB004 propose two probabilistic variants of the CC/AC methods. The first one, called Probability Average (PA), is the counterpart of CC and will be denoted here as Probabilistic CC (PCC). The only difference with respect to CC is that PCC learns a probabilistic classifier instead of a crisp one. If the classifier returns the probability that an example belongs to the positive class, then the prevalence is computed as the average of such probability for all the examples in the test sample:p