A Review on Quantification Learning <s> Ensembles for Quantification Learning <s> Many business applications track changes over time, for example, measuring the monthly prevalence of influenza incidents. In situations where a classifier is needed to identify the relevant incidents, imperfect classification accuracy can cause substantial bias in estimating class prevalence. The paper defines two research challenges for machine learning. The `quantification' task is to accurately estimate the number of positive cases (or class distribution) in a test set, using a training set that may have a substantially different distribution. The `cost quantification' variant estimates the total cost associated with the positive class, where each case is tagged with a cost attribute, such as the expense to resolve the case. Quantification has a very different utility model from traditional classification research. For both forms of quantification, the paper describes a variety of methods and evaluates them with a suitable methodology, revealing which methods give reliable estimates when training data is scarce, the testing class distribution differs widely from training, and the positive class is rare, e.g., 1% positives. These strengths can make quantification practical for business use, even where classification accuracy is poor. <s> BIB001 </s> A Review on Quantification Learning <s> Ensembles for Quantification Learning <s> Our hypothesis: ensembles are well suited for problems with distribution changes.If those changes are characterizable, ensembles can be designed to tackle them.Idea: to generate different samples based on the expected distribution changes.Case study: we present ensembles versions of two binary quantification algorithms.Ensembles outperform original counterpart algorithms using trivial aggregation rules. Display Omitted Ensemble methods are widely applied to supervised learning tasks. Based on a simple strategy they often achieve good performance, especially when the single models comprising the ensemble are diverse. Diversity can be introduced into the ensemble by creating different training samples for each model. In that case, each model is trained with a data distribution that may be different from the original training set distribution. Following that idea, this paper analyzes the hypothesis that ensembles can be especially appropriate in problems that: (i) suffer from distribution changes, (ii) it is possible to characterize those changes beforehand. The idea consists in generating different training samples based on the expected distribution changes, and to train one model with each of them. As a case study, we shall focus on binary quantification problems, introducing ensembles versions for two well-known quantification algorithms. Experimental results show that these ensemble adaptations outperform the original counterpart algorithms, even when trivial aggregation rules are used. <s> BIB002
(1) Sample generation. This step generates the samples for training the models of the ensemble. Each training sample has the same size that the training dataset, but its prevalence is different. The process to generate each sample works as follows. First, p j , the sample prevalence, is fixed randomly in a predefined range [p min , p max ] that depends on the expected drift in the distribution. Then, the examples for the sample are selected using random sampling with replacement (to guarantee that P (x |y) is constant) until the distribution [p j , 1 − p j ] is obtained. (2) Training phase. The quantifiers of the ensemble, {h 1 , . . . ,h k }, are learned by training the selected quantification algorithm over each generated training sample. Two different quantification methods are considered in the article: AC (Equation (32)) (see Section 6.2) and HDy (see Section 8.4). (3) Prediction. Each modelh j of the ensemble is applied to obtain the prevalence estimate, p j , of the unlabeled test set T . The proposed aggregation function is the arithmetic mean, thus the final prediction is just mean(p 1 , . . . ,p k ). The authors argue that the application of an ensemble approach to quantification tasks should produce more advantages than its use in other learning problems, for example, classification. The advantage of using a collection of models instead of a single one is common to all learning problems where ensemble algorithms are applied: an individual model presents the risk of performing poorly in some situations; theoretically, an ensemble reduces such risk. However, there are additional benefits in the quantification case: (i) The knowledge about the learning problem helps to introduce an adequate diversity into the ensemble and (ii) using multiple models reduces the issues of the base quantification algorithm, especially for those methods that correct their estimates. The AC correction (Equation (32)) is sometimes unstable, despite it theoretically returns perfect estimations. When the estimates of tpr and f pr are inaccurate, the correction will cause inappropriate changes BIB001 BIB002 ) (this is in part the motivation of the methods in Section 6.5). Using ensembles reduces both risks: the risk of having a bad single classifier and the risk of a poor posterior correction.