A Review on Quantification Learning <s> Changes in Data Distribution <s> It sometimes happens (for instance in case control studies) that a classifier is trained on a data set that does not reflect the true a priori probabilities of the target classes on real-world data. This may have a negative effect on the classification accuracy obtained on the real-world data set, especially when the classifier's decisions are based on the a posteriori probabilities of class membership. Indeed, in this case, the trained classifier provides estimates of the a posteriori probabilities that are not valid for this real-world data set (they rely on the a priori probabilities of the training set). Applying the classifier as is (without correcting its outputs with respect to these new conditions) on this new data set may thus be suboptimal. In this note, we present a simple iterative procedure for adjusting the outputs of the trained classifier with respect to these new a priori probabilities without having to refit the model, even when these probabilities are not known in advance. As a by-product, estimates of the new a priori probabilities are also obtained. This iterative algorithm is a straightforward instance of the expectation-maximization (EM) algorithm and is shown to maximize the likelihood of the new data. Thereafter, we discuss a statistical test that can be applied to decide if the a priori class probabilities have changed from the training set to the real-world data. The procedure is illustrated on different classification problems involving a multilayer neural network, and comparisons with a standard procedure for a priori probability estimation are provided. Our original method, based on the EM algorithm, is shown to be superior to the standard one for a priori probability estimation. Experimental results also indicate that the classifier with adjusted outputs always performs better than the original one in terms of classification accuracy, when the a priori probability conditions differ from the training set to the real-world data. The gain in classification accuracy can be significant. <s> BIB001 </s> A Review on Quantification Learning <s> Changes in Data Distribution <s> In an article in this issue, Webb and Ting criticize ROC analysis for its inability to handle certain changes in class distributions. They imply that the ability of ROC graphs to depict performance in the face of changing class distributions has been overstated. In this editorial response, we describe two general types of domains and argue that Webb and Ting's concerns apply primarily to only one of them. Furthermore, we show that there are interesting real-world domains of the second type, in which ROC analysis may be expected to hold in the face of changing class distributions. <s> BIB002 </s> A Review on Quantification Learning <s> Changes in Data Distribution <s> Many business applications track changes over time, for example, measuring the monthly prevalence of influenza incidents. In situations where a classifier is needed to identify the relevant incidents, imperfect classification accuracy can cause substantial bias in estimating class prevalence. The paper defines two research challenges for machine learning. The `quantification' task is to accurately estimate the number of positive cases (or class distribution) in a test set, using a training set that may have a substantially different distribution. The `cost quantification' variant estimates the total cost associated with the positive class, where each case is tagged with a cost attribute, such as the expense to resolve the case. Quantification has a very different utility model from traditional classification research. For both forms of quantification, the paper describes a variety of methods and evaluates them with a suitable methodology, revealing which methods give reliable estimates when training data is scarce, the testing class distribution differs widely from training, and the positive class is rare, e.g., 1% positives. These strengths can make quantification practical for business use, even where classification accuracy is poor. <s> BIB003 </s> A Review on Quantification Learning <s> Changes in Data Distribution <s> The most basic assumption used in statistical learning theory is that training data and test data are drawn from the same underlying distribution. Unfortunately, in many applications, the "in-domain" test data is drawn from a distribution that is related, but not identical, to the "out-of-domain" distribution of the training data. We consider the common case in which labeled out-of-domain data is plentiful, but labeled in-domain data is scarce. We introduce a statistical formulation of this problem in terms of a simple mixture model and present an instantiation of this framework to maximum entropy classifiers and their linear chain counterparts. We present efficient inference algorithms for this special case based on the technique of conditional expectation maximization. Our experimental results show that our approach leads to improved performance on three real world tasks on four different data sets from the natural language processing domain. <s> BIB004 </s> A Review on Quantification Learning <s> Changes in Data Distribution <s> The field of dataset shift has received a growing amount of interest in the last few years. The fact that most real-world applications have to cope with some form of shift makes its study highly relevant. The literature on the topic is mostly scattered, and different authors use different names to refer to the same concepts, or use the same name for different concepts. With this work, we attempt to present a unifying framework through the review and comparison of some of the most important works in the literature. <s> BIB005 </s> A Review on Quantification Learning <s> Changes in Data Distribution <s> Class distribution estimation (quantification) plays an important role in many practical classification problems. Firstly, it is important in order to adapt the classifier to the operational conditions when they differ from those assumed in learning. Additionally, there are some real domains where the quantification task is itself valuable due to the high variability of the class prior probabilities. Our novel quantification approach for two-class problems is based on distributional divergence measures. The mismatch between the test data distribution and validation distributions generated in a fully controlled way is measured by the Hellinger distance in order to estimate the prior probability that minimizes this divergence. Experimental results on several binary classification problems show the benefits of this approach when compared to such approaches as counting the predicted class labels and other methods based on the classifier confusion matrix or on posterior probability estimations. We also illustrate these techniques as well as their robustness against the base classifier performance (a neural network) with a boar semen quality control setting. Empirical results show that the quantification can be conducted with a mean absolute error lower than 0.008, which seems very promising in this field. <s> BIB006 </s> A Review on Quantification Learning <s> Changes in Data Distribution <s> Concept drift primarily refers to an online supervised learning scenario when the relation between the input data and the target variable changes over time. Assuming a general knowledge of supervised learning in this article, we characterize adaptive learning processes; categorize existing strategies for handling concept drift; overview the most representative, distinct, and popular techniques and algorithms; discuss evaluation methodology of adaptive algorithms; and present a set of illustrative applications. The survey covers the different facets of concept drift in an integrated way to reflect on the existing scattered state of the art. Thus, it aims at providing a comprehensive introduction to the concept drift adaptation for researchers, industry analysts, and practitioners. <s> BIB007 </s> A Review on Quantification Learning <s> Changes in Data Distribution <s> Most machine learning models are static, but the world is dynamic, and increasing online deployment of learned models gives increasing urgency to the development of efficient and effective mechanisms to address learning in the context of non-stationary distributions, or as it is commonly called concept drift. However, the key issue of characterizing the different types of drift that can occur has not previously been subjected to rigorous definition and analysis. In particular, while some qualitative drift categorizations have been proposed, few have been formally defined, and the quantitative descriptions required for precise and objective understanding of learner performance have not existed. We present the first comprehensive framework for quantitative analysis of drift. This supports the development of the first comprehensive set of formal definitions of types of concept drift. The formal definitions clarify ambiguities and identify gaps in previous definitions, giving rise to a new comprehensive taxonomy of concept drift types and a solid foundation for research into mechanisms to detect and address concept drift. <s> BIB008 </s> A Review on Quantification Learning <s> Changes in Data Distribution <s> For evolving populations the training data and the test data need not follow the same distribution. Thus, the performance of a prediction model will deteriorate over the course of time. This requires the re-estimation of the prediction model after some time. However, in many applications e.g. credit scoring, new labelled data are not available for re-estimation due to verification latency, i.e. label delay. Thus, methods which enable a prediction model to adapt to distributional changes by using only unlabelled data are highly desirable. A shift adaptation method for binary classification is presented here. The model is based on mixture distributions. The conditional feature distributions are determined at the time where labelled data are available, and the unconditional feature distribution is determined at the time where new unlabelled data are accessible. These mixture distributions provide information on the old and the new positions of subpopulations. A transition model then describes how the subpopulations of each class have drifted to form the new unconditional feature distribution. Assuming that the conditional distributions are reorganised using a minimum of energy, a two-step estimation procedure results. First, for a given class prior distribution the transfer of probability mass is estimated such that the energy required to obtain the new unconditional distribution by a local transfer of the old conditional distributions is a minimum. Since the optimal solution of the resulting transportation problem measures the distance between the old and the new distributions, the change of the class prior distribution is found in a second step by solving the transportation problem for varying class prior distributions and selecting the value for which the objective function is a minimum. Using the solution of the transportation problem and the component parameters of the unconditional feature distribution, the new conditional feature distribution can be determined. This thus allows for a shift adaptation of the classification rule. The performance of the proposed model is investigated using a large real-world dataset on default rates in Danish companies. The results show that the shift adaptation improves classification results. <s> BIB009
Lately, there has been an increasing interest in the real applications that present challenges arising from data distribution changes, also named dataset shift problems. Several machine-learning problems fall in this category, and probably the most popular are concept drift BIB007 , domain adaptation BIB004 , and transfer learning (Pan and Yang 2010;  . All these problems share that the distribution used for training our model is different than the data the model will face later. Different taxonomies for these kinds of problems have been proposed according to the drift type, see BIB005 ,  , and BIB008 . These articles characterize the changes that occur in the joint probability distribution of the covariates and the class variables, P (X × Y). Quantification is one of those problems but its dataset-shift setting appears under different names, with prior probability shift being the most common BIB005 . For some authors, the causal relationship between x and y determines the type of expected changes in the distribution. In this sense, BIB002 distinguish two types of problems: Y → X problems, in which the covariates x are causally determined by the class y, and X → Y problems, where y causally depends on the covariates x. Spam detection is an example of the latter group, because an email is considered spam or not depending on its content, and a medical diagnosis problem is a paradigmatic example of the first group: to be affected by a disease y will produce a series of symptoms x. The joint probability of x and y, P (x, y), can be written as P (y|x )P (x ) in X → Y problems and P (x |y)P (y) in Y → X problems. Dataset shift occurs when at least one of these four terms changes between the training data and the test sets, and, thus, the joint distribution also varies, in symbols P tr (x, y) P tst (x, y). Therefore, there are several types of dataset-shift problems depending on the elements that change. In the case of quantification tasks, it is obvious that P (y) changes, otherwise the desired quantifier would be trivial: It returns always a set of constants equal to the prior probabilities, P tr (y). Also, some authors restrict quantification tasks to Y → X problems; see, for instance, the definition of prior probability shift tasks in BIB005 . We think that this is not correct; one may find examples of quantification tasks for both group of problems. For instance, quantifying consumers' opinion belongs to X → Y problems, while quantifying plankton morphological groups belongs to Y → X problems. However, one of the main reason to identify quantification tasks with Y → X problems is because almost all quantification algorithms BIB001 BIB003 BIB006 ) assume that P (x |y) does not change. This learning assumption seems easier to be fulfilled in Y → X problems, because y determines x. In our opinion, the property that P (x |y) remains unaltered must be analysed for each particular application, independently that it belongs to X → Y or Y → X problems. Only a few methods assume that P (x |y) may change, for instance that in BIB009 .