A Review on Quantification Learning <s> Forman's Mixture Model Approach <s> Class distribution estimation (quantification) plays an important role in many practical classification problems. Firstly, it is important in order to adapt the classifier to the operational conditions when they differ from those assumed in learning. Additionally, there are some real domains where the quantification task is itself valuable due to the high variability of the class prior probabilities. Our novel quantification approach for two-class problems is based on distributional divergence measures. The mismatch between the test data distribution and validation distributions generated in a fully controlled way is measured by the Hellinger distance in order to estimate the prior probability that minimizes this divergence. Experimental results on several binary classification problems show the benefits of this approach when compared to such approaches as counting the predicted class labels and other methods based on the classifier confusion matrix or on posterior probability estimations. We also illustrate these techniques as well as their robustness against the base classifier performance (a neural network) with a boar semen quality control setting. Empirical results show that the quantification can be conducted with a mean absolute error lower than 0.008, which seems very promising in this field. <s> BIB001 </s> A Review on Quantification Learning <s> Forman's Mixture Model Approach <s> Quantification is the machine learning task of estimating test-data class proportions that are not necessarily similar to those in training. Apart from its intrinsic value as an aggregate statistic, quantification output can also be used to optimize classifier probabilities, thereby increasing classification accuracy. We unify major quantification approaches under a constrained multi-variate regression framework, and use mathematical programming to estimate class proportions for different loss functions. With this modeling approach, we extend existing binary-only quantification approaches to multi-class settings as well. We empirically verify our unified framework by experimenting with several multi-class datasets including the Stanford Sentiment Treebank and CIFAR-10. <s> BIB002
The main idea behind this method proposed by Forman (2005) is to employ the scores produced by a binary classifier (for instance, the scores produced by a SVM model to decide when an example is positive or negative). The distribution of the scores over the testing examples is modeled using a mixture of two distributions obtained using the training examples, one from the positives and the other from the negatives. The method, called Mixture Model (MM), trains a classifier but just to represent the three distributions manipulated by the algorithm. The first step is to obtain the distributions of the positives and the negatives, D + and D − , respectively. These distributions are computed using the scores provided by binary classifiers. Instead of training just one classifier using the whole training data, and then obtaining the scores of those training examples to compute D + and D − , the method performs a many-fold cross-validation (CV); the author uses 50-CV to obtain the scores. This reduces the risk of overfitting. When the number of folds is large, the obtained classifiers will be similar, because they have in common the most part of their training examples. The next step is to represent the distribution of the testing examples. For this, the MM method learns a classifier (using the whole training dataset) that is applied to estimate such distribution, T . The key idea is to model this distribution as a mixture of D + and D − , Notice that this model assumes that P (x |y) is constant, because D + and D − change uniformly. The method returns the value ofp that better matches the distribution ofT and the mixture distribution. One important issue for this group of algorithms, and MM is not an exception, is how to describe the different distributions involved. Forman considers several choices, including the cumulative distribution function (CDF), the empirical probability density function (PDF), or a parametric model. Finally, the author selects CDF, arguing that PDF requires (i) discretizing the scores into artificial bins (especially if a non-probabilistic classifier is used) and (ii) additional parameters for tuning. The last reason is also used to discard a parametric model. The other key element of MM is the procedure to measure the quality of the fit between the mixture model and the distribution of T . The Kolmogorov-Smirnov statistic is commonly used to measure the difference between two CDFs. It computes the maximum difference among all pairwise values of two CDFs. However, the author prefers another difference metric, called PP-Area. The metric is inspired in a visual comparison between the plot of the two CDFs (see Figure 3(c) ). Plotting both, one versus the other, a Probability-Probability (PP) curve is drawn. A perfect 45 line is obtained when both CDFs give the same probability for each input. The level of agreement between the two distributions can be measured comparing the perfect case with the PP curve obtained. There are, again, several options to compare these two functions, like the mean error or the mean-squared error. However, the author prefers to use the area between the PP curve and the 45 line (see Figure 3) . This metric presents the advantage of being commutative and monotonic. Minimizing the PP-Area corresponds to minimizing the L1-norm BIB002 . BIB001 present two algorithms based also on the idea of comparing and matching distributions. They difference between them is how to represent the distributions. However, both algorithms share a key ingredient that is the use of the Hellinger distance as the metric to assess the difference between the distributions.