A Review on Quantification Learning <s> Performance Measures for Binary Quantification <s> This paper promotes a new task for supervised machine learning research: quantification - the pursuit of learning methods for accurately estimating the class distribution of a test set, with no concern for predictions on individual cases. A variant for cost quantification addresses the need to total up costs according to categories predicted by imperfect classifiers. These tasks cover a large and important family of applications that measure trends over time.The paper establishes a research methodology, and uses it to evaluate several proposed methods that involve selecting the classification threshold in a way that would spoil the accuracy of individual classifications. In empirical tests, Median Sweep methods show outstanding ability to estimate the class distribution, despite wide disparity in testing and training conditions. The paper addresses shifting class priors and costs, but not concept drift in general. <s> BIB001 </s> A Review on Quantification Learning <s> Performance Measures for Binary Quantification <s> Insemination techniques in the veterinary field demand more objective methods to control the quality of sperm samples. In particular, different factors may damage a number of sperm cells that is difficult to predict in advance. This paper addresses the problem of quantifying the proportion of damaged/intact sperm cells in a given sample based on computer vision techniques and supervised learning. Unlike common supervised classification approaches, neither the individual example classification is important nor the class distribution assumed in learning can be considered stationary. To deal with this, an estimation process based on Posterior Probability estimates (PP), and known to increase the classifier accuracy under changes in class distributions, is assessed here for quantification purposes. It is compared with two approaches based on the classifier confusion matrix (Adjusted Count and Median Sweep) and the naive approach based on classifying and counting. Experimental results with boar sperm samples and back propagation neural networks show that the PP based quantification outperforms any of the previously considered approaches in terms of the Mean Absolute Error, Kullback Leibler divergence and Mean Relative Error. Moreover, in spite of an imperfect classification, the PP approach guarantees a uniform Mean Absolute Error (around 3%) for whatever combination of training and test class distributions, what is very promising in this practical field. <s> BIB002 </s> A Review on Quantification Learning <s> Performance Measures for Binary Quantification <s> Many business applications track changes over time, for example, measuring the monthly prevalence of influenza incidents. In situations where a classifier is needed to identify the relevant incidents, imperfect classification accuracy can cause substantial bias in estimating class prevalence. The paper defines two research challenges for machine learning. The `quantification' task is to accurately estimate the number of positive cases (or class distribution) in a test set, using a training set that may have a substantially different distribution. The `cost quantification' variant estimates the total cost associated with the positive class, where each case is tagged with a cost attribute, such as the expense to resolve the case. Quantification has a very different utility model from traditional classification research. For both forms of quantification, the paper describes a variety of methods and evaluates them with a suitable methodology, revealing which methods give reliable estimates when training data is scarce, the testing class distribution differs widely from training, and the positive class is rare, e.g., 1% positives. These strengths can make quantification practical for business use, even where classification accuracy is poor. <s> BIB003 </s> A Review on Quantification Learning <s> Performance Measures for Binary Quantification <s> The increasing availability of participatory web and social media presents enormous opportunities to study human relations and collective behaviors. Many applications involving decision making want to obtain certain generalized properties about the population in a network, such as the proportion of actors given a category, instead of the category of individuals. While data mining and machine learning researchers have developed many methods for link-based classification or relational learning, most are optimized to classify individual nodes in a network. In order to accurately estimate the prevalence of one class in a network, some quantification method has to be used. In this work, two kinds of approaches are presented: quantification based on classification or quantification based on link analysis. Extensive experiments are conducted on several representative network data, with interesting findings reported concerning efficacy and robustness of different quantification methods, providing insights to further quantify the ebb and flow of online collective behaviors at macro-level. <s> BIB004 </s> A Review on Quantification Learning <s> Performance Measures for Binary Quantification <s> Quantification is the name given to a novel machine learning task which deals with correctly estimating the number of elements of one class in a set of examples. The output of a quantifier is a real value, since training instances are the same as a classification problem, a natural approach is to train a classifier and to derive a quantifier from it. Some previous works have shown that just classifying the instances and counting the examples belonging to the class of interest classify count typically yields bad quantifiers, especially when the class distribution may vary between training and test. Hence, adjusted versions of classify count have been developed by using modified thresholds. However, previous works have explicitly discarded (without a deep analysis) any possible approach based on the probability estimations of the classifier. In this paper, we present a method based on averaging the probability estimations of a classifier with a very simple scaling that does perform reasonably well, showing that probability estimators for quantification capture a richer view of the problem than methods based on a threshold. <s> BIB005 </s> A Review on Quantification Learning <s> Performance Measures for Binary Quantification <s> We study the classifier quantification problem in the context of the topical opinion retrieval, that consists in estimating proportions of the sentiment categories in the result set of a topic. We propose a methodology to circumvent individual classification allowing a real-time sentiment analysis for huge volumes of data. After discussing existing approaches to quantification, the novel proposed methodology is applied to Microblogging Retrieval and provides statistically significant estimates of sentiment category proportions. Our solution modifies Hopkins and King’s approach in order to remove manual intervention, and making sentiment analysis feasible in real time. Evaluation is conduced with a test collection made up of about 3,2M tweets. <s> BIB006 </s> A Review on Quantification Learning <s> Performance Measures for Binary Quantification <s> Surveys show that the mean absolute percentage error (MAPE) is the most widely used measure of prediction accuracy in businesses and organizations. It is, however, biased: when used to select among competing prediction methods it systematically selects those whose predictions are too low. This has not been widely discussed and so is not generally known among practitioners. We explain why this happens. We investigate an alternative relative accuracy measure which avoids this bias: the log of the accuracy ratio, that is, log (prediction/actual). Relative accuracy is particularly relevant if the scatter in the data grows as the value of the variable grows (heteroscedasticity). We demonstrate using simulations that for heteroscedastic data (modelled by a multiplicative error factor) the proposed metric is far superior to MAPE for model selection. Another use for accuracy measures is in fitting parameters to prediction models. Minimum MAPE models do not predict a simple statistic and so theoretical analysis is limited. We prove that when the proposed metric is used instead, the resulting least squares regression model predicts the geometric mean. This important property allows its theoretical properties to be understood. <s> BIB007 </s> A Review on Quantification Learning <s> Performance Measures for Binary Quantification <s> Sentiment classification has become a ubiquitous enabling technology in the Twittersphere, since classifying tweets according to the sentiment they convey towards a given entity (be it a product, a person, a political party, or a policy) has many applications in political science, social science, market research, and many others. In this paper, we contend that most previous studies dealing with tweet sentiment classification (TSC) use a suboptimal approach. The reason is that the final goal of most such studies is not estimating the class label (e.g., Positive, Negative, or Neutral) of individual tweets, but estimating the relative frequency (a.k.a. “prevalence”) of the different classes in the dataset. The latter task is called quantification, and recent research has convincingly shown that it should be tackled as a task of its own, using learning algorithms and evaluation measures different from those used for classification. In this paper, we show (by carrying out experiments using two learners, seven quantification-specific algorithms, and 11 TSC datasets) that using quantification-specific algorithms produces substantially better class frequency estimates than a state-of-the-art classification-oriented algorithm routinely used in TSC. We thus argue that researchers interested in tweet sentiment prevalence should switch to quantification-specific (instead of classification-specific) learning algorithms and evaluation measures. <s> BIB008 </s> A Review on Quantification Learning <s> Performance Measures for Binary Quantification <s> long range forecasting from crystal ball to computer is available in our book collection an online access to it is set as public so you can get it instantly. Our digital library hosts in multiple locations, allowing you to get the most less latency time to download any of our books like this one. Merely said, the long range forecasting from crystal ball to computer is universally compatible with any devices to read. <s> BIB009 </s> A Review on Quantification Learning <s> Performance Measures for Binary Quantification <s> The estimation of class prevalence, i.e., the fraction of the population that belongs to a certain class, is a very useful tool in data analytics and learning, and finds applications in many domains, such as sentiment analysis, epidemiology, etc. For example, in sentiment analysis, the objective is often not to estimate whether a specific text conveys positive or negative sentiment, but rather estimate the overall distribution of positive and negative sentiment during an event window. A popular way of performing the above task, often dubbed quantification, is to use supervised learning to train a prevalence estimator from labelled data. In this paper we propose the first online stochastic algorithms for directly optimizing (i) performance measures for quantification, and (ii) hybrid performance measures that seek to balance quantification and classification performance. We prove rigorous bounds for our algorithms which demonstrate that they exhibit optimal convergence. Our algorithms present a significant advancement in the theory of multivariate optimization. We also report extensive experiments on benchmark and real data sets which demonstrate that our methods significantly outperform existing optimization techniques used for the quantification problem. <s> BIB010
From the point of view of evaluation, binary quantification presents maybe more choices than the other quantification problems to select our target performance measure. The reason is that binary quantification is one of the simplest quantification problems and it can be reduced to the prediction of the percentage of the positive class, p. Thus, for a model,h, and a given testing set T , we just need to compare the prevalence predicted,p, with the actual one, p. Most of the measures discussed here shall be extended for multi-class quantification in the next section. Perhaps the most popular BIB004 BIB002  performance measure for binary (and multi-class) quantification is Kullback-Leibler (KL) Divergence, also known as discrimination information, relative entropy, or normalized cross-entropy (see  and BIB003 ). KL Divergence is a special case of the family of f-divergences and it can be defined for binary quantification as This metric, ranging from 0 (optimum) to +∞, measures the divergence between the predicted distribution, [p,n = 1 −p], and the actual distribution, [p, n = 1 − p]. The main advantage of KL divergence is that it seems more suitable for averaging over different test distributions than the regression-based metrics described below. However, KL divergence also has some drawbacks: It is not a true metric (it does not obey the triangle inequality and it is not symmetric) and it is less interpretable than other performance metrics. Besides, KLD is undefined whenp is 0 or 1. To resolve these situations, it can be normalized via the logistic function BIB008 : NKLD ranges between 1 (no divergence, p =p) and 0 (total divergence). Other choice in binary quantification is to employ regression performance measures. The regression metrics used in quantification literature are -Bias measures when a binary quantifier tends to overestimate or underestimate the proportion of positives: (12) When a method overestimates p, the bias is positive, and negative otherwise. Bias is used in Forman (2005), BIB001 , and BIB004 . -Absolute Error (AE) is just the difference between both magnitudes: This is one of the most employed measure, because it is simple and easily interpretable. The averaged version over a group of test sets, Mean Absolute Error (MAE) is used in Tang  Mean Squared Error (MSE) is preferred for some authors BIB005 BIB006  ) over MAE. The differences between both is that MAE is more robust to outliers and it is more intuitive and easier to interpret than MSE, and the advantage of MSE is that it does not assign equal weight to all mistakes, emphasizing the extreme values whose consequences may be much bigger than the equivalent smaller ones for a particular application. However, the problem of MAE and MSE is that averaging across testing samples with different prevalences presents certain issues that must be taken into account. For instance, an absolute error of 0.1 produced when the actual prevalence is 1 is not the same as when the actual value is 0.2. In the latter case, the error can be considered worse. This leads to the group of relative measures: -Relative Absolute Error (RAE) is the relation between the absolute error and p, The advantage of RAE is that it diminish the importance of errors to true class prevalence. On the other side, RAE presents some problems: Its range of values can be awkward depending on the value of p and, more importantly, it is asymmetric, unbounded, and undefined when p = 0. The common solution to solve this last issue is to smooth the expression, adding a small constant, ϵ, to the numerator and the denominator. BIB008 propose ϵ =  SAPE tries to solve some of the issues of RAE. A recent article BIB007 has shown that MRAE prefers those models that systematically under-forecast when it is used in model BIB009 , that is, the mean of (16) over a group of testing sets applying (9), seems the best alternative considering all the above factors: it is a percentage, it is always defined and its reliability for model selection purposes is comparable to that of ln(p/p) according to BIB007 . However, SMAPE is less popular than MRAE, it only appears in . Finally, the last group of performance measures for binary quantification are those that provide normalized values  BIB010 ). The goal is to obtain score functions that range between 1 (the optimum) and 0. Analyzing equations (13) and (14), we can observe that their respective upper bounds are max(p, 1 − p) and max(p, 1 − p) 2 . Both can be normalized using those values. Besides, these loss functions can be redefined as their counterpart score functions. Considering both factors, two normalized measures can be derived as: -Normalized Squared Score (NSS)