In this paper, we study the accuracy of values aggregated over classes predicted by a classification algorithm. The problem is that the resulting aggregates (e.g., sums of a variable) are known to be biased. The bias can be large even for highly accurate classification algorithms, in particular when dealing with class-imbalanced data. To correct this bias, the algorithm's classification error rates have to be estimated. In this estimation, two issues arise when applying existing bias correction methods. First, inaccuracies in estimating classification error rates have to be taken into account. Second, impermissible estimates, such as a negative estimate for a positive value, have to be dismissed. We show that both issues are relevant in applications where the true labels are known only for a small set of data points. We propose a novel bias correction method using Bayesian inference. The novelty of our method is that it imposes constraints on the model parameters. We show that our method solves the problem of biased classification-based aggregates as well as the two issues above, in the general setting of multi-class classification. In the empirical evaluation, using a binary classifier on a real-world dataset of company tax returns, we show that our method outperforms existing methods in terms of mean squared error.
An alternative is an that measures the bias on the aggregate level instead of on the individual level @cite_8 . Other alternatives include averaging multiple (biased) estimators into a single, more accurate estimator @cite_9 . Such alternatives will reduce the bias of classification-based aggregates, but our proposed method will completely remove it for sufficiently large test sets.
@cite_9 @cite_8
2070256050 2063746322
The bias correction to the maximum likelihood extimates of the parameters for logistic discrimination is examined under mixture and separate sampling schemes. An existing adjustment developed under mixture sampling and based on higher derivatives of the log likelihood is modified slightly for use under separate sampling. The effect of this bias correction on the sampling properties of the misclassification rates and of the estimated posterior probabilities is discussed. Given a collection of @math different estimators or classifiers, we study the problem of model selection type aggregation, i.e., we construct a new estimator or classifier, called aggregate, which is nearly as good as the best among them with respect to a given risk criterion. We define our aggregate by a simple recursive procedure which solves an auxiliary stochastic linear programming problem related to the original non-linear one and constitutes a special case of the mirror averaging algorithm. We show that the aggregate satisfies sharp oracle inequalities under some general assumptions. The results allow one to construct in an easy way sharp adaptive nonparametric estimators for several problems including regression, classification and density estimation.