A Survey of old and New Results for the Test Error Estimation of a Classifier <s> Normal Approximation <s> We revisit the problem of interval estimation of a binomial proportion. The erratic behavior of the coverage probability of the stan- d ardWaldconfid ence interval has previously been remarkedon in the literature (Blyth andStill, Agresti andCoull, Santner andothers). We begin by showing that the chaotic coverage properties of the Waldinter- val are far more persistent than is appreciated. Furthermore, common textbook prescriptions regarding its safety are misleading and defective in several respects andcannot be trusted . This leads us to consideration of alternative intervals. A number of natural alternatives are presented, each with its motivation and con- text. Each interval is examinedfor its coverage probability andits length. Basedon this analysis, we recommendthe Wilson interval or the equal- tailedJeffreys prior interval for small n andthe interval suggestedin Agresti andCoull for larger n. We also provide an additional frequentist justification for use of the Jeffreys interval. <s> BIB001 </s> A Survey of old and New Results for the Test Error Estimation of a Classifier <s> Normal Approximation <s> In this paper, we review the k–Fold Cross Validation (KCV) technique, applied to the Support Vector Machine (SVM) classification algorithm. We compare several variations on the KCV technique: some of them are often used by practitioners, but without any theoretical justification, while others are less used but more rigorous in finding a correct classifier. The last ones allow to establish an upper– bound of the error rate of the SVM, which represent a way to guarantee, in a statistical sense, the reliability of the classifier and, therefore, turns out to be quite important in many real–world applications. Some experimental results on well–known benchmarking datasets allow to perform the comparison and support our claims. <s> BIB002
A simple formula for the estimation of the confidence interval can be derived by the application of the central limit theorem and, consequently, from the approximation of the binomial by a normal distribution BIB001 . Under this hypothesis, the inequality, which represents the upper-bound of the confidence interval, can be written as: where n is the sample size, σ 2 is the variance and z 1−δ denotes the (1 − δ)-th percentile of the standard normal distribution. As the variance is unknown, we can approximate σ 2 by exploiting the definition of sample variance of the empirical error, according to Eq. BIB002 . Then, an explicit upper-bound for the generalization error formulation can be obtained: or, equivalently: according to Eq. (7). Unfortunately, as the binomial distribution could noticeably differ from the normal approximation, especially when the number of test patterns n is small, and the sample variance could differ from the true one, the bound is not rigorous and its coverage probability could fall well below the userdefined confidence level (1 − δ).