A Survey of old and New Results for the Test Error Estimation of a Classifier <s> Chebyshev Bound <s> Part 1 Probability and Random Variables 1 The Meaning of Probability 2 The Axioms of Probability 3 Repeated Trials 4 The Concept of a Random Variable 5 Functions of One Random Variable 6 Two Random Variables 7 Sequences of Random Variables 8 Statistics Part 2 Stochastic Processes 9 General Concepts 10 Random Walk and Other Applications 11 Spectral Representation 12 Spectral Estimation 13 Mean Square Estimation 14 Entropy 15 Markov Chains 16 Markov Processes and Queueing Theory <s> BIB001 </s> A Survey of old and New Results for the Test Error Estimation of a Classifier <s> Chebyshev Bound <s> We revisit the problem of interval estimation of a binomial proportion. The erratic behavior of the coverage probability of the stan- d ardWaldconfid ence interval has previously been remarkedon in the literature (Blyth andStill, Agresti andCoull, Santner andothers). We begin by showing that the chaotic coverage properties of the Waldinter- val are far more persistent than is appreciated. Furthermore, common textbook prescriptions regarding its safety are misleading and defective in several respects andcannot be trusted . This leads us to consideration of alternative intervals. A number of natural alternatives are presented, each with its motivation and con- text. Each interval is examinedfor its coverage probability andits length. Basedon this analysis, we recommendthe Wilson interval or the equal- tailedJeffreys prior interval for small n andthe interval suggestedin Agresti andCoull for larger n. We also provide an additional frequentist justification for use of the Jeffreys interval. <s> BIB002
The Chebyshev inequality represents the generalization of the Wilson Score approach to the case of continuous bounded loss functions BIB001 : but, differently from Eq. BIB002 , this bound is rigorous, i.e. it holds with a coverage probability at least equal to (1 − δ). By applying Eq. (29) we get: Then, we can solve Eq. (33) with respect to L and obtain a rigorous and explicit upper bound for the generalization error. In particular, the unluckiness factor assumes the same formulation as in Eq.  (as expected, because the Chebyshev bound is a generalization of the Wilson Score interval to the case of continuous bounded soft loss functions) but, in this case, A and B are constants such that The Chebyshev inequality represents the generalization of the Wilson Score approach to the case of continuous bounded loss functions BIB001 : but, differently from Eq. BIB002 , this bound is rigorous, i.e. it holds with a coverage probability at least equal to (1 − δ). By applying Eq. (29) we get: Then, we can solve Eq. (33) with respect to L and obtain a rigorous and explicit upper bound for the generalization error. In particular, the unluckiness factor assumes the same formulation as in Eq. (20) (as expected, because the Chebyshev bound is a generalization of the Wilson Score interval to the case of continuous bounded soft loss functions) but, in this case, A and B are constants such that