Teaching dimension is a learning theoretic quantity that specifies the minimum training set size to teach a target model to a learner. Previous studies on teaching dimension focused on version-space learners which maintain all hypotheses consistent with the training data, and cannot be applied to modern machine learners which select a specific hypothesis via optimization. This paper presents the first known teaching dimension for ridge regression, support vector machines, and logistic regression. We also exhibit optimal training sets that match these teaching dimensions. Our approach generalizes to other linear learners.
Teaching dimension is distinct from VC dimension. For a finite hypothesis space @math , Goldman and Kearns @cite_1 proved the relation These inequalities are somewhat weak, as Goldman and Kearns had shown both cases where one quantity is much larger than the other. The distinction between TD and VC dimension is also present in our setting. For example, by inspecting the inhomogeneous SVM column in Table we note that TD does not depend on the dimensionality @math of the feature space @math . To see why this makes intuitive sense, note two @math -dimensional points are sufficient to specify any bisecting hyperplane in @math . On the other hand, recall that the VC dimension for inhomogeneous hyperplanes in @math is @math . Further quantification of the relation between TD and VC (and other capacity measures) remains an open research question.
@cite_1
2146250995 1999441752 2912171497 92982764
This paper is concerned with various combinatorial parameters of classes that can be learned from a small set of examples. We show that the recursive teaching dimension, recently introduced by (2008), is strongly connected to known complexity notions in machine learning, e.g., the self-directed learning complexity and the VC-dimension. To the best of our knowledge these are the first results unveiling such relations between teaching and query learning as well as between teaching and the VC-dimension. It will turn out that for many natural classes the RTD is upper-bounded by the VCD, e.g., classes of VC-dimension 1, intersection-closed classes and finite maximum classes. However, we will also show that there are certain (but rare) classes for which the recursive teaching dimension exceeds the VC-dimension. Moreover, for maximum classes, the combinatorial structure induced by the RTD, called teaching plan, is highly similar to the structure of sample compression schemes. Indeed one can transform any repetition-free teaching plan for a maximum class C into an unlabeled sample compression scheme for C and vice versa, while the latter is produced by (i) the corner-peeling algorithm of Rubinstein and Rubinstein (2012) and (ii) the tail matching algorithm of Kuzmin and Warmuth (2007). where de denotes normalized surface measure, V is the conformal gradient and q = (2n) (n 2). A modern folklore theorem is that by taking the infinitedimensional limit of this inequality, one obtains the Gross logarithmic Sobolev inequality for Gaussian measure, which determines Nelson's hypercontractive estimates for the Hermite semigroup (see [8]). One observes using conformal invariance that the above inequality is equivalent to the sharp Sobolev inequality on Rn for which boundedness and extremal functions can be easily calculated using dilation invariance and geometric symmetrization. The roots here go back to Hardy and Littlewood. The advantage of casting the problem on the sphere is that the role of the constants is evident, and one is led immediately to the conjecture that this inequality should hold whenever possible (for example, 2 < q < 0o if n = 2). This is in fact true and will be demonstrated in Section 2. A clear question at this point is "What is the situation in dimension 2?" Two important arguments ([25], [26], [27]) dealt with this issue, both motivated by geometric variational problems. Because q goes to infinity for dimension 2, the appropriate function space is the exponential class. Responding in part We study the problem of learning conjunctive concepts from examples on structural domains like the blocks world. This class of concepts is formally defined, and it is shown that even for samples in which each example (positive or negative) is a two-object scene, it is NP-complete to determine if there is any concept in this class that is consistent with the sample. We demonstrate how this result affects the feasibility of Mitchell's version of space approach and how it shows that it is unlikely that this class of concepts is polynomially learnable from random examples alone in the PAC framework of Valiant. On the other hand, we show that for any fixed bound on the number of objects per scene, this class is polynomially learnable if, in addition to providing random examples, we allow the learning algorithm to make subset queries. In establishing this result, we calculate the capacity of the hypothesis space of conjunctive concepts in a structural domain and use a general theorem of Vapnik and Chervonenkis. This latter result can also be used to estimate a sample size sufficient for heuristic learning techniques that do not use queries. This chapter presents a brief introduction to Vapnik-Chervonenkis (VC) dimension, a quantity which characterizes the difficulty of distribution-independent learning. The chapter establishes various elementary results, and discusses how to estimate the VC dimension in several examples of interest in neural network theory.