Consistent Interpolating Ensembles via the Manifold-Hilbert Kernel
Recent research in the theory of overparametrized learning has sought to
establish generalization guarantees in the interpolating regime. Such results
have been established for a few common classes of methods, but so far not for
ensemble methods. We devise an ensemble classification method that
simultaneously interpolates the training data, and is consistent for a broad
class of data distributions. To this end, we define the manifold-Hilbert kernel
for data distributed on a Riemannian manifold. We prove that kernel smoothing
regression using the manifold-Hilbert kernel is weakly consistent in the
setting of Devroye et al. 1998. For the sphere, we show that the
manifold-Hilbert kernel can be realized as a weighted random partition kernel,
which arises as an infinite ensemble of partition-based classifiers.