LIDL: Local Intrinsic Dimension Estimation Using Approximate Likelihood
Most of the existing methods for estimating the local intrinsic dimension of
a data distribution do not scale well to high-dimensional data. Many of them
rely on a non-parametric nearest neighbors approach which suffers from the
curse of dimensionality. We attempt to address that challenge by proposing a
novel approach to the problem: Local Intrinsic Dimension estimation using
approximate Likelihood (LIDL). Our method relies on an arbitrary density
estimation method as its subroutine and hence tries to sidestep the
dimensionality challenge by making use of the recent progress in parametric
neural methods for likelihood estimation. We carefully investigate the
empirical properties of the proposed method, compare them with our theoretical
predictions, and show that LIDL yields competitive results on the standard
benchmarks for this problem and that it scales to thousands of dimensions. What
is more, we anticipate this approach to improve further with the continuing
advances in the density estimation literature.