The Intrinsic Manifolds of Radiological Images and their Role in Deep Learning
The manifold hypothesis is a core mechanism behind the success of deep
learning, so understanding the intrinsic manifold structure of image data is
central to studying how neural networks learn from the data. Intrinsic dataset
manifolds and their relationship to learning difficulty have recently begun to
be studied for the common domain of natural images, but little such research
has been attempted for radiological images. We address this here. First, we
compare the intrinsic manifold dimensionality of radiological and natural
images. We also investigate the relationship between intrinsic dimensionality
and generalization ability over a wide range of datasets. Our analysis shows
that natural image datasets generally have a higher number of intrinsic
dimensions than radiological images. However, the relationship between
generalization ability and intrinsic dimensionality is much stronger for
medical images, which could be explained as radiological images having
intrinsic features that are more difficult to learn. These results give a more
principled underpinning for the intuition that radiological images can be more
challenging to apply deep learning to than natural image datasets common to
machine learning research. We believe rather than directly applying models
developed for natural images to the radiological imaging domain, more care
should be taken to developing architectures and algorithms that are more
tailored to the specific characteristics of this domain. The research shown in
our paper, demonstrating these characteristics and the differences from natural
images, is an important first step in this direction.