Very early work on unsupervised learning by [1]} shows that "convolutionally extracted" features achieve good performance on unsupervised tasks because of the similarity between the extracted tiles/features making them easier to cluster using k-means. This correlates well with our theory that the predicted labels from convolutionally extracted features on small tiles gives the CNNs their advantage. However no theoretical models were proposed and this work was before CNNs were introduced.
