Many of the last decade's greatest successes in computer vision have been driven by Convolutional Neural Networks (CNNs) [1]}, [2]}, [3]}. Despite difficulty theoretically justifying their ability to generalize beyond the training set, there is no doubt that massive datasets, coupled with efficient training using GPUs, have made it possible to fit discriminative and generative image models with historically unprecedented capabilities. A fundamental property of CNNs' is their locality bias (features at each layer are computed based on features from the previous layer within a small receptive field) and approximate translation equivariance (the parameters of each linear layer are shared across spatial positions, although boundary effects and commonly used pooling layers break exact translation equivariance [4]}). Their constrained structure leads CNNs to learn a hierarchy of feature extractors useful for minimizing an objective function (for example correctly classifying an input image). Typically this hierarchy of features entangles spatial extent and semantic complexity (early layers contain simple features with small spatial extent such as edges or corners, and deep layers contain complex features with large spatial extent such as face detectors), although there have been efforts to design architectures which break the entanglement of features' spatial extent and complexity [5]}.
<FIGURE>