The available resolution in our visual world is extremely high, if not infinite. Existing CNNs can be applied in a fully convolutional way to images of arbitrary resolution, but as the size of the input increases, they can not capture contextual information. In addition, computational requirements scale linearly to the number of input pixels, and resources are allocated uniformly across the input, no matter how informative different image regions are. We attempt to address these problems by proposing a novel architecture that traverses an image pyramid in a top-down fashion, while it uses a hard attention mechanism to selectively process only the most informative image parts. We conduct experiments on MNIST and ImageNet datasets, and we show that our models can significantly outperform fully convolutional counterparts, when the resolution of the input is that big that the receptive field of the baselines can not adequately cover the objects of interest. Gains in performance come for less FLOPs, because of the selective processing that we follow. Furthermore, our attention mechanism makes our predictions more interpretable, and creates a trade-off between accuracy and complexity that can be tuned both during training and testing time. Our visual world is very rich, and there is information of interest in an almost infinite number of different scales. As a result, we would like our models to be able to process images of arbitrary resolution, in order to capture visual information with arbitrary level of detail. This is possible with existing CNN architectures, since we can use fully convolutional processing (Long et al. (2015) ), coupled with global pooling. However, global pooling ignores the spatial configuration of feature maps, and the output essentially becomes a bag of features 1 . To demonstrate why this an important problem, in Figure 1 (a) and (b) we provide an example of a simple CNN that is processing an image in two different resolutions. In (a) we see that the receptive field of neurons from the second layer suffices to cover half of the kid's body, while in (b) the receptive field of the same neurons cover area that corresponds to the size of a foot. This shows that as the input size increases, the final representation becomes a bag of increasingly more local features, leading to the absence of coarselevel information, and potentially harming performance. We call this phenomenon the receptive field problem of fully convolutional processing. An additional problem is that computational resources are allocated uniformly to all image regions, no matter how important they are for the task at hand. For example, in Figure 1 (b), the same amount of computation is dedicated to process both the left half of the image that contains the kid, and the right half that is merely background. We also have to consider that computational complexity scales linearly with the number of input pixels, and as a result, the bigger the size of the input, the more resources are wasted on processing uninformative regions. We attempt to resolve the aforementioned problems by proposing a novel architecture that traverses an image pyramid in a top-down fashion, while it visits only the most informative regions along the way. The receptive field problem of fully convolutional processing. A simple CNN consisted of 2 convolutional layers (colored green), followed by a global pooling layer (colored red), processes an image in two different resolutions. The shaded regions indicate the receptive fields of neurons from different layers. As the resolution of the input increases, the final latent representation becomes a bag of increasingly more local features, lacking coarse information. (c) A sketch of our proposed architecture. The arrows on the left side of the image demonstrate how we focus on image sub-regions in our top-down traversal, while the arrows on the right show how we combine the extracted features in a bottom-up fashion. In Figure 1 (c) we provide a simplified sketch of our approach. We start at level 1, where we process the input image in low resolution, to get a coarse description of its content. The extracted features (red cube) are used to select out of a predefined grid, the image regions that are worth processing in higher resolution. This process constitutes a hard attention mechanism, and the arrows on the left side of the image show how we extend processing to 2 additional levels. All extracted features are combined together as denoted by the arrows on the right, to create the final image representation that is used for classification (blue cube). We evaluate our model on synthetic variations of MNIST (LeCun et al., 1998 ) and on ImageNet (Deng et al., 2009 ), while we compare it against fully convolutional baselines. We show that when the resolution of the input is that big, that the receptive field of the baseline 2 covers a relatively small portion of the object of interest, our network performs significantly better. We attribute this behavior to the ability of our model to capture both contextual and local information by extracting features from different pyramid levels, while the baselines suffer from the receptive field problem. Gains in accuracy are achieved for less floating point operations (FLOPs) compared to the baselines, due to the attention mechanism that we use. If we increase the number of attended image locations, computational requirements increase, but the probability of making a correct prediction is expected to increase as well. This is a trade-off between accuracy and computational complexity, that can be tuned during training through regularization, and during testing by stopping processing on early levels. Finally, by inspecting attended regions, we are able to get insights about the image parts that our networks value the most, and to interpret the causes of missclassifications. We proposed a novel architecture that is able to process images of arbitrary resolution without sacrificing spatial information, as it typically happens with fully convolutional processing. This is achieved by approaching feature extraction as a top-down image pyramid traversal, that combines information from multiple different scales. The employed attention mechanism allows us to adjust the computational requirements of our models, by changing the number of locations they attend. This way we can exploit the existing trade-off between computational complexity and accuracy. Furthermore, by inspecting the image regions that our models attend, we are able to get important insights about the causes of their decisions. Finally, there are multiple future research directions that we would like to explore. These include the improvement of the localization capabilities of our attention mechanism, and the application of our model to the problem of budgeted batch classification. In addition, we would like our feature extraction process to become more adaptive, by allowing already extracted features to affect the processing of image regions that are attended later on. Figure 8 we provide the parsing tree that our model implicitly creates.
We propose a novel architecture that traverses an image pyramid in a top-down fashion, while it visits only the most informative regions along the way.
{"src": "scitldr-AIC_train", "paper_id": "rkeH6AEYvr"}