Capturing long-range feature relations has been a central issue on convolutional neural networks(CNNs). To tackle this, attempts to integrate end-to-end trainable attention module on CNNs are widespread. Main goal of these works is to adjust feature maps considering spatial-channel correlation inside a convolution layer. In this paper, we focus on modeling relationships among layers and propose a novel structure, 'Recurrent Layer Attention network,' which stores the hierarchy of features into recurrent neural networks(RNNs) that concurrently propagating with CNN and adaptively scales feature volumes of all layers. We further introduce several structural derivatives for demonstrating the compatibility on recent attention modules and the expandability of proposed network. For semantic understanding on learned features, we also visualize intermediate layers and plot the curve of layer scaling coefficients(i.e., layer attention). Recurrent Layer Attention network achieves significant performance enhancement requiring a slight increase on parameters in an image classification task with CIFAR and ImageNet-1K 2012 dataset and an object detection task with Microsoft COCO 2014 dataset. Concatenating all features in the order of layers in convolutional neural network (CNN) provides new interpretation, features form a sequence, consisting of features with small receptive fields to large receptive fields. Interestingly, recurrent neural network (RNN) is one of the representatives for modeling the sequential information. (Sutskever et al., 2014; Hochreiter & Schmidhuber, 1997) . On the other hands, Recent attempts to utilize attention mechanism for empowering CNN a better representational power are prevalent (Hu et al., 2018b; Wang et al., 2017) . Motivated by intrinsic characteristics of CNN and RNN, and recent attention works on computer vision, we present the Recurrent Layer Attention Network (RLA network), which is differentiable and light-weight while improving the representational power of CNN by a slightly different way from other attention works in computer vision. The main goal of our work is applying global weights balance among layers by inheriting the feature hierarchy from previous CNN layers. We accomplish the goal by two main structural designs: employing our inter-layer attention mechanism to make the network re-adjust features, and utilizing RNN to memorize the feature hierarchy with concurrently propagating parallel with CNN. We hypothesize that RLA network gains additional class discriminability through inheriting the informative feature hierarchy, such as repetitive appearances of important features or relevant features with different receptive fields. For example, our network raises the activation of neurons that is responsible to the whole body of zebra, using a history of features of relatively smaller receptive field. We demonstrate our hypothesis through the Grad-CAM visualization of intermediate features and corresponding layer attention value (i.e., the importance of layer.) We evaluate RLA network on image classification and object detection tasks using benchmark datasets: CIFAR, ImageNet-1K, and Microsoft COCO. On both tasks, RLA network gets comparable results with the state-of-the-arts, Squeeze-and-Excitation network (Hu et al., 2018b) , and superior results than original ResNet architecture. Moreover, we suggest the compatibility of the RLA network by introducing the expansion of RLA network combining our inter-layer attention mechanism toward recent attention works (Hu et al., 2018b) on ablation study. Incorporating RLA network to recent attention works (Call these networks as intra-layer attention networks), further variations of RLA network can be recognized as the generalized attention network considering correlations among both inside and outside a layer. We summarize our contributions as follows: • We propose two new concepts: the weight balancing of CNN features along layers (call it inter-layer attention), and the connection from shallow CNN layers to deep CNN layers through concurrently propagating RNN. • We demonstrate the effectiveness of proposed RLA network for an image classification task and an object detection task on benchmark datasets. RLA network achieves similar or superior results compared with leading deep network while requiring small model parameters increase. • Ablation studies investigate the effectiveness of two proposed concepts and show the compatibility towards existing intra-attention models, and the further expandability by tuning architectural designs. We show and discuss how RLA network is learned to interpret images by visualizing intermediate layers and plotting layer attention values. • RLA network is easy to implement requiring basic operations of modern deep learning frameworks. In this paper, we first propose inter-layer attention mechansim to enhance the representational power of CNN. We structurized our mechanism as 'Recurrent Layer Attention network' by utilizing two new concepts: the weight balancing of CNN features along layers and the link from shallow CNN layers to deep CNN layers via RNN for directly conveying the feature hierarchy. We introduce structural derivatives of RLA network: 'IA+RLA' for proving an applicability of our work toward recent intra-layer attention mechanism, and 'RLA-vector' for distilling the impacts of proposed two new concepts. We also precisely select statistics for the context by focusing local patterns preserved in feature summarization procedure. We evaluate RLA network using CIFAR and ImageNet-1k 2012 datasets for an image classification task, and also verify it's generalization ability toward an object detection task via experiments on Microsoft COCO dataset. For demonstrating our hypothesis that RLA network gains additional class discriminability and semantically understanding how RLA network induce their model parameters to be learned, we visualize RLA network utilizing Grad-CAM visualization, plot the layer attention value curve, and report several interesting findings. For future works, we are planning to integrate our inter-layer attention mechanism to intra-layer attention mechanism with heavier experiments first, and to utilize the concept of making any kinds of arbitrary connection from earlier layer to latter layer through RNN in other domains.
We propose a new type of end-to-end trainable attention module, which applies global weight balances among layers by utilizing co-propagating RNN with CNN.
{"src": "scitldr-AIC_train", "paper_id": "Bke5aJBKvH"}