Recent studies in attention modules have enabled higher performance in computer vision tasks by capturing global contexts and accordingly attending important features. In this paper, we propose a simple and highly parametrically efficient module named Tree-structured Attention Module (TAM) which recursively encourages neighboring channels to collaborate in order to produce a spatial attention map as an output. Unlike other attention modules which try to capture long-range dependencies at each channel, our module focuses on imposing non-linearities be- tween channels by utilizing point-wise group convolution. This module not only strengthens representational power of a model but also acts as a gate which controls signal flow. Our module allows a model to achieve higher performance in a highly parameter-efficient manner. We empirically validate the effectiveness of our module with extensive experiments on CIFAR-10/100 and SVHN datasets. With our proposed attention module employed, ResNet50 and ResNet101 models gain 2.3% and 1.2% accuracy improvement with less than 1.5% parameter over- head. Our PyTorch implementation code is publicly available. Advancements in attention modules have boosted up the performance where they are employed over broad fields in deep learning such as machine translation, image generation, image and video classification, object detection, segmentation, etc (Vaswani et al., 2017; Hu et al., 2018a; b; c; Wang et al., 2018; Cao et al., 2019; Zhang et al., 2019) . In the fields of computer vision tasks, numerous attention modules have been proposed in a way that one can attach it to a backbone network obtaining an efficient trade-off between additional parameters of the attached attention module and the model's performance. SENet (Hu et al., 2018b) encodes global spatial information using global average pooling and captures channel-wise dependencies using two fully-connected layers over the previously encoded values at each channel. Input feature maps of the SE module are recalibrated with output values corresponding to each channel after applying a sigmoid activation function to produce output feature maps of the module. In this manner, the model can distinguish which channels to attend than others. GENet (Hu et al., 2018a) shows simply gathering spatial information with depth-wise strided convolution and redistributing each gathered value across all positions with nearest neighbor upsampling can significantly help a network to understand global feature context. NLNet (Wang et al., 2018) aggregates query-specific global context and adds values to each corresponding channel. GCNet (Cao et al., 2019) simplifies NLNet in a computationally efficient way using the fact that a non-local block used in the NLNet tends to produce attention map independent of query position. BAM efficiently enhances backbone networks by placing attention modules in bottleneck regions, which requires few increase in both parameters and computation. CBAM incorporates channel and spatial attentions and employs a max descriptor as well as an average descriptor for more precise attention. It is clear that proposed modules in aforementioned studies have brought remarkable results, most of their main focus has been on how to capture long-range dependencies across spatial dimension. That is, they mainly focus on contextual modeling rather than capturing inter-channel relations both of which are regarded indispensable for an attention module as depicted in Cao et al. (2019) . In this work, we propose a module which strengthens model representational power by imposing nonlinearities between neighboring channels in a parameter efficient manner. While this work deviates Figure 1 : An instance of our proposed module with group size 2. f p denotes a point-wise convolution followed by an activation function which combines neighboring channels. C m n denotes a n-th channel after applying m point-wise group convolutions to the input feature map. One channel attention map followed by a sigmoid σ is produced. A color refers to information a channel contains. The repetition of point-wise group convolution yields a tree-like structure. from the current trend of capturing long-range dependencies within spatial dimension, we argue that taking consideration of inter-channel relations can also achieve highly competitive results even without capturing any kind of spatial dependencies. Our module incorporates all channels to produce a single meaningful attention map as an output whereas most previous studies restore the input channel dimension in order to attend important channels and to suppress less meaningful ones. For this, we repeatedly apply light-weight point-wise group convolution with a fixed group size to an input feature map until the number of channels becomes one. While the increased parameters and computation are almost negligible, we find this simple design remarkably boosts up the performance of various backbone networks. As we see in section 3, the module performance is highly competitive to other attention modules and enhances baseline models with few additional parameter overhead. This gives one a clue to another notion for attention deviating from the current trend of taking global context. Our contributions are two-fold: • we propose Tree-structured Attention Module (TAM) which allows the network to learn inter-channel relationships using light-weight point-wise group convolutions. This treestructure enables convolution filters in the mid and later phase of a network to have a higher variance so that it can have more presentation power. • by proving validity of TAM with extensive experiments, we highlight the potential importance of inter-channel relations. In this paper, we propose Tree-structure Attention module which enables a network to learn interchannel relationships which deviates from the current trend of capturing long-range dependencies in attention literature. TAM adopts light-weight point-wise group convolutions to allow communication between neighboring channels. Once trained, TAM acts as a static gate controlling signal at a certain location which does not depend on input feature but on the location where it is placed. Moreover, TAM permits higher variances in filter weights in the early and mid phase and helps the filters to focus on important ones at the last phase before classifier. On top of that, TAM produces favorable performance gains with only a few additional parameters to a backbone network. These advantages of TAM shed a light on a new way to attend features.
Our paper proposes an attention module which captures inter-channel relationships and offers large performance gains.
{"src": "scitldr-AIC_train", "paper_id": "r1xBoxBYDH"}