We introduce the concept of channel aggregation in ConvNet architecture, a novel compact representation of CNN features useful for explicitly modeling the nonlinear channels encoding especially when the new unit is embedded inside of deep architectures for action recognition. The channel aggregation is based on multiple-channels features of ConvNet and aims to be at the spot finding the optical convergence path at fast speed. We name our proposed convolutional architecture “nonlinear channels aggregation networks (NCAN)” and its new layer “nonlinear channels aggregation layer (NCAL)”. We theoretically motivate channels aggregation functions and empirically study their effect on convergence speed and classification accuracy. Another contribution in this work is an efficient and effective implementation of the NCAL, speeding it up orders of magnitude. We evaluate its performance on standard benchmarks UCF101 and HMDB51, and experimental results demonstrate that this formulation not only obtains a fast convergence but stronger generalization capability without sacrificing performance. With modern learnable representations such as deep convolutional neural networks (CNNs) matured in many image understanding tasks BID7 , human action recognition has received a significant amount of attentions BID11 BID2 BID9 BID3 BID12 . Due to the fact that video itself provides an additional temporal clue and that the parameters and the calculations of CNNs grow exponentially, training CNNs with such large-scale parameters in video domain is time-consuming. However, it remains unclear how the effective convergence accelerators could be conducted for the optimal path by formulizing the handcrafted rules. Since videos consist of still images, training tricks and methods, such as Relu, BN, have been shown to transfer to videos directly. Recent theoretical and empirical works have demonstrated the importance of quickly training deep architectures successfully, and the effective convergence accelerators advanced in the 2D image, such as relu BID4 and batch normalization BID5 , have been developed for fast convergence. This is in part inspired by observations of the limited GPU memory and computing power, especially when confronting the large-scale video dataset which may introduce a large majority of parameters. Another pipeline of algorithms focuses on the training optimizer of CNNs, for examples, sgd, momentum, nesterov, adagrad and adadelta. However, training CNNs utilizing the large-scale video datasets is still nontrivial in video task, particularly if one seeks a compact but fast long termporal dynamic representation that can be processed efficiently.Our current work reconsiders the means of facilitating convergence of ConvNets to increase the understanding of how to embed some hand-crafted rules inside of CNNs for fast convergence in a more thorough fashion. In addition to the accelerators and effective optimizers, we tend to explore a thorough method causing the value of the loss function to descend rapidly. Intuitively, we argue that CNNs will accelerate training process once the complex relationship across convolutional features channels is modeled, explicitly, by the hand-crafted rules. In the existing units 3D convolution implements a linear partial sum of channels BID6 , 3D max-pooling takes the maximum feature by channels and 3D average-pooling make a spatial-channel average of features. Unfortunately, all the 3D units conduct a linear channels aggregation, implicitly and locally. Despite that the implicit linear aggregation has been applied to broad fields, there seems to be less works explicitly taking modeling the complex nonlinear relationship across channels into account. In fact, either one-stream or two-stream algorithms ignore the channel-level encoding. For video recognition task, a very tricky problem is how to train the CNN architectures for the sake of making a lower loss rapidly in the scarcity of videos. We conjecture that there is complex nonlinear relationship among the channels of CNN features. Once this implicit relationship is explicitly modeled, such accomplishment will facilitate converging with faster search to the optimal trajectory.In this paper, we proposed a nonlinear channels aggregation layer (NCAL), which explicitly models the complex nonlinear relationships across channels. Since a standard CNN provides a whole hierarchy of video representations, the first question worthy exploring is where the NACL should take place. For example, we can aggregate the output of the fully-connected layers of CNN architecture pre-trained on videos. A drawback of such implementation is that the convolutional features channels of CNN itself are still implicitly encoded and are unaware of the lower level channels relationship. The alternative is to model the nonlinear channels aggregation of some intermediate network layer. In this case, the lower layers fail to extract the representative features from video sequences, but the upper layers can reason about the overall dynamics in the video. The former is prone to sacrificing the recognition performance while the latter is thus thought of as the appropriate convolutional features for the compact aggregation. Here we build our methods on top of the successful Inception V1 architecture. More specifically, three main contributions are provided in this work. Our first contribution is to introduce the concept of nonlinear channels aggregation for fast convergence. We also show that, in this manner, it is possible to apply the concept of nonlinear channels aggregation to the intermediate layers of a CNN representation by constructing an efficient nonlinear channels aggregation layer (NCAL).Here we build our methods on top of the successful Inception V1 architecture. More specifically, three main contributions are provided in this work. Our first contribution is to introduce the concept of nonlinear channels aggregation for fast convergence. We also show that, in this manner, it is possible to construct an efficient nonlinear channels aggregation by applying the concept of nonlinear channels aggregation to the intermediate layers of the standard CNN. More importantly, it is explicitly and globally that the nonlinear channels relationship is modeled compared to the traditional local and implicit units.Our second contribution is to simplify the process of nonlinear channels aggregation layer (NCAL) and make a fast yet accurate implementation of it. Notably , the proposed NCAL can be embodied inside of any standard CNN architectures, and not break the rest components of structures. More broadly , the proposed NCAL is not limited to action recognition, that is, it can be applied to any task with CNNs. Here we introduce it into action recognition, and leave the explorations of it on the other domains in the future.Our third contribution is to leverage these ideas to construct a novel nonlinear channels aggregation network, perform the training process end-to-end. We show that such nonlinear channels encoding results in a fast decline in the value of the loss function of CNNs while obtains efficient and accurate classification of actions in videos.The rest of the paper is organized as follows: Section 2 describes the related works, and section 3 represents the principle of the nonlinear channels aggregation networks (NCAN) and the backward propagation of NCAN. This is followed by the experiments in section 4. Finally, we conclude this paper in Section 6. We present nonlinear channels aggregation, a powerful and new, yet simple concept in the context of deep learning that captures the global channels relationship. We introduce a novel nonlinear channels aggregation layer (NCAL) and make a fast yet accurate implementation of NCAL, which allows us to embed the principle of complex channels encoding to the mainstream CNN architectures and back-propagate the gradients through NCALs. Experiments on video sequences demonstrate the effective power of nonlinear channels aggregation on facilitating training CNNs.In this paper we fit the complex channels relationships by capturing the global channels aggregation. Still, there seems to be some possible research directions that can be further expanded, modeling the nonlinear functions across channels. In the future it is beneficial to explore multiple-scale channel-levels by pyramid coding across channels. In sublimation, we can embed any hand-crafted rules, channels aggregation in the mainstream architectures, to making CNN working as we expect.
An architecture enables CNN trained on the video sequences converging rapidly 
{"src": "scitldr-AIC_train", "paper_id": "rJgdHs05FQ"}