Convolutional Neural Networks continuously advance the progress of 2D and 3D image and object classification. The steadfast usage of this algorithm requires constant evaluation and upgrading of foundational concepts to maintain progress. Network regularization techniques typically focus on convolutional layer operations, while leaving pooling layer operations without suitable options. We introduce Wavelet Pooling as another alternative to traditional neighborhood pooling. This method decomposes features into a second level decomposition, and discards the first-level subbands to reduce feature dimensions. This method addresses the overfitting problem encountered by max pooling, while reducing features in a more structurally compact manner than pooling via neighborhood regions. Experimental results on four benchmark classification datasets demonstrate our proposed method outperforms or performs comparatively with methods like max, mean, mixed, and stochastic pooling. Convolutional Neural Networks (CNNs) have become the standard-bearer in image and object classification BID18 . Due to the layer structures conforming to the shape of the inputs, CNNs consistently classify images, objects, videos, etc. at a higher accuracy rate than vector-based deep learning techniques BID18 . The strength of this algorithm motivates researchers to constantly evaluate and upgrade foundational concepts to continue growth and progress. The key components of CNN, the convolutional layer and pooling layer, consistently undergo modifications and innovations to elevate accuracy and efficiency of CNNs beyond previous benchmarks.Pooling has roots in predecessors to CNN such as Neocognitron, which manual subsampling by the user occurs BID5 , and Cresceptron, which introduces the first max pooling operation in deep learning BID28 . Pooling subsamples the results of the convolutional layers, gradually reducing spatial dimensions of the data throughout the network. The benefits of this operation are to reduce parameters, increase computational efficiency, and regulate overfitting BID1 .Methods of pooling vary, with the most popular form being max pooling, and secondarily, average pooling BID18 BID13 . These forms of pooling are deterministic, efficient, and simple, but have weaknesses hindering the potential for optimal network learning BID13 BID30 . Other pooling operations, notably mixed pooling and stochastic pooling, use probabilistic approaches to correct some of the issues of the prior methods BID30 BID31 .However, one commonality all these pooling operations employ a neighborhood approach to subsampling, reminiscent of nearest neighbor interpolation in image processing. Neighborhood interpolation techniques perform fast, with simplicity and efficiency, but introduce artifacts such as edge halos, blurring, and aliasing BID20 . Minimizing discontinuities in the data are critical to aiding in network regularization, and increasing classification accuracy.We propose a wavelet pooling algorithm that uses a second-level wavelet decomposition to subsample features. Our approach forgoes the nearest neighbor interpolation method in favor of an organic, subband method that more accurately represents the feature contents with less artifacts. We compare our proposed pooling method to max, mean, mixed, and stochastic pooling to verify its validity, and ability to produce near equal or superior results. We test these methods on benchmark image classification datasets such as Mixed National Institute of Standards and Technology (MNIST) BID12 , Canadian Institute for Advanced Research (CIFAR-10) BID11 , Street House View Numbers (SHVN) BID17 , and Karolinska Directed Emotional Faces (KDEF) (Lundqvist et al., 1998) . We perform all simulations in MATLAB R2016b.The rest of this paper organizes as follows: Section 2 gives the background, Section 3 describes the proposed methods, Section 4 discusses the experimental results, and Section 5 gives the summary and conclusion. All CNN experiments use MatConvNet BID26 . All training uses stochastic gradient descent BID0 . For our proposed method, the wavelet basis is the Haar wavelet, mainly for its even, square subbands. All experiments are run on a 64-bit operating system, with an Intel Core i7-6800k CPU @ 3.40 GHz processor, with 64.0 GB of RAM. We utilize two GeForce Titan X Pascal GPUs with 12 GB of video memory for all training. All CNN structures except for MNIST use a network loosely based on Zeilers network BID31 . We repeat the experiments with Dropout BID22 and replace Local Response Normalization BID11 with Batch Normalization BID7 for CIFAR-10 and SHVN (Dropout only) to examine how these regularization techniques change the pooling results. To test the effectiveness of each pooling method on each dataset, we solely pool with that method for all pooling layers in that network. All pooling methods use a 2x2 window for an even comparison to the proposed method. Figure 6 gives a selection of each of the datasets. We prove wavelet pooling has potential to equal or eclipse some of the traditional methods currently utilized in CNNs. Our proposed method outperforms all others in the MNIST dataset, outperforms all but one in the CIFAR-10 and KDEF datasets, and performs within respectable ranges of the pooling methods that outdo it in the SHVN dataset. The addition of dropout and batch normalization show our proposed methods response to network regularization. Like the non-dropout cases, it outperforms all but one in both the CIFAR-10 & KDEF datasets, and performs within respectable ranges of the pooling methods that outdo it in the SHVN dataset. Our results confirm previous studies proving that no one pooling method is superior, but some perform better than others depending on the dataset and network structure BID1 ; BID13 . Furthermore, many networks alternate between different pooling methods to maximize the effectiveness of each method.Future work and improvements in this area could be to vary the wavelet basis to explore which basis performs best for the pooling. Altering the upsampling and downsampling factors in the decomposition and reconstruction can lead to better image feature reductions outside of the 2x2 scale.Retention of the subbands we discard for the backpropagation could lead to higher accuracies and fewer errors. Improving the method of FTW we use could greatly increase computational efficiency. Finally, analyzing the structural similarity (SSIM) of wavelet pooling versus other methods could further prove the vitality of using our approach.
Pooling is achieved using wavelets instead of traditional neighborhood approaches (max, average, etc).
{"src": "scitldr-AIC_train", "paper_id": "rkhlb8lCZ"}