Deep Learning and Its Applications to Machine Health Monitoring: A Survey <s> C. Convolutioanl Neural Network <s> We trained a large, deep convolutional neural network to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes. On the test data, we achieved top-1 and top-5 error rates of 37.5% and 17.0% which is considerably better than the previous state-of-the-art. The neural network, which has 60 million parameters and 650,000 neurons, consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax. To make training faster, we used non-saturating neurons and a very efficient GPU implementation of the convolution operation. To reduce overriding in the fully-connected layers we employed a recently-developed regularization method called "dropout" that proved to be very effective. We also entered a variant of this model in the ILSVRC-2012 competition and achieved a winning top-5 test error rate of 15.3%, compared to 26.2% achieved by the second-best entry. <s> BIB001 </s> Deep Learning and Its Applications to Machine Health Monitoring: A Survey <s> C. Convolutioanl Neural Network <s> Convolutional Neural Networks (CNN) have showed success in achieving translation invariance for many image processing tasks. The success is largely attributed to the use of local filtering and max-pooling in the CNN architecture. In this paper, we propose to apply CNN to speech recognition within the framework of hybrid NN-HMM model. We propose to use local filtering and max-pooling in frequency domain to normalize speaker variance to achieve higher multi-speaker speech recognition performance. In our method, a pair of local filtering layer and max-pooling layer is added at the lowest end of neural network (NN) to normalize spectral variations of speech signals. In our experiments, the proposed CNN architecture is evaluated in a speaker independent speech recognition task using the standard TIMIT data sets. Experimental results show that the proposed CNN method can achieve over 10% relative error reduction in the core TIMIT test sets when comparing with a regular NN using the same number of hidden layers and weights. Our results also show that the best result of the proposed CNN model is better than previously published results on the same TIMIT test sets that use a pre-trained deep NN model. <s> BIB002
Convolutional neural networks (CNNs) were firstly proposed by LeCun  for image processing, which is featured by two key properties: spatially shared weights and spatial pooling. CNN models have shown their success in various computer vision applications  ,  , BIB001 where input data are usually 2D data. CNN has also been introduced to address sequential data including Natural Language Processing and Speech Recognition BIB002 ,  . CNN aims to learn abstract features by alternating and stacking convolutional kernels and pooling operation. In CNN, the convolutional layers (convolutional kernels) convolve multiple local filters with raw input data and generate invariant local features and the subsequent pooling layers extract most significant features with a fixed-length over sliding windows of the raw input data. Considering 2D-CNN have been illustrated extensively in previous research compared to 1D-CNN, here, only the mathematical details behind 1D-CNN is given as follows: Firstly, we assume that the input sequential data is x = [x 1 , . . . , x T ] that T is the length of the sequence and x i ∈ R d at each time step. Convolution: the dot product between a filter vector u ∈ R md and an concatenation vector representation x i:i+m−1 defines the convolution operation as follows: where * T represents the transpose of a matrix * , b and ϕ denotes bias term and non-linear activation function, respectively. x i:i+m−1 is a m-length window starting from the i-th time step, which is described as: As defined in Eq. (10), the output scale c i can be regarded as the activation of the filter u on the corresponding subsequence x i:i+m−1 . By sliding the filtering window from the beginning time step to the ending time step, a feature map as a vector can be given as follows: where the index j represents the j-th filter. It corresponds to multi-windows as {x 1:m , x 2:m+1 , . . . , x l−m+1:l }. Max-pooling: Pooling layer is able to reduce the length of the feature map, which can further minimize the number of model parameters. The hyper-parameter of pooling layer is pooling length denoted as s. MAX operation is taking a max over the s consecutive values in feature map c j . Then, the compressed feature vector can be obtained as: where h j = max(c (j−1)s , c (j−1)s+1 , . . . , c js−1 ). Then, via alternating the above two layers: convolution and max-pooling ones, fully connected layers and a softmax layer are usually added as the top layers to make predictions. To give a clear illustration, the framework for a one-layer CNN has been displayed in Figure. 4.