Deep Learning and Its Applications to Machine Health Monitoring: A Survey <s> A. Auto-encoders (AE) and its variants <s> Complexity theory of circuits strongly suggests that deep architectures can be much more efficient (sometimes exponentially) than shallow architectures, in terms of computational elements required to represent some functions. Deep multi-layer neural networks have many levels of non-linearities allowing them to compactly represent highly non-linear and highly-varying functions. However, until recently it was not clear how to train such deep networks, since gradient-based optimization starting from random initialization appears to often get stuck in poor solutions. Hinton et al. recently introduced a greedy layer-wise unsupervised learning algorithm for Deep Belief Networks (DBN), a generative model with many layers of hidden causal variables. In the context of the above optimization problem, we study this algorithm empirically and explore variants to better understand its success and extend it to cases where the inputs are continuous or where the structure of the input distribution is not revealing enough about the variable to be predicted in a supervised task. Our experiments also confirm the hypothesis that the greedy layer-wise unsupervised training strategy mostly helps the optimization, by initializing weights in a region near a good local minimum, giving rise to internal distributed representations that are high-level abstractions of the input, bringing better generalization. <s> BIB001 </s> Deep Learning and Its Applications to Machine Health Monitoring: A Survey <s> A. Auto-encoders (AE) and its variants <s> Previous work has shown that the difficulties in learning deep generative or discriminative models can be overcome by an initial unsupervised learning step that maps inputs to useful intermediate representations. We introduce and motivate a new training principle for unsupervised learning of a representation based on the idea of making the learned representations robust to partial corruption of the input pattern. This approach can be used to train autoencoders, and these denoising autoencoders can be stacked to initialize deep architectures. The algorithm can be motivated from a manifold learning and information theoretic perspective or from a generative model perspective. Comparative experiments clearly show the surprising advantage of corrupting the input of autoencoders on a pattern classification benchmark suite. <s> BIB002
As a feed-forward neural network, auto-encoder consists of two phases including encoder and decoder. Encoder takes an input x and transforms it to a hidden representation h via a non-linear mapping as follows: where ϕ is a non-linear activation function. Then, decoder maps the hidden representation back to the original representation in a similar way as follows: Model parameters including θ = [W, b, W , b ] are optimized to minimize the reconstruction error between z = f θ (x) and x. One commonly adopted measure for the average reconstruction error over a collection of N data samples is squared error and the corresponding optimization problem can be written as follows: where x i is the i-th sample. It is clearly shown that AE can be trained in an unsupervised way. And the hidden representation h can be regarded as a more abstract and meaningful representation for data sample x. Usually, the hidden size should be set to be larger than the input size in AE, which is verified empirically BIB001 . Addition of Sparsity: To prevent the learned transformation is the identity one and regularize auto-encoders, the sparsity constraint is imposed on the hidden units  . The corrsponding optimization function is updated as: where m is the hidden layer size and the second term is the summation of the KL-divergence over the hidden units. The KL-divergence on j-th hidden neuron is given as: where p is the predefined mean activation target and p j is the average activation of the j-th hidden neuron over the whole datasets. Given a small p, the addition of sparsity constraint can lead the learned hidden representation to be a sparse representation. Therefore, the variant of AE is named sparse auto-encoder. Addition of Denoising: Different from conventional AE, denoising AE takes a corrupted version of data as input and is trained to reconstruct/denoise the clean input x from its corrupted samplex. The most common adopted noise is dropout noise/binary masking noise, which randomly sets a fraction of the input features to be zero BIB002 . The variant of AE is denoising auto-encoder (DA), which can learn more robust representation and prevent it from learning the identity transformation. Stacking Structure: Several DA can be stacked together to form a deep network and learn high-level representations by feeding the outputs of the l-th layer as inputs to the (l + 1)-th layer BIB002 . And the training is done one layer greedily at a time. Since auto-encoder can be trained in an unsupervised way, auto-encoder, especially stacked denoising auto-encoder (SDA), can provide an effective pre-training solution via initializing the weights of deep neural network (DNN) to train the model. After layer-wise pre-training of SDA, the parameters of auto-encoders can be set to the initialization for all the hidden layers of DNN. And then, the supervised finetuning is performed to minimize prediction error on a labeled training data. Usually, a softmax/regression layer is added on top of the network to map the output of the last layer in AE to targets. The whole process is shown in Figure 2 . The pretraining protocol based on SDA can make DNN models have better convergence capability compared to arbitrary random initialization.