Deep Learning and Its Applications to Machine Health Monitoring: A Survey <s> D. Recurrent Neural Network <s> We show that a recurrent, second-order neural network using a real-time, forward training algorithm readily learns to infer small regular grammars from positive and negative string training samples. We present simulations that show the effect of initial conditions, training set size and order, and neural network architecture. All simulations were performed with random initial weight strengths and usually converge after approximately a hundred epochs of training. We discuss a quantization algorithm for dynamically extracting finite state automata during and after training. For a well-trained neural net, the extracted automata constitute an equivalence class of state machines that are reducible to the minimal machine of the inferred grammar. We then show through simulations that many of the neural net state machines are dynamically stable, that is, they correctly classify many long unseen strings. In addition, some of these extracted automata actually outperform the trained neural network for classification... <s> BIB001 </s> Deep Learning and Its Applications to Machine Health Monitoring: A Survey <s> D. Recurrent Neural Network <s> Learning to store information over extended time intervals by recurrent backpropagation takes a very long time, mostly because of insufficient, decaying error backflow. We briefly review Hochreiter's (1991) analysis of this problem, then address it by introducing a novel, efficient, gradient based method called long short-term memory (LSTM). Truncating the gradient where this does not do harm, LSTM can learn to bridge minimal time lags in excess of 1000 discrete-time steps by enforcing constant error flow through constant error carousels within special units. Multiplicative gate units learn to open and close access to the constant error flow. LSTM is local in space and time; its computational complexity per time step and weight is O. 1. Our experiments with artificial data involve local, distributed, real-valued, and noisy pattern representations. In comparisons with real-time recurrent learning, back propagation through time, recurrent cascade correlation, Elman nets, and neural sequence chunking, LSTM leads to many more successful runs, and learns much faster. LSTM also solves complex, artificial long-time-lag tasks that have never been solved by previous recurrent network algorithms. <s> BIB002 </s> Deep Learning and Its Applications to Machine Health Monitoring: A Survey <s> D. Recurrent Neural Network <s> Long short-term memory (LSTM) can solve many tasks not solvable by previous learning algorithms for recurrent neural networks (RNNs). We identify a weakness of LSTM networks processing continual input streams without explicitly marked sequence ends. Without resets, the internal state values may grow indefinitely and eventually cause the network to break down. Our remedy is an adaptive "forget gate" that enables an LSTM cell to learn to reset itself at appropriate times, thus releasing internal resources. We review an illustrative benchmark problem on which standard LSTM outperforms other RNN algorithms. All algorithms (including LSTM) fail to solve a continual version of that problem. LSTM with forget gates, however, easily solves it in an elegant way. <s> BIB003 </s> Deep Learning and Its Applications to Machine Health Monitoring: A Survey <s> D. Recurrent Neural Network <s> The temporal distance between events conveys information essential for numerous sequential tasks such as motor control and rhythm detection. While Hidden Markov Models tend to ignore this information, recurrent neural networks (RNNs) can in principle learn to make use of it. We focus on Long Short-Term Memory (LSTM) because it has been shown to outperform other RNNs on tasks involving long time lags. We find that LSTM augmented by "peephole connections" from its internal cells to its multiplicative gates can learn the fine distinction between sequences of spikes spaced either 50 or 49 time steps apart without the help of any short training exemplars. Without external resets or teacher forcing, our LSTM variant also learns to generate stable streams of precisely timed spikes and other highly nonlinear periodic patterns. This makes LSTM a promising approach for tasks that require the accurate measurement or generation of time intervals. <s> BIB004 </s> Deep Learning and Its Applications to Machine Health Monitoring: A Survey <s> D. Recurrent Neural Network <s> In recent years, deep artificial neural networks (including recurrent ones) have won numerous contests in pattern recognition and machine learning. This historical survey compactly summarizes relevant work, much of it from the previous millennium. Shallow and Deep Learners are distinguished by the depth of their credit assignment paths, which are chains of possibly learnable, causal links between actions and effects. I review deep supervised learning (also recapitulating the history of backpropagation), unsupervised learning, reinforcement learning & evolutionary computation, and indirect search for short programs encoding deep and large networks. <s> BIB005 </s> Deep Learning and Its Applications to Machine Health Monitoring: A Survey <s> D. Recurrent Neural Network <s> In this paper, we propose a novel neural network model called RNN Encoder‐ Decoder that consists of two recurrent neural networks (RNN). One RNN encodes a sequence of symbols into a fixedlength vector representation, and the other decodes the representation into another sequence of symbols. The encoder and decoder of the proposed model are jointly trained to maximize the conditional probability of a target sequence given a source sequence. The performance of a statistical machine translation system is empirically found to improve by using the conditional probabilities of phrase pairs computed by the RNN Encoder‐Decoder as an additional feature in the existing log-linear model. Qualitatively, we show that the proposed model learns a semantically and syntactically meaningful representation of linguistic phrases. <s> BIB006 </s> Deep Learning and Its Applications to Machine Health Monitoring: A Survey <s> D. Recurrent Neural Network <s> In this paper we compare different types of recurrent units in recurrent neural networks (RNNs). Especially, we focus on more sophisticated units that implement a gating mechanism, such as a long short-term memory (LSTM) unit and a recently proposed gated recurrent unit (GRU). We evaluate these recurrent units on the tasks of polyphonic music modeling and speech signal modeling. Our experiments revealed that these advanced recurrent units are indeed better than more traditional recurrent units such as tanh units. Also, we found GRU to be comparable to LSTM. <s> BIB007
As stated in BIB005 , recurrent neural networks (RNN) are the deepest of all neural networks, which can generate and address memories of arbitrary-length sequences of input patterns. RNN is able to build connections between units from a directed cycle. Different from basic neural network: multi-layer perceptron that can only map from input data to target vectors, RNN is able to map from the entire history of previous inputs to target vectors in principal and allows a memory of previous inputs to be kept in the network's internal state. RNNs can be trained via backpropagation through time for supervised tasks with sequential input data and target outputs  ,  , BIB001 . RNN can address the sequential data using its internal memory, as shown in Figure 5 . (a). The transition function defined in each time step t takes the current time information x t and the previous hidden output h t−1 and updates the current hidden output as follows: where H defines a nonlinear and differentiable transformation function. After processing the whole sequence, the hidden output at the last time step i.e. h T is the learned representation of the input sequential data whose length is T . A conventional Multilayer perceptron (MLP) is added on top to map the obtained representation h T to targets. Various transition functions can lead to various RNN models. The most simple one is vanilla RNN that is given as follows: where W and H denote transformation matrices and b is the bias vector. And ϕ denote the nonlinear activation function such as sigmoid and tanh functions. Due to the vanishing gradient problem during backpropagation for model training, vanilla RNN may not capture long-term dependencies. Therefore, Long-short term memory (LSTM) and gated recurrent neural networks (GRU) were presented to prevent backpropagated errors from vanishing or exploding BIB002 , BIB003 , BIB004 , BIB006 , BIB007 . The core idea behind these advanced RNN variants is that gates are introduced to avoid the long-term dependency problem and enable each recurrent unit to adaptively capture dependencies of different time scales. Besides these proposed advanced transition functions such as LSTMs and GRUs, multi-layer and bi-directional recurrent structure can increase the model capacity and flexibility. As shown in Figure 5 .(b), multi-layer structure can enable the hidden output of one recurrent layer to be propagated through time and used as the input data to the next recurrent layer. And the bidirectional recurrent structure is able to process the sequence data in two directions including forward and backward ways with two separate hidden layers, which is illustrated in Figure 5.(c) . The following equations define the corresponding hidden layer function and the → and ← denote forward and backward process, respectively. Then, the final vector h T is the concatenated vector of the outputs of forward and backward processes as follows: Convolutional Layer . Illustrations for one-layer CNN that contains one convolutional layer, one pooling layer, one fully-connected layer, and one softmax layer. 