Time Series Data Cleaning: A Survey <s> E. LONG SHORT-TERM MEMORY <s> Long short-term memory (LSTM) can solve many tasks not solvable by previous learning algorithms for recurrent neural networks (RNNs). We identify a weakness of LSTM networks processing continual input streams without explicitly marked sequence ends. Without resets, the internal state values may grow indefinitely and eventually cause the network to break down. Our remedy is an adaptive "forget gate" that enables an LSTM cell to learn to reset itself at appropriate times, thus releasing internal resources. We review an illustrative benchmark problem on which standard LSTM outperforms other RNN algorithms. All algorithms (including LSTM) fail to solve a continual version of that problem. LSTM with forget gates, however, easily solves it in an elegant way. <s> BIB001 </s> Time Series Data Cleaning: A Survey <s> E. LONG SHORT-TERM MEMORY <s> This book will be most useful to applied mathematicians, communication engineers, signal processors, statisticians, and time series researchers, both applied and theoretical. Readers should have some background in complex function theory and matrix algebra and should have successfully completed the equivalent of an upper division course in statistics. <s> BIB002 </s> Time Series Data Cleaning: A Survey <s> E. LONG SHORT-TERM MEMORY <s> We adopted an approach based on an LSTM neural network to monitor and detect faults in industrial multivariate time series data. To validate the approach we created a Modelica model of part of a real gasoil plant. By introducing hacks into the logic of the Modelica model, we were able to generate both the roots and causes of fault behavior in the plant. Having a self-consistent data set with labeled faults, we used an LSTM architecture with a forecasting error threshold to obtain precision and recall quality metrics. The dependency of the quality metric on the threshold level is considered. An appropriate mechanism such as "one handle" was introduced for filtering faults that are outside of the plant operator field of interest. <s> BIB003 </s> Time Series Data Cleaning: A Survey <s> E. LONG SHORT-TERM MEMORY <s> Mechanical devices such as engines, vehicles, aircrafts, etc., are typically instrumented with numerous sensors to capture the behavior and health of the machine. However, there are often external factors or variables which are not captured by sensors leading to time-series which are inherently unpredictable. For instance, manual controls and/or unmonitored environmental conditions or load may lead to inherently unpredictable time-series. Detecting anomalies in such scenarios becomes challenging using standard approaches based on mathematical models that rely on stationarity, or prediction models that utilize prediction errors to detect anomalies. We propose a Long Short Term Memory Networks based Encoder-Decoder scheme for Anomaly Detection (EncDec-AD) that learns to reconstruct 'normal' time-series behavior, and thereafter uses reconstruction error to detect anomalies. We experiment with three publicly available quasi predictable time-series datasets: power demand, space shuttle, and ECG, and two real-world engine datasets with both predictive and unpredictable behavior. We show that EncDec-AD is robust and can detect anomalies from predictable, unpredictable, periodic, aperiodic, and quasi-periodic time-series. Further, we show that EncDec-AD is able to detect anomalies from short time-series (length as small as 30) as well as long time-series (length as large as 500). <s> BIB004
Since Recurrent Neural Network (RNN) also has the problem of gradient disappearance, it is difficult to process long-sequence data. Gers et al. BIB001 improve RNN and got the RNN special case Long Short-Term Memory (LSTM), which can avoid the disappearance of the regular RNN gradient. It has been widely used in industry and BIB003 -  use LSTM to perform anomaly detection on time series data. The left picture is a simple RNN structure diagram, and the right picture is a simple LSTM structure diagram in Figure 9 , where given function as shown in equation BIB002 . In equation (15), x t is the input of data in the current state, h t−1 (hidden state) indicates the input of the previous node received, y t is the output in the current state and h t is the output passed to the next node. As can be seen from the Figure 9 , the output h t is related to the values of x t and h t−1 . y t is often used to invest in a linear layer (mainly for dimension mapping), and then use softmax to classify the required data. As shown in Figure 9 , RNN has only one delivery state h t , LSTM also has a delivery status c t (cell state). There are three main stages within LSTM: (1) Forgotten phase. The forgetting phase is mainly to forget the input that is passed in from the previous node. A simple principle is: forget the unimportant, remember the important one. More specifically, z f is calculated as a forgotten gate, which is used to control the previous state c t−1 , and then decide whether to retain the data or forget it. (2) Selective memory phase. At this stage, the input is selectively memorized. Mainly to remember the input x, the more important the data needs to be more reserved. (3) Output phase. This phase determines which outputs would be treated as current states. Similar to the normal RNN, the output y t is often also obtained by h t change. Filonov et al. BIB003 and Pankaj et al. BIB004 provide recurrent neural networks by providing network time series data. The recurrent neural network understands what the normal expected network activity is. When an unfamiliar activity from the network is provided to a trained network, it can distinguish whether the activity is expected or invaded.