Time Series Data Cleaning: A Survey <s> II. SMOOTHING BASED CLEANING ALGORITHM <s> This paper describes the use of the Kalman Filter in a certain ciass of forecasting problems. The time series is assumed to be modeled as a time varying mean with additive noise. The mean of the time series is assumed to be a linear combination of known functions. The coefficients appearing in the linear combination are unknown. Under such assumptions, the time series can be described as a linear system with the state vector of the system being the unknown parameters and present value of the mean of the process. The Kalman Filter can be used under these circumstances to obtain an “optimal” estimate of the state vector. One of the distinct advantages of the Kalman Filter is that time varying coefficients can be permitted in the model. Examples using the Kalman Filter in forecasting are presented. <s> BIB001 </s> Time Series Data Cleaning: A Survey <s> II. SMOOTHING BASED CLEANING ALGORITHM <s> This book will be most useful to applied mathematicians, communication engineers, signal processors, statisticians, and time series researchers, both applied and theoretical. Readers should have some background in complex function theory and matrix algebra and should have successfully completed the equivalent of an upper division course in statistics. <s> BIB002 </s> Time Series Data Cleaning: A Survey <s> II. SMOOTHING BASED CLEANING ALGORITHM <s> We are concerned with the issues of outlier detection and change point detection from a data stream. In the area of data mining, there have been increased interest in these issues since the former is related to fraud detection, rare event discovery, etc., while the latter is related to event/trend by change detection, activity monitoring, etc. Specifically, it is important to consider the situation where the data source is non-stationary, since the nature of data source may change over time in real applications. Although in most previous work outlier detection and change point detection have not been related explicitly, this paper presents a unifying framework for dealing with both of them on the basis of the theory of on-line learning of non-stationary time series. In this framework a probabilistic model of the data source is incrementally learned using an on-line discounting learning algorithm, which can track the changing data source adaptively by forgetting the effect of past data gradually. Then the score for any given data is calculated to measure its deviation from the learned model, with a higher score indicating a high possibility of being an outlier. Further change points in a data stream are detected by applying this scoring method into a time series of moving averaged losses for prediction using the learned model. Specifically we develop an efficient algorithms for on-line discounting learning of auto-regression models from time series data, and demonstrate the validity of our framework through simulation and experimental applications to stock market data analysis. <s> BIB003 </s> Time Series Data Cleaning: A Survey <s> II. SMOOTHING BASED CLEANING ALGORITHM <s> The deployment of environmental sensors has generated an interest in real-time applications of the data they collect. This research develops a real-time anomaly detection method for environmental data streams that can be used to identify data that deviate from historical patterns. The method is based on an autoregressive data-driven model of the data stream and its corresponding prediction interval. It performs fast, incremental evaluation of data as it becomes available, scales to large quantities of data, and requires no pre-classification of anomalies. Furthermore, this method can be easily deployed on a large heterogeneous sensor network. Sixteen instantiations of this method are compared based on their ability to identify measurement errors in a windspeed data stream from Corpus Christi, Texas. The results indicate that a multilayer perceptron model of the data stream, coupled with replacement of anomalous data points, performs well at identifying erroneous data in this data stream. <s> BIB004 </s> Time Series Data Cleaning: A Survey <s> II. SMOOTHING BASED CLEANING ALGORITHM <s> This article presents a robust augmented Kalman filter that extends the data– cleaning filter (Masreliez and Martin, 1977) to the general state space model featuring nonstationary and regression effects. The robust filter shrinks the observations towards their one–step–ahead prediction based on the past, by bounding the effect of the information carried by a new observation according to an influence function. When maximum likelihood estimation is carried out on the replacement data, an M–type estimator is obtained. We investigate the performance of the robust AKF in two applications using as a modeling framework the basic structural time series model, a popular unobserved components model in the analysis of seasonal time series. First, a Monte Carlo experiment is conducted in order to evaluate the comparative accuracy of the proposed method for estimating the variance parameters. Second, the method is applied in a forecasting context to a large set of European trade statistics series. <s> BIB005
Smoothing techniques are often used to eliminate data noise, especially numerical data noise. Low-pass filtering, which filters out the lower frequency of the data set, is a simple algorithm. The characteristic of this type of technology is that the time overhead is small, but because the original data may be modified much, which makes the data distorted and leads to the uncertainty of the analysis results, there are not many applications used in time series cleaning. The research of smoothing technology mainly focuses on algorithms such as Moving Average (MA) BIB002 , Autoregressive (AR)  , BIB004 , BIB003 and Kalman filter model  , BIB005 , BIB001 . Thereby, this chapter mainly introduce these three technologies and their extensions.