Time Series Data Cleaning: A Survey <s> B. AUTOREGRESSIVE <s> Many statistical models, and in particular autoregressive-moving average time series models, can be regarded as means of transforming the data to white noise, that is, to an uncorrelated sequence of errors. If the parameters are known exactly, this random sequence can be computed directly from the observations; when this calculation is made with estimates substituted for the true parameter values, the resulting sequence is referred to as the "residuals," which can be regarded as estimates of the errors. If the appropriate model has been chosen, there will be zero autocorrelation in the errors. In checking adequacy of fit it is therefore logical to study the sample autocorrelation function of the residuals. For large samples the residuals from a correctly fitted model resemble very closely the true errors of the process; however, care is needed in interpreting the serial correlations of the residuals. It is shown here that the residual autocorrelations are to a close approximation representable as a singular linear transformation of the autocorrelations of the errors so that they possess a singular normal distribution. Failing to allow for this results in a tendency to overlook evidence of lack of fit. Tests of fit and diagnostic checks are devised which take these facts into account. <s> BIB001 </s> Time Series Data Cleaning: A Survey <s> B. AUTOREGRESSIVE <s> Autoregressive moving-average (ARMA) models are of great interest in speech processing. This paper presents new stochastic realization algorithms for identification of such models, by use of a special canonical filter form in the state space, directly and simply connected with ARMA models. We take advantage of certain matrix properties to develop algorithms, which eliminate a matrix inversion, using either the autoeorrelation function of the signal {y(k)} , or the autocorrelation function of a pseudo-innovation sequence {\tilde{y}(k)} , or a cross-correlation function between {y(k)} and {\tilde{y}(k)} . We also present a new algorithm for optimal joint state and parameter estimation in the important case of autoregressive (AR) models. Results obtained with all these algorithms are given for simulated examples. <s> BIB002 </s> Time Series Data Cleaning: A Survey <s> B. AUTOREGRESSIVE <s> Outliers, level shifts, and variance changes are commonplace in applied time series analysis. However, their existence is often ignored and their impact is overlooked, for the lack of simple and useful methods to detect and handle those extraordinary events. The problem of detecting outliers, level shifts, and variance changes in a univariate time series is considered. The methods employed are extremely simple yet useful. Only the least squares techniques and residual variance ratios are used. The effectiveness of these simple methods is demonstrated by analysing three real data sets. <s> BIB003 </s> Time Series Data Cleaning: A Survey <s> B. AUTOREGRESSIVE <s> Abstract This paper presents the use of statistically rigorous algorithms combined with active-sensing impedance methods for damage identification in engineering systems. In particular, we propose to use statistical pattern recognition methods to address damage classification and data mining issues associated with the examination of large numbers of impedance signals for health monitoring applications. In this paper, in order to diagnose damage with statistical confidence, the impedance-based monitoring is cast in the context of an outlier detection framework. A modified auto-regressive model with exogenous inputs (ARX) in the frequency domain is developed and the damage-sensitive feature is obtained by quantifying the differences between the measured impedance and the output of the ARX model. Furthermore, because of the non-Gaussian nature of the feature distribution tails, extreme value statistics is employed to develop a robust damage classifier. This paper concludes with a numerical example on a 5 degree-of-freedom system and an experimental investigation on a multi-story building model to demonstrate the performance of the proposed concept. <s> BIB004 </s> Time Series Data Cleaning: A Survey <s> B. AUTOREGRESSIVE <s> The deployment of environmental sensors has generated an interest in real-time applications of the data they collect. This research develops a real-time anomaly detection method for environmental data streams that can be used to identify data that deviate from historical patterns. The method is based on an autoregressive data-driven model of the data stream and its corresponding prediction interval. It performs fast, incremental evaluation of data as it becomes available, scales to large quantities of data, and requires no pre-classification of anomalies. Furthermore, this method can be easily deployed on a large heterogeneous sensor network. Sixteen instantiations of this method are compared based on their ability to identify measurement errors in a windspeed data stream from Corpus Christi, Texas. The results indicate that a multilayer perceptron model of the data stream, coupled with replacement of anomalous data points, performs well at identifying erroneous data in this data stream. <s> BIB005 </s> Time Series Data Cleaning: A Survey <s> B. AUTOREGRESSIVE <s> This paper proposes a stochastic wind power model based on an autoregressive integrated moving average (ARIMA) process. The model takes into account the nonstationarity and physical limits of stochastic wind power generation. The model is constructed based on wind power measurement of one year from the Nysted offshore wind farm in Denmark. The proposed limited-ARIMA (LARIMA) model introduces a limiter and characterizes the stochastic wind power generation by mean level, temporal correlation and driving noise. The model is validated against the measurement in terms of temporal correlation and probability distribution. The LARIMA model outperforms a first-order transition matrix based discrete Markov model in terms of temporal correlation, probability distribution and model parameter number. The proposed LARIMA model is further extended to include the monthly variation of the stochastic wind power generation. <s> BIB006 </s> Time Series Data Cleaning: A Survey <s> B. AUTOREGRESSIVE <s> In declarative data cleaning, data semantics are encoded as constraints and errors arise when the data violates the constraints. Various forms of statistical and logical inference can be used to reason about and repair inconsistencies (errors) in data. Recently, unified approaches that repair both errors in data and errors in semantics (the constraints) have been proposed. However, both data-only approaches and unified approaches are by and large static in that they apply cleaning to a single snapshot of the data and constraints. We introduce a continuous data cleaning framework that can be applied to dynamic data and constraint environments. Our approach permits both the data and its semantics to evolve and suggests repairs based on the accumulated evidence to date. Importantly, our approach uses not only the data and constraints as evidence, but also considers the past repairs chosen and applied by a user (user repair preferences). We introduce a repair classifier that predicts the type of repair needed to resolve an inconsistency, and that learns from past user repair preferences to recommend more accurate repairs in the future. Our evaluation shows that our techniques achieve high prediction accuracy and generate high quality repairs. Of independent interest, our work makes use of a set of data statistics that are shown to be sensitive to predicting particular repair types. <s> BIB007 </s> Time Series Data Cleaning: A Survey <s> B. AUTOREGRESSIVE <s> RFID is widely used for object tracking in indoor environments, e.g., airport baggage tracking. Analyzing RFID data offers insight into the underlying tracking systems as well as the associated business processes. However, the inherent uncertainty in RFID data, including noise (cross readings) and incompleteness (missing readings), pose challenges to high-level RFID data querying and analysis. In this paper, we address these challenges by proposing a learning-based data cleansing approach that, unlike existing approaches, requires no detailed prior knowledge about the spatio-temporal properties of the indoor space and the RFID reader deployment. Requiring only minimal information about RFID deployment, the approach learns relevant knowledge from raw RFID data and uses it to cleanse the data. In particular, we model raw RFID readings as time series that are sparse because the indoor space is only partly covered by a limited number of RFID readers. We propose the Indoor RFID Multi-variate Hidden Markov Model (IR-MHMM) to capture the uncertainties of indoor RFID data as well as the correlation of moving object locations and object RFID readings. We propose three state space design methods for IR-MHMM that enable the learning of parameters while contending with raw RFID data time series. We solely use raw uncleansed RFID data for the learning of model parameters, requiring no special labeled data or ground truth. The resulting IR-MHMM based RFID data cleansing approach is able to recover missing readings and reduce cross readings with high effectiveness and efficiency, as demonstrated by extensive experimental studies with both synthetic and real data. Given enough indoor RFID data for learning, the proposed approach achieves a data cleansing accuracy comparable to or even better than state-of-the-art techniques requiring very detailed prior knowledge, making our solution superior in terms of both effectiveness and employability. <s> BIB008 </s> Time Series Data Cleaning: A Survey <s> B. AUTOREGRESSIVE <s> Anomalous data can negatively impact energy forecasting by causing model parameters to be incorrectly estimated. This paper presents two approaches for the detection and imputation of anomalies in time series data. Autoregressive with exogenous inputs (ARX) and artificial neural network (ANN) models are used to extract the characteristics of time series. Anomalies are detected by performing hypothesis testing on the extrema of the residuals, and the anomalous data points are imputed using the ARX and ANN models. Because the anomalies affect the model coefficients, the data cleaning process is performed iteratively. The models are re-learned on “cleaner” data after an anomaly is imputed. The anomalous data are reimputed to each iteration using the updated ARX and ANN models. The ARX and ANN data cleaning models are evaluated on natural gas time series data. This paper demonstrates that the proposed approaches are able to identify and impute anomalous data points. Forecasting models learned on the unclean data and the cleaned data are tested on an uncleaned out-of-sample dataset. The forecasting model learned on the cleaned data outperforms the model learned on the unclean data with 1.67% improvement in the mean absolute percentage errors and a 32.8% improvement in the root mean squared error. Existing challenges include correctly identifying specific types of anomalies such as negative flows. <s> BIB009 </s> Time Series Data Cleaning: A Survey <s> B. AUTOREGRESSIVE <s> Abstract High resolution velocity profiling instruments have enabled a new generation of turbulence studies by greatly increasing the amount and quality of simultaneous velocity measurements that can be obtained. As with all velocity profiling instruments, however, the collected data are susceptible to erroneous spikes and poor quality time series that must be corrected or removed prior to analysis and interpretation of the results. In the current study, ARMA models are investigated for their potential to provide a comprehensive approach to data cleaning. Specific objectives are to: i) describe the cleaning algorithms and their integration with an existing open-source Matlab toolbox, ii) test the algorithms using a range of published data sets from two different instruments, and iii) recommend metrics to compare cleaning results. The recommended approach to detect and replace outliers in profiled velocity data is to use a spatial ‘seasonal’ filter that takes advantage of information available in neighboring cells and a low order ARMA model. Recommended data quality metrics are the spike frequency and the coefficients of the model. This approach is more precise than the most common current despiking method, offers a seamless method for generating replacement values that does not change the statistics of the velocity time series, and provides simple metrics with clear physical interpretation that can be used to compare the quality of different datasets. <s> BIB010
The Autoregressive (AR) Model is a process that uses itself as a regression variable and uses the linear combination of the previous k random variables to describe the linear regression model of the random variable at the time t. The definition of AR model BIB005 , BIB007 as shown in equation BIB008 . In equation (4), x t is the predicted value of x t , x t represents the true value at time t, k is the order, µ is mean value of the process, t is white noise, ω i is the parameter of the model, a is a constant. Park et al. BIB004 use labeled data y to propose an autoregressive with exogenous input (ARX) model based on the AR model: In equation (5), y t is the possible repair of x t , and others are the same to the aforesaid AR model. Alengrin and Favier BIB002 propose Autoregressive moving average (ARMA) model), which is composed of the AR model and MA model. Besides that, the Gaussian Autoregressive Moving Average model is defined as shown in equation (6) BIB003 . In equation (6), , and x t is a sequence of independent Gaussian variates with mean µ = 0 and variance σ 2 x . Box and Pierce BIB001 propose a more complex Autoregressive Integrated Moving Average (ARIMA) model based on the ARMA model, which is not described in detail here. Akouemo and Povinelli BIB009 propose a method combining ARX and Artificial Neural Network (ANN) model for cleaning time series, which performs a hypothesis test to detect anomalies the extrema of the residuals, and repairs anomalous data points by using the ARX and ANN models. Dilling and MacVicar BIB010 clean high-frequency velocity profile data with ARMA model and Chen et al. BIB006 use the ARIMA model to clean wind power data.