Time Series Data Cleaning: A Survey <s> A. MOVING AVERAGE <s> This book will be most useful to applied mathematicians, communication engineers, signal processors, statisticians, and time series researchers, both applied and theoretical. Readers should have some background in complex function theory and matrix algebra and should have successfully completed the equivalent of an upper division course in statistics. <s> BIB001 </s> Time Series Data Cleaning: A Survey <s> A. MOVING AVERAGE <s> Abstract The paper provides a systematic development of the forecasting expressions for exponential weighted moving averages. Methods for series with no trend, or additive or multiplicative trend are examined. Similarly, the methods cover non-seasonal, and seasonal series with additive or multiplicative error structures. The paper is a reprinted version of the 1957 report to the Office of Naval Research (ONR 52) and is being published here to provide greater accessibility. <s> BIB002 </s> Time Series Data Cleaning: A Survey <s> A. MOVING AVERAGE <s> Abstract In Gardner [Gardner, E. S., Jr. (1985). Exponential smoothing: The state of the art. Journal of Forecasting 4, 1–28], I reviewed the research in exponential smoothing since the original work by Brown and Holt. This paper brings the state of the art up to date. The most important theoretical advance is the invention of a complete statistical rationale for exponential smoothing based on a new class of state-space models with a single source of error. The most important practical advance is the development of a robust method for smoothing damped multiplicative trends. We also have a new adaptive method for simple smoothing, the first such method to demonstrate credible improved forecast accuracy over fixed-parameter smoothing. Longstanding confusion in the literature about whether and how to renormalize seasonal indices in the Holt–Winters methods has finally been resolved. There has been significant work in forecasting for inventory control, including the development of new predictive distributions for total lead-time demand and several improved versions of Croston's method for forecasting intermittent time series. Regrettably, there has been little progress in the identification and selection of exponential smoothing methods. The research in this area is best described as inconclusive, and it is still difficult to beat the application of a damped trend to every time series. <s> BIB003 </s> Time Series Data Cleaning: A Survey <s> A. MOVING AVERAGE <s> Nowadays, wireless sensor networks have been widely used in many monitoring applications. Due to the low quality of sensors and random effects of the environments, however, it is well known that the collected sensor data are noisy. Therefore, it is very critical to clean the sensor data before using them to answer queries or conduct data analysis. Popular data cleaning approaches, such as the moving average, cannot meet the requirements of both energy efficiency and quick response time in many sensor related applications. In this paper, we propose a hybrid sensor data cleaning approach with confidence. Specifically, we propose a smart weighted moving average (WMA) algorithm that collects confidence data from sensors and computes the weighted moving average. The rationale behind the WMA algorithm is to draw more samples for a particular value that is of great importance to the moving average, and provide higher confidence weight for this value, such that this important value can be quickly reflected in the moving average. Based on our extensive simulation results, we demonstrate that, compared to the simple moving average (SMA), our WMA approach can effectively clean data and offer quick response time. <s> BIB004 </s> Time Series Data Cleaning: A Survey <s> A. MOVING AVERAGE <s> Managing inconsistency in databases has long been recognized as an important problem. One of the most promising approaches to coping with inconsistency in databases is the framework of database repairs, which has been the topic of an extensive investigation over the past several years. Intuitively, a repair of an inconsistent database is a consistent database that differs from the given inconsistent database in a minimal way. So far, most of the work in this area has addressed the problem of obtaining the consistent answers to a query posed on an inconsistent database. Repair checking is the following decision problem: given two databases r and r', is r' a repair of r? Although repair checking is a fundamental algorithmic problem about inconsistent databases, it has not received as much attention as consistent query answering. In this paper, we give a polynomial-time algorithm for subset-repair checking under integrity constraints that are the union of a weakly acyclic set of local-as-view (LAV) tuple-generating dependencies and a set of equality-generating dependencies. This result significantly generalizes earlier work for subset-repair checking when the integrity constraints are the union of an acyclic set of inclusion dependencies and a set of functional dependencies. We also give a polynomial-time algorithm for symmetric-difference repair checking, when the integrity constraints form a weakly acyclic set of LAV tgds. After this, we establish a number of complexity-theoretic results that delineate the boundary between tractability and intractability for the repair-checking problem. Specifically, we show that the aforementioned tractability results are optimal; in particular, subset-repair checking for arbitrary weakly acyclic sets of tuple-generating dependencies is a coNP-complete problem. We also study cardinality-based repairs and show that cardinality-repair checking is coNP-complete for various classes of integrity constraints encountered in database design and data exchange. <s> BIB005 </s> Time Series Data Cleaning: A Survey <s> A. MOVING AVERAGE <s> Probe vehicle data are being increasingly applied in urban dynamic traffic data collection. However, the mobility and scale limit of probe vehicles may lead to incomplete or inaccurate data and thus influence the measurement of the state of traffic. At present, probe vehicle data are usually repaired by linear interpolation or a historical average method, but the repair accuracy is relatively low. To address the given problems, the multi-threshold control repair method (MTCRM) was proposed to clean and repair the probe vehicle data. The MTCRM adopts threshold control and a rule based on the approximate normalization transform to clean abnormal traffic data and to fill in the missing data by a weighted average method and an exponential smoothing method. In this approach, we combine topological road network characteristics to fill in the missing data from data for neighboring road sections and repair noisy data by reconstructing the principal components. This paper mainly focuses on analyzing the component of the recurring pattern of probe vehicle data, which can provide guidelines for the subsequent traffic forecasts. The findings of data repair for different grades of road in Beijing, China, demonstrate that the mean repair error may meet the requirements of traffic-state measurement, demonstrating that MTCRM can effectively clean probe vehicle data. <s> BIB006 </s> Time Series Data Cleaning: A Survey <s> A. MOVING AVERAGE <s> The data cleaning of electrical power big data can improve the correctness, the completeness, the consistency and the reliability of the data. Aiming at the difficulties of the extracting of the unified anomaly detection pattern and the low accuracy and continuity of the anomaly data correction in the process of the electrical power big data cleaning, the data cleaning model of the electrical power big data based on Spark is proposed. Firstly, the normal clusters and the corresponding boundary samples are obtained by the improved CURE clustering algorithm. Then, the anomaly data identification algorithm based on boundary samples is designed. Finally, the anomaly data modification is realized by using exponential weighting moving mean value. The high efficiency and accuracy is proved by the experiment of the data cleaning of the wind power generation monitoring data from the wind power station. <s> BIB007 </s> Time Series Data Cleaning: A Survey <s> A. MOVING AVERAGE <s> Anomalous data can negatively impact energy forecasting by causing model parameters to be incorrectly estimated. This paper presents two approaches for the detection and imputation of anomalies in time series data. Autoregressive with exogenous inputs (ARX) and artificial neural network (ANN) models are used to extract the characteristics of time series. Anomalies are detected by performing hypothesis testing on the extrema of the residuals, and the anomalous data points are imputed using the ARX and ANN models. Because the anomalies affect the model coefficients, the data cleaning process is performed iteratively. The models are re-learned on “cleaner” data after an anomaly is imputed. The anomalous data are reimputed to each iteration using the updated ARX and ANN models. The ARX and ANN data cleaning models are evaluated on natural gas time series data. This paper demonstrates that the proposed approaches are able to identify and impute anomalous data points. Forecasting models learned on the unclean data and the cleaned data are tested on an uncleaned out-of-sample dataset. The forecasting model learned on the cleaned data outperforms the model learned on the unclean data with 1.67% improvement in the mean absolute percentage errors and a 32.8% improvement in the root mean squared error. Existing challenges include correctly identifying specific types of anomalies such as negative flows. <s> BIB008
The moving average (MA) series algorithm BIB001 is widely used in time series for smoothing and time series prediction. A simple moving average (SMA) algorithm: Calculate the average of the most recently observed N time series values, which is used to predict the value at time t. A simple definition as shown in equation BIB005 . In equation (1), x t is the predicted value of x t , x t represents the true value at time t, 2 * n + 1 is the window size(count). For a time series x(t) = 9. 8, 8.5, 5.4, 5.6, 5.9, 9.2, 7.4 , for simplicity, we use SMA with n = 3 and k = 4, then calculate according to equation BIB008 . To eliminate errors or noises, the data x(t) can be considered as a certain time window of sliding window, then continuously calculate the local average over a given interval 2n + 1 to filter out the noise (erroneous data) and get a smoother measurement. In the weighted moving average (WMA) algorithm, data points at different relative positions in the window have different weights. Generally defined as: In equation (3), ω i represents the weight of the influence of the i position data point on the t position data point, other definitions follow the example above. A simple strategy is that the farther away from the two data points, the smaller the mutual influence. For instance, a natural idea is the reciprocal of the distance between two data points as the weight of their mutual influence. Similarly, the weight of each data point in the exponential weighted moving average (EWMA) algorithm BIB003 decreases exponentially with increasing distance, which is mainly used for unsteady time series  , BIB002 . Aiming at the need for the rapid response of sensor data cleaning, Zhuang et al. BIB004 propose an intelligent weighted moving average algorithm, which calculates weighted moving averages via collected confidence data from sensors. Zhang et al. BIB006 propose a method based on multi-threshold control and approximate positive transform to clean the probe vehicle data, and fill the lost data with the weighted average method and exponential smoothing method. Qu et al. BIB007 first use cluster-based methods for anomaly detection and then use exponentially weighted averaging for data repair, which is used to clean power data in a distributed environment.