Time Series Data Cleaning: A Survey <s> D. SUMMARY AND DISCUSSION <s> Outliers in time series can be regarded as being generated by dynamic intervention models at unknown time points. Two special cases, innovational outlier (IO) and additive outlier (AO), are studied in this article. The likelihood ratio criteria for testing the existence of outliers of both types, and the criteria for distinguishing between them are derived. An iterative procedure is proposed for detecting IO and AO in practice and for estimating the time series parameters in autoregressive-integrated-moving-average models in the presence of outliers. The powers of the procedure in detecting outliers are investigated by simulation experiments. The performance of the proposed procedure for estimating the autoregressive coefficient of a simple AR(l) model compares favorably with robust estimation procedures proposed in the literature. Two real examples are presented. <s> BIB001 </s> Time Series Data Cleaning: A Survey <s> D. SUMMARY AND DISCUSSION <s> Several algorithms are developed to estimate the parameters of a causal nonminimum-phase autoregressive moving average (ARMA) (p,q) system which is excited by an unobservable independently identically distributed non-Gaussian process. The output is contaminated by additive colored Gaussian noise of unknown power spectral density. A fundamental result is presented pertaining to the identifiability of AR parameters, based on the Yule-Walker equations drawn from a (specific) set of (p+1) 1-D slices of the kth (k>2) order output cumulant. Several MA parameter estimation algorithms are developed: one method uses q 1-D slices of the output cumulant; a second method uses only two 1-D cumulant slices. These methods do not involve computation of the residual (i.e. AR compensated) time series or polynomial factorization. Multidimensional versions of the various algorithms are presented. A simulation study demonstrating the effectiveness of the algorithms is included. > <s> BIB002 </s> Time Series Data Cleaning: A Survey <s> D. SUMMARY AND DISCUSSION <s> In recent years, there has been an explosion of interest in mining time-series databases. As with most computer science problems, representation of the data is the key to efficient and effective solutions. One of the most commonly used representations is piecewise linear approximation. This representation has been used by various researchers to support clustering, classification, indexing and association rule mining of time-series data. A variety of algorithms have been proposed to obtain this representation, with several algorithms having been independently rediscovered several times. In this paper, we undertake the first extensive review and empirical comparison of all proposed techniques. We show that all these algorithms have fatal flaws from a data-mining perspective. We introduce a novel algorithm that we empirically show to be superior to all others in the literature. <s> BIB003 </s> Time Series Data Cleaning: A Survey <s> D. SUMMARY AND DISCUSSION <s> Abstract Battery management systems in hybrid-electric-vehicle battery packs must estimate values descriptive of the pack’s present operating condition. These include: battery state-of-charge, power fade, capacity fade, and instantaneous available power. The estimation mechanism must adapt to changing cell characteristics as cells age and therefore provide accurate estimates over the lifetime of the pack. In a series of three papers, we propose methods, based on extended Kalman filtering (EKF), that are able to accomplish these goals for a lithium ion polymer battery pack. We expect that they will also work well on other battery chemistries. These papers cover the required mathematical background, cell modeling and system identification requirements, and the final solution, together with results. This third paper concludes the series by presenting five additional applications where either an EKF or results from EKF may be used in typical BMS algorithms: initializing state estimates after the vehicle has been idle for some time; estimating state-of-charge with dynamic error bounds on the estimate; estimating pack available dis/charge power; tracking changing pack parameters (including power fade and capacity fade) as the pack ages, and therefore providing a quantitative estimate of state-of-health; and determining which cells must be equalized. Results from pack tests are presented. <s> BIB004 </s> Time Series Data Cleaning: A Survey <s> D. SUMMARY AND DISCUSSION <s> Accuracy and robustness with respect to missing or corrupt input data are two key characteristics for any travel time prediction model that is to be applied in a real-time environment (e.g. for display on variable message signs on freeways). This article proposes a freeway travel time prediction framework that exhibits both qualities. The framework exploits a recurrent neural network topology, the so-called state-space neural network (SSNN), with preprocessing strategies based on imputation. Although the SSNN model is a neural network, its design (in terms of input- and model selection) is not “black box” nor location-specific. Instead, it is based on the lay-out of the freeway stretch of interest. In this sense, the SSNN model combines the generality of neural network approaches, with traffic related (“white-box”) design. Robustness to missing data is tackled by means of simple imputation (data replacement) schemes, such as exponential forecasts and spatial interpolation. Although there are clear theoretical shortcomings to “simple” imputation schemes to remedy input failure, our results indicate that their use is justified in this particular application. The SSNN model appears to be robust to the “damage” done by these imputation schemes. This is true for both incidental (random) and structural input failure. We demonstrate that the SSNN travel time prediction framework yields good accurate and robust travel time predictions on both synthetic and real data. <s> BIB005 </s> Time Series Data Cleaning: A Survey <s> D. SUMMARY AND DISCUSSION <s> Abstract In the past decades, process engineers are facing increasingly more data analytics challenges and having difficulties obtaining valuable information from a wealth of process variable data trends. The raw data of different formats stored in databases are not useful until they are cleaned and transformed. Generally, data cleaning consists of four steps: missing data imputation, outlier detection, noise removal, and time alignment and delay estimation. This paper discusses available data cleaning methods that can be used in data pre-processing and help overcome challenges of “Big Data”. <s> BIB006 </s> Time Series Data Cleaning: A Survey <s> D. SUMMARY AND DISCUSSION <s> This article presents a robust augmented Kalman filter that extends the data– cleaning filter (Masreliez and Martin, 1977) to the general state space model featuring nonstationary and regression effects. The robust filter shrinks the observations towards their one–step–ahead prediction based on the past, by bounding the effect of the information carried by a new observation according to an influence function. When maximum likelihood estimation is carried out on the replacement data, an M–type estimator is obtained. We investigate the performance of the robust AKF in two applications using as a modeling framework the basic structural time series model, a popular unobserved components model in the analysis of seasonal time series. First, a Monte Carlo experiment is conducted in order to evaluate the comparative accuracy of the proposed method for estimating the variance parameters. Second, the method is applied in a forecasting context to a large set of European trade statistics series. <s> BIB007 </s> Time Series Data Cleaning: A Survey <s> D. SUMMARY AND DISCUSSION <s> Missing numerical values are prevalent, e.g., owing to unreliable sensor reading, collection and transmission among heterogeneous sources. Unlike categorized data imputation over a limited domain, the numerical values suffer from two issues: (1) sparsity problem, the incomplete tuple may not have sufficient complete neighbors sharing the same/similar values for imputation, owing to the (almost) infinite domain; (2) heterogeneity problem, different tuples may not fit the same (regression) model. In this study, enlightened by the conditional dependencies that hold conditionally over certain tuples rather than the whole relation, we propose to learn a regression model individually for each complete tuple together with its neighbors. Our IIM, Imputation via Individual Models, thus no longer relies on sharing similar values among the k complete neighbors for imputation, but utilizes their regression results by the aforesaid learned individual (not necessary the same) models. Remarkably, we show that some existing methods are indeed special cases of our IIM, under the extreme settings of the number l of learning neighbors considered in individual learning. In this sense, a proper number l of neighbors is essential to learn the individual models (avoid over-fitting or under-fitting). We propose to adaptively learn individual models over various number l of neighbors for different complete tuples. By devising efficient incremental computation, the time complexity of learning a model reduces from linear to constant. Experiments on real data demonstrate that our IIM with adaptive learning achieves higher imputation accuracy than the existing approaches. <s> BIB008 </s> Time Series Data Cleaning: A Survey <s> D. SUMMARY AND DISCUSSION <s> Incomplete information often occurs along with many database applications, e.g., in data integration, data cleaning, or data exchange. The idea of data imputation is often to fill the missing data with the values of its neighbors who share the same/similar information. Such neighbors could either be identified certainly by editing rules or extensively by similarity relationships. Owing to data sparsity, the number of neighbors identified by editing rules w.r.t. value equality is rather limited, especially in the presence of data values with variances . To enrich the imputation candidates, a natural idea is to extensively consider the neighbors with similarity relationship. However, the candidates suggested by these (heterogenous) similarity neighbors may conflict with each other. In this paper, we propose to utilize the similarity rules with tolerance to small variations (instead of the aforesaid editing rules with strict equality constraints) to rule out the invalid candidates provided by similarity neighbors. To enrich the data imputation, i.e., imputing the missing values more , we study the problem of maximizing the missing data imputation. Our major contributions include (1) the np -hardness analysis on solving as well as approximating the problem, (2) exact algorithms for tackling the problem, and (3) efficient approximation with performance guarantees. Experiments on real and synthetic data sets demonstrate the superiority of our proposal in filling accuracy. We also demonstrate that the record matching application is indeed improved, after applying the proposed imputation. <s> BIB009
As shown in Table 2 , there are many methods based on smoothing, such as the state-space model  , BIB007 , BIB005 and Interpolation BIB003 , BIB006 . The state-space model assumes that the system's change over time can be determined by an unobservable vector sequence, the relationship between the time series and the observable sequence can be determined by the state-space model. By establishing state equations and observation equations, the state-space model provides a model framework to fully describe the temporal characteristics of dynamic systems. To make this kind of smoothing algorithm have a better effect, many studies BIB001 , BIB004 , BIB002 have also proposed various techniques to estimate the parameters in the above methods. Most smoothing techniques, when cleaning time series, have a small-time overhead, but it is very easy to change the original correct data points, which greatly affects the accuracy of cleaning. In other words, correct data are altered, which can distort the results of the analysis and lead to uncertainty in the results. In the field of relational databases, there are many cleaning algorithms based on integrity constraints, which are difficult to apply in the field of time series in which the observed values are substantially numerical, because they follow a strict equality relationship. A few methods, which we summarize in Table (5), can be used for time series data cleaning, for instance ODs and SDs can be used to solve problems in some scenarios, such as the number of miles in a car is nondecreasing. Further speed-based constraints can be used to process data such as GPS and stock prices, but only with VOLUME 8, 2020 relevant domain knowledge can give a reasonable constraint. Therefore, the constraint-based cleaning algorithm needs to be further improved to have better robustness. The Similarity Rule Constraints proposed by Song et al. BIB009 and the Learning Individual Models proposed by Zhang et al. BIB008 are suitable for repairing missing data. One possible future direction is to use anomaly detection methods to detect anomalies first, and then treat outliers as missing to repair. We will discuss anomaly detection in Section V.