Time Series Data Cleaning: A Survey <s> I. INTRODUCTION <s> Preface 1 INTRODUCTION 1.1 Examples of Time Series 1.2 Objectives of Time Series Analysis 1.3 Some Simple Time Series Models 1.3.3 A General Approach to Time Series Modelling 1.4 Stationary Models and the Autocorrelation Function 1.4.1 The Sample Autocorrelation Function 1.4.2 A Model for the Lake Huron Data 1.5 Estimation and Elimination of Trend and Seasonal Components 1.5.1 Estimation and Elimination of Trend in the Absence of Seasonality 1.5.2 Estimation and Elimination of Both Trend and Seasonality 1.6 Testing the Estimated Noise Sequence 1.7 Problems 2 STATIONARY PROCESSES 2.1 Basic Properties 2.2 Linear Processes 2.3 Introduction to ARMA Processes 2.4 Properties of the Sample Mean and Autocorrelation Function 2.4.2 Estimation of $\gamma(\cdot)$ and $\rho(\cdot)$ 2.5 Forecasting Stationary Time Series 2.5.3 Prediction of a Stationary Process in Terms of Infinitely Many Past Values 2.6 The Wold Decomposition 1.7 Problems 3 ARMA MODELS 3.1 ARMA($p,q$) Processes 3.2 The ACF and PACF of an ARMA$(p,q)$ Process 3.2.1 Calculation of the ACVF 3.2.2 The Autocorrelation Function 3.2.3 The Partial Autocorrelation Function 3.3 Forecasting ARMA Processes 1.7 Problems 4 SPECTRAL ANALYSIS 4.1 Spectral Densities 4.2 The Periodogram 4.3 Time-Invariant Linear Filters 4.4 The Spectral Density of an ARMA Process 1.7 Problems 5 MODELLING AND PREDICTION WITH ARMA PROCESSES 5.1 Preliminary Estimation 5.1.1 Yule-Walker Estimation 5.1.3 The Innovations Algorithm 5.1.4 The Hannan-Rissanen Algorithm 5.2 Maximum Likelihood Estimation 5.3 Diagnostic Checking 5.3.1 The Graph of $\t=1,\ldots,n\ 5.3.2 The Sample ACF of the Residuals <s> BIB001 </s> Time Series Data Cleaning: A Survey <s> I. INTRODUCTION <s> Pervasive applications rely on data captured from the physical world through sensor devices. Data provided by these devices, however, tend to be unreliable. The data must, therefore, be cleaned before an application can make use of them, leading to additional complexity for application development and deployment. Here we present Extensible Sensor stream Processing (ESP), a framework for building sensor data cleaning infrastructures for use in pervasive applications. ESP is designed as a pipeline using declarative cleaning mechanisms based on spatial and temporal characteristics of sensor data. We demonstrate ESP's effectiveness and ease of use through three real-world scenarios. <s> BIB002 </s> Time Series Data Cleaning: A Survey <s> I. INTRODUCTION <s> Timestamps are often found to be dirty in various scenarios, e.g., in distributed systems with clock synchronization problems or unreliable RFID readers. Without cleaning the imprecise timestamps, temporal-related applications such as provenance analysis or pattern queries are not reliable. To evaluate the correctness of timestamps, temporal constraints could be employed, which declare the distance restrictions between timestamps. Guided by such constraints on timestamps, in this paper, we study a novel problem of repairing inconsistent timestamps that do not conform to the required temporal constraints. Following the same line of data repairing, the timestamp repairing problem is to minimally modify the timestamps towards satisfaction of temporal constraints. This problem is practically challenging, given the huge space of possible timestamps. We tackle the problem by identifying a concise set of promising candidates, where an optimal repair solution can always be found. Repair algorithms with efficient pruning are then devised over the identified candidates. Experiments on real datasets demonstrate the superiority of our proposal compared to the state-of-the-art approaches. <s> BIB003 </s> Time Series Data Cleaning: A Survey <s> I. INTRODUCTION <s> Detecting and repairing dirty data is one of the perennial challenges in data analytics, and failure to do so can result in inaccurate analytics and unreliable decisions. Over the past few years, there has been a surge of interest from both industry and academia on data cleaning problems including new abstractions, interfaces, approaches for scalability, and statistical techniques. To better understand the new advances in the field, we will first present a taxonomy of the data cleaning literature in which we highlight the recent interest in techniques that use constraints, rules, or patterns to detect errors, which we call qualitative data cleaning. We will describe the state-of-the-art techniques and also highlight their limitations with a series of illustrative examples. While traditionally such approaches are distinct from quantitative approaches such as outlier detection, we also discuss recent work that casts such approaches into a statistical estimation framework including: using Machine Learning to improve the efficiency and accuracy of data cleaning and considering the effects of data cleaning on statistical analysis. <s> BIB004
Time series data can be defined  as a sequence of random variables, x 1 , x 2 , . . . , x n , where the random variable x 1 denotes the value taken by the series at the first time point, the variable x 2 denotes the value for the second time period, x n denotes the value for the n-th time period, and so on. Time series have been widely used in many fields  , BIB001 , [49] such as financial economy, meteorology and hydrology, signal processing, industrial manufacturing, and so on. Time series data are important in industry, where there are all kinds of sensor devices capturing data from the industrial environment uninterruptedly. Owing to the fact that data of the sensor devices are often unreliable BIB002 , time series data are often large and dirty. In the financial field, the most important application of time series data is to predict future commodity (stock) price movements. However, time series errors in the financial field are also very prevalent, even some data sets, which are considered quite accurate, still have erroneous data. For instance, the correct rate of stock information on Yahoo Finance is 93%. The costs and risks of errors, conflicts, and inconsistencies in time series have drawn widespread attention from businesses and government agencies. In recent work, the data quality issues in time series data are studied, since they pose unique data quality challenges due to the presence of autocorrelations, trends, seasonality, and gaps in the time series data  . According to Shilakes and Tylman  , the relevant market growth rate The associate editor coordinating the review of this manuscript and approving it for publication was Feng Xia . of data quality is about 17%, which is much higher than the 7% annual growth rate of the IT industry. For instance, approximately 30% to 80% of the time and cost are spent on data cleaning in data warehousing project development. The time series errors can be either timestamp errors or observed value errors. For possible timestamp errors, Song et al. BIB003 propose a method for cleaning timestamps. In this survey, we focus on the existing methods of dealing with observed value errors, thereby, the time series errors mentioned in the following are observation errors. There are two types of processing methods commonly used in the industry when dealing with time series data errors: (1) Discarding erroneous data. First, the time series is detected via using an anomaly detection algorithm, and then the detected abnormal data are discarded. (2) Cleaning data. Data cleaning is divided into manual cleaning and automatic cleaning. There is no doubt that manual cleaning has a high accuracy rate, but it is difficult to implement because it takes more time and effort. The existing surveys of data cleaning BIB004 ,  mainly summarize the methods of dealing with data missing, data inconsistence, data integration and erroneous data in the database. Karkouch et al.  review the generation of sensor data, the reasons for the formation of data quality problems, and the techniques for improving data quality. However, Karkouch et al.  do not provide a detailed overview of the existing state-of-the-art of erroneous data cleaning. Thereby, we review the state-of-the-art of time series data error value cleaning, which may provide a tutorial for others.