Time Series Data Cleaning: A Survey <s> A. TOOLS <s> Analysts often clean dirty data iteratively--cleaning some data, executing the analysis, and then cleaning more data based on the results. We explore the iterative cleaning process in the context of statistical model training, which is an increasingly popular form of data analytics. We propose ActiveClean, which allows for progressive and iterative cleaning in statistical modeling problems while preserving convergence guarantees. ActiveClean supports an important class of models called convex loss models (e.g., linear regression and SVMs), and prioritizes cleaning those records likely to affect the results. We evaluate ActiveClean on five real-world datasets UCI Adult, UCI EEG, MNIST, IMDB, and Dollars For Docs with both real and synthetic errors. The results show that our proposed optimizations can improve model accuracy by up-to 2.5x for the same amount of data cleaned. Furthermore for a fixed cleaning budget and on all real dirty datasets, ActiveClean returns more accurate models than uniform sampling and Active Learning. <s> BIB001 </s> Time Series Data Cleaning: A Survey <s> A. TOOLS <s> We introduce HoloClean, a framework for holistic data repairing driven by probabilistic inference. HoloClean unifies qualitative data repairing, which relies on integrity constraints or external data sources, with quantitative data repairing methods, which leverage statistical properties of the input data. Given an inconsistent dataset as input, HoloClean automatically generates a probabilistic program that performs data repairing. Inspired by recent theoretical advances in probabilistic inference, we introduce a series of optimizations which ensure that inference over HoloClean's probabilistic model scales to instances with millions of tuples. We show that HoloClean finds data repairs with an average precision of ∼ 90% and an average recall of above ∼ 76% across a diverse array of datasets exhibiting different types of errors. This yields an average F1 improvement of more than 2× against state-of-the-art methods. <s> BIB002 </s> Time Series Data Cleaning: A Survey <s> A. TOOLS <s> Time series visualization of streaming telemetry (i.e., charting of key metrics such as server load over time) is increasingly prevalent in modern data platforms and applications. However, many existing systems simply plot the raw data streams as they arrive, often obscuring large-scale trends due to small-scale noise. We propose an alternative: to better prioritize end users' attention, smooth time series visualizations as much as possible to remove noise, while retaining large-scale structure to highlight significant deviations. We develop a new analytics operator called ASAP that automatically smooths streaming time series by adaptively optimizing the trade-off between noise reduction (i.e., variance) and trend retention (i.e., kurtosis). We introduce metrics to quantitatively assess the quality of smoothed plots and provide an efficient search strategy for optimizing these metrics that combines techniques from stream processing, user interface design, and signal processing via autocorrelation-based pruning, pixel-aware preaggregation, and on-demand refresh. We demonstrate that ASAP can improve users' accuracy in identifying long-term deviations in time series by up to 38.4% while reducing response times by up to 44.3%. Moreover, ASAP delivers these results several orders of magnitude faster than alternative search strategies. <s> BIB003 </s> Time Series Data Cleaning: A Survey <s> A. TOOLS <s> Data cleaning consumes up to 80% of the data analysis pipeline. This is a significant overhead for organizations where data cleaning is still a manually driven process requiring domain expertise. Recent advances have fueled a new computing paradigm called Database-as-a-Service, where data management tasks are outsourced to large service providers. We propose a new Data Cleaning-as-a-Service model that allows a client to interact with a data cleaning provider who hosts curated, and sensitive data. We present PACAS: a Privacy-Aware data Cleaning-As-a-Service framework that facilitates communication between the client and the service provider via a data pricing scheme where clients issue queries, and the service provider returns clean answers for a price while protecting her data. We propose a practical privacy model in such interactive settings called (X,Y,L)-anonymity that extends existing data publishing techniques to consider the data semantics while protecting sensitive values. Our evaluation over real data shows that PACAS effectively safeguards semantically related sensitive values, and provides improved accuracy over existing privacy-aware cleaning techniques. <s> BIB004 </s> Time Series Data Cleaning: A Survey <s> A. TOOLS <s> The wide use of machine learning is fundamentally changing the software development paradigm (a.k.a. Software 2.0) where data becomes a first-class citizen, on par with code. As machine learning is used in sensitive applications, it becomes imperative that the trained model is accurate, fair, and robust to attacks. While many techniques have been proposed to improve the model training process (in-processing approach) or the trained model itself (post-processing), we argue that the most effective method is to clean the root cause of error: the data the model is trained on (pre-processing). Historically, there are at least three research communities that have been separately studying this problem: data management, machine learning (model fairness), and security. Although a significant amount of research has been done by each community, ultimately the same datasets must be preprocessed, and there is little understanding how the techniques relate to each other and can possibly be integrated. We contend that it is time to extend the notion of data cleaning for modern machine learning needs. We identify dependencies among the data preprocessing techniques and propose MLClean, a unified data cleaning framework that integrates the techniques and helps train accurate and fair models. This work is part of a broader trend of Big data -- Artificial Intelligence (AI) integration. <s> BIB005 </s> Time Series Data Cleaning: A Survey <s> A. TOOLS <s> With the dramatic increasing interest in data analysis, ensuring data quality becomes one of the most important topics in data science. Data Cleaning, the process of ensuring data quality, is composed of two stages: error detection and error repair. Despite decades of research in data cleaning, existing cleaning systems still have limitations in terms of usability and error coverage. We propose PIClean, a probabilistic and interactive data cleaning system that aims at addressing the aforementioned limitations. PIClean produces probabilistic errors and probabilistic fixes using low-rank approximation, which implicitly discovers and uses relationships between columns of a dataset for cleaning. The probabilistic errors and fixes are confirmed or rejected by users, and the user feedbacks are constantly incorporated by PIClean to produce more accurate and higher-coverage cleaning results. <s> BIB006 </s> Time Series Data Cleaning: A Survey <s> A. TOOLS <s> The application of social network has produced a large amount of entity data in different formats, which accompanied by problems such as data offset and attribute missing. The existing research on dealing with multiple data scenarios and accuracy performance is insufficient. To solve the problem of data cleaning, EDCleaner is carried out on transforming entity information into structured data with attribute labeling. The method of attribute recognition and data normalization for semi-structured data is proposed in EDCleaner, which efficiently identifies the attribute tag relationship of data and obtains the structured data with uniform specifications. Furthermore, a data cleaning model with active learning extension is established. The machine learning classifier is used to further improve the accuracy of attribute recognition, and finally form an efficient and accurate data cleaning method. Experimental results show that EDCleaner improves the cleaning accuracy and other performing indicators of entity information and exceeds the level of state of the art. <s> BIB007 </s> Time Series Data Cleaning: A Survey <s> A. TOOLS <s> IoT data with timestamps are often found with outliers, such as GPS trajectories or sensor readings. While existing systems mostly focus on detecting temporal outliers without explanations, a decision maker may be more interested in the cause of the outlier appearance such that subsequent actions would be taken, e.g., cleaning unreliable readings or repairing broken devices. Such outlier detection and explanation are expected to be performed in either offline (batch) or online modes (over streaming IoT data with timestamps). In this work, we present TsOutlier, a new prototype system for detecting outliers with explanations over IoT data. The framework defines uniform profiles to explain the outliers detected by various algorithms, including the outliers with variant time intervals. Both batch and streaming processing are supported in a uniform framework. In particular, by varying the block size, it provides a tradeoff between computing the accurate results and approximating with efficient incremental computation. In this paper, we present several case studies of applying TsOutlier in industry, e.g., how this framework works in detecting outliers over the operation data of Shanghai Subway, and how to get reasonable explanations for the detected outliers in tracking excavators. <s> BIB008
There are many tools or systems for data cleaning, but they are not effective on time series cleaning problems. In Table 8 we investigate some tools that might be used for time series cleaning because they BIB001 , BIB002 , BIB005 , BIB006 are originally used to solve traditional database cleaning problems. Ding et al.  present Cleanits, which is an industrial time series cleaning system and implements an integrated cleaning strategy for detecting and repairing in industrial time series. Cleanits provides a user-friendly interface so users can use results and logging visualization over every cleaning process. Besides, the algorithm design of Cleanits also considers the characteristics of industrial time series and domain knowledge. The ASPA proposed by Rong and Bailis BIB003 violates the principle of minimum modification and distort the data, which is not suitable for being used widely. EDCleaner proposed by Wang et al. BIB007 is designed for social network data, detection and cleaning are performed through the characteristics of statistical data fields. Huang et al. BIB004 propose PACAS which is a framework for data cleaning between service providers and customers. Huang et al. BIB008 present TsOutlier, a new framework for detecting outliers with explanations over IoT data. TsOutlier uses multiple algorithms to detect anomalies in time series data, and supports both batch and streaming processing. There is not much research on time series cleaning tools or systems, and we discuss further in Future Directions in Section VII.