Time Series Data Cleaning: A Survey <s> E. OTHERS <s> Vibratory power unit for vibrating conveyers and screens comprising an asynchronous polyphase motor, at least one pair of associated unbalanced masses disposed on the shaft of said motor, with the first mass of a pair of said unbalanced masses being rigidly fastened to said shaft and with said second mass of said pair being movably arranged relative to said first mass, means for controlling and regulating the conveying rate during conveyer operation by varying the rotational speed of said motor between predetermined minimum and maximum values, said second mass being movably outwardly by centrifugal force against the pressure of spring means, said spring means being prestressed in such a manner that said second mass is, at rotational motor speeds lower than said minimum speed, held in its initial position, and at motor speeds between said lower and upper values in positions which are radially offset with respect to the axis of said motor to an extent depending on the value of said rotational motor speed. <s> BIB001 </s> Time Series Data Cleaning: A Survey <s> E. OTHERS <s> This paper describes the use of the Kalman Filter in a certain ciass of forecasting problems. The time series is assumed to be modeled as a time varying mean with additive noise. The mean of the time series is assumed to be a linear combination of known functions. The coefficients appearing in the linear combination are unknown. Under such assumptions, the time series can be described as a linear system with the state vector of the system being the unknown parameters and present value of the mean of the process. The Kalman Filter can be used under these circumstances to obtain an “optimal” estimate of the state vector. One of the distinct advantages of the Kalman Filter is that time varying coefficients can be permitted in the model. Examples using the Kalman Filter in forecasting are presented. <s> BIB002 </s> Time Series Data Cleaning: A Survey <s> E. OTHERS <s> Abstract. An approach to smoothing and forecasting for time series with missing observations is proposed. For an underlying state-space model, the EM algorithm is used in conjunction with the conventional Kalman smoothed estimators to derive a simple recursive procedure for estimating the parameters by maximum likelihood. An example is given which involves smoothing and forecasting an economic series using the maximum likelihood estimators for the parameters. <s> BIB003 </s> Time Series Data Cleaning: A Survey <s> E. OTHERS <s> An Armstrong relation for a set of functional dependencies (FDs) is a relation that satisfies each FD implied by the set but no FD that is not implied by it. The structure and size (number of tuples) of Armstrong relatsons are investigated. Upper and lower bounds on the size of minimal-sized Armstrong relations are derived, and upper and lower bounds on the number of distinct entries that must appear m an Armstrong relation are given. It is shown that the time complexity of finding an Armstrong relation, gwen a set of functional dependencies, is precisely exponential in the number of attributes. Also shown ,s the falsity of a natural conjecture which says that almost all relations obeying a given set of FDs are Armstrong relations for that set of FDs. Finally, Armstrong relations are used to generahze a result, obtained by Demetrovics using quite complicated methods, about the possible sets of keys for a relauon. <s> BIB004 </s> Time Series Data Cleaning: A Survey <s> E. OTHERS <s> Most real-world data is stored in relational form. In contrast, most statistical learning methods work with “flat” data representations, forcing us to convert our data into a form that loses much of the relational structure. The recently introduced framework of probabilistic relational models (PRMs) allows us to represent probabilistic models over multiple entities that utilize the relations between them. In this paper, we propose the use of probabilistic models not only for the attributes in a relational model, but for the relational structure itself. We propose two mechanisms for modeling structural uncertainty: reference uncertainty and existence uncertainty. We describe the appropriate conditions for using each model and present learning algorithms for each. We present experimental results showing that the learned models can be used to predict relational structure and, moreover, the observed relational structure can be used to provide better predictions for the attributes in the model. <s> BIB005 </s> Time Series Data Cleaning: A Survey <s> E. OTHERS <s> Recent work on graphical models for relational data has demonstrated significant improvements in classification and inference when models represent the dependencies among instances. Despite its use in conventional statistical models, the assumption of instance independence is contradicted by most relational data sets. For example, in citation data there are dependencies among the topics of a paper's references, and in genomic data there are dependencies among the functions of interacting proteins. In this paper, we present relational dependency networks (RDNs), graphical models that are capable of expressing and reasoning with such dependencies in a relational setting. We discuss RDNs in the context of relational Bayes networks and relational Markov networks and outline the relative strengths of RDNs---namely, the ability to represent cyclic dependencies, simple methods for parameter estimation, and efficient structure learning techniques. The strengths of RDNs are due to the use of pseudolikelihood learning techniques, which estimate an efficient approximation of the full joint distribution. We present learned RDNs for a number of real-world data sets and evaluate the models in a prediction context, showing that RDNs identify and exploit cyclic relational dependencies to achieve significant performance gains over conventional conditional models. In addition, we use synthetic data to explore model performance under various relational data characteristics, showing that RDN learning and inference techniques are accurate over a wide range of conditions. <s> BIB006 </s> Time Series Data Cleaning: A Survey <s> E. OTHERS <s> Real-world databases often contain syntactic and semantic errors, in spite of integrity constraints and other safety measures incorporated into modern DBMSs. We present ERACER, an iterative statistical framework for inferring missing information and correcting such errors automatically. Our approach is based on belief propagation and relational dependency networks, and includes an efficient approximate inference algorithm that is easily implemented in standard DBMSs using SQL and user defined functions. The system performs the inference and cleansing tasks in an integrated manner, using shrinkage techniques to infer correct values accurately even in the presence of dirty data. We evaluate the proposed methods empirically on multiple synthetic and real-world data sets. The results show that our framework achieves accuracy comparable to a baseline statistical method using Bayesian networks with exact inference. However, our framework has wider applicability than the Bayesian network baseline, due to its ability to reason with complex, cyclic relational dependencies. <s> BIB007 </s> Time Series Data Cleaning: A Survey <s> E. OTHERS <s> A lot of systems and applications are data-driven, and the correctness of their operation relies heavily on the correctness of their data. While existing data cleaning techniques can be quite effective at purging datasets of errors, they disregard the fact that a lot of errors are systematic, inherent to the process that produces the data, and thus will keep occurring unless the problem is corrected at its source. In contrast to traditional data cleaning, in this paper we focus on data diagnosis: explaining where and how the errors happen in a data generative process. We develop a large-scale diagnostic framework called DATA X-RAY. Our contributions are three-fold. First, we transform the diagnosis problem to the problem of finding common properties among erroneous elements, with minimal domain-specific assumptions. Second, we use Bayesian analysis to derive a cost model that implements three intuitive principles of good diagnoses. Third, we design an efficient, highly-parallelizable algorithm for performing data diagnosis on large-scale data. We evaluate our cost model and algorithm using both real-world and synthetic data, and show that our diagnostic framework produces better diagnoses and is orders of magnitude more efficient than existing techniques. <s> BIB008 </s> Time Series Data Cleaning: A Survey <s> E. OTHERS <s> As key decisions are often made based on information contained in a database, it is important for the database to be as complete and correct as possible. For this reason, many data cleaning tools have been developed to automatically resolve inconsistencies in databases. However, data cleaning tools provide only best-effort results and usually cannot eradicate all errors that may exist in a database. Even more importantly, existing data cleaning tools do not typically address the problem of determining what information is missing from a database. To overcome the limitations of existing data cleaning techniques, we present QOCO, a novel query-oriented system for cleaning data with oracles. Under this framework, incorrect (resp. missing) tuples are removed from (added to) the result of a query through edits that are applied to the underlying database, where the edits are derived by interacting with domain experts which we model as oracle crowds. We show that the problem of determining minimal interactions with oracle crowds to derive database edits for removing (adding) incorrect (missing) tuples to the result of a query is NP-hard in general and present heuristic algorithms that interact with oracle crowds. Finally, we implement our algorithms in our prototype system QOCO and show that it is effective and efficient through a comprehensive suite of experiments. <s> BIB009 </s> Time Series Data Cleaning: A Survey <s> E. OTHERS <s> It is useful to predict future values in time series data, for example when there are many sensors monitoring environments such as urban space. The Gaussian Process (GP) model is considered as a promising technique for this setting. However, the GP model requires too high a training cost to be tractable for large data. Though approximation methods have been proposed to improve GP's scalability, they usually can only capture global trends in the data and fail to preserve small-scale patterns, resulting in unsatisfactory performance. We propose a new method to apply the GP for sensor time series prediction. Instead of (eagerly) training GPs on entire datasets, we custom-build query-dependent GPs on small fractions of the data for each prediction request. Implementing this idea in practice at scale requires us to overcome two obstacles. On the one hand, a central challenge with such a semi-lazy learning model is the substantial model-building effort at kNN query time, which could lead to unacceptable latency. We propose a novel two-level inverted-like index to support kNN search using the DTW on the GPU, making such "just-in-time" query-dependent model construction feasible for real-time applications. On the other hand, several parameters should be tuned for each time series individually since different sensors have different data generating processes in diverse environments. Manually configuring the parameters is usually not feasible due to the large number of sensors. To address this, we devise an adaptive auto-tuning mechanism to automatically determine and dynamically adjust the parameters for each time series with little human assistance. Our method has the following strong points: (a) it can make prediction in real time without a training phase; (b) it can yield superior prediction accuracy; and (c) it can effectively estimate the analytical predictive uncertainty. To illustrate our points, we present SMiLer, a semi-lazy time series prediction system for sensors. Extensive experiments on real-world datasets demonstrate its effectiveness and efficiency. In particular, by devising a two-level inverted-like index on the GPU with an enhanced lower bound of the DTW, SMiLer accelerates the efficiency of kNN search by one order of magnitude over its baselines. The prediction accuracy of SMiLer is better than the state-of-the-art competitors (up to 10 competitors) with better estimation of predictive uncertainty. <s> BIB010 </s> Time Series Data Cleaning: A Survey <s> E. OTHERS <s> With energy scarcity and environmental pollution becoming increasingly serious, the accurate estimation of fuel consumption of vehicles has been important in vehicle management and transportation planning toward a sustainable green transition. Fuel consumption is calculated by fuel-level data collected from high-precision fuel-level sensors. However, in the vehicle monitor platform, there are many types of error in the data collection and transmission processes, such as the noise, interference, and collision errors that are common in the high speed and dynamic vehicle environment. In this paper, an effective method for cleaning and repairing the fuel-level data is proposed, which adopts the threshold to acquire abnormal fuel data, the time quantum to identify abnormal data, and linear interpolation based algorithm to correct data errors. Specifically, a modified Gaussian mixture model (GMM) based on the synchronous iteration method is proposed to acquire the thresholds, which uses the particle swarm optimization algorithm and the steepest descent algorithm to optimize the parameters of GMM. The experiment results based on the fuel-level data of vehicles collected over one month prove that the modified GMM is superior to GMM-expectation maximization on fuel-level data, and the proposed method is effective for cleaning and repairing outliers of fuel-level data. <s> BIB011
Firstly, we summarize the methods described above in Table BIB004 . In fact, Bayesian prediction model is a technique based on Bayesian statistics. The Bayesian prediction model utilizes model information, data information, and prior information, so the prediction effect is good, there this model is widely used, including in the field of time series data cleaning. Wang et al. BIB008 establish a cost model for Bayesian analysis which is used to analyze errors in the data. Bergman et al. BIB009 consider the user's participation and use the user's feedback on the query results to clean the data. Mayfield et al. BIB007 propose a more complex relationship-dependent network (RDN BIB006 ) model to model the probability relationships between attributes. The difference between RDN and traditional relational dependencies (such as Bayesian networks BIB005 ) is that RDNs can contain ring structures. The method iteratively cleans the data set and observes the change in the probability distribution of the data set before and after each wash. When the probability distribution of the data set converges, the cleaning process is aborted. Zhou and Tung BIB010 argue a technique for accelerating the learning of Gaussian models via using GPU. The article believes that in the case of excessive data, it is not necessary to use all the data to learn the model. Also, the author provides a method of automatic tuning. In order to clean and repair fuel level data, Tian et al. BIB011 propose a modified Gaussian mixture model (GMM) based on the synchronous iteration method, which uses the particle swarm optimization algorithm and the steepest descent algorithm to optimize the parameters of GMM and uses linear interpolation-based algorithm to correct data errors. Shumway and Stoffer BIB003 use the EM BIB001 algorithm combined with the spatial state model  , BIB002 to predict and smooth the time series.