Big Data Quality: A systematic literature review and future research directions <s> Model-based techniques <s> The deployment of environmental sensors has generated an interest in real-time applications of the data they collect. This research develops a real-time anomaly detection method for environmental data streams that can be used to identify data that deviate from historical patterns. The method is based on an autoregressive data-driven model of the data stream and its corresponding prediction interval. It performs fast, incremental evaluation of data as it becomes available, scales to large quantities of data, and requires no pre-classification of anomalies. Furthermore, this method can be easily deployed on a large heterogeneous sensor network. Sixteen instantiations of this method are compared based on their ability to identify measurement errors in a windspeed data stream from Corpus Christi, Texas. The results indicate that a multilayer perceptron model of the data stream, coupled with replacement of anomalous data points, performs well at identifying erroneous data in this data stream. <s> BIB001 </s> Big Data Quality: A systematic literature review and future research directions <s> Model-based techniques <s> Wireless sensor network WSN applications require efficient, accurate and timely data analysis in order to facilitate near real-time critical decision-making and situation awareness. Accurate analysis and decision-making relies on the quality of WSN data as well as on the additional information and context. Raw observations collected from sensor nodes, however, may have low data quality and reliability due to limited WSN resources and harsh deployment environments. This article addresses the quality of WSN data focusing on outlier detection. These are defined as observations that do not conform to the expected behaviour of the data. The developed methodology is based on time-series analysis and geostatistics. Experiments with a real data set from the Swiss Alps showed that the developed methodology accurately detected outliers in WSN data taking advantage of their spatial and temporal correlations. It is concluded that the incorporation of tools for outlier detection in WSNs can be based on current statistical methodology. This provides a usable and important tool in a novel scientific field. <s> BIB002 </s> Big Data Quality: A systematic literature review and future research directions <s> Model-based techniques <s> In recent years, RFID technologies have been used in many applications, such as inventory checking and object tracking. However, raw RFID data are inherently unreliable due to physical device limitations and different kinds of environmental noise. Currently, existing work mainly focuses on RFID data cleansing in a static environment (e.g. inventory checking). It is therefore difficult to cleanse RFID data streams in a mobile environment (e.g. object tracking) using the existing solutions, which do not address the data missing issue effectively. In this paper, we study how to cleanse RFID data streams for object tracking, which is a challenging problem, since a significant percentage of readings are routinely dropped. We propose a probabilistic model for object tracking in a mobile environment. We develop a Bayesian inference based approach for cleansing RFID data using the model. In order to sample data from the movement distribution, we devise a sequential sampler that cleans RFID data with high accuracy and efficiency. We validate the effectiveness and robustness of our solution through extensive simulations and demonstrate its performance by using two real RFID applications of human tracking and conveyor belt monitoring. <s> BIB003 </s> Big Data Quality: A systematic literature review and future research directions <s> Model-based techniques <s> Low quality sensor data limits WSN capabilities for providing reliable real-time situation-awareness. Outlier detection is a solution to ensure the quality of sensor data. An effective and efficient outlier detection technique for WSNs not only identifies outliers in a distributed and online manner with high detection accuracy and low false alarm, but also satisfies WSN constraints in terms of communication, computational and memory complexity. In this paper, we take into account the correlation between sensor data attributes and propose two distributed and online outlier detection techniques based on a hyperellipsoidal one-class support vector machine (SVM). We also take advantage of the theory of spatio-temporal correlation to identify outliers and update the ellipsoidal SVM-based model representing the changed normal behavior of sensor data for further outlier identification. Simulation results show that our adaptive ellipsoidal SVM-based outlier detection technique achieves better detection accuracy and lower false alarm as compared to existing SVM-based techniques designed for WSNs. <s> BIB004 </s> Big Data Quality: A systematic literature review and future research directions <s> Model-based techniques <s> Wireless sensor networks (WSNs) are increasingly used as platforms for collecting data from unattended environments and monitoring important events in phenomena. However, sensor data is affected by anomalies that occur due to various reasons, such as, node software or hardware failures, reading errors, unusual events, and malicious attacks. Therefore, effective, efficient, and real time detection of anomalous measurement is required to guarantee the quality of data collected by these networks. In this paper, two efficient and effective anomaly detection models PCCAD and APCCAD are proposed for static and dynamic environments, respectively. Both models utilize the One-Class Principal Component Classifier (OCPCC) to measure the dissimilarity between sensor measurements in the feature space. The proposed APCCAD model incorporates an incremental learning method that is able to track the dynamic normal changes of data streams in the monitored environment. The efficiency and effectiveness of the proposed models are demonstrated using real life datasets collected by real sensor network projects. Experimental results show that the proposed models have advantages over existing models in terms of efficient utilization of sensor limited resources. The results further reveal that the proposed models achieve better detection effectiveness in terms of high detection accuracy with low false alarms especially for dynamic environmental data streams compared to some existing models. <s> BIB005 </s> Big Data Quality: A systematic literature review and future research directions <s> Model-based techniques <s> Trust and reputation systems are widely employed in WSNs to help decision making processes by assessing trustworthiness of sensors as well as the reliability of the reported data. Iterative filtering (IF) algorithms hold great promise for such a purpose; they simultaneously estimate the aggregate value of the readings and assess the trustworthiness of the nodes. Such algorithms, however, operate by batch processing over a widow of data reported by the nodes, which represents a difficulty in applications involving streaming data. In this paper, we propose STRIF (Streaming IF) which extends IF algorithms to data streaming by leveraging a novel method for updating the sensors' variances. We compare the performance of STRIF algorithm to several batch processing IF algorithms through extensive experiments across a wide variety of configurations over both real-world and synthetic datasets. Our experimental results demonstrate that STRIF can process data streams much more efficiently than the batch algorithms while keeping the accuracy of the data aggregation close to that of the batch IF algorithm. <s> BIB006 </s> Big Data Quality: A systematic literature review and future research directions <s> Model-based techniques <s> Abstract Vast and ever increasing quantities of data are produced by sensors in the Internet of Things (IoT). The quality of this data can be very variable due to problems with sensors, incorrect calibration etc. Data quality can be greatly enhanced by cleaning the data before it reaches its end user. This paper reports on the construction of a distributed cleaning system (DCS) to clean data streams in real-time for an environmental case-study. A combination of declarative and statistical model based cleaning methods are applied and initial results are reported. <s> BIB007 </s> Big Data Quality: A systematic literature review and future research directions <s> Model-based techniques <s> Despite advances in sensor technology, there are a number of problems that continue to require attention. Sensors fail due to low battery power, poor calibration, exposure to the elements and interference to name but a few factors. This can have a negative effect on data quality, which can however be improved by data cleaning. In particular, models can learn characteristics of data to detect and replace incorrect values. The research presented in this paper focuses on the building of models of environmental sensor data that can incorporate context awareness about the sampling locations. These models have been tested and validated both for static and streaming data. We show that contextual models demonstrate favourable outcomes when used to clean streaming data. <s> BIB008 </s> Big Data Quality: A systematic literature review and future research directions <s> Model-based techniques <s> An environmental monitoring process consists of a regular collection and analysis of sensors data streams. It aims to infer new knowledge about the environment, enabling the explorer to supervise the network and to take right decisions. Different data mining techniques are then applied to the collected data in order to infer aggregated statistics useful for anomalies detection and forecasting. The obtained results are closely dependent on the collected data quality. In fact, the data are often dirty, they contain noisy, erroneous and missing values. Poor data quality leads to defective and faulty results. One solution to overcome this problem will be presented in this paper. It consists of evaluating and improving the data quality, to be able to obtain reliable results. In this paper, we first introduce the data quality concept. Then, we discuss the existing related research studies. Finally, we propose a complete sensors data quality management system. <s> BIB009 </s> Big Data Quality: A systematic literature review and future research directions <s> Model-based techniques <s> With the increasing awareness of the harmful impacts of urban air pollution, air quality monitoring stations have been deployed in many metropolitan areas. These stations provide air quality data to the public. However, due to sampling device failures and data processing errors, missing data in air quality measurements is common. Data integrity becomes a critical challenge when such data are employed for public services. In this paper, we investigate the mathematical property of air quality measurements, and attempt to recover the missing data. First, we empirically study the low rank property of these measurements. Second, we formulate the low rank matrix completion (LRMC) optimization problem to reconstruct the missing air quality data. The problem is transformed using duality theory, and singular value thresholding (SVT) is employed to develop sub-optimal solutions. Third, to evaluate the performance of our methodology, we conduct a series of case studies including different types of missing data patterns. The simulation results demonstrate that the proposed SVT methodology can effectively recover missing air quality data, and outperform the existing Interpolation. Finally, we investigate the parameter sensitivity of SVT. Our study can serve as a guideline for missing data recovery in the real world. <s> BIB010 </s> Big Data Quality: A systematic literature review and future research directions <s> Model-based techniques <s> This paper presents a new framework to improve the quality of streaming synchrophasor measurements with the existence of missing data and bad data. The method exploits the low-rank property of the Hankel structure to identify and correct bad data, as well as to estimate and fill in the missing data. The method is advantageous compared to existing methods in the literature that only estimate missing data by leveraging the low-rank property of the synchrophasor data observation matrix. The proposed algorithm can efficiently differentiate event data from bad data, even in the existence of simultaneous and consecutive bad data. The algorithm has been verified through numerical experiments on recorded synchrophasor datasets. <s> BIB011
First, we explain the Model-based papers. There are six papers in this sub-category, which the common feature of all of them is to use data to build a model and predict future data based on the model to detect outliers. A spatial-temporal analysis based approach is presented in BIB002 . First, a temporal-based method (TOD) is proposed for detecting outliers. In this way, each sensor analyzes its data in terms of time and predicts the next value. Then, if the new data is far from the predicted value, it will be considered as an outlier. In the second method, data are analyzed for spatial analysis (SOD). In this way, the sensor data of the neighbors is examined and then the model is constructed and eventually the new value is predicted. In the third method, which combines the two previous methods (TSOD), the data in a sensor network is initially investigated by each sensor in terms of the temporal analysis. Subsequently, by receiving data from its neighbors, the spatial analysis is also done, and ultimately, the outlier is denoted. The evaluations are conducted in terms of accuracy, model complexity, and false positive rate. The results show that TOD, even though the communication complexity is low, obtains incorrect results, SOD, although predicting the correct results, but the communication complexity is exceptionally high, and in the TSOD, there is a right balance between these two criteria. Another model-based method is presented in BIB001 to detect anomalies in the data stream. The proposed method has four main steps; in the first step, a predictive model predicts new data. The prediction algorithm can be the nearest cluster (NC), single-layer linear network (LN) or multilayer perceptron (MLP). In these algorithms, there are some q data for constructing a model and predicting new data and specifying a range. When viewing new data, that data is compared with the predicted value and the allowed range. If it is outside of this range, data is anomalous. Otherwise, the data is normal. In the following, the values of the q data must be updated to provide further data prediction. There are two strategies in this situation. If we add the new anomaly data to the data set, the strategy is AD, and if we add the predicted value to the data set, then we have adopted the ADAM. In order to evaluate, wind speed data has been used, and q is equal to 30, that is, from the last 30 data, the new data is predicted. The efficiency and accuracy of the mentioned algorithms have been evaluated. The ADAM strategy has been able to reduce the false positive rate of algorithms LN and MLP, but using this strategy has a negative impact on the performance of algorithms. In BIB006 , the method presented in  is developed. In  , the IF (Iterative Filtering) algorithm is used to model data. This algorithm predicts new data, along with weight, based on a data model. Since IF-based algorithms use the entire data in order to improve the prediction accuracy, they are suitable for static data. This paper attempts to apply this method to streaming data. For this purpose, the first window data is used by the IF algorithm to construct the model. Then, if the new data differs from the measured variance, it is an outlier. In addition, it is said that if the number of outliers exceeds the normal data, the model is wrong and should be updated. The prediction accuracy, as well as the number of reconstruction of the model in the proposed model and the IF, have been evaluated. The results show that the proposed method has been able to have a better result in the streaming environment. PCA based methods are presented in  , BIB005 , which have mutual authors, to detect the outliers of each sensor. In  first, each sensor builds its model with a PCA algorithm locally. Then the model is sent to cluster head, and the cluster head combines local models and builds a global model then sends it back to the sensors. Each sensor updates its model after receiving the global model, and when the new data comes, the data that is far from the threshold is considered as an outlier. In BIB005 only local model is provided, and there is no global and distributed model. The accuracy, the false positive rate, and the detection rate are used for evaluation, which results in many tests show an accuracy of at least 97%. Paper BIB004 , similar to  , uses local and global models. In this paper, the model is based on the SVM algorithm, and, unlike  , which transmits the model from the sensors to the cluster head, the model is sent from each sensor to its neighbors. The detection rate and the false positive rate are evaluated in this paper, which has a good false positive rate and a better detection rate than other methods. Next, we explain the Learning-based methods. The latest sub-category on cleaning is Model-based papers. There are six papers in this subcategory, which are explained below. A method of tracking objects is presented in BIB003 . In this paper, one of the model-based methods is presented. First, a baseline algorithm is introduced, in which, according to the current position of the object, a model is constructed, and future locations are predicted. Then it is determined that if the data is corrupted, the prediction process will encounter an error. As a result, a solution to this problem is presented. In this method, in addition to the available information, a radius is considered for each object, depending on which radius of motion of each object is determined. As a result, the process of predicting the next location of each object can be done correctly. It is also said that the exact amount of radius is difficult to determine because if a large radius is selected, the prediction of the next location is difficult and if small, it may not be accurate to predict. Finally, this method is compared with the baseline algorithm. The effect of missing rate and moving speed on the accuracy of prediction and run-time are estimated. The results indicate that the proposed method has been able to control and reduce the impact of missing data in the prediction process. In BIB007 , another model-based approach is presented. First, the data value whose error is recognizable is determined. Then a statistical model of the existing data is created which is used to estimate the noisy data. In order to implement both R and Spark are used. Different algorithms are used to construct the model and the most accuracy one is chosen. In the evaluation section, the accuracy of the prediction has been measured, and the 87% prediction is obtained. Also, in BIB008 , presented by the same authors, the model, in addition to the data that receives from each sensor, is made up of other information, such as wind speed and the distance of buses with specified locations, taken from external sources. The reference BIB009 is also a model-based approach in which an abstract framework is provided to examine the quality of wireless sensor data as well as its cleaning. The authors say that the framework is part of a project that will be carried out later. In this framework, after collecting data from sensors, outliers are first detected, and then the new value is determined by the smoothing models. There is no description of what algorithms are used in each section. In order to clean the data, a low-rank matrix used in BIB010 and BIB011 . The paper BIB010 presents a method for cleaning the missing data in air quality data sets. For this purpose, since the data is stored in the matrix, a mathematically based method called low-rank matrix is used. So the issue becomes an optimization problem. If the given and the optimal matrices are close together, the rank of the given matrix is less and, therefore, is complete. The SVT algorithm is used to recover the matrix. This algorithm continues until it reaches the optimal answer and finally delivers the complete matrix. The simulation results show that the proposed method can effectively recover and improve the missing data in the air quality data set.