Big Data Quality: A systematic literature review and future research directions <s> Cleaning Task <s> AbstractWireless Sensor Networks (WSNs) are widely used for monitoring physical happenings of the environment. However, the data gathered by the WSNs may be inaccurate and unreliable due to power exhaustion, noise and other reasons. Unnecessary data such as erroneous data and redundant data transmission causes a lot of extra energy consumption. To improve the data reliability and reduce the energy consumption, we proposed an in-network processing architecture for data cleaning, which divides the task into four stages implemented in different nodes respectively. This strategy guaranteed the cleaning algorithms were computationally lightweight in local nodes and energy-efficient due to almost no communication overhead. In addition, we presented the detection algorithms for data faults and event outliers, which were conducted by utilizing the related attributes from the local sensor node and the cooperation with its relaying neighbor. Experiment results show that our proposed approach is accurate and energy-... <s> BIB001 </s> Big Data Quality: A systematic literature review and future research directions <s> Cleaning Task <s> With the evolution of data and increasing popularity of IoT (Internet of Things), stream data mining has gained immense popularity. Researchers and developers are trying to analyze data patterns obtained from various devices. Stream data have several characteristics, the most important being its huge volume and high velocity. Although, a lot of research is being conducted in order to develop more efficient stream data mining techniques, pre-processing of stream data is an area that is under-studied. Real time applications generate data which is rather noisy and contain missing values. Apart from this, there is the issue of data evolution, which is a concern when dealing with stream data. To deal with the evolution of data, the proposed solution offers a hybrid of preprocessing techniques which are adaptive in nature. As a result of the study, an adaptive preprocessing and learning approach is implemented. The case study with sensor weather data demonstrates the results and accuracy of the proposed solution. <s> BIB002 </s> Big Data Quality: A systematic literature review and future research directions <s> Cleaning Task <s> Existing scalable data cleaning approaches have focused on batch data cleaning. However, batch data cleaning is not suitable for streaming big data systems, in which dynamic data is generated continuously. Despite the increasing popularity of stream-processing systems, few stream data cleaning techniques have been proposed so far. In this paper, we bridge this gap by addressing the problem of rule-based stream data cleaning, which sets stringent requirements on latency, rule dynamics and ability to cope with the continuous nature of data streams. We design a system, called Bleach, which achieves real-time violation detection and data repair on a dirty data stream. Bleach relies on efficient, compact and distributed data structures to maintain the necessary state to repair data. Additionally, it supports rule dynamics and uses a "cumulative" sliding window operation to improve cleaning accuracy. We evaluate a prototype of Bleach using both synthetic and real data streams and experimentally validate its high throughput, low latency and high cleaning accuracy, which are preserved even with rule dynamics. <s> BIB003 </s> Big Data Quality: A systematic literature review and future research directions <s> Cleaning Task <s> Since a low-quality data may influence the effectiveness and reliability of applications, data quality is required to be guaranteed. Data quality assessment is considered as the foundation of the promotion of data quality, so it is essential to access the data quality before any other data related activities. In the electric power industry, more and more electric power data is continuously accumulated, and many electric power applications have been developed based on these data. In China, the power grid has many special characteristic, traditional big data assessment frameworks cannot be directly applied. Therefore, a big data framework for electric power data quality assessment is proposed. Based on big data techniques, the framework can accumulate both the real-time data and the history data, provide an integrated computation environment for electric power big data assessment, and support the storage of different types of data. <s> BIB004
The methods used to clean data stream are discussed in this section. Papers in this field are divided into three sub-categories: Learning-based, Context-aware, and Model-based. In this section, papers that focus on the cleaning task are presented, which are divided into two sub-categories: Learning-based and Rule-based. Hybrid cleaning is also presented in four papers, each of which is described in detail below. In BIB002 , a hybrid method for data preprocessing is presented. The k-nearest neighbor algorithm runs on historical data and is applied to a window of a data stream. After cleaning the data stream by MSSA algorithm, the model, adaptively, is updated with cleaned data in order to make the next prediction more accurate. The prediction accuracy of the proposed method is evaluated, and the results indicate a precise prediction of the algorithm when the method uses an adaptive algorithm for predictive purposes. In BIB003 a distributed rule-based method is presented to clean streaming data. The proposed method has two basic modules. First, the detection module and the second, the cleaning module. A set of rules is defined in the data. The number of detect workers is defined by the number of rules. The input data is entered by the router to the corresponding detect worker. Each detect worker is responsible for investigating the violation of the corresponding rule. In each worker, historical data is also available to compare the new data with them, and if there is a violation, it will be announced. Then the noisy data will enter to the cleaning module. In this module, the equivalence class algorithm is used, in order to clean data. In this algorithm, the same data is placed in a group, and the majority value is introduced as new data. The throughput and time delay of the method is one of the experiments that the authors of this paper have done and the results show that the proposed method has been able to provide a little delay, in addition to the accuracy of the results. Paper BIB004 also provides a framework for assessing and cleaning Chinese electricity data. This framework has three main modules. First is a data collection module where data is accumulated from sensors, power grids, and databases. Second, the storage module, this module is designed to store different types of data in an integrated storage environment, which includes Oracle, HBase, and HDFS. The third is the calculation module in which historical and real-time data are processed. In addition to data processing, this module also assesses the quality of the data as well as its cleaning. The proposed framework has been implemented, but its evaluation results are not presented in the paper. Another hybrid method for detecting and cleaning sensor data is presented in BIB001 . At first, each sensor individually analyzes time series data, and those who are not similar to the majority are marked as an outlier. These data may be identified as normal data when compared with other sensor data. So in the second step, each sensor compares its neighbors' data with its own data to detect the event outliers, and if a conflict is observed, the data is tagged. Thirdly, the data is sent to the sink, and in the sink, after analyzing the correlation between all sensor data, the smoothing process is done. Moreover, the cleaned data is transmitted to the server for processing phase. The proposed method is compared with other methods, and the results indicate that the algorithm is more accurate and more efficient.