Big Data Quality: A systematic literature review and future research directions <s> Learning based techniques <s> Data preprocessing and cleansing play a vital role in data mining by ensuring good quality of data. Data-cleansing tasks include imputation of missing values, identification of outliers, and identification and correction of noisy data. In this paper, we present a novel technique called A Fuzzy Expectation Maximization and Fuzzy Clustering-based Missing Value Imputation Framework for Data Pre-processing (FEMI). It imputes numerical and categorical missing values by making an educated guess based on records that are similar to the record having a missing value. While identifying a group of similar records and making a guess based on the group, it applies a fuzzy clustering approach and our novel fuzzy expectation maximization algorithm. We evaluate FEMI on eight publicly available natural data sets by comparing its performance with the performance of five high-quality existing techniques, namely EMI, GkNN, FKMI, SVR and IBLLS. We use thirty-two types (patterns) of missing values for each data set. Two evaluation criteria namely root mean squared error and mean absolute error are used. Our experimental results indicate (according to a confidence interval and $$t$$t test analysis) that FEMI performs significantly better than EMI, GkNN, FKMI, SVR, and IBLLS. <s> BIB001 </s> Big Data Quality: A systematic literature review and future research directions <s> Learning based techniques <s> The data cleaning of electrical power big data can improve the correctness, the completeness, the consistency and the reliability of the data. Aiming at the difficulties of the extracting of the unified anomaly detection pattern and the low accuracy and continuity of the anomaly data correction in the process of the electrical power big data cleaning, the data cleaning model of the electrical power big data based on Spark is proposed. Firstly, the normal clusters and the corresponding boundary samples are obtained by the improved CURE clustering algorithm. Then, the anomaly data identification algorithm based on boundary samples is designed. Finally, the anomaly data modification is realized by using exponential weighting moving mean value. The high efficiency and accuracy is proved by the experiment of the data cleaning of the wind power generation monitoring data from the wind power station. <s> BIB002 </s> Big Data Quality: A systematic literature review and future research directions <s> Learning based techniques <s> AbstractWireless sensor network (WSN) has been widely used in the areas such as health care and industrial monitoring. However, WSN systems still suffer from inevitable problems of communication interference and data failure. In this paper, an improved Hidden Markov Model (HMM) is proposed to enhance the quality of WSN sensor data. This model can be used to recover the missing data and predict the upcoming data in order to improve the data integrity and reliability ultimately. K-means clustering is firstly used to group sensor data series on the basis of different patterns. Next, Particle Swarm Optimization is applied for optimizing HMM parameters, which is enhanced by a hybrid mutation strategy. Experiments on two real data-sets show that the proposed approach can outperform the baseline models (Naive Bayes, Grey System, BP-Neural Network and Traditional HMM) on precision of both single-step prediction and multiple-step prediction. The results also demonstrate that the proposed approach can improve data ... <s> BIB003 </s> Big Data Quality: A systematic literature review and future research directions <s> Learning based techniques <s> In the era of big data and Internet of things, massive sensor data are gathered with Internet of things. Quantity of data captured by sensor networks are considered to contain highly useful and valuable information. However, for a variety of reasons, received sensor data often appear abnormal. Therefore, effective anomaly detection methods are required to guarantee the quality of data collected by those sensor nodes. Since sensor data are usually correlated in time and space, not all the gathered data are valuable for further data processing and analysis. Preprocessing is necessary for eliminating the redundancy in gathered massive sensor data. In this paper, the proposed work defines a sensor data preprocessing framework. It is mainly composed of two parts, i.e., sensor data anomaly detection and sensor data redundancy elimination. In the first part, methods based on principal statistic analysis and Bayesian network is proposed for sensor data anomaly detection. Then, approaches based on static Bayesian network (SBN) and dynamic Bayesian networks (DBNs) are proposed for sensor data redundancy elimination. Static sensor data redundancy detection algorithm (SSDRDA) for eliminating redundant data in static datasets and real-time sensor data redundancy detection algorithm (RSDRDA) for eliminating redundant sensor data in real-time are proposed. The efficiency and effectiveness of the proposed methods are validated using real-world gathered sensor datasets. <s> BIB004 </s> Big Data Quality: A systematic literature review and future research directions <s> Learning based techniques <s> With the development of Smart Grid Technology, more and more electric power sensor data are utilized in various electric power systems. To guarantee the effectiveness of such systems, it is necessary to ensure the quality of electric power sensor data, especially when the scale of electric power sensor data is large. In the field of large-scale electric power sensor data cleaning, the computational efficiency and accuracy of data cleaning are two vital requirements. In order to satisfy these requirements, this paper presents an electric power sensor data oriented data cleaning solution, which is composed of a data cleaning framework and a data cleaning method. Based on Hadoop, the given framework is able to support large-scale electric power sensor data acquisition, storage and processing. Meanwhile, the proposed method which achieves outlier detection and reparation is implemented on the basis of a time-relevant k-means clustering algorithm in Spark. The feasibility and effectiveness of the proposed method is evaluated on a data set which originates from charging piles. Experimental results show that the proposed data cleaning method is able to improve the data quality of electric power sensor data by finding and repairing most outliers. For large-scale electric power sensor data, the proposed data cleaning method has high parallel performance and strong scalability. <s> BIB005
There are five papers in this sub-category, the first four papers are based on clustering, and the last one is based on classification. A distributed clustering approach is presented in  , which has six components. The first one is called System Adapter and is responsible for registering and executing the system. Event Extractor is the second component and is responsible for reading incoming events, setting a timestamp for each data and adding metadata to it. The third component, Central Splitter, distributes the data to existing nodes in order to make it scalable. Next, the k-means clustering algorithm is responsible for grouping the data and finding the cluster center. Then, the Markov model is used to detect outliers. This algorithm considers data outside of each cluster as an outlier. Finally, in Ordering Operator, the collected data is merged and then sent to the sink. Both parallel and sequential implementations are considered. Latency and throughput are evaluated, and the results indicate that the distributed algorithm has lower latency and higher efficiency. The paper  also used the k-means clustering and Markov model to detect anomalies like  . The difference is that the input data is in triple format (RDF) and must first be parsed and then enter the window. In the end, the output must also be transformed into a triple form. The latency of the proposed method in multi-threaded mode has been evaluated that the system with four threads has been able to have good performance. Another clustering-based approach is presented in BIB003 . First, the received signal is converted into a data format that can be preprocessed. Then the k-means algorithm groups data, moreover the HMM and PSO algorithms are used to optimize the parameters, and ultimately, new data is predicted. The average prediction rate is evaluated, and results show enhancement of the proposed method in prediction accuracy and practical feasibility. The last clustering method, which has several innovations, is presented in  . First, an adaptive distance-based algorithm called SOStream is presented, which is an efficient streaming algorithm that is capable of distinguishing overlapping cluster in an online manner. Also, an online spatiotemporal clustering algorithm, called ASMM is proposed, which is capable of dynamically adjusting its clustering structure based on the incoming data. In this clustering, by observing the new data, its Euclidean distance is compared with the centroids of the existing cluster, and the closest one is chosen. If the distance of the selected centroids is less than the desired threshold, then the data is transferred to that cluster and otherwise returned to the input buffer. If the number of data transfers to the buffer exceeds the threshold, then that data is considered as an outlier. The algorithm is tested with different real-life datasets. The results show that ASMM outperformed both EMM and LOF regarding different confusion matrix measures. In BIB004 , which is a classification method, eigenvectors and eigenvalues are obtained by PCA, and data with a low eigenvalue is considered as an outlier. Then, the sensors that generate the outlier must be detected, which Bayesian Network is used for this purpose to identify the relationship between the sensors and find the corresponding sensors. The dependencies between sensors and anomaly detection rate are evaluated, and the results show that the proposed method can improve the precision of anomaly detection. There is one paper in this sub-category, which is BIB005 . In these papers, a learning-based cleaning framework is presented that has three primary modules: storage, computing, and data collection. In the storage module, MySQL is used to store structured data, HBase to store unstructured data and HDFS to store the data in Hadoop. The computing module is a combination of Yarn, Map Reduction and Spark for distributed computing. The collection module has been used to collect data from sensors, which consists of four components, called communication service, data transmission service, structured data accessing service, and un-structured data accessing service. Kafka is also used to transfer data from the collection module to the calculation module. The proposed cleaning method consists of three parts: time-relevant k-means clustering, outlier detection, and data warning and reparation. First, k-means clustering algorithm groups data in distinct clusters. In each cluster, the values of max and min are introduced as the cluster boundary. The data outside this range is called an outlier. Then the outlier data is replaced by the min or mean. In order to evaluate, recall, run-time, speed-up and scalability of the method has been evaluated. The results show that the proposed method has been able to increase the quality of power data by detecting and replacing most outliers. In this section, we briefly review two papers that employ clustering to develop a learning technique for the task of big data cleaning. In BIB001 , a fuzzy-based clustering algorithm is proposed to clean the incomplete data. After normalization, the data is divided into two categories, complete and incomplete data. A fuzzy clustering algorithm is executed on complete data, and cluster heads are identified. Then, the membership degree of all data to all clusters is computed. Subsequently, the fuzzy algorithm is executed on incomplete data to determine the appropriate values to reduce data incompleteness based on membership degrees. The performance, the proper number of clusters and the runtime of the proposed method are compared with five other methods, and the experimental results indicate that the proposed method outperforms the other five techniques. Another clustering based method is presented in BIB002 . In the proposed method, by the Cure clustering, the center of the clusters as well as the borderline data are detected. In order to determine the cluster boundary points, the center of each cluster is first defined, and the distance of each data with the center is determined, then the farthest data is a border point. When analyzing the new data, the data is examined by the cluster boundary, and if exceeded, it is considered as an outlier. Since big electrical data is being generated over time, it changes slowly. Therefore, in order to handle the outliers, we can use the centroid of the cluster for corresponding outliers. In order to evaluate, the authors use Spark, six Ubuntu servers, one for master and others for slave servers. In this paper wind power data is used for examination. The detection rate, the false positive rate, and the performance of the proposed method are evaluated with three other methods, which the results indicate that the proposed method has a higher detection rate and a lower false positive rate compared to other methods.