Big Data Quality: A systematic literature review and future research directions <s> Batch Processing Methods <s> Data preprocessing and cleansing play a vital role in data mining by ensuring good quality of data. Data-cleansing tasks include imputation of missing values, identification of outliers, and identification and correction of noisy data. In this paper, we present a novel technique called A Fuzzy Expectation Maximization and Fuzzy Clustering-based Missing Value Imputation Framework for Data Pre-processing (FEMI). It imputes numerical and categorical missing values by making an educated guess based on records that are similar to the record having a missing value. While identifying a group of similar records and making a guess based on the group, it applies a fuzzy clustering approach and our novel fuzzy expectation maximization algorithm. We evaluate FEMI on eight publicly available natural data sets by comparing its performance with the performance of five high-quality existing techniques, namely EMI, GkNN, FKMI, SVR and IBLLS. We use thirty-two types (patterns) of missing values for each data set. Two evaluation criteria namely root mean squared error and mean absolute error are used. Our experimental results indicate (according to a confidence interval and $$t$$t test analysis) that FEMI performs significantly better than EMI, GkNN, FKMI, SVR, and IBLLS. <s> BIB001 </s> Big Data Quality: A systematic literature review and future research directions <s> Batch Processing Methods <s> Although most NoSQL Data Stores are schema-less, information on the structural properties of the persisted data is nevertheless essential during application development. Otherwise, accessing the data becomes simply impractical. In this paper, we introduce an algorithm for schema extraction that is operating outside of the NoSQL data store. Our method is specifically targeted at semi-structured data persisted in NoSQL stores, e.g., in JSON format. Rather than designing the schema up front, extracting a schema in hindsight can be seen as a reverse-engineering step. Based on the extracted schema information, we propose set of similarity measures that capture the degree of heterogeneity of JSON data and which reveal structural outliers in the data. We evaluate our implementation on two real-life datasets: a database from the Wendelstein 7-X project and Web Performance Data. <s> BIB002 </s> Big Data Quality: A systematic literature review and future research directions <s> Batch Processing Methods <s> The data cleaning of electrical power big data can improve the correctness, the completeness, the consistency and the reliability of the data. Aiming at the difficulties of the extracting of the unified anomaly detection pattern and the low accuracy and continuity of the anomaly data correction in the process of the electrical power big data cleaning, the data cleaning model of the electrical power big data based on Spark is proposed. Firstly, the normal clusters and the corresponding boundary samples are obtained by the improved CURE clustering algorithm. Then, the anomaly data identification algorithm based on boundary samples is designed. Finally, the anomaly data modification is realized by using exponential weighting moving mean value. The high efficiency and accuracy is proved by the experiment of the data cleaning of the wind power generation monitoring data from the wind power station. <s> BIB003 </s> Big Data Quality: A systematic literature review and future research directions <s> Batch Processing Methods <s> We address the problem of dealing with inconsistencies in fusion of big data sources using Resource Description Framework (RDF) and ontologies. We propose a scalable approach ensuring data quality for query answering over big RDF data in a distributed way on a Spark ecosystem. In so doing, the cleaning inconsistent big RDF data approach is built on the following steps (1) modeling consistency rules to detect the inconsistency triples even if it is implicitly hidden including inference and inconsistent rules (2) detecting inconsistency through rule evaluation based on Apache Spark framework to discover the minimally sub-set of inconsistent triples (3) cleaning the inconsistency through finding the best repair for consistent query answering. <s> BIB004 </s> Big Data Quality: A systematic literature review and future research directions <s> Batch Processing Methods <s> The concept of Top-N local outlier that focuses on the detection of the N points with the largest Local Outlier Factor (LOF) score has been shown to be very effective for identifying outliers in big datasets. However, detecting Top-N local outliers is computationally expensive, since the computation of LOF scores for all data points requires a huge number of high complexity k-nearest neighbor (kNN) searches. In this work, we thus present the first distributed solution to tackle this problem of Top-N local outlier detection (DTOLF). First, DTOLF features an innovative safe elimination strategy that efficiently identifies dually-safe points, namely those that are guaranteed to (1) not be classified as Top-N outliers and (2) not be needed as neighbors of points residing on other machines. Therefore, it effectively minimizes both the processing and communication costs of the Top-N outlier detection process. Further, based on the well-accepted observation that strong correlations among attributes are prevalent in real world datasets, we propose correlation-aware optimization strategies that ensure the effectiveness of grid-based partitioning and of the safe elimination strategy in multi-dimensional datasets. Our extensive experimental evaluation on OpenStreetMap, SDSS, and TIGER datasets demonstrates the effectiveness of DTOLF — up to 10 times faster than the alternative methods and scaling to terabyte level datasets. <s> BIB005 </s> Big Data Quality: A systematic literature review and future research directions <s> Batch Processing Methods <s> Abstract Big data changed the way in which we collect and analyze data. In particular, the amount of available information is constantly growing and organizations rely more and more on data analysis in order to achieve their competitive advantage. However, such amount of data can create a real value only if combined with quality: good decisions and actions are the results of correct, reliable and complete data. In such a scenario, methods and techniques for the Data Quality assessment can support the identification of suitable data to process. If for traditional database numerous assessment methods are proposed, in the Big Data scenario new algorithms have to be designed in order to deal with novel requirements related to variety, volume and velocity issues. In particular, in this paper we highlight that dealing with heterogeneous sources requires an adaptive approach able to trigger the suitable quality assessment methods on the basis of the data type and context in which data have to be used. Furthermore, we show that in some situations it is not possible to evaluate the quality of the entire dataset due to performance and time constraints. For this reason, we suggest to focus the Data Quality assessment only on a portion of the dataset and to take into account the consequent loss of accuracy by introducing a confidence factor as a measure of the reliability of the quality assessment procedure. We propose a methodology to build a Data Quality adapter module, which selects the best configuration for the Data Quality assessment based on the user main requirements: time minimization, confidence maximization, and budget minimization. Experiments are performed by considering real data gathered from a smart city case study. <s> BIB006
This section describes the papers that use batch processing model to improve the quality of big data. As indicated in Figure 8 , based on the primary purpose of the proposed methods, we have divided these works into four categories: outlier detection, evaluation, cleaning, and review papers. In Table 4 the list of selected papers, in terms of each technique in the batch processing category is provided. BIB001 , BIB003 Rule-based BIB004 -  Distance-based BIB002 - BIB005 Instance-based  -  Schema-based BIB006 -  Next, we briefly review the papers in each category.