Big Data Quality: A systematic literature review and future research directions <s> Batch Review Papers <s> Organizations are nowadays immersed in the Big Data Era. Beyond the hype of the concept of Big Data, it is true that something in the way of doing business is really changing. Although some challenges keep being the same as for regular data, with big data, the focus has changed. The reason is due to Big Data is not only data, but also a complete framework including data themselves, storage, formats, and ways of provisioning, processing and analytics. A challenge that becomes even trickier is the one concerning to the management of the quality of big data. More than ever the need for assessing the quality-in-use of big datasets gains importance since the real contribution – business value- of a dataset to a business can be only estimated in its context of use. Although there exists different data quality models to assess the quality of data there still lacks of a quality-in-use model adapted to big data. To fill this gap, and based on ISO 25012 and ISO 25024, we propose the 3Cs model, which is composed of three data quality dimensions for assessing the quality-in-use of big datasets: Contextual Consistency, Operational Consistency and Temporal Consistency. <s> BIB001 </s> Big Data Quality: A systematic literature review and future research directions <s> Batch Review Papers <s> In our Big Data era, data is being generated, collected and analyzed at an unprecedented scale, and data-driven decision making is sweeping through all aspects of society. Recent studies have shown that poor quality data is prevalent in large databases and on the Web. Since poor quality data can have serious consequences on the results of data analyses, the importance of veracity, the fourth ‘V’ of big data is increasingly being recognized. In this tutorial, we highlight the substantial challenges that the first three ‘V’s, volume, velocity and variety, bring to dealing with veracity in big data. Due to the sheer volume and velocity of data, one needs to understand and (possibly) repair erroneous data in a scalable and timely manner. With the variety of data, often from a diversity of sources, data quality rules cannot be specified a priori; one needs to let the “data to speak for itself” in order to discover the semantics of the data. This tutorial presents recent results that are relevant to big data quality management, focusing on the two major dimensions of (i) discovering quality issues from the data itself, and (ii) trading-off accuracy vs efficiency, and identifies a range of open problems for the community. <s> BIB002 </s> Big Data Quality: A systematic literature review and future research directions <s> Batch Review Papers <s> A USAF sponsored MITRE research team undertook four separate, domain-specific case studies about Big Data applications. Those case studies were initial investigations into the question of whether or not data quality issues encountered in Big Data collections are substantially different in cause, manifestation, or detection than those data quality issues encountered in more traditionally sized data collections. The study addresses several factors affecting Big Data Quality at multiple levels, including collection, processing, and storage. Though not unexpected, the key findings of this study reinforce that the primary factors affecting Big Data reside in the limitations and complexities involved with handling Big Data while maintaining its integrity. These concerns are of a higher magnitude than the provenance of the data, the processing, and the tools used to prepare, manipulate, and store the data. Data quality is extremely important for all data analytics problems. From the study's findings, the "truth about Big Data" is there are no fundamentally new DQ issues in Big Data analytics projects. Some DQ issues exhibit return-s-to-scale effects, and become more or less pronounced in Big Data analytics, though. Big Data Quality varies from one type of Big Data to another and from one Big Data technology to another. <s> BIB003 </s> Big Data Quality: A systematic literature review and future research directions <s> Batch Review Papers <s> Though the issues of data quality trace back their origin to the early days of computing, the recent emergence of Big Data has added more dimensions. Furthermore, given the range of Big Data applications, potential consequences of bad data quality can be for more disastrous and widespread. This paper provides a perspective on data quality issues in the Big Data context. it also discusses data integration issues that arise in biological databases and attendant data quality issues. <s> BIB004 </s> Big Data Quality: A systematic literature review and future research directions <s> Batch Review Papers <s> This article investigates the evolution of data quality issues from traditional structured data managed in relational databases to Big Data. In particular, the paper examines the nature of the relationship between Data Quality and several research coordinates that are relevant in Big Data, such as the variety of data types, data sources and application domains, focusing on maps, semi-structured texts, linked open data, sensor & sensor networks and official statistics. Consequently a set of structural characteristics is identified and a systematization of the a posteriori correlation between them and quality dimensions is provided. Finally, Big Data quality issues are considered in a conceptual framework suitable to map the evolution of the quality paradigm according to three core coordinates that are significant in the context of the Big Data phenomenon: the data type considered, the source of data, and the application domain. Thus, the framework allows ascertaining the relevant changes in data quality emerging with the Big Data phenomenon, through an integrative and theoretical literature review. <s> BIB005 </s> Big Data Quality: A systematic literature review and future research directions <s> Batch Review Papers <s> Data quality management systems are thoroughly researched topics and have resulted in many tools and techniques developed by both academia and industry. However, the advent of Big Data might pose some serious questions pertaining to the applicability of existing data quality concepts. There is a debate concerning the importance of data quality for Big Data; one school of thought argues that high data quality methods are essential for deriving higher level analytics while another school of thought argues that data quality level will not be so important as the volume of Big Data would be used to produce patterns and some amount of dirty data will not mask the analytic results which might be derived. This paper aims to investigate various components and activities forming part of data quality management such as dimensions, metrics, data quality rules, data profiling and data cleansing. The result list existing challenges and future research areas associated with Big Data for data quality management. <s> BIB006 </s> Big Data Quality: A systematic literature review and future research directions <s> Batch Review Papers <s> High-quality data are the precondition for analyzing and using big data and for guaranteeing the value of the data. Currently, comprehensive analysis and research of quality standards and quality assessment methods for big data are lacking. First, this paper summarizes reviews of data quality research. Second, this paper analyzes the data characteristics of the big data environment, presents quality challenges faced by big data, and formulates a hierarchical data quality framework from the perspective of data users. This framework consists of big data quality dimensions, quality characteristics, and quality indexes. Finally, on the basis of this framework, this paper constructs a dynamic assessment process for data quality. This process has good expansibility and adaptability and can meet the needs of big data quality assessment. The research results enrich the theoretical scope of big data and lay a solid foundation for the future by establishing an assessment model and studying evaluation algorithms. <s> BIB007 </s> Big Data Quality: A systematic literature review and future research directions <s> Batch Review Papers <s> Big Data refers to data volumes in the range of Exabyte (1018) and beyond. Such volumes exceed the capacity of current on-line storage and processing systems. With characteristics like volume, velocity and variety big data throws challenges to the traditional IT establishments. Computer assisted innovation, real time data analytics, customer-centric business intelligence, industry wide decision making and transparency are possible advantages, to mention few, of Big Data. There are many issues with Big Data that warrant quality assessment methods. The issues are pertaining to storage and transport, management, and processing. This paper throws light into the present state of quality issues related to Big Data. It provides valuable insights that can be used to leverage Big Data science activities. <s> BIB008 </s> Big Data Quality: A systematic literature review and future research directions <s> Batch Review Papers <s> The massive growth in the scale of data has been observed in recent years being a key factor of the Big Data scenario. Big Data can be defined as high volume, velocity and variety of data that require a new high-performance processing. Addressing big data is a challenging and time-demanding task that requires a large computational infrastructure to ensure successful data processing and analysis. The presence of data preprocessing methods for data mining in big data is reviewed in this paper. The definition, characteristics, and categorization of data preprocessing approaches in big data are introduced. The connection between big data and data preprocessing throughout all families of methods and big data technologies are also examined, including a review of the state-of-the-art. In addition, research challenges are discussed, with focus on developments on different big data framework, such as Hadoop, Spark and Flink and the encouragement in devoting substantial research efforts in some families of data preprocessing methods and applications on new big data learning paradigms. <s> BIB009 </s> Big Data Quality: A systematic literature review and future research directions <s> Batch Review Papers <s> With the fast advance of big data technology and analytics solutions, big data computing and service is becoming a very hot research and application subject in academic research, industry community, and government services. Nevertheless, there are increasing data quality problems resulting in erroneous data costs in enterprises and businesses. Current research seldom discusses how to effectively validate big data to ensure data quality. This paper provides informative discussions for big data validation and quality assurance, including the essential concepts, focuses, and validation process. Moreover, the paper presents a comparison among big data validation tools and several major players in industry are discussed. Furthermore, the primary issues, challenges, and needs are discussed. <s> BIB010 </s> Big Data Quality: A systematic literature review and future research directions <s> Batch Review Papers <s> With the rapid development of social networks, Internet of things, Cloud computing as well as other technologies, big data age is arriving. The increasing number of data has brought great value to the public and enterprises. Meanwhile how to manage and use big data better has become the focus of all walks of life. The 4V characteristics of big data have brought a lot of issues to the big data processing. The key to big data processing is to solve data quality issue, and to ensure data quality is a prerequisite for the successful application of big data technique. In this paper, we use recommendation systems and prediction systems as typical big data applications, and try to find out the data quality issues during data collection, data preprocessing, data storage and data analysis stages of big data processing. According to the elaboration and analysis of the proposed issues, the corresponding solutions are also put forward. Finally, some open problems to be solved in the future are also raised. <s> BIB011 </s> Big Data Quality: A systematic literature review and future research directions <s> Batch Review Papers <s> With the advances in communication technologies and the high amount of data generated, collected, and stored, it becomes crucial to manage the quality of this data deluge in an efficient and cost-effective way. The storage, processing, privacy and analytics are the main keys challenging aspects of Big Data that require quality evaluation and monitoring. Quality has been recognized by the Big Data community as an essential facet of its maturity. Yet, it is a crucial practice that should be implemented at the earlier stages of its lifecycle and progressively applied across the other key processes. The earlier we incorporate quality the full benefit we can get from insights. In this paper, we first identify the key challenges that necessitates quality evaluation. We then survey, classify and discuss the most recent work on Big Data management. Consequently, we propose an across-the-board quality management framework describing the key quality evaluation practices to be conducted through the different Big Data stages. The framework can be used to leverage the quality management and to provide a roadmap for Data scientists to better understand quality practices and highlight the importance of managing the quality. We finally, conclude the paper and point to some future research directions on quality of Big Data. <s> BIB012
There are thirteen papers in this section that expressing the challenges and issues in the area of big data quality are the common point of most of them. Some authors propose solutions and some else have provided an abstract framework. More explanation is as follows. There are various quality models available to assess the quality of the data. Researchers continue to suffer from the lack of a standardized model for the assessment of big data quality. So, in this paper BIB001 , a quality model for big data that has been claimed to be integrated with any big data project is introduced. This model places the quality dimensions in three categories Contextual Adequacy, which is used in a specific area, Operational Adequacy, which means accessibility of data, and Temporal Adequacy, The appropriate time to analyze data. Accordingly, the 15 dimensions of quality referred to in ISO/IEC 25012 have been assigned in these three categories. In this paper BIB008 , first, big data characteristics and their definitions are expressed. Then the challenges associated with big data, especially data quality, have been reported. For example, challenges such as heterogeneity, incompleteness, incompatibility, diversity of data sources, lack of structure, lack of standardization in data quality, security, authenticity, etc. are mentioned. The presence of preprocessing algorithms for data mining in big data are reviewed in this paper BIB009 . Definitions, characteristics, and categorization of big data methods are also outlined. The relationship between big data and preprocessing data is also investigated, including a review of the state-of-the-art. In this paper  , a preliminary study was initially presented to develop a conceptual model for describing the effects of data quality in the Big Data area. Also, the relationship between big data characteristics and data quality is discussed in the proposed model. This paper BIB010 offers informative discussions for big data validation and quality assurance, including basic concepts, and validation process. Some of these challenges are big data management and big data processing. Also, it provides a comparative on big data validation tools. Different tools are compared in terms of the operation environment, supported data source and essential data validation functions. This study BIB003 focuses on several factors that affect the quality of big data at various levels, including collection, processing, and storage. The purpose of these case studies is to provide information on issues related to the quality of the various data related to the design, implementation, and operation of large data plans. The authors have identified data quality issues during data acquisition, preprocessing, storage and analysis stages of big data processing in BIB011 . Some of these issues are Data Sparseness, Noise Data, Incomplete Data, Limitations of Storage Technology, Data Timeliness, Accuracy, and Scalability. In the end, there are also solutions to the challenges mentioned. For example, using clustering algorithms in order to complete data, Cloud Storage to deal with storage limitation and Spark platform to speed up the processing. This paper BIB004 provides a perspective on data quality challenges in big data. Five quality dimensions that are mentioned are accuracy, confidence, completeness, volume, and timeliness. In the following, some of the challenges in this area are mentioned. The paper BIB002 describes the general description and some tutorials of how to handle data quality in big data as well as existing problems such as how to assess data quality in a distributed and realtime manner. The authors of this paper BIB005 point to the evolution of data quality issues from structured data to big data. Also examines the relationship between data quality and topics that are related to big data such as data type diversity, data sources, semi-structured data, linked open data and sensors. Data quality management involves specifying dimensions, metrics, data quality rules, data profiling and, finally, data cleaning. This paper BIB006 discusses how data quality is managed, as well as the challenges in data quality associated with big data. These issues include computational complexity, volume, variety, and online cleaning. This paper BIB007 analyzes the characteristics of big data, also examines the quality challenges such as volume, variety, and velocity. Ultimately authors provide an abstract model for data quality assessment, which the goals and quality dimensions are first defined, then the indicators of each dimension are taken into account and when data comes, are cleaned, and the report is generated. In BIB012 , firstly, characteristics of big data and the relevant challenges are examined and then the recent methods for big data management are categorized and discussed from the perspective of Big Data Value Chain, Big Data Management & Characteristics, Big Data Problems & Data Quality Issues, Data Quality, and Big Data Applications and Quality Improvements. Finally, solutions have been proposed, including fast, continuous, and online data quality assessment.