Big Data Quality: A systematic literature review and future research directions <s> Introduction <s> The amount of data in our world has been exploding, and analyzing large data sets—so-called big data— will become a key basis of competition, underpinning new waves of productivity growth, innovation, and consumer surplus, according to research by MGI and McKinsey's Business Technology Office. Leaders in every sector will have to grapple with the implications of big data, not just a few data-oriented managers. The increasing volume and detail of information captured by enterprises, the rise of multimedia, social media, and the Internet of Things will fuel exponential growth in data for the foreseeable future. <s> BIB001 </s> Big Data Quality: A systematic literature review and future research directions <s> Introduction <s> In many companies, product management struggles in getting accurate customer feedback. Often, validation and confirmation of functionality with customers takes place only after the product has been deployed, and there are no mechanisms that help product managers to continuously learn from customers. Although there are techniques available for collecting customer feedback, these are typically not applied as part of a continuous feedback loop. As a result, the selection and prioritization of features becomes far from optimal, and product deviates from what the customers need. In this paper, we present a literature review of currently recognized techniques for collecting customer feedback. We develop a model in which we categorize the techniques according to their characteristics. The purpose of this literature review is to provide an overview of current software engineering research in this area and to better understand the different techniques that are used for collecting customer feedback. <s> BIB002 </s> Big Data Quality: A systematic literature review and future research directions <s> Introduction <s> The Data Warehousing Institute (TDWI) estimates that data quality problems cost U.S. businesses more than $600 billion a year. The problem with data is that its quality quickly degenerates over time. Experts say 2 percent of records in a customer file become obsolete in one month because customers die, divorce, marry, and move. In addition, data entry errors, system migrations, and changes in source systems, among other things, generate bucket loads of errors. More complex, as organizations fragment into different divisions and units, interpretations of data elements change to meet the local business needs. However, there are several ways that the Company should concern, such as to treat data as a strategic corporate resource; develop a program for managing data quality with a commitment from the top; and hire, train, or outsource experienced data quality professionals to oversee and carry out the program. The Organizations can sustain a commitment to managing data quality over time and adjust monitoring and cleansing processes to changes in the business and underlying systems by using the Commercial data quality tools. Data is a vital resource. Companies that invest proportionally to manage this resource will stand a stronger chance of succeeding in today's competitive global economy than those that squander this critical resource by neglecting to ensure adequate levels of quality. This paper reviews the characteristics of big data quality and the managing processes that are involved in it. <s> BIB003 </s> Big Data Quality: A systematic literature review and future research directions <s> Introduction <s> In recent years, big data have become a hot research topic. The increasing amount of big data also increases the chance of breaching the privacy of individuals. Since big data require high computational power and large storage, distributed systems are used. As multiple parties are involved in these systems, the risk of privacy violation is increased. There have been a number of privacy-preserving mechanisms developed for privacy protection at different stages (e.g., data generation, data storage, and data processing) of a big data life cycle. The goal of this paper is to provide a comprehensive overview of the privacy preservation mechanisms in big data and present the challenges for existing mechanisms. In particular, in this paper, we illustrate the infrastructure of big data and the state-of-the-art privacy-preserving mechanisms in each stage of the big data life cycle. Furthermore, we discuss the challenges and future research directions related to privacy preservation in big data. <s> BIB004 </s> Big Data Quality: A systematic literature review and future research directions <s> Introduction <s> Big data has been acknowledged for its enormous potential. In contrast to the potential, in a recent survey more than half of financial service organizations reported that big data has not delivered the expected value. One of the main reasons for this is related to data quality. The objective of this research is to identify the antecedents of big data quality in financial institutions. This will help to understand how data quality from big data analysis can be improved. For this, a literature review was performed and data was collected using three case studies, followed by content analysis. The overall findings indicate that there are no fundamentally new data quality issues in big data projects. Nevertheless, the complexity of the issues is higher, which makes it harder to assess and attain data quality in big data projects compared to the traditional projects. Ten antecedents of big data quality were identified encompassing data, technology, people, process and procedure, organization, and external aspects. <s> BIB005 </s> Big Data Quality: A systematic literature review and future research directions <s> Introduction <s> In the era of big data and Internet of things, massive sensor data are gathered with Internet of things. Quantity of data captured by sensor networks are considered to contain highly useful and valuable information. However, for a variety of reasons, received sensor data often appear abnormal. Therefore, effective anomaly detection methods are required to guarantee the quality of data collected by those sensor nodes. Since sensor data are usually correlated in time and space, not all the gathered data are valuable for further data processing and analysis. Preprocessing is necessary for eliminating the redundancy in gathered massive sensor data. In this paper, the proposed work defines a sensor data preprocessing framework. It is mainly composed of two parts, i.e., sensor data anomaly detection and sensor data redundancy elimination. In the first part, methods based on principal statistic analysis and Bayesian network is proposed for sensor data anomaly detection. Then, approaches based on static Bayesian network (SBN) and dynamic Bayesian networks (DBNs) are proposed for sensor data redundancy elimination. Static sensor data redundancy detection algorithm (SSDRDA) for eliminating redundant data in static datasets and real-time sensor data redundancy detection algorithm (RSDRDA) for eliminating redundant sensor data in real-time are proposed. The efficiency and effectiveness of the proposed methods are validated using real-world gathered sensor datasets. <s> BIB006 </s> Big Data Quality: A systematic literature review and future research directions <s> Introduction <s> Abstract With the rapid development of the Internet of Things (IoT), Big Data technologies have emerged as a critical data analytics tool to bring the knowledge within IoT infrastructures to better meet the purpose of the IoT systems and support critical decision making. Although the topic of Big Data analytics itself is extensively researched, the disparity between IoT domains (such as healthcare, energy, transportation and others) has isolated the evolution of Big Data approaches in each IoT domain. Thus, the mutual understanding across IoT domains can possibly advance the evolution of Big Data research in IoT. In this work, we therefore conduct a survey on Big Data technologies in different IoT domains to facilitate and stimulate knowledge sharing across the IoT domains. Based on our review, this paper discusses the similarities and differences among Big Data technologies used in different IoT domains, suggests how certain Big Data technology used in one IoT domain can be re-used in another IoT domain, and develops a conceptual framework to outline the critical Big Data technologies across all the reviewed IoT domains. <s> BIB007 </s> Big Data Quality: A systematic literature review and future research directions <s> Introduction <s> After the rise of big data to national strategy, the application of big data in every industry is increasing. The quality of data will directly affect the value of data and influence the analysis and decision of managers. Aiming at the characteristics of big data, such as volume, velocity, variety and value, a quality management framework of big data based on application scenario is proposed, which includes data quality assessment and quality management of structured data, unstructured data and data integration stage. In view of the current structured data leading to the core business of the enterprise, we use the research method to extend the peripheral data layer by layer on the main data. Big data processing technology, such as Hadoop and Storm, is used to construct a big data cleaning system based on semantics. Combined with JStorm platform, a real-time control system for big data quality is given. Finally, a big data quality evaluation system is built to detect the effect of data integration. The framework can guarantee the output of high quality big data on the basis of traditional data quality system. It helps enterprises to understand data rules and increase the value of core data, which has practical application value. <s> BIB008
Recent advances in information and communication technology (ICT) have led to mass production of data by social networks, sensor networks, and other Internet-based applications of various domains like healthcare. The vast amount of data that is generated with a high speed from various internet sources is called Big Data BIB004 . Recently, organizations have concluded that processing big data, especially the data coming from Twitter and Facebook can provide a significant impact on increasing the business's effectiveness and added values  - BIB002 . However, due to the exponential growth of the available data, traditional systems and methods are not able to meet the requirements for processing this data. In 2011, and in McKinney Global Institute report, three characteristics of data, i.e. Volume (size of data), Variety (different types of data from several sources) and Velocity (real-time data generation), which are called the 3 V's, were introduced as the three major dimensions of big data BIB001 . In addition to these 3 V's, two other dimensions are also introduced. These are Veracity (uncertainty of data) and Value (benefits to various industrial field)  - BIB007 . While it is clear that valuable information can be extracted through analysis of big data, the results of such an analysis are hardly reliable unless well-defined and proper verification and quality control mechanisms are applied on this data before it is used. As a result, the importance of data quality assessment in big data needs to be taken into consideration by both researchers and practitioners. For example, in the domain of Internet of things (IoT), which is an active area of big data production and consumption, the application scenarios include deployment of many sensors that should collect the data. Quality aspects of the collected data in these networks should be at an appropriate level, but for different reasons, the data may have a low quality BIB006 . Since inadequate and inaccurate data creates serious problems, one of which is the incorrect decisions that organizations might make BIB008 , big data quality assessment is one of the most critical challenges BIB005 . So without data quality assessment, organizations do not have a proper understanding of what happens on the market BIB003 . Figure 1 shows the significant problems caused by the poor quality of big data. BIB003 ) As shown in Figure 1 , low data quality may cause problems such as extra cost and allocating more time to reconcile data. California Independent System Operator (CAISO), which oversees the operation of California's bulk electric power system, transmission lines, and electricity market generated and transmitted by its member utilities, reported that 10%-17% of the data received in 2011 had quality issues  . Also, IBM has reported that only one in three corporate executives trust their analytics results  . Moreover, a recent study has shown that low data quality costs the USA three trillion dollars per year  . It is also stated in  that two-thirds of European and American businesses are not able to unlock value from big data. All these reports emphasize the importance of assessing the data quality in the big data domain. Data quality assessment is an essential prerequisite for data improvement, and the purpose of data quality evaluation is to determine the quality level of data  . Considering the importance of the data quality assessment in the big data era, we have decided to review the papers in this field to determine what methods have been proposed by different scholars and what challenges they encountered. For this purpose, we provided a Systematic Literature Review (SLR) to search and access high-quality papers in this era. Afterward, we have identified a research tree and various profiles that details of the methodology and results, which can provide useful information to other researchers interested in this field, are explained in the following sections. The rest of the paper is organized as follows: the search methodology, which includes the planning and conducting phases, is described in Section 2. In Section 3, the research tree obtained from the study of the papers is shown and explained. Studies related to stream processing in Section 4, batch processing in Section 5 and the hybrid methods in Section 6 are described. The review papers are described and compared with the systematic review presented in this paper in Section 7. In Section 8, the results of the systematic review are presented and provide useful information to those researchers who are interested in the big data quality area. Challenges and future work in Section 9, and finally, Section 10 concludes the paper.