Big Data Quality: A systematic literature review and future research directions <s> Schema-based techniques <s> The use of freely available online data is rapidly increasing, as companies have detected the possibilities and the value of these data in their businesses. In particular, data from social media are seen as interesting as they can, when properly treated, assist in achieving customer insight into business decision making. However, the unstructured and uncertain nature of this kind of big data presents a new kind of challenge: how to evaluate the quality of data and manage the value of data within a big data architecture? This paper contributes to addressing this challenge by introducing a new architectural solution to evaluate and manage the quality of social media data in each processing phase of the big data pipeline. The proposed solution improves business decision making by providing real-time, validated data for the user. The solution is validated with an industrial case example, in which the customer insight is extracted from social media data in order to determine the customer satisfaction regarding the quality of a product. <s> BIB001 </s> Big Data Quality: A systematic literature review and future research directions <s> Schema-based techniques <s> Social media data has provided various insights into the behaviour of consumers and businesses. However, extracted data may be erroneous, or could have originated from a malicious source. Thus, quality of social media should be managed. Also, it should be understood how data quality can be managed across a big data pipeline, which may consist of several processing and analysis phases. The contribution of this paper is evaluation of data quality management architecture for social media data. The theoretical concepts based on previous work have been implemented for data quality evaluation of Twitter-based data sets. Particularly, reference architecture for quality management in social media data has been extended and evaluated based on the implementation architecture. Experiments indicate that 150–800 tweets/s can be evaluated with two cloud nodes depending on the configuration. <s> BIB002 </s> Big Data Quality: A systematic literature review and future research directions <s> Schema-based techniques <s> In the Big Data Era, data is the core for any governmental, institutional, and private organization. Efforts were geared towards extracting highly valuable insights that cannot happen if data is of poor quality. Therefore, data quality (DQ) is considered as a key element in Big data processing phase. In this stage, low quality data is not penetrated to the Big Data value chain. This paper, addresses the data quality rules discovery (DQR) after the evaluation of quality and prior to Big Data pre-processing. We propose a DQR discovery model to enhance and accurately target the pre-processing activities based on quality requirements. We defined, a set of pre-processing activities associated with data quality dimensions (DQD's) to automatize the DQR generation process. Rules optimization are applied on validated rules to avoid multi-passes pre-processing activities and eliminates duplicate rules. Conducted experiments showed an increased quality scores after applying the discovered and optimized DQR's on data. <s> BIB003 </s> Big Data Quality: A systematic literature review and future research directions <s> Schema-based techniques <s> Abstract Big data changed the way in which we collect and analyze data. In particular, the amount of available information is constantly growing and organizations rely more and more on data analysis in order to achieve their competitive advantage. However, such amount of data can create a real value only if combined with quality: good decisions and actions are the results of correct, reliable and complete data. In such a scenario, methods and techniques for the Data Quality assessment can support the identification of suitable data to process. If for traditional database numerous assessment methods are proposed, in the Big Data scenario new algorithms have to be designed in order to deal with novel requirements related to variety, volume and velocity issues. In particular, in this paper we highlight that dealing with heterogeneous sources requires an adaptive approach able to trigger the suitable quality assessment methods on the basis of the data type and context in which data have to be used. Furthermore, we show that in some situations it is not possible to evaluate the quality of the entire dataset due to performance and time constraints. For this reason, we suggest to focus the Data Quality assessment only on a portion of the dataset and to take into account the consequent loss of accuracy by introducing a confidence factor as a measure of the reliability of the quality assessment procedure. We propose a methodology to build a Data Quality adapter module, which selects the best configuration for the Data Quality assessment based on the user main requirements: time minimization, confidence maximization, and budget minimization. Experiments are performed by considering real data gathered from a smart city case study. <s> BIB004 </s> Big Data Quality: A systematic literature review and future research directions <s> Schema-based techniques <s> The combination of data and technology is having a high impact on the way we live. The world is getting smarter thanks to the quantity of collected and analyzed data. However, it is necessary to consider that such amount of data is continuously increasing and it is necessary to deal with novel requirements related to variety, volume, velocity, and veracity issues. In this paper we focus on veracity that is related to the presence of uncertain or imprecise data: errors, missing or invalid data can compromise the usefulness of the collected values. In such a scenario, new methods and techniques able to evaluate the quality of the available data are needed. In fact, the literature provides many data quality assessment and improvement techniques, especially for structured data, but in the Big Data era new algorithms have to be designed. We aim to provide an overview of the issues and challenges related to Data Quality assessment in the Big Data scenario. We also propose a possible solution developed by considering a smart city case study and we describe the lessons learned in the design and implementation phases. <s> BIB005
A data quality evaluation architecture is presented in BIB004 , BIB005 , which has several modules. The profiling module profiles data based on its features and gains statistical data. The results of this step are stored in the Quality Metadata module. On the other hand, the user specifies his requirements through the user interface, for example, confidence level and so on. The data are evaluated by the DQ Assessment module in terms of quality dimensions, and if the data volume and a large number of computations are time-consuming, the Adapter module will sample the data and perform the evaluation on fewer data to spend less time. In experiments, the impact of confidence has been evaluated on quality dimensions as well as on runtime. The presented architecture has been able to evaluate the quality of the data by the defined requirements by optimizing the user's parameters. Data extraction, data preprocessing, data processing, data analysis, data transformation, and visualization are big data management steps. In BIB001 , BIB002 , authors have tried to evaluate the quality of data by metadata in each of these steps. Metadata is obtained after extracting data and analyzing its attributes. Then the quality of the data is evaluated by calculating the quality dimensions and with metadata assistance. Extracting metadata in the processing, analyzing and deciding making phases are also performed. In each of these steps, after data selecting, the metadata is obtained by examining the attributes of the data, then quality evaluation takes place as long as the decisionmaking step is complete. The quality evaluation of Twitter data is presented in BIB002 . The manager first lists the required quality dimensions. Then the data expert identifies the criteria for evaluating these dimensions and also specifies the metadata for the tweets. The manager in the final step considers a threshold for each quality dimension. When a data is taken from Twitter, its schema first is examined by the metadata, and if it is within the range, its quality dimensions are evaluated. In the end, only good enough data is sent to the analytic process. The paper BIB003 presents a framework for the discovery of data quality rules. In the first step, and from a large amount of data, data sampling and profiling are performed. After profiling and reviewing data features, quality dimensions are identified, and sampled data are evaluated using quality dimensions. Then the score for each feature is calculated. By analyzing the quality score, a set of rules is generatedfor example, a rule to delete incomplete data and so on. Then the rules apply to the sampled data, and the changes are inspected. If the rules need to be changed, they will be updated. This thesis  presents a tool for evaluating big data quality. In this tool, data is first loaded from the storage location, and then, after profiling the data, the system evaluates them based on four data quality dimensions, such as completeness, timeliness, accuracy, and reputation. The reason for choosing these four dimensions is their applicability in some domains, such as e-Business, e-Science and Information Systems. The focus of this research is on the scalability and costeffectively of the tool, so the tool has been developed on Apache Spark in order to be more effective. The performance of the tool has been evaluated on different dataset sizes, and the results demonstrate that big data framework can be used to improve performance and scalability of a data quality query system on small, medium, and large workloads.