Big Data Quality: A systematic literature review and future research directions <s> Technique dependent limitations <s> Sensors are increasingly embedded into physical products in order to capture data about their conditions and usage for decision making in business applications. However, a major issue for such applications is the limited quality of the captured data due to inherently restricted precision and performance of the sensors. Moreover, the data quality is further decreased by data processing to meet resource constraints in streaming environments and ultimately influences business decisions. The issue of how to efficiently provide applications with information about data quality (DQ) is still an open research problem. In my Ph.D. thesis, I address this problem by developing a system to provide business applications with accurate information on data quality. Furthermore, the system will be able to incorporate and guarantee user-defined data quality levels. In this paper, I will present the major results from my research so far. This includes a novel jumping-window-based approach for the efficient transfer of data quality information as well as a flexible metamodel for storage and propagation of data quality. The comprehensive analysis of common data processing operators w.r.t. their impact on data quality allows a fruitful knowledge evaluation and thus diminishes incorrect business decisions. <s> BIB001 </s> Big Data Quality: A systematic literature review and future research directions <s> Technique dependent limitations <s> Sensors in smart-item environments capture data about product conditions and usage to support business decisions as well as production automation processes. A challenging issue in this application area is the restricted quality of sensor data due to limited sensor precision and sensor failures. Moreover, data stream processing to meet resource constraints in streaming environments introduces additional noise and decreases the data quality. In order to avoid wrong business decisions due to dirty data, quality characteristics have to be captured, processed, and provided to the respective business task. However, the issue of how to efficiently provide applications with information about data quality is still an open research problem. In this article, we address this problem by presenting a flexible model for the propagation and processing of data quality. The comprehensive analysis of common data stream processing operators and their impact on data quality allows a fruitful data evaluation and diminishes incorrect business decisions. Further, we propose the data quality model control to adapt the data quality granularity to the data stream interestingness. <s> BIB002 </s> Big Data Quality: A systematic literature review and future research directions <s> Technique dependent limitations <s> In the field of wireless sensor networks, those measurements that significantly deviate from the normal pattern of sensed data are considered as outliers. The potential sources of outliers include noise and errors, events, and malicious attacks on the network. Traditional outlier detection techniques are not directly applicable to wireless sensor networks due to the nature of sensor data and specific requirements and limitations of the wireless sensor networks. This survey provides a comprehensive overview of existing outlier detection techniques specifically developed for the wireless sensor networks. Additionally, it presents a technique-based taxonomy and a comparative table to be used as a guideline to select a technique suitable for the application at hand based on characteristics such as data type, outlier type, outlier identity, and outlier degree. <s> BIB003 </s> Big Data Quality: A systematic literature review and future research directions <s> Technique dependent limitations <s> Wireless Sensor Networks (WSNs) are important and necessary platforms for the future as the concept “Internet of Things” has emerged lately. They are used for monitoring, tracking, or controlling of many applications in industry, health care, habitat, and military. However, the quality of data collected by sensor nodes is affected by anomalies that occur due to various reasons, such as node failures, reading errors, unusual events, and malicious attacks. Therefore, anomaly detection is a necessary process to ensure the quality of sensor data before it is utilized for making decisions. In this review, we present the challenges of anomaly detection in WSNs and state the requirements to design efficient and effective anomaly detection models. We then review the latest advancements of data anomaly detection research in WSNs and classify current detection approaches in five main classes based on the detection methods used to design these approaches. Varieties of the state-of-the-art models for each class are covered and their limitations are highlighted to provide ideas for potential future works. Furthermore, the reviewed approaches are compared and evaluated based on how well they meet the stated requirements. Finally, the general limitations of current approaches are mentioned and further research opportunities are suggested and discussed. <s> BIB004 </s> Big Data Quality: A systematic literature review and future research directions <s> Technique dependent limitations <s> In the statistics community, outlier detection for time series data has been studied for decades. Recently, with advances in hardware and software technology, there has been a large body of work on temporal outlier detection from a computational perspective within the computer science community. In particular, advances in hardware technology have enabled the availability of various forms of temporal data collection mechanisms, and advances in software technology have enabled a variety of data management mechanisms. This has fueled the growth of different kinds of data sets such as data streams, spatio-temporal data, distributed streams, temporal networks, and time series data, generated by a multitude of applications. There arises a need for an organized and detailed study of the work done in the area of outlier detection with respect to such temporal datasets. In this survey, we provide a comprehensive and structured overview of a large set of interesting outlier definitions for various forms of temporal data, novel techniques, and application scenarios in which specific definitions and techniques have been widely used. <s> BIB005 </s> Big Data Quality: A systematic literature review and future research directions <s> Technique dependent limitations <s> Performing predictive modelling, such as anomaly detection, in Big Data is a difficult task. This problem is compounded as more and more sources of Big Data are generated from environmental sensors, logging applications, and the Internet of Things. Further, most current techniques for anomaly detection only consider the content of the data source, i.e. the data itself, without concern for the context of the data. As data becomes more complex it is increasingly important to bias anomaly detection techniques for the context, whether it is spatial, temporal, or semantic. The work proposed in this paper outlines a contextual anomaly detection technique for use in streaming sensor networks. The technique uses a well-defined content anomaly detection algorithm for real-time point anomaly detection. Additionally, we present a post-processing context-aware anomaly detection algorithm based on sensor profiles, which are groups of contextually similar sensors generated by a multivariate clustering algorithm. Our proposed research has been implemented and evaluated with real-world data provided by Powersmiths, located in Brampton, Ontario, Canada. <s> BIB006 </s> Big Data Quality: A systematic literature review and future research directions <s> Technique dependent limitations <s> The ability to detect and process anomalies for Big Data in real-time is a difficult task. The volume and velocity of the data within many systems makes it difficult for typical algorithms to scale and retain their real-time characteristics. The pervasiveness of data combined with the problem that many existing algorithms only consider the content of the data source; e.g. a sensor reading itself without concern for its context, leaves room for potential improvement. The proposed work defines a contextual anomaly detection framework. It is composed of two distinct steps: content detection and context detection. The content detector is used to determine anomalies in real-time, while possibly, and likely, identifying false positives. The context detector is used to prune the output of the content detector, identifying those anomalies which are considered both content and contextually anomalous. The context detector utilizes the concept of profiles, which are groups of similarly grouped data points generated by a multivariate clustering algorithm. The research has been evaluated against two real-world sensor datasets provided by a local company in Brampton, Canada. Additionally, the framework has been evaluated against the open-source Dodgers dataset, available at the UCI machine learning repository, and against the R statistical toolbox. <s> BIB007 </s> Big Data Quality: A systematic literature review and future research directions <s> Technique dependent limitations <s> The use of freely available online data is rapidly increasing, as companies have detected the possibilities and the value of these data in their businesses. In particular, data from social media are seen as interesting as they can, when properly treated, assist in achieving customer insight into business decision making. However, the unstructured and uncertain nature of this kind of big data presents a new kind of challenge: how to evaluate the quality of data and manage the value of data within a big data architecture? This paper contributes to addressing this challenge by introducing a new architectural solution to evaluate and manage the quality of social media data in each processing phase of the big data pipeline. The proposed solution improves business decision making by providing real-time, validated data for the user. The solution is validated with an industrial case example, in which the customer insight is extracted from social media data in order to determine the customer satisfaction regarding the quality of a product. <s> BIB008 </s> Big Data Quality: A systematic literature review and future research directions <s> Technique dependent limitations <s> Real time data analysis in data streams is a highly challenging area in big data. The surge in big data techniques has recently attracted considerable interest to the detection of significant changes or anomalies in data streams. There is a variety of literature across a number of fields relevant to anomaly detection. The growing number of techniques, from seemingly disconnected areas, prevents a comprehensive review. Many interesting techniques may therefore remain largely unknown to the anomaly detection community at large. The survey presents a compact, but comprehensive overview of diverse strategies for anomaly detection in evolving data streams. A number of recommendations based performance and applicability to use cases are provided. We expect that our classification and recommendations will provide useful guidelines to practitioners in this rapidly evolving field. <s> BIB009 </s> Big Data Quality: A systematic literature review and future research directions <s> Technique dependent limitations <s> The data cleaning of electrical power big data can improve the correctness, the completeness, the consistency and the reliability of the data. Aiming at the difficulties of the extracting of the unified anomaly detection pattern and the low accuracy and continuity of the anomaly data correction in the process of the electrical power big data cleaning, the data cleaning model of the electrical power big data based on Spark is proposed. Firstly, the normal clusters and the corresponding boundary samples are obtained by the improved CURE clustering algorithm. Then, the anomaly data identification algorithm based on boundary samples is designed. Finally, the anomaly data modification is realized by using exponential weighting moving mean value. The high efficiency and accuracy is proved by the experiment of the data cleaning of the wind power generation monitoring data from the wind power station. <s> BIB010 </s> Big Data Quality: A systematic literature review and future research directions <s> Technique dependent limitations <s> ABSTRACTNowadays technological advances have promoted big data streams common in many applications, including mobile internet applications, internet of things, and industry production process. Outl... <s> BIB011 </s> Big Data Quality: A systematic literature review and future research directions <s> Technique dependent limitations <s> With the rapid development of social networks, Internet of things, Cloud computing as well as other technologies, big data age is arriving. The increasing number of data has brought great value to the public and enterprises. Meanwhile how to manage and use big data better has become the focus of all walks of life. The 4V characteristics of big data have brought a lot of issues to the big data processing. The key to big data processing is to solve data quality issue, and to ensure data quality is a prerequisite for the successful application of big data technique. In this paper, we use recommendation systems and prediction systems as typical big data applications, and try to find out the data quality issues during data collection, data preprocessing, data storage and data analysis stages of big data processing. According to the elaboration and analysis of the proposed issues, the corresponding solutions are also put forward. Finally, some open problems to be solved in the future are also raised. <s> BIB012 </s> Big Data Quality: A systematic literature review and future research directions <s> Technique dependent limitations <s> Social media data has provided various insights into the behaviour of consumers and businesses. However, extracted data may be erroneous, or could have originated from a malicious source. Thus, quality of social media should be managed. Also, it should be understood how data quality can be managed across a big data pipeline, which may consist of several processing and analysis phases. The contribution of this paper is evaluation of data quality management architecture for social media data. The theoretical concepts based on previous work have been implemented for data quality evaluation of Twitter-based data sets. Particularly, reference architecture for quality management in social media data has been extended and evaluated based on the implementation architecture. Experiments indicate that 150–800 tweets/s can be evaluated with two cloud nodes depending on the configuration. <s> BIB013 </s> Big Data Quality: A systematic literature review and future research directions <s> Technique dependent limitations <s> In the Big Data Era, data is the core for any governmental, institutional, and private organization. Efforts were geared towards extracting highly valuable insights that cannot happen if data is of poor quality. Therefore, data quality (DQ) is considered as a key element in Big data processing phase. In this stage, low quality data is not penetrated to the Big Data value chain. This paper, addresses the data quality rules discovery (DQR) after the evaluation of quality and prior to Big Data pre-processing. We propose a DQR discovery model to enhance and accurately target the pre-processing activities based on quality requirements. We defined, a set of pre-processing activities associated with data quality dimensions (DQD's) to automatize the DQR generation process. Rules optimization are applied on validated rules to avoid multi-passes pre-processing activities and eliminates duplicate rules. Conducted experiments showed an increased quality scores after applying the discovered and optimized DQR's on data. <s> BIB014 </s> Big Data Quality: A systematic literature review and future research directions <s> Technique dependent limitations <s> Abstract Big data changed the way in which we collect and analyze data. In particular, the amount of available information is constantly growing and organizations rely more and more on data analysis in order to achieve their competitive advantage. However, such amount of data can create a real value only if combined with quality: good decisions and actions are the results of correct, reliable and complete data. In such a scenario, methods and techniques for the Data Quality assessment can support the identification of suitable data to process. If for traditional database numerous assessment methods are proposed, in the Big Data scenario new algorithms have to be designed in order to deal with novel requirements related to variety, volume and velocity issues. In particular, in this paper we highlight that dealing with heterogeneous sources requires an adaptive approach able to trigger the suitable quality assessment methods on the basis of the data type and context in which data have to be used. Furthermore, we show that in some situations it is not possible to evaluate the quality of the entire dataset due to performance and time constraints. For this reason, we suggest to focus the Data Quality assessment only on a portion of the dataset and to take into account the consequent loss of accuracy by introducing a confidence factor as a measure of the reliability of the quality assessment procedure. We propose a methodology to build a Data Quality adapter module, which selects the best configuration for the Data Quality assessment based on the user main requirements: time minimization, confidence maximization, and budget minimization. Experiments are performed by considering real data gathered from a smart city case study. <s> BIB015 </s> Big Data Quality: A systematic literature review and future research directions <s> Technique dependent limitations <s> The combination of data and technology is having a high impact on the way we live. The world is getting smarter thanks to the quantity of collected and analyzed data. However, it is necessary to consider that such amount of data is continuously increasing and it is necessary to deal with novel requirements related to variety, volume, velocity, and veracity issues. In this paper we focus on veracity that is related to the presence of uncertain or imprecise data: errors, missing or invalid data can compromise the usefulness of the collected values. In such a scenario, new methods and techniques able to evaluate the quality of the available data are needed. In fact, the literature provides many data quality assessment and improvement techniques, especially for structured data, but in the Big Data era new algorithms have to be designed. We aim to provide an overview of the issues and challenges related to Data Quality assessment in the Big Data scenario. We also propose a possible solution developed by considering a smart city case study and we describe the lessons learned in the design and implementation phases. <s> BIB016 </s> Big Data Quality: A systematic literature review and future research directions <s> Technique dependent limitations <s> The internet of Things (IoT) is a data stream environment where a large scale deployment of smart things continuously report readings. These data streams are then consumed by pervasive applications, i.e. data consumers, to offer ubiquitous services. The data quality (DQ) is a key criteria for IoT data consumers especially when considering the inherent uncertainty of sensor-enabled data. However, DQ is a highly subjective concept and there is no standard agreement on how to determine “good” data. Moreover, the combinations of considered measured attributes and associated DQ information are as diverse as the needs of data consumers. This introduces expensive overheads for developers tasked with building DQ-aware IoT software systems which are capable of managing their own DQ information. To effectively handle these various perceptions of DQ, we propose a Model-Driven Architecture-based approach that allows each developer to easily and efficiently express, through models and other provided resources, the data consumer’s vision of DQ and its requirements using an easy-to-use graphical model editor. The defined DQ specifications are then automatically transformed to generate an entire infrastructure for DQ management that fits perfectly the data consumer’s requirements. We demonstrate the flexibility and the efficiency of our approach by generating two DQ management infrastructures built on top of different platforms and testing them through a real life data stream environment scenario. <s> BIB017
Some challenges are related to the technique used in the quality evaluation algorithm, since any technique with its advantages has some limitations. As briefly explained in Figure 6 , the advantages and disadvantages of each technique are presented below:  Learning based: Clustering and classification algorithms challenges are discussed in  , BIB004 , BIB009 , BIB003 , BIB011 , BIB012 . Clustering algorithms can be adapted to complex data types and can be used to obtain context information. However, in most clustering algorithms, the number of clusters is one of the crucial parameters to be defined. Since there is no access to all data in the data stream, it is impossible to specify the number of clusters. Not specifying the number of clusters leads to the production of arbitrary shapes, and ultimately it will be difficult to analyze these clusters. Classification algorithms do not need to set the parameter. In these algorithms, the existence of a label in a dataset is necessary. In the batch, we can test and classify the data with reasonable accuracy by constructing the model, but in the streaming data, specifying the size of test data and updating the model are major challenges. Also, the computational complexity of this kind of methods is higher than clustering algorithms.  Model-based: In some studies  , BIB004 , BIB009 ,  - BIB011 model-based methods (also known as statistical-based technique) are explained. These methods use temporal correlation in order to build the model, and the least change in data distribution results in the detection of poor quality data. The simplicity of this method makes it more used in streaming data than batch. On the other hand, updating the model is one of the challenges of this technique, mainly when the concept drift occurs; this method cannot adapt itself to the new data distribution. These methods are suitable for univariate data and cannot find the interactions between attributes in multivariate data.  Distance-based: As in  , BIB005 - BIB009 ,  - BIB011 explained, these methods (also known as nearest neighbor based) do not assume any data distribution. Applying this technique for different types of data is simple because it only needs to define the appropriate distance criterion. However, it is difficult to define the distance criterion for complex data. The computation cost in multivariate data is prohibitive, and they are not suitable for high dimensional data. Also, the scalability of these methods is a significant concern. Typically, in these methods, a threshold value is used to detect outliers; the inappropriate threshold setting results in a lot of false positives.  Rule-based: The rules are identified by the expert, so the precision of evaluation is more than other methods. Since these rules are obtained by examining historical data, the corresponding algorithm can be performed in batch or hybrid processing. One of the disadvantages of this approach is human intervention which is discussed in BIB010 , BIB004 ,  , which may take much time to define the rules. Also, since the results are obtained by the rules, the learning process, as well as the proper approach to changing the distribution of data, is not achieved.  Context-aware: Context information can improve the quality of the results BIB006 and is useful for identifying low-quality data whose pattern is not obtained locally BIB007 . However, it is difficult to adapt to the local pattern with global patterns, especially when other data sources have different structures and schemas. Also, data integration in this situation is arduous.  Instance and Schema-based: In these methods, if the schema is available, then constraints of each attribute are obtained by profiling BIB008 , BIB015 , BIB016 , BIB013 , BIB014 . Otherwise, it is done by analyzing and inspecting data values BIB001 - BIB002 . The presence of the schema, which often exists in batch data, can help with the accuracy of profiling results. On the other hand, and in streaming data with no data schema, the evaluation process becomes onerous, and generally with scholars evaluate the data by identifying a series of quality dimensions and defining their measures BIB001 - BIB017 . Finding the appropriate data quality dimensions that are suitable for the target data is another challenge of data assessment. In the next subsection, we discuss the issues that have not yet been appropriately addressed in the studies.