Big Data Quality: A systematic literature review and future research directions <s> Source dependent limitations <s> Sensors are increasingly embedded into physical products in order to capture data about their conditions and usage for decision making in business applications. However, a major issue for such applications is the limited quality of the captured data due to inherently restricted precision and performance of the sensors. Moreover, the data quality is further decreased by data processing to meet resource constraints in streaming environments and ultimately influences business decisions. The issue of how to efficiently provide applications with information about data quality (DQ) is still an open research problem. In my Ph.D. thesis, I address this problem by developing a system to provide business applications with accurate information on data quality. Furthermore, the system will be able to incorporate and guarantee user-defined data quality levels. In this paper, I will present the major results from my research so far. This includes a novel jumping-window-based approach for the efficient transfer of data quality information as well as a flexible metamodel for storage and propagation of data quality. The comprehensive analysis of common data processing operators w.r.t. their impact on data quality allows a fruitful knowledge evaluation and thus diminishes incorrect business decisions. <s> BIB001 </s> Big Data Quality: A systematic literature review and future research directions <s> Source dependent limitations <s> Sensors in smart-item environments capture data about product conditions and usage to support business decisions as well as production automation processes. A challenging issue in this application area is the restricted quality of sensor data due to limited sensor precision and sensor failures. Moreover, data stream processing to meet resource constraints in streaming environments introduces additional noise and decreases the data quality. In order to avoid wrong business decisions due to dirty data, quality characteristics have to be captured, processed, and provided to the respective business task. However, the issue of how to efficiently provide applications with information about data quality is still an open research problem. In this article, we address this problem by presenting a flexible model for the propagation and processing of data quality. The comprehensive analysis of common data stream processing operators and their impact on data quality allows a fruitful data evaluation and diminishes incorrect business decisions. Further, we propose the data quality model control to adapt the data quality granularity to the data stream interestingness. <s> BIB002 </s> Big Data Quality: A systematic literature review and future research directions <s> Source dependent limitations <s> In the field of wireless sensor networks, those measurements that significantly deviate from the normal pattern of sensed data are considered as outliers. The potential sources of outliers include noise and errors, events, and malicious attacks on the network. Traditional outlier detection techniques are not directly applicable to wireless sensor networks due to the nature of sensor data and specific requirements and limitations of the wireless sensor networks. This survey provides a comprehensive overview of existing outlier detection techniques specifically developed for the wireless sensor networks. Additionally, it presents a technique-based taxonomy and a comparative table to be used as a guideline to select a technique suitable for the application at hand based on characteristics such as data type, outlier type, outlier identity, and outlier degree. <s> BIB003 </s> Big Data Quality: A systematic literature review and future research directions <s> Source dependent limitations <s> In recent years, RFID technologies have been used in many applications, such as inventory checking and object tracking. However, raw RFID data are inherently unreliable due to physical device limitations and different kinds of environmental noise. Currently, existing work mainly focuses on RFID data cleansing in a static environment (e.g. inventory checking). It is therefore difficult to cleanse RFID data streams in a mobile environment (e.g. object tracking) using the existing solutions, which do not address the data missing issue effectively. In this paper, we study how to cleanse RFID data streams for object tracking, which is a challenging problem, since a significant percentage of readings are routinely dropped. We propose a probabilistic model for object tracking in a mobile environment. We develop a Bayesian inference based approach for cleansing RFID data using the model. In order to sample data from the movement distribution, we devise a sequential sampler that cleans RFID data with high accuracy and efficiency. We validate the effectiveness and robustness of our solution through extensive simulations and demonstrate its performance by using two real RFID applications of human tracking and conveyor belt monitoring. <s> BIB004 </s> Big Data Quality: A systematic literature review and future research directions <s> Source dependent limitations <s> Low quality sensor data limits WSN capabilities for providing reliable real-time situation-awareness. Outlier detection is a solution to ensure the quality of sensor data. An effective and efficient outlier detection technique for WSNs not only identifies outliers in a distributed and online manner with high detection accuracy and low false alarm, but also satisfies WSN constraints in terms of communication, computational and memory complexity. In this paper, we take into account the correlation between sensor data attributes and propose two distributed and online outlier detection techniques based on a hyperellipsoidal one-class support vector machine (SVM). We also take advantage of the theory of spatio-temporal correlation to identify outliers and update the ellipsoidal SVM-based model representing the changed normal behavior of sensor data for further outlier identification. Simulation results show that our adaptive ellipsoidal SVM-based outlier detection technique achieves better detection accuracy and lower false alarm as compared to existing SVM-based techniques designed for WSNs. <s> BIB005 </s> Big Data Quality: A systematic literature review and future research directions <s> Source dependent limitations <s> Wireless Sensor Networks (WSNs) are important and necessary platforms for the future as the concept “Internet of Things” has emerged lately. They are used for monitoring, tracking, or controlling of many applications in industry, health care, habitat, and military. However, the quality of data collected by sensor nodes is affected by anomalies that occur due to various reasons, such as node failures, reading errors, unusual events, and malicious attacks. Therefore, anomaly detection is a necessary process to ensure the quality of sensor data before it is utilized for making decisions. In this review, we present the challenges of anomaly detection in WSNs and state the requirements to design efficient and effective anomaly detection models. We then review the latest advancements of data anomaly detection research in WSNs and classify current detection approaches in five main classes based on the detection methods used to design these approaches. Varieties of the state-of-the-art models for each class are covered and their limitations are highlighted to provide ideas for potential future works. Furthermore, the reviewed approaches are compared and evaluated based on how well they meet the stated requirements. Finally, the general limitations of current approaches are mentioned and further research opportunities are suggested and discussed. <s> BIB006 </s> Big Data Quality: A systematic literature review and future research directions <s> Source dependent limitations <s> In the statistics community, outlier detection for time series data has been studied for decades. Recently, with advances in hardware and software technology, there has been a large body of work on temporal outlier detection from a computational perspective within the computer science community. In particular, advances in hardware technology have enabled the availability of various forms of temporal data collection mechanisms, and advances in software technology have enabled a variety of data management mechanisms. This has fueled the growth of different kinds of data sets such as data streams, spatio-temporal data, distributed streams, temporal networks, and time series data, generated by a multitude of applications. There arises a need for an organized and detailed study of the work done in the area of outlier detection with respect to such temporal datasets. In this survey, we provide a comprehensive and structured overview of a large set of interesting outlier definitions for various forms of temporal data, novel techniques, and application scenarios in which specific definitions and techniques have been widely used. <s> BIB007 </s> Big Data Quality: A systematic literature review and future research directions <s> Source dependent limitations <s> Wireless sensor networks (WSNs) are increasingly used as platforms for collecting data from unattended environments and monitoring important events in phenomena. However, sensor data is affected by anomalies that occur due to various reasons, such as, node software or hardware failures, reading errors, unusual events, and malicious attacks. Therefore, effective, efficient, and real time detection of anomalous measurement is required to guarantee the quality of data collected by these networks. In this paper, two efficient and effective anomaly detection models PCCAD and APCCAD are proposed for static and dynamic environments, respectively. Both models utilize the One-Class Principal Component Classifier (OCPCC) to measure the dissimilarity between sensor measurements in the feature space. The proposed APCCAD model incorporates an incremental learning method that is able to track the dynamic normal changes of data streams in the monitored environment. The efficiency and effectiveness of the proposed models are demonstrated using real life datasets collected by real sensor network projects. Experimental results show that the proposed models have advantages over existing models in terms of efficient utilization of sensor limited resources. The results further reveal that the proposed models achieve better detection effectiveness in terms of high detection accuracy with low false alarms especially for dynamic environmental data streams compared to some existing models. <s> BIB008 </s> Big Data Quality: A systematic literature review and future research directions <s> Source dependent limitations <s> Performing predictive modelling, such as anomaly detection, in Big Data is a difficult task. This problem is compounded as more and more sources of Big Data are generated from environmental sensors, logging applications, and the Internet of Things. Further, most current techniques for anomaly detection only consider the content of the data source, i.e. the data itself, without concern for the context of the data. As data becomes more complex it is increasingly important to bias anomaly detection techniques for the context, whether it is spatial, temporal, or semantic. The work proposed in this paper outlines a contextual anomaly detection technique for use in streaming sensor networks. The technique uses a well-defined content anomaly detection algorithm for real-time point anomaly detection. Additionally, we present a post-processing context-aware anomaly detection algorithm based on sensor profiles, which are groups of contextually similar sensors generated by a multivariate clustering algorithm. Our proposed research has been implemented and evaluated with real-world data provided by Powersmiths, located in Brampton, Ontario, Canada. <s> BIB009 </s> Big Data Quality: A systematic literature review and future research directions <s> Source dependent limitations <s> In our Big Data era, data is being generated, collected and analyzed at an unprecedented scale, and data-driven decision making is sweeping through all aspects of society. Recent studies have shown that poor quality data is prevalent in large databases and on the Web. Since poor quality data can have serious consequences on the results of data analyses, the importance of veracity, the fourth ‘V’ of big data is increasingly being recognized. In this tutorial, we highlight the substantial challenges that the first three ‘V’s, volume, velocity and variety, bring to dealing with veracity in big data. Due to the sheer volume and velocity of data, one needs to understand and (possibly) repair erroneous data in a scalable and timely manner. With the variety of data, often from a diversity of sources, data quality rules cannot be specified a priori; one needs to let the “data to speak for itself” in order to discover the semantics of the data. This tutorial presents recent results that are relevant to big data quality management, focusing on the two major dimensions of (i) discovering quality issues from the data itself, and (ii) trading-off accuracy vs efficiency, and identifies a range of open problems for the community. <s> BIB010 </s> Big Data Quality: A systematic literature review and future research directions <s> Source dependent limitations <s> Trust and reputation systems are widely employed in WSNs to help decision making processes by assessing trustworthiness of sensors as well as the reliability of the reported data. Iterative filtering (IF) algorithms hold great promise for such a purpose; they simultaneously estimate the aggregate value of the readings and assess the trustworthiness of the nodes. Such algorithms, however, operate by batch processing over a widow of data reported by the nodes, which represents a difficulty in applications involving streaming data. In this paper, we propose STRIF (Streaming IF) which extends IF algorithms to data streaming by leveraging a novel method for updating the sensors' variances. We compare the performance of STRIF algorithm to several batch processing IF algorithms through extensive experiments across a wide variety of configurations over both real-world and synthetic datasets. Our experimental results demonstrate that STRIF can process data streams much more efficiently than the batch algorithms while keeping the accuracy of the data aggregation close to that of the batch IF algorithm. <s> BIB011 </s> Big Data Quality: A systematic literature review and future research directions <s> Source dependent limitations <s> Ensemble Stream Modeling and Data-cleaning are sensor information processing systems have different training and testing methods by which their goals are cross-validated. This research examines a mechanism, which seeks to extract novel patterns by generating ensembles from data. The main goal of label-less stream processing is to process the sensed events to eliminate the noises that are uncorrelated, and choose the most likely model without over fitting thus obtaining higher model confidence. Higher quality streams can be realized by combining many short streams into an ensemble which has the desired quality. The framework for the investigation is an existing data mining tool. ::: First, to accommodate feature extraction such as a bush or natural forest-fire event we make an assumption of the burnt area (BA*), sensed ground truth as our target variable obtained from logs. Even though this is an obvious model choice the results are disappointing. The reasons for this are two: One, the histogram of fire activity is highly skewed. Two, the measured sensor parameters are highly correlated. Since using non descriptive features does not yield good results, we resort to temporal features. By doing so we carefully eliminate the averaging effects; the resulting histogram is more satisfactory and conceptual knowledge is learned from sensor streams. ::: Second is the process of feature induction by cross-validating attributes with single or multi-target variables to minimize training error. We use F-measure score, which combines precision and accuracy to determine the false alarm rate of fire events. The multi-target data-cleaning trees use information purity of the target leaf-nodes to learn higher order features. A sensitive variance measure such as ƒ-test is performed during each node's split to select the best attribute. Ensemble stream model approach proved to improve when using complicated features with a simpler tree classifier. ::: The ensemble framework for data-cleaning and the enhancements to quantify quality of fitness (30% spatial, 10% temporal, and 90% mobility reduction) of sensor led to the formation of streams for sensor-enabled applications. Which further motivates the novelty of stream quality labeling and its importance in solving vast amounts of real-time mobile streams generated today. <s> BIB012 </s> Big Data Quality: A systematic literature review and future research directions <s> Source dependent limitations <s> Data mining is one of the most exciting fields of research for the researcher. As data is getting digitized, systems are getting connected and integrated, scope of data generation and analytics has increased exponentially. Today, most of the systems generate non-stationary data of huge, size, volume, occurrence speed, fast changing etc. these kinds of data are called data streams. One of the most recent trend i.e. IOT (Internet Of Things) is also promising lots of expectation of people which will ease the use of day to day activities and it could also connect systems and people together. This situation will also lead to generation of data streams, thus present and future scope of data stream mining is highly promising. Characteristics of data stream possess many challenges for the researcher; this makes analytics of such data difficult and also acts as source of inspiration for researcher. Outlier detection plays important role in any application. In this paper we reviewed different techniques of outlier detection for stream data and their issues in detail and presented results of the same. <s> BIB013 </s> Big Data Quality: A systematic literature review and future research directions <s> Source dependent limitations <s> This article investigates the evolution of data quality issues from traditional structured data managed in relational databases to Big Data. In particular, the paper examines the nature of the relationship between Data Quality and several research coordinates that are relevant in Big Data, such as the variety of data types, data sources and application domains, focusing on maps, semi-structured texts, linked open data, sensor & sensor networks and official statistics. Consequently a set of structural characteristics is identified and a systematization of the a posteriori correlation between them and quality dimensions is provided. Finally, Big Data quality issues are considered in a conceptual framework suitable to map the evolution of the quality paradigm according to three core coordinates that are significant in the context of the Big Data phenomenon: the data type considered, the source of data, and the application domain. Thus, the framework allows ascertaining the relevant changes in data quality emerging with the Big Data phenomenon, through an integrative and theoretical literature review. <s> BIB014 </s> Big Data Quality: A systematic literature review and future research directions <s> Source dependent limitations <s> A USAF sponsored MITRE research team undertook four separate, domain-specific case studies about Big Data applications. Those case studies were initial investigations into the question of whether or not data quality issues encountered in Big Data collections are substantially different in cause, manifestation, or detection than those data quality issues encountered in more traditionally sized data collections. The study addresses several factors affecting Big Data Quality at multiple levels, including collection, processing, and storage. Though not unexpected, the key findings of this study reinforce that the primary factors affecting Big Data reside in the limitations and complexities involved with handling Big Data while maintaining its integrity. These concerns are of a higher magnitude than the provenance of the data, the processing, and the tools used to prepare, manipulate, and store the data. Data quality is extremely important for all data analytics problems. From the study's findings, the "truth about Big Data" is there are no fundamentally new DQ issues in Big Data analytics projects. Some DQ issues exhibit return-s-to-scale effects, and become more or less pronounced in Big Data analytics, though. Big Data Quality varies from one type of Big Data to another and from one Big Data technology to another. <s> BIB015 </s> Big Data Quality: A systematic literature review and future research directions <s> Source dependent limitations <s> To effectively clean the large-scale, mixed and inaccurate monitoring or collective data, reduce the cost of data cache and ensure the consistent deviation detection on timing data of each cycle, a big data online cleaning algorithm based on dynamic outlier detection has been proposed. The data cleaning method is improved by local outliner detection upon density, sampling cluster uniformly dilution Euclidean distance matrix retaining some corrections into next cycle of cleaning, which avoids a sampling causing overall cleaning deviation and reduces amount of calculation within data cleaning stable time, enhancing the speed greatly. Finally, the distributed solutions on online cleaning algorithm based on Hadoop platform. <s> BIB016 </s> Big Data Quality: A systematic literature review and future research directions <s> Source dependent limitations <s> AbstractWireless Sensor Networks (WSNs) are widely used for monitoring physical happenings of the environment. However, the data gathered by the WSNs may be inaccurate and unreliable due to power exhaustion, noise and other reasons. Unnecessary data such as erroneous data and redundant data transmission causes a lot of extra energy consumption. To improve the data reliability and reduce the energy consumption, we proposed an in-network processing architecture for data cleaning, which divides the task into four stages implemented in different nodes respectively. This strategy guaranteed the cleaning algorithms were computationally lightweight in local nodes and energy-efficient due to almost no communication overhead. In addition, we presented the detection algorithms for data faults and event outliers, which were conducted by utilizing the related attributes from the local sensor node and the cooperation with its relaying neighbor. Experiment results show that our proposed approach is accurate and energy-... <s> BIB017 </s> Big Data Quality: A systematic literature review and future research directions <s> Source dependent limitations <s> With the fast advance of big data technology and analytics solutions, big data computing and service is becoming a very hot research and application subject in academic research, industry community, and government services. Nevertheless, there are increasing data quality problems resulting in erroneous data costs in enterprises and businesses. Current research seldom discusses how to effectively validate big data to ensure data quality. This paper provides informative discussions for big data validation and quality assurance, including the essential concepts, focuses, and validation process. Moreover, the paper presents a comparison among big data validation tools and several major players in industry are discussed. Furthermore, the primary issues, challenges, and needs are discussed. <s> BIB018 </s> Big Data Quality: A systematic literature review and future research directions <s> Source dependent limitations <s> Since a low-quality data may influence the effectiveness and reliability of applications, data quality is required to be guaranteed. Data quality assessment is considered as the foundation of the promotion of data quality, so it is essential to access the data quality before any other data related activities. In the electric power industry, more and more electric power data is continuously accumulated, and many electric power applications have been developed based on these data. In China, the power grid has many special characteristic, traditional big data assessment frameworks cannot be directly applied. Therefore, a big data framework for electric power data quality assessment is proposed. Based on big data techniques, the framework can accumulate both the real-time data and the history data, provide an integrated computation environment for electric power big data assessment, and support the storage of different types of data. <s> BIB019 </s> Big Data Quality: A systematic literature review and future research directions <s> Source dependent limitations <s> In the era of big data and Internet of things, massive sensor data are gathered with Internet of things. Quantity of data captured by sensor networks are considered to contain highly useful and valuable information. However, for a variety of reasons, received sensor data often appear abnormal. Therefore, effective anomaly detection methods are required to guarantee the quality of data collected by those sensor nodes. Since sensor data are usually correlated in time and space, not all the gathered data are valuable for further data processing and analysis. Preprocessing is necessary for eliminating the redundancy in gathered massive sensor data. In this paper, the proposed work defines a sensor data preprocessing framework. It is mainly composed of two parts, i.e., sensor data anomaly detection and sensor data redundancy elimination. In the first part, methods based on principal statistic analysis and Bayesian network is proposed for sensor data anomaly detection. Then, approaches based on static Bayesian network (SBN) and dynamic Bayesian networks (DBNs) are proposed for sensor data redundancy elimination. Static sensor data redundancy detection algorithm (SSDRDA) for eliminating redundant data in static datasets and real-time sensor data redundancy detection algorithm (RSDRDA) for eliminating redundant sensor data in real-time are proposed. The efficiency and effectiveness of the proposed methods are validated using real-world gathered sensor datasets. <s> BIB020 </s> Big Data Quality: A systematic literature review and future research directions <s> Source dependent limitations <s> With the rapid development of social networks, Internet of things, Cloud computing as well as other technologies, big data age is arriving. The increasing number of data has brought great value to the public and enterprises. Meanwhile how to manage and use big data better has become the focus of all walks of life. The 4V characteristics of big data have brought a lot of issues to the big data processing. The key to big data processing is to solve data quality issue, and to ensure data quality is a prerequisite for the successful application of big data technique. In this paper, we use recommendation systems and prediction systems as typical big data applications, and try to find out the data quality issues during data collection, data preprocessing, data storage and data analysis stages of big data processing. According to the elaboration and analysis of the proposed issues, the corresponding solutions are also put forward. Finally, some open problems to be solved in the future are also raised. <s> BIB021 </s> Big Data Quality: A systematic literature review and future research directions <s> Source dependent limitations <s> With the development of Smart Grid Technology, more and more electric power sensor data are utilized in various electric power systems. To guarantee the effectiveness of such systems, it is necessary to ensure the quality of electric power sensor data, especially when the scale of electric power sensor data is large. In the field of large-scale electric power sensor data cleaning, the computational efficiency and accuracy of data cleaning are two vital requirements. In order to satisfy these requirements, this paper presents an electric power sensor data oriented data cleaning solution, which is composed of a data cleaning framework and a data cleaning method. Based on Hadoop, the given framework is able to support large-scale electric power sensor data acquisition, storage and processing. Meanwhile, the proposed method which achieves outlier detection and reparation is implemented on the basis of a time-relevant k-means clustering algorithm in Spark. The feasibility and effectiveness of the proposed method is evaluated on a data set which originates from charging piles. Experimental results show that the proposed data cleaning method is able to improve the data quality of electric power sensor data by finding and repairing most outliers. For large-scale electric power sensor data, the proposed data cleaning method has high parallel performance and strong scalability. <s> BIB022 </s> Big Data Quality: A systematic literature review and future research directions <s> Source dependent limitations <s> The concept of Top-N local outlier that focuses on the detection of the N points with the largest Local Outlier Factor (LOF) score has been shown to be very effective for identifying outliers in big datasets. However, detecting Top-N local outliers is computationally expensive, since the computation of LOF scores for all data points requires a huge number of high complexity k-nearest neighbor (kNN) searches. In this work, we thus present the first distributed solution to tackle this problem of Top-N local outlier detection (DTOLF). First, DTOLF features an innovative safe elimination strategy that efficiently identifies dually-safe points, namely those that are guaranteed to (1) not be classified as Top-N outliers and (2) not be needed as neighbors of points residing on other machines. Therefore, it effectively minimizes both the processing and communication costs of the Top-N outlier detection process. Further, based on the well-accepted observation that strong correlations among attributes are prevalent in real world datasets, we propose correlation-aware optimization strategies that ensure the effectiveness of grid-based partitioning and of the safe elimination strategy in multi-dimensional datasets. Our extensive experimental evaluation on OpenStreetMap, SDSS, and TIGER datasets demonstrates the effectiveness of DTOLF — up to 10 times faster than the alternative methods and scaling to terabyte level datasets. <s> BIB023 </s> Big Data Quality: A systematic literature review and future research directions <s> Source dependent limitations <s> Abstract Context: Big Data systems are a class of software systems that ingest, store, process and serve massive amounts of heterogeneous data, from multiple sources. Despite their undisputed impact in current society, their engineering is still in its infancy and companies find it difficult to adopt them due to their inherent complexity. Existing attempts to provide architectural guidelines for their engineering fail to take into account important Big Data characteristics, such as the management, evolution and quality of the data. Objective: In this paper, we follow software engineering principles to refine the λ -architecture, a reference model for Big Data systems, and use it as seed to create Bolster , a software reference architecture (SRA) for semantic-aware Big Data systems. Method: By including a new layer into the λ -architecture, the Semantic Layer, Bolster is capable of handling the most representative Big Data characteristics (i.e., Volume, Velocity, Variety, Variability and Veracity). Results: We present the successful implementation of Bolster in three industrial projects, involving five organizations. The validation results show high level of agreement among practitioners from all organizations with respect to standard quality factors. Conclusion: As an SRA, Bolster allows organizations to design concrete architectures tailored to their specific needs. A distinguishing feature is that it provides semantic-awareness in Big Data Systems. These are Big Data system implementations that have components to simplify data definition and exploitation. In particular, they leverage metadata (i.e., data describing data) to enable (partial) automation of data exploitation and to aid the user in their decision making processes. This simplification supports the differentiation of responsibilities into cohesive roles enhancing data governance. <s> BIB024 </s> Big Data Quality: A systematic literature review and future research directions <s> Source dependent limitations <s> The internet of Things (IoT) is a data stream environment where a large scale deployment of smart things continuously report readings. These data streams are then consumed by pervasive applications, i.e. data consumers, to offer ubiquitous services. The data quality (DQ) is a key criteria for IoT data consumers especially when considering the inherent uncertainty of sensor-enabled data. However, DQ is a highly subjective concept and there is no standard agreement on how to determine “good” data. Moreover, the combinations of considered measured attributes and associated DQ information are as diverse as the needs of data consumers. This introduces expensive overheads for developers tasked with building DQ-aware IoT software systems which are capable of managing their own DQ information. To effectively handle these various perceptions of DQ, we propose a Model-Driven Architecture-based approach that allows each developer to easily and efficiently express, through models and other provided resources, the data consumer’s vision of DQ and its requirements using an easy-to-use graphical model editor. The defined DQ specifications are then automatically transformed to generate an entire infrastructure for DQ management that fits perfectly the data consumer’s requirements. We demonstrate the flexibility and the efficiency of our approach by generating two DQ management infrastructures built on top of different platforms and testing them through a real life data stream environment scenario. <s> BIB025 </s> Big Data Quality: A systematic literature review and future research directions <s> Source dependent limitations <s> Abstract Big data changed the way in which we collect and analyze data. In particular, the amount of available information is constantly growing and organizations rely more and more on data analysis in order to achieve their competitive advantage. However, such amount of data can create a real value only if combined with quality: good decisions and actions are the results of correct, reliable and complete data. In such a scenario, methods and techniques for the Data Quality assessment can support the identification of suitable data to process. If for traditional database numerous assessment methods are proposed, in the Big Data scenario new algorithms have to be designed in order to deal with novel requirements related to variety, volume and velocity issues. In particular, in this paper we highlight that dealing with heterogeneous sources requires an adaptive approach able to trigger the suitable quality assessment methods on the basis of the data type and context in which data have to be used. Furthermore, we show that in some situations it is not possible to evaluate the quality of the entire dataset due to performance and time constraints. For this reason, we suggest to focus the Data Quality assessment only on a portion of the dataset and to take into account the consequent loss of accuracy by introducing a confidence factor as a measure of the reliability of the quality assessment procedure. We propose a methodology to build a Data Quality adapter module, which selects the best configuration for the Data Quality assessment based on the user main requirements: time minimization, confidence maximization, and budget minimization. Experiments are performed by considering real data gathered from a smart city case study. <s> BIB026 </s> Big Data Quality: A systematic literature review and future research directions <s> Source dependent limitations <s> The combination of data and technology is having a high impact on the way we live. The world is getting smarter thanks to the quantity of collected and analyzed data. However, it is necessary to consider that such amount of data is continuously increasing and it is necessary to deal with novel requirements related to variety, volume, velocity, and veracity issues. In this paper we focus on veracity that is related to the presence of uncertain or imprecise data: errors, missing or invalid data can compromise the usefulness of the collected values. In such a scenario, new methods and techniques able to evaluate the quality of the available data are needed. In fact, the literature provides many data quality assessment and improvement techniques, especially for structured data, but in the Big Data era new algorithms have to be designed. We aim to provide an overview of the issues and challenges related to Data Quality assessment in the Big Data scenario. We also propose a possible solution developed by considering a smart city case study and we describe the lessons learned in the design and implementation phases. <s> BIB027 </s> Big Data Quality: A systematic literature review and future research directions <s> Source dependent limitations <s> With the advances in communication technologies and the high amount of data generated, collected, and stored, it becomes crucial to manage the quality of this data deluge in an efficient and cost-effective way. The storage, processing, privacy and analytics are the main keys challenging aspects of Big Data that require quality evaluation and monitoring. Quality has been recognized by the Big Data community as an essential facet of its maturity. Yet, it is a crucial practice that should be implemented at the earlier stages of its lifecycle and progressively applied across the other key processes. The earlier we incorporate quality the full benefit we can get from insights. In this paper, we first identify the key challenges that necessitates quality evaluation. We then survey, classify and discuss the most recent work on Big Data management. Consequently, we propose an across-the-board quality management framework describing the key quality evaluation practices to be conducted through the different Big Data stages. The framework can be used to leverage the quality management and to provide a roadmap for Data scientists to better understand quality practices and highlight the importance of managing the quality. We finally, conclude the paper and point to some future research directions on quality of Big Data. <s> BIB028
Some limitations depend on the sources that generate the data. For example, if data is generated by a sensor, the constraints of the sensor, e.g. its lack of required precision, can affect the data quality. Regarding the source-dependent limitations, we have identified the following groups:  Resource constraints: The process of quality assessment requires computational capacity and enough time in order to have better quality data, which is mentioned in BIB011 ,  , BIB005 , BIB004 , BIB012 , BIB001 - BIB002 , BIB025 , BIB019 , BIB017 , BIB007 - BIB013 ,  - BIB003 , BIB014 . In some data production environments such as the sensor networks and the Internet of things that the sensors are responsible for generating data, there are computational, communication and memory limitations that do not allow a powerful evaluation process to be performed.  Source heterogeneity: Source heterogeneity is another challenge to achieving the assessment goal BIB020 , BIB008 , BIB026 , BIB027 , BIB007 , BIB006 ,  - BIB003 , BIB015 , BIB021 . On the social networks, the heterogeneity challenge of the network structure is a serious problem. The difference in structure may lead to unusual behaviors, and this creates different data distributions. As a result of the different distributions, there is a severe problem of detecting and cleaning data.  Scalability: Another critical challenge, at the source level, is scalability limitation that is discussed in  , BIB022 , BIB016 - BIB009 , BIB023 ,  , BIB019 , BIB024 , BIB007 , BIB006 ,  , BIB028 , BIB018 , BIB021 , BIB010 . In environments where it is not possible, or cost effective, to distribute data or employ a distributed algorithm, evaluation methods cannot be implemented in a scalable manner. The use of distributed mechanisms can help to increase the scalability and efficiency of the algorithm.