Big Data Quality: A systematic literature review and future research directions <s> Inherent limitations <s> Outlier detection is a well established area of statistics but most of the existing outlier detection techniques are designed for applications where the entire dataset is available for random access. A typical outlier detection technique constructs a standard data distribution or model and identifies the deviated data points from the model as outliers. Evidently these techniques are not suitable for online data streams where the entire dataset, due to its unbounded volume, is not available for random access. Moreover, the data distribution in data streams change over time which challenges the existing outlier detection techniques that assume a constant standard data distribution for the entire dataset. In addition, data streams are characterized by uncertainty which imposes further complexity. In this paper we propose an adaptive, online outlier detection technique addressing the aforementioned characteristics of data streams, called Adaptive Outlier Detection for Data Streams (A-ODDS), which identifies outliers with respect to all the received data points as well as temporally close data points. The temporally close data points are selected based on time and change of data distribution. We also present an efficient and online implementation of the technique and a performance study showing the superiority of A-ODDS over existing techniques in terms of accuracy and execution time on a real-life dataset collected from meteorological applications. <s> BIB001 </s> Big Data Quality: A systematic literature review and future research directions <s> Inherent limitations <s> In our Big Data era, data is being generated, collected and analyzed at an unprecedented scale, and data-driven decision making is sweeping through all aspects of society. Recent studies have shown that poor quality data is prevalent in large databases and on the Web. Since poor quality data can have serious consequences on the results of data analyses, the importance of veracity, the fourth ‘V’ of big data is increasingly being recognized. In this tutorial, we highlight the substantial challenges that the first three ‘V’s, volume, velocity and variety, bring to dealing with veracity in big data. Due to the sheer volume and velocity of data, one needs to understand and (possibly) repair erroneous data in a scalable and timely manner. With the variety of data, often from a diversity of sources, data quality rules cannot be specified a priori; one needs to let the “data to speak for itself” in order to discover the semantics of the data. This tutorial presents recent results that are relevant to big data quality management, focusing on the two major dimensions of (i) discovering quality issues from the data itself, and (ii) trading-off accuracy vs efficiency, and identifies a range of open problems for the community. <s> BIB002 </s> Big Data Quality: A systematic literature review and future research directions <s> Inherent limitations <s> Data mining is one of the most exciting fields of research for the researcher. As data is getting digitized, systems are getting connected and integrated, scope of data generation and analytics has increased exponentially. Today, most of the systems generate non-stationary data of huge, size, volume, occurrence speed, fast changing etc. these kinds of data are called data streams. One of the most recent trend i.e. IOT (Internet Of Things) is also promising lots of expectation of people which will ease the use of day to day activities and it could also connect systems and people together. This situation will also lead to generation of data streams, thus present and future scope of data stream mining is highly promising. Characteristics of data stream possess many challenges for the researcher; this makes analytics of such data difficult and also acts as source of inspiration for researcher. Outlier detection plays important role in any application. In this paper we reviewed different techniques of outlier detection for stream data and their issues in detail and presented results of the same. <s> BIB003 </s> Big Data Quality: A systematic literature review and future research directions <s> Inherent limitations <s> Abstract Vast and ever increasing quantities of data are produced by sensors in the Internet of Things (IoT). The quality of this data can be very variable due to problems with sensors, incorrect calibration etc. Data quality can be greatly enhanced by cleaning the data before it reaches its end user. This paper reports on the construction of a distributed cleaning system (DCS) to clean data streams in real-time for an environmental case-study. A combination of declarative and statistical model based cleaning methods are applied and initial results are reported. <s> BIB004 </s> Big Data Quality: A systematic literature review and future research directions <s> Inherent limitations <s> With the abundance of raw data generated from various sources, Big Data has become a preeminent approach in acquiring, processing, and analyzing large amounts of heterogeneous data to derive valuable evidences. The size, speed, and formats in which data is generated and processed affect the overall quality of information. Therefore, Quality of Big Data (QBD) has become an important factor to ensure that the quality of data is maintained at all Big data processing phases. This paper addresses the QBD at the pre-processing phase, which includes sub-processes like cleansing, integration, filtering, and normalization. We propose a QBD model incorporating processes to support Data quality profile selection and adaptation. In addition, it tracks and registers on a data provenance repository the effect of every data transformation happened in the pre-processing phase. We evaluate the data quality selection module using large EEG dataset. The obtained results illustrate the importance of addressing QBD at an early phase of Big Data processing lifecycle since it significantly save on costs and perform accurate data analysis. <s> BIB005 </s> Big Data Quality: A systematic literature review and future research directions <s> Inherent limitations <s> Although most NoSQL Data Stores are schema-less, information on the structural properties of the persisted data is nevertheless essential during application development. Otherwise, accessing the data becomes simply impractical. In this paper, we introduce an algorithm for schema extraction that is operating outside of the NoSQL data store. Our method is specifically targeted at semi-structured data persisted in NoSQL stores, e.g., in JSON format. Rather than designing the schema up front, extracting a schema in hindsight can be seen as a reverse-engineering step. Based on the extracted schema information, we propose set of similarity measures that capture the degree of heterogeneity of JSON data and which reveal structural outliers in the data. We evaluate our implementation on two real-life datasets: a database from the Wendelstein 7-X project and Web Performance Data. <s> BIB006 </s> Big Data Quality: A systematic literature review and future research directions <s> Inherent limitations <s> Though the issues of data quality trace back their origin to the early days of computing, the recent emergence of Big Data has added more dimensions. Furthermore, given the range of Big Data applications, potential consequences of bad data quality can be for more disastrous and widespread. This paper provides a perspective on data quality issues in the Big Data context. it also discusses data integration issues that arise in biological databases and attendant data quality issues. <s> BIB007 </s> Big Data Quality: A systematic literature review and future research directions <s> Inherent limitations <s> The main aim of data stream outlier detection is to find the data stream outliers in rational time accurately. The existing outlier detection algorithms can find outliers in static data sets efficiently, but they are inapplicable for the dynamic data stream, and cannot find the abnormal data effectively. Due to the requirements of real-time detection, dynamic adjustment and the inapplicability of existing algorithms on data stream outlier detection, we propose a new data stream outlier detection algorithm, ODGrid, which can find the abnormal data in data stream in real time and adjust the detection results dynamically. According to the experiments on real datasets and synthetic datasets, ODGrid is superior to the existing data stream outlier detection algorithms, and it has good scalability to the dimensionality of data space. <s> BIB008 </s> Big Data Quality: A systematic literature review and future research directions <s> Inherent limitations <s> Among many Big Data applications are those that deal with data streams. A data stream is a sequence of data points with timestamps that possesses the properties of transiency, infiniteness, uncertainty, concept drift, and multi-dimensionality. In this paper we propose an outlier detection technique called Orion that addresses all the characteristics of data streams. Orion looks for a projected dimension of multi-dimensional data points with the help of an evolutionary algorithm, and identifies a data point as an outlier if it resides in a low-density region in that dimension. Experiments comparing Orion with existing techniques using both real and synthetic datasets show that Orion achieves an average of 7X the precision, 5X the recall, and a competitive execution time compared to existing techniques. <s> BIB009 </s> Big Data Quality: A systematic literature review and future research directions <s> Inherent limitations <s> Big Data refers to data volumes in the range of Exabyte (1018) and beyond. Such volumes exceed the capacity of current on-line storage and processing systems. With characteristics like volume, velocity and variety big data throws challenges to the traditional IT establishments. Computer assisted innovation, real time data analytics, customer-centric business intelligence, industry wide decision making and transparency are possible advantages, to mention few, of Big Data. There are many issues with Big Data that warrant quality assessment methods. The issues are pertaining to storage and transport, management, and processing. This paper throws light into the present state of quality issues related to Big Data. It provides valuable insights that can be used to leverage Big Data science activities. <s> BIB010 </s> Big Data Quality: A systematic literature review and future research directions <s> Inherent limitations <s> With the fast advance of big data technology and analytics solutions, big data computing and service is becoming a very hot research and application subject in academic research, industry community, and government services. Nevertheless, there are increasing data quality problems resulting in erroneous data costs in enterprises and businesses. Current research seldom discusses how to effectively validate big data to ensure data quality. This paper provides informative discussions for big data validation and quality assurance, including the essential concepts, focuses, and validation process. Moreover, the paper presents a comparison among big data validation tools and several major players in industry are discussed. Furthermore, the primary issues, challenges, and needs are discussed. <s> BIB011 </s> Big Data Quality: A systematic literature review and future research directions <s> Inherent limitations <s> In the era of big data and Internet of things, massive sensor data are gathered with Internet of things. Quantity of data captured by sensor networks are considered to contain highly useful and valuable information. However, for a variety of reasons, received sensor data often appear abnormal. Therefore, effective anomaly detection methods are required to guarantee the quality of data collected by those sensor nodes. Since sensor data are usually correlated in time and space, not all the gathered data are valuable for further data processing and analysis. Preprocessing is necessary for eliminating the redundancy in gathered massive sensor data. In this paper, the proposed work defines a sensor data preprocessing framework. It is mainly composed of two parts, i.e., sensor data anomaly detection and sensor data redundancy elimination. In the first part, methods based on principal statistic analysis and Bayesian network is proposed for sensor data anomaly detection. Then, approaches based on static Bayesian network (SBN) and dynamic Bayesian networks (DBNs) are proposed for sensor data redundancy elimination. Static sensor data redundancy detection algorithm (SSDRDA) for eliminating redundant data in static datasets and real-time sensor data redundancy detection algorithm (RSDRDA) for eliminating redundant sensor data in real-time are proposed. The efficiency and effectiveness of the proposed methods are validated using real-world gathered sensor datasets. <s> BIB012 </s> Big Data Quality: A systematic literature review and future research directions <s> Inherent limitations <s> Abstract Context: Big Data systems are a class of software systems that ingest, store, process and serve massive amounts of heterogeneous data, from multiple sources. Despite their undisputed impact in current society, their engineering is still in its infancy and companies find it difficult to adopt them due to their inherent complexity. Existing attempts to provide architectural guidelines for their engineering fail to take into account important Big Data characteristics, such as the management, evolution and quality of the data. Objective: In this paper, we follow software engineering principles to refine the λ -architecture, a reference model for Big Data systems, and use it as seed to create Bolster , a software reference architecture (SRA) for semantic-aware Big Data systems. Method: By including a new layer into the λ -architecture, the Semantic Layer, Bolster is capable of handling the most representative Big Data characteristics (i.e., Volume, Velocity, Variety, Variability and Veracity). Results: We present the successful implementation of Bolster in three industrial projects, involving five organizations. The validation results show high level of agreement among practitioners from all organizations with respect to standard quality factors. Conclusion: As an SRA, Bolster allows organizations to design concrete architectures tailored to their specific needs. A distinguishing feature is that it provides semantic-awareness in Big Data Systems. These are Big Data system implementations that have components to simplify data definition and exploitation. In particular, they leverage metadata (i.e., data describing data) to enable (partial) automation of data exploitation and to aid the user in their decision making processes. This simplification supports the differentiation of responsibilities into cohesive roles enhancing data governance. <s> BIB013 </s> Big Data Quality: A systematic literature review and future research directions <s> Inherent limitations <s> Since a low-quality data may influence the effectiveness and reliability of applications, data quality is required to be guaranteed. Data quality assessment is considered as the foundation of the promotion of data quality, so it is essential to access the data quality before any other data related activities. In the electric power industry, more and more electric power data is continuously accumulated, and many electric power applications have been developed based on these data. In China, the power grid has many special characteristic, traditional big data assessment frameworks cannot be directly applied. Therefore, a big data framework for electric power data quality assessment is proposed. Based on big data techniques, the framework can accumulate both the real-time data and the history data, provide an integrated computation environment for electric power big data assessment, and support the storage of different types of data. <s> BIB014 </s> Big Data Quality: A systematic literature review and future research directions <s> Inherent limitations <s> A data stream is a sequence of data generated continuously over time. A data stream is too big to be saved in memory and its underlying data distribution may change over time. Outlier detection aims to find data instances which significantly deviate from the underlying data distribution. While outlier detection is performed at an individual instance level, anomalous pattern detection involves detecting a point in time where the behavior of the data becomes unusual and differs from normal behavior. Most outlier detection methods work in unsupervised mode, where the class labels of the data samples are not known. Alternatively, concept drift detection methods find a drifting point in the streaming data and try to adapt the model to the new emerging pattern. In this paper, we provide a review of outlier detection, anomaly pattern detection and concept drift detection approaches for streaming data. <s> BIB015
Some challenges are inherent to the nature of big data, and the quality evaluation algorithm has to face these constraints. Some of the problems are related to the nature of the big data, such as data volume, velocity, and variety. In this section, we have tried to highlight the challenges of big data quality assessment, so we have avoided describing such problems and only explains the inherent problems in big data quality:  Variety of arrival rate: The arrival rate can be fixed or variable. The evaluation algorithm should provide a mechanism for processing existing data before the arrival of the new incoming data. Studies  ,  , BIB003 discuss this problem.  Infinite: As in  ,  explained, in streaming data, data is continuously being received, and the evaluation process must be done online and without interruption of the main retrieval process.  Transient: In data stream, transient data is a significant challenge that is mentioned in  ,  , BIB003 . Data expires after a while and lose credibility, so data processing should be done before it expires.  Concept drift: Data distribution may change after some time, and if the evaluation algorithm fails to find the new distribution, it will not be adequately performed. One of the significant challenges in designing the algorithm is attention to the distribution of data which is considered in BIB004 , BIB001 , BIB009 ,  ,  , BIB003 , BIB015 .  Heterogeneous schema: In some cases BIB012 ,  , BIB005 , BIB006 ,  , BIB013 ,  , BIB003 , BIB010 , BIB011 , BIB007 , BIB002 , we face the heterogeneity problem of the data schema. One of the complexities of data preprocessing is to provide the right solution to deal with this issue.  Distributed data points: Data may be received from different sources, in which case it is necessary to integrate and extract the correlation of these data in order to obtain the appropriate context information. Studies  , BIB004 , BIB008 , BIB014 face this challenge.