Big Data Quality: A systematic literature review and future research directions <s> Instance-based techniques <s> Sensors are increasingly embedded into physical products in order to capture data about their conditions and usage for decision making in business applications. However, a major issue for such applications is the limited quality of the captured data due to inherently restricted precision and performance of the sensors. Moreover, the data quality is further decreased by data processing to meet resource constraints in streaming environments and ultimately influences business decisions. The issue of how to efficiently provide applications with information about data quality (DQ) is still an open research problem. In my Ph.D. thesis, I address this problem by developing a system to provide business applications with accurate information on data quality. Furthermore, the system will be able to incorporate and guarantee user-defined data quality levels. In this paper, I will present the major results from my research so far. This includes a novel jumping-window-based approach for the efficient transfer of data quality information as well as a flexible metamodel for storage and propagation of data quality. The comprehensive analysis of common data processing operators w.r.t. their impact on data quality allows a fruitful knowledge evaluation and thus diminishes incorrect business decisions. <s> BIB001 </s> Big Data Quality: A systematic literature review and future research directions <s> Instance-based techniques <s> In smart item environments, multitude of sensors are applied to capture data about product conditions and usage to guide business decisions as well as production automation processes. A big issue in this application area is posed by the restricted quality of sensor data due to limited sensor precision as well as sensor failures and malfunctions. Decisions derived on incorrect or misleading sensor data are likely to be faulty. The issue of how to efficiently provide applications with information about data quality (DQ) is still an open research problem. In this paper, we present a flexible model for the efficient transfer and management of data quality for streaming as well as static data. We propose a data stream metamodel to allow for the propagation of data quality from the sensors up to the respective business application without a significant overhead of data. Furthermore, we present the extension of the traditional RDBMS metamodel to permit the persistent storage of data quality information in a relational database. Finally, we demonstrate a data quality metadata mapping to close the gap between the streaming environment and the target database. Our solution maintains a flexible number of DQ dimensions and supports applications directly consuming streaming data or processing data filed in a persistent database. <s> BIB002 </s> Big Data Quality: A systematic literature review and future research directions <s> Instance-based techniques <s> Sensors in smart-item environments capture data about product conditions and usage to support business decisions as well as production automation processes. A challenging issue in this application area is the restricted quality of sensor data due to limited sensor precision and sensor failures. Moreover, data stream processing to meet resource constraints in streaming environments introduces additional noise and decreases the data quality. In order to avoid wrong business decisions due to dirty data, quality characteristics have to be captured, processed, and provided to the respective business task. However, the issue of how to efficiently provide applications with information about data quality is still an open research problem. In this article, we address this problem by presenting a flexible model for the propagation and processing of data quality. The comprehensive analysis of common data stream processing operators and their impact on data quality allows a fruitful data evaluation and diminishes incorrect business decisions. Further, we propose the data quality model control to adapt the data quality granularity to the data stream interestingness. <s> BIB003 </s> Big Data Quality: A systematic literature review and future research directions <s> Instance-based techniques <s> Data Stream Management Systems (DSMS) have been proposed to address the challenges of applications which produce continuous, rapid streams of data that have to be processed in real-time. Data quality (DQ) plays an important role in DSMS as there is usually a trade-off between accuracy and consistency on the one hand, and timeliness and completeness on the other hand. Previous work on data quality in DSMS has focused only on specific aspects of DQ. In this paper, we present a flexible, holistic ontology-based data quality framework for data stream applications. Our DQ model is based on a threefold notion of DQ. First, content-based evaluation of DQ uses semantic rules which can be user- defined in an extensible ontology. Second, query-based evaluation adds DQ information to the query results and updates it while queries are being processed. Third, the application-based evaluation can use any kind of function which computes an application-specific DQ value. The whole DQ process is driven by the metadata managed in an ontology which provides a semantically clear definition of the DQ features of the DSMS. The evaluation of our approach in two case studies in the domain of traffic information systems has shown that our framework provides the required flexibility, extensibility, and performance for DQ management in DSMS. <s> BIB004 </s> Big Data Quality: A systematic literature review and future research directions <s> Instance-based techniques <s> Data Stream Management Systems (DSMS) provide real-time data processing in an effective way, but there is always a tradeoff between data quality (DQ) and performance. We propose an ontology-based data quality framework for relational DSMS that includes DQ measurement and monitoring in a transparent, modular, and flexible way. We follow a threefold approach that takes the characteristics of relational data stream management for DQ metrics into account. While (1) Query Metrics respect changes in data quality due to query operations, (2) Content Metrics allow the semantic evaluation of data in the streams. Finally, (3) Application Metrics allow easy user-defined computation of data quality values to account for application specifics. Additionally, a quality monitor allows us to observe data quality values and take counteractions to balance data quality and performance. The framework has been designed along a DQ management methodology suited for data streams. It has been evaluated in the domains of transportation systems and health monitoring. <s> BIB005 </s> Big Data Quality: A systematic literature review and future research directions <s> Instance-based techniques <s> While the potential benefits of Big Data adoption are significant, and some initial successes have already been realized, there remain many research and technical challenges that must be addressed to fully realize this potential. The Big Data processing, storage and analytics, of course, are major challenges that are most easily recognized. However, there are additional challenges related for instance to Big Data collection, integration, and quality enforcement. This paper proposes a hybrid approach to Big Data quality evaluation across the Big Data value chain. It consists of assessing first the quality of Big Data itself, which involve processes such as cleansing, filtering and approximation. Then, assessing the quality of process handling this Big Data, which involve for example processing and analytics process. We conduct a set of experiments to evaluate Quality of Data prior and after its pre-processing, and the Quality of the pre-processing and processing on a large dataset. Quality metrics have been measured to access three Big Data quality dimensions: accuracy, completeness, and consistency. The results proved that combination of data-driven and process-driven quality evaluation lead to improved quality enforcement across the Big Data value chain. Hence, we recorded high prediction accuracy and low processing time after we evaluate 6 well-known classification algorithms as part of processing and analytics phase of Big Data value chain. <s> BIB006 </s> Big Data Quality: A systematic literature review and future research directions <s> Instance-based techniques <s> The internet of Things (IoT) is a data stream environment where a large scale deployment of smart things continuously report readings. These data streams are then consumed by pervasive applications, i.e. data consumers, to offer ubiquitous services. The data quality (DQ) is a key criteria for IoT data consumers especially when considering the inherent uncertainty of sensor-enabled data. However, DQ is a highly subjective concept and there is no standard agreement on how to determine “good” data. Moreover, the combinations of considered measured attributes and associated DQ information are as diverse as the needs of data consumers. This introduces expensive overheads for developers tasked with building DQ-aware IoT software systems which are capable of managing their own DQ information. To effectively handle these various perceptions of DQ, we propose a Model-Driven Architecture-based approach that allows each developer to easily and efficiently express, through models and other provided resources, the data consumer’s vision of DQ and its requirements using an easy-to-use graphical model editor. The defined DQ specifications are then automatically transformed to generate an entire infrastructure for DQ management that fits perfectly the data consumer’s requirements. We demonstrate the flexibility and the efficiency of our approach by generating two DQ management infrastructures built on top of different platforms and testing them through a real life data stream environment scenario. <s> BIB007 </s> Big Data Quality: A systematic literature review and future research directions <s> Instance-based techniques <s> Currently, on-line monitoring and measuring system of power quality has accumulated a huge amount of data. In the age of big data, those data integrated from various systems will face big data application problems. This paper proposes a data quality assessment system method for on-line monitoring and measuring system of power quality based on big data and data provenance to assess integrity, redundancy, accuracy, timeliness, intelligence and consistency of data set and single data. Specific assessment rule which conforms to the situation of on-line monitoring and measuring system of power quality will be devised to found data quality problems. Thus it will provide strong data support for big data application of power quality. <s> BIB008
There are six papers in this category, which all of them are instance based, which means that in order to evaluate data stream, just the data record itself is used, not its schema. In BIB001 for the first time, the concept of the quality window has been proposed. This implies that data quality evaluation should be done at the window level. Accuracy, completeness, and confidence are considered as data quality dimensions. At first, each data record in the window is evaluated in terms of three dimensions, and then all data results are aggregated based on the timestamp and data attributes. Finally, the measured values are used to analyze the data and compare it with the corresponding threshold whether the data is good enough. The idea of the quality window is also used in BIB002 . The authors of the paper have argued that data analysis may not be sufficient only by the quality values of a window. Therefore, the data quality of more windows needs to be investigated. Therefore, it is suggested that not only stream data, but also quality values should be stored in relational tables. The authors have developed a database management system to store evaluated quality results. The results of data quality are calculated by the method BIB001 and are stored in a table. Finally, more data can be considered for data quality analysis. If the data quality of a window is evaluated and it is determined that the corresponding value is less than the threshold, does all the window data have low quality or because of a particular data, the final result is lower than the threshold. In BIB003 , the answer to this question is that, as soon as the data is of poor quality, the window size should be reduced. If the quality of the window data is low, then it can be easy to exclude all window data for analysis. In this paper, four functions are considered, by which they identify the suspicious data values and decide to reduce or increase the size of the window by comparing the amount of data with the given threshold. In BIB004 and BIB005 , which have the same authors, an ontology was used to evaluate the quality of data. In this architecture, three components of query, application, and content are defined. Data is first processed by the data processor. This component corrects the query using the Query-based Quality Service component. Then, the data quality evaluation is done by the Data Quality Processor and its interaction with the Application-based Quality Service and the Content-Based Quality Service. The Data Quality Monitor also analyzes the evaluated result. The information that enables this analysis is provided by the DQ Ontology component. In the Query-based Quality Service component, existing queries can also be rewritten. In the Application-based Quality Service, the user-defined quality dimensions (such as accuracy, etc.), can be defined and in the Content-based Quality Service component, there are some specific rules to evaluate the data values. For example, the maximum speed of a region is 280 km / h. The difference between the two papers is in the case study that authors have done. System performance and memory usage have been evaluated with and without data quality assessment. In the case of quality assessment, there is no significant difference in consumption. A quality assessment architecture is presented in BIB007 . In this paper, there is a system by which the data consumer defines his requirements and the system automatically stores valid data in the database. In this system that is graphically available to the user, the data consumer can select the quality dimensions and the specific data attributes. Finally, the system transforms the requirements to the executable code and executes it on the data. In the following, we explain the instance based methods. A model for improving the quality of open data is presented in  . In the first step, the data quality dimensions are evaluated, then the data quality criteria are defined, how each quality dimension is calculated. In the third step, each quality dimension is weighed to define the data quality index. In order to select a dataset, the popularity index is defined by the number of downloads, number of uses, and so on. Then the evaluations carried out by the end users are measured, and their comments are collected to be analyzed, and the quality level evaluated. A model for assessing the quality of health data is presented in BIB006 . After data collection, data quality is evaluated before and after the preprocessing stage. The quality dimensions such as accuracy, correctness and completeness are evaluated before the pre-processing, then the transformation, filtering, and other pre-processing techniques are performed, and after this stage, again the accuracy, correctness, and completeness of the data are evaluated to determine the degree of quality improvement. In order to evaluate the method, the data quality improvement has been conducted, and the results indicate that data quality has increased in big health data. In BIB008 , the number of six quality dimensions are specified for the evaluation of electrical data. These dimensions include accuracy, redundancy, integrity, consistency, timeliness, and intelligence. The data are evaluated by the specified dimensions, and the following results are obtained: 1. the number of duplicate data is very high and can affect the results, 2-In the conversion process, there is an inconsistency and 3-In the process of integrating transformer accounts to headquarters, data acquisition and data conversion are most readily caused by untimely data. The thesis  first addresses the big data quality problems such as big data management and big data processing. Then it goes into the big data validation process. This process involves the following steps: 1-Data collection; 2-Data cleaning; 3-Data transformation; 4-Data loading; and 5-Data analysis. Also, the data quality assessment tools are discussed like MS-Excel, Zoho Reports, DataCleaner, Tableau, etc. Then some quality evaluation frameworks are also mentioned. In this thesis, eight data quality dimensions have been proposed, including the completeness, accuracy, timeliness, uniqueness, validity, consistency, the reliability of gauge/sensor and usability. Each of these dimensions is evaluated in a dataset, and the output results are examined, and it is said that the results can be extended in the future.