Big Data Quality: A systematic literature review and future research directions <s> Outlier Detection Task <s> Big data has been acknowledged for its enormous potential. In contrast to the potential, in a recent survey more than half of financial service organizations reported that big data has not delivered the expected value. One of the main reasons for this is related to data quality. The objective of this research is to identify the antecedents of big data quality in financial institutions. This will help to understand how data quality from big data analysis can be improved. For this, a literature review was performed and data was collected using three case studies, followed by content analysis. The overall findings indicate that there are no fundamentally new data quality issues in big data projects. Nevertheless, the complexity of the issues is higher, which makes it harder to assess and attain data quality in big data projects compared to the traditional projects. Ten antecedents of big data quality were identified encompassing data, technology, people, process and procedure, organization, and external aspects. <s> BIB001 </s> Big Data Quality: A systematic literature review and future research directions <s> Outlier Detection Task <s> Abstract Context: Big Data systems are a class of software systems that ingest, store, process and serve massive amounts of heterogeneous data, from multiple sources. Despite their undisputed impact in current society, their engineering is still in its infancy and companies find it difficult to adopt them due to their inherent complexity. Existing attempts to provide architectural guidelines for their engineering fail to take into account important Big Data characteristics, such as the management, evolution and quality of the data. Objective: In this paper, we follow software engineering principles to refine the λ -architecture, a reference model for Big Data systems, and use it as seed to create Bolster , a software reference architecture (SRA) for semantic-aware Big Data systems. Method: By including a new layer into the λ -architecture, the Semantic Layer, Bolster is capable of handling the most representative Big Data characteristics (i.e., Volume, Velocity, Variety, Variability and Veracity). Results: We present the successful implementation of Bolster in three industrial projects, involving five organizations. The validation results show high level of agreement among practitioners from all organizations with respect to standard quality factors. Conclusion: As an SRA, Bolster allows organizations to design concrete architectures tailored to their specific needs. A distinguishing feature is that it provides semantic-awareness in Big Data Systems. These are Big Data system implementations that have components to simplify data definition and exploitation. In particular, they leverage metadata (i.e., data describing data) to enable (partial) automation of data exploitation and to aid the user in their decision making processes. This simplification supports the differentiation of responsibilities into cohesive roles enhancing data governance. <s> BIB002 </s> Big Data Quality: A systematic literature review and future research directions <s> Outlier Detection Task <s> Abstract With the rapid development of the Internet of Things (IoT), Big Data technologies have emerged as a critical data analytics tool to bring the knowledge within IoT infrastructures to better meet the purpose of the IoT systems and support critical decision making. Although the topic of Big Data analytics itself is extensively researched, the disparity between IoT domains (such as healthcare, energy, transportation and others) has isolated the evolution of Big Data approaches in each IoT domain. Thus, the mutual understanding across IoT domains can possibly advance the evolution of Big Data research in IoT. In this work, we therefore conduct a survey on Big Data technologies in different IoT domains to facilitate and stimulate knowledge sharing across the IoT domains. Based on our review, this paper discusses the similarities and differences among Big Data technologies used in different IoT domains, suggests how certain Big Data technology used in one IoT domain can be re-used in another IoT domain, and develops a conceptual framework to outline the critical Big Data technologies across all the reviewed IoT domains. <s> BIB003 </s> Big Data Quality: A systematic literature review and future research directions <s> Outlier Detection Task <s> After the rise of big data to national strategy, the application of big data in every industry is increasing. The quality of data will directly affect the value of data and influence the analysis and decision of managers. Aiming at the characteristics of big data, such as volume, velocity, variety and value, a quality management framework of big data based on application scenario is proposed, which includes data quality assessment and quality management of structured data, unstructured data and data integration stage. In view of the current structured data leading to the core business of the enterprise, we use the research method to extend the peripheral data layer by layer on the main data. Big data processing technology, such as Hadoop and Storm, is used to construct a big data cleaning system based on semantics. Combined with JStorm platform, a real-time control system for big data quality is given. Finally, a big data quality evaluation system is built to detect the effect of data integration. The framework can guarantee the output of high quality big data on the basis of traditional data quality system. It helps enterprises to understand data rules and increase the value of core data, which has practical application value. <s> BIB004
The papers in this category are divided into four sub-categories: Model-based, Learning-based, Distance based, and Context-aware. Papers that use static data in order to detect outliers are described in this section, and as mentioned earlier, review papers are discussed and compared in Section 7. There is only one paper in this category that we explain below. A big data processing platform is implemented in BIB002 which is based on the Lambda architecture BIB003 and is intended to improve it. In lambda architecture, there are three main layers: batch layer, speed layer and serving layer. In the batch layer, a copy of the input data is taken, and then the data is processed in batch, in the speed layer, the data stream is entered and processed in real-time and the results are stored and in the serving layer, the results have been delivered to the user. In the proposed method, it is claimed that the lambda architecture can handle the volume and velocity characteristics of big data, but not variety. So a layer called the metadata management system (MDM) is added to the Lambda architecture and called the new architecture Bolster. In MDM, there is a metadata repository in which all the relevant metadata are stored in an ontology. In this ontology, there are features of the input data, where they come from and what attributes they should have. When data is ingested, they are first validated by the MDM in terms of their attributes, and if they have the desired features, the processing phase begins. Otherwise, they will either be deleted or stored in the data storage. The architecture has been used in several use cases such as BDAL 11 , SUPERSEDE BIB004 , and WISCC BIB001 . In each use case, several roles have been participated to test the method. Authors asked a number of questions and participants evaluated the system based on their factors.