A Survey of Distributed Data Stream Processing Frameworks <s> I. INTRODUCTION <s> MapReduce is a programming model and an associated implementation for processing and generating large data sets. Users specify a map function that processes a key/value pair to generate a set of intermediate key/value pairs, and a reduce function that merges all intermediate values associated with the same intermediate key. Many real world tasks are expressible in this model, as shown in the paper. ::: ::: Programs written in this functional style are automatically parallelized and executed on a large cluster of commodity machines. The run-time system takes care of the details of partitioning the input data, scheduling the program's execution across a set of machines, handling machine failures, and managing the required inter-machine communication. This allows programmers without any experience with parallel and distributed systems to easily utilize the resources of a large distributed system. ::: ::: Our implementation of MapReduce runs on a large cluster of commodity machines and is highly scalable: a typical MapReduce computation processes many terabytes of data on thousands of machines. Programmers find the system easy to use: hundreds of MapReduce programs have been implemented and upwards of one thousand MapReduce jobs are executed on Google's clusters every day. <s> BIB001 </s> A Survey of Distributed Data Stream Processing Frameworks <s> I. INTRODUCTION <s> There are many academic and commercial stream processing engines (SPEs) today, each of them with its own execution semantics. This variation may lead to seemingly inexplicable differences in query results. In this paper, we present SECRET, a model of the behavior of SPEs. SECRET is a descriptive model that allows users to analyze the behavior of systems and understand the results of window-based queries for a broad range of heterogeneous SPEs. The model is the result of extensive analysis and experimentation with several commercial and academic engines. In the paper, we describe the types of heterogeneity found in existing engines, and show with experiments on real systems that our model can explain the key differences in windowing behavior. <s> BIB002 </s> A Survey of Distributed Data Stream Processing Frameworks <s> I. INTRODUCTION <s> In recent years, real-time processing and analytics systems for big data--in the context of Business Intelligence (BI)--have received a growing attention. The traditional BI platforms that perform regular updates on daily, weekly or monthly basis are no longer adequate to satisfy the fast-changing business environments. However, due to the nature of big data, it has become a challenge to achieve the real-time capability using the traditional technologies. The recent distributed computing technology, MapReduce, provides off-the-shelf high scalability that can significantly shorten the processing time for big data; Its open-source implementation such as Hadoop has become the de-facto standard for processing big data, however, Hadoop has the limitation of supporting real-time updates. The improvements in Hadoop for the real-time capability, and the other alternative real-time frameworks have been emerging in recent years. This paper presents a survey of the open source technologies that support big data processing in a real-time/near real-time fashion, including their system architectures and platforms. <s> BIB003 </s> A Survey of Distributed Data Stream Processing Frameworks <s> I. INTRODUCTION <s> Unbounded, unordered, global-scale datasets are increasingly common in day-to-day business (e.g. Web logs, mobile usage statistics, and sensor networks). At the same time, consumers of these datasets have evolved sophisticated requirements, such as event-time ordering and windowing by features of the data themselves, in addition to an insatiable hunger for faster answers. Meanwhile, practicality dictates that one can never fully optimize along all dimensions of correctness, latency, and cost for these types of input. As a result, data processing practitioners are left with the quandary of how to reconcile the tensions between these seemingly competing propositions, often resulting in disparate implementations and systems. ::: ::: We propose that a fundamental shift of approach is necessary to deal with these evolved requirements in modern data processing. We as a field must stop trying to groom unbounded datasets into finite pools of information that eventually become complete, and instead live and breathe under the assumption that we will never know if or when we have seen all of our data, only that new data will arrive, old data may be retracted, and the only way to make this problem tractable is via principled abstractions that allow the practitioner the choice of appropriate tradeoffs along the axes of interest: correctness, latency, and cost. ::: ::: In this paper, we present one such approach, the Dataflow Model, along with a detailed examination of the semantics it enables, an overview of the core principles that guided its design, and a validation of the model itself via the real-world experiences that led to its development. <s> BIB004 </s> A Survey of Distributed Data Stream Processing Frameworks <s> I. INTRODUCTION <s> For getting up-to-date insight into online services, extracted data has to be processed in near real time. For example, major big data companies (Facebook, LinkedIn, Twitter) analyse streaming data for development of new services. Several technologies have been developed, which could be selected for implementation of stream processing functionalities. The contribution of this paper is feasibility analysis of technologies for stream-based processing of semi-structured data. Particularly, feasibility of a Big Data management system for semi-structured data (AsterixDB) will be compared to Spark streaming, which has been integrated with Cassandra NoSQL database for persistence. The study focuses on stream processing in a simulated social media use case (tweet analysis), which has been implemented to Eucalyptus cloud computing environment on a distributed shared memory multiprocessor platform. The results indicate that AsterixDB is able to provide significantly better performance both in terms of throughput and latency, when data feed functionality of AsterixDB is used, and stream processing has been implemented with Java. AsterixDB also scaled on the same level or better, when the amount of nodes on the cloud platform was increased. However, stream processing in AsterixDB was delayed by batching of data, when tweets were streamed into the database with data feeds. <s> BIB005 </s> A Survey of Distributed Data Stream Processing Frameworks <s> I. INTRODUCTION <s> The immense growth of data demands switching from traditional data processing solutions to systems, which can process a continuous stream of real time data. Various applications employ stream processing systems to provide solutions to emerging Big Data problems. Open-source solutions such as Storm, Spark Streaming, and S4 are the attempts to answer key stream processing questions. The recent introduction of real time stream processing commercial solutions such as Amazon Kinesis, IBM Infosphere Stream reflect industry requirements. The system and application related challenges to handle massive stream of real time data analytics are an active field of research. ::: In this paper, we present a comparative analysis of the existing state-of-the-art stream processing solutions. We also include various application domains, which are transforming their business model to benefit from these large scale stream processing systems. <s> BIB006 </s> A Survey of Distributed Data Stream Processing Frameworks <s> I. INTRODUCTION <s> Large and complex amounts of data are being generated where traditional data processing applications are inadequate to deal with them. On the other hand, there is a growing need for extracting information from operational data in real time, which is crucial in fast changing situation. The faster one can harness insights from data, the greater the benefit in driving value, taking, reducing costs, and increasing efficiency. This paper surveys different approaches in real-time analytics of Big Data or near real-time in the specific fields of application, as well as tools and techniques being used. The survey results indicate what technologies have been used in each of the fields of application and what the reason for the choice was. <s> BIB007 </s> A Survey of Distributed Data Stream Processing Frameworks <s> I. INTRODUCTION <s> Under several emerging application scenarios, such as in smart cities, operational monitoring of large infrastructure, wearable assistance, and Internet of Things, continuous data streams must be processed under very short delays. Several solutions, including multiple software engines, have been developed for processing unbounded data streams in a scalable and efficient manner. More recently, architecture has been proposed to use edge computing for data stream processing. This paper surveys state of the art on stream processing engines and mechanisms for exploiting resource elasticity features of cloud computing in stream processing. Resource elasticity allows for an application or service to scale out/in according to fluctuating demands. Although such features have been extensively investigated for enterprise applications, stream processing poses challenges on achieving elastic systems that can make efficient resource management decisions based on current load. Elasticity becomes even more challenging in highly distributed environments comprising edge and cloud computing resources. This work examines some of these challenges and discusses solutions proposed in the literature to address them. <s> BIB008 </s> A Survey of Distributed Data Stream Processing Frameworks <s> I. INTRODUCTION <s> Data Stream Processing (DSP) has emerged as a key enabler to develop pervasive services that require to process data in a near real-time fashion. DSP applications keep up with the high volume of produced data by scaling their execution on multiple computing nodes, so as to process the incoming data flow in parallel. Workloads variability requires to elastically adapt the application parallelism at run-time in order to avoid over-provisioning. Elasticity policies for DSP have been widely investigated, but mostly under the simplifying assumption of homogeneous infrastructures. The resulting solutions do not capture the richness and inherent complexity of modern infrastructures, where heterogeneous computing resources are available on-demand. In this paper, we formulate the problem of controlling elasticity on heterogeneous resources as a Markov Decision Process (MDP). The resulting MDP is not easily solved by traditional techniques due to state space explosion, and thus we show how linear Function Approximation and Tile Coding can be used to efficiently compute elasticity policies at run-time. In order to deal with parameters uncertainty, we integrate the proposed approach with Reinforcement Learning algorithms. Our numerical evaluation shows the efficacy of the presented solutions compared to standard methods in terms of accuracy and convergence speed. <s> BIB009 </s> A Survey of Distributed Data Stream Processing Frameworks <s> I. INTRODUCTION <s> Recently, big data streams have become ubiquitous due to the fact that a number of applications generate a huge amount of data at a great velocity. This made it difficult for existing data mining tools, technologies, methods, and techniques to be applied directly on big data streams due to the inherent dynamic characteristics of big data. In this paper, a systematic review of big data streams analysis which employed a rigorous and methodical approach to look at the trends of big data stream tools and technologies as well as methods and techniques employed in analysing big data streams. It provides a global view of big data stream tools and technologies and its comparisons. Three major databases, Scopus, ScienceDirect and EBSCO, which indexes journals and conferences that are promoted by entities such as IEEE, ACM, SpringerLink, and Elsevier were explored as data sources. Out of the initial 2295 papers that resulted from the first search string, 47 papers were found to be relevant to our research questions after implementing the inclusion and exclusion criteria. The study found that scalability, privacy and load balancing issues as well as empirical analysis of big data streams and technologies are still open for further research efforts. We also found that although, significant research efforts have been directed to real-time analysis of big data stream not much attention has been given to the preprocessing stage of big data streams. Only a few big data streaming tools and technologies can do all of the batch, streaming, and iterative jobs; there seems to be no big data tool and technology that offers all the key features required for now and standard benchmark dataset for big data streaming analytics has not been widely adopted. In conclusion, it was recommended that research efforts should be geared towards developing scalable frameworks and algorithms that will accommodate data stream computing mode, effective resource allocation strategy and parallelization issues to cope with the ever-growing size and complexity of data. <s> BIB010
The ability to handle and process continuous data streams is becoming an essential part of building a data-driven organization. Data streams are sequences of unbounded tuples generated continuously in time BIB002 . Unbounded and global datasets such as Web logs, mobile usage statistics, and sensor networks are increasingly becoming common in day-to-day businesses BIB004 . Streaming data is also increasingly getting important for social media platforms such as Facebook, Twitter, and LinkedIn BIB005 . Unlike traditional batch processing which involves processing of static data, streaming or online processing involves processing dynamic or continuous data  . Extracting meaningful and timely insights from unbounded data is very challenging. Recent demands for scalable and low latency The associate editor coordinating the review of this manuscript and approving it for publication was Fabrizio Messina . analytics tools  have revealed various shortcomings of traditional batch processing systems such as MapReduce BIB001 and Hive  . Batch processing systems suffer from latency problems due to the need to collect input data into batches before it can be processed. Under several application scenarios such as fraud detection in financial transactions and healthcare analytics involving digital sensors and Internet of Things (IoT), continuous data streams must be processed under very short delays BIB008 . This is because certain types of data streams such as stock values, credit card transactions, traffic conditions, time-sensitive patient data, and trending news rapidly depreciate in value if not processed instantly. Thus, the ability to handle and process continuous streams of data is becoming an essential part of building today's datadriven organizations. Currently many tools exist for ingesting, processing, storing, indexing, and managing streaming data, which makes it a difficult task for practitioners to select the right combination of tools and platforms for building data stream analytics applications  . The main motivation of this paper is to strive to alleviate this task by first identifying the major components of a modern Data Stream Processing System (DSPS). Before choosing a DSPS or any of its components, it is important to understand and evaluate why it is necessary, what is needed and what are the available options. Data Stream Processing Engines (DSPEs) lie at the core of DSPSs and enable the definition and execution of stream processing pipelines. According to Russo et al. BIB009 , DSPEs have emerged as key enablers to develop pervasive services that process data in a near real-time fashion. However, with the myriad of DSPEs that exist today, the large design space for DSPSs, and the intermingle of technical and marketing content from the different vendors and platforms, it is challenging to map design requirements to efficient and compatible hybrid solutions. This underscores the need for systematic studies to showcase advances in tools enabling data stream processing and open challenges, which form the basis of further research. A number of literature reviews on DSPEs have emerged in recent years. Some of the early DSPE survey papers include the reports by Kamburugamuve et al.  and Bockermann  , a selective survey of tools enabling data stream processing by Gorawski et al.  and a survey of open source near real-time technologies by Liu et al. BIB003 . Other surveys of DSPEs and their applications in various domains have been outlined in BIB006 - BIB007 . Recent surveys on DSPEs include a taxonomy of DSPEs by Zhao et al.  , a global view of some popular DSPEs by Kolajo et al BIB010 and a performance comparison of some of the existing DSPEs with regards to time-series analysis by Gehring et al.  . Our work extends these recent studies by reviewing both design and implementation aspects of DSPEs unlike most of the previous surveys which focus either on the design, or the implementation aspect. Besides explaining the usefulness and the key features of DSPEs, this paper also provides insights on the design and implementation choices such as when DSPEs are not the most suitable paradigms, and when to choose distributed or single-node approach. The main contributions of this paper are: 1. An overview of DSPS including usability and architecture. 2. The listing of the key features of DSPEs based on an extensive study of the state-of-the-art DSPEs, which we used to propose a taxonomy of DSPEs. 3. A literature review of DSPEs based on the proposed taxonomy and a critical review of some of the popular DSPEs. 4. A comparison of DSPEs based on the key features. 5. A real-world DSPS use case scenario and the rational for selecting the most appropriate DSPE. 6. A discussion on the open research problems in this area. The rest of the paper is organized as follows. Section 2 describes an overview of the architecture and the main components of a general DSPS. Section 3 presents the key features and taxonomy of DSPEs. A literature review of DSPEs encompassing the work of both academic researchers and industry professionals is presented in Section 4. A comparative analysis of some of the cutting edge open source distributed DSPEs (Storm, Spark Streaming, Flink, Kafka Streams, and IBM Streams) is also presented in Section 4. A use case scenario and design considerations for developing a DSPS is presented in Section 5 along with discussions and recommendations for choosing a DSPE for different applications. Section 6 summarizes the paper and draws the conclusion.