A Survey of Distributed Data Stream Processing Frameworks <s> A. LITERATURE REVIEW <s> In recent years, real-time processing and analytics systems for big data--in the context of Business Intelligence (BI)--have received a growing attention. The traditional BI platforms that perform regular updates on daily, weekly or monthly basis are no longer adequate to satisfy the fast-changing business environments. However, due to the nature of big data, it has become a challenge to achieve the real-time capability using the traditional technologies. The recent distributed computing technology, MapReduce, provides off-the-shelf high scalability that can significantly shorten the processing time for big data; Its open-source implementation such as Hadoop has become the de-facto standard for processing big data, however, Hadoop has the limitation of supporting real-time updates. The improvements in Hadoop for the real-time capability, and the other alternative real-time frameworks have been emerging in recent years. This paper presents a survey of the open source technologies that support big data processing in a real-time/near real-time fashion, including their system architectures and platforms. <s> BIB001 </s> A Survey of Distributed Data Stream Processing Frameworks <s> A. LITERATURE REVIEW <s> The immense growth of data demands switching from traditional data processing solutions to systems, which can process a continuous stream of real time data. Various applications employ stream processing systems to provide solutions to emerging Big Data problems. Open-source solutions such as Storm, Spark Streaming, and S4 are the attempts to answer key stream processing questions. The recent introduction of real time stream processing commercial solutions such as Amazon Kinesis, IBM Infosphere Stream reflect industry requirements. The system and application related challenges to handle massive stream of real time data analytics are an active field of research. ::: In this paper, we present a comparative analysis of the existing state-of-the-art stream processing solutions. We also include various application domains, which are transforming their business model to benefit from these large scale stream processing systems. <s> BIB002 </s> A Survey of Distributed Data Stream Processing Frameworks <s> A. LITERATURE REVIEW <s> For getting up-to-date insight into online services, extracted data has to be processed in near real time. For example, major big data companies (Facebook, LinkedIn, Twitter) analyse streaming data for development of new services. Several technologies have been developed, which could be selected for implementation of stream processing functionalities. The contribution of this paper is feasibility analysis of technologies for stream-based processing of semi-structured data. Particularly, feasibility of a Big Data management system for semi-structured data (AsterixDB) will be compared to Spark streaming, which has been integrated with Cassandra NoSQL database for persistence. The study focuses on stream processing in a simulated social media use case (tweet analysis), which has been implemented to Eucalyptus cloud computing environment on a distributed shared memory multiprocessor platform. The results indicate that AsterixDB is able to provide significantly better performance both in terms of throughput and latency, when data feed functionality of AsterixDB is used, and stream processing has been implemented with Java. AsterixDB also scaled on the same level or better, when the amount of nodes on the cloud platform was increased. However, stream processing in AsterixDB was delayed by batching of data, when tweets were streamed into the database with data feeds. <s> BIB003 </s> A Survey of Distributed Data Stream Processing Frameworks <s> A. LITERATURE REVIEW <s> Large and complex amounts of data are being generated where traditional data processing applications are inadequate to deal with them. On the other hand, there is a growing need for extracting information from operational data in real time, which is crucial in fast changing situation. The faster one can harness insights from data, the greater the benefit in driving value, taking, reducing costs, and increasing efficiency. This paper surveys different approaches in real-time analytics of Big Data or near real-time in the specific fields of application, as well as tools and techniques being used. The survey results indicate what technologies have been used in each of the fields of application and what the reason for the choice was. <s> BIB004 </s> A Survey of Distributed Data Stream Processing Frameworks <s> A. LITERATURE REVIEW <s> Streaming data processing has been gaining attention due to its application into a wide range of scenarios. To serve the booming demands of streaming data processing, many computation engines have been developed. However, there is still a lack of real-world benchmarks that would be helpful when choosing the most appropriate platform for serving real-time streaming needs. In order to address this problem, we developed a streaming benchmark for three representative computation engines: Flink, Storm and Spark Streaming. Instead of testing speed-of-light event processing, we construct a full data pipeline using Kafka and Redis in order to more closely mimic the real-world production scenarios. Based on our experiments, we provide a performance comparison of the three data engines in terms of 99th percentile latency and throughput for various configurations. <s> BIB005
Gualtieri and Curran  grouped about 26 criteria into three high-level categories namely current offering, strategy, and market offering to evaluate the strengths and weaknesses of 15 top commercial and open-source DSPE vendors and their products. The authors used a combination of product briefings and demos, vendor surveys, and customer reference calls for their study. This evaluation by Forrester uncovered a market with seven leaders (IBM, Software AG, SAP, TIBCO Software, Oracle, DataTorrent, and SQLstream) and eight strong performers (Impetus Technologies, SAS, Striim, Informatica, WSO2, Cisco Systems, data Artisans, and EsperTech). The similarities, differences, implementation trade-offs, and the intended use-cases of five open-source distributed streaming frameworks were described by Zapletal in a theoretical overview  and an expert recommendation was made based on fault tolerance, state management and performance observations  . The study included Apache Storm, Apache Storm Trident, Apache Spark Streaming, Apache Samza, and Apache Fink. The author recommended that a careful evaluation of streaming application requirements should be conducted before choosing a framework. Wähner   . Braida reported a comparative analysis of five top-level Apache projects, Apache Storm, Samza, Apex, Spark Streaming, and Flink, which provide stream processing capabilities, with IBM Streams. The study concluded that suitability of the options for stream processing depends on the specific use cases. According to the study by Kamburugamuve et al.  , the requirements of a good stream processing framework revolves around two important attributes; the latency of the system and the high availability of the system. The study also identified some expectations of a distributed DSPEs, which include (i) high data mobility, a key for maintaining low latency, (ii) high availability and data processing guarantees, (iii) effective data partitioning and parallel handling, (iv) support for high level languages and tools for querying data streams, (v) ability to recover from failures in the case of non-deterministic processing, (vi) support for data persistence, and (vii) ability to handle stream imperfections such as delay, out of order, duplicates or loss. These attributes and the architecture were used as the criteria to compare several streaming frameworks which included Apache Aurora, Borealis (no longer active as a research project), Storm, S4, and Samza. The study by Gorawski et al.   compared several DSPEs (Storm, Samza, S4, MillWheel, Stratosphere, Streams) using the following criteria: message processing semantic, state handling and fault tolerance, scalability, support for distributed processing, and whether it is embeddable. Bockermann reported that the compared DSPEs differ from one another in terms of the guarantees they provide and the modelling capabilities they address. According to the study, Streams does not provide any fault-tolerance features by its own runtime, Storm, however, provides fault-tolerance similar to transactions and expects data to arrive in order. According to Bockermann, MillWheel builds upon low watermark timestamps and does treat all data streams as unordered. Liu et al. BIB001 studied the shortcomings of Hadoop and the potential of lambda architecture with regards to data stream processing. Liu Samza based on their language, stream abstraction, latency, throughput, message processing guarantees, and main components. According to Hesse and Lorenz, even though there are features that are common to all the compared DSPEs such as the use of Java Virtual Machine (JVM), a clear ranking cannot be created based on the presented results. Developing a use case and feature based ranking and recommendation system for DSPEs would be a good future research. The study by Singh et al. BIB002 compared several stream processing solutions namely, Storm, Spark Streaming, S4, Amazon Kinesis, and IBM Streams based on the type of framework, implementation language, supported languages for application development, abstraction or primitiveness, data sources, model of computation or transformation, persistence entity, execution reliability, fault tolerance, latency, and vendor. Pääkkönen BIB003 compared AsterixDB and Spark Streaming with Cassandra in terms of their latency and throughput in processing semi-structured social media data streams. Pääkkönen reported that AsterixDB scaled better as more nodes were added to the cluster. AsterixDB with Java also achieved significantly higher throughput and lower latency when data feeds were utilized. AsterixDB stream processing was, however, delayed by batching of streamed tweets. The study by Yadranjiaghdam et al. BIB004 compared several DSPEs based on their application domains and recommend the extension of the use of DSPEs to other evolving application areas that may be beneficial to the society. Chintapalli et al. BIB005 described the need for a benchmark for DSPEs and proposed one for selecting platforms for streaming data analytic needs. The benchmark focused on Apache Flink, Storm, and Spark Streaming within the context of a complete DSPS that utilized Kafka for ingestion and filtering of JSON events, and Redis for storing windowed count and timestamps of relevant events. Each of these frameworks has its advantages and disadvantages according to this study, for example, Storm and Flink have much lower latency than Spark Streaming at high throughput, while in terms of throughput rate, Spark Streaming can handle higher maximum throughput rate with its performance quite sensitive to the batch duration setting. Chintapalli et al. also added that tuning is required for Spark to achieve the desired latency to meet the service level agreements and recommended further work to look beyond just performance by incorporating other features such as security, and integration with other tools and libraries. Shukla What is missing in all of the above studies is a systematic taxonomy based on which the DSPEs were compared. Zhao et al.  identified the need to develop a set of standard criteria for characterizing DSPEs. Zhao et al. developed a taxonomy for both open source and commercial data stream processing frameworks. The taxonomy developed by Zhao et al., however, does not fully cover the technical depth and majority of the ingredients that are essential for characterizing data stream engines. In our proposed taxonomy in Section 3, we dived deeper into the key and technical features of the DSPE's such as programming and data models as well as the integration of these frameworks with other big data processing systems.