A Survey of Distributed Data Stream Processing Frameworks <s> 2) DATA STREAM PROCESSING LAYER <s> Traditional database management systems are best equipped to run one-time queries over finite stored data sets. However, many modern applications such as network monitoring, financial analysis, manufacturing, and sensor networks require long-running, or continuous, queries over continuous unbounded streams of data. In the STREAM project at Stanford, we are investigating data management and query processing for this class of applications. As part of the project we are building a general-purpose prototype Data Stream Management System (DSMS), also called STREAM, that supports a large class of declarative continuous queries over continuous streams and traditional stored data sets. The STREAM prototype targets environments where streams may be rapid, stream characteristics and query loads may vary over time, and system resources may be limited. <s> BIB001 </s> A Survey of Distributed Data Stream Processing Frameworks <s> 2) DATA STREAM PROCESSING LAYER <s> Applications that require real-time processing of high-volume data steams are pushing the limits of traditional data processing infrastructures. These stream-based applications include market feed processing and electronic trading on Wall Street, network and infrastructure monitoring, fraud detection, and command and control in military environments. Furthermore, as the "sea change" caused by cheap micro-sensor technology takes hold, we expect to see everything of material significance on the planet get "sensor-tagged" and report its state or location in real time. This sensorization of the real world will lead to a "green field" of novel monitoring and control applications with high-volume and low-latency processing requirements.Recently, several technologies have emerged---including off-the-shelf stream processing engines---specifically to address the challenges of processing high-volume, real-time data without requiring the use of custom code. At the same time, some existing software technologies, such as main memory DBMSs and rule engines, are also being "repurposed" by marketing departments to address these applications.In this paper, we outline eight requirements that a system software should meet to excel at a variety of real-time stream processing applications. Our goal is to provide high-level guidance to information technologists so that they will know what to look for when evaluation alternative stream processing solutions. As such, this paper serves a purpose comparable to the requirements papers in relational DBMSs and on-line analytical processing. We also briefly review alternative system software technologies in the context of our requirements.The paper attempts to be vendor neutral, so no specific commercial products are mentioned. <s> BIB002 </s> A Survey of Distributed Data Stream Processing Frameworks <s> 2) DATA STREAM PROCESSING LAYER <s> Abstract In the last few years, we have seen much interest in data processing technologies. Although initial interests focused on batch processing technologies like MapReduce, people have realized the need for more responsive technologies such as stream processing and complex event processing. Complex event processing has been historically used within a single node or a cluster of tightly interconnected nodes. However, to be effective with Big Data use-cases, CEP technologies need to be able to scale up to handle large use-cases. This paper presents several approaches to scale complex event processing by distributing it across several nodes. Wihidum discusses how to balance the workload among nodes efficiently, how complex event processing queries can be broken up into simple sub queries, and how queries can be efficiently deployed in the cluster. The paper focuses on three techniques used for scaling queries: pipelining, partitioning and distributed operators. Then it discusses in detail the distribution of few CEP operators: filters, joins, pattern matching, and partitions. Empirical results show that the techniques followed in Wihidum have improved the performance of the CEP solution. <s> BIB003 </s> A Survey of Distributed Data Stream Processing Frameworks <s> 2) DATA STREAM PROCESSING LAYER <s> As the data-driven economy evolves, enterprises have come to realize a competitive advantage in being able to act on high volume, high velocity streams of data. Technologies such as distributed message queues and streaming processing platforms that can scale to thousands of data stream partitions on commodity hardware are a response. However, the programming API provided by these systems is often low-level, requiring substantial custom code that adds to the programmer learning curve and maintenance overhead. Additionally, these systems often lack SQL querying capabilities that have proven popular on Big Data systems like Hive, Impala or Presto. We define a minimal set of extensions to standard SQL for data stream querying and manipulation. These extensions are prototyped in SamzaSQL, a new tool for streaming SQL that compiles streaming SQL into physical plans that are executed on Samza, an open-source distributed stream processing framework. We compare the performance of streaming SQL queries against native Samza applications and discuss usability improvements. SamzaSQL is a part of the open source Apache Samza project and will be available for general use. <s> BIB004 </s> A Survey of Distributed Data Stream Processing Frameworks <s> 2) DATA STREAM PROCESSING LAYER <s> Realtime data processing powers many use cases at Facebook, including realtime reporting of the aggregated, anonymized voice of Facebook users, analytics for mobile applications, and insights for Facebook page administrators. Many companies have developed their own systems; we have a realtime data processing ecosystem at Facebook that handles hundreds of Gigabytes per second across hundreds of data pipelines. Many decisions must be made while designing a realtime stream processing system. In this paper, we identify five important design decisions that affect their ease of use, performance, fault tolerance, scalability, and correctness. We compare the alternative choices for each decision and contrast what we built at Facebook to other published systems. Our main decision was targeting seconds of latency, not milliseconds. Seconds is fast enough for all of the use cases we support and it allows us to use a persistent message bus for data transport. This data transport mechanism then paved the way for fault tolerance, scalability, and multiple options for correctness in our stream processing systems Puma, Swift, and Stylus. We then illustrate how our decisions and systems satisfy our requirements for multiple use cases at Facebook. Finally, we reflect on the lessons we learned as we built and operated these systems. <s> BIB005 </s> A Survey of Distributed Data Stream Processing Frameworks <s> 2) DATA STREAM PROCESSING LAYER <s> Under several emerging application scenarios, such as in smart cities, operational monitoring of large infrastructure, wearable assistance, and Internet of Things, continuous data streams must be processed under very short delays. Several solutions, including multiple software engines, have been developed for processing unbounded data streams in a scalable and efficient manner. More recently, architecture has been proposed to use edge computing for data stream processing. This paper surveys state of the art on stream processing engines and mechanisms for exploiting resource elasticity features of cloud computing in stream processing. Resource elasticity allows for an application or service to scale out/in according to fluctuating demands. Although such features have been extensively investigated for enterprise applications, stream processing poses challenges on achieving elastic systems that can make efficient resource management decisions based on current load. Elasticity becomes even more challenging in highly distributed environments comprising edge and cloud computing resources. This work examines some of these challenges and discusses solutions proposed in the literature to address them. <s> BIB006
The data stream processing layer is where the streaming data processing applications or jobs are executed. It can host loosely coupled disjoint applications or a DSPE or both. DSPEs generally offer a set of streaming data processing operators which can be configured and threaded together to build a stream data processing pipeline to analyze incoming data streams  . Many DSPEs exist today some of which are marketed by large software vendors while others were created as a part of open-source research projects. The main categories of DSPEs include Data Stream Management Engines (DSME), Complex Event Processing Engines (CEPE) and general-purpose DSPEs (GDSPE) BIB006 . Traditional Database Management Systems (DBMSs) are best equipped to run periodical queries over finite stored data sets. However, many modern data analytics applications such as financial data analytics, and network and sensor data monitoring, need to support continuous queries over dynamic unbounded data streams BIB001 . DSMEs provide extensions to DBMSs by enabling long-running queries over dynamic data, providing support for SQL-like operations and declarative interfaces. The basic requirements that a DSME should meet were outlined by Stonebraker et al. BIB002 . These requirements include the ability to (i) process continuous data on-the-fly without any requirement to store them, (ii) support high-level languages such as SQL, (iii) handle imperfections such as delayed, missing and out-of-order data, (iv) guarantee predictable and repeatable outcomes, (v) efficiently store, access, modify, and combine (with live streaming data) state information, (vi) ensure that the integrity of the data is maintained at all times and relevant applications are up and available despite failures, (vii) automatically and transparently distribute the data processing load across multiple processors and machines, and (viii) respond to high-volume data processing applications in real-time using a highly optimized execution path. Cutting edge DSMEs include Samza-SQL BIB004 , KSQL, and SQLstream Blaze  ,  . CEPE is the core logic of Complex Event Processing (CEP) which denotes application of business rules to streaming event data (such as log or sensor data streams) associated with a timestamp  . This enables useful real-time actions to be taken over data streams  . A CEPE supports exploring relationships among events using specific rules as well as declarative interfaces BIB006 . Cutting edge CEPEs include Esper, StreamBase, and WSO2 CEP  ,  , BIB003 . Generic-purpose DSPEs (GDSPE) are being developed to perform distributed stream processing with the aim of achieving scalable and fault-tolerant execution on cluster environments. Unlike DBMSs or CEPEs, GDSPEs do not provide declarative interfaces, thereby requiring developers to program applications rather than writing queries. Most modern GDSPEs support streaming SQL and CEPE either natively or through an add-on library. GDSPEs are commonly used to process high volume and velocity of streaming data. A good DSPE should have the ability to process data in real-time and for any given time interval, as well as to peek into the data within a sliding window of time  . Cutting edge GDSPEs include Kafka Streams, Spark Streaming, Storm, Flink, and Samza as listed in TABLE 1 BIB006 ,  . There are also many commercial GDSPEs including IBM Streams, Amazon Kinesis, Azure Streams, Google Cloud Dataflow, Facebook's Stylus, Swift, and Puma BIB006 , BIB005 . The detailed discussion of DSPEs will follow later in the paper.