A Survey of Distributed Data Stream Processing Frameworks <s> III. DATA STREAM PROCESSING ENGINES (DSPEs) <s> Continuous queries are persistent queries that allow users to receive new results when they become available. While continuous query systems can transform a passive web into an active environment, they need to be able to support millions of queries due to the scale of the Internet. No existing systems have achieved this level of scalability. NiagaraCQ addresses this problem by grouping continuous queries based on the observation that many web queries share similar structures. Grouped queries can share the common computation, tend to fit in memory and can reduce the I/O cost significantly. Furthermore, grouping on selection predicates can eliminate a large number of unnecessary query invocations. Our grouping technique is distinguished from previous group optimization approaches in the following ways. First, we use an incremental group optimization strategy with dynamic re-grouping. New queries are added to existing query groups, without having to regroup already installed queries. Second, we use a query-split scheme that requires minimal changes to a general-purpose query engine. Third, NiagaraCQ groups both change-based and timer-based queries in a uniform way. To insure that NiagaraCQ is scalable, we have also employed other techniques including incremental evaluation of continuous queries, use of both pull and push models for detecting heterogeneous data source changes, and memory caching. This paper presents the design of NiagaraCQ system and gives some experimental results on the system's performance and scalability. <s> BIB001 </s> A Survey of Distributed Data Stream Processing Frameworks <s> III. DATA STREAM PROCESSING ENGINES (DSPEs) <s> AsterixDB is a new, full-function BDMS (Big Data Management System) with a feature set that distinguishes it from other platforms in today's open source Big Data ecosystem. Its features make it well-suited to applications like web data warehousing, social data storage and analysis, and other use cases related to Big Data. AsterixDB has a flexible NoSQL style data model; a query language that supports a wide range of queries; a scalable runtime; partitioned, LSM-based data storage and indexing (including B+-tree, R-tree, and text indexes); support for external as well as natively stored data; a rich set of built-in types; support for fuzzy, spatial, and temporal types and queries; a built-in notion of data feeds for ingestion of data; and transaction support akin to that of a NoSQL store. ::: ::: Development of AsterixDB began in 2009 and led to a mid-2013 initial open source release. This paper is the first complete description of the resulting open source AsterixDB system. Covered herein are the system's data model, its query language, and its software architecture. Also included are a summary of the current status of the project and a first glimpse into how AsterixDB performs when compared to alternative technologies, including a parallel relational DBMS, a popular NoSQL store, and a popular Hadoop-based SQL data analytics platform, for things that both technologies can do. Also included is a brief description of some initial trials that the system has undergone and the lessons learned (and plans laid) based on those early "customer" engagements. <s> BIB002 </s> A Survey of Distributed Data Stream Processing Frameworks <s> III. DATA STREAM PROCESSING ENGINES (DSPEs) <s> It is already true that Big Data has drawn huge attention from researchers in information sciences, policy and decision makers in governments and enterprises. As the speed of information growth exceeds Moore’s Law at the beginning of this new century, excessive data is making great troubles to human beings. However, there are so much potential and highly useful values hidden in the huge volume of data. A new scientific paradigm is born as dataintensive scientific discovery (DISD), also known as Big Data problems. A large number of fields and sectors, ranging from economic and business activities to public administration, from national security to scientific researches in many areas, involve with Big Data problems. On the one hand, Big Data is extremely valuable to produce productivity in businesses and evolutionary breakthroughs in scientific disciplines, which give us a lot of opportunities to make great progresses in many fields. There is no doubt that the future competitions in business productivity and technologies will surely converge into the Big Data explorations. On the other hand, Big Data also arises with many challenges, such as difficulties in data capture, data storage, data analysis and data visualization. This paper is aimed to demonstrate a close-up view about Big Data, including Big Data applications, Big Data opportunities and challenges, as well as the state-of-the-art techniques and technologies we currently adopt to deal with the Big Data problems. We also discuss several underlying methodologies to handle the data deluge, for example, granular computing, cloud computing, bio-inspired computing, and quantum computing. <s> BIB003 </s> A Survey of Distributed Data Stream Processing Frameworks <s> III. DATA STREAM PROCESSING ENGINES (DSPEs) <s> SAMOA (SCALABLE ADVANCED MASSIVE ONLINE ANALYSIS) is a platform for mining big data streams. It provides a collection of distributed streaming algorithms for the most common data mining and machine learning tasks such as classification, clustering, and regression, as well as programming abstractions to develop new algorithms. It features a pluggable architecture that allows it to run on several distributed stream processing engines such as Storm, S4, and Samza. samoa is written in Java, is open source, and is available at http://samoa-project.net under the Apache Software License version 2.0. <s> BIB004 </s> A Survey of Distributed Data Stream Processing Frameworks <s> III. DATA STREAM PROCESSING ENGINES (DSPEs) <s> As a new trend for data-intensive computing, real-time stream computing is gaining significant attention in the big data era. In theory, stream computing is an effective way to support big data by providing extremely low-latency processing tools and massively parallel processing architectures in real-time data analysis. However, in most existing stream computing environments, how to efficiently deal with big data stream computing, and how to build efficient big data stream computing systems are posing great challenges to big data computing research. First, the data stream graphs and the system architecture for big data stream computing, and some related key technologies, such as system structure, data transmission, application interfaces, and high availability, are systemically researched. Then, we give a classification of the latest research and depict the development status of some popular big data stream computing systems, including Twitter Storm, Yahoo! S4, Microsoft TimeStream, and Microsoft Naiad. Finally, the potential challenges and future directions of big data stream computing are discussed. 11. <s> BIB005 </s> A Survey of Distributed Data Stream Processing Frameworks <s> III. DATA STREAM PROCESSING ENGINES (DSPEs) <s> Along with the spread of the World Wide Web, social networks and the Internet of Things in the past decades, the need for systems and algorithms that can process massive graphs has been continuously increasing. There has been considerable amount of research done in distributed graph processing since the emergence of such large-scale graphs. Another steadily growing field in the past years has been stream processing. This rise of interest can be attributed to the need to process large amounts of continuously streaming data with scalability, fault tolerance and very low latency. Graph streaming, the unification of these two fields is a rather new idea, with some research already being done on it. Processing graphs that are unbounded, and so large that they cannot be stored in memory or even on the disk, is only possible with a distributed graph streaming model. Our goal is to provide a graph streaming model and API that can handle common transformations and provide statistics on streamed graphs. This graph streaming API is created on top of Flink streaming and provides similar interfaces to Gelly, which is the graph library on the batch processing part of Flink. <s> BIB006 </s> A Survey of Distributed Data Stream Processing Frameworks <s> III. DATA STREAM PROCESSING ENGINES (DSPEs) <s> This paper presents GraphJet, a new graph-based system for generating content recommendations at Twitter. As motivation, we trace the evolution of our formulation and approach to the graph recommendation problem, embodied in successive generations of systems. Two trends can be identified: supplementing batch with real-time processing and a broadening of the scope of recommendations from users to content. Both of these trends come together in Graph-Jet, an in-memory graph processing engine that maintains a real-time bipartite interaction graph between users and tweets. The storage engine implements a simple API, but one that is sufficiently expressive to support a range of recommendation algorithms based on random walks that we have refined over the years. Similar to Cassovary, a previous graph recommendation engine developed at Twitter, GraphJet assumes that the entire graph can be held in memory on a single server. The system organizes the interaction graph into temporally-partitioned index segments that hold adjacency lists. GraphJet is able to support rapid ingestion of edges while concurrently serving lookup queries through a combination of compact edge encoding and a dynamic memory allocation scheme that exploits power-law characteristics of the graph. Each GraphJet server ingests up to one million graph edges per second, and in steady state, computes up to 500 recommendations per second, which translates into several million edge read operations per second. <s> BIB007 </s> A Survey of Distributed Data Stream Processing Frameworks <s> III. DATA STREAM PROCESSING ENGINES (DSPEs) <s> Streaming data processing has been gaining attention due to its application into a wide range of scenarios. To serve the booming demands of streaming data processing, many computation engines have been developed. However, there is still a lack of real-world benchmarks that would be helpful when choosing the most appropriate platform for serving real-time streaming needs. In order to address this problem, we developed a streaming benchmark for three representative computation engines: Flink, Storm and Spark Streaming. Instead of testing speed-of-light event processing, we construct a full data pipeline using Kafka and Redis in order to more closely mimic the real-world production scenarios. Based on our experiments, we provide a performance comparison of the three data engines in terms of 99th percentile latency and throughput for various configurations. <s> BIB008 </s> A Survey of Distributed Data Stream Processing Frameworks <s> III. DATA STREAM PROCESSING ENGINES (DSPEs) <s> With almost everything now online, organizations look at the Big Data collected to gain insights for improving their services. In the analytics process, derivation of such insights requires experimenting-with and integrating different analytics techniques, while handling the Big Data high arrival velocity and large volumes. Existing solutions cover bits-and-pieces of the analytics process, leaving it to organizations to assemble their own ecosystem or buy an off-the-shelf ecosystem that can have unnecessary components to them. We build on this point by dividing the Big Data Analytics problem into six main pillars. We characterize and show examples of solutions designed for each of these pillars. We then integrate these six pillars into a taxonomy to provide an overview of the possible state-of-the-art analytics ecosystems. In the process, we highlight a number of ecosystems to meet organizations different needs. Finally, we identify possible areas of research for building future Big Data Analytics Ecosystems. <s> BIB009 </s> A Survey of Distributed Data Stream Processing Frameworks <s> III. DATA STREAM PROCESSING ENGINES (DSPEs) <s> Velocity is one of the 4 Vs commonly used to characterize Big Data [5]. In this regard, Forrester remarked the following in Q3 2014 [8]: "The high velocity, white-water flow of data from innumerable real-time data sources such as market data, Internet of Things, mobile, sensors, click-stream, and even transactions remain largely unnavigated by most firms. The opportunity to leverage streaming analytics has never been greater." Example use cases of streaming analytics include, but not limited to: (a) visualization of business metrics in real-time (b) facilitating highly personalized experiences (c) expediting response during emergencies. Streaming analytics is extensively used in a wide variety of domains such as healthcare, e-commerce, financial services, telecommunications, energy and utilities, manufacturing, government and transportation. ::: ::: In this tutorial, we shall present an in-depth overview of streaming analytics -- applications, algorithms and platforms -- landscape. We shall walk through how the field has evolved over the last decade and then discuss the current challenges -- the impact of the other three Vs, viz., Volume, Variety and Veracity, on Big Data streaming analytics. The tutorial is intended for both researchers and practitioners in the industry. We shall also present state-of-the-affairs of streaming analytics at Twitter. <s> BIB010
DSPEs are complex stream processing engines which are composed of multiple components as described below in Section A. These components differ in the underlying programming models, configurations, and operations. Research in DSPEs has diverged  into (i) query-based systems such as NiagaraCQ BIB001 , TelegraphCQ  , and AsterixDB BIB002 , (ii) online distributed machine learning systems such as Scalable Advanced Massive Online Analysis (SAMOA) BIB004 , (iii) streaming graph analytics systems such as GraphJet BIB007 , and (iv) general purpose streaming data processing frameworks BIB008 such as Flink and Spark Streaming, having low-latency BIB009 and a distributed parallel processing architecture BIB005 , BIB006 . Here, we explain the key features of DSPEs in Section B and propose a taxonomy for comparing the state of the art DSPEs in Section C. For more information on use case scenarios and advantages of DSPEs such as Storm, S4, SQLstream, Splunk, Kafka, and SAP HANA refer to Khalifa et al. BIB009 , Chen and Zhang BIB003 . Kejariwal et al. BIB010 , and Sakr  provide detailed descriptions of design choices and challenges of selected DSPEs which include S4, Storm, Trident, M3, Samza, Akka, Flink, Spark Streaming, MillWheel, Pulsar, Heron, IBM Streams, Microsoft Trill, and Stream Insight.