A Survey of Distributed Data Stream Processing Frameworks <s> II. MOTIVATION AND AN OVERVIEW OF DSPS <s> Unbounded, unordered, global-scale datasets are increasingly common in day-to-day business (e.g. Web logs, mobile usage statistics, and sensor networks). At the same time, consumers of these datasets have evolved sophisticated requirements, such as event-time ordering and windowing by features of the data themselves, in addition to an insatiable hunger for faster answers. Meanwhile, practicality dictates that one can never fully optimize along all dimensions of correctness, latency, and cost for these types of input. As a result, data processing practitioners are left with the quandary of how to reconcile the tensions between these seemingly competing propositions, often resulting in disparate implementations and systems. ::: ::: We propose that a fundamental shift of approach is necessary to deal with these evolved requirements in modern data processing. We as a field must stop trying to groom unbounded datasets into finite pools of information that eventually become complete, and instead live and breathe under the assumption that we will never know if or when we have seen all of our data, only that new data will arrive, old data may be retracted, and the only way to make this problem tractable is via principled abstractions that allow the practitioner the choice of appropriate tradeoffs along the axes of interest: correctness, latency, and cost. ::: ::: In this paper, we present one such approach, the Dataflow Model, along with a detailed examination of the semantics it enables, an overview of the core principles that guided its design, and a validation of the model itself via the real-world experiences that led to its development. <s> BIB001 </s> A Survey of Distributed Data Stream Processing Frameworks <s> II. MOTIVATION AND AN OVERVIEW OF DSPS <s> Services like social networks, web analytics, and intelligent e-commerce often need to manage data at a scale too big for a traditional database. As scale and demand increase, so does Complexity. Fortunately, scalability and simplicity are not mutually exclusiverather than using some trendy technology, a different approach is needed. Big data systems use many machines working in parallel to store and process data, which introduces fundamental challenges unfamiliar to most developers. Big Data shows how to build these systems using an architecture that takes advantage of clustered hardware along with new tools designed specifically to capture and analyze web-scale data. It describes a scalable, easy to understand approach to big data systems that can be built and run by a small team. Following a realistic example, this book guides readers through the theory of big data systems, how to use them in practice, and how to deploy and operate them once they're built. Purchase of the print book comes with an offer of a free PDF, ePub, and Kindle eBook from Manning. Also available is all code from the book. <s> BIB002 </s> A Survey of Distributed Data Stream Processing Frameworks <s> II. MOTIVATION AND AN OVERVIEW OF DSPS <s> Sensor and smart phone technologies present opportunities for data explosion, streaming and collecting from heterogeneous devices every second. Analyzing these large datasets can unlock multiple behaviors previously unknown, and help optimize approaches to city wide applications or societal use cases. However, collecting and handling of these massive datasets presents challenges in how to perform optimized online data analysis ‘on-the-fly’, as current approaches are often limited by capability, expense and resources. This presents a need for developing new methods for data management particularly using public clouds to minimize cost, network resources and on-demand availability. This paper presents an implementation of the lambda architecture design pattern to construct a data-handling backend on Amazon EC2, providing high throughput, dense and intense data demand delivered as services, minimizing the cost of the network maintenance. This paper combines ideas from database management, cost models, query management and cloud computing to present a general architecture that could be applied in any given scenario where affordable online data processing of Big Datasets is needed. The results are presented with a case study of processing router sensor data on the current ESnet network data as a working example of the approach. The results showcase a reduction in cost and argue benefits for performing online analysis and anomaly detection for sensor data. <s> BIB003
Modern Data Stream Processing Systems (DSPS) try to combine batch and stream processing capabilities into a single or multiple parallel data processing pipelines BIB001 . The motivation behind developing such infrastructures is that massive processing can be done on historical data to train machine learning models using batch data analytic pipelines, which can then be deployed on real-time incoming data streams for scoring using a separate data analytic pipeline. Marz and Warren propose the lambda architecture BIB002 , which includes batch and stream processing within a single framework. The lambda architecture was designed for applications that have delays in data collection and online processing due to the use of interactive interfaces or dashboards and data validation steps BIB003 . It is, however, limited in application because of the requirement of having to build, maintain, and integrate two separate systems. It is, therefore, desirable to reduce this complexity by providing a single interface at a higher level of abstraction to manage the underlying batch and streaming systems BIB001 . In this section, we first motivate the reader by describing data analytics scenarios that will necessitate the use of streaming analytics approach. Next, we give an overview of the general architecture of a DSPS, and finally, describe the main data processing components of a DSPS.