A Survey of Distributed Data Stream Processing Frameworks <s> B. GENERAL ARCHITECTURE OF A DSPS <s> Realtime data processing powers many use cases at Facebook, including realtime reporting of the aggregated, anonymized voice of Facebook users, analytics for mobile applications, and insights for Facebook page administrators. Many companies have developed their own systems; we have a realtime data processing ecosystem at Facebook that handles hundreds of Gigabytes per second across hundreds of data pipelines. Many decisions must be made while designing a realtime stream processing system. In this paper, we identify five important design decisions that affect their ease of use, performance, fault tolerance, scalability, and correctness. We compare the alternative choices for each decision and contrast what we built at Facebook to other published systems. Our main decision was targeting seconds of latency, not milliseconds. Seconds is fast enough for all of the use cases we support and it allows us to use a persistent message bus for data transport. This data transport mechanism then paved the way for fault tolerance, scalability, and multiple options for correctness in our stream processing systems Puma, Swift, and Stylus. We then illustrate how our decisions and systems satisfy our requirements for multiple use cases at Facebook. Finally, we reflect on the lessons we learned as we built and operated these systems. <s> BIB001 </s> A Survey of Distributed Data Stream Processing Frameworks <s> B. GENERAL ARCHITECTURE OF A DSPS <s> Under several emerging application scenarios, such as in smart cities, operational monitoring of large infrastructure, wearable assistance, and Internet of Things, continuous data streams must be processed under very short delays. Several solutions, including multiple software engines, have been developed for processing unbounded data streams in a scalable and efficient manner. More recently, architecture has been proposed to use edge computing for data stream processing. This paper surveys state of the art on stream processing engines and mechanisms for exploiting resource elasticity features of cloud computing in stream processing. Resource elasticity allows for an application or service to scale out/in according to fluctuating demands. Although such features have been extensively investigated for enterprise applications, stream processing poses challenges on achieving elastic systems that can make efficient resource management decisions based on current load. Elasticity becomes even more challenging in highly distributed environments comprising edge and cloud computing resources. This work examines some of these challenges and discusses solutions proposed in the literature to address them. <s> BIB002
A DSPS can be deployed on a single machine consisting of multiple components, where each component directly calls the next component(s) in the processing pipeline. For greater throughput, efficiency, and fault tolerance, large stream processing systems are set up on multiple distributed machines with the help of additional software stack. A simple Extract, Transform, and Load (ETL) transactional DSPS architecture for processing IoT streams was proposed by Meehan et al.  . The architecture comprises three main components (i) stream collection for data ingestion, (ii) streaming ETL engine for real-time query processing, and (iii) an Online Analytical Processing (OLAP) backend for handling long-running queries. A data migrator component is used to connect the ETL and the OLAP components to facilitate data transformations. The DSPS at Facebook BIB001 is relatively complex and powers many use cases such as the real-time reporting of aggregated voice of Facebook users, analytics for mobile applications, and insights for Facebook page administrators. It is made up of data sources such as mobile and web products; Scribe as a data distribution tool; stream processing systems such as Puma, Stylus, and Swift; and data stores such as Laser, Scuba, and Hive. The Facebook DSPS flow is a complex Directed Acyclic Graph (DAG) comprising of data streams from the mobile and web products that are fed into Scribe. Real-time data processing systems read from and write to Scribe. The data stores also use Scribe for data ingestion and serve different types of queries. According to Psaltis and de Assuncao et al. BIB002 ,  , the architecture of a DSPS is generally multi-tiered and is composed of many loosely coupled components that include data sources, data collection systems, data storage systems, messaging systems, and stream processing and delivery systems.