A Survey of Distributed Data Stream Processing Frameworks <s> 3) STATE MANAGEMENT COMPARISON <s> Stream processors are emerging in industry as an apparatus that drives analytical but also mission critical services handling the core of persistent application logic. Thus, apart from scalability and low-latency, a rising system need is first-class support for application state together with strong consistency guarantees, and adaptivity to cluster reconfigurations, software patches and partial failures. Although prior systems research has addressed some of these specific problems, the practical challenge lies on how such guarantees can be materialized in a transparent, non-intrusive manner that relieves the user from unnecessary constraints. Such needs served as the main design principles of state management in Apache Flink, an open source, scalable stream processor. ::: ::: We present Flink's core pipelined, in-flight mechanism which guarantees the creation of lightweight, consistent, distributed snapshots of application state, progressively, without impacting continuous execution. Consistent snapshots cover all needs for system reconfiguration, fault tolerance and version management through coarse grained rollback recovery. Application state is declared explicitly to the system, allowing efficient partitioning and transparent commits to persistent storage. We further present Flink's backend implementations and mechanisms for high availability, external state queries and output commit. Finally, we demonstrate how these mechanisms behave in practice with metrics and large-deployment insights exhibiting the low performance trade-offs of our approach and the general benefits of exploiting asynchrony in continuous, yet sustainable system deployments. <s> BIB001 </s> A Survey of Distributed Data Stream Processing Frameworks <s> 3) STATE MANAGEMENT COMPARISON <s> With the ubiquity of real-time data, organizations need streaming systems that are scalable, easy to use, and easy to integrate into business applications. Structured Streaming is a new high-level streaming API in Apache Spark based on our experience with Spark Streaming. Structured Streaming differs from other recent streaming APIs, such as Google Dataflow, in two main ways. First, it is a purely declarative API based on automatically incrementalizing a static relational query (expressed using SQL or DataFrames), in contrast to APIs that ask the user to build a DAG of physical operators. Second, Structured Streaming aims to support end-to-end real-time applications that integrate streaming with batch and interactive analysis. We found that this integration was often a key challenge in practice. Structured Streaming achieves high performance via Spark SQL's code generation engine and can outperform Apache Flink by up to 2x and Apache Kafka Streams by 90x. It also offers rich operational features such as rollbacks, code updates, and mixed streaming/batch execution. We describe the system's design and use cases from several hundred production deployments on Databricks, the largest of which process over 1 PB of data per month. <s> BIB002
Storm provides abstractions for bolts to manage (save and retrieve) the state of its operations. Currently, Storm has a default in-memory based state implementation and also a Redis backed implementation that provides state persistence. The only supported state implementation in Storm, at the time of writing this paper, is the KeyValueState, which provides key-value mapping. At a high level, structured streaming tracks state in a manner similar in both its micro-batch and continuous modes. The state of an application is tracked using two external storage systems, a write-ahead log that supports durable, atomic writes at low latency, and a state store that can store larger amounts of data durably and allows parallel access (e.g., S3 or HDFS). Structured streaming uses these systems together to recover from failure BIB002 . Flink's ecosystem of modules and services built on its core offers different flavours of external state access and isolation. Each stream operation in Flink can declare its own state and update it continuously in order to maintain a summary of the data seen so far BIB001 . State information during Flink's operations is maintained in an embedded key/value store [78] . Kafka Streams provides applications with powerful, elastic, highly scalable, and fault-tolerant stateful processing capabilities. Kafka Streams provides state stores, which can be used by stream processing applications to store and query data, which is an important capability for implementing stateful operations. These state stores can either be a RocksDB database, an in-memory hashmap, or another convenient data structure [79] . IBM Streams operators can be configured to keep state information between subsequent tuples. This is achieved using tuple history, operator windows, operator custom logic, or primitive operators. The tuple history approach means that expressions in parameters or output assignments of operator invocations can refer directly to tuples received in the past. This is because history access maintains a list of tuples that can be subscripted by using the input port name. The operator windows approach uses a window which is a logical container for tuples recently received by an input port of an operator. The operator custom logic approach is made up of local operators' states that persist across operator firings and statements. An operator fires and statements get executed when an input port receives a tuple or a punctuation. Finally, the primitive operators which can either be generic or non-generic are user-written operators with some state mechanisms built into them. These are often controlled by the developers of the operators  .