A Survey of Distributed Data Stream Processing Frameworks <s> 4) STATE MANAGEMENT <s> Stream processors are emerging in industry as an apparatus that drives analytical but also mission critical services handling the core of persistent application logic. Thus, apart from scalability and low-latency, a rising system need is first-class support for application state together with strong consistency guarantees, and adaptivity to cluster reconfigurations, software patches and partial failures. Although prior systems research has addressed some of these specific problems, the practical challenge lies on how such guarantees can be materialized in a transparent, non-intrusive manner that relieves the user from unnecessary constraints. Such needs served as the main design principles of state management in Apache Flink, an open source, scalable stream processor. ::: ::: We present Flink's core pipelined, in-flight mechanism which guarantees the creation of lightweight, consistent, distributed snapshots of application state, progressively, without impacting continuous execution. Consistent snapshots cover all needs for system reconfiguration, fault tolerance and version management through coarse grained rollback recovery. Application state is declared explicitly to the system, allowing efficient partitioning and transparent commits to persistent storage. We further present Flink's backend implementations and mechanisms for high availability, external state queries and output commit. Finally, we demonstrate how these mechanisms behave in practice with metrics and large-deployment insights exhibiting the low performance trade-offs of our approach and the general benefits of exploiting asynchrony in continuous, yet sustainable system deployments. <s> BIB001 </s> A Survey of Distributed Data Stream Processing Frameworks <s> 4) STATE MANAGEMENT <s> The concept of state and its applications vary widely across big data processing systems. This is evident in both the research literature and existing systems, such as Apache Flink, Apache Heron, Apache Samza, Apache Spark, and Apache Storm. Given the pivotal role that state management plays, particularly, for iterative batch and stream processing, in this survey, we present examples of state as an enabler, discuss the alternative approaches used to handle and implement state, capture the many facets of state management, and highlight new research directions. Our aim is to provide insight into disparate state management techniques, motivate others to pursue research in this area, and draw attention to open problems. <s> BIB002
A data streaming analytics program can be modelled as a directed tree where the analysis results are the roots, operators are the intermediate nodes, and data are the leaves. The data flows from the leaves through the operator nodes to the roots. Each operator node performs an operation that transforms inputs flowing through it into outputs. As described at the beginning of this section, operators can either be stateless or stateful. Stateless operators are purely functional, they produce output solely based on their input. On the other hand, stateful operators compute their output on a sequence of inputs and potentially use additional side information, maintained in an internal data structure called state. A state in a data stream application is a data structure that preserves the history of past operations and influences the processing logic for future computations BIB002 . In traditional DSPEs, state was separated from the application logic that computed and transformed the data, and the state information was stored in a centralized database management system to be shared among applications BIB001 . Handling state efficiently presents numerous technical challenges BIB002 . The state management facilities in various DSPEs naturally fall along a complexity continuum from naive in-memory-only choice to a persistent state that can be queried and replicated  .