The pervasive availability of streaming data is driving Fast Data platforms for low-latency streaming applications. Such applications need to respond to dynamism in the input rates and task behavior using scale-in and -out on elastic Cloud resources. Platforms like Apache Storm do not provide robust means to respond to such dynamism and for rapid task migration across VMs. We propose several dataflow checkpoint and migration approaches that allow a running streaming dataflow to migrate, without any loss of in-flight messages or their internal tasks states, while reducing the time to recover and stabilize. We implement these strategies on Storm and evaluate them using micro and application dataflows for scaling in and out on 2 – 21 Cloud VMs. Our results show that we can migrate large dataflows and catchup with their processing 75 faster than Storm, which takes over 140 secs. We also find that our approaches stabilize the application up to 42 faster, and there is no failure and re-processing of messages.
@cite_19 uses a hybrid Cloud model for stream processing to address a lack of resources on the local stream computing cluster. They dynamically adjust the resources required to respond to dynamic rates, with the goal of minimizing the cost for using the Cloud while satisfying their QoS. The implementation on is able to assign or remove computational resources dynamically. Though the system transfers data stream processing to Cloud, they do not talk about message reliabilty and state handling during the migration. This may lead to loss of in-flight messages and internal state of stateful tasks during migration.
@cite_19
2132367116
Stream computing, also known as data stream processing, has emerged as a new processing paradigm that processes incoming data streams from tremendous numbers of sensors in a real-time fashion. Data stream applications must have low latency even when the incoming data rate fluctuates wildly. This is almost impossible with a local stream computing environment because its computational resources are finite. To address this kind of problem, we have devised a method and an architecture that transfers data stream processing to a Cloud environment as required in response to the changes of the data rate in the input data stream. Since a trade-off exists between application's latency and the economic costs when using the Cloud environment, we treat it as an optimization problem that minimizes the economic cost of using the Cloud. We implemented a prototype system using Amazon EC2 and an IBM System S stream computing system to evaluate the effectiveness of our approach. Our experimental results show that our approach reduces the costs by 80 while keeping the application's response latency low.