Benchmarking Streaming Computation Engines: Storm, Flink and Spark Streaming
Streaming data processing has been gaining attention due to its application into a wide range of scenarios. To serve the booming demands of streaming data processing, many computation engines have been developed. However, there is still a lack of real world benchmarks that would be helpful when choosing the most appropriate platform for serving real time streaming needs. In order to address this problem, we developed a streaming benchmark for three representative computation engines: Flink, Storm and Spark Streaming. Instead of testing speed of light event processing, we construct a full data pipeline using Kafka and Redis in order to more closely mimic the real world production scenarios. Based on our experiments, we provide a performance comparison of the three data engines in terms of 99th percentile latency and throughput for various configurations.
2016 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)