Spark GPU: An accelerated in memory data processing engine on clusters
Apache Spark is an in memory data processing system that supports both SQL queries and advanced analytics over large data sets. In this paper, we present our design and implementation of Spark GPU that enables Spark to utilize GPU's massively parallel processing ability to achieve both high performance and high throughput. Spark GPU transforms a general purpose data processing system into a GPU supported system by addressing several real world technical challenges including minimizing internal and external data transfers, preparing a suitable data format and a batching mode for efficient GPU execution, and determining the suitability of workloads for GPU with a task scheduling capability between CPU and GPU. We have comprehensively evaluated Spark GPU with a set of representative analytical workloads to show its effectiveness. Our results show that Spark GPU improves the performance of machine learning workloads by up to 16.13x and the performance of SQL queries by up to 4.83x.
2016 IEEE International Conference on Big Data (Big Data)