Understanding Software Platforms for In Memory Scientific Data Analysis: A Case Study of the Spark System
Over the last five years, Apache Spark has become a major software platform for in memory data analysis. Acknowledging its widespread use, we present a comprehensive study of system characteristics of Spark with a focus on scientific data analytics performing large scale matrix operations. We compare its performance to SciDB, a disk based platform for array data analysis. A benchmark, ArrayBench, is developed to evaluate the performance of matrix processing for scientific data analytics. ArrayBench is applied to data from a real biological workflow whose data inputs are in matrix form. Herein, we report the findings, which shed light on the improvement of Spark and SciDB and future development of large scale scientific data analytics.
2016 IEEE 22nd International Conference on Parallel and Distributed Systems (ICPADS)