DFS Perf A Scalable and Unified Benchmarking Framework for Distributed File Systems
A distributed file system (DFS) is a key component of virtually any cluster computing system. The performance of such system depends heavily on the underlying DFS design and deployment. As a result, it is critical to characterize the performance and design trade offs of DFSes with respect to cluster configurations and real world workloads. To this end, we present DFS Perf, a scalable, extensible, and low overhead benchmarking framework to evaluate the properties and the performance of various DFS implementations. DFS Perf uses a highly parallel architecture to cover a large variety of workloads at different scales, and provides an extensible interface to incorporate user defined workloads and integrate with various DFSes. As a proof of concept, our current DFS Perf implementation includes several built in benchmarks and workloads, including machine learning and SQL applications. We present performance comparisons of four stateof the art DFS designs, namely Alluxio, CephFS, GlusterFS, and HDFS, on a cluster with 40 nodes (960 cores) We demonstrate that DFS Perf can provide guidance on existing DFS designs and implementations, while adding 5.7% overhead.