Big Data frameworks were designed with dedicated commodity infrastructure
in mind, and, with the exception of Dask [1]}, do not support batch
HPC schedulers such as the Oracle Grid Engine (OGE),
Slurm and TORQUE, which
are commonly available to scientists. Therefore, to run Big Data
applications on HPC schedulers, it is also necessary to start an overlay
cluster that will schedule application tasks on resources provisioned
through batch schedulers. Our experiments quantify the effect of pilot jobs
when combined with such an overlay cluster.
