Apache Spark is a popular Big Data framework, commonly used in both
industrial and academic settings. Although it is a Scala-based framework,
it also has APIs for Java, Python (PySpark) and R. Spark's Resilient
Distributed Dataset (RDD) abstraction enabled in-memory processing of
pipelines by co-locating tasks and data, which provided important
performance improvements compared to its predecessor Hadoop
MapReduce [1]}. Through the use of RDDs, it also became possible
to execute iterative workflows – something not easily doable in older
frameworks. Schedulers for Spark include its built-in standalone schedule,
Yet Another Resource Negotiator (YARN [2]}), and Mesos [3]}. As a
result of the sustained growth in data volumes, Apache Spark and other
Big Data engines are increasingly used for scientific applications,
including in neuroimaging, our primary field of
interest [4]}, [5]}, [6]}.
