Incrementation
[1]
\(x\)a sleep delay in seconds
\(n\)a number of iterations
\(C\)a set of image chunks
\(chunk\)  in \(C\) 
read \(chunk\) 
\(i \in \llbracket 1, n \rrbracket \) 
\(chunk\leftarrow chunk+1\) 
sleep \(x\) 
write \(chunk\) 


To determine the added value of pilot scheduling over batch scheduling of Spark
applications, we required a Spark application operating on a large dataset with
an important processing time to emulate what would be the average requirements
of a scientific Spark application. For this, we created a synthetic application
that would process the Big Brain [1]}, a 76 GB 3D histological
image of a human brain. The algorithm is a chain of map transformations that
at each transformation increment the voxels of the image by 1 (see Algorithm REF ).
We chose such a synthetic algorithm
as the focus of our experiments is pilot scheduling and not the application in
itself. Furthermore, this algorithm enabled us to have control over the task duration
which was representative of scientific applications. Additionally, it was important
that the overall application duration did not vary between the different levels of
parallelism within our experiments. Being able to adjust the task duration based on
level of parallelism allowed us to achieve this.
