
Distributed PyTorch [1]} on CPUs. Due to the lack of hardware accelerators such as GPUs and TPUs [2]} in
IBM Cloud Functions, we run PyTorch v1.8.1 with Intel MKL enabled in a cluster of VM servers utilizing all the available cores. We use the all-reduce operator
of Gloo [3]}—rule of thumb for CPU training—, a MPI-like library for cross-machine communication.
Mini-batches are downloaded from IBM COS.

PyWren-IBM [4]}. We use PyWren-IBM as non-specialized serverless ML representative. PyWren-IBM has been optimized
to run on IBM Cloud Functions. Since it is a MapReduce framework, we leverage the map phase to process mini-batches in parallel
and reduce tasks to aggregate the local updates. All communication is done through IBM COS, including the sharing of updates, to keep
its pure serverless, general-purpose architecture.

