Our scheme provides robustness under powerful attacks and sophisticated distortion methods at the expense of increased computation and communication time. Note that each worker has to perform \(l\)  forward/backward propagation computations and transmit \(l\)  gradients per iteration. In related baseline [1]}, [2]} and redundancy-based methods [3]}, [4]} each worker is responsible for a single such computation. Experimentally, we have observed that Aspis needs up to \(5\times \)  overall training time compared to other schemes to complete the same number of training epochs. We emphasize that the training time incurred by each scheme depends on a wide range of parameters including the utilized defense, the batch size and the number of iterations and can vary significantly. We believe that implementation-related improvements such as utilizing GPUs can alleviate some of the overhead. Communication-related algorithmic improvements are also worth exploring. Finally, our implementation natively supports resuming from a checkpoint (trained model) and hence, when new data becomes available we can only use that data to perform more epochs of training.
