Distributed training. Distributed training is a heavily studied topic [1]}, [2]}.
Our work focuses on synchronous and distributed training techniques [3]}, [4]}, [5]}.
Some examples of synchronous, distributed training approaches include data parallel training, parallel SGD [6]}, [7]}, and local SGD [8]}, [9]}.
Our methodology holds similarities to model parallel training techniques, which have been heavily explored [10]}, [11]}, [12]}, [13]}, [14]}, [15]}, [16]}.
More closely, our approach is inspired by independent subnetwork training [17]}, explored for multi-layer perceptrons.
