The benchmark methods are DPSGD [1]}, as well as the stochastic versions of ByRDiE [2]} and BRIDGE [3]} (denoted by ByRDiE-S and BRIDGE-S, respectively). In DPSGD, the mixing matrix is constructed following the equal neighbor weights rule [4]}. In ByRDiE-S, the coordinates of the model are updated sequentially. We set the number of inner-loop iterations to update every coordinate to be 1, as suggested by [2]}. For fair comparison, in ByRDiE-S one iteration refers to that all the coordinates have been updated once. Step sizes of the benchmark methods are hand-tuned to the best. In all the compared methods, the batch-size is set as 32.
