Hyperparameters. The learning rates of the local SGD solver for all algorithms are selected from the candidate set \(\lbrace 0.01, 0.1, 0.2, 0.5 \rbrace \)  for best performance. Recall that both FedADMM and FedProx make use of a quadratic penalty term and require the setting of hyperparameter \(\rho \) . It has been observed in [1]}, [2]}, and is further supported in our experiments, that careful tuning of \(\rho \)  is required to achieve the best performance for FedProx (see discussion on Proximal Parameter \(\rho \)  in Section V-B). For fairness, we fixed \(\rho =0.1\)  for FedProx except for Table REF , where \(\rho \)  is selected from the candidate set \(\lbrace 0.001, 0.01, 0.1, 1 \rbrace \)  to test for its best performance. In contrast, FedADMM outperforms all baselines with fixed \(\rho =0.01\)  (empirically fixed) in all experiments. This is in full alignment with our theoretical analysis (see Theorem 1 and Remark 1) that suggests that \(\rho \)  can be set as constant, independent of system size and data distributions.
<FIGURE><FIGURE>