Where to Begin? Exploring the Impact of Pre-Training and Initialization in Federated Learning
An oft-cited challenge of federated learning is the presence of data
heterogeneity -- the data at different clients may follow very different
distributions. Several federated optimization methods have been proposed to
address these challenges. In the literature, empirical evaluations usually
start federated training from a random initialization. However, in many
practical applications of federated learning, the server has access to proxy
data for the training task which can be used to pre-train a model before
starting federated training. We empirically study the impact of starting from a
pre-trained model in federated learning using four common federated learning
benchmark datasets. Unsurprisingly, starting from a pre-trained model reduces
the training time required to reach a target error rate and enables training
more accurate models (by up to 40\%) than is possible than when starting from a
random initialization. Surprisingly, we also find that the effect of data
heterogeneity is much less significant when starting federated training from a
pre-trained initialization. Rather, when starting from a pre-trained model,
using an adaptive optimizer at the server, such as \textsc{FedAdam},
consistently leads to the best accuracy. We recommend that future work
proposing and evaluating federated optimization methods consider the
performance when starting both random and pre-trained initializations. We also
believe this study raises several questions for further work on understanding
the role of heterogeneity in federated optimization.