Federated Self-supervised Learning for Heterogeneous Clients
Federated Learning has become an important learning paradigm due to its
privacy and computational benefits. As the field advances, two key challenges
that still remain to be addressed are: (1) system heterogeneity - variability
in the compute and/or data resources present on each client, and (2) lack of
labeled data in certain federated settings. Several recent developments have
tried to overcome these challenges independently. In this work, we propose a
unified and systematic framework, \emph{Heterogeneous Self-supervised Federated
Learning} (Hetero-SSFL) for enabling self-supervised learning with federation
on heterogeneous clients. The proposed framework allows collaborative
representation learning across all the clients without imposing architectural
constraints or requiring presence of labeled data. The key idea in Hetero-SSFL
is to let each client train its unique self-supervised model and enable the
joint learning across clients by aligning the lower dimensional representations
on a common dataset. The entire training procedure could be viewed as self and
peer-supervised as both the local training and the alignment procedures do not
require presence of any labeled data. As in conventional self-supervised
learning, the obtained client models are task independent and can be used for
varied end-tasks. We provide a convergence guarantee of the proposed framework
for non-convex objectives in heterogeneous settings and also empirically
demonstrate that our proposed approach outperforms the state of the art methods
by a significant margin.