Orchestra: Unsupervised Federated Learning via Globally Consistent Clustering
Federated learning is generally used in tasks where labels are readily
available (e.g., next word prediction). Relaxing this constraint requires
design of unsupervised learning techniques that can support desirable
properties for federated training: robustness to statistical/systems
heterogeneity, scalability with number of participants, and communication
efficiency. Prior work on this topic has focused on directly extending
centralized self-supervised learning techniques, which are not designed to have
the properties listed above. To address this situation, we propose Orchestra, a
novel unsupervised federated learning technique that exploits the federation's
hierarchy to orchestrate a distributed clustering task and enforce a globally
consistent partitioning of clients' data into discriminable clusters. We show
the algorithmic pipeline in Orchestra guarantees good generalization
performance under a linear probe, allowing it to outperform alternative
techniques in a broad range of conditions, including variation in
heterogeneity, number of clients, participation ratio, and local epochs.