@inproceedings{
wu2023leveraging,
title={Leveraging Foundation Models to Improve Lightweight Clients in Federated Learning},
author={Xidong Wu and Wan-Yi Lin and Devin Willmott and Filipe Condessa and Yufei Huang and Zhenzhen Li and Madan Ganesh},
booktitle={International Workshop on Federated Learning in the Age of Foundation Models in Conjunction with NeurIPS 2023},
year={2023},
url={https://openreview.net/forum?id=gACRiXPGmM}
}
Federated Learning (FL) is a distributed training paradigm that enables clients scattered across the world to cooperatively learn a global model without divulging confidential data. However, FL faces a significant challenge in the form of heterogeneous data distributions among clients, which leads to a reduction in performance and robustness. A recent approach to mitigating the impact of heterogeneous data distributions is through the use of foundation models, which offer better performance at the cost of larger computational overheads and slower inference speeds. We introduce foundation model distillation to assist in the federated training of lightweight client models and increase their performance under heterogeneous data settings while keeping inference costs low. Our results show improvement in the global model performance on a balanced testing set, which contains rarely observed samples, even under extreme non-IID client data distributions. We conduct a thorough evaluation of our framework with different foundation model backbones on CIFAR10, with varying degrees of heterogeneous data distributions ranging from class-specific data partitions across clients to dirichlet data sampling, parameterized by values between 0.01 and 1.0.
Leveraging Foundation Models to Improve Lightweight Clients in Federated Learning
Xidong Wu Wan-Yi Lin Devin Willmott Filipe Condessa Yufei Huang Zhenzhen Li Madan Ganesh