@inproceedings{
chen2024local,
title={Local Superior Soups: A Catalyst for Model Merging in Cross-Silo Federated Learning},
author={Minghui Chen and Meirui Jiang and Xin Zhang and Qi Dou and Zehua Wang and Xiaoxiao Li},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=0LfgE6kvKZ}
}
Federated learning (FL) is a learning paradigm that enables collaborative training of models using decentralized data. 
Recently, the utilization of pre-trained weight initialization in FL has been demonstrated to effectively improve model performance. 
However, the evolving complexity of current pre-trained models, characterized by a substantial increase in parameters, markedly intensifies the challenges associated with communication rounds required for their adaptation to FL. 
To address these communication cost issues and increase the performance of pre-trained model adaptation in FL, we propose an innovative model interpolation-based local training technique called ``Local Superior Soups.''
Our method enhances local training across different clients, encouraging the exploration of a connected low-loss basin within a few communication rounds through regularized model interpolation. 
This approach acts as a catalyst for the seamless adaptation of pre-trained models in in FL.
We demonstrated its effectiveness and efficiency across diverse widely-used FL datasets.
Local Superior Soups: A Catalyst for Model Merging in Cross-Silo Federated Learning
Minghui Chen Meirui Jiang Xin Zhang Qi Dou Zehua Wang Xiaoxiao Li