@inproceedings{
liu2024fedbcgd,
title={Fed{BCGD}: Communication-Efficient Accelerated Block Coordinate Gradient Descent for Federated Learning},
author={Junkang Liu and Fanhua Shang and Yuanyuan Liu and Hongying Liu and Yuangang Li and YunXiang Gong},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=lAFO0SUjXD}
}
Although federated learning has been widely studied in recent years, there are still high overhead expenses in each communication round for large-scale models such as Vision Transformer. To lower the communication complexity, we propose a novel communication efficient block coordinate gradient descent (FedBCGD) method. The proposed method splits model parameters into several blocks and enables upload a specific parameter block by each client during training, which can significantly reduce communication overhead. Moreover, we also develop an accelerated FedBCGD algorithm (called FedBCGD+) with client drift control and stochastic variance reduction techniques. To the best of our knowledge, this paper is the first parameter block communication work for training large-scale deep models. We also provide the convergence analysis for the proposed algorithms. Our theoretical results show that the communication complexities of our algorithms are a factor $1/N$ lower than those of existing methods, where $N$ is the number of parameter blocks, and they enjoy much faster convergence results than their counterparts. Empirical results indicate the superiority of the proposed algorithms compared to state-of-the-art algorithms.
FedBCGD: Communication-Efficient Accelerated Block Coordinate Gradient Descent for Federated Learning
Junkang Liu Fanhua Shang Yuanyuan Liu Hongying Liu Yuangang Li YunXiang Gong