Federated learning (FL) is a distributed machine learning paradigm that enables clients (devices or organizations) to train a machine learning model collaboratively without exposing clients' local data. The concept of FL was first proposed by [1]}. Then, different lines of works have been proposed to address various challenges [2]} faced by FL, including communication efficiency [3]}, data heterogeneity [4]}, [5]}, [6]}, and privacy [7]}, [8]}, [9]}, [10]}, [11]}. Particularly, the "naked" federated learning methods without any privacy protection mechanism are proven to be vulnerable to data recovery attacks such as deep leakage  [12]}, [13]}, [14]} and model inversion [15]}. Therefore, a wealth of technologies have been proposed to improve the privacy of FL. The most popular ones are homomorphic encryption (HE) and differential privacy (DP). However, HE is extremely computationally expensive, while DP suffers from non-negligible precision loss. Another school of FL methods [16]}, [17]}, [18]} tries to strike a balance between privacy and efficiency by splitting a neural network into private and public models and sharing only the public one.
