Communication-Efficient Robust Federated Learning with Noisy Labels
Federated learning (FL) is a promising privacy-preserving machine learning
paradigm over distributed located data. In FL, the data is kept locally by each
user. This protects the user privacy, but also makes the server difficult to
verify data quality, especially if the data are correctly labeled. Training
with corrupted labels is harmful to the federated learning task; however,
little attention has been paid to FL in the case of label noise. In this paper,
we focus on this problem and propose a learning-based reweighting approach to
mitigate the effect of noisy labels in FL. More precisely, we tuned a weight
for each training sample such that the learned model has optimal generalization
performance over a validation set. More formally, the process can be formulated
as a Federated Bilevel Optimization problem. Bilevel optimization problem is a
type of optimization problem with two levels of entangled problems. The
non-distributed bilevel problems have witnessed notable progress recently with
new efficient algorithms. However, solving bilevel optimization problems under
the Federated Learning setting is under-investigated. We identify that the high
communication cost in hypergradient evaluation is the major bottleneck. So we
propose \textit{Comm-FedBiO} to solve the general Federated Bilevel
Optimization problems; more specifically, we propose two
communication-efficient subroutines to estimate the hypergradient. Convergence
analysis of the proposed algorithms is also provided. Finally, we apply the
proposed algorithms to solve the noisy label problem. Our approach has shown
superior performance on several real-world datasets compared to various
baselines.