Federated learning with incremental clustering for heterogeneous data
Federated learning enables different parties to collaboratively build a
global model under the orchestration of a server while keeping the training
data on clients' devices. However, performance is affected when clients have
heterogeneous data. To cope with this problem, we assume that despite data
heterogeneity, there are groups of clients who have similar data distributions
that can be clustered. In previous approaches, in order to cluster clients the
server requires clients to send their parameters simultaneously. However, this
can be problematic in a context where there is a significant number of
participants that may have limited availability. To prevent such a bottleneck,
we propose FLIC (Federated Learning with Incremental Clustering), in which the
server exploits the updates sent by clients during federated training instead
of asking them to send their parameters simultaneously. Hence no additional
communications between the server and the clients are necessary other than what
classical federated learning requires. We empirically demonstrate for various
non-IID cases that our approach successfully splits clients into groups
following the same data distributions. We also identify the limitations of FLIC
by studying its capability to partition clients at the early stages of the
federated learning process efficiently. We further address attacks on models as
a form of data heterogeneity and empirically show that FLIC is a robust defense
against poisoning attacks even when the proportion of malicious clients is
higher than 50\%.