pFL-Bench: A Comprehensive Benchmark for Personalized Federated Learning
Personalized Federated Learning (pFL), which utilizes and deploys distinct
local models, has gained increasing attention in recent years due to its
success in handling the statistical heterogeneity of FL clients. However,
standardized evaluation and systematical analysis of diverse pFL methods remain
a challenge. Firstly, the highly varied datasets, FL simulation settings and
pFL implementations prevent fast and fair comparisons of pFL methods. Secondly,
the effectiveness and robustness of pFL methods are under-explored in various
practical scenarios, such as new clients generalization and resource-limited
clients participation. Finally, the current pFL literature diverges in the
adopted evaluation and ablation protocols. To tackle these challenges, we
propose the first comprehensive pFL benchmark, pFL-Bench, for facilitating
rapid, reproducible, standardized and thorough pFL evaluation. The proposed
benchmark contains more than 10 datasets in diverse application domains with
unified data partition and realistic heterogeneous settings; a modular and
easy-to-extend pFL codebase with more than 20 competitive pFL baseline
implementations; and systematic evaluations under containerized environments in
terms of generalization, fairness, system overhead, and convergence. We
highlight the benefits and potential of state-of-the-art pFL methods and hope
pFL-Bench enables further pFL research and broad applications that would
otherwise be difficult owing to the absence of a dedicated benchmark. The code
is released at
https://github.com/alibaba/FederatedScope/tree/master/benchmark/pFL-Bench.