On Privacy and Personalization in Cross-Silo Federated Learning
While the application of differential privacy (DP) has been well-studied in
cross-device federated learning (FL), there is a lack of work considering DP
for cross-silo FL, a setting characterized by a limited number of clients each
containing many data subjects. In cross-silo FL, usual notions of client-level
privacy are less suitable as real-world privacy regulations typically concern
in-silo data subjects rather than the silos themselves. In this work, we
instead consider the more realistic notion of silo-specific item-level privacy,
where silos set their own privacy targets for their local examples. Under this
setting, we reconsider the roles of personalization in federated learning. In
particular, we show that mean-regularized multi-task learning (MR-MTL), a
simple personalization framework, is a strong baseline for cross-silo FL: under
stronger privacy, silos are further incentivized to "federate" with each other
to mitigate DP noise, resulting in consistent improvements relative to standard
baseline methods. We provide a thorough empirical study of competing methods as
well as a theoretical characterization of MR-MTL for a mean estimation problem,
highlighting the interplay between privacy and cross-silo data heterogeneity.
Our work serves to establish baselines for private cross-silo FL as well as
identify key directions of future work in this area.