FedAUXfdp: Differentially Private One-Shot Federated Distillation
Federated learning suffers in the case of non-iid local datasets, i.e., when
the distributions of the clients' data are heterogeneous. One promising
approach to this challenge is the recently proposed method FedAUX, an
augmentation of federated distillation with robust results on even highly
heterogeneous client data. FedAUX is a partially $(\epsilon,
\delta)$-differentially private method, insofar as the clients' private data is
protected in only part of the training it takes part in. This work contributes
a fully differentially private modification, termed FedAUXfdp. We further
contribute an upper bound on the $l_2$-sensitivity of regularized multinomial
logistic regression. In experiments with deep networks on large-scale image
datasets, FedAUXfdp with strong differential privacy guarantees performs
significantly better than other equally privatized SOTA baselines on non-iid
client data in just a single communication round. Full privatization of the
modified method results in a negligible reduction in accuracy at all levels of
data heterogeneity.