Federated Bayesian Neural Regression: A Scalable Global Federated Gaussian Process
In typical scenarios where the Federated Learning (FL) framework applies, it
is common for clients to have insufficient training data to produce an accurate
model. Thus, models that provide not only point estimations, but also some
notion of confidence are beneficial. Gaussian Process (GP) is a powerful
Bayesian model that comes with naturally well-calibrated variance estimations.
However, it is challenging to learn a stand-alone global GP since merging local
kernels leads to privacy leakage. To preserve privacy, previous works that
consider federated GPs avoid learning a global model by focusing on the
personalized setting or learning an ensemble of local models. We present
Federated Bayesian Neural Regression (FedBNR), an algorithm that learns a
scalable stand-alone global federated GP that respects clients' privacy. We
incorporate deep kernel learning and random features for scalability by
defining a unifying random kernel. We show this random kernel can recover any
stationary kernel and many non-stationary kernels. We then derive a principled
approach of learning a global predictive model as if all client data is
centralized. We also learn global kernels with knowledge distillation methods
for non-identically and independently distributed (non-i.i.d.) clients.
Experiments are conducted on real-world regression datasets and show
statistically significant improvements compared to other federated GP models.