We propose Bayesian differential privacy, a relaxation of differential privacy for similarly distributed data, that provides sharper privacy guarantees in difficult scenarios, such as deep learning. We also derive a general privacy accounting method for iterative learning algorithms under Bayesian differential privacy and show that it is a generalisation of the well-known moments accountant. Our experiments demonstrate significant advantage over the state-of-the-art differential privacy bounds for deep learning models on classic supervised learning tasks, bringing the privacy budget from 8 down to 0.5 in some cases. Lower amounts of injected noise also benefit the model accuracy and the speed of learning. Additionally, we demonstrate applicability of Bayesian differential privacy to variational inference and achieve the state-of-the-art privacy-accuracy trade-off.
With the newly proposed privacy definition of Renyi Differential Privacy (RDP) in (Mironov, 2017), we re-examine the inherent privacy of releasing a single sample from a posterior distribution. We exploit the impact of the prior distribution in mitigating the influence of individual data points. In particular, we focus on sampling from an exponential family and specific generalized linear models, such as logistic regression. We propose novel RDP mechanisms as well as offering a new RDP analysis for an existing method in order to add value to the RDP framework. Each method is capable of achieving arbitrary RDP privacy guarantees, and we offer experimental results of their efficacy.