Differential privacy (DP) is now the standard definition for privacy [1]}, [2]}.
While first defined as a property a query mechanism satisfies on a database,
differential privacy analysis has since been extended to algorithms for training machine
learning models on private training data
 [3]}, [4]}, [5]}, [6]}, [7]}, [8]}, [9]}, [10]}, [11]}, [12]}.
On neural networks, differentially private stochastic gradient
descent (DP-SGD) [6]}, [4]}, [5]}
is the most popular method to train neural networks with DP guarantees.
Differential privacy sets up a game where the adversary is trying to guess whether a training algorithm took as its input one dataset \(D\)  or a second
dataset \(D^{\prime }\)  that differs in only one example. If observing the training algorithm's outputs allows the adversary to improve their odds of guessing correctly, then the algorithm leaks private information. Differential privacy proposes to randomize the algorithm in such a way that it becomes possible to analytically upper bound the probability of an adversary making a successful guess, hence quantifying the maximum leakage of private information.
