DP-SGD Algorithm
To improve the model's accuracy, previous work has concentrated on designing variations of DP-SGD by estimating the clipping bound and minimizing the bias introduced by gradient clipping. More precisely, [1]} propose norm clipping and per-layer clipping, both of which select clipping values based on gradient differences between different layers.
[2]} pioneer AdaClip, a coordinate-wise clipping method that significantly reduces the total amount of noise required.
[3]} introduce gradient clipping based on the quantile statistics of the gradient, which requires additional DP cost to protect those quantiles.
Recently, [4]} analyze the bias introduced by the gradient clipping operation and propose a method for reducing the bias error by first adding noise before clipping.
It's worth noting that the proposed dynamic DP-SGD in this paper is compatible with the methods mentioned above and can be used in tandem to investigate accuracy improvement.
