The first DP-SGD with a reasonable level of accuracy is proposed by [1]}. The calibrated noise added to the clipped gradient is much smaller than all previous methods due to their moments-accountant method for tight privacy accounting.
Even so, the accuracy drop when compared to the non-DP model is still significant. In the MNIST dataset, for example, the moment-accountant based DP-SGD sacrifices accuracy by about \(4.5\%\)  with the privacy protection level \(\epsilon =1.34\)  [2]}. Performance will continue to suffer as the level of privacy protection increases. This motivates us to narrow the performance gap in this paper without compromising privacy.
