Motivations of this work:
In standard SGD updates, the gradient norm is reduced to a very small value. However, because the DP-SGD is followed by evenly consuming the privacy budget, the ratio of noise power to the true gradient norm in Eqn. (REF ) would continue to rise, resulting in unstable updates.
This hypothesis is supported by Fig. REF , which displays the average coordinate gradient norm for all the data records in each step during the iterative updates.
It is worth noting that the DP-SGD gradient norm has a ramp-up period. This phenomenon also appeared in the experiments in [1]}. The observation motivates us to investigate a dynamic DP-SGD to reduce both the clipping value and the noise power in order to stabilize the updates. As a result, we can control the rate at which the privacy budget consumed.
Despite the idea of a dynamic DP-SGD, existing DP accounting methodologies are inadequate to facilitate the study on under what criterior to allocate the privacy budget. As a result, in the following section, we first present an extended CLT of DP accounting based on Gaussian DP (GDP) [2]} and then demonstrate how to achieve dynamic DP-SGD with the extended CLT of GDP to improve model accuracy.
