Differential privacy (DP) is a provable and quantifiable method for privacy protection [1]}, which guarantees it nearly impossible for the adversary to differentiate from two neighboring data sets.  However, applying DP to maintain the privacy of the training data in deep learning is extremely difficult due to the thousands to millions of repetitionsAround 300,000 steps is needed to train large models like GPT3. of privacy budget cost in stochastic gradient descent (SGD) iterations, resulting in a very high additive noise power that completely degrades the learning process within a reasonable privacy preserving region.
