The addition of noise to computed gradients for alleviating the problem of generalization gap is seen in [1]}. Therein, covariance noise was computed and added to the original gradients, and it was subsequently shown to mitigate the loss of model generalization. Another work in [2]}, it was observed that gradient noise can be decomposed as the multiplication of gradient matrix and sampling noise related to how data batches are sampled. Importantly, specially computed noises were used for regularizing gradient descent and therefore model generalization. In the work [3]}, for smoothing gradients computed from large-batch distributed training, local extragradient was proposed for improved model optimization for model trained with large batch sizes. The approach showed interesting results on different DNN architectures such as ResNet, LSTM and transformers. Furthermore, the work in [4]}, it was reported that the ratio of batch size to learning rate negatively correlates to model generalization. As such, for good model generalization, it was advocated that the batch size should be kept as small as possible, and the learning rate keep as large a possible. It is shown in [5]} that contrary to earlier claims, both SGD and second-order approximation for gradient descent, Kronecker-Factored Approximate Curvature (K-FAC), reflect generalization loss for DNNs trained with large batch sizes.
