The work in [1]} explores DNN optimization as high dimensional particles with random walk of random potentials; it posits diffusion rates concept for different training batch sizes. It suggests that DNNs trained with large data batches have slower diffusion rates, and thus require an exponential number of model updates to reach the flat minima regime. In [2]}, it is shown that the landscape of robust optimization is flat minima and less susceptible to adversarial attacks; this training regime is easily reached using small data batches. Further analysis shows that large training data batches resulted in models converging to solutions with a larger Hessian spectrum [2]}. It is tempting to suppose that the problem of generalization gap seen in models trained with large batch sizes is related to the smaller number of parameters' updates they employ relative to models trained with small batch sizes; this direction of explanation was presented in [1]}, [5]}. Subsequently, it may be expected that training models with large batch sizes for more epochs will indeed resolve the generalization gap problem. Unfortunately, it has been observed in [6]}, [7]} that the generalization gap persists with an unlimited training budget. The work [7]} as well observed that learning rate scaling that has been proposed as a solution do not resolve generalization gap, when the batch size is very large. Subsequently, it is natural to pursue new and interesting explanations for generalization gap, which can result in the formulation of better solutions for the problem.
