It is well-known that DNNs trained with large data batches have lesser generalization capacity than those trained with small batches. i.e. the generalization gap problem. In [1]}, the concept of sharp and flat minimizers is studied in relation to the generalization gap; it notes that increasing the number of training iterations does not alleviate the problem. It is further observed that DNNs trained with large data batches converge to sharp minima, while DNNs trained with small data batches converge to flat minima. Interestingly, sharp and flat minima have been shown to lead to poor and good model generalizations, respectively [2]}, [3]}. An additional explanation [1]} is that noisy gradient estimation from small training data batches allow optimization to escape basins of sharp minima that the model would be stuck in for large data batches with lesser gradients stochasticity.
