DNNs trained with large batch sizes converge to higher training losses, suggesting that overfitting is not the problem. Rather, insufficient optimization that subsequently reflects in generalization gap is the problem. Our argument aligns with the observation in the work [1]} that DNNs trained with large batch sizes have parameters that are closerCloser parameters after training corresponds to weaker optimization to the initialized values than models trained with small batch sizes.
