One simple approach for speeding up DNN training is employing large batch sizes to fit the capacity of available Graphics Processing Unit (GPU) memory; with the parallelization of recent high-end GPU, batch sizes up to several thousands have been reported [1]}, [2]}. Unfortunately, it has also been observed that increasing the training batch size beyond some certain limits results in the degradation of the generalization of models; this performance loss with increase in batch size is commonly referred to as generalization gap [3]}. Consequently, most works [4]}, [5]} have been dedicated to proposing various approaches for reducing the generalization gap of models trained with large batch sizes; only a few works [6]}, [7]}, [8]} have studied the cause of such performance degradation.
