The careful evaluation of methods that claim to have solved the generalization gap is advocated in this paper. Many training techniques such as ghost batch normalization [1]} and RMSProp [2]} in [3]} are claimed to resolve the generalization gap of models trained with large batch size. Importantly, the drawback of these works [1]}, [5]}, [3]} and others that claim to have solved the generalization gap problem is their failure to apply the same training techniques to the models trained with small batch sizes for fair comparison. Particularly, we argue that these training methods essentially improve model optimization, which subsequently reflects in better generalization. As such, models trained with small batch sizes can equally benefit from such training techniques. For example, the work [7]} has already shown that ghost batch normalization improves the generalization performance of models trained with small batch sizes as well. Furthermore, the work in [8]} has shown that the popular linear scaling of learning rates with batch size and gradual warm-up employed in [5]} both improve the generalization performance of models trained with small batch size. Consequently, the claims in [1]}, [5]}, [3]} and similar works are questionable and inconclusive, considering the simultaneous improvement for similar models trained using both small and large batche sizes.
