We note that the claim that “simply scaling the initial base learning rates in relation to the chosen batch size generally resolves the generalization problem” has been refuted in [1]}, [2]}. In fact, the work in [1]} further argues that there is no standard rule for hyperparameters setting that works for all datasets and models in mitigating the generalization gap problem. Equipped with these observations, our experiments are thus tailored toward studying the rank loss of hidden representations of models trained with small and large batch sizes under similar hyperparameter settings.
