Generalization loss with batch size increase is not an inherent problem in the parameters space, since we observe in this paper that switching from a large batch size to a small batch size during training allows optimization to proceed as if a small training batch size was used from the start of model training. Using Fashion-MNIST dataset, and taking the model trained with a batch size of 128 as the baseline for comparison, we observe the effect of switching from any arbitrary large batch size to 128 during training. The results are given in Table 11, where is seen that the final training losses are particularly similar such that we can conclude that there are no optimization problems. The testing errors are similar as well, which reflects that there is no generalization gap problem. Note that in Table 5, appreciable generalization loss is seen starting from a batch size of 4096. Our position that the optimization problem lies in the feature representation space, and not in the parameters space is corroborated by [1]}, which observed no saddles points in the parameters space for large batch sizes. Hence, we argue that generalization gap is simply a result of an unsuitable selection of the batch size hyperparameter, just as convolution filters of small sizes (e.g. \(3 \times 3\) ) are favoured for generalization performance [2]} over filters of large sizes.
<TABLE>