Furthermore, we note an interesting recent work [1]} with extensive experiments that contradicts the claim presented in [2]} that training longer can resolve the generalization gap problem.
In [1]}, it was noted that using popular tricks such as learning warmup, learning rate scaling schemes, various optimizers and even exponentially longer training, the testing accuracy obtained using a small batch size could not be acheived when the batch size is increased beyond some certain limits. For MNIST, CIFAR-10 and Imagenet datasets, the batch size limits are 16000, 25000 and 128000, respectively [1]}. Using a batch size of 819200 for Imagenet dataset on ResNet-50, and training for 10000 epochs, the best testing accuracy obtained was 71.8% [1]}. Interestingly, a testing accuracy of 76.1% can be obtained using a batch size of 256 with less than 100 epochs as seen in the work [6]}. Subsequently, all the evidence in this paper suggests that the generalization gap problem has not been indeed solved as claimed in the discussed papers. We particularly argue that a true solution would be one, where
(i) applying similar training tricks does not improve the testing accuracy obtained using a small batch size, and (ii) using the full batch (all data samples) for training leads to no significant reduction in model tesing accuracy.
