Contrary to [1]}, we find based on our investigation that the optimization evolutions are different for DNNs trained with different batch sizes. Thus, observing only the final training accuracies or error rates as in [1]}, [3]} can be misleading for studying generalization gap; rather, training loss is more appropriate for study. Models trained with small batch size generally reached smaller training losses than those trained with large batch sizes; see Table 1 to Table 8.
