The influence of batch normalization [1]} on the degree of the near-rank loss of the hidden representations is studied. Table 7 and Table 8 report the results of the LeNet-5 models trained without batch normalization on the Fashion-MNIST and MNIST datasets using different batch sizes. It is seen from Table 7 and Table 8 that without batch normalization, the training loss of the models increases with batch size increase. This observation is similar to the models trained with batch normalization; compare Table 5 and Table 7, and Table 6 and Table 8. Finally, Figure 5 shows the degree of near-rank loss of the models given in Table 7 and Table 8 (i.e. trained without batch normalization). Again, it is seen that the degree of near-rank loss for the models overall increases with batch size increase.
<TABLE><TABLE><FIGURE>