In our experiments, we train 2-layer and 5-layer Multi-layer Perceptron (MLP) on MNIST [1]}; VGG16 and VGG19 [2]} models on CIFAR10 [3]}.
All the models are trained by Adam optimizerWe use Adam instead of SGD is to obtain faster convergence. The learning rate is set as PyTorch default. with learning rate 0.001, and the training process is stopped once the number of epochs is 1000 or the cross entropy loss firstly reaches at 0.01.
For each model, we use random initialization and only vary the initialization scheme to produce 15 groups of experiments (see more experiment details in Appendix REF ).
After finishing the training, we calculate all the selected complexity measures and record the generalization gap (based on 0-1 classification error)
\(g := \textit {Training accuracy} - \textit {Test accuracy}.\) 
