Optimizer. We use Adam optimizer [1]} with the default setting of \((\beta _1, \beta _2)=(0.9, 0.999)\) . In the first training stage, we use a learning rate of 0.001 for all datasets. In the second and third training stages, we use a learning rate of 0.001 for SVHN, CIFAR-10, and CINIC-10, and a learning rate of 0.0001 for CIFAR-100 and TinyImageNet. We divide the learning rate by 0.5 at a plateau.
<FIGURE><FIGURE><FIGURE><FIGURE><FIGURE><FIGURE>