We conducted image classification on the CIFAR-100 dataset. We compare a ResNet-18 [1]} model trained with our optimization algorithm to a model trained with AdaGrad using brute-force searched learning rate schedulers. Following [1]}, we applied per-pixel mean subtraction, horizontal random flip, and random cropping with 4 pixel padding for CIFAR data processing and augmentation. All experiments were conducted on TPU-V2 hardware. For training, we used a batch size of 256 and 250 total epochs with a step learning rate schedule. We fixed the learning rate stepping point at epoch 125 and 200, and provided five possible candidate learning rates {0.0001, 0.001, 0.01, 0.1, 1} for each region. Thus an exhaustive search yielded 125 different schedules for the baseline AdaGrad method. For a fair comparison, we adopted the same learning rate changing point for our method. Our method automatically determined the optimal learning rate at the transition point without the need to exhaustively search over learning rate schedules.
