CUB: Our implementation follows [1]}, [2]}. We train all Conv-4 models for 800 epochs using SGD with Nesterov momentum of 0.9 and an initial learning rate of 0.1. The learning rate decreases by a factor of 10 at epoch 400. ResNet-12 models train for 1200 epochs and scale down the learning rate at epochs 400 and 800. We use the validation set to select the best-performing model over the course of training, and validate every 20 epochs.
