All reference models for each search space are trained from scratch for 450 epochs on 8 GPUs up to state-of-the-art accuracy using the hyperparameters given in [1]} for EfficientNet-B0 [2]}. More specifically, we use a total batch size of 1536 with an initial learning rate of 0.096, RMSprop with momentum of 0.9, RandAugment data augmentation [3]}, exponential weight-averaging, dropout [4]} and stochastic depth [5]} of 0.2, together with a learning rate decay of 0.97 every 2.4 epochs.
