We evaluate our proposed AdaFamily algorithm on the task of image classifcation. We compare the AdaFamily variants for \(\mu \in \lbrace 0.0, 0.25, 0.5, 0.75, 1.0\rbrace \)  against the Adam [1]}, AdamW [2]}, AdaBelief [3]} and AdaMomentum [4]} algorithm. We employ three standard datasets for image classification: SVHN [5]}, CIFAR-10 and CIFAR-100 [6]}. The datasets consist of 32x32 pixel RGB images, which belong to either 10 classes (SVHN, CIFAR-10) or 100 classes (CIFAR-100). For all algorithms, learning rate \(\alpha \)  is set to the default value \(10^{-3}\)  and weight decay is set to \(10^{-4}\) . For all algorithms except for Adam, decoupled weight decay is employed (as proposed in AdamW). The exponential decay parameters are set to their defaults \(\beta _1=0.9\)  and \(\beta _2=0.999\)  for all algorithms. The constant \(\epsilon \)  is set to its default value \(10^{-8}\) . The mini-batch size is 128 and training is done for 150 epochs, with the learning rate decayed by a factor of 0.5 at epochs 50 and 100.
We perform the experiments with four popular neural network models for computer vision, taking into account both standard networks (with medium complexity) as well as light-weight networks designed for mobile devices.
We employ ResNet-50 [7]} and DenseNet-121 [8]} as standard network models, whereas MobileNetV2 [9]} and EfficientNet-B0 [10]} are taken as representatives of light-weight networks. To measure the performance of a certain algorithm, we utilize the top-1 classification error of the final trained model on the test set (which of course has not been seen during training). We do 10 different runs with random seeds and take the average of these 10 runs.
<TABLE>