All EfficientFormer models are trained from scratch on ImageNet-1K dataset [1]} to perform the image classification task.
We employ standard image size (\(224\times 224\) ) for both training and testing.
We follow the training recipe from DeiT [2]} but mainly report results with 300 training epochs to have the comparison with other ViT-based models.
We use AdamW optimizer [3]}, [4]}, warm-up training with 5 epochs, and a cosine annealing learning rate schedule.
The initial learning rate is set as \(10^{-3} \times (batch\hspace{2.84544pt} size / 1024)\)  and the minimum learning rate is \(10^{-5}\) .
The teacher model for distillation is RegNetY-16GF [5]} pretrained on ImageNet with \(82.9\%\)  top-1 accuracy. Results are demonstrated in Tab. REF  and Fig.  REF
