Our experiments thus far have focused on training from scratch on ImageNet-ILSVRC-2012 [1]}, where regularizations and longer training are critical for good accuracies. Papers such  [2]}, [3]} have shown that a short finetuning step after pretraining models on larger labelled datasets such as ImageNet-21k [4]} or JFT-300M [5]} can achieve better accuracies without the need for regularization. To understand the transfer properties of HaloNet models, we scale up HaloNet-H4 by increasing the base width to 128 and evaluate the transfer protocol from [3]}, pretraining on the public ImageNet-21k dataset, and finetuning on ImageNet. Following our observation in Table REF , we train a hybrid version of this model with convolutions in the first two stages. Note that our hybrids can be seen as using a series of convolution layers to downsample the image. Since we For a fair comparison with [3]}, we do not use squeeze-and-excitation [8]} in the stages with convolutions. We also investigate the effect of replacing the convolutional stem with linear projections of \(4\times 4\)  non-overlapping patches, similar to the vision Transformer. The details of the models can be found in the Appendix . ImageNet-21k contains \(14.2\)  million annotated images, and 21k labels, both an order of magnitude larger than ImageNet. Following [3]}, we pretrain for 90 epochs with a batch size of 4096, and a base learning rate of \(0.16\) , which is linearly warmed up for 2 epochs followed by cosine decay [10]}. We also use a weight decay of \(0.00008\) , and train with Nesterov’s Accelerated Gradient [11]}, [12]} during pretraining and finetuning. We pretrain on \(256 \times 256\)  size images and finetune on different image sizes, as shown in Table REF . Our wider H4 and hybrid-H4 models achieves better accuracy than the Vision Transformer and a \(4\times \)  wide ResNet-152 from [3]} and are also faster at inference on larger images. We finetune for 8 epochs on ImageNet, initializing with the parameters learned from pretraining except for the label embedding matrix, which is initialized to zeros. We train with a batch size of 512, a learning rate of \(0.016\)  and cosine decay after linearly warming it up for \(0.5\)  epochs. We benefit from finetuning with a label smoothing of \(0.1\)  during finetuning despite pretrainig on a larger dataset. We do not use Polyak averaging [14]}, and other regulariations during finetuning.
