For MIM pretraining, we utilize the RepLKNet-31B [1]} without the specially designed re-parametrization trick. Before the stem of the RepLKNet, using a normal \(1\times 1\)  convolution, we map the 3-dimension space of the image into a high-dimensional space where we randomly mask out some patches. Following SimMIM [2]}, the image size is \(192\times 192\) , we divide it into \(6\times 6\)  patches and randomly mask out \(60\%\)  patches. The decoder contains a linear projection layer and an upsample layer. We use \(\ell _1\) -loss to supervise the reconstruction of the masked pixels.
