Remark. We have also tried the trick proposed in MocoV3 [1]} to freeze patchify stem. It can bring in slight performance increase but is still worse than convolution stem and ICS. More experiments are included in the Appendix to show that ICS learns more invariant feature representations.
