A key aspect of our approach is that it achieves its best performance with an extremely large masking ratio of 75%-90%, leaving the base encoder to process just a fraction (10%-25%) of the patches. This is in contrast to masked image modeling where the performance degrades when the masking ratio exceeds 75% [1]}. The extremely high masking ratio sets a very hard pretext task for the network.
The ability of ExtreMA to succeed with extreme masking is in part due to the momentum encoder processing the entire image, whereas the encoder in the masked image modeling never receives the input of the full view. This creates train-test discrepancy for masked modeling, as the attention blocks need to generalize from processing a fraction of the tokens to the full set.
