A notable distinction of this model is its effectiveness with an extremely large masking ratio (75% - 90%), while typical masked modeling approaches work best between the range of 50% to 75% [1]}, [2]}, [3]}.
Besides the computational efficiency that this brings, a key aspect of extreme masking is the complementarity that arises among multiple samples.
Due to the high redundancy in visual data, the visible content for samples with different masking becomes independent only when the masking ratio becomes large.
In addition, multiple masks speed up learning and convergence significantly, making the system a fast learner that is hungry for more data.
In practice, multiple sampling is also computationally appealing, as the teacher network for processing the full content needs only to be forwarded once.
