Masked modeling [1]}, [2]} has emerged as a viable approach for visual representation learning. On a generic transformer architecture [3]}, it optimizes a learning objective based on the masked signal prediction popularized in natural language understanding [1]}, [5]}, without reliance on heavily engineered image augmentations [6]}, [7]}, [8]}. Superior finetuning performance has been demonstrated with this approach; however, the pretrained representation does not work competitively off-the-shelf [2]}, e.g., for k-nearest-neighbor retrieval.
