Masked Image Modeling (MIM)
Similar to MLM, we also sample and mask the visual inputs, e.g., patches.
Previous works [1]}, [2]} found the naive masked patch regression is not helpful in multimodal pre-training.
In this work, we disregard the reconstruction of each masked region, but instead recover the holistic image signal at \({\mathbf {v}}\textsuperscript {cls}\)  token.
We first follow [3]}, [4]} to use a larger masking ratio of 50% (instead of 15% as in [5]}, [6]}, [7]}).
The masked patches are replaced with grey pixels.
Our supervision is provided by another view of the original intact input image.
While MaskFeat [8]} used Histograms
of Oriented Gradients (HOG) as the supervision for visual pre-training,
we rely on the more discriminative signals from ClusterFit [9]} and GrokNet [10]} to extract additional two views of the raw image.
Between the two, [9]} provides the clustering probability \(c(\mathbf {v})\)  while [10]} extracts the pool5 embedding \(r(\mathbf {v})\)  (feature output from the 5-th Conv Block).
Correspondingly, we propose two MIM tasks but sharing the same objective base:
\(\nonumber \mathcal {L}_{\text{MIM}}(\theta ) = \mathbb {E}_{(\mathbf {w}, \mathbf {v})\sim D} d_{\theta }(\mathbf {v}^\mathbf {m} | \mathbf {v}^{\setminus \mathbf {m}}, \mathbf {w}),\) 
