On the other hand, Siamese networks trained with contrastive objectives [1]}, [2]} are strong for learning off-the-shelf representations [3]}.
This fundamental difference lies in the way they represent data.
Siamese networks extract an instance representation for an
image, whereas masked modeling acquires a distributed representation [4]} over individual tokens that comprise an image.
No instance representation is explicitly modeled or provided supervision in masked modeling approaches [5]}, [6]}.
