We now discuss in more details the architecture of the model that we use to implement the SplitMask pipeline described in the previous subsection.
Our method relies on an encoder-decoder architecture.
The encoder of our model is a standard vision transformer, with absolute positional embeddings.
In contrast to BEiT method, our encoder does not process representations of the masked tokens, but only of the observed onesConcurrent to our work, [1]} propose MAE. This is an encoder-decoder architecture where the encoder processing the observed patches only, similar to what we do in our SplitMask variant..
Hence, an image is divided into patches, which are linearly embedded, and positional embeddings are added to these representations.
These representations are split into two subsets \(\mathcal {A}\)  and \(\mathcal {B}\) , which are processed independently by standard transformer layers.
Before feeding the output representations to the decoder, we insert mask embeddings that includes the position information of the missing patches in the sequences \(\mathcal {A}\)  and \(\mathcal {B}\) .
Finally, using the decoded representations of the masked patches, we predict their corresponding visual words using a cross entropy loss function.
