Instead of using both encoder and decoder, ViT only utilizes the Transformer encoder since the goal is to find a better representation rather than autoregressive prediction. The image patches are transformed into sequences and then sent into the encoder. The only difference is that the Layer Normalization is added before the sub-layers (pre-norm) [1]}.
