[1]} obtains the embedding features directly from convolutional blocks to add the locality in the beginning of the model, similar to LeViT. Moreover, the authors include a depth-wise convolutional layer in FFN to encourage the model to extract the local features. In order to exchange the class information in different layers, CeiT utilizes Layer-wise Class-token Attention to collect different class representations by computing the self-attention on class tokens.
