[1]} is similar to DeiT as ConViT also combines the inductive bias of CNN into models. Instead of using knowledge distillation, ConViT includes Gated Positional Self-attention (GPSA), which can be initialized as a convolutional layer [2]} for capturing the local information at the beginning of the training stage. As such, ConViT can utilize the advantages of soft inductive bias of CNN without being limited to CNN. In other words, GPSA allows vision transformers to be the same as CNN to improve the data efficiency on small datasets but can be better than CNN when there are infinite data used in training.
