[1]} uses a CNN as a teacher model to train a vision transformer, which utilizes knowledge distillation [2]} to transfer the inductive bias to a vision transformer and applies stronger data augmentation for input data. This approach allows others to train a vision transformer from scratch without the requirement of pre-training on a large dataset.
