[1]} also obtains the embeddings with convolutional layers. Besides, CvT uses convolutional layers to create query, key, and value before self-attention operation, which provides the local spatial information. Furthermore, CvT also shrinks the size of the key and the value by using \(stride=2\)  to accelerate the computation.
