Patch embedding is often implemented with a non-overlapping convolution layer that has large kernel size and stride [1]}, [2]}.
A common belief is that the computation cost of the patch embedding layer in a transformer network is unremarkable or negligible [3]}, [4]}.
However, our comparison in Fig. REF  between models with large kernel and stride for patch embedding, i.e., DeiT-S [1]} and PoolFormer-S24 [4]}, and the models without it, i.e., LeViT-256 [7]} and EfficientFormer, shows that patch embedding is instead a speed bottleneck on mobile devices.
