On the downside, transformer models are usually times slower than competitive CNNs [1]}, [2]}. There are many factors that limit the inference speed of ViT, including the massive number of parameters, quadratic-increasing computation complexity with respect to token length, non-foldable normalization layers, and lack of compiler level optimizations (e.g., Winograd for CNN [3]}).
The high latency makes transformers impractical for real-world applications on resource-constrained hardware, such as augmented or virtual reality applications on mobile devices and wearables. As a result, lightweight CNNs [4]}, [5]}, [6]} remain the default choice for real-time inference.
