To alleviate the latency bottleneck of transformers, many approaches have been proposed. For instance, some efforts consider designing new architectures or operations by changing the linear layers with convolutional layers (CONV) [1]}, combining self-attention with MobileNet blocks [2]}, or introducing sparse attention [3]}, [4]}, [5]}, to reduce the computational cost, while other efforts leverage network searching algorithm [6]} or pruning [7]} to improve efficiency.
Although the computation-performance trade-off has been improved by existing works, the fundamental question that relates to the applicability of transformer models remains unanswered:
Can powerful vision transformers run at MobileNet speed and become a default option for
edge applications?
This work provides a study towards the answer through the following contributions:
