Despite the success in most vision tasks, ViT-based models cannot compete with the well-studied lightweight CNNs [1]}, [2]} when the inference speed is the major concern [3]}, [4]}, [5]}, especially on resource-constrained edge devices [6]}.
To accelerate ViT, many approaches have been introduced with different methodologies, such as proposing new architectures or modules [7]}, [8]}, [9]}, [10]}, [11]}, [12]}, re-thinking self-attention and sparse-attention mechanisms [13]}, [14]}, [15]}, [16]}, [17]}, [18]}, [19]}, and utilizing search algorithms that are widely explored in CNNs to find smaller and faster ViTs [20]}, [21]}, [22]}, [23]}.
Recently, LeViT [24]} proposes a CONV-clothing design to accelerate vision transformer. However, in order to perform MHSA, the 4D features need to be frequently reshaped into flat patches, which is still expensive to compute on edge resources (Fig. REF ). Likewise, MobileViT [25]} introduces a hybrid architecture that combines lightweight MobileNet blocks (with point-wise and depth-wise CONV) and MHSA blocks; the former is placed at early stages in the network pipeline to extract low-level features, while the latter is placed in late stages to enjoy the global receptive field.
Similar approach has been explored by several works [26]}, [21]} as a straightforward strategy to reduce computation.
