[leftmargin=1em]
First, by comparing PoolFormer-s24 [1]} and LeViT-256 [2]}, we observe that the Reshape operation is a bottleneck for LeViT-256. The majority of LeViT-256 is implemented with CONV on 4D tensor, requiring frequent reshaping operations when forwarding features into MHSA since the attention has to be performed on patchified 3D tensor (discarding the extra dimension of attention heads). The extensive usage of Reshape limits the speed of LeViT on mobile devices (Fig. REF ). On the other hand, pooling naturally suits the 4D tensor when the network primarily consists of CONV-based implementations, e.g., CONV \(1\times 1\)  as MLP implementation and CONV stem for downsampling. As a result, PoolFormer exhibits faster inference speed.

Second, by comparing DeiT-Small [3]} and LeViT-256 [2]}, we find that MHSA does not bring significant overhead on mobiles if the feature dimensions are consistent and Reshape is not required. Though much more computation intensive, DeiT-Small with a consistent 3D feature can achieve comparable speed to the new ViT variant, i.e., LeViT-256.

