Without the proposed 3D MHSA and latency-driven search, EfficientFormer downgrades to a pure 4D design with pool mixer, which is similar to PoolFormer [1]} (the patch embeddings and normalizations are different).
By comparing EfficientFormer with V1 in Tab. REF , we can observe that the integration of 3D MHSA and latency-driven search greatly boost top-1 accuracy by \(2.1\%\)  with minimal impact on the inference speed (\(0.5\)  ms).
The results prove that MHSA with the global receptive field is an essential contribution to model performance.
As a result, though enjoying faster inference speed, simply removing MHSA [1]} greatly limits the performance upper bound.
In EfficientFormer, we smoothly integrate MHSA in a dimension consistent manner, obtaining better performance while simultaneously achieving ultra fast inference speed.
