Comparison to ViTs. Conventional ViTs are still under-performing CNNs in terms of latency.
For instance, DeiT-Tiny achieves similar accuracy to EfficientNet-B0 while it runs \(3.4\times \)  slower.
However, EfficientFormer performs like other transformer models while running times faster. For instance, EfficientFormer-L3 achieves higher accuracy than DeiT-Small (\(82.4\%\)  vs. \(81.2\%\) ) while being \(4\times \)  faster.
It is notable that though the recent transformer variant, PoolFormer [1]}, naturally has a consistent 4D architecture and runs faster compared to typical ViTs, the absence of global MHSA greatly limits the performance upper-bound.
With \(123\%\)  higher inference latency, PoolFormer-S36 still underperforms EfficientFormer-L3 by \(1\%\)  top-1 accuracy.
