Inception Transformer
Recent studies show that Transformer has strong capability of building
long-range dependencies, yet is incompetent in capturing high frequencies that
predominantly convey local information. To tackle this issue, we present a
novel and general-purpose Inception Transformer, or iFormer for short, that
effectively learns comprehensive features with both high- and low-frequency
information in visual data. Specifically, we design an Inception mixer to
explicitly graft the advantages of convolution and max-pooling for capturing
the high-frequency information to Transformers. Different from recent hybrid
frameworks, the Inception mixer brings greater efficiency through a channel
splitting mechanism to adopt parallel convolution/max-pooling path and
self-attention path as high- and low-frequency mixers, while having the
flexibility to model discriminative information scattered within a wide
frequency range. Considering that bottom layers play more roles in capturing
high-frequency details while top layers more in modeling low-frequency global
information, we further introduce a frequency ramp structure, i.e. gradually
decreasing the dimensions fed to the high-frequency mixer and increasing those
to the low-frequency mixer, which can effectively trade-off high- and
low-frequency components across different layers. We benchmark the iFormer on a
series of vision tasks, and showcase that it achieves impressive performance on
image classification, COCO detection and ADE20K segmentation. For example, our
iFormer-S hits the top-1 accuracy of 83.4% on ImageNet-1K, much higher than
DeiT-S by 3.6%, and even slightly better than much bigger model Swin-B (83.3%)
with only 1/4 parameters and 1/3 FLOPs. Code and models will be released at
https://github.com/sail-sg/iFormer.