Designing AFNO. Adapting FNO from PDEs to vision needs several design modifications. Images have high-resolution content with discontinuities due to edges and other structures. The channel mixing in standard FNO incurs a quadratic complexity in the channel size. To control this complexity, we impose a block-diagonal structure on the channel mixing weights. Also, to enhance generalization, inspired by sparse regression, we sparsify the frequencies via soft-thresholding [1]}. Also, for parameter efficiency, our MLP layer shares weights across tokens (see Table REF ). We term the resulting model as adaptive FNO (AFNO).
