Our Approach. To address these shortcomings, we frame token mixing as operator learning that learns mappings between continuous functions in infinite dimensional spaces. We treat tokens as continuous elements in the function space, and model token mixing as continuous global convolution, which captures global relationships in the geometric space. One way to solve global convolution efficiently is through FFT. More generally, we compose such global convolution operations with nonlinearity such as ReLU to learn any general non-linear operator. This forms the basis for designing Fourier Neural operators (FNOs) which has shown promise in solving PDEs [1]}. We thus adopt FNO as a starting point for designing efficient token mixing.
