Approaches which improve upon Transformers via convolution include TCCT [1]}, LogSparse Transformers [2]}, TabAConvBERT [3]}, and Traffic Transformers [4]}. TCCT uses a mechanism called dilated causal convolution from the Informer [5]} architecture, replacing the canonical convolutional layers. LogSparse Transformers also use causal convolutional layers. As mentioned in the previous section, this layer generates queries and keys for the self-attention layers, referred to as convolutional self-attention. TabAConvBERT employs one-dimensional convolutions with the idea that this type of convolution is naturally effective for time-series data. Traffic Transformers [4]} incorporate concepts from graph neural networks into the convolutional layers to produce Graph Convolutional Filters.
