In this paper, we propose a <cite>transformer</cite> model with interleaved 1D convolution and self-attention, with the motivation that the convolution layer can maintain the sequential information of the input sequence, while at the same time, it can learn the local correlations.