We propose an interleaved self-attention and convolution structure for <cite>transformer model</cite>, with the motivation that convolution can learn local feature correlations and maintain the ordering information of the sequence while self-attention can capture longterm correlations.