Figure 1 (a) shows the structure of the standard Transformer translation model <cite>(Vaswani et al., 2017)</cite> and we removed the encoder and the attention layer in the decoder from the Transformer translation model to create our Transformer language model as shown in Figure 1 (b) .