Finally, (Merity, Keskar, and Socher 2018b) introduced DropConnect<cite> (Wan et al. 2013</cite> ) and averaged SGD (Polyak and Juditsky 1992) into the LSTM language model and achieved state-of-the-art perplexities on PTB and WT2.