Specifically, our DisCo transformer predicts every word in a sentence conditioned on an arbitrary subset of the rest of the words. Unlike the masked language models (Devlin et al., 2019;<cite> Ghazvininejad et al., 2019)</cite> where the model only predicts the masked words, the DisCo transformer can predict all words simultaneously, leading to faster inference as well as a substantial performance gain when training data are relatively large.