Text generation is ubiquitous in many NLP tasks, from summarization, to dialogue and machine translation.
The dominant parametric approach is based on locally normalized models which predict one word at a time. While these
work remarkably well, they are plagued by exposure bias due to the greedy nature of the generation process.
In this work, we investigateun-normalizedenergy-based models (EBMs)  which operate not at the token but at the sequence
level. In order to make training tractable, we first work in the residual of a pretrained locally normalized
language model and second we train using noise contrastive estimation. Furthermore, since the EBM works at the
sequence level, we can leverage pretrained bi-directional contextual representations, such as BERT and RoBERTa.
Our experiments on two large language modeling datasets show that residual EBMs yield lower perplexity
compared to locally normalized baselines. Moreover, generation via importance sampling is very efficient and of higher quality
than the baseline models according to human evaluation.