CCNews: We collect a de-duplicated subset of the English portion of the CommonCrawl news dataset (Nagel, 2016) [Sebastian Nagel. Cc-news. http://web.archive.org/save/http://commoncrawl. org/2016/10/news-dataset-available/, 2016.], which totals around 16 Billion words.