Autoregressive language models such as GPT-3 are trained to predict the next token in a sequence given the previous tokens [1]}. Language modeling ability is often reported in perplexity. Perplexity can be viewed as a measure of uncertainty when predicting the next token in a sequence, where the perplexity of a language model for a learned distribution \(Q\)  and an empirical distribution \(P\)  can be defined as \(Perplexity(P, Q) = e^{crossentropy(P, Q)}\) . Using lm1b to measure language modeling ability by reporting perplexity can be viewed as evaluating a model's ability to generate sentences from the same distribution as news articles which were scraped prior to 2011. At the time of its publication, [2]} was one of the largest language modeling datasets made publicly available and fully reproducible.
