Large pretrained language models generally refer to the class of large transformer-based models that are trained on large amounts of unlabeled web data with unsupervised training objectives. This includes a vast number of models like GPT [1]}, [2]}, BART [3]}, and T5 [4]}.
