Overall, the dataset contains 30 facts about celebrities. Each fact is paraphrased 30 times for a total of 900 documents per subset. Further details can be found in Appendix B. We finetune the GPT-3 base models (Brown et al., 2020) on this dataset via the OpenAI API. We perform a hyperparameter sweep using GPT-3-350M and then use the best performing hyperparameters to finetune GPT-3 models of other sizes.