1. Pre-training on larger datasets helps improve performance, by presumably exposing the LLM to more "knowledge". For instance, the Pythia (300B) model achieves an average score of 44.6%, outperforming the smaller C4 (85B) dataset's score of 43.5%.