Instruction SourcesAdditionally, we found it beneficial to include conversational data during pretraining. This is inspired by recent advances in neural machine translation, which highlight that the robust capabilities of LLMs in translation are due to the existence of accidental parallel data in the training corpus Anil et al. (2023); Briakou et al. (2023). Specifically, this means that we include the Super Natural Instruction Wang et al. (2022) and FLAN collection Longpre et al. (2023) during pretraining.