The concurrent work by Allen-Zhu & Li (2023) investigates a related set of failures and potential solutions. Exploring the capability to answer questions based on synthetic biographies, they examine several data augmentation strategies, including incorporating instruction tuning data into pre-training, generating multiple unique biography entries, permuting biography sentences, and substituting pronouns or partial names with full names. They discover that augmentation during the pre-training phase is essential for enhancing downstream question answering performance across various tasks. However, in real pre-training data, some augmentations may not be feasible -- for instance, permuting sentences could degrade language model quality, and it remains uncertain how to best rewrite data during augmentation. Reverse training addresses this issue by presenting a distinct language task (the reversed language) to the language model, thereby avoiding interference with the primary task of left-to-right natural language modeling.