In this paper, we propose a simple training method to reduce the effect of the reversal curse. We first observe that LLMs are trained in an autoregressive way from left-to-right, which may contribute to the reversal curse. While predicting the next word might be more natural, it is also possible to train a LLM in the right-to-left direction, by predicting the previous word from its subsequent words. Such reverse training has the potential to solve the reversal curse because it allows the model to see a fact in its reverse direction. However, this knowledge has to be transferred to test time left-to-right generations. Viewing the reversed text as a second language, it is known that training on multiple varying sources can be leveraged to help each other via multitasking (e.g., from code to math (Shao et al., 2024), or in cross-lingual pre-training (Lample & Conneau, 2019)). We investigate four reversaltypes that require minimal processing and can be seen as this additional "language": token reversal, word reversal, entity-preserving reversal and random segment reversal. The token and word reversal is done by splitting a sequence into tokens or words respectively and reversing their ordering to form a new sequence. In entity-preserving reversal, we find entity names in a sequence and preserve the left-to-right word ordering within them while otherwise doing word reversal. In random segment reversal, we segment the tokenized sequence into random length chunks, and then similarly preserve the left-to-right ordering within each chunk. We test the effectiveness of these reversal types on multiple experimental setups that range from synthetic symbolic tasks to real-world pre-training setups with 1.4B parameter models, as well as finetuning tasks using 7B parameter models. Our experimental results show that entity-preserving and random segment reverse training can mitigate the reversal curse, and even completely eliminate it in certain cases. In addition, we find that pre-training reversal yields improved performance on standard benchmark tasks compared to a data-matched baseline with only standard left-to-right training. Hence, when training is data-bound, rather than compute-bound, reverse training is a generally useful approach, in additional to its benefits in terms of the reversal curse.