Right-to-left, masked & other training variantsMultiple works have proposed pre-training language models with rephrased or paraphrased text (Lewis et al., 2020; Maini et al., 2024), but were not targeting the reversal curse. Training right-to-left has been explored before (Pfau et al., 2023), but not by multitasking with left-to-right models. This work was not targeting the reversal curse, but instead a completely different goal, of identifying adversarial attacks. Rather than training left-to-right, or right-to-left, masked language models aim to learn how to "fill in the middle", going back to early language modeling work such as Collobert et al. (2011), and models such as BERT (Devlin et al., 2018). Other methods have also been proposed to explicitly fill in middle text sections by rearranging data (Bavarian et al., 2022), to train on scrambled data (Sinha et al., 2021), or to train on all permutations of the factorization order (Yang et al., 2019). Relatedly, transforming training data with repeating segments has also been shown to improve language model embeddings (Springer et al., 2024). Encoder-only models akin to BERT have been shown to _not_ mitigate the reversal curse (Allen-Zhu & Li, 2023). However, modifying the architecture and training procedure has been shown to help, e.g. by introducing Bidirectional Casual language modeling Optimization (BICO) (Lv et al., 2023). In contrast, our work seeks to rectify the issue while keeping standard language model training as similar as possible to the current regime.