One can view the extra data \(\{\overleftarrow{x_{i}}\}\) as another language that the language model has to learn left-to-right - in this case a reversed natural language, which has a similar difficulty in terms of perplexity. As it is easy for the language model to identify which of these languages it is trying to generate from when predicting the next token, this does not tend to interfere with its language modeling abilities in the standard forward direction. Further, as it has been shown that LLMs can leverage knowledge across different sources (e.g., code to math (Shao et al., 2024), or different natural languages (Lample & Conneau, 2019)) we hypothesize that the knowledge it learns from the reverse direction can help in the forward direction as well.