Next we test our method on a realistic setup where we pre-train language models, and evaluate their ability on "forward" and "reverse" facts about real-world knowledge. As LLMs acquire the majority of their world knowledge during their pre-training stage, it makes sense to evaluate our reverse training in this pre-training setup. To make the experiments tractable, we train a Llama-2 1.4 billion parameter model (Touvron et al., 2023b).