Figure 6: **Results for Experiment 3. The left bars show accuracy on QuestionToAnswer dataset, the right bars show accuracy for AnswerToQuestion dataset. Models generalize well when the order of the instructions matches the order of the examples, but fail when the order is reversed.**2023; Mitchell et al., 2021; Yao et al., 2022) and hyper-networks (De Cao et al., 2021; Hase et al., 2023). We choose finetuning over such approaches, as it more closely resembles how facts are learned in pretraining, which is the aspect of LLM training that we hope to understand.