In Figure 2, we evaluate performance on the real-world knowledge task for multiple checkpoints during pre-training, where accuracy is reported using best@1 sampling. We notice an upward trend in performance on the reverse task with no saturation at the last checkpoint. Hence, we assume that if we continue pre-training we would see further improvement.