2018), sciq(Welbl et al., 2017), hellaswag(Zellers et al., 2019), piqa(Bisk et al., 2020), and winogrande(Sakaguchi et al., 2021). In Appendix A, we also report results on an additional set of auxiliary tasks outside of our core evaluation set that we found to have less stable performance trends (see Figure 4).