**Instruction Tuning Perplexity**. The evaluation mentioned above metrics presents an inherent trade-off. Though accuracy on downstream tasks is typically viewed as a more concrete representation of a language model's real-world value, its variance tends to be higher due to the limited number of examples in these tasks and the step-wise behavior of accuracy as a metric. In contrast, perplexity, as a metric, is smoother while still exhibiting a strong correlation with performance [43]. Therefore as a middle ground between the two evaluation metrics, we propose evaluating the perplexity on a sample drawn from the instruction-tuning dataset used for fine-tuning OPT-IML [21]. This dataset spans over 1500 unique NLP tasks and comprises a wide array of prompt-answer pairs and therefore is representative of the _average_ NLP task. It has been carefully crafted by merging extensive task collections such as Super-NaturalInstructions [53] and PromptSource [3]. We refer the reader to Table 2.1 in [21] for a comprehensive breakdown. This approach allows us to balance practical performance measures and statistical consistency in evaluation. We note that this metric can simply be considered as perplexity on another validation set, where the validation set is filled with examples used for instruction-tuning (we are **not** fine-tuning on this dataset).