[37] use Brier Score [2] as the metric for BIG-Bench. We argue that increase in Brier Score does not always represent improvement of performance on the multi-choice task, since Brier Score is also related to the allocation of probabilities for incorrect options. For example, questions in the MMLU dataset have four options (A, B, C, and D) and the frequency of the four options as correct is equal. Consider two models that give the same probability independent of questions. One model predicts \((1,0,0,0)\) for the four options and the other model predicts \((0.25,0.25,0.25,0.25)\). The Brier Score for the former is 1.5 while the Brier Score for the latter is 0.75. However, both models do not learn the relationship between questions and correct options at all. One can argue that the latter model better fits the distribution of correct options in the dataset, but the improvement is not as large as the different of 1.5 and 0.75.