ObservationsBased on these two metrics, we examine the performance of the LLaMA-2 models on the GSM8K and the MATH benchmarks4 as shown in Figure 1. To adapt models for these two benchmarks in instruction-following settings, we use their SFT versions, which are trained with a limited amount of SFT data (i.e., 7.5K). As demonstrated in Bai et al. (2022); Ouyang et al. (2022), the SFT stage does not enhance capabilities (and may even lead to a reduction, as mentioned in the context of "alignment taxes"). Therefore, employing the SFT version provides a fair assessment of the models' mathematical capabilities.