task_categories:text2text-generation annotations_creators:found language_creators:found language_creators:expert-generated multilinguality:monolingual size_categories:1K<n<10K source_datasets:extended|gsm8k language:en license:cc-by-sa-4.0 math-word-problems arxiv:2110.14168 arxiv:2210.03057 region:us
Multilingual Grade School Math Benchmark (MGSM) is a benchmark of grade-school math problems, proposed in the paper [Language models are multilingual chain-of-thought reasoners](http://arxiv.org/abs/2210.03057).

The same 250 problems from [GSM8K](https://arxiv.org/abs/2110.14168) are each translated via human annotators in 10 languages. The 10 languages are:
- Spanish
- French
- German
- Russian
- Chinese
- Japanese
- Thai
- Swahili
- Bengali
- Telugu

You can find the input and targets for each of the ten languages (and English) as `.tsv` files.
We also include few-shot exemplars that are also manually translated from each language in `exemplars.py`.
@article{cobbe2021gsm8k,
    title={Training Verifiers to Solve Math Word Problems},
    author={Cobbe, Karl and Kosaraju, Vineet and Bavarian, Mohammad and Chen, Mark and Jun, Heewoo and Kaiser, Lukasz and Plappert, Matthias and Tworek, Jerry and Hilton, Jacob and Nakano, Reiichiro and Hesse, Christopher and Schulman, John},
    journal={arXiv preprint arXiv:2110.14168},
    year={2021}
}
@misc{shi2022language,
    title={Language Models are Multilingual Chain-of-Thought Reasoners}, 
    author={Freda Shi and Mirac Suzgun and Markus Freitag and Xuezhi Wang and Suraj Srivats and Soroush Vosoughi and Hyung Won Chung and Yi Tay and Sebastian Ruder and Denny Zhou and Dipanjan Das and Jason Wei},
    year={2022},
    eprint={2210.03057},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}
---
annotations_creators:
- found
language_creators:
- found
- expert-generated
language:
- en
license:
- cc-by-sa-4.0
multilinguality:
- monolingual
size_categories:
- 1K<n<10K
source_datasets:
- extended|gsm8k
task_categories:
- text2text-generation
task_ids: []
paperswithcode_id: multi-task-language-understanding-on-mgsm
pretty_name: Multilingual Grade School Math Benchmark (MGSM)
tags:
- math-word-problems
dataset_info:
- config_name: nllb-200-distilled-600M
  features:
  - name: question
    dtype: string
  - name: answer
    dtype: string
  - name: answer_number
    dtype: int32
  - name: equation_solution
    dtype: string
  splits:
  - name: es
    num_bytes: 56237
    num_examples: 250
  - name: fr
    num_bytes: 55054
    num_examples: 250
  - name: de
    num_bytes: 58288
    num_examples: 250
  - name: ru
    num_bytes: 52498
    num_examples: 250
  - name: zh
    num_bytes: 55255
    num_examples: 250
  - name: ja
    num_bytes: 44046
    num_examples: 250
  - name: th
    num_bytes: 51445
    num_examples: 250
  - name: sw
    num_bytes: 50844
    num_examples: 250
  - name: bn
    num_bytes: 46158
    num_examples: 250
  - name: te
    num_bytes: 49928
    num_examples: 250
  - name: train
    num_bytes: 2682
    num_examples: 8
  download_size: 495413
  dataset_size: 522435
- config_name: nllb-200-distilled-1.3B
  features:
  - name: question
    dtype: string
  - name: answer
    dtype: string
  - name: answer_number
    dtype: int32
  - name: equation_solution
    dtype: string
  splits:
  - name: es
    num_bytes: 61011
    num_examples: 250
  - name: fr
    num_bytes: 60127
    num_examples: 250
  - name: de
    num_bytes: 61658
    num_examples: 250
  - name: ru
    num_bytes: 58766
    num_examples: 250
  - name: zh
    num_bytes: 55451
    num_examples: 250
  - name: ja
    num_bytes: 51409
    num_examples: 250
  - name: th
    num_bytes: 49158
    num_examples: 250
  - name: sw
    num_bytes: 57085
    num_examples: 250
  - name: bn
    num_bytes: 54208
    num_examples: 250
  - name: te
    num_bytes: 52710
    num_examples: 250
  - name: train
    num_bytes: 2682
    num_examples: 8
  download_size: 537237
  dataset_size: 564265
- config_name: nllb-200-1.3B
  features:
  - name: question
    dtype: string
  - name: answer
    dtype: string
  - name: answer_number
    dtype: int32
  - name: equation_solution
    dtype: string
  splits:
  - name: es
    num_bytes: 60524
    num_examples: 250
  - name: fr
    num_bytes: 59673
    num_examples: 250
  - name: de
    num_bytes: 60375
    num_examples: 250
  - name: ru
    num_bytes: 57837
    num_examples: 250
  - name: zh
    num_bytes: 58165
    num_examples: 250
  - name: ja
    num_bytes: 58423
    num_examples: 250
  - name: th
    num_bytes: 51044
    num_examples: 250
  - name: sw
    num_bytes: 58507
    num_examples: 250
  - name: bn
    num_bytes: 53901
    num_examples: 250
  - name: te
    num_bytes: 51593
    num_examples: 250
  - name: train
    num_bytes: 2682
    num_examples: 8
  download_size: 545702
  dataset_size: 572724
- config_name: nllb-200-3.3B
  features:
  - name: question
    dtype: string
  - name: answer
    dtype: string
  - name: answer_number
    dtype: int32
  - name: equation_solution
    dtype: string
  splits:
  - name: es
    num_bytes: 62012
    num_examples: 250
  - name: fr
    num_bytes: 60219
    num_examples: 250
  - name: de
    num_bytes: 61821
    num_examples: 250
  - name: ru
    num_bytes: 58382
    num_examples: 250
  - name: zh
    num_bytes: 58931
    num_examples: 250
  - name: ja
    num_bytes: 58752
    num_examples: 250
  - name: th
    num_bytes: 57139
    num_examples: 250
  - name: sw
    num_bytes: 60391
    num_examples: 250
  - name: bn
    num_bytes: 55057
    num_examples: 250
  - name: te
    num_bytes: 54888
    num_examples: 250
  - name: train
    num_bytes: 2682
    num_examples: 8
  download_size: 563242
  dataset_size: 590274
- config_name: xglm-564M
  features:
  - name: question
    dtype: string
  - name: answer
    dtype: string
  - name: answer_number
    dtype: int32
  - name: equation_solution
    dtype: string
  splits:
  - name: es
    num_bytes: 42608
    num_examples: 250
  - name: fr
    num_bytes: 45691
    num_examples: 250
  - name: de
    num_bytes: 51470
    num_examples: 250
  - name: ru
    num_bytes: 60715
    num_examples: 250
  - name: zh
    num_bytes: 45629
    num_examples: 250
  - name: ja
    num_bytes: 43786
    num_examples: 250
  - name: th
    num_bytes: 35269
    num_examples: 250
  - name: sw
    num_bytes: 37892
    num_examples: 250
  - name: bn
    num_bytes: 51002
    num_examples: 250
  - name: te
    num_bytes: 98158
    num_examples: 250
  - name: train
    num_bytes: 2682
    num_examples: 8
  download_size: 487886
  dataset_size: 514902
- config_name: xglm-1.7B
  features:
  - name: question
    dtype: string
  - name: answer
    dtype: string
  - name: answer_number
    dtype: int32
  - name: equation_solution
    dtype: string
  splits:
  - name: es
    num_bytes: 59727
    num_examples: 250
  - name: fr
    num_bytes: 59811
    num_examples: 250
  - name: de
    num_bytes: 60222
    num_examples: 250
  - name: ru
    num_bytes: 58039
    num_examples: 250
  - name: zh
    num_bytes: 44307
    num_examples: 250
  - name: ja
    num_bytes: 40936
    num_examples: 250
  - name: th
    num_bytes: 44383
    num_examples: 250
  - name: sw
    num_bytes: 53708
    num_examples: 250
  - name: bn
    num_bytes: 76978
    num_examples: 250
  - name: te
    num_bytes: 56112
    num_examples: 250
  - name: train
    num_bytes: 2682
    num_examples: 8
  download_size: 529882
  dataset_size: 556905
- config_name: xglm-2.9B
  features:
  - name: question
    dtype: string
  - name: answer
    dtype: string
  - name: answer_number
    dtype: int32
  - name: equation_solution
    dtype: string
  splits:
  - name: es
    num_bytes: 60811
    num_examples: 250
  - name: fr
    num_bytes: 58777
    num_examples: 250
  - name: de
    num_bytes: 60297
    num_examples: 250
  - name: ru
    num_bytes: 58133
    num_examples: 250
  - name: zh
    num_bytes: 43453
    num_examples: 250
  - name: ja
    num_bytes: 48201
    num_examples: 250
  - name: th
    num_bytes: 39620
    num_examples: 250
  - name: sw
    num_bytes: 56296
    num_examples: 250
  - name: bn
    num_bytes: 50937
    num_examples: 250
  - name: te
    num_bytes: 46948
    num_examples: 250
  - name: train
    num_bytes: 2682
    num_examples: 8
  download_size: 499131
  dataset_size: 526155
- config_name: xglm-4.5B
  features:
  - name: question
    dtype: string
  - name: answer
    dtype: string
  - name: answer_number
    dtype: int32
  - name: equation_solution
    dtype: string
  splits:
  - name: es
    num_bytes: 68793
    num_examples: 250
  - name: fr
    num_bytes: 68088
    num_examples: 250
  - name: de
    num_bytes: 76522
    num_examples: 250
  - name: ru
    num_bytes: 63439
    num_examples: 250
  - name: zh
    num_bytes: 58577
    num_examples: 250
  - name: ja
    num_bytes: 56872
    num_examples: 250
  - name: th
    num_bytes: 58692
    num_examples: 250
  - name: sw
    num_bytes: 72348
    num_examples: 250
  - name: bn
    num_bytes: 63835
    num_examples: 250
  - name: te
    num_bytes: 58979
    num_examples: 250
  - name: train
    num_bytes: 2682
    num_examples: 8
  download_size: 621817
  dataset_size: 648827
- config_name: xglm-7.5B
  features:
  - name: question
    dtype: string
  - name: answer
    dtype: string
  - name: answer_number
    dtype: int32
  - name: equation_solution
    dtype: string
  splits:
  - name: es
    num_bytes: 56510
    num_examples: 250
  - name: fr
    num_bytes: 56170
    num_examples: 250
  - name: de
    num_bytes: 56587
    num_examples: 250
  - name: ru
    num_bytes: 55870
    num_examples: 250
  - name: zh
    num_bytes: 53385
    num_examples: 250
  - name: ja
    num_bytes: 51831
    num_examples: 250
  - name: th
    num_bytes: 49858
    num_examples: 250
  - name: sw
    num_bytes: 55484
    num_examples: 250
  - name: bn
    num_bytes: 51975
    num_examples: 250
  - name: te
    num_bytes: 51737
    num_examples: 250
  - name: train
    num_bytes: 2682
    num_examples: 8
  download_size: 515073
  dataset_size: 542089
- config_name: bloom-560m
  features:
  - name: question
    dtype: string
  - name: answer
    dtype: string
  - name: answer_number
    dtype: int32
  - name: equation_solution
    dtype: string
  splits:
  - name: es
    num_bytes: 47987
    num_examples: 250
  - name: fr
    num_bytes: 43992
    num_examples: 250
  - name: de
    num_bytes: 56995
    num_examples: 250
  - name: ru
    num_bytes: 72240
    num_examples: 250
  - name: zh
    num_bytes: 61450
    num_examples: 250
  - name: ja
    num_bytes: 73445
    num_examples: 250
  - name: th
    num_bytes: 180123
    num_examples: 250
  - name: sw
    num_bytes: 50369
    num_examples: 250
  - name: bn
    num_bytes: 86465
    num_examples: 250
  - name: te
    num_bytes: 75244
    num_examples: 250
  - name: train
    num_bytes: 2682
    num_examples: 8
  download_size: 724012
  dataset_size: 750992
- config_name: bloom-1b1
  features:
  - name: question
    dtype: string
  - name: answer
    dtype: string
  - name: answer_number
    dtype: int32
  - name: equation_solution
    dtype: string
  splits:
  - name: es
    num_bytes: 56625
    num_examples: 250
  - name: fr
    num_bytes: 53998
    num_examples: 250
  - name: de
    num_bytes: 56874
    num_examples: 250
  - name: ru
    num_bytes: 32323
    num_examples: 250
  - name: zh
    num_bytes: 50902
    num_examples: 250
  - name: ja
    num_bytes: 38347
    num_examples: 250
  - name: th
    num_bytes: 20754
    num_examples: 250
  - name: sw
    num_bytes: 27779
    num_examples: 250
  - name: bn
    num_bytes: 34663
    num_examples: 250
  - name: te
    num_bytes: 24958
    num_examples: 250
  - name: train
    num_bytes: 2682
    num_examples: 8
  download_size: 372897
  dataset_size: 399905
- config_name: bloom-1b7
  features:
  - name: question
    dtype: string
  - name: answer
    dtype: string
  - name: answer_number
    dtype: int32
  - name: equation_solution
    dtype: string
  splits:
  - name: es
    num_bytes: 44595
    num_examples: 250
  - name: fr
    num_bytes: 48809
    num_examples: 250
  - name: de
    num_bytes: 57435
    num_examples: 250
  - name: ru
    num_bytes: 45954
    num_examples: 250
  - name: zh
    num_bytes: 47375
    num_examples: 250
  - name: ja
    num_bytes: 51493
    num_examples: 250
  - name: th
    num_bytes: 24154
    num_examples: 250
  - name: sw
    num_bytes: 41557
    num_examples: 250
  - name: bn
    num_bytes: 37503
    num_examples: 250
  - name: te
    num_bytes: 42682
    num_examples: 250
  - name: train
    num_bytes: 2682
    num_examples: 8
  download_size: 417273
  dataset_size: 444239
- config_name: bloom-3b
  features:
  - name: question
    dtype: string
  - name: answer
    dtype: string
  - name: answer_number
    dtype: int32
  - name: equation_solution
    dtype: string
  splits:
  - name: es
    num_bytes: 60956
    num_examples: 250
  - name: fr
    num_bytes: 61243
    num_examples: 250
  - name: de
    num_bytes: 60337
    num_examples: 250
  - name: ru
    num_bytes: 61329
    num_examples: 250
  - name: zh
    num_bytes: 57078
    num_examples: 250
  - name: ja
    num_bytes: 64180
    num_examples: 250
  - name: th
    num_bytes: 24167
    num_examples: 250
  - name: sw
    num_bytes: 45735
    num_examples: 250
  - name: bn
    num_bytes: 45720
    num_examples: 250
  - name: te
    num_bytes: 40840
    num_examples: 250
  - name: train
    num_bytes: 2682
    num_examples: 8
  download_size: 497369
  dataset_size: 524267
- config_name: bloom-7b1
  features:
  - name: question
    dtype: string
  - name: answer
    dtype: string
  - name: answer_number
    dtype: int32
  - name: equation_solution
    dtype: string
  splits:
  - name: es
    num_bytes: 63425
    num_examples: 250
  - name: fr
    num_bytes: 61340
    num_examples: 250
  - name: de
    num_bytes: 61858
    num_examples: 250
  - name: ru
    num_bytes: 60070
    num_examples: 250
  - name: zh
    num_bytes: 59410
    num_examples: 250
  - name: ja
    num_bytes: 57485
    num_examples: 250
  - name: th
    num_bytes: 24974
    num_examples: 250
  - name: sw
    num_bytes: 58232
    num_examples: 250
  - name: bn
    num_bytes: 57178
    num_examples: 250
  - name: te
    num_bytes: 57703
    num_examples: 250
  - name: train
    num_bytes: 2682
    num_examples: 8
  download_size: 537348
  dataset_size: 564357
- config_name: llama-7B
  features:
  - name: question
    dtype: string
  - name: answer
    dtype: string
  - name: answer_number
    dtype: int32
  - name: equation_solution
    dtype: string
  splits:
  - name: es
    num_bytes: 55313
    num_examples: 250
  - name: fr
    num_bytes: 61302
    num_examples: 250
  - name: de
    num_bytes: 62152
    num_examples: 250
  - name: ru
    num_bytes: 60929
    num_examples: 250
  - name: zh
    num_bytes: 59157
    num_examples: 250
  - name: ja
    num_bytes: 57356
    num_examples: 250
  - name: th
    num_bytes: 41148
    num_examples: 250
  - name: sw
    num_bytes: 56414
    num_examples: 250
  - name: bn
    num_bytes: 52156
    num_examples: 250
  - name: te
    num_bytes: 7360
    num_examples: 250
  - name: train
    num_bytes: 2682
    num_examples: 8
  download_size: 488983
  dataset_size: 515969
- config_name: llama-13B
  features:
  - name: question
    dtype: string
  - name: answer
    dtype: string
  - name: answer_number
    dtype: int32
  - name: equation_solution
    dtype: string
  splits:
  - name: es
    num_bytes: 62592
    num_examples: 250
  - name: fr
    num_bytes: 61965
    num_examples: 250
  - name: de
    num_bytes: 62148
    num_examples: 250
  - name: ru
    num_bytes: 61099
    num_examples: 250
  - name: zh
    num_bytes: 59858
    num_examples: 250
  - name: ja
    num_bytes: 55759
    num_examples: 250
  - name: th
    num_bytes: 51280
    num_examples: 250
  - name: sw
    num_bytes: 56081
    num_examples: 250
  - name: bn
    num_bytes: 48204
    num_examples: 250
  - name: te
    num_bytes: 6128
    num_examples: 250
  - name: train
    num_bytes: 2682
    num_examples: 8
  download_size: 500978
  dataset_size: 527796
- config_name: llama-30B
  features:
  - name: question
    dtype: string
  - name: answer
    dtype: string
  - name: answer_number
    dtype: int32
  - name: equation_solution
    dtype: string
  splits:
  - name: es
    num_bytes: 36577
    num_examples: 250
  - name: fr
    num_bytes: 50763
    num_examples: 250
  - name: de
    num_bytes: 63141
    num_examples: 250
  - name: ru
    num_bytes: 58198
    num_examples: 250
  - name: zh
    num_bytes: 61880
    num_examples: 250
  - name: ja
    num_bytes: 55989
    num_examples: 250
  - name: th
    num_bytes: 53253
    num_examples: 250
  - name: sw
    num_bytes: 59724
    num_examples: 250
  - name: bn
    num_bytes: 51345
    num_examples: 250
  - name: te
    num_bytes: 6546
    num_examples: 250
  - name: train
    num_bytes: 2682
    num_examples: 8
  download_size: 473194
  dataset_size: 500098
- config_name: RedPajama-INCITE-Base-3B-v1
  features:
  - name: question
    dtype: string
  - name: answer
    dtype: string
  - name: answer_number
    dtype: int32
  - name: equation_solution
    dtype: string
  splits:
  - name: es
    num_bytes: 61548
    num_examples: 250
  - name: fr
    num_bytes: 61357
    num_examples: 250
  - name: de
    num_bytes: 58325
    num_examples: 250
  - name: ru
    num_bytes: 61655
    num_examples: 250
  - name: zh
    num_bytes: 61669
    num_examples: 250
  - name: ja
    num_bytes: 59500
    num_examples: 250
  - name: th
    num_bytes: 31415
    num_examples: 250
  - name: sw
    num_bytes: 72056
    num_examples: 250
  - name: bn
    num_bytes: 26241
    num_examples: 250
  - name: te
    num_bytes: 26116
    num_examples: 250
  - name: train
    num_bytes: 2682
    num_examples: 8
  download_size: 495561
  dataset_size: 522564
- config_name: RedPajama-INCITE-7B-Base
  features:
  - name: question
    dtype: string
  - name: answer
    dtype: string
  - name: answer_number
    dtype: int32
  - name: equation_solution
    dtype: string
  splits:
  - name: es
    num_bytes: 63198
    num_examples: 250
  - name: fr
    num_bytes: 61124
    num_examples: 250
  - name: de
    num_bytes: 60728
    num_examples: 250
  - name: ru
    num_bytes: 60378
    num_examples: 250
  - name: zh
    num_bytes: 50030
    num_examples: 250
  - name: ja
    num_bytes: 57939
    num_examples: 250
  - name: th
    num_bytes: 25615
    num_examples: 250
  - name: sw
    num_bytes: 60635
    num_examples: 250
  - name: bn
    num_bytes: 18704
    num_examples: 250
  - name: te
    num_bytes: 21116
    num_examples: 250
  - name: train
    num_bytes: 2682
    num_examples: 8
  download_size: 455157
  dataset_size: 482149
- config_name: open_llama_3b
  features:
  - name: question
    dtype: string
  - name: answer
    dtype: string
  - name: answer_number
    dtype: int32
  - name: equation_solution
    dtype: string
  splits:
  - name: es
    num_bytes: 59734
    num_examples: 250
  - name: fr
    num_bytes: 59925
    num_examples: 250
  - name: de
    num_bytes: 60270
    num_examples: 250
  - name: ru
    num_bytes: 62725
    num_examples: 250
  - name: zh
    num_bytes: 34013
    num_examples: 250
  - name: ja
    num_bytes: 28163
    num_examples: 250
  - name: th
    num_bytes: 13190
    num_examples: 250
  - name: sw
    num_bytes: 46125
    num_examples: 250
  - name: bn
    num_bytes: 5721
    num_examples: 250
  - name: te
    num_bytes: 5605
    num_examples: 250
  - name: train
    num_bytes: 2682
    num_examples: 8
  download_size: 351125
  dataset_size: 378153
- config_name: open_llama_7b
  features:
  - name: question
    dtype: string
  - name: answer
    dtype: string
  - name: answer_number
    dtype: int32
  - name: equation_solution
    dtype: string
  splits:
  - name: es
    num_bytes: 61962
    num_examples: 250
  - name: fr
    num_bytes: 60687
    num_examples: 250
  - name: de
    num_bytes: 60474
    num_examples: 250
  - name: ru
    num_bytes: 61525
    num_examples: 250
  - name: zh
    num_bytes: 36631
    num_examples: 250
  - name: ja
    num_bytes: 29926
    num_examples: 250
  - name: th
    num_bytes: 11176
    num_examples: 250
  - name: sw
    num_bytes: 61601
    num_examples: 250
  - name: bn
    num_bytes: 5080
    num_examples: 250
  - name: te
    num_bytes: 5899
    num_examples: 250
  - name: train
    num_bytes: 2682
    num_examples: 8
  download_size: 370615
  dataset_size: 397643
- config_name: open_llama_13b
  features:
  - name: question
    dtype: string
  - name: answer
    dtype: string
  - name: answer_number
    dtype: int32
  - name: equation_solution
    dtype: string
  splits:
  - name: es
    num_bytes: 63245
    num_examples: 250
  - name: fr
    num_bytes: 61569
    num_examples: 250
  - name: de
    num_bytes: 62071
    num_examples: 250
  - name: ru
    num_bytes: 60086
    num_examples: 250
  - name: zh
    num_bytes: 37475
    num_examples: 250
  - name: ja
    num_bytes: 32072
    num_examples: 250
  - name: th
    num_bytes: 12902
    num_examples: 250
  - name: sw
    num_bytes: 58870
    num_examples: 250
  - name: bn
    num_bytes: 5624
    num_examples: 250
  - name: te
    num_bytes: 5647
    num_examples: 250
  - name: train
    num_bytes: 2682
    num_examples: 8
  download_size: 375230
  dataset_size: 402243
- config_name: open_llama_7b_v2
  features:
  - name: question
    dtype: string
  - name: answer
    dtype: string
  - name: answer_number
    dtype: int32
  - name: equation_solution
    dtype: string
  splits:
  - name: es
    num_bytes: 62306
    num_examples: 250
  - name: fr
    num_bytes: 61168
    num_examples: 250
  - name: de
    num_bytes: 60439
    num_examples: 250
  - name: ru
    num_bytes: 60916
    num_examples: 250
  - name: zh
    num_bytes: 57891
    num_examples: 250
  - name: ja
    num_bytes: 53155
    num_examples: 250
  - name: th
    num_bytes: 34743
    num_examples: 250
  - name: sw
    num_bytes: 58901
    num_examples: 250
  - name: bn
    num_bytes: 34548
    num_examples: 250
  - name: te
    num_bytes: 5253
    num_examples: 250
  - name: train
    num_bytes: 2682
    num_examples: 8
  download_size: 464986
  dataset_size: 492002
- config_name: falcon-7b
  features:
  - name: question
    dtype: string
  - name: answer
    dtype: string
  - name: answer_number
    dtype: int32
  - name: equation_solution
    dtype: string
  splits:
  - name: es
    num_bytes: 46760
    num_examples: 250
  - name: fr
    num_bytes: 33877
    num_examples: 250
  - name: de
    num_bytes: 51277
    num_examples: 250
  - name: ru
    num_bytes: 59591
    num_examples: 250
  - name: zh
    num_bytes: 37624
    num_examples: 250
  - name: ja
    num_bytes: 46601
    num_examples: 250
  - name: th
    num_bytes: 37107
    num_examples: 250
  - name: sw
    num_bytes: 31857
    num_examples: 250
  - name: bn
    num_bytes: 18472
    num_examples: 250
  - name: te
    num_bytes: 18376
    num_examples: 250
  - name: train
    num_bytes: 2682
    num_examples: 8
  download_size: 357224
  dataset_size: 384224
- config_name: xgen-7b-4k-base
  features:
  - name: question
    dtype: string
  - name: answer
    dtype: string
  - name: answer_number
    dtype: int32
  - name: equation_solution
    dtype: string
  splits:
  - name: es
    num_bytes: 63837
    num_examples: 250
  - name: fr
    num_bytes: 62076
    num_examples: 250
  - name: de
    num_bytes: 62146
    num_examples: 250
  - name: ru
    num_bytes: 61401
    num_examples: 250
  - name: zh
    num_bytes: 60295
    num_examples: 250
  - name: ja
    num_bytes: 57008
    num_examples: 250
  - name: th
    num_bytes: 18524
    num_examples: 250
  - name: sw
    num_bytes: 56158
    num_examples: 250
  - name: bn
    num_bytes: 25948
    num_examples: 250
  - name: te
    num_bytes: 5803
    num_examples: 250
  - name: train
    num_bytes: 2682
    num_examples: 8
  download_size: 448853
  dataset_size: 475878
- config_name: xgen-7b-8k-base
  features:
  - name: question
    dtype: string
  - name: answer
    dtype: string
  - name: answer_number
    dtype: int32
  - name: equation_solution
    dtype: string
  splits:
  - name: es
    num_bytes: 63243
    num_examples: 250
  - name: fr
    num_bytes: 60948
    num_examples: 250
  - name: de
    num_bytes: 61832
    num_examples: 250
  - name: ru
    num_bytes: 59217
    num_examples: 250
  - name: zh
    num_bytes: 60354
    num_examples: 250
  - name: ja
    num_bytes: 57012
    num_examples: 250
  - name: th
    num_bytes: 28194
    num_examples: 250
  - name: sw
    num_bytes: 56686
    num_examples: 250
  - name: bn
    num_bytes: 27221
    num_examples: 250
  - name: te
    num_bytes: 5460
    num_examples: 250
  - name: train
    num_bytes: 2682
    num_examples: 8
  download_size: 455836
  dataset_size: 482849
- config_name: xgen-7b-8k-inst
  features:
  - name: question
    dtype: string
  - name: answer
    dtype: string
  - name: answer_number
    dtype: int32
  - name: equation_solution
    dtype: string
  splits:
  - name: es
    num_bytes: 63113
    num_examples: 250
  - name: fr
    num_bytes: 60264
    num_examples: 250
  - name: de
    num_bytes: 59762
    num_examples: 250
  - name: ru
    num_bytes: 59374
    num_examples: 250
  - name: zh
    num_bytes: 62900
    num_examples: 250
  - name: ja
    num_bytes: 60877
    num_examples: 250
  - name: th
    num_bytes: 26089
    num_examples: 250
  - name: sw
    num_bytes: 57640
    num_examples: 250
  - name: bn
    num_bytes: 24301
    num_examples: 250
  - name: te
    num_bytes: 5290
    num_examples: 250
  - name: train
    num_bytes: 2682
    num_examples: 8
  download_size: 455320
  dataset_size: 482292
- config_name: polylm-1.7b
  features:
  - name: question
    dtype: string
  - name: answer
    dtype: string
  - name: answer_number
    dtype: int32
  - name: equation_solution
    dtype: string
  splits:
  - name: es
    num_bytes: 55706
    num_examples: 250
  - name: fr
    num_bytes: 55751
    num_examples: 250
  - name: de
    num_bytes: 54071
    num_examples: 250
  - name: ru
    num_bytes: 37159
    num_examples: 250
  - name: zh
    num_bytes: 47577
    num_examples: 250
  - name: ja
    num_bytes: 38931
    num_examples: 250
  - name: th
    num_bytes: 40203
    num_examples: 250
  - name: sw
    num_bytes: 20814
    num_examples: 250
  - name: bn
    num_bytes: 24317
    num_examples: 250
  - name: te
    num_bytes: 7420
    num_examples: 250
  - name: train
    num_bytes: 2682
    num_examples: 8
  download_size: 357603
  dataset_size: 384631
- config_name: polylm-13b
  features:
  - name: question
    dtype: string
  - name: answer
    dtype: string
  - name: answer_number
    dtype: int32
  - name: equation_solution
    dtype: string
  splits:
  - name: es
    num_bytes: 63444
    num_examples: 250
  - name: fr
    num_bytes: 62136
    num_examples: 250
  - name: de
    num_bytes: 63002
    num_examples: 250
  - name: ru
    num_bytes: 62522
    num_examples: 250
  - name: zh
    num_bytes: 59722
    num_examples: 250
  - name: ja
    num_bytes: 55541
    num_examples: 250
  - name: th
    num_bytes: 57684
    num_examples: 250
  - name: sw
    num_bytes: 46889
    num_examples: 250
  - name: bn
    num_bytes: 28704
    num_examples: 250
  - name: te
    num_bytes: 7883
    num_examples: 250
  - name: train
    num_bytes: 2682
    num_examples: 8
  download_size: 483392
  dataset_size: 510209
- config_name: polylm-multialpaca-13b
  features:
  - name: question
    dtype: string
  - name: answer
    dtype: string
  - name: answer_number
    dtype: int32
  - name: equation_solution
    dtype: string
  splits:
  - name: es
    num_bytes: 62502
    num_examples: 250
  - name: fr
    num_bytes: 60978
    num_examples: 250
  - name: de
    num_bytes: 62310
    num_examples: 250
  - name: ru
    num_bytes: 60440
    num_examples: 250
  - name: zh
    num_bytes: 57642
    num_examples: 250
  - name: ja
    num_bytes: 55315
    num_examples: 250
  - name: th
    num_bytes: 59002
    num_examples: 250
  - name: sw
    num_bytes: 51728
    num_examples: 250
  - name: bn
    num_bytes: 31947
    num_examples: 250
  - name: te
    num_bytes: 12891
    num_examples: 250
  - name: train
    num_bytes: 2682
    num_examples: 8
  download_size: 490498
  dataset_size: 517437
- config_name: open_llama_3b_v2
  features:
  - name: question
    dtype: string
  - name: answer
    dtype: string
  - name: answer_number
    dtype: int32
  - name: equation_solution
    dtype: string
  splits:
  - name: es
    num_bytes: 62474
    num_examples: 250
  - name: fr
    num_bytes: 60493
    num_examples: 250
  - name: de
    num_bytes: 59760
    num_examples: 250
  - name: ru
    num_bytes: 57592
    num_examples: 250
  - name: zh
    num_bytes: 54634
    num_examples: 250
  - name: ja
    num_bytes: 53936
    num_examples: 250
  - name: th
    num_bytes: 38960
    num_examples: 250
  - name: sw
    num_bytes: 57320
    num_examples: 250
  - name: bn
    num_bytes: 27394
    num_examples: 250
  - name: te
    num_bytes: 4680
    num_examples: 250
  - name: train
    num_bytes: 2682
    num_examples: 8
  download_size: 452910
  dataset_size: 479925
- config_name: Llama-2-7b-hf
  features:
  - name: question
    dtype: string
  - name: answer
    dtype: string
  - name: answer_number
    dtype: int32
  - name: equation_solution
    dtype: string
  splits:
  - name: es
    num_bytes: 63035
    num_examples: 250
  - name: fr
    num_bytes: 61128
    num_examples: 250
  - name: de
    num_bytes: 61496
    num_examples: 250
  - name: ru
    num_bytes: 59918
    num_examples: 250
  - name: zh
    num_bytes: 59415
    num_examples: 250
  - name: ja
    num_bytes: 54466
    num_examples: 250
  - name: th
    num_bytes: 37269
    num_examples: 250
  - name: sw
    num_bytes: 53461
    num_examples: 250
  - name: bn
    num_bytes: 42955
    num_examples: 250
  - name: te
    num_bytes: 7122
    num_examples: 250
  - name: train
    num_bytes: 2682
    num_examples: 8
  download_size: 475925
  dataset_size: 502947
- config_name: Llama-2-13b-hf
  features:
  - name: question
    dtype: string
  - name: answer
    dtype: string
  - name: answer_number
    dtype: int32
  - name: equation_solution
    dtype: string
  splits:
  - name: es
    num_bytes: 63347
    num_examples: 250
  - name: fr
    num_bytes: 62187
    num_examples: 250
  - name: de
    num_bytes: 63309
    num_examples: 250
  - name: ru
    num_bytes: 62772
    num_examples: 250
  - name: zh
    num_bytes: 62210
    num_examples: 250
  - name: ja
    num_bytes: 59083
    num_examples: 250
  - name: th
    num_bytes: 57690
    num_examples: 250
  - name: sw
    num_bytes: 57538
    num_examples: 250
  - name: bn
    num_bytes: 54947
    num_examples: 250
  - name: te
    num_bytes: 7062
    num_examples: 250
  - name: train
    num_bytes: 2682
    num_examples: 8
  download_size: 525803
  dataset_size: 552827
- config_name: Llama-2-7b-chat-hf
  features:
  - name: question
    dtype: string
  - name: answer
    dtype: string
  - name: answer_number
    dtype: int32
  - name: equation_solution
    dtype: string
  splits:
  - name: es
    num_bytes: 58203
    num_examples: 250
  - name: fr
    num_bytes: 40149
    num_examples: 250
  - name: de
    num_bytes: 57587
    num_examples: 250
  - name: ru
    num_bytes: 47777
    num_examples: 250
  - name: zh
    num_bytes: 50018
    num_examples: 250
  - name: ja
    num_bytes: 54107
    num_examples: 250
  - name: th
    num_bytes: 41549
    num_examples: 250
  - name: sw
    num_bytes: 61414
    num_examples: 250
  - name: bn
    num_bytes: 37996
    num_examples: 250
  - name: te
    num_bytes: 10156
    num_examples: 250
  - name: train
    num_bytes: 2682
    num_examples: 8
  download_size: 434632
  dataset_size: 461638
- config_name: Llama-2-13b-chat-hf
  features:
  - name: question
    dtype: string
  - name: answer
    dtype: string
  - name: answer_number
    dtype: int32
  - name: equation_solution
    dtype: string
  splits:
  - name: es
    num_bytes: 63304
    num_examples: 250
  - name: fr
    num_bytes: 61708
    num_examples: 250
  - name: de
    num_bytes: 63291
    num_examples: 250
  - name: ru
    num_bytes: 62305
    num_examples: 250
  - name: zh
    num_bytes: 61994
    num_examples: 250
  - name: ja
    num_bytes: 58226
    num_examples: 250
  - name: th
    num_bytes: 60256
    num_examples: 250
  - name: sw
    num_bytes: 58108
    num_examples: 250
  - name: bn
    num_bytes: 55180
    num_examples: 250
  - name: te
    num_bytes: 6525
    num_examples: 250
  - name: train
    num_bytes: 2682
    num_examples: 8
  download_size: 526574
  dataset_size: 553579
---

# Dataset Card for MGSM MT

## Table of Contents
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Supported Tasks](#supported-tasks-and-leaderboards)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-instances)
  - [Data Splits](#data-instances)
- [Dataset Creation](#dataset-creation)
  - [Curation Rationale](#curation-rationale)
  - [Source Data](#source-data)
  - [Annotations](#annotations)
  - [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
  - [Social Impact of Dataset](#social-impact-of-dataset)
  - [Discussion of Biases](#discussion-of-biases)
  - [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
  - [Dataset Curators](#dataset-curators)
  - [Licensing Information](#licensing-information)
  - [Citation Information](#citation-information)

## Dataset Description

- **Homepage:** https://openai.com/blog/grade-school-math/
- **Repository:** https://github.com/openai/grade-school-math
- **Paper:** https://arxiv.org/abs/2110.14168
- **Leaderboard:** [Needs More Information]
- **Point of Contact:** [Needs More Information]

### Dataset Summary

Multilingual Grade School Math Benchmark (MGSM) is a benchmark of grade-school math problems, proposed in the paper [Language models are multilingual chain-of-thought reasoners](http://arxiv.org/abs/2210.03057). This dataset is the machine-translated version of MGSM in English from each language.

The same 250 problems from [GSM8K](https://arxiv.org/abs/2110.14168) are each translated via human annotators in 10 languages. The 10 languages are:
- Spanish
- French
- German
- Russian
- Chinese
- Japanese
- Thai
- Swahili
- Bengali
- Telugu

GSM8K (Grade School Math 8K) is a dataset of 8.5K high quality linguistically diverse grade school math word problems. The dataset was created to support the task of question answering on basic mathematical problems that require multi-step reasoning.

You can find the input and targets for each of the ten languages (and English) as `.tsv` files.
We also include few-shot exemplars that are also manually translated from each language in `exemplars.py`.

### Supported Tasks and Leaderboards

[Needs More Information]

### Languages

The same 250 problems from [GSM8K](https://arxiv.org/abs/2110.14168) are each translated via human annotators in 10 languages. The 10 languages are:
- Spanish
- French
- German
- Russian
- Chinese
- Japanese
- Thai
- Swahili
- Bengali
- Telugu

This dataset is the machine-translated version of MGSM in English from each language.

## Dataset Structure

### Data Instances

Each instance in the train split contains:
- a string for the grade-school level math question
- a string for the corresponding answer with chain-of-thought steps.
- the numeric solution to the question
- the equation solution to the question

```python
{'question': 'Question: Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. How many tennis balls does he have now?',
 'answer': 'Step-by-Step Answer: Roger started with 5 balls. 2 cans of 3 tennis balls each is 6 tennis balls. 5 + 6 = 11. The answer is 11.',
 'answer_number': 11,
 'equation_solution': '5 + 6 = 11.'}
```

Each instance in the test split contains:
- a string for the grade-school level math question
- the numeric solution to the question

```python
{'question': "Janet’s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market?",
 'answer': None,
 'answer_number': 18,
 'equation_solution': None}
```

### Data Fields

The data fields are the same among `train` and `test` splits.

- question: The question string to a grade school math problem.

- answer: The full solution string to the `question`. It contains multiple steps of reasoning with calculator annotations and the final numeric solution.

- answer_number: The numeric solution to the `question`.

- equation_solution: The equation solution to the `question`.

### Data Splits

- The train split includes 8 few-shot exemplars that are also manually translated from each language.
- The test split includes the same 250 problems from GSM8K translated via human annotators in 10 languages.

| name   |train|test      |
|--------|----:|---------:|
|en      |  8  |   250    |
|es      |  8  |   250    |
|fr      |  8  |   250    |
|de      |  8  |   250    |
|ru      |  8  |   250    |
|zh      |  8  |   250    |
|ja      |  8  |   250    |
|th      |  8  |   250    |
|sw      |  8  |   250    |
|bn      |  8  |   250    |
|te      |  8  |   250    |

## Dataset Creation

### Curation Rationale

[Needs More Information]

### Source Data

#### Initial Data Collection and Normalization

From the paper:

> We initially collected a starting set of a thousand problems and natural language solutions by hiring freelance contractors on Upwork (upwork.com). We then worked with Surge AI (surgehq.ai), an NLP data labeling platform, to scale up our data collection. After collecting the full dataset, we asked workers to re-solve all problems, with no workers re-solving problems they originally wrote. We checked whether their final answers agreed with the original solu- tions, and any problems that produced disagreements were either repaired or discarded. We then performed another round of agreement checks on a smaller subset of problems, finding that 1.7% of problems still produce disagreements among contractors. We estimate this to be the fraction of problems that con- tain breaking errors or ambiguities. It is possible that a larger percentage of problems contain subtle errors.

#### Who are the source language producers?

[Needs More Information]

### Annotations

#### Annotation process

[Needs More Information]

#### Who are the annotators?

Surge AI (surgehq.ai)

### Personal and Sensitive Information

[Needs More Information]

## Considerations for Using the Data

### Social Impact of Dataset

[Needs More Information]

### Discussion of Biases

[Needs More Information]

### Other Known Limitations

[Needs More Information]

## Additional Information

### Dataset Curators

[Needs More Information]

### Licensing Information

The GSM8K dataset is licensed under the [MIT License](https://opensource.org/licenses/MIT).

### Citation Information

```bibtex
@article{cobbe2021gsm8k,
    title={Training Verifiers to Solve Math Word Problems},
    author={Cobbe, Karl and Kosaraju, Vineet and Bavarian, Mohammad and Chen, Mark and Jun, Heewoo and Kaiser, Lukasz and Plappert, Matthias and Tworek, Jerry and Hilton, Jacob and Nakano, Reiichiro and Hesse, Christopher and Schulman, John},
    journal={arXiv preprint arXiv:2110.14168},
    year={2021}
}
@misc{shi2022language,
    title={Language Models are Multilingual Chain-of-Thought Reasoners}, 
    author={Freda Shi and Mirac Suzgun and Markus Freitag and Xuezhi Wang and Suraj Srivats and Soroush Vosoughi and Hyung Won Chung and Yi Tay and Sebastian Ruder and Denny Zhou and Dipanjan Das and Jason Wei},
    year={2022},
    eprint={2210.03057},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}
```

### Contributions

Thanks to [@juletx](https://github.com/juletx) for adding this dataset.