task_categories:text-classification task_ids:semantic-similarity-classification task_ids:semantic-similarity-scoring task_ids:text-scoring task_ids:multi-input-text-classification annotations_creators:expert-generated annotations_creators:machine-generated language_creators:expert-generated language_creators:machine-generated multilinguality:multilingual size_categories:10K<n<100K source_datasets:extended|other-paws language:en license:other paraphrase-identification arxiv:1908.11828 region:us
PAWS-X, a multilingual version of PAWS (Paraphrase Adversaries from Word Scrambling) for six languages.

This dataset contains 23,659 human translated PAWS evaluation pairs and 296,406 machine
translated training pairs in six typologically distinct languages: French, Spanish, German,
Chinese, Japanese, and Korean. English language is available by default. All translated
pairs are sourced from examples in PAWS-Wiki.

For further details, see the accompanying paper: PAWS-X: A Cross-lingual Adversarial Dataset
for Paraphrase Identification (https://arxiv.org/abs/1908.11828)

NOTE: There might be some missing or wrong labels in the dataset and we have replaced them with -1.
@InProceedings{pawsx2019emnlp,
  title = {{PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification}},
  author = {Yang, Yinfei and Zhang, Yuan and Tar, Chris and Baldridge, Jason},
  booktitle = {Proc. of EMNLP},
  year = {2019}
}
---
annotations_creators:
- expert-generated
- machine-generated
language_creators:
- expert-generated
- machine-generated
language:
- en
license:
- other
multilinguality:
- multilingual
size_categories:
- 10K<n<100K
source_datasets:
- extended|other-paws
task_categories:
- text-classification
task_ids:
- semantic-similarity-classification
- semantic-similarity-scoring
- text-scoring
- multi-input-text-classification
paperswithcode_id: paws-x
pretty_name: 'PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification'
tags:
- paraphrase-identification
dataset_info:
- config_name: nllb-200-distilled-600M
  features:
  - name: id
    dtype: int32
  - name: sentence1
    dtype: string
  - name: sentence2
    dtype: string
  - name: label
    dtype:
      class_label:
        names:
          '0': '0'
          '1': '1'
  splits:
  - name: de
    num_bytes: 470424
    num_examples: 2000
  - name: es
    num_bytes: 477895
    num_examples: 2000
  - name: fr
    num_bytes: 478044
    num_examples: 2000
  - name: ja
    num_bytes: 461718
    num_examples: 2000
  - name: ko
    num_bytes: 467649
    num_examples: 2000
  - name: zh
    num_bytes: 481919
    num_examples: 2000
  download_size: 2704143
  dataset_size: 2837649
- config_name: nllb-200-distilled-1.3B
  features:
  - name: id
    dtype: int32
  - name: sentence1
    dtype: string
  - name: sentence2
    dtype: string
  - name: label
    dtype:
      class_label:
        names:
          '0': '0'
          '1': '1'
  splits:
  - name: de
    num_bytes: 469810
    num_examples: 2000
  - name: es
    num_bytes: 477848
    num_examples: 2000
  - name: fr
    num_bytes: 476036
    num_examples: 2000
  - name: ja
    num_bytes: 465219
    num_examples: 2000
  - name: ko
    num_bytes: 469779
    num_examples: 2000
  - name: zh
    num_bytes: 481685
    num_examples: 2000
  download_size: 2706871
  dataset_size: 2840377
- config_name: nllb-200-1.3B
  features:
  - name: id
    dtype: int32
  - name: sentence1
    dtype: string
  - name: sentence2
    dtype: string
  - name: label
    dtype:
      class_label:
        names:
          '0': '0'
          '1': '1'
  splits:
  - name: de
    num_bytes: 472562
    num_examples: 2000
  - name: es
    num_bytes: 480329
    num_examples: 2000
  - name: fr
    num_bytes: 479096
    num_examples: 2000
  - name: ja
    num_bytes: 465418
    num_examples: 2000
  - name: ko
    num_bytes: 468672
    num_examples: 2000
  - name: zh
    num_bytes: 480250
    num_examples: 2000
  download_size: 2712821
  dataset_size: 2846327
- config_name: nllb-200-3.3B
  features:
  - name: id
    dtype: int32
  - name: sentence1
    dtype: string
  - name: sentence2
    dtype: string
  - name: label
    dtype:
      class_label:
        names:
          '0': '0'
          '1': '1'
  splits:
  - name: de
    num_bytes: 475185
    num_examples: 2000
  - name: es
    num_bytes: 482022
    num_examples: 2000
  - name: fr
    num_bytes: 480477
    num_examples: 2000
  - name: ja
    num_bytes: 468442
    num_examples: 2000
  - name: ko
    num_bytes: 475577
    num_examples: 2000
  - name: zh
    num_bytes: 483772
    num_examples: 2000
  download_size: 2731969
  dataset_size: 2865475
- config_name: xglm-564M
  features:
  - name: id
    dtype: int32
  - name: sentence1
    dtype: string
  - name: sentence2
    dtype: string
  - name: label
    dtype:
      class_label:
        names:
          '0': '0'
          '1': '1'
  splits:
  - name: de
    num_bytes: 405887
    num_examples: 2000
  - name: es
    num_bytes: 433475
    num_examples: 2000
  - name: fr
    num_bytes: 451810
    num_examples: 2000
  - name: ja
    num_bytes: 480321
    num_examples: 2000
  - name: ko
    num_bytes: 430501
    num_examples: 2000
  - name: zh
    num_bytes: 536783
    num_examples: 2000
  download_size: 2605271
  dataset_size: 2738777
- config_name: xglm-1.7B
  features:
  - name: id
    dtype: int32
  - name: sentence1
    dtype: string
  - name: sentence2
    dtype: string
  - name: label
    dtype:
      class_label:
        names:
          '0': '0'
          '1': '1'
  splits:
  - name: de
    num_bytes: 448117
    num_examples: 2000
  - name: es
    num_bytes: 470068
    num_examples: 2000
  - name: fr
    num_bytes: 478245
    num_examples: 2000
  - name: ja
    num_bytes: 462409
    num_examples: 2000
  - name: ko
    num_bytes: 410803
    num_examples: 2000
  - name: zh
    num_bytes: 455754
    num_examples: 2000
  download_size: 2591890
  dataset_size: 2725396
- config_name: xglm-2.9B
  features:
  - name: id
    dtype: int32
  - name: sentence1
    dtype: string
  - name: sentence2
    dtype: string
  - name: label
    dtype:
      class_label:
        names:
          '0': '0'
          '1': '1'
  splits:
  - name: de
    num_bytes: 450076
    num_examples: 2000
  - name: es
    num_bytes: 471853
    num_examples: 2000
  - name: fr
    num_bytes: 475575
    num_examples: 2000
  - name: ja
    num_bytes: 435278
    num_examples: 2000
  - name: ko
    num_bytes: 407905
    num_examples: 2000
  - name: zh
    num_bytes: 437874
    num_examples: 2000
  download_size: 2545055
  dataset_size: 2678561
- config_name: xglm-4.5B
  features:
  - name: id
    dtype: int32
  - name: sentence1
    dtype: string
  - name: sentence2
    dtype: string
  - name: label
    dtype:
      class_label:
        names:
          '0': '0'
          '1': '1'
  splits:
  - name: de
    num_bytes: 466986
    num_examples: 2000
  - name: es
    num_bytes: 483691
    num_examples: 2000
  - name: fr
    num_bytes: 485910
    num_examples: 2000
  - name: ja
    num_bytes: 485014
    num_examples: 2000
  - name: ko
    num_bytes: 459562
    num_examples: 2000
  - name: zh
    num_bytes: 502672
    num_examples: 2000
  download_size: 2750329
  dataset_size: 2883835
- config_name: xglm-7.5B
  features:
  - name: id
    dtype: int32
  - name: sentence1
    dtype: string
  - name: sentence2
    dtype: string
  - name: label
    dtype:
      class_label:
        names:
          '0': '0'
          '1': '1'
  splits:
  - name: de
    num_bytes: 457033
    num_examples: 2000
  - name: es
    num_bytes: 471085
    num_examples: 2000
  - name: fr
    num_bytes: 474534
    num_examples: 2000
  - name: ja
    num_bytes: 455080
    num_examples: 2000
  - name: ko
    num_bytes: 432714
    num_examples: 2000
  - name: zh
    num_bytes: 462024
    num_examples: 2000
  download_size: 2618964
  dataset_size: 2752470
- config_name: bloom-560m
  features:
  - name: id
    dtype: int32
  - name: sentence1
    dtype: string
  - name: sentence2
    dtype: string
  - name: label
    dtype:
      class_label:
        names:
          '0': '0'
          '1': '1'
  splits:
  - name: de
    num_bytes: 422431
    num_examples: 2000
  - name: es
    num_bytes: 407925
    num_examples: 2000
  - name: fr
    num_bytes: 417238
    num_examples: 2000
  - name: ja
    num_bytes: 541097
    num_examples: 2000
  - name: ko
    num_bytes: 305526
    num_examples: 2000
  - name: zh
    num_bytes: 467990
    num_examples: 2000
  download_size: 2428701
  dataset_size: 2562207
- config_name: bloom-1b1
  features:
  - name: id
    dtype: int32
  - name: sentence1
    dtype: string
  - name: sentence2
    dtype: string
  - name: label
    dtype:
      class_label:
        names:
          '0': '0'
          '1': '1'
  splits:
  - name: de
    num_bytes: 420950
    num_examples: 2000
  - name: es
    num_bytes: 440695
    num_examples: 2000
  - name: fr
    num_bytes: 444933
    num_examples: 2000
  - name: ja
    num_bytes: 383160
    num_examples: 2000
  - name: ko
    num_bytes: 309106
    num_examples: 2000
  - name: zh
    num_bytes: 427093
    num_examples: 2000
  download_size: 2292431
  dataset_size: 2425937
- config_name: bloom-1b7
  features:
  - name: id
    dtype: int32
  - name: sentence1
    dtype: string
  - name: sentence2
    dtype: string
  - name: label
    dtype:
      class_label:
        names:
          '0': '0'
          '1': '1'
  splits:
  - name: de
    num_bytes: 441068
    num_examples: 2000
  - name: es
    num_bytes: 455189
    num_examples: 2000
  - name: fr
    num_bytes: 458970
    num_examples: 2000
  - name: ja
    num_bytes: 471554
    num_examples: 2000
  - name: ko
    num_bytes: 387729
    num_examples: 2000
  - name: zh
    num_bytes: 434684
    num_examples: 2000
  download_size: 2515688
  dataset_size: 2649194
- config_name: bloom-3b
  features:
  - name: id
    dtype: int32
  - name: sentence1
    dtype: string
  - name: sentence2
    dtype: string
  - name: label
    dtype:
      class_label:
        names:
          '0': '0'
          '1': '1'
  splits:
  - name: de
    num_bytes: 452342
    num_examples: 2000
  - name: es
    num_bytes: 468924
    num_examples: 2000
  - name: fr
    num_bytes: 469477
    num_examples: 2000
  - name: ja
    num_bytes: 450059
    num_examples: 2000
  - name: ko
    num_bytes: 371349
    num_examples: 2000
  - name: zh
    num_bytes: 443763
    num_examples: 2000
  download_size: 2522408
  dataset_size: 2655914
- config_name: bloom-7b1
  features:
  - name: id
    dtype: int32
  - name: sentence1
    dtype: string
  - name: sentence2
    dtype: string
  - name: label
    dtype:
      class_label:
        names:
          '0': '0'
          '1': '1'
  splits:
  - name: de
    num_bytes: 460868
    num_examples: 2000
  - name: es
    num_bytes: 476090
    num_examples: 2000
  - name: fr
    num_bytes: 477681
    num_examples: 2000
  - name: ja
    num_bytes: 462541
    num_examples: 2000
  - name: ko
    num_bytes: 410996
    num_examples: 2000
  - name: zh
    num_bytes: 452755
    num_examples: 2000
  download_size: 2607425
  dataset_size: 2740931
- config_name: llama-7B
  features:
  - name: id
    dtype: int32
  - name: sentence1
    dtype: string
  - name: sentence2
    dtype: string
  - name: label
    dtype:
      class_label:
        names:
          '0': '0'
          '1': '1'
  splits:
  - name: de
    num_bytes: 467040
    num_examples: 2000
  - name: es
    num_bytes: 479857
    num_examples: 2000
  - name: fr
    num_bytes: 481692
    num_examples: 2000
  - name: ja
    num_bytes: 469209
    num_examples: 2000
  - name: ko
    num_bytes: 460027
    num_examples: 2000
  - name: zh
    num_bytes: 492611
    num_examples: 2000
  download_size: 2716930
  dataset_size: 2850436
- config_name: llama-13B
  features:
  - name: id
    dtype: int32
  - name: sentence1
    dtype: string
  - name: sentence2
    dtype: string
  - name: label
    dtype:
      class_label:
        names:
          '0': '0'
          '1': '1'
  splits:
  - name: de
    num_bytes: 464622
    num_examples: 2000
  - name: es
    num_bytes: 475395
    num_examples: 2000
  - name: fr
    num_bytes: 475380
    num_examples: 2000
  - name: ja
    num_bytes: 455735
    num_examples: 2000
  - name: ko
    num_bytes: 446006
    num_examples: 2000
  - name: zh
    num_bytes: 477833
    num_examples: 2000
  download_size: 2661465
  dataset_size: 2794971
- config_name: llama-30B
  features:
  - name: id
    dtype: int32
  - name: sentence1
    dtype: string
  - name: sentence2
    dtype: string
  - name: label
    dtype:
      class_label:
        names:
          '0': '0'
          '1': '1'
  splits:
  - name: de
    num_bytes: 471142
    num_examples: 2000
  - name: es
    num_bytes: 480239
    num_examples: 2000
  - name: fr
    num_bytes: 480078
    num_examples: 2000
  - name: ja
    num_bytes: 473976
    num_examples: 2000
  - name: ko
    num_bytes: 468087
    num_examples: 2000
  - name: zh
    num_bytes: 498795
    num_examples: 2000
  download_size: 2738811
  dataset_size: 2872317
- config_name: RedPajama-INCITE-Base-3B-v1
  features:
  - name: id
    dtype: int32
  - name: sentence1
    dtype: string
  - name: sentence2
    dtype: string
  - name: label
    dtype:
      class_label:
        names:
          '0': '0'
          '1': '1'
  splits:
  - name: de
    num_bytes: 454468
    num_examples: 2000
  - name: es
    num_bytes: 474260
    num_examples: 2000
  - name: fr
    num_bytes: 477493
    num_examples: 2000
  - name: ja
    num_bytes: 463806
    num_examples: 2000
  - name: ko
    num_bytes: 455166
    num_examples: 2000
  - name: zh
    num_bytes: 520240
    num_examples: 2000
  download_size: 2711927
  dataset_size: 2845433
- config_name: RedPajama-INCITE-7B-Base
  features:
  - name: id
    dtype: int32
  - name: sentence1
    dtype: string
  - name: sentence2
    dtype: string
  - name: label
    dtype:
      class_label:
        names:
          '0': '0'
          '1': '1'
  splits:
  - name: de
    num_bytes: 467209
    num_examples: 2000
  - name: es
    num_bytes: 482675
    num_examples: 2000
  - name: fr
    num_bytes: 479674
    num_examples: 2000
  - name: ja
    num_bytes: 469695
    num_examples: 2000
  - name: ko
    num_bytes: 427807
    num_examples: 2000
  - name: zh
    num_bytes: 475045
    num_examples: 2000
  download_size: 2668599
  dataset_size: 2802105
- config_name: open_llama_3b
  features:
  - name: id
    dtype: int32
  - name: sentence1
    dtype: string
  - name: sentence2
    dtype: string
  - name: label
    dtype:
      class_label:
        names:
          '0': '0'
          '1': '1'
  splits:
  - name: de
    num_bytes: 459906
    num_examples: 2000
  - name: es
    num_bytes: 474097
    num_examples: 2000
  - name: fr
    num_bytes: 477589
    num_examples: 2000
  - name: ja
    num_bytes: 462664
    num_examples: 2000
  - name: ko
    num_bytes: 434739
    num_examples: 2000
  - name: zh
    num_bytes: 490475
    num_examples: 2000
  download_size: 2665964
  dataset_size: 2799470
- config_name: open_llama_7b
  features:
  - name: id
    dtype: int32
  - name: sentence1
    dtype: string
  - name: sentence2
    dtype: string
  - name: label
    dtype:
      class_label:
        names:
          '0': '0'
          '1': '1'
  splits:
  - name: de
    num_bytes: 464258
    num_examples: 2000
  - name: es
    num_bytes: 476895
    num_examples: 2000
  - name: fr
    num_bytes: 475470
    num_examples: 2000
  - name: ja
    num_bytes: 467530
    num_examples: 2000
  - name: ko
    num_bytes: 420696
    num_examples: 2000
  - name: zh
    num_bytes: 471007
    num_examples: 2000
  download_size: 2642350
  dataset_size: 2775856
- config_name: open_llama_13b
  features:
  - name: id
    dtype: int32
  - name: sentence1
    dtype: string
  - name: sentence2
    dtype: string
  - name: label
    dtype:
      class_label:
        names:
          '0': '0'
          '1': '1'
  splits:
  - name: de
    num_bytes: 466772
    num_examples: 2000
  - name: es
    num_bytes: 480354
    num_examples: 2000
  - name: fr
    num_bytes: 480221
    num_examples: 2000
  - name: ja
    num_bytes: 460154
    num_examples: 2000
  - name: ko
    num_bytes: 443434
    num_examples: 2000
  - name: zh
    num_bytes: 467898
    num_examples: 2000
  download_size: 2665327
  dataset_size: 2798833
- config_name: xgen-7b-4k-base
  features:
  - name: id
    dtype: int32
  - name: sentence1
    dtype: string
  - name: sentence2
    dtype: string
  - name: label
    dtype:
      class_label:
        names:
          '0': '0'
          '1': '1'
  splits:
  - name: de
    num_bytes: 466109
    num_examples: 2000
  - name: es
    num_bytes: 480599
    num_examples: 2000
  - name: fr
    num_bytes: 481774
    num_examples: 2000
  - name: ja
    num_bytes: 455601
    num_examples: 2000
  - name: ko
    num_bytes: 441720
    num_examples: 2000
  - name: zh
    num_bytes: 473661
    num_examples: 2000
  download_size: 2665958
  dataset_size: 2799464
- config_name: xgen-7b-8k-base
  features:
  - name: id
    dtype: int32
  - name: sentence1
    dtype: string
  - name: sentence2
    dtype: string
  - name: label
    dtype:
      class_label:
        names:
          '0': '0'
          '1': '1'
  splits:
  - name: de
    num_bytes: 464831
    num_examples: 2000
  - name: es
    num_bytes: 478903
    num_examples: 2000
  - name: fr
    num_bytes: 481199
    num_examples: 2000
  - name: ja
    num_bytes: 458928
    num_examples: 2000
  - name: ko
    num_bytes: 448148
    num_examples: 2000
  - name: zh
    num_bytes: 475878
    num_examples: 2000
  download_size: 2674381
  dataset_size: 2807887
- config_name: xgen-7b-8k-inst
  features:
  - name: id
    dtype: int32
  - name: sentence1
    dtype: string
  - name: sentence2
    dtype: string
  - name: label
    dtype:
      class_label:
        names:
          '0': '0'
          '1': '1'
  splits:
  - name: de
    num_bytes: 472749
    num_examples: 2000
  - name: es
    num_bytes: 483956
    num_examples: 2000
  - name: fr
    num_bytes: 487250
    num_examples: 2000
  - name: ja
    num_bytes: 485563
    num_examples: 2000
  - name: ko
    num_bytes: 476502
    num_examples: 2000
  - name: zh
    num_bytes: 507723
    num_examples: 2000
  download_size: 2780237
  dataset_size: 2913743
- config_name: open_llama_7b_v2
  features:
  - name: id
    dtype: int32
  - name: sentence1
    dtype: string
  - name: sentence2
    dtype: string
  - name: label
    dtype:
      class_label:
        names:
          '0': '0'
          '1': '1'
  splits:
  - name: de
    num_bytes: 464268
    num_examples: 2000
  - name: es
    num_bytes: 476576
    num_examples: 2000
  - name: fr
    num_bytes: 478153
    num_examples: 2000
  - name: ja
    num_bytes: 460932
    num_examples: 2000
  - name: ko
    num_bytes: 456955
    num_examples: 2000
  - name: zh
    num_bytes: 467587
    num_examples: 2000
  download_size: 2670965
  dataset_size: 2804471
- config_name: falcon-7b
  features:
  - name: id
    dtype: int32
  - name: sentence1
    dtype: string
  - name: sentence2
    dtype: string
  - name: label
    dtype:
      class_label:
        names:
          '0': '0'
          '1': '1'
  splits:
  - name: de
    num_bytes: 456304
    num_examples: 2000
  - name: es
    num_bytes: 474821
    num_examples: 2000
  - name: fr
    num_bytes: 448537
    num_examples: 2000
  - name: ja
    num_bytes: 373442
    num_examples: 2000
  - name: ko
    num_bytes: 425657
    num_examples: 2000
  - name: zh
    num_bytes: 449866
    num_examples: 2000
  download_size: 2495121
  dataset_size: 2628627
- config_name: polylm-1.7b
  features:
  - name: id
    dtype: int32
  - name: sentence1
    dtype: string
  - name: sentence2
    dtype: string
  - name: label
    dtype:
      class_label:
        names:
          '0': '0'
          '1': '1'
  splits:
  - name: de
    num_bytes: 459992
    num_examples: 2000
  - name: es
    num_bytes: 466048
    num_examples: 2000
  - name: fr
    num_bytes: 470826
    num_examples: 2000
  - name: ja
    num_bytes: 448180
    num_examples: 2000
  - name: ko
    num_bytes: 415816
    num_examples: 2000
  - name: zh
    num_bytes: 438679
    num_examples: 2000
  download_size: 2566035
  dataset_size: 2699541
- config_name: polylm-13b
  features:
  - name: id
    dtype: int32
  - name: sentence1
    dtype: string
  - name: sentence2
    dtype: string
  - name: label
    dtype:
      class_label:
        names:
          '0': '0'
          '1': '1'
  splits:
  - name: de
    num_bytes: 473536
    num_examples: 2000
  - name: es
    num_bytes: 482328
    num_examples: 2000
  - name: fr
    num_bytes: 481341
    num_examples: 2000
  - name: ja
    num_bytes: 452146
    num_examples: 2000
  - name: ko
    num_bytes: 457546
    num_examples: 2000
  - name: zh
    num_bytes: 464947
    num_examples: 2000
  download_size: 2678338
  dataset_size: 2811844
- config_name: polylm-multialpaca-13b
  features:
  - name: id
    dtype: int32
  - name: sentence1
    dtype: string
  - name: sentence2
    dtype: string
  - name: label
    dtype:
      class_label:
        names:
          '0': '0'
          '1': '1'
  splits:
  - name: de
    num_bytes: 472264
    num_examples: 2000
  - name: es
    num_bytes: 477291
    num_examples: 2000
  - name: fr
    num_bytes: 474987
    num_examples: 2000
  - name: ja
    num_bytes: 465751
    num_examples: 2000
  - name: ko
    num_bytes: 465889
    num_examples: 2000
  - name: zh
    num_bytes: 461985
    num_examples: 2000
  download_size: 2684661
  dataset_size: 2818167
- config_name: open_llama_3b_v2
  features:
  - name: id
    dtype: int32
  - name: sentence1
    dtype: string
  - name: sentence2
    dtype: string
  - name: label
    dtype:
      class_label:
        names:
          '0': '0'
          '1': '1'
  splits:
  - name: de
    num_bytes: 454405
    num_examples: 2000
  - name: es
    num_bytes: 475689
    num_examples: 2000
  - name: fr
    num_bytes: 476410
    num_examples: 2000
  - name: ja
    num_bytes: 447704
    num_examples: 2000
  - name: ko
    num_bytes: 435675
    num_examples: 2000
  - name: zh
    num_bytes: 466981
    num_examples: 2000
  download_size: 2623358
  dataset_size: 2756864
- config_name: Llama-2-7b-hf
  features:
  - name: id
    dtype: int32
  - name: sentence1
    dtype: string
  - name: sentence2
    dtype: string
  - name: label
    dtype:
      class_label:
        names:
          '0': '0'
          '1': '1'
  splits:
  - name: de
    num_bytes: 468952
    num_examples: 2000
  - name: es
    num_bytes: 481463
    num_examples: 2000
  - name: fr
    num_bytes: 481620
    num_examples: 2000
  - name: ja
    num_bytes: 452968
    num_examples: 2000
  - name: ko
    num_bytes: 448819
    num_examples: 2000
  - name: zh
    num_bytes: 476890
    num_examples: 2000
  download_size: 2677206
  dataset_size: 2810712
- config_name: Llama-2-13b-hf
  features:
  - name: id
    dtype: int32
  - name: sentence1
    dtype: string
  - name: sentence2
    dtype: string
  - name: label
    dtype:
      class_label:
        names:
          '0': '0'
          '1': '1'
  splits:
  - name: de
    num_bytes: 471040
    num_examples: 2000
  - name: es
    num_bytes: 480439
    num_examples: 2000
  - name: fr
    num_bytes: 479753
    num_examples: 2000
  - name: ja
    num_bytes: 457856
    num_examples: 2000
  - name: ko
    num_bytes: 459972
    num_examples: 2000
  - name: zh
    num_bytes: 478780
    num_examples: 2000
  download_size: 2694334
  dataset_size: 2827840
- config_name: Llama-2-7b-chat-hf
  features:
  - name: id
    dtype: int32
  - name: sentence1
    dtype: string
  - name: sentence2
    dtype: string
  - name: label
    dtype:
      class_label:
        names:
          '0': '0'
          '1': '1'
  splits:
  - name: de
    num_bytes: 429595
    num_examples: 2000
  - name: es
    num_bytes: 395137
    num_examples: 2000
  - name: fr
    num_bytes: 338615
    num_examples: 2000
  - name: ja
    num_bytes: 448313
    num_examples: 2000
  - name: ko
    num_bytes: 429424
    num_examples: 2000
  - name: zh
    num_bytes: 425094
    num_examples: 2000
  download_size: 2332672
  dataset_size: 2466178
- config_name: Llama-2-13b-chat-hf
  features:
  - name: id
    dtype: int32
  - name: sentence1
    dtype: string
  - name: sentence2
    dtype: string
  - name: label
    dtype:
      class_label:
        names:
          '0': '0'
          '1': '1'
  splits:
  - name: de
    num_bytes: 476183
    num_examples: 2000
  - name: es
    num_bytes: 481248
    num_examples: 2000
  - name: fr
    num_bytes: 480349
    num_examples: 2000
  - name: ja
    num_bytes: 475454
    num_examples: 2000
  - name: ko
    num_bytes: 482906
    num_examples: 2000
  - name: zh
    num_bytes: 492532
    num_examples: 2000
  download_size: 2755166
  dataset_size: 2888672
---

# Dataset Card for PAWS-X MT

## Table of Contents
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
  - [Curation Rationale](#curation-rationale)
  - [Source Data](#source-data)
  - [Annotations](#annotations)
  - [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
  - [Social Impact of Dataset](#social-impact-of-dataset)
  - [Discussion of Biases](#discussion-of-biases)
  - [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
  - [Dataset Curators](#dataset-curators)
  - [Licensing Information](#licensing-information)
  - [Citation Information](#citation-information)
  - [Contributions](#contributions)

## Dataset Description

- **Homepage:** [PAWS-X](https://github.com/google-research-datasets/paws/tree/master/pawsx)
- **Repository:** [PAWS-X](https://github.com/google-research-datasets/paws/tree/master/pawsx)
- **Paper:** [PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification](https://arxiv.org/abs/1908.11828)
- **Point of Contact:** [Yinfei Yang](yinfeiy@google.com)

### Dataset Summary

This dataset contains 23,659 **human** translated PAWS evaluation pairs and
296,406 **machine** translated training pairs in six typologically distinct
languages: French, Spanish, German, Chinese, Japanese, and Korean. All
translated pairs are sourced from examples in
[PAWS-Wiki](https://github.com/google-research-datasets/paws#paws-wiki).

For further details, see the accompanying paper:
[PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase
Identification](https://arxiv.org/abs/1908.11828)

This is a machine-translated version of the original dataset into English from each langauge.

### Supported Tasks and Leaderboards

It has been majorly used for paraphrase identification for English and other 6 languages namely French, Spanish, German, Chinese, Japanese, and Korean

### Languages

The dataset is in English, French, Spanish, German, Chinese, Japanese, and Korean

## Dataset Structure

### Data Instances

For en:
```
id		    :   1
sentence1	:	In Paris , in October 1560 , he secretly met the English ambassador , Nicolas Throckmorton , asking him for a passport to return to England through Scotland .
sentence2	:	In October 1560 , he secretly met with the English ambassador , Nicolas Throckmorton , in Paris , and asked him for a passport to return to Scotland through England .
label       :   0
```
For fr:
```
id		    :   1
sentence1	:	À Paris, en octobre 1560, il rencontra secrètement l'ambassadeur d'Angleterre, Nicolas Throckmorton, lui demandant un passeport pour retourner en Angleterre en passant par l'Écosse.
sentence2	:	En octobre 1560, il rencontra secrètement l'ambassadeur d'Angleterre, Nicolas Throckmorton, à Paris, et lui demanda un passeport pour retourner en Écosse par l'Angleterre.
label       :   0
```

### Data Fields

All files are in tsv format with four columns:

Column Name | Data
:---------- | :--------------------------------------------------------
id          | An ID that matches the ID of the source pair in PAWS-Wiki
sentence1   | The first sentence
sentence2   | The second sentence
label       | Label for each pair

The source text of each translation can be retrieved by looking up the ID in the
corresponding file in PAWS-Wiki.

### Data Splits

The numbers of examples for each of the seven languages are shown below:

Language | Train   | Dev    | Test
:------- | ------: | -----: | -----:
en       | 49,401  | 2,000  | 2,000
fr       | 49,401  | 2,000  | 2,000
es       | 49,401  | 2,000  | 2,000
de       | 49,401  | 2,000  | 2,000
zh       | 49,401  | 2,000  | 2,000
ja       | 49,401  | 2,000  | 2,000
ko       | 49,401  | 2,000  | 2,000


> **Caveat**: please note that the dev and test sets of PAWS-X are both sourced
> from the dev set of PAWS-Wiki. As a consequence, the same `sentence 1` may
> appear in both the dev and test sets. Nevertheless our data split guarantees
> that there is no overlap on sentence pairs (`sentence 1` + `sentence 2`)
> between dev and test.

## Dataset Creation

### Curation Rationale

Most existing work on adversarial data generation focuses on English. For example, PAWS (Paraphrase Adversaries from Word Scrambling) (Zhang et al., 2019) consists of challenging English paraphrase identification pairs from Wikipedia and Quora. They remedy this gap with PAWS-X, a new dataset of 23,659 human translated PAWS evaluation pairs in six typologically distinct languages: French, Spanish, German, Chinese, Japanese, and Korean. They provide baseline numbers for three models with different capacity to capture non-local context and sentence structure, and using different multilingual training and evaluation regimes. Multilingual BERT (Devlin et al., 2019) fine-tuned on PAWS English plus machine-translated data performs the best, with a range of 83.1-90.8 accuracy across the non-English languages and an average accuracy gain of 23% over the next best model. PAWS-X shows the effectiveness of deep, multilingual pre-training while also leaving considerable headroom as a new challenge to drive multilingual research that better captures structure and contextual information.

### Source Data

PAWS (Paraphrase Adversaries from Word Scrambling)

#### Initial Data Collection and Normalization

All translated pairs are sourced from examples in [PAWS-Wiki](https://github.com/google-research-datasets/paws#paws-wiki)

#### Who are the source language producers?

This dataset contains 23,659 human translated PAWS evaluation pairs and 296,406 machine translated training pairs in six typologically distinct languages: French, Spanish, German, Chinese, Japanese, and Korean.

### Annotations

#### Annotation process

If applicable, describe the annotation process and any tools used, or state otherwise. Describe the amount of data annotated, if not all. Describe or reference annotation guidelines provided to the annotators. If available, provide interannotator statistics. Describe any annotation validation processes.

#### Who are the annotators?

The paper mentions the translate team, especially Mengmeng Niu, for the help with the annotations.

### Personal and Sensitive Information

[More Information Needed]

## Considerations for Using the Data

### Social Impact of Dataset

[More Information Needed]

### Discussion of Biases

[More Information Needed]

### Other Known Limitations

[More Information Needed]

## Additional Information

### Dataset Curators

List the people involved in collecting the dataset and their affiliation(s). If funding information is known, include it here.

### Licensing Information

The dataset may be freely used for any purpose, although acknowledgement of Google LLC ("Google") as the data source would be appreciated. The dataset is provided "AS IS" without any warranty, express or implied. Google disclaims all liability for any damages, direct or indirect, resulting from the use of the dataset.

### Citation Information

```
@InProceedings{pawsx2019emnlp,
  title = {{PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification}},
  author = {Yang, Yinfei and Zhang, Yuan and Tar, Chris and Baldridge, Jason},
  booktitle = {Proc. of EMNLP},
  year = {2019}
}
```
### Contributions

Thanks to [@bhavitvyamalik](https://github.com/bhavitvyamalik), [@gowtham1997](https://github.com/gowtham1997) for adding this dataset.