task_categories:text-classification task_ids:semantic-similarity-classification task_ids:semantic-similarity-scoring task_ids:text-scoring task_ids:multi-input-text-classification annotations_creators:expert-generated annotations_creators:machine-generated language_creators:machine-generated multilinguality:multilingual size_categories:10K<n<100K source_datasets:extended|other-paws language:nb language:nn license:cc-by-4.0 region:us
Norwegian PAWS-X, Bokmaal and Nynorsk machine-translated versions of PAWS-X.

PAWS-X, a multilingual version of PAWS (Paraphrase Adversaries from Word Scrambling) for six languages.

This dataset contains 23,659 human translated PAWS evaluation pairs and 296,406 machine
translated training pairs in six typologically distinct languages: French, Spanish, German,
Chinese, Japanese, and Korean. English language is available by default. All translated
pairs are sourced from examples in PAWS-Wiki.

For further details, see the accompanying paper: PAWS-X: A Cross-lingual Adversarial Dataset
for Paraphrase Identification (https://arxiv.org/abs/1908.11828)

NOTE: There might be some missing or wrong labels in the dataset and we have replaced them with -1.
@InProceedings{pawsx2019emnlp,
  title = {{PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification}},
  author = {Yang, Yinfei and Zhang, Yuan and Tar, Chris and Baldridge, Jason},
  booktitle = {Proc. of EMNLP},
  year = {2019}
}
---
annotations_creators:
- expert-generated
- machine-generated
language_creators:
- machine-generated
language:
- nb
- nn
license:
- cc-by-4.0
multilinguality:
- multilingual
size_categories:
- 10K<n<100K
source_datasets:
- extended|other-paws
task_categories:
- text-classification
task_ids:
- semantic-similarity-classification
- semantic-similarity-scoring
- text-scoring
- multi-input-text-classification
pretty_name: 'NbAiLab/norwegian-paws-x'
---

# Dataset Card for Norwegian PAWS-X

## Table of Contents
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Languages](#languages)
  - [Dataset Structure](#dataset-structure)
    - [Data Instances](#data-instances)
    - [Data Fields](#data-fields)
    - [Data Splits](#data-splits)
  - [Dataset Creation](#dataset-creation)
    - [Curation Rationale](#curation-rationale)
    - [Source Data](#source-data)
    - [Annotations](#annotations)
    - [Personal and Sensitive Information](#personal-and-sensitive-information)
  - [Considerations for Using the Data](#considerations-for-using-the-data)
    - [Social Impact of Dataset](#social-impact-of-dataset)
    - [Discussion of Biases](#discussion-of-biases)
    - [Other Known Limitations](#other-known-limitations)
  - [Additional Information](#additional-information)
    - [Dataset Curators](#dataset-curators)
    - [Licensing Information](#licensing-information)
    - [Citation Information](#citation-information)
    - [Contributions](#contributions)

## Dataset Description

- **Homepage:** [NB AiLab](https://ai.nb.no/)
- **Repository:** [Norwegian PAWS-X Repository](#)
- **Point of Contact:** [ai-lab@nb.no](mailto:ai-lab@nb.no)

### Dataset Summary

Norwegian PAWS-X is an extension of the PAWS-X dataset. PAWS-X is a multilingual version of PAWS (Paraphrase Adversaries from Word Scrambling) for six languages. The Norwegian PAWS-X dataset has machine-translated versions of the original PAWS-X dataset into Norwegian Bokmål and Nynorsk.

### Languages

- Norwegian Bokmål (`nb`)
- Norwegian Nynorsk (`nn`)

## Dataset Structure

### Data Instances

Each instance includes a pair of sentences in Norwegian along with a binary label indicating whether the sentences are paraphrases of each other.

### Data Fields

- `id`: An identifier for each example (int32)
- `sentence1`: The first sentence in Norwegian (string)
- `sentence2`: The second sentence in Norwegian (string)
- `label`: Binary label, where '1' indicates the sentences are paraphrases and '0' indicates they are not (class_label: '0', '1')

### Data Splits

The dataset is divided into training, validation, and test sets. The exact numbers of instances in each split will be as per the original PAWS-X dataset.

## Dataset Creation

### Curation Rationale

Norwegian PAWS-X was created to extend the PAWS paraphrase identification task to the Norwegian language, including both Bokmål and Nynorsk standards. This promotes multilingual and cross-lingual research in paraphrase identification.

### Source Data

The source data consists of human-translated PAWS pairs in six languages. For the Norwegian PAWS-X dataset, these pairs were translated into Norwegian Bokmål and Nynorsk using FAIR’s No Language Left Behind 3.3B parameters model.

### Annotations

The dataset retains the original PAWS labels, which were created through a combination of expert and machine-generated annotations.

### Personal and Sensitive Information

There is no known personal or sensitive information in this dataset.

## Considerations for Using the Data

### Social Impact of Dataset

This dataset helps in promoting the development of NLP technologies in Norwegian.

### Other Known Limitations

There may be potential issues related to the translation quality, as the translations were generated using a machine translation model.

## Additional Information

### Dataset Curators

The dataset was curated by researcher Javier de la Rosa.

### Licensing Information

Original PAWS-X License:
- The dataset may be freely used for any purpose, with acknowledgment of Google LLC as the data source being appreciated. The dataset is provided "AS IS" without any warranty, express or implied. Google disclaims all liability for any damages, direct or indirect, resulting from the use of the dataset.

Norwegian PAWS-X License:
- CC BY 4.0

