task_categories:text-classification annotations_creators:expert-generated language_creators:found multilinguality:multilingual size_categories:1M<n<10M source_datasets:extended|xnli language:en language:de language:fr mode classification aligned code-mixed region:us
This dataset was generated from XNLI using the CodeMixed Text Generator for a binary text classification task.
# @InProceedings{huggingface:dataset,
# title = {A great new dataset},
# author={huggingface, Inc.
# },
# year={2020}
# }
---
annotations_creators:
- expert-generated
language:
- en
- de
- fr
language_creators:
- found
license: []
multilinguality:
- multilingual
pretty_name: XNLI Code-Mixed Corpus
size_categories:
- 1M<n<10M
source_datasets:
- extended|xnli
tags:
- mode classification
- aligned
- code-mixed
task_categories:
- text-classification
task_ids: []
dataset_info:
  - config_name: de_ec
    features:
      - name: text
        dtype: string
      - name: label
        dtype: int64
          # class_label:
          #   names:
          #     '0': spoken
          #     '1': written
    splits:
      - name: train
        num_bytes: 576
        num_examples: 2490
      - name: test
        num_bytes: 194139776
        num_examples: 1610549

  - config_name: de_ml
    features:
      - name: text
        dtype: string
      - name: label
        dtype: int64
          # class_label:
          #   names:
          #     '0': spoken
          #     '1': written
    splits:
      - name: train
        num_bytes: 576
        num_examples: 2490
      - name: test
        num_bytes: 87040
        num_examples: 332326

  - config_name: fr_ec
    features:
      - name: text
        dtype: string
      - name: label
        dtype: int64
          # class_label:
          #   names:
          #     '0': spoken
          #     '1': written
    splits:
      - name: train
        num_bytes: 576
        num_examples: 2490
      - name: test
        num_bytes: 564416
        num_examples: 2562631

  - config_name: fr_ml
    features:
      - name: text
        dtype: string
      - name: label
        dtype: int64
          # class_label:
          #   names:
          #     '0': spoken
          #     '1': written
    splits:
      - name: train
        num_bytes: 576
        num_examples: 2490
      - name: test
        num_bytes: 361472
        num_examples: 1259159

download_size: 1376728
dataset_size: 1376704
---

# Dataset Card for XNLI Code-Mixed Corpus

## Dataset Description

- **Homepage:** 
- **Repository:** 
- **Paper:** 
- **Leaderboard:** 
- **Point of Contact:** 

### Dataset Summary


### Supported Tasks and Leaderboards

Binary mode classification (spoken vs written)

### Languages
- English
- German
- French
- German-English code-mixed by Equivalence Constraint Theory
- German-English code-mixed by Matrix Language Theory
- French-English code-mixed by Equivalence Constraint Theory
- German-English code-mixed by Matrix Language Theory

## Dataset Structure

### Data Instances

{
  'text': "And he said , Mama , I 'm home", 
  'label': 0
}

### Data Fields
- text: sentence
- label: binary label of text (0: spoken 1: written)

### Data Splits
  - de-ec
    - train (English, German, French monolingual): 
    - test (German-English code-mixed by Equivalence Constraint Theory):
  - de-ml: 
    - train (English, German, French monolingual):
    - test (German-English code-mixed by Matrix Language Theory):
  - fr-ec
    - train (English, German, French monolingual):
    - test (French-English code-mixed by Equivalence Constraint Theory):
  - fr-ml: 
    - train (English, German, French monolingual):
    - test (French-English code-mixed by Matrix Language Theory):

### Other Statistics


#### Average Sentence Length
 
- German
  - train: 
  - test: 
    
- French
  - train: 
  - test: 

#### Label Split
  - train:
    - 0: 
    - 1: 
  - test:
    - 0: 
    - 1: 

## Dataset Creation

### Curation Rationale

Using the XNLI Parallel Corpus, we generated a code-mixed corpus using CodeMixed Text Generator.

The XNLI Parallel Corpus is available here:
https://huggingface.co/datasets/nanakonoda/xnli_parallel
It was created from the XNLI corpus.
More information is available in the datacard for the XNLI Parallel Corpus.

Here is the link and citation for the original CodeMixed Text Generator paper.
https://github.com/microsoft/CodeMixed-Text-Generator

```
@inproceedings{rizvi-etal-2021-gcm,
    title = "{GCM}: A Toolkit for Generating Synthetic Code-mixed Text",
    author = "Rizvi, Mohd Sanad Zaki  and
      Srinivasan, Anirudh  and
      Ganu, Tanuja  and
      Choudhury, Monojit  and
      Sitaram, Sunayana",
    booktitle = "Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations",
    month = apr,
    year = "2021",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.eacl-demos.24",
    pages = "205--211",
    abstract = "Code-mixing is common in multilingual communities around the world, and processing it is challenging due to the lack of labeled and unlabeled data. We describe a tool that can automatically generate code-mixed data given parallel data in two languages. We implement two linguistic theories of code-mixing, the Equivalence Constraint theory and the Matrix Language theory to generate all possible code-mixed sentences in the language-pair, followed by sampling of the generated data to generate natural code-mixed sentences. The toolkit provides three modes: a batch mode, an interactive library mode and a web-interface to address the needs of researchers, linguists and language experts. The toolkit can be used to generate unlabeled text data for pre-trained models, as well as visualize linguistic theories of code-mixing. We plan to release the toolkit as open source and extend it by adding more implementations of linguistic theories, visualization techniques and better sampling techniques. We expect that the release of this toolkit will help facilitate more research in code-mixing in diverse language pairs.",
}
```

### Source Data

XNLI Parallel Corpus
https://huggingface.co/datasets/nanakonoda/xnli_parallel

#### Original Source Data
XNLI Parallel Corpus was created using the XNLI Corpus.
https://github.com/facebookresearch/XNLI

Here is the citation for the original XNLI paper.

```
@InProceedings{conneau2018xnli,
  author = "Conneau, Alexis
        and Rinott, Ruty
        and Lample, Guillaume
        and Williams, Adina
        and Bowman, Samuel R.
        and Schwenk, Holger
        and Stoyanov, Veselin",
  title = "XNLI: Evaluating Cross-lingual Sentence Representations",
  booktitle = "Proceedings of the 2018 Conference on Empirical Methods
               in Natural Language Processing",
  year = "2018",
  publisher = "Association for Computational Linguistics",
  location = "Brussels, Belgium",
}
```

#### Initial Data Collection and Normalization

We removed all punctuation from the XNLI Parallel Corpus except apostrophes.

#### Who are the source language producers?

N/A

### Annotations

#### Annotation process

N/A

#### Who are the annotators?

N/A

### Personal and Sensitive Information

N/A

## Considerations for Using the Data

### Social Impact of Dataset

N/A

### Discussion of Biases

N/A

### Other Known Limitations

N/A

## Additional Information

### Dataset Curators

N/A

### Licensing Information

N/A

### Citation Information


### Contributions

N/A