task_categories:text-classification annotations_creators:expert-generated language_creators:found multilinguality:multilingual size_categories:1M<n<10M source_datasets:extended|xnli language:en language:de language:fr mode classification aligned code-mixed region:us
This dataset was generated from XNLI using the CodeMixed Text Generator for a binary text classification task.
# @InProceedings{huggingface:dataset,
# title = {A great new dataset},
# author={huggingface, Inc.
# },
# year={2020}
# }
---
annotations_creators:
- expert-generated
language:
- en
- de
- fr
language_creators:
- found
license: []
multilinguality:
- multilingual
pretty_name: XNLI Code-Mixed Corpus (Sampled)
size_categories:
- 1M<n<10M
source_datasets:
- extended|xnli
tags:
- mode classification
- aligned
- code-mixed
task_categories:
- text-classification
task_ids: []
dataset_info:
- config_name: monolingual
  features:
  - name: text
    dtype: string
  - name: label
    dtype: int64
  splits:
  - name: train
    num_bytes: 317164
    num_examples: 2490
  - name: test
    num_bytes: 641496
    num_examples: 5007
  download_size: 891209
  dataset_size: 958660
- config_name: de_ec
  features:
  - name: text
    dtype: string
  - name: label
    dtype: int64
  splits:
  - name: train
    num_bytes: 317164
    num_examples: 2490
  - name: test
    num_bytes: 1136549
    num_examples: 14543
  download_size: 1298619
  dataset_size: 1453713
- config_name: de_ml
  features:
  - name: text
    dtype: string
  - name: label
    dtype: int64
  splits:
  - name: train
    num_bytes: 317164
    num_examples: 2490
  - name: test
    num_bytes: 1068937
    num_examples: 12750
  download_size: 1248962
  dataset_size: 1386101
- config_name: fr_ec
  features:
  - name: text
    dtype: string
  - name: label
    dtype: int64
  splits:
  - name: train
    num_bytes: 317164
    num_examples: 2490
  - name: test
    num_bytes: 1520429
    num_examples: 18653
  download_size: 1644995
  dataset_size: 1837593
- config_name: fr_ml
  features:
  - name: text
    dtype: string
  - name: label
    dtype: int64
  splits:
  - name: train
    num_bytes: 317164
    num_examples: 2490
  - name: test
    num_bytes: 1544539
    num_examples: 17381
  download_size: 1682885
  dataset_size: 1861703

download_size: 891209
dataset_size: 958660
---

# Dataset Card for XNLI Code-Mixed Corpus (Sampled)

## Dataset Description

- **Homepage:** 
- **Repository:** 
- **Paper:** 
- **Leaderboard:** 
- **Point of Contact:** 

### Dataset Summary


### Supported Tasks and Leaderboards

Binary mode classification (spoken vs written)

### Languages
- English
- German
- French
- German-English code-mixed by Equivalence Constraint Theory
- German-English code-mixed by Matrix Language Theory
- French-English code-mixed by Equivalence Constraint Theory
- German-English code-mixed by Matrix Language Theory

## Dataset Structure

### Data Instances

{
  'text': "And he said , Mama , I 'm home", 
  'label': 0
}

### Data Fields
- text: sentence
- label: binary label of text (0: spoken 1: written)

### Data Splits
  - monolingual
    - train (English, German, French monolingual): 2490
    - test (English, German, French monolingual): 5007
  - de_ec
    - train (English, German, French monolingual): 2490
    - test (German-English code-mixed by Equivalence Constraint Theory): 14543
  - de_ml
    - train (English, German, French monolingual): 2490
    - test (German-English code-mixed by Matrix Language Theory): 12750
  - fr_ec
    - train (English, German, French monolingual): 2490
    - test (French-English code-mixed by Equivalence Constraint Theory): 18653
  - fr_ml
    - train (English, German, French monolingual): 2490
    - test (French-English code-mixed by Matrix Language Theory): 17381

### Other Statistics


#### Average Sentence Length

- monolingual
  - train: 19.18714859437751
  - test: 19.321150389454765
 
- de_ec
  - train: 19.18714859437751
  - test: 11.24314103004882
    
- de_ml
  - train: 19.18714859437751
  - test: 12.159450980392156

- fr_ec
  - train: 19.18714859437751
  - test: 12.26526564091567
    
- fr_ml
  - train: 19.18714859437751
  - test: 13.486968528853346


#### Label Split
  - monolingual
    - train
      - 0: 498
      - 1: 1992
      
    - test
      - 0: 1002
      - 1: 4005

  - de_ec
    - train
      - 0: 498
      - 1: 1992
  
    - test
      - 0: 2777
      - 1: 11766
  
  - de_ml
    - train
      - 0: 498
      - 1: 1992
  
    - test
      - 0: 2329
      - 1: 10421

  - fr_ec
    - train
      - 0: 498
      - 1: 1992
  
    - test
      - 0: 3322
      - 1: 15331

  - fr_ml
    - train
      - 0: 498
      - 1: 1992
  
    - test
      - 0: 2788
      - 1: 14593

## Dataset Creation

### Curation Rationale

Using the XNLI Parallel Corpus, we generated a code-mixed corpus using CodeMixed Text Generator, and sampled a maximum of 30 sentences per original English sentence.

The XNLI Parallel Corpus is available here:
https://huggingface.co/datasets/nanakonoda/xnli_parallel
It was created from the XNLI corpus.
More information is available in the datacard for the XNLI Parallel Corpus.

Here is the link and citation for the original CodeMixed Text Generator paper.
https://github.com/microsoft/CodeMixed-Text-Generator

```
@inproceedings{rizvi-etal-2021-gcm,
    title = "{GCM}: A Toolkit for Generating Synthetic Code-mixed Text",
    author = "Rizvi, Mohd Sanad Zaki  and
      Srinivasan, Anirudh  and
      Ganu, Tanuja  and
      Choudhury, Monojit  and
      Sitaram, Sunayana",
    booktitle = "Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations",
    month = apr,
    year = "2021",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.eacl-demos.24",
    pages = "205--211",
    abstract = "Code-mixing is common in multilingual communities around the world, and processing it is challenging due to the lack of labeled and unlabeled data. We describe a tool that can automatically generate code-mixed data given parallel data in two languages. We implement two linguistic theories of code-mixing, the Equivalence Constraint theory and the Matrix Language theory to generate all possible code-mixed sentences in the language-pair, followed by sampling of the generated data to generate natural code-mixed sentences. The toolkit provides three modes: a batch mode, an interactive library mode and a web-interface to address the needs of researchers, linguists and language experts. The toolkit can be used to generate unlabeled text data for pre-trained models, as well as visualize linguistic theories of code-mixing. We plan to release the toolkit as open source and extend it by adding more implementations of linguistic theories, visualization techniques and better sampling techniques. We expect that the release of this toolkit will help facilitate more research in code-mixing in diverse language pairs.",
}
```

### Source Data

XNLI Code-Mixed Corpus
https://huggingface.co/datasets/nanakonoda/xnli_cm

XNLI Parallel Corpus
https://huggingface.co/datasets/nanakonoda/xnli_parallel

#### Original Source Data
XNLI Parallel Corpus was created using the XNLI Corpus.
https://github.com/facebookresearch/XNLI

Here is the citation for the original XNLI paper.

```
@InProceedings{conneau2018xnli,
  author = "Conneau, Alexis
        and Rinott, Ruty
        and Lample, Guillaume
        and Williams, Adina
        and Bowman, Samuel R.
        and Schwenk, Holger
        and Stoyanov, Veselin",
  title = "XNLI: Evaluating Cross-lingual Sentence Representations",
  booktitle = "Proceedings of the 2018 Conference on Empirical Methods
               in Natural Language Processing",
  year = "2018",
  publisher = "Association for Computational Linguistics",
  location = "Brussels, Belgium",
}
```

#### Initial Data Collection and Normalization

We removed all punctuation from the XNLI Parallel Corpus except apostrophes.

#### Who are the source language producers?

N/A

### Annotations

#### Annotation process

N/A

#### Who are the annotators?

N/A

### Personal and Sensitive Information

N/A

## Considerations for Using the Data

### Social Impact of Dataset

N/A

### Discussion of Biases

N/A

### Other Known Limitations

N/A

## Additional Information

### Dataset Curators

N/A

### Licensing Information

N/A

### Citation Information


### Contributions

N/A