region:us
---
dataset_info:
  features:
  - name: premise
    dtype: string
  - name: entailment
    dtype: string
  - name: contradiction
    dtype: string
  splits:
  - name: train
    num_bytes: 90132700
    num_examples: 413837
  - name: valid
    num_bytes: 10572025
    num_examples: 48686
  - name: test
    num_bytes: 5289636
    num_examples: 24345
  download_size: 64195317
  dataset_size: 105994361
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
  - split: valid
    path: data/valid-*
  - split: test
    path: data/test-*
---
# Korean Natural Language Inference (KorNLI) for SimCSE Dataset

For a better dataset description, please visit this GitHub repository prepared by the authors of the article: [LINK](https://github.com/kakaobrain/kor-nlu-datasets) <br>
<br>
**This dataset was prepared by converting KorNLI dataset**. I took every unique premise of the dataset and searched for its entailment (positive example) and contradiction (negative example).
These changes have been made in order to apply SimCSE method.

**I additionaly share the code, which I used to convert the KorNLI dataset to make everything more clear**
```
from datasets import load_dataset
from tqdm import tqdm
import pandas as pd
import numpy as np

def create_trios(df, save_path):
    list_of_examples = []
    unique_premises = df.drop_duplicates("premise")["premise"]
    for premise in tqdm(unique_premises):
        premise_dataset = df[(df["premise"] == premise)] 
        positive_examples = premise_dataset[premise_dataset["label"] == "entailment"]["hypothesis"]
        negative_examples = premise_dataset[premise_dataset["label"] == "contradiction"]["hypothesis"]
        
        if len(positive_examples) == 0 or len(negative_examples) == 0:
            continue
        
        for positive_example in positive_examples:
            for negative_example in negative_examples:
                list_of_examples.append((premise, positive_example, negative_example))
        
        
    examples_df = pd.DataFrame(list_of_examples, columns=["premise", "entailment", "contradiction"])
    examples_df.to_csv(save_path)


if __name__ == "__main__":
    dataset1 = load_dataset("kor_nli", "snli")["train"]
    dataset2 = load_dataset("kor_nli", "multi_nli")["train"]
    df_1 = pd.DataFrame(dataset1)
    df_2 = pd.DataFrame(dataset2)
    df_full = pd.concat([df_1, df_2])
    df_full.dropna(inplace=True)
    df_full["label"] = ["neutral" if label == 1 else "contradiction" if label == 2 else "entailment" for label in df_full["label"]]
    create_trios(df_full, <insert your path>)
```

**How to download**

```
from datasets import load_dataset
data = load_dataset("dkoterwa/kor_nli_simcse")
```


**If you use this dataset for research, please cite this paper:**


```
@article{ham2020kornli,
  title={KorNLI and KorSTS: New Benchmark Datasets for Korean Natural Language Understanding},
  author={Ham, Jiyeon and Choe, Yo Joong and Park, Kyubyong and Choi, Ilji and Soh, Hyungjoon},
  journal={arXiv preprint arXiv:2004.03289},
  year={2020}
}
```
[More Information needed](https://github.com/huggingface/datasets/blob/main/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)