task_categories:text-classification task_categories:question-answering task_categories:sentence-similarity task_categories:zero-shot-classification size_categories:10K<n<100K language:en language:fr license:cc-by-sa-4.0 ChatGPT Bing LM Detection Detection OOD arxiv:2301.07597 region:us
Human ChatGPT Comparison Corpus (HC3) Translated To French.
The translation is done by Google Translate API.
We also add the native french QA pairs from ChatGPT, BingGPT and FAQ pages.

This dataset was used in our TALN 2023 paper.
Towards a Robust Detection of Language Model-Generated Text: Is ChatGPT that easy to detect?
# TODO: Add BibTeX citation for our TALN 2023 paper:
Towards a Robust Detection of Language Model-Generated Text: Is ChatGPT that easy to detect?

@article{guo-etal-2023-hc3,
    title = "How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection",
    author = "Guo, Biyang  and
      Zhang, Xin  and
      Wang, Ziyuan  and
      Jiang, Minqi  and
      Nie, Jinran  and
      Ding, Yuxuan  and
      Yue, Jianwei  and
      Wu, Yupeng",
    journal={arXiv preprint arxiv:2301.07597}
    year = "2023",
}
---
task_categories:
  - text-classification
  - question-answering
  - sentence-similarity
  - zero-shot-classification

language:
  - en
  - fr
size_categories:
  - 10K<n<100K

tags:
- ChatGPT
- Bing
- LM Detection
- Detection
- OOD

license: cc-by-sa-4.0
---
 Dataset card for the dataset used in :
## Towards a Robust Detection of Language Model-Generated Text: Is ChatGPT that easy to detect?

Paper: https://gitlab.inria.fr/wantoun/robust-chatgpt-detection/-/raw/main/towards_chatgpt_detection.pdf

Source Code: https://gitlab.inria.fr/wantoun/robust-chatgpt-detection
## Dataset Summary

#### overview:
This dataset is made of two parts:
- First, an extension of the [Human ChatGPT Comparison Corpus (HC3) dataset](https://huggingface.co/datasets/Hello-SimpleAI/HC3) with French data automatically translated from the English source.
- Second, out-of-domain and adversarial French data set have been gathereed (Human adversarial, BingGPT, Native French ChatGPT responses).



#### Details:  
- We first format the data into three subsets: `sentence`, `question` and `full` following the original paper.
- We then extend the data by translating the English questions and answers to French.
- We provide native French ChatGPT responses to a sample of the translated questions.
- We added a subset with QA pairs from BingGPT
- We included an adversarial subset with human-written answers in the style of conversational LLMs like Bing/ChatGPT.



## Available Subsets

### Out-of-domain:
- `hc3_fr_qa_chatgpt`: Translated French questions and native French ChatGPT answers pairs from HC3. This is the `ChatGPT-Native` subset from the paper.
    - Features: `id`, `question`, `answer`, `chatgpt_answer`, `label`, `source`
    - Size:
        - test: `113` examples, `25592` words
- `qa_fr_binggpt`: French questions and BingGPT answers pairs. This is the `BingGPT` subset from the paper.
    - Features: `id`, `question`, `answer`, `label`, `deleted_clues`, `deleted_sources`, `remarks`
    - Size:
        - test: `106` examples, `26291` words
- `qa_fr_binglikehuman`: French questions and human written BingGPT-like answers pairs. This is the `Adversarial` subset from the paper.
    - Features: `id`, `question`, `answer`, `label`, `source`
    - Size:
        - test: `61` examples, `17328` words
-  `faq_fr_gouv`: French FAQ questions and answers pairs from domain ending with `.gouv` from the MQA dataset (subset 'fr-faq-page'). https://huggingface.co/datasets/clips/mqa. This is the `FAQ-Gouv` subset from the paper.
    - Features: `id`, `page_id`, `question_id`, `answer_id`, `bucket`, `domain`, `question`, `answer`, `label`
    - Size:
        - test: `235` examples, `22336` words
- `faq_fr_random`: French FAQ questions and answers pairs from random domain from the MQA dataset (subset 'fr-faq-page'). https://huggingface.co/datasets/clips/mqa. This is the `FAQ-Rand` subset from the paper.
    - Features: `id`, `page_id`, `question_id`, `answer_id`, `bucket`, `domain`, `question`, `answer`, `label`
    - Size:
        - test: `4454` examples, `271823` words



### In-domain:

- `hc3_en_qa`: English questions and answers pairs from HC3.
    - Features: `id`, `question`, `answer`, `label`, `source`
    - Size:
        - train: `68335` examples, `12306363` words
        - validation: `17114` examples, `3089634` words
        - test: `710` examples, `117001` words
- `hc3_en_sentence`: English answers split into sentences from HC3.
    - Features: `id`, `text`, `label`, `source`
    - Size:
        - train: `455320` examples, `9983784` words
        - validation: `113830` examples, `2510290` words
        - test: `4366` examples, `99965` words
- `hc3_en_full`: English questions and answers pairs concatenated from HC3.
    - Features: `id`, `text`, `label`, `source`
    - Size:
        - train: `68335` examples, `9982863` words
        - validation: `17114` examples, `2510058` words
        - test: `710` examples, `99926` words
- `hc3_fr_qa`: Translated French questions and answers pairs from HC3.
    - Features: `id`, `question`, `answer`, `label`, `source`
    - Size:
        - train: `68283` examples, `12660717` words
        - validation: `17107` examples, `3179128` words
        - test: `710`  examples, `127193` words
- `hc3_fr_sentence`: Translated French answers split into sentences from HC3.
    - Features: `id`, `text`, `label`, `source`
    - Size:
        - train: `464885` examples, `10189606` words
        - validation: `116524` examples, `2563258` words
        - test:  `4366` examples, `108374` words
- `hc3_fr_full`: Translated French questions and answers pairs concatenated from HC3.
    - Features: `id`, `text`, `label`, `source`
    - Size:
        - train: `68283` examples, `10188669` words
        - validation: `17107` examples, `2563037` words
        - test: `710` examples, `108352` words



## How to load

```python
from datasets import load_dataset

dataset = load_dataset("almanach/hc3_multi", "hc3_fr_qa")
```

## Dataset Copyright

If the source datasets used in this corpus has a specific license which is stricter than CC-BY-SA, our products follow the same.
If not, they follow CC-BY-SA license.

| English Split       | Source | Source License | Note |
|----------|-------------|--------|-------------|
| reddit_eli5 | [ELI5](https://github.com/facebookresearch/ELI5)   | BSD License    |     |
| open_qa  | [WikiQA](https://www.microsoft.com/en-us/download/details.aspx?id=52419)  | [PWC Custom](https://paperswithcode.com/datasets/license)   |      |
| wiki_csai   | Wikipedia | CC-BY-SA |   | [Wiki FAQ](https://en.wikipedia.org/wiki/Wikipedia:FAQ/Copyright) |
| medicine    | [Medical Dialog](https://github.com/UCSD-AI4H/Medical-Dialogue-System) | Unknown|  [Asking](https://github.com/UCSD-AI4H/Medical-Dialogue-System/issues/10)|
| finance     | [FiQA](https://paperswithcode.com/dataset/fiqa-1) | Unknown |  Asking by 📧  |
| FAQ | [MQA]( https://huggingface.co/datasets/clips/mqa) | CC0 1.0| |
| ChatGPT/BingGPT | | Unknown | This is ChatGPT/BingGPT generated data. |
| Human | | CC-BY-SA | |

## Citation

```bibtex
@proceedings{towards-a-robust-2023-antoun,
    title = "Towards a Robust Detection of Language Model-Generated Text: Is ChatGPT that easy to detect?",
    editor = "Antoun, Wissam  and
      Mouilleron, Virginie  and
      Sagot, Benoit  and
      Seddah, Djam{\'e}",
    month = "6",
    year = "2023",
    address = "Paris, France",
    publisher = "ATALA",
    url = "https://gitlab.inria.fr/wantoun/robust-chatgpt-detection/-/raw/main/towards_chatgpt_detection.pdf",
}
```


```bibtex
@article{guo-etal-2023-hc3,
    title = "How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection",
    author = "Guo, Biyang  and
      Zhang, Xin  and
      Wang, Ziyuan  and
      Jiang, Minqi  and
      Nie, Jinran  and
      Ding, Yuxuan  and
      Yue, Jianwei  and
      Wu, Yupeng",
    journal={arXiv preprint arxiv:2301.07597}
    year = "2023",
    url ="https://arxiv.org/abs/2301.07597"
}
```