task_categories:text-classification size_categories:10K<n<100K language:en license:openrail++ scientific paper fake papers science scientific text region:us
TODO
---
viewer: true
task_categories:
- text-classification
language:
- en
tags:
- scientific paper
- fake papers
- science
- scientific text
pretty_name: ' A Benchmark Dataset for Identifying Machine-Generated Scientific Papers in the LLM Era (IDMGSP)'
size_categories:
- 10K<n<100K
dataset_info:
- config_name: classifier_input
  features:
  - name: id
    dtype: string
  - name: year
    dtype: string
  - name: title
    dtype: string
  - name: abstract
    dtype: string
  - name: introduction
    dtype: string
  - name: conclusion
    dtype: string
  - name: categories
    dtype: string
  - name: src
    dtype: string
  - name: label
    dtype: int64
  splits:
  - name: train
    num_bytes: 70117904
    num_examples: 16000
  - name: test
    num_bytes: 34724993
    num_examples: 8000
  download_size: 32157176
  dataset_size: 104842897
- config_name: tecg
  features:
  - name: id
    dtype: string
  - name: year
    dtype: string
  - name: title
    dtype: string
  - name: abstract
    dtype: string
  - name: introduction
    dtype: string
  - name: conclusion
    dtype: string
  - name: categories
    dtype: string
  - name: src
    dtype: string
  - name: label
    dtype: int64
  splits:
  - name: test
    num_bytes: 2408633
    num_examples: 1000
  download_size: 582824
  dataset_size: 2408633
- config_name: train+gpt3
  features:
  - name: id
    dtype: string
  - name: year
    dtype: string
  - name: title
    dtype: string
  - name: abstract
    dtype: string
  - name: introduction
    dtype: string
  - name: conclusion
    dtype: string
  - name: categories
    dtype: string
  - name: src
    dtype: string
  - name: label
    dtype: int64
  splits:
  - name: train
    num_bytes: 73586425
    num_examples: 17200
  download_size: 22487536
  dataset_size: 73586425
- config_name: train-cg
  features:
  - name: id
    dtype: string
  - name: year
    dtype: string
  - name: title
    dtype: string
  - name: abstract
    dtype: string
  - name: introduction
    dtype: string
  - name: conclusion
    dtype: string
  - name: categories
    dtype: string
  - name: src
    dtype: string
  - name: label
    dtype: int64
  splits:
  - name: train
    num_bytes: 65261576
    num_examples: 14000
  download_size: 20272344
  dataset_size: 65261576
- config_name: ood_gpt3
  features:
  - name: title
    dtype: string
  - name: abstract
    dtype: string
  - name: introduction
    dtype: string
  - name: conclusion
    dtype: string
  - name: src
    dtype: string
  - name: label
    dtype: int64
  splits:
  - name: train
    num_bytes: 3454121
    num_examples: 1200
  - name: test
    num_bytes: 2837275
    num_examples: 1000
  download_size: 1708501
  dataset_size: 6291396
- config_name: ood_real
  features:
  - name: abstract
    dtype: string
  - name: introduction
    dtype: string
  - name: conclusion
    dtype: string
  - name: src
    dtype: string
  - name: label
    dtype: int64
  splits:
  - name: test
    num_bytes: 15808225
    num_examples: 4000
  download_size: 5336873
  dataset_size: 15808225
license: openrail++
---
# Dataset Card for A Benchmark Dataset for Identifying Machine-Generated Scientific Papers in the LLM Era

## Dataset Description

- **Repository:** https://github.com/qwenzo/-IDMGSP
- **Paper:** TODO 

### Dataset Summary

A benchmark for detecting machine-generated scientific papers based on their abstract, introduction and conclusion sections.

### Supported Tasks and Leaderboards

current benchmark results in terms of accuracy:
| Model                       | Train Dataset   | TEST    | OOD-GPT3 | OOD-REAL | TECG    | TEST-CC |
|-----------------------------|-----------------|---------|----------|----------|---------|---------|
| LR-1gram (tf-idf) (our)    | TRAIN           | 95.3%   | 4.0%     | 94.6%    | 96.1%   | 7.8%    |
| LR-1gram (tf-idf) (our)    | TRAIN+GPT3      | 94.6%   | 86.5%    | 86.2%    | 97.8%   | 13.7%   |
| LR-1gram (tf-idf) (our)    | TRAIN-CG        | 86.6%   | 0.8%     | 97.8%    | 32.6%   | 1.2%    |
| RF-1gram (tf-idf) (our)    | TRAIN           | 94.8%   | 24.7%    | 87.3%    | 100.0%  | 8.1%    |
| RF-1gram (tf-idf) (our)    | TRAIN+GPT3      | 91.7%   | 95.0%    | 69.3%    | 100.0%  | 15.1%   |
| RF-1gram (tf-idf) (our)    | TRAIN-CG        | 97.6%   | 7.0%     | 95.0%    | 57.0%   | 1.7%    |
| [IDMGSP-Galactica-TRAIN](https://huggingface.co/tum-nlp/IDMGSP-Galactica-TRAIN) (our)        | TRAIN           | 98.4%   | 25.9%    | 95.5%    | 84.0%   | 6.8%    |
| [IDMGSP-Galactica-TRAIN_GPT3](https://huggingface.co/tum-nlp/IDMGSP-Galactica-TRAIN_GPT3) (our)            | TRAIN+GPT3      | 98.5%   | 71.2%    | 95.1%    | 84.0%   | 12.0%   |
| [IDMGSP-Galactica-TRAIN-CG](https://huggingface.co/tum-nlp/IDMGSP-Galactica-TRAIN-CG) (our)            | TRAIN-CG        | 96.4%   | 12.4%    | 97.6%    | 61.3%   | 2.4%    |
| [IDMGSP-RoBERTa-TRAIN-ABSTRACT](https://huggingface.co/tum-nlp/IDMGSP-RoBERTa-TRAIN-ABSTRACT) + [IDMGSP-RoBERTa-TRAIN-INTRODUCTION](https://huggingface.co/tum-nlp/IDMGSP-RoBERTa-TRAIN-INTRODUCTION) + [IDMGSP-RoBERTa-TRAIN-CONCLUSION](https://huggingface.co/tum-nlp/IDMGSP-RoBERTa-TRAIN-CONCLUSION) (our)              | TRAIN           | 72.3%   | 55.5%    | 50.0%    | 100.0%  | 63.5%   |
| [IDMGSP-RoBERTa-TRAIN_GPT3-ABSTRACT](https://huggingface.co/tum-nlp/IDMGSP-RoBERTa-TRAIN_GPT3-ABSTRACT) + [IDMGSP-RoBERTa-TRAIN_GPT3-INTRODUCTION](https://huggingface.co/tum-nlp/IDMGSP-RoBERTa-TRAIN_GPT3-INTRODUCTION) + [IDMGSP-RoBERTa-TRAIN_GPT3-CONCLUSION](https://huggingface.co/tum-nlp/IDMGSP-RoBERTa-TRAIN_GPT3-CONCLUSION) (our)              | TRAIN+GPT3      | 65.7%   | 100.0%   | 29.1%    | 100.0%  | 75.0%   |
| [IDMGSP-RoBERTa-TRAIN-CG-ABSTRACT](https://huggingface.co/tum-nlp/IDMGSP-RoBERTa-TRAIN-CG-ABSTRACT) + [IDMGSP-RoBERTa-TRAIN-CG-INTRODUCTION](https://huggingface.co/tum-nlp/IDMGSP-RoBERTa-TRAIN-CG-INTRODUCTION) + [IDMGSP-RoBERTa-TRAIN-CG-CONCLUSION](https://huggingface.co/tum-nlp/IDMGSP-RoBERTa-TRAIN-CG-CONCLUSION) (our)              | TRAIN-CG        | 86.0%   | 2.0%     | 92.5%    | 76.5%   | 9.2%    |
| GPT-3 (our)                | TRAIN-SUB       | 100.0%  | 25.9%    | 99.0%    | 100.0%  | N/A     |
| DetectGPT                   | -               | 61.5%   | 0.0%     | 99.9%    | 68.7%   | N/A     |
| ChatGPT-IO (our)*          | -               | 69.0%   | 49.0%    | 89.0%    | 0.0%    | 3.0%    |
| LLMFE (our)*                | TRAIN+GPT3      | 80.0%   | 62.0%    | 70.0%    | 90.0%   | 33.0%   |

### Languages

English

## Dataset Structure

### Data Instances
Each instance in the dataset corresponds to a row in a CSV file, encompassing the features of a paper, its label, and the paper's source.

### Data Fields
#### classifier_input
  - name: id
  description: The ID of the provided paper corresponds to the identifier assigned by the arXiv database if the paper's source is marked as "real".
  dtype: string
- name: year
  description: year of the publication as given by the arXiv database.
  dtype: string
- name: title
  description: title of the paper given by the arXiv database.
  dtype: string
- name: abstract
  description: abstract of the paper given by the arXiv database.
  dtype: string
- name: introduction
  description: introduction section of the paper. extracted by the PDF parser.
  dtype: string
- name: conclusion
  description: conclusion section of the paper. extracted by the PDF parser.
  dtype: string
- name: categories
  description: topics/domains of the paper given by the arXiv database. This field is null if the src field is not "real".
  dtype: string
- name: src
  description: indicator of the source of the paper. This can have the values "chatgpt", "gpt2", "real", "scigen" or "galactica".
  dtype: string
- name: label
  description: 0 for real/human-written papers and 1 for fake/machine-generated papers.
  dtype: int64

#### train+gpt3
  - name: id
  description: The ID of the provided paper corresponds to the identifier assigned by the arXiv database if the paper's source is marked as "real".
  dtype: string
- name: year
  description: year of the publication as given by the arXiv database.
  dtype: string
- name: title
  description: title of the paper given by the arXiv database.
  dtype: string
- name: abstract
  description: abstract of the paper given by the arXiv database.
  dtype: string
- name: introduction
  description: introduction section of the paper. extracted by the PDF parser.
  dtype: string
- name: conclusion
  description: conclusion section of the paper. extracted by the PDF parser.
  dtype: string
- name: categories
  description: topics/domains of the paper given by the arXiv database. This field is null if the src field is not "real".
  dtype: string
- name: src
  description: indicator of the source of the paper. This can have the values "chatgpt", "gpt2", "real", "scigen" or "galactica", "gpt3".
  dtype: string
- name: label
  description: 0 for real/human-written papers and 1 for fake/machine-generated papers.
  dtype: int64

#### tecg
  - name: id
  description: The ID of the provided paper corresponds to the identifier assigned by the arXiv database if the paper's source is marked as "real".
  dtype: string
- name: year
  description: year of the publication as given by the arXiv database.
  dtype: string
- name: title
  description: title of the paper given by the arXiv database.
  dtype: string
- name: abstract
  description: abstract of the paper given by the arXiv database.
  dtype: string
- name: introduction
  description: introduction section of the paper. extracted by the PDF parser.
  dtype: string
- name: conclusion
  description: conclusion section of the paper. extracted by the PDF parser.
  dtype: string
- name: categories
  description: topics/domains of the paper given by the arXiv database. This field is null if the src field is not "real".
  dtype: string
- name: src
  description: indicator of the source of the paper. Always has the value "chatgpt".
  dtype: string
- name: label
  description: always having the value 1.
  dtype: int64

#### train-cg
  - name: id
  description: The ID of the provided paper corresponds to the identifier assigned by the arXiv database if the paper's source is marked as "real".
  dtype: string
- name: year
  description: year of the publication as given by the arXiv database.
  dtype: string
- name: title
  description: title of the paper given by the arXiv database.
  dtype: string
- name: abstract
  description: abstract of the paper given by the arXiv database.
  dtype: string
- name: introduction
  description: introduction section of the paper. extracted by the PDF parser.
  dtype: string
- name: conclusion
  description: conclusion section of the paper. extracted by the PDF parser.
  dtype: string
- name: categories
  description: topics/domains of the paper given by the arXiv database. This field is null if the src field is not "real".
  dtype: string
- name: src
  description: indicator of the source of the paper. This can have the values "gpt2", "real", "scigen" or "galactica".
  dtype: string
- name: label
  description: 0 for real/human-written papers and 1 for fake/machine-generated papers.
  dtype: int64

#### ood_gpt3
- name: title
  description: title of the paper given by the arXiv database.
  dtype: string
- name: abstract
  description: abstract of the paper given by the arXiv database.
  dtype: string
- name: introduction
  description: introduction section of the paper. extracted by the PDF parser.
  dtype: string
- name: conclusion
  description: conclusion section of the paper. extracted by the PDF parser.
  dtype: string
- name: src
  description: indicator of the source of the paper. Has the value "gpt3".
  dtype: string
- name: label
  description: always having the value 1.
  dtype: int64

#### ood_real
  dtype: string
- name: abstract
  description: abstract of the paper given by the arXiv database.
  dtype: string
- name: introduction
  description: introduction section of the paper. extracted by the PDF parser.
  dtype: string
- name: conclusion
  description: conclusion section of the paper. extracted by the PDF parser.
  dtype: string
- name: src
  description: indicator of the source of the paper. Has the value "ood_real".
  dtype: string
- name: label
  description: always having the value 0.
  dtype: int64

#### test-cc
  - name: id
  description: The ID of the provided paper corresponds to the identifier assigned by the arXiv database if the paper's source is marked as "real".
  dtype: string
- name: year
  description: year of the publication as given by the arXiv database.
  dtype: string
- name: title
  description: title of the paper given by the arXiv database.
  dtype: string
- name: abstract
  description: abstract of the paper given by the arXiv database.
  dtype: string
- name: introduction
  description: introduction section of the paper. extracted by the PDF parser.
  dtype: string
- name: conclusion
  description: conclusion section of the paper. extracted by the PDF parser.
  dtype: string
- name: categories
  description: topics/domains of the paper given by the arXiv database. This field is null if the src field is not "real".
  dtype: string
- name: src
  description: indicator of the source of the paper. Always has the value "chatgpt-paraphrased".
  dtype: string
- name: paraphrased_sections
  description: indicator of which sections are paraphrased. Can have the values "introduction", "conclusion", "introduction, conclusion", "abstract, introduction, conclusion".
  dtype: string
- name: label
  description: 0 for real/human-written papers and 1 for fake/machine-generated papers. Always has the value 1.
  dtype: int64

### Data Splits

Table: Overview of the datasets used to train and evaluate the classifiers.

| Dataset                              | arXiv  | ChatGPT | GPT-2  | SCIgen | Galactica | GPT-3  | ChatGPT (co-created) |
|--------------------------------------|--------|---------|--------|--------|-----------|--------|-----------------------|
| Standard train (TRAIN)               | 8k     | 2k      | 2k     | 2k     | 2k        | -      | -                     |
| Standard train subset (TRAIN-SUB)    | 4k     | 1k      | 1k     | 1k     | 1k        | -      | -                     |
| TRAIN without ChatGPT (TRAIN-CG)     | 8k     | -       | 2k     | 2k     | 2k        | -      | -                     |
| TRAIN plus GPT-3 (TRAIN+GPT3)        | 8k     | 2k      | 2k     | 2k     | 2k        | 1.2k   | -                     |
| Standard test (TEST)                 | 4k     | 1k      | 1k     | 1k     | 1k        | -      | -                     |
| Out-of-domain GPT-3 only (OOD-GPT3)  | -      | -       | -      | -      | -         | 1k     | -                     |
| Out-of-domain real (OOD-REAL)        | 4k (parsing 2) | -  | -      | -      | -         | -      | -                     |
| ChatGPT only (TECG)                  | -      | 1k      | -      | -      | -         | -      | -                     |
| Co-created test (TEST-CC)            | -      | -       | -      | -      | -         | -      | 4k                   |


[More Information Needed]

## Dataset Creation

### Curation Rationale

[More Information Needed]

### Source Data

#### Initial Data Collection and Normalization

[More Information Needed]

#### Who are the source language producers?

[More Information Needed]

### Annotations

#### Annotation process

[More Information Needed]

#### Who are the annotators?

[More Information Needed]

### Personal and Sensitive Information

[More Information Needed]

## Considerations for Using the Data

### Social Impact of Dataset

[More Information Needed]

### Discussion of Biases

[More Information Needed]

### Other Known Limitations

[More Information Needed]

## Additional Information

### Dataset Curators

[More Information Needed]

### Licensing Information

[More Information Needed]

### Citation Information

[More Information Needed]

### Contributions

[More Information Needed]