task_categories:text-classification size_categories:1K<n<10K language:en license:cc AIGC for education arxiv:2304.07666 region:us
---
license: cc
task_categories:
- text-classification
language:
- en
tags:
- AIGC for education
size_categories:
- 1K<n<10K
---

# Machine-essays generation pipeline

Please check out our [github repo](https://github.com/huhailinguist/ArguGPT).

This document only introduces how we collected **machine-generated essays**. 

| model            | timestamp   | # total | # valid | # short | # repetitive | # overlapped |
|------------------|-------------|---------|---------|---------|--------------|--------------|
| gpt2-xl          | Nov, 2019   | 4,573   | 563     | 1,637   | 0            | 2,373        |
| text-babbage-001 | April, 2022 | 917     | 479     | 181     | 240          | 17           |
| text-curie-001   | April, 2022 | 654     | 498     | 15      | 110          | 31           |
| text-davinci-001 | April, 2022 | 632     | 493     | 1       | 41           | 97           |
| text-davinci-002 | April, 2022 | 621     | 495     | 1       | 56           | 69           |
| text-davinci-003 | Nov, 2022   | 1,130   | 1,090   | 0       | 30           | 10           |
| gpt-3.5-turbo    | Mar, 2023   | 1,122   | 1,090   | 0       | 4            | 28           |
| total            | -           | 9,647   | 4,708   | 1,835   | 481          | 2,625        |


## Models

We chose 7 models from GPT family: 1) `gpt2-xl`, 2) `text-babbage-001`, 3) `text-curie-001`, 4) `text-davinci-001`, 5) `text-davinci-002`, 
6) `text-davinci-003`, and 7) `gpt-3.5-turbo`. 
More information about these models can be seen in [OpenAI documentation](https://platform.openai.com/docs/model-index-for-researchers). 

For WECCL and TOEFL, we used all 7 models to generate argumentative essays. 
As for GRE, of which the writing task is more difficult than WECCL and TOEFL, we only used `text-davinci-003` and `gpt-3.5-turbo`. 

**Notes**: Since `gpt2-xl` cannot respond to prompts as InstructGPTs and other later models, 
we fed `gpt2-xl` the prompt along with one beginning sentence randomly extracted from human essays for continuous writing. 
Therefore, the first sentence of each essay generated by `gpt2-xl` is actually human-authored. 

## Prompts selection

Our writing topics are collected from human-WECCL, human-TOEFL, and human-GRE. 
In a writing task, a topic statement is presented for students (or machines) to attack or defend. 
The topic statement here is refered to `ESSAY_PROMPT`, and our added instructions for machine is refered to `ADDED_PROMPT`.

Therefore, our prompt format is as follow: `ESSAY_PROMPT` + `ADDED_PROMPT`. 

For instance, 
- `ESSAY_PROMPT`: It is better to have broad knowledge of many academic subjects than to specialize in one specific subject. 
- `ADDED_PROMPT`: Do you agree or disagree? Use specific reasons and examples to support your answer. Write an essay of roughly {300/400/500} words.

We asked the machine to write 300 words for writing tasks in WECCL, 400 for TOEFL, and 500 for GRE. 

## Essays filtering, preprocessing, and automated scoring

We then filtered out the essays that are short, repetitive and overlapped. 
- Short: we set the threshold of 50 words for `gpt2-xl`, and 100 words for others.
- Repetitive: 40% of sentences are *similar*.
- Overlapped: 40% of sentences are *similar* with any other essay already generated.
- Definition of *similar*: "I like a dog." and "I don't like a cat." have 3 words in common. The similarity therefore is 6 / 9 = 0.67. If the similarity is greater than 0.8, the two sentences are *similar*.

We deleted "As an AI model, ..." generated by gpt-3.5-turbo. 
And we used [YouDao automated scoring system](https://ai.youdao.com/) to score all the essays, 
and categorized them into low, mid, and high levels.

## Citation

Please cite our work [arXiv:2304.07666](https://arxiv.org/abs/2304.07666) as  

```
@misc{liu2023argugpt,
      title={ArguGPT: evaluating, understanding and identifying argumentative essays generated by GPT models}, 
      author={Yikang Liu and Ziyin Zhang and Wanyang Zhang and Shisen Yue and Xiaojing Zhao and Xinyuan Cheng and Yiwen Zhang and Hai Hu},
      year={2023},
      eprint={2304.07666},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```
