task_categories:question-answering task_categories:text-generation task_categories:conversational size_categories:100K<n<1M region:us
---
task_categories:
- question-answering
- text-generation
- conversational
size_categories:
- 100K<n<1M
---
# Dataset Card for Dataset Name

## Dataset Description

- **Repository: {https://github.com/ar852/chatgpt-scraping}** 

### Dataset Summary

scraped-chatgpt-conversations contains ~100k conversations between a user and chatgpt that were shared online through reddit, twitter, or sharegpt. For sharegpt, the conversations were directly scraped from the website. For reddit and twitter, images were downloaded from submissions, segmented, and run through an OCR pipeline to obtain a conversation list. For information on how the each json file is structured, please see `json_guides.md`

### Languages

 - twitter 1, twitter 2, and sharegpt json files are multilingual
 - reddit and twitter 2 json files are english only

## Dataset Structure

 - refer to *json_guide.txt*

## Dataset Creation

This dataset was created by scraping images from twitter, reddit, and sharegpt.com using the pushshift and twitter APIs, respectively. The images are run through a filter to check if they contain a chatgpt conversation, then the image is processed and run through an OCR pipeline to obtain the conversation text. More info can be found in the repository. 

### Source Data
 - twitter.com
 - reddit.com
 - sharegpt.com

## Considerations for Using the Data

A significant amount of dicts created from parsing reddit and twitter images may be parsed incorrectly for a number of reasons: cropping done by the image poster, incorrectly classifying the image as containing a chatgpt conversation, incorrect image parsing (segmentation) by the parser, incorrect OCR by pytesseract. 

### Licensing Information

[More Information Needed]

### Contributions

[More Information Needed]