task_categories:text-generation task_categories:zero-shot-classification size_categories:10B<n<100B language:en region:us
---
dataset_info:
  features:
  - name: title
    sequence: string
  - name: author
    sequence: string
  - name: authoraffiliation
    sequence: string
  - name: venue
    sequence: string
  - name: abstract
    dtype: string
  - name: doi
    dtype: string
  - name: pdfurls
    sequence: string
  - name: corpusid
    dtype: int64
  - name: arxivid
    dtype: string
  - name: pdfsha
    dtype: string
  - name: text
    dtype: string
  - name: github_urls
    sequence: string
  splits:
  - name: train
    num_bytes: 89132091867
    num_examples: 1671614
  download_size: 35993359504
  dataset_size: 89132091867
task_categories:
- text-generation
- zero-shot-classification
language:
- en
pretty_name: arxiv_s2orc_parsed
size_categories:
- 10B<n<100B
---
# Dataset Card for "ArtifactAI/arxiv_s2orc_parsed"


## Dataset Description

https://huggingface.co/datasets/ArtifactAI/arxiv_s2orc_parsed


### Dataset Summary

ArtifactAI/arxiv_s2orc_parsed is a subset of the [AllenAI S2ORC dataset](https://github.com/allenai/s2orc), a general-purpose corpus for NLP and text mining research over scientific papers, 
The dataset is filtered strictly for ArXiv papers, including the full text for each paper. Github links have been extracted from each paper to aid in the development of [ArtifactAI/arxiv_python_research_code](https://huggingface.co/datasets/ArtifactAI/arxiv_python_research_code)

### How to use it
```python
from datasets import load_dataset

ds = load_dataset("ArtifactAI/arxiv_s2orc_parsed", split="train")

# dataset streaming (will only download the data as needed)
ds = load_dataset("ArtifactAI/arxiv_s2orc_parsed", streaming=True, split="train")
```

## Dataset Structure
### Data Instances
Each data instance corresponds to one file. The content of the file is in the `text` feature, and other features provide some metadata.
### Data Fields
- `title` (sequence): list of titles.
- `author` (sequence): list of authors.
- `authoraffiliation` (sequence): list of institution affiliations for each author.
- `venue`: (integer): paper publication venue.
- `doi`: (float): paper doi.
- `pdfurls`: (integer): url link to the paper.
- `corpusid`: (int): corpus ID as defined by s2orc.
- `arxivid`: (int): arxiv paper id.
- `pdfsha`: (string): unique pdf hash.
- `text`: (string): full text of the arxiv paper.
- github_urls: (sequence): list of github urls referenced within the text

### Data Splits

The dataset has no splits and all data is loaded as train split by default.

## Additional Information

### Dataset Curators
Matthew Kenney, Artifact AI, matt@artifactai.com

### Citation Information
```
@misc{arxiv_s2orc_parsed,
    title={arxiv_s2orc_parsed},
    author={Matthew Kenney},
    year={2023}
}
```