region:us
---
dataset_info:
  features:
  - name: code
    dtype: string
  - name: package
    dtype: string
  - name: path
    dtype: string
  - name: filename
    dtype: string
  splits:
  - name: train
    num_bytes: 31543801750
    num_examples: 2438172
  download_size: 9201420527
  dataset_size: 31543801750
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
---
# Dataset Card for "pypi_clean"

All of the latest package versions from pypi. The original data came from [here](https://py-code.org/datasets). I pulled the latest versions of each package, then extracted only `md`, `rst`, `ipynb`, and `py` files.

I then applied some cleaning:

- rendering notebooks
- removing leading comments/licenses