python code region:us
---
tags:
- python
- code
---

# CodeParrot 🦜 Dataset Cleaned

## What is it?

A dataset of Python files from Github. This is the deduplicated version of the [codeparrot](https://huggingface.co/datasets/transformersbook/codeparrot).

## Processing
The original dataset contains a lot of duplicated and noisy data. Therefore, the dataset was cleaned with the following steps:
- Deduplication 
  - Remove exact matches
- Filtering
  - Average line length < 100
  - Maximum line length < 1000
  - Alpha numeric characters fraction > 0.25
  - Remove auto-generated files (keyword search)
  
For more details see the preprocessing script in the transformers repository [here](https://github.com/huggingface/transformers/tree/master/examples/research_projects/codeparrot).

## Splits
The dataset is split in a [train](https://huggingface.co/datasets/codeparrot/codeparrot-clean-train) and [validation](https://huggingface.co/datasets/codeparrot/codeparrot-clean-valid) split used for training and evaluation.

## Structure
This dataset has ~50GB of code and 5361373 files.
```python
DatasetDict({
    train: Dataset({
        features: ['repo_name', 'path', 'copies', 'size', 'content', 'license', 'hash', 'line_mean', 'line_max', 'alpha_frac', 'autogenerated'],
        num_rows: 5361373
    })
})
```