region:us
# Jam-CGPT

Jam-CGPT dataset contains the summary generated by using GPT-3.5. The dataset size ranges from 170k to 2.15m. We follow [Jam](https://huggingface.co/apcl/jam)'s procedure to compile the dataset for finetuning.

## Jam-CGPT dataset files

| Filename | Description |
| ------- | ------- |
|170k.tar.gz|170k summary train and val bin file|
|620k.tar.gz | 620k summary train and val bin file|
|1.25m.tar.gz | 1.25m summary train and val bin file |
|2.15m.tar.gz | 2.15m summary train and val bin file |
|jam_cgpt_test.tar.gz |Jam-CGPT testset |
|raw_data.tar.gz |Jam-CGPT raw data without compiling to bin file and test and val function id pickle files |

## Jam-CGPT dataset tokens
We provide the number of tokens for each dataset the following table:
| Dataset | Number of tokens |
| ------- | ------- |
|170k|37,399,419 tokens|
|620k | 118,941,493 tokens|
|1.25m | 233,964,910 tokens |
|2.15m | 403,770,021 |

We tokenize our data using scripts provided in our [github repository](https://github.com/apcl-research/Jam-CGPT/blob/main/data/jam_cgpt_170k/prepare_fc_raw.py).
