region:us
# jm52m
jm52m is a dataset we created containing 52m Java methods from 52k Java projects. The source code originated from the Merobase and Sourcerer data releases, supplemented by our own prior work in LeClair et al. It contains code uploaded to code repositories between 2008 and 2018. We then extracted every Java method from every file and project. We removed empty methods, methods from corrupt files, and methods with parsing errors.

---
## jm52m file list

- fc_lsh_parts_0.X0 -- lsh directory where: X = threshold e.g. if threshold=0.5, X will be 5.
  
- fundats-j1.pkl -- a pickle file that is a dictionary for raw function code files with key = function id and value = raw code
 
- fundats-j1.json.gz -- a compressed version of json file that is a dictionary for raw function code files with key = function id and value = raw code
 
- q90testfids.pkl -- funcom Java methods test set ID files

- train.bin; val.bin -- bin files for training and fine-tuning models

- jm52m.sql.gz -- a compressed version of sql files for 52m Java methods

---
## jm52m dataset details

We provide the size of our dataset in the following table:

| Config | Value |
| ------- | ------- |
|number of tokens | 8,752,695,577|
|number of documents | 51,841,717|
|number of files | 8,402,038 |
|number of projects | 52,933 |
|megabytes after processing |16,695 |

We tokenize our data using scripts provided in our [github repository](https://github.com/apcl-research/jam/blob/main/data/jam_jm52m/prepare_fc_raw.py). 

