task_categories:text-generation task_categories:text2text-generation task_categories:text-classification size_categories:100K<n<1M language:en license:cc solidity test case smart contract ethereum doi:10.57967/hf/0808 region:us
---
dataset_info:
  features:
  - name: hash
    dtype: string
  - name: size
    dtype: int64
  - name: ext
    dtype: string
  - name: lang
    dtype: string
  - name: is_test
    dtype: bool
  - name: repo_id
    dtype: string
  - name: repo_name
    dtype: string
  - name: repo_head
    dtype: string
  - name: repo_path
    dtype: string
  - name: content_tokens
    dtype: int64
  - name: content_chars
    dtype: int64
  - name: content
    dtype: string
  - name: __index_level_0__
    dtype: int64
  splits:
  - name: train
    num_bytes: 5736925269
    num_examples: 284112
  - name: test
    num_bytes: 710770657
    num_examples: 35514
  - name: eval
    num_bytes: 721961344
    num_examples: 35514
  download_size: 2050339485
  dataset_size: 7169657270
license: cc
task_categories:
- text-generation
- text2text-generation
- text-classification
language:
- en
tags:
- solidity
- test case
- smart contract
- ethereum
pretty_name: Solidity Dataset
size_categories:
- 100K<n<1M
---
# Solidity Dataset

## Dataset Description
This dataset is collected from public GitHub repositories written in Solidity programming language.
The list of the repositories is available at [repositories.json](https://huggingface.co/datasets/seyyedaliayati/solidity-dataset/blob/main/repositories.json) file.

It contains useful data about smart contracts written in Solidity along with test cases (and unit tests) written to test smart contracts.

## Dataset Summary
The dataset contains of [355,540 rows](#data-splits) in total. Each row includes the following features:

- `hash` (string): The sha256 hash value of the file content before any pre-processing.
- `size` (integer):  File size in bytes.
- `ext` (string): File extention.
- `lang` (string): The name of the programming language that the file is written with. (Solidity or Python or JavaScript)
- `is_test` (bool): Indicates whether this file is test case (test file) or the smart contract main code.
- `repo_id` (string): GitHub's repository identifer fetched from GitHub's API.
- `repo_name` (string): GitHub's repository name.
- `repo_head` (string): The head commit of the repository that the file is fetched.
- `repo_path` (string): Relative file path.
- `content_tokens` (integer): Number of tokens in the file content.
- `content_chars` (integer): Number of characters in the file content.
- `content` (string): File content.
- `__index_level_0__` (integer): Ignore this field please!

## Supported Tasks and Leaderboards

This dataset can be used for tasks related to analyzing smart contracts, test cases in smart contracts, and improving language models on Solidity language.
As of now, there are no specific leaderboards associated with this dataset.

## Languages

- The dataset is in the English language (en).
- Smart contracts (`is_test=false`) are in Solidity programming language.
- Test cases (`is_test=true`) are in Solidity, Python, or JavaScript programming language.

## Data Splits

The dataset is split into three splits:

- `train`: 284112 rows (80% of the dataset)
- `test`: 35514 rows (10% of the dataset)
- 'eval': 35514 rows (10% of the dataset)

## Dataset Creation

The `content_token` is generated via [StarCoderBase tokenizer](https://huggingface.co/bigcode/starcoderbase) using the following code snippet:
```python
from transformers import AutoTokenizer

checkpoint = "bigcode/starcoderbase"

tokenizer = AutoTokenizer.from_pretrained(checkpoint)

def count_tokens(code: str) -> int:
    tokens = tokenizer.tokenize(code)
    return len(tokens)
```

The `is_test` calculated by detecting some regex patterns in the file content. More details will publish soon.


## License

This dataset is released under the [Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) license](https://creativecommons.org/licenses/by-nc/4.0/).

## Citation

Please use the following citation when referencing the this dataset:
```
@misc {seyyed_ali_ayati_2023,
	author       = { {Seyyed Ali Ayati} },
	title        = { solidity-dataset (Revision 77e80ad) },
	year         = 2023,
	url          = { https://huggingface.co/datasets/seyyedaliayati/solidity-dataset },
	doi          = { 10.57967/hf/0808 },
	publisher    = { Hugging Face }
}
```