task_categories:text-generation task_ids:language-modeling annotations_creators:crowdsourced language_creators:crowdsourced size_categories:100K<n<1M language:code license:other code, kotlin, native Android development doi:10.57967/hf/0779 region:us
---
annotations_creators: 
- crowdsourced
license: other
language_creators:
- crowdsourced
language:
- code
task_categories:
- text-generation
tags:
- code, kotlin, native Android development
size_categories:
- 100K<n<1M
source_datasets: []
pretty_name: iva-kotlin-codeint-raw
task_ids:
- language-modeling
---

# IVA Kotlin GitHub Code Dataset

## Dataset Description

This is the raw IVA Kotlin dataset extracted from GitHub. 
It contains uncurated Kotlin files gathered with the purpose to train a code generation model.

The dataset consists of 464215 kotlin code files from GitHub totaling ~361 MB of data. 
The dataset was created from the public GitHub dataset on Google BiqQuery.

### How to use it

To download the full dataset:

```python
from datasets import load_dataset
dataset = load_dataset('mvasiliniuc/iva-kotlin-codeint', split='train')
```

```python

from datasets import load_dataset
dataset = load_dataset('mvasiliniuc/iva-kotlin-codeint', split='train')
print(dataset[723])

#OUTPUT:
{
   "repo_name":"nemerosa/ontrack",
   "path":"ontrack-extension-notifications/src/main/java/net/nemerosa/ontrack/extension/notifications/webhooks/WebhookController.kt",
   "copies":"1",
   "size":"3248",
   "content":"...@RestController\n@RequestMapping(\"/extension/notifications/webhook\")\nclass WebhookController(\n    private val webhookAdminService: WebhookAdminService,\n    private val webhookExecutionService: ",
   "license":"mit"
}
```

## Data Structure

### Data Fields

|Field|Type|Description|
|---|---|---|
|repo_name|string|name of the GitHub repository|
|path|string|path of the file in GitHub repository|
|copies|string|number of occurrences in dataset|
|code|string|content of source file|
|size|string|size of the source file in bytes|
|license|string|license of GitHub repository|

### Instance

```json
{
   "repo_name":"nemerosa/ontrack",
   "path":"ontrack-extension-notifications/src/main/java/net/nemerosa/ontrack/extension/notifications/webhooks/WebhookController.kt",
   "copies":"1",
   "size":"3248",
   "content":"...@RestController\n@RequestMapping(\"/extension/notifications/webhook\")\nclass WebhookController(\n    private val webhookAdminService: WebhookAdminService,\n    private val webhookExecutionService: ",
   "license":"mit"
}
```

## Languages

The dataset contains only Kotlin files.

```json
{
    "Kotlin": [".kt"]
}
```

## Licenses

Each entry in the dataset contains the associated license. The following is a list of licenses involved and their occurrences.

```json
{
    "agpl-3.0": 9146,
    "apache-2.0": 272388,
    "artistic-2.0": 219,
    "bsd-2-clause": 896,
    "bsd-3-clause": 12328,
    "cc0-1.0": 411,
    "epl-1.0": 2111,
    "gpl-2.0": 11080,
    "gpl-3.0": 48911,
    "isc": 997,
    "lgpl-2.1": 297,
    "lgpl-3.0": 7749,
    "mit": 92540,
    "mpl-2.0": 3386,
    "unlicense": 1756
}
```

## Dataset Statistics

```json
{
    "Total size": "~361 MB",
    "Number of files": 464215,
    "Number of files under 500 bytes": 99845,
    "Average file size in bytes": 3252,
}
```

## Dataset Creation

The dataset was created using Google Query for Github: 
https://cloud.google.com/blog/topics/public-datasets/github-on-bigquery-analyze-all-the-open-source-code

The following steps were pursued for data
gathering:
1. Creation of a dataset and a table in Google Big Query Project.
2. Creation of a bucket in Google Cloud Storage.
3. Creation of a query in Google Big Query Project.
4. Running the query with the setting to output the results in the dataset and table
created at step one.
5. Exporting the resulting dataset into the bucket created in step 2. Export format of JSON with gzip compression.

The result of these steps leads to the following results:
* 2.7 TB Processed, 
* number of extracted rows/files was 464,215 
* total logical bytes  1.46 GB. 
* the result amounts to 7 json.gz files in a total of 361 MB.

The SQL Query used is:
```sql
SELECT
  f.repo_name, f.path, c.copies, c.size, c.content, l.license
FROM
 (select f.*, row_number() over (partition by id order by path desc) as seqnum from `bigquery-public-data.github_repos.files` AS f) f
JOIN
  `bigquery-public-data.github_repos.contents` AS c
ON
  f.id = c.id AND seqnum=1
JOIN
  `bigquery-public-data.github_repos.licenses` AS l
ON
  f.repo_name = l.repo_name
WHERE
  NOT c.binary AND ((f.path LIKE '%.kt') AND (c.size BETWEEN 0 AND 1048575))
```

## Data Splits
The dataset only contains a train split.

Using the curated version of this dataset, a split was made into multiple repositories:
* Clean Version: https://huggingface.co/datasets/mvasiliniuc/iva-kotlin-codeint-clean
* Clean Version Train: https://huggingface.co/datasets/mvasiliniuc/iva-kotlin-codeint-clean-train
* Clean Version Valid: https://huggingface.co/datasets/mvasiliniuc/iva-kotlin-codeint-clean-valid

# Considerations for Using the Data

The dataset comprises source code from various repositories, potentially containing harmful or biased code, 
along with sensitive information such as passwords or usernames.

# Additional Information

## Dataset Curators

[mircea.dev@icloud.com](mircea.dev@icloud.com)

## Licensing Information

* The license of this open-source dataset is: other.
* The dataset is gathered from open-source repositories on [GitHub using BigQuery](https://cloud.google.com/blog/topics/public-datasets/github-on-bigquery-analyze-all-the-open-source-code). 
* Find the license of each entry in the dataset in the corresponding license column.

## Citation Information

```json
@misc {mircea_vasiliniuc_2023,
	author       = { {Mircea Vasiliniuc} },
	title        = { iva-kotlin-codeint (Revision 1af5124) },
	year         = 2023,
	url          = { https://huggingface.co/datasets/mvasiliniuc/iva-kotlin-codeint },
	doi          = { 10.57967/hf/0779 },
	publisher    = { Hugging Face }
}
```