task_categories:text-generation task_ids:language-modeling annotations_creators:crowdsourced language_creators:crowdsourced size_categories:10K<n<100K language:code license:other code, swift, native iOS development, curated, training region:us
---
annotations_creators: 
- crowdsourced
license: other
language_creators:
- crowdsourced
language:
- code
task_categories:
- text-generation
tags:
- code, swift, native iOS development, curated, training
size_categories:
- 10K<n<100K
source_datasets: []
pretty_name: iva-swift-codeint-clean
task_ids:
- language-modeling
---

# IVA Swift GitHub Code Dataset

## Dataset Description

This is the curated train split of IVA Swift dataset extracted from GitHub. 
It contains curated Swift files gathered with the purpose to train a code generation model.

The dataset consists of 320000 Swift code files from GitHub.
[Here is the unsliced curated dataset](https://huggingface.co/datasets/mvasiliniuc/iva-swift-codeint-clean) and 
[here is the raw dataset](https://huggingface.co/datasets/mvasiliniuc/iva-swift-codeint).

### How to use it

To download the full dataset:

```python
from datasets import load_dataset
dataset = load_dataset('mvasiliniuc/iva-swift-codeint-clean', split='train')
```


## Data Structure

### Data Fields

|Field|Type|Description|
|---|---|---|
|repo_name|string|name of the GitHub repository|
|path|string|path of the file in GitHub repository|
|copies|string|number of occurrences in dataset|
|content|string|content of source file|
|size|string|size of the source file in bytes|
|license|string|license of GitHub repository|
|hash|string|Hash of content field.|
|line_mean|number|Mean line length of the content.
|line_max|number|Max line length of the content.
|alpha_frac|number|Fraction between mean and max line length of content.
|ratio|number|Character/token ratio of the file with tokenizer.
|autogenerated|boolean|True if the content is autogenerated by looking for keywords in the first few lines of the file.
|config_or_test|boolean|True if the content is a configuration file or a unit test.
|has_no_keywords|boolean|True if a file has none of the keywords for Swift Programming Language.
|has_few_assignments|boolean|True if file uses symbol '=' less than `minimum` times.

### Instance

```json
{
   "repo_name":"...",
   "path":".../BorderedButton.swift",
   "copies":"2",
   "size":"2649",
   "content":"...",
   "license":"mit",
   "hash":"db1587fd117e9a835f58cf8203d8bf05",
   "line_mean":29.1136363636,
   "line_max":87,
   "alpha_frac":0.6700641752,
   "ratio":5.298,
   "autogenerated":false,
   "config_or_test":false,
   "has_no_keywords":false,
   "has_few_assignments":false
}
```

## Languages

The dataset contains only Swift files.

```json
{
    "Swift": [".swift"]
}
```

## Licenses

Each entry in the dataset contains the associated license. The following is a list of licenses involved and their occurrences.

```json
{
   "agpl-3.0":1415,
   "apache-2.0":71451,
   "artistic-2.0":169,
   "bsd-2-clause":2628,
   "bsd-3-clause":5492,
   "cc0-1.0":1176,
   "epl-1.0":498,
   "gpl-2.0":7846,
   "gpl-3.0":15716,
   "isc":676,
   "lgpl-2.1":932,
   "lgpl-3.0":2553,
   "mit":201134,
   "mpl-2.0":6846,
   "unlicense":1468
}
```

## Dataset Statistics

```json
{
    "Total size": "~453 MB",
    "Number of files": 320000,
    "Number of files under 500 bytes": 3116,
    "Average file size in bytes": 5940,
}
```

## Curation Process

See [the unsliced curated dataset](https://huggingface.co/datasets/mvasiliniuc/iva-swift-codeint-clean) for mode details.

## Data Splits

The dataset only contains a train split focused only on training data. For validation and unspliced versions, please check the following links:
* Clean Version Unsliced: https://huggingface.co/datasets/mvasiliniuc/iva-swift-codeint-clean
* Clean Version Valid: https://huggingface.co/datasets/mvasiliniuc/iva-swift-codeint-clean-valid

# Considerations for Using the Data

The dataset comprises source code from various repositories, potentially containing harmful or biased code, 
along with sensitive information such as passwords or usernames.
