task_categories:text-generation annotations_creators:no-annotation language_creators:machine-generated size_categories:100K<n<1M language:en language:code license:cc-by-nc-sa-3.0 code region:us
---
annotations_creators:
- no-annotation
language:
- en
- code
language_creators:
- machine-generated
license:
- cc-by-nc-sa-3.0
multilinguality: []
pretty_name: Shadertoys-fine
size_categories:
- 100K<n<1M
source_datasets: []
tags:
- code
task_categories:
- text-generation
task_ids: []
dataset_info:
- config_name: default
  features:
  - name: name
    dtype: string
  - name: code
    dtype: string
  - name: source
    dtype: string
  - name: author
    dtype: string
  splits:
  - name: train
  - name: test
  download_size: 154529204
  dataset_size: 0
- config_name: fine
  features:
  - name: name
    dtype: string
  - name: code
    dtype: string
  - name: source
    dtype: string
  - name: author
    dtype: string
  splits:
  - name: train
    num_bytes: 119963236
    num_examples: 226910
  - name: test
    num_bytes: 20003783
    num_examples: 38356
  download_size: 154529204
  dataset_size: 139967019
- config_name: return_completion
  features:
  - name: body
    dtype: string
  - name: return_statement
    dtype: string
  splits:
  - name: train
    num_bytes: 37597125
    num_examples: 84843
  - name: test
    num_bytes: 6360131
    num_examples: 14248
  download_size: 154529204
  dataset_size: 43957256
---

# Dataset Card for Shadertoys-fine

## Table of Contents
- [Table of Contents](#table-of-contents)
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
  - [Source Data](#source-data)
- [Licensing Information](#licensing-information)

## Dataset Description

- **Repository:** https://github.com/Vipitis/project (private placeholder)

### Dataset Summary

fine variant of the Shadertoys dataset (still WIP), where individual functions are avaialable as Datapoints.

### Supported Tasks and Leaderboards

 `language-modeling`: The dataset can be used to train a model for modelling programming languages, which consists in building language models for programming languages.

### Languages

- English (names, comments)
- Shadercode **programming** language

## Dataset Structure

### Data Instances

A data point consists of the function string, it's name as well as a bit of metadata like the author and source URL. (in the future there might be a function string without comments)
```
{
  'name': '<type> <name>',
  'code': '<type> <name>(<inputs>) { <body> return <outputs>; }\n',
  'source': 'https://shadertoy.com/view/<shaderID>',
  'author': '<username>'
}
```

A data point in the `return_completion` subset for the return-completion task in [ShaderEval](https://huggingface.co/spaces/Vipitis/ShaderEval) includes just two features:
```
{
  'body': '<type> <name> <type> <name>(<inputs>) { <body> return',
  'return_statment': ' <outputs>: }\n',
}
```


### Data Fields

- 'name' funciton identifier composed of the type and the name of the function
- 'code' the raw code (including comments) of function.
- 'source' URL to the shader. It might be on a different renderpass
- 'author' username of the shader author

- 'body' the body of the function without the return statement (no comments)
- 'return_statment' the return statement of the function. everything infront of the semicolon is kept and white sapces are stripped in the custome Evaluator.

### Data Splits

Currently available (shuffled):
  - train (85.0%)
  - test (15.0%)

These splits should be indexed the same across both subsets. So if you are fine-tuning on the `fine` subset you won't get exposed to the `return_completion` test split. However there are many duplicates among both subsets and splits.

## Dataset Creation

Data retrieved starting 2022-07-20

### Source Data

#### Initial Data Collection and Normalization

All data was collected via the [Shadertoy.com API](https://www.shadertoy.com/howto#q2) and then by looking for keywords and counting curly brackets to figure out what is part of a function and what isn't.

#### Who are the source language producers?

Shadertoy.com contributers which publish shaders as 'public+API'

## Licensing Information

The Default [licnese for each Shader](https://www.shadertoy.com/terms) is CC BY-NC-SA 3.0. However, some Shaders might have a different license attached. The Dataset is currently not filtering for any licensis. 