task_categories:text-generation task_categories:text-to-image annotations_creators:no-annotation language_creators:machine-generated size_categories:10K<n<100K language:en language:code license:cc-by-nc-sa-3.0 code region:us
---
annotations_creators:
- no-annotation
language:
- en
- code
language_creators:
- machine-generated
license:
- cc-by-nc-sa-3.0
multilinguality: []
pretty_name: Shadertoys
size_categories:
- 10K<n<100K
source_datasets: []
tags:
- code
task_categories:
- text-generation
- text-to-image
task_ids: []
dataset_info:
  features:
  - name: num_passes
    dtype: int64
  - name: has_inputs
    dtype: bool
  - name: name
    dtype: string
  - name: type
    dtype: string
  - name: code
    dtype: string
  - name: title
    dtype: string
  - name: description
    dtype: string
  - name: tags
    sequence: string
  - name: author
    dtype: string
  - name: license
    dtype: string
  - name: source
    dtype: string
  splits:
  - name: train
    num_bytes: 162960894
    num_examples: 37841
  - name: test
    num_bytes: 26450429
    num_examples: 6617
  download_size: 86294414
  dataset_size: 189411323
---

# Dataset Card for Shadertoys

## Table of Contents
- [Table of Contents](#table-of-contents)
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
  - [Source Data](#source-data)
- [Licensing Information](#licensing-information)

## Dataset Description

- **Repository:** https://github.com/Vipitis/project (private placeholder)

### Dataset Summary

The Shadertoys dataset contains over 44k renderpasses collected from the Shadertoy.com API. Some shader programm contain multiple render passes.
To browse a subset of this dataset, look at the [ShaderEval](https://huggingface.co/spaces/Vipitis/ShaderCoder) space. A finer variant of this dataset is [Shadertoys-fine](https://huggingface.co/datasets/Vipitis/Shadertoys-fine).

### Supported Tasks and Leaderboards

 `text-generation` the dataset can be used to train generative language models, for code completion tasks.
 `ShaderEval` [task1](https://huggingface.co/spaces/Vipitis/ShaderEval) from ShaderEval uses a dataset derived from Shadertoys to test return completion of autoregressive language models.

### Languages

- English (title, description, tags, comments)
- Shadercode **programming** language, a subset of GLSL specifically for Shadertoy.com

## Dataset Structure

### Data Instances

A data point consists of the whole shadercode, some information from the API as well as additional metadata.
```
{
 'num_passes': 1,
 'has_inputs': False,
 'name': 'Image',
 'type': 'image',
 'code': '<full code>',
 'title': '<title of the shader>',
 'description': '<description of the shader>',
 'tags': ['tag1','tag2','tag3', ... ],
 'license': 'unknown',
 'author': '<username>',
 'source': 'https://shadertoy.com/view/<shaderID>'
}
```

### Data Fields
- 'num_passes' number of passes the parent shader program has
- 'has_inputs' if any inputs were used like textures, audio streams,
- 'name' Name of the renderpass, usually Image, Buffer A, Common, etc
- 'type' type of the renderpass; one of `{'buffer', 'common', 'cubemap', 'image', 'sound'}`
- 'code' the raw code (including comments) the whole renderpass.
- 'title' Name of the Shader
- 'description' description given for the Shader
- 'tags' List of tags assigned to the Shader (by it's creator); there are more than 10000 unique tags.
- 'license' currently in development
- 'author' username of the shader author
- 'source' URL to the shader. Not to the specific renderpass.

### Data Splits

Currently available (shuffled):
  - train (85.0%)
  - test (15.0%)

## Dataset Creation

Data retrieved starting 2022-07-20

### Source Data

#### Initial Data Collection and Normalization

All data was collected via the [Shadertoy.com API](https://www.shadertoy.com/howto#q2) and then iterated over the items in 'renderpass' while adding some of the fields from 'info'.
The code to generate these datasets should be published on the GitHub repository in the near future. 

#### Who are the source language producers?

Shadertoy.com contributers which publish shaders as 'public+API'

## Licensing Information

The Default [license for each Shader](https://www.shadertoy.com/terms) is CC BY-NC-SA 3.0. However, some Shaders might have a different license attached.
The Dataset is currently not filtering for any licenses but gives a license tag, if easily recognizeable by naive means.
Please check the first comment of each shader program yourself as to not violate any copyrights for downstream use. The main license requires share alike and by attribution.
Attribution of every data field can be found in the 'author' column, but might not include further attribution within the code itself or parents from forked shaders.