task_categories:text-generation task_ids:language-modeling language_creators:crowdsourced multilinguality:multilingual size_categories:unknown language:code region:us
---
annotations_creators: []
language_creators:
- crowdsourced
language: ["code"]
multilinguality:
- multilingual
size_categories:
- unknown
source_datasets: []
task_categories:
- text-generation
task_ids:
- language-modeling

extra_gated_prompt: |-
  ## Terms of Use for The Stack

  The Stack dataset is a collection of  3.1 TB of source code in 30 programming languages. We ask that you read and acknowledge the following points before using the dataset:
  1. The Stack is a collection of source code from repositories with various licenses. Any use of all or part of the code gathered in The Stack must abide by the terms of the original licenses, including attribution clauses when relevant. We facilitate this by providing provenance information for each data point.
  2. The Stack is regularly updated to enact validated data removal requests. By clicking on "Access repository", you agree to update your own version of The Stack to the most recent usable version specified by the maintainers in [the following thread](https://huggingface.co/datasets/bigcode/the-stack/discussions/7). If you have questions about dataset versions and allowed uses, please also ask them in the dataset’s [community discussions](https://huggingface.co/datasets/bigcode/the-stack/discussions/new). We will also notify users via email when the latest usable version changes.
  3. To host, share, or otherwise provide access to The Stack dataset, you must include [these Terms of Use](https://huggingface.co/datasets/bigcode/the-stack#terms-of-use-for-the-stack) and require users to agree to it.

  By clicking on "Access repository" below, you accept that your contact information (email address and username) can be shared with the dataset maintainers as well.
    
extra_gated_fields:
 Email: text
 I have read the License and agree with its terms: checkbox
---

## Dataset Description
![Smol](https://huggingface.co/datasets/bigcode/admin/resolve/main/smol.png)

A small subset (~0.1%) of [the-stack](https://huggingface.co/datasets/bigcode/the-stack) dataset, each programming language has 10,000 random samples from the original dataset. The dataset has 2.6GB of text (code).


## Languages

The dataset contains 30 programming languages:
````
"assembly", "batchfile", "c++", "c", "c-sharp", "cmake", "css", "dockerfile", "fortran", "go", "haskell", "html", "java",
"javascript", "julia", "lua", "makefile", "markdown", "perl", "php", "powershell", "python", "ruby", "rust", 
"scala", "shell", "sql", "tex", "typescript", "visual-basic"
`````
## Dataset Structure

```python
from datasets import load_dataset
load_dataset("bigcode/the-stack-smol")

DatasetDict({
    train: Dataset({
        features: ['content', 'avg_line_length', 'max_line_length', 'alphanum_fraction', 'licenses', 'repository_name', 'path', 'size', 'lang'],
        num_rows: 300000
    })
})
```
### How to use it
You can either load the whole dataset like above, or load a specific language such as python by specifying the folder directory:
```python
load_dataset("bigcode/the-stack-smol", data_dir="data/python")

DatasetDict({
    train: Dataset({
        features: ['content', 'avg_line_length', 'max_line_length', 'alphanum_fraction', 'licenses', 'repository_name', 'path', 'size', 'lang'],
        num_rows: 10000
    })
})
```

