task_categories:text-generation task_ids:language-modeling language_creators:crowdsourced multilinguality:multilingual size_categories:unknown language:code region:us
---
annotations_creators: []
language_creators:
- crowdsourced
language: ["code"]
multilinguality:
- multilingual
size_categories:
- unknown
source_datasets: []
task_categories:
- text-generation
task_ids:
- language-modeling
extra_gated_prompt: >-
  ## Terms of Use for The Stack


  The Stack dataset is a collection of source code in over 300 programming
  languages. We ask that you read and acknowledge the following points before
  using the dataset:

  1. The Stack is a collection of source code from repositories with various
  licenses. Any use of all or part of the code gathered in The Stack must abide
  by the terms of the original licenses, including attribution clauses when
  relevant. We facilitate this by providing provenance information for each data
  point.

  2. The Stack is regularly updated to enact validated data removal requests. By
  clicking on "Access repository", you agree to update your own version of The
  Stack to the most recent usable version specified by the maintainers in [the
  following
  thread](https://huggingface.co/datasets/bigcode/the-stack/discussions/7). If
  you have questions about dataset versions and allowed uses, please also ask
  them in the dataset’s [community
  discussions](https://huggingface.co/datasets/bigcode/the-stack/discussions/new).
  We will also notify users via email when the latest usable version changes.

  3. To host, share, or otherwise provide access to The Stack dataset, you must
  include [these Terms of
  Use](https://huggingface.co/datasets/bigcode/the-stack#terms-of-use-for-the-stack)
  and require users to agree to it.


  By clicking on "Access repository" below, you accept that your contact
  information (email address and username) can be shared with the dataset
  maintainers as well.
extra_gated_fields:
  Email: text
  I have read the License and agree with its terms: checkbox
---

## Dataset Description
This is the Python, Java and JavaScript subsets of The Stack (v1.1) after cleaning* and agressive deduplication from [stack-dedup-alt-decontaminate](https://huggingface.co/datasets/bigcode/stack-dedup-alt-decontaminate) with 
filtering on [comment to code ratio](https://github.com/bigcode-project/bigcode-dataset/tree/main/preprocessing) with minimum of 0.01 and maximum of 0.8.

The additional comments filtering removes 26.5% of the dataset's volume which goes from 215GB of text to 170GB.

(*) cleaning: near deduplication + PII redaction + line length & percentage of alphanumeric characters filtering + data decontamination
