Spaces:

HuggingFaceFW
/

blogpost-fineweb-v1

Running

App Files Files Community

guipenedo HF staff commited on Jun 1

Commit

f7efb82

•

1 Parent(s): ac80290

styling and small text changes

Browse files

Files changed (7) hide show

dist/distill.bundle.js +0 -0
dist/distill.bundle.js.map +0 -0
dist/index.html +32 -137
dist/style.css +99 -0
src/distill.js +1 -0
src/index.html +32 -137
src/style.css +99 -0

dist/distill.bundle.js CHANGED Viewed

The diff for this file is too large to render. See raw diff

dist/distill.bundle.js.map CHANGED Viewed

The diff for this file is too large to render. See raw diff

dist/index.html CHANGED Viewed

@@ -1,125 +1,20 @@
 <!doctype html>
 <head>
-    <link rel="stylesheet" href="style.css">
     <script src="distill.bundle.js" type="module" fetchpriority="high" blocking></script>
     <script src="main.bundle.js" type="module" fetchpriority="low" defer></script>
     <meta name="viewport" content="width=device-width, initial-scale=1">
     <meta charset="utf8">
     <base target="_blank">
     <title>FineWeb: decanting the web for the finest text data at scale</title>
-    <style>
-        /* ****************************************
-         * TOC
-         ******************************************/
-        @media (max-width: 1199px) {
-            d-contents {
-                display: none;
-                justify-self: start;
-                align-self: start;
-                padding-bottom: 0.5em;
-                margin-bottom: 1em;
-                padding-left: 0.25em;
-                border-bottom: 1px solid rgba(0, 0, 0, 0.1);
-                border-bottom-width: 1px;
-                border-bottom-style: solid;
-                border-bottom-color: rgba(0, 0, 0, 0.1);
-            }
-        }
-        d-contents a:hover {
-            border-bottom: none;
-        }
-        @media (min-width: 1200px) {
-            d-article {
-                /* Ensure d-article does not prevent sticky positioning */
-                overflow: visible;
-            }
-            d-contents {
-                align-self: start;
-                grid-column-start: 1 !important;
-                grid-column-end: 4 !important;
-                grid-row: auto / span 6;
-                justify-self: end;
-                margin-top: 0em;
-                padding-right: 3em;
-                padding-left: 2em;
-                border-right: 1px solid rgba(0, 0, 0, 0.1);
-                border-right-width: 1px;
-                border-right-style: solid;
-                border-right-color: rgba(0, 0, 0, 0.1);
-                position: -webkit-sticky; /* For Safari */
-                position: sticky;
-                top: 10px; /* Adjust this value if needed */
-            }
-        }
-        d-contents nav h3 {
-            margin-top: 0;
-            margin-bottom: 1em;
-        }
-        d-contents nav div {
-            color: rgba(0, 0, 0, 0.8);
-            font-weight: bold;
-        }
-        d-contents nav a {
-            color: rgba(0, 0, 0, 0.8);
-            border-bottom: none;
-            text-decoration: none;
-        }
-        d-contents li {
-            list-style-type: none;
-        }
-        d-contents ul, d-article d-contents ul {
-            padding-left: 1em;
-        }
-        d-contents nav ul li {
-            margin-bottom: .25em;
-        }
-        d-contents nav a:hover {
-            text-decoration: underline solid rgba(0, 0, 0, 0.6);
-        }
-        d-contents nav ul {
-            margin-top: 0;
-            margin-bottom: 6px;
-        }
-        d-contents nav > div {
-            display: block;
-            outline: none;
-            margin-bottom: 0.5em;
-        }
-        d-contents nav > div > a {
-            font-size: 13px;
-            font-weight: 600;
-        }
-        d-contents nav > div > a:hover,
-        d-contents nav > ul > li > a:hover {
-            text-decoration: none;
-        }
-    </style>
 </head>
 <body>
 <d-front-matter>
     <script id='distill-front-matter' type="text/json">{
     "title": "🍷 FineWeb: decanting the web for the finest text data at scale",
-    "description": "This blog covers a discussion on processing and evaluating data quality at scale, the 🍷 FineWeb recipe (listing and explaining all of our design choices), and the process followed to create 📚 FineWeb-Edu.",
     "published": "May 28, 2024",
     "affiliation": {"name": "HuggingFace"},
     "authors": [
@@ -187,17 +82,18 @@
     <p>Both datasets are released under the permissive <a href="https://opendatacommons.org/licenses/by/1-0/">ODC-By 1.0 license</a></p>
     <p>As 🍷 FineWeb has gathered a lot of interest from the
-        community, we decided to explain in all details the steps involved in creating it as well as our processing decisions and
-        many lessons learned along the way. Hence the present (lengthy) technical report. Read on for all the juicy details on large text dataset creation!</p>
     <p><strong>TLDR:</strong> This blog covers a discussion on processing and evaluating data quality at scale, the 🍷 FineWeb
-        recipe (listing and explaining all of our design choices), and the process followed to create its 📚 FineWeb-Edu subset as well.</p>
     <h2>General considerations on web data</h2>
     <h3>Sourcing the data</h3>
     <p>A common question often asked regarding web datasets used
         to train LLMs is “where do they even get all that data?”. There are generally two options:</p>
     <ul>
-        <li>you either crawl it yourself, like companies like OpenAI or Anthropic (among other) do (see hints <a
                 href="https://platform.openai.com/docs/gptbot">here</a> and <a
                 href="https://darkvisitors.com/agents/claudebot">here</a>)
         </li>
@@ -206,15 +102,15 @@
         <li>you use a public repository of crawled webpages, like the one maintained by
             the non-profit <a href="https://commoncrawl.org/">CommonCrawl</a></li>
     </ul>
-    <p>To build and filter 🍷 FineWeb, following what had done in the past by number of LLM training teams,
       we used <a href="https://commoncrawl.org/">CommonCrawl</a> (CC) as a starting point.
         The Common Crawl non–profit organization has been crawling the web since 2007 and
-        release a new crawl containing 200 to 300 TB of textual content obtained via automatic web crawling usually
         every 1 or 2 months. </p>
-    <p>As an example, the latest CC crawl (id 2024-18) contains 2.7
-        billion web pages, totaling 386 TiB of uncompressed HTML text content (Note that the size changes from dump to dump).
-      Ninety-six crawls have been released since 2013 and 3 dumps from 2008 to 2012, which are in a different (older) format.
-      <d-footnote>We have not processed these 3 older dumps.</d-footnote> </p>
     <h3>Processing at scale</h3>
     <p>Given the sheer size of the data involved, one of the main
@@ -225,40 +121,39 @@
             href="https://github.com/huggingface/datatrove"><code>datatrove</code></a><d-cite bibtex-key="penedo2024datatrove"></d-cite>, an open-source data
         processing library that allowed us to seamlessly scale our filtering and deduplication setup to thousands of
         CPU cores. All the data processing steps involved in the creation of 🍷 FineWeb used this <a
-                href="https://github.com/huggingface/datatrove">library</a>. In most cases, you'll find the exact same scripts we used in the <a
-                href="https://github.com/huggingface/datatrove"><code>datatrove</code></a> repository.</p>
     <h3>What is good data?</h3>
     <p>This is probably the main question to keep in mind when
-        creating a dataset. In most context and in particular in the context of large language model pretraining <d-footnote>Note that all our discussion in this report is focused on the special field of web-scale datasets ("web-scale" typically meaning >100 billion tokens) used to pretrained a Large Language Models (by pretraining we mean the very first step of training of a model, starting from random weights). We don't pretend to cover any other field of dataset creation nor that the lessons or hypothesis we develop in this document can extend to any field beside this specific field.</d-footnote>, "high quality" is not a very well defined term<d-cite bibtex-key="albalak2024survey"></d-cite>, and not even a property of documents that can always be clearly perceived through direct, human, observation alone.<d-cite bibtex-key="longpre2023pretrainers"></d-cite></p>
     <p>It is still common to train a model on a given corpus considered "clean"
         (typically wikipedia<d-footnote>Even though as we mentioned above the notion of "clean" is so ill-defined that it should probably not been seen as equivalent to wikipedia-type of text</d-footnote>) and use it to check the perplexity on the dataset
         that we were trying to curate<d-cite bibtex-key="wenzek2019ccnet"></d-cite>. Unfortunately this does not always correlate with improved performance on a set of downstream
-        tasks of interest<d-cite bibtex-key="soldaini2024dolma"></d-cite>, and as a result another often used approach is to train small models<d-footnote>"Small" in comparison to standard sizes of today's LLM, i.e. small in comparison to 7-70 billion parameters. In this work "small" means about 1-2 billion parameters</d-footnote> on a representative subset of our dataset and evaluate them on
-        a set of evaluation tasks. The reason small model are used is because training models is
-        expensive and time consuming as a function of model size. In this second approach, it is important to
         choose a diverse and representative set of dataset-evaluation tasks and try not to overfit to any one individual benchmark as it would risk hurting the generality of the obtained LLM after pretraining.</p>
     <p>Yet another way to compare different datasets would be to
         train a model on each dataset and have humans rate and compare the generations of the models (like on the <a
                 href="https://chat.lmsys.org/">LMSYS Chatbot Arena</a>)<d-cite bibtex-key="chiang2024chatbot"></d-cite>. This would arguably provide the most
         reliable results in terms of representing real model usage, but getting ablation results this way is unfortunately
-        expensive and slow. It also often requires for the models to have undergone through an instruction finetuning stage to acquire conversational capabilities, as pretrained models are not directly-designed to follow instructions and thus much more sensitive to prompt details.<d-cite bibtex-key="ouyang2022training"></d-cite></p>
     <p>In this work, we went with the approach of training small
         models and evaluating them on a set of "early-signal" benchmark tasks. We believe this is a reasonable proxy for the quality
-        of the data used to train these models with the above mentioned caveat around overfitting on the evaluation benchmarks.</p>
     <h3>Ablations and evaluation setup</h3>
-    <p>To be able to compare the impact of a given processing
-        step, we typically train two models on two versions of the dataset, one version processed with the extra –ablated– step and another version with this step
-        ablated (cut/removed). Apart from the data, these two models are otherwise identical: same number of parameters, architecture hyper-parameters, and are trained
         on an equal number of randomly sampled tokens from each version of the data, for a single epoch — the only difference being thus the
-        training data. We then evaluate each model on the same set of tasks and compare average
         scores.</p>
-    <p>Our ablation models are trained using <a
             href="https://github.com/huggingface/nanotron"><code>nanotron</code></a> with this config [<strong>TODO:
-        INSERT SIMPLIFIED NANOTRON CONFIG HERE</strong>]. Ablation models have 1.82B parameters (including embeddings), used the Llama
-        architecture with a 2048 sequence length, a global batch size of ~2 million tokens and GPT2 tokenizer as mentioned above. For most
         ablations we trained on ~28B tokens (roughly the Chinchilla<d-cite bibtex-key="hoffmann2022training"></d-cite> optimal training size for this
-        model size). To confirm relative performance after several steps of filtering we conducted longer training runs in 350 billion tokens as mentioned further below.</p>
     <p>We evaluated the models using <a
             href="https://github.com/huggingface/lighteval/"><code>lighteval</code></a>. We carefully selected a set of benchmark for ablations by selecting
         benchmarks that would provide good signal at a relatively small scale ("small" models trained on only "a few
@@ -266,17 +161,17 @@
     <ul>
         <li>small variance between runs trained on different samplings of the same
             dataset: we want our runs on a subset of the data to be representative of the whole dataset, and the
-            resulting scores to have as little evaluation noise as possible and sensitive to exact data choice (apart from larger ablation that we are concerned with)
         </li>
     </ul>
     <ul>
         <li>performance increasing monotonically (or close) over a training run:
-            ideally, as the number of seen tokens increases, the performance of a high-signal benchmark should not decrease
             (which would be indicative of unreliable results at a small scale)
         </li>
     </ul>
     <ul>
-        <li>performance above the random noise level with a few standard deviations at least. Given our small ablation models and trainings we usually don't reach extremely high scores on any benchmark, but we want to make sure that the scores we get are above random noise.
         </li>
     </ul>
     <p>After consideration, we selected the following list of benchmarks:</p>
@@ -291,7 +186,7 @@
         <li>MMLU<d-cite bibtex-key="hendrycks2021measuring"></d-cite></li>
     </ul>
     <p>To
-        compute our checkpoint evaluation in a constrained time, we capped the longer benchmarks at 1000 samples (wall-clock evaluation taking less than 5
         min on a single node of 8 GPUs - done in parallel to the training).</p>
     <aside>You can find the full list of tasks and prompts we used <a
             href="https://huggingface.co/datasets/HuggingFaceFW/fineweb/blob/main/lighteval_tasks.py">here</a>.</aside>

 <!doctype html>
 <head>
     <script src="distill.bundle.js" type="module" fetchpriority="high" blocking></script>
     <script src="main.bundle.js" type="module" fetchpriority="low" defer></script>
     <meta name="viewport" content="width=device-width, initial-scale=1">
     <meta charset="utf8">
     <base target="_blank">
     <title>FineWeb: decanting the web for the finest text data at scale</title>
+    <link rel="stylesheet" href="style.css">
 </head>
 <body>
 <d-front-matter>
     <script id='distill-front-matter' type="text/json">{
     "title": "🍷 FineWeb: decanting the web for the finest text data at scale",
+    "description": "This blog covers a discussion on processing and evaluating data quality at scale, the 🍷 FineWeb recipe (listing and explaining all of our design choices), and the process followed to create its 📚 FineWeb-Edu subset.",
     "published": "May 28, 2024",
     "affiliation": {"name": "HuggingFace"},
     "authors": [
     <p>Both datasets are released under the permissive <a href="https://opendatacommons.org/licenses/by/1-0/">ODC-By 1.0 license</a></p>
     <p>As 🍷 FineWeb has gathered a lot of interest from the
+        community, we decided to explain in full detail the steps involved in creating it as well as our processing decisions and
+        many lessons learned along the way. Hence, the present (lengthy) technical report. Read on for all the juicy details on large text dataset creation!</p>
+    <aside>For the best possible reading experience, we recommend not using a mobile phone.</aside>
     <p><strong>TLDR:</strong> This blog covers a discussion on processing and evaluating data quality at scale, the 🍷 FineWeb
+        recipe (listing and explaining all of our design choices), and the process followed to create its 📚 FineWeb-Edu subset.</p>
     <h2>General considerations on web data</h2>
     <h3>Sourcing the data</h3>
     <p>A common question often asked regarding web datasets used
         to train LLMs is “where do they even get all that data?”. There are generally two options:</p>
     <ul>
+        <li>you either crawl it yourself, like companies such as OpenAI or Anthropic (among others) do (see <a
                 href="https://platform.openai.com/docs/gptbot">here</a> and <a
                 href="https://darkvisitors.com/agents/claudebot">here</a>)
         </li>
         <li>you use a public repository of crawled webpages, like the one maintained by
             the non-profit <a href="https://commoncrawl.org/">CommonCrawl</a></li>
     </ul>
+    <p>To build 🍷 FineWeb, following what has been done in the past by a number of LLM training teams,
       we used <a href="https://commoncrawl.org/">CommonCrawl</a> (CC) as a starting point.
         The Common Crawl non–profit organization has been crawling the web since 2007 and
+        releases a new crawl containing 200 to 400 TiB of textual content obtained via automatic web crawling usually
         every 1 or 2 months. </p>
+    <p>As an example, the latest CC crawl (April 2024) contains 2.7
+        billion web pages, totaling 386 TiB of uncompressed HTML text content<d-footnote>Note that the size changes from crawl to crawl</d-footnote>.
+      Ninety-six crawls have been released since 2013 and 3 crawls from 2008 to 2012, which are in a different (older) format.
+      <d-footnote>We have not processed these 3 older crawls.</d-footnote> </p>
     <h3>Processing at scale</h3>
     <p>Given the sheer size of the data involved, one of the main
             href="https://github.com/huggingface/datatrove"><code>datatrove</code></a><d-cite bibtex-key="penedo2024datatrove"></d-cite>, an open-source data
         processing library that allowed us to seamlessly scale our filtering and deduplication setup to thousands of
         CPU cores. All the data processing steps involved in the creation of 🍷 FineWeb used this <a
+                href="https://github.com/huggingface/datatrove">library</a>. You will find the exact scripts we used in the
+        <a href="https://github.com/huggingface/datatrove/blob/main/examples/fineweb.py"><code>datatrove</code> repository</a>.</p>
     <h3>What is good data?</h3>
     <p>This is probably the main question to keep in mind when
+        creating a dataset. In most contexts and, in particular, in the context of large language model pretraining <d-footnote>Note that this report is focused on the special field of web-scale datasets ("web-scale" typically meaning >100 billion tokens obtained from the web) used to pretrain a Large Language Model (by pretraining we mean the very first step in the training of a model, starting from random weights). We don't pretend to cover any other field of dataset creation nor that the lessons or hypothesis we develop in this document can extend to any field besides this specific field.</d-footnote>, "high quality" is not a very well defined term<d-cite bibtex-key="albalak2024survey"></d-cite>, and not even a property of documents that can always be clearly perceived through direct human observation alone.<d-cite bibtex-key="longpre2023pretrainers"></d-cite></p>
     <p>It is still common to train a model on a given corpus considered "clean"
         (typically wikipedia<d-footnote>Even though as we mentioned above the notion of "clean" is so ill-defined that it should probably not been seen as equivalent to wikipedia-type of text</d-footnote>) and use it to check the perplexity on the dataset
         that we were trying to curate<d-cite bibtex-key="wenzek2019ccnet"></d-cite>. Unfortunately this does not always correlate with improved performance on a set of downstream
+        tasks of interest<d-cite bibtex-key="soldaini2024dolma"></d-cite>, and as a result another often used approach is to train small models<d-footnote>"Small" in comparison to standard sizes of today's LLMs, i.e. small in comparison to 7-70 billion parameters. In this work "small" means about 1-2 billion parameters</d-footnote> on a representative subset of our dataset and evaluate them on
+        a set of evaluation tasks. Small models are used because training costs and time are a function of model size. In this second approach, it is important to
         choose a diverse and representative set of dataset-evaluation tasks and try not to overfit to any one individual benchmark as it would risk hurting the generality of the obtained LLM after pretraining.</p>
     <p>Yet another way to compare different datasets would be to
         train a model on each dataset and have humans rate and compare the generations of the models (like on the <a
                 href="https://chat.lmsys.org/">LMSYS Chatbot Arena</a>)<d-cite bibtex-key="chiang2024chatbot"></d-cite>. This would arguably provide the most
         reliable results in terms of representing real model usage, but getting ablation results this way is unfortunately
+        expensive and slow. It also often requires for the models to have undergone an instruction finetuning stage to acquire conversational capabilities, as pretrained models are not directly designed to follow instructions and are thus much more sensitive to prompt details.<d-cite bibtex-key="ouyang2022training"></d-cite></p>
     <p>In this work, we went with the approach of training small
         models and evaluating them on a set of "early-signal" benchmark tasks. We believe this is a reasonable proxy for the quality
+        of the data used to train these models, when keeping in mind the above-mentioned caveat around overfitting on the evaluation benchmarks.</p>
     <h3>Ablations and evaluation setup</h3>
+    <p>To compare the impact of a given processing
+        step, we trained two models on two versions of the dataset, one version processed with the extra step (the one we wish to evaluate) and another version with this step
+        ablated (cut/removed). Apart from the data, these two models would be otherwise identical: the same number of parameters, architecture hyper-parameters, and trained
         on an equal number of randomly sampled tokens from each version of the data, for a single epoch — the only difference being thus the
+        training data. We then evaluated each model on the same set of tasks and compared average
         scores.</p>
+    <p>Our ablation models were trained using <a
             href="https://github.com/huggingface/nanotron"><code>nanotron</code></a> with this config [<strong>TODO:
+        INSERT SIMPLIFIED NANOTRON CONFIG HERE</strong>]. Our ablation models have 1.82B parameters (including embeddings), used the Llama
+        architecture with a 2048 sequence length, a global batch size of ~2 million tokens, and the GPT2 tokenizer. For most
         ablations we trained on ~28B tokens (roughly the Chinchilla<d-cite bibtex-key="hoffmann2022training"></d-cite> optimal training size for this
+        model size). To confirm relative performance improvements after each step of filtering we conducted longer training runs on 350 billion tokens as mentioned further below.</p>
     <p>We evaluated the models using <a
             href="https://github.com/huggingface/lighteval/"><code>lighteval</code></a>. We carefully selected a set of benchmark for ablations by selecting
         benchmarks that would provide good signal at a relatively small scale ("small" models trained on only "a few
     <ul>
         <li>small variance between runs trained on different samplings of the same
             dataset: we want our runs on a subset of the data to be representative of the whole dataset, and the
+            resulting scores to have as little sensitivity to exact data choice as possible (apart from larger ablations that we are concerned with)
         </li>
     </ul>
     <ul>
         <li>performance increasing monotonically (or close) over a training run:
+            ideally, as the number of seen tokens increases, the performance on a high-signal benchmark should not decrease
             (which would be indicative of unreliable results at a small scale)
         </li>
     </ul>
     <ul>
+        <li>performance above random baseline for this task by at least a few standard deviations: given our small ablation models and trainings we usually don't reach extremely high scores on any benchmark, but we want to make sure that the scores we get are above random noise.
         </li>
     </ul>
     <p>After consideration, we selected the following list of benchmarks:</p>
         <li>MMLU<d-cite bibtex-key="hendrycks2021measuring"></d-cite></li>
     </ul>
     <p>To
+        ensure our checkpoint evaluation stayed within a limited timeframe, we capped the longer benchmarks at 1000 samples (wall-clock evaluation taking less than 5
         min on a single node of 8 GPUs - done in parallel to the training).</p>
     <aside>You can find the full list of tasks and prompts we used <a
             href="https://huggingface.co/datasets/HuggingFaceFW/fineweb/blob/main/lighteval_tasks.py">here</a>.</aside>

dist/style.css CHANGED Viewed

@@ -141,4 +141,103 @@ d-byline .byline {
 d-contents > nav a.active {
     text-decoration: underline;
 }

 d-contents > nav a.active {
     text-decoration: underline;
+}
+@media (max-width: 1199px) {
+    d-contents {
+        display: none;
+        justify-self: start;
+        align-self: start;
+        padding-bottom: 0.5em;
+        margin-bottom: 1em;
+        padding-left: 0.25em;
+        border-bottom: 1px solid rgba(0, 0, 0, 0.1);
+        border-bottom-width: 1px;
+        border-bottom-style: solid;
+        border-bottom-color: rgba(0, 0, 0, 0.1);
+    }
+}
+d-contents a:hover {
+    border-bottom: none;
+}
+@media (min-width: 1200px) {
+    d-article {
+        /* Ensure d-article does not prevent sticky positioning */
+        overflow: visible;
+    }
+    d-contents {
+        align-self: start;
+        grid-column-start: 1 !important;
+        grid-column-end: 4 !important;
+        grid-row: auto / span 6;
+        justify-self: end;
+        margin-top: 0em;
+        padding-right: 3em;
+        padding-left: 2em;
+        border-right: 1px solid rgba(0, 0, 0, 0.1);
+        border-right-width: 1px;
+        border-right-style: solid;
+        border-right-color: rgba(0, 0, 0, 0.1);
+        position: -webkit-sticky; /* For Safari */
+        position: sticky;
+        top: 10px; /* Adjust this value if needed */
+    }
+}
+d-contents nav h3 {
+    margin-top: 0;
+    margin-bottom: 1em;
+}
+d-contents nav div {
+    color: rgba(0, 0, 0, 0.8);
+    font-weight: bold;
+}
+d-contents nav a {
+    color: rgba(0, 0, 0, 0.8);
+    border-bottom: none;
+    text-decoration: none;
+}
+d-contents li {
+    list-style-type: none;
+}
+d-contents ul, d-article d-contents ul {
+    padding-left: 1em;
+}
+d-contents nav ul li {
+    margin-bottom: .25em;
+}
+d-contents nav a:hover {
+    text-decoration: underline solid rgba(0, 0, 0, 0.6);
+}
+d-contents nav ul {
+    margin-top: 0;
+    margin-bottom: 6px;
+}
+d-contents nav > div {
+    display: block;
+    outline: none;
+    margin-bottom: 0.5em;
+}
+d-contents nav > div > a {
+    font-size: 13px;
+    font-weight: 600;
+}
+d-contents nav > div > a:hover,
+d-contents nav > ul > li > a:hover {
+    text-decoration: none;
 }

src/distill.js CHANGED Viewed

@@ -4382,6 +4382,7 @@ d-footnote-list a.footnote-backlink {
           const backlink = document.createElement('a');
           backlink.setAttribute('class', 'footnote-backlink');
           backlink.textContent = '[↩]';
           backlink.href = '#' + footnote.id;

           const backlink = document.createElement('a');
           backlink.setAttribute('class', 'footnote-backlink');
+          backlink.setAttribute('target', '_self');
           backlink.textContent = '[↩]';
           backlink.href = '#' + footnote.id;

src/index.html CHANGED Viewed

@@ -1,125 +1,20 @@
 <!doctype html>
 <head>
-    <link rel="stylesheet" href="style.css">
     <script src="distill.bundle.js" type="module" fetchpriority="high" blocking></script>
     <script src="main.bundle.js" type="module" fetchpriority="low" defer></script>
     <meta name="viewport" content="width=device-width, initial-scale=1">
     <meta charset="utf8">
     <base target="_blank">
     <title>FineWeb: decanting the web for the finest text data at scale</title>
-    <style>
-        /* ****************************************
-         * TOC
-         ******************************************/
-        @media (max-width: 1199px) {
-            d-contents {
-                display: none;
-                justify-self: start;
-                align-self: start;
-                padding-bottom: 0.5em;
-                margin-bottom: 1em;
-                padding-left: 0.25em;
-                border-bottom: 1px solid rgba(0, 0, 0, 0.1);
-                border-bottom-width: 1px;
-                border-bottom-style: solid;
-                border-bottom-color: rgba(0, 0, 0, 0.1);
-            }
-        }
-        d-contents a:hover {
-            border-bottom: none;
-        }
-        @media (min-width: 1200px) {
-            d-article {
-                /* Ensure d-article does not prevent sticky positioning */
-                overflow: visible;
-            }
-            d-contents {
-                align-self: start;
-                grid-column-start: 1 !important;
-                grid-column-end: 4 !important;
-                grid-row: auto / span 6;
-                justify-self: end;
-                margin-top: 0em;
-                padding-right: 3em;
-                padding-left: 2em;
-                border-right: 1px solid rgba(0, 0, 0, 0.1);
-                border-right-width: 1px;
-                border-right-style: solid;
-                border-right-color: rgba(0, 0, 0, 0.1);
-                position: -webkit-sticky; /* For Safari */
-                position: sticky;
-                top: 10px; /* Adjust this value if needed */
-            }
-        }
-        d-contents nav h3 {
-            margin-top: 0;
-            margin-bottom: 1em;
-        }
-        d-contents nav div {
-            color: rgba(0, 0, 0, 0.8);
-            font-weight: bold;
-        }
-        d-contents nav a {
-            color: rgba(0, 0, 0, 0.8);
-            border-bottom: none;
-            text-decoration: none;
-        }
-        d-contents li {
-            list-style-type: none;
-        }
-        d-contents ul, d-article d-contents ul {
-            padding-left: 1em;
-        }
-        d-contents nav ul li {
-            margin-bottom: .25em;
-        }
-        d-contents nav a:hover {
-            text-decoration: underline solid rgba(0, 0, 0, 0.6);
-        }
-        d-contents nav ul {
-            margin-top: 0;
-            margin-bottom: 6px;
-        }
-        d-contents nav > div {
-            display: block;
-            outline: none;
-            margin-bottom: 0.5em;
-        }
-        d-contents nav > div > a {
-            font-size: 13px;
-            font-weight: 600;
-        }
-        d-contents nav > div > a:hover,
-        d-contents nav > ul > li > a:hover {
-            text-decoration: none;
-        }
-    </style>
 </head>
 <body>
 <d-front-matter>
     <script id='distill-front-matter' type="text/json">{
     "title": "🍷 FineWeb: decanting the web for the finest text data at scale",
-    "description": "This blog covers a discussion on processing and evaluating data quality at scale, the 🍷 FineWeb recipe (listing and explaining all of our design choices), and the process followed to create 📚 FineWeb-Edu.",
     "published": "May 28, 2024",
     "affiliation": {"name": "HuggingFace"},
     "authors": [
@@ -187,17 +82,18 @@
     <p>Both datasets are released under the permissive <a href="https://opendatacommons.org/licenses/by/1-0/">ODC-By 1.0 license</a></p>
     <p>As 🍷 FineWeb has gathered a lot of interest from the
-        community, we decided to explain in all details the steps involved in creating it as well as our processing decisions and
-        many lessons learned along the way. Hence the present (lengthy) technical report. Read on for all the juicy details on large text dataset creation!</p>
     <p><strong>TLDR:</strong> This blog covers a discussion on processing and evaluating data quality at scale, the 🍷 FineWeb
-        recipe (listing and explaining all of our design choices), and the process followed to create its 📚 FineWeb-Edu subset as well.</p>
     <h2>General considerations on web data</h2>
     <h3>Sourcing the data</h3>
     <p>A common question often asked regarding web datasets used
         to train LLMs is “where do they even get all that data?”. There are generally two options:</p>
     <ul>
-        <li>you either crawl it yourself, like companies like OpenAI or Anthropic (among other) do (see hints <a
                 href="https://platform.openai.com/docs/gptbot">here</a> and <a
                 href="https://darkvisitors.com/agents/claudebot">here</a>)
         </li>
@@ -206,15 +102,15 @@
         <li>you use a public repository of crawled webpages, like the one maintained by
             the non-profit <a href="https://commoncrawl.org/">CommonCrawl</a></li>
     </ul>
-    <p>To build and filter 🍷 FineWeb, following what had done in the past by number of LLM training teams,
       we used <a href="https://commoncrawl.org/">CommonCrawl</a> (CC) as a starting point.
         The Common Crawl non–profit organization has been crawling the web since 2007 and
-        release a new crawl containing 200 to 300 TB of textual content obtained via automatic web crawling usually
         every 1 or 2 months. </p>
-    <p>As an example, the latest CC crawl (id 2024-18) contains 2.7
-        billion web pages, totaling 386 TiB of uncompressed HTML text content (Note that the size changes from dump to dump).
-      Ninety-six crawls have been released since 2013 and 3 dumps from 2008 to 2012, which are in a different (older) format.
-      <d-footnote>We have not processed these 3 older dumps.</d-footnote> </p>
     <h3>Processing at scale</h3>
     <p>Given the sheer size of the data involved, one of the main
@@ -225,40 +121,39 @@
             href="https://github.com/huggingface/datatrove"><code>datatrove</code></a><d-cite bibtex-key="penedo2024datatrove"></d-cite>, an open-source data
         processing library that allowed us to seamlessly scale our filtering and deduplication setup to thousands of
         CPU cores. All the data processing steps involved in the creation of 🍷 FineWeb used this <a
-                href="https://github.com/huggingface/datatrove">library</a>. In most cases, you'll find the exact same scripts we used in the <a
-                href="https://github.com/huggingface/datatrove"><code>datatrove</code></a> repository.</p>
     <h3>What is good data?</h3>
     <p>This is probably the main question to keep in mind when
-        creating a dataset. In most context and in particular in the context of large language model pretraining <d-footnote>Note that all our discussion in this report is focused on the special field of web-scale datasets ("web-scale" typically meaning >100 billion tokens) used to pretrained a Large Language Models (by pretraining we mean the very first step of training of a model, starting from random weights). We don't pretend to cover any other field of dataset creation nor that the lessons or hypothesis we develop in this document can extend to any field beside this specific field.</d-footnote>, "high quality" is not a very well defined term<d-cite bibtex-key="albalak2024survey"></d-cite>, and not even a property of documents that can always be clearly perceived through direct, human, observation alone.<d-cite bibtex-key="longpre2023pretrainers"></d-cite></p>
     <p>It is still common to train a model on a given corpus considered "clean"
         (typically wikipedia<d-footnote>Even though as we mentioned above the notion of "clean" is so ill-defined that it should probably not been seen as equivalent to wikipedia-type of text</d-footnote>) and use it to check the perplexity on the dataset
         that we were trying to curate<d-cite bibtex-key="wenzek2019ccnet"></d-cite>. Unfortunately this does not always correlate with improved performance on a set of downstream
-        tasks of interest<d-cite bibtex-key="soldaini2024dolma"></d-cite>, and as a result another often used approach is to train small models<d-footnote>"Small" in comparison to standard sizes of today's LLM, i.e. small in comparison to 7-70 billion parameters. In this work "small" means about 1-2 billion parameters</d-footnote> on a representative subset of our dataset and evaluate them on
-        a set of evaluation tasks. The reason small model are used is because training models is
-        expensive and time consuming as a function of model size. In this second approach, it is important to
         choose a diverse and representative set of dataset-evaluation tasks and try not to overfit to any one individual benchmark as it would risk hurting the generality of the obtained LLM after pretraining.</p>
     <p>Yet another way to compare different datasets would be to
         train a model on each dataset and have humans rate and compare the generations of the models (like on the <a
                 href="https://chat.lmsys.org/">LMSYS Chatbot Arena</a>)<d-cite bibtex-key="chiang2024chatbot"></d-cite>. This would arguably provide the most
         reliable results in terms of representing real model usage, but getting ablation results this way is unfortunately
-        expensive and slow. It also often requires for the models to have undergone through an instruction finetuning stage to acquire conversational capabilities, as pretrained models are not directly-designed to follow instructions and thus much more sensitive to prompt details.<d-cite bibtex-key="ouyang2022training"></d-cite></p>
     <p>In this work, we went with the approach of training small
         models and evaluating them on a set of "early-signal" benchmark tasks. We believe this is a reasonable proxy for the quality
-        of the data used to train these models with the above mentioned caveat around overfitting on the evaluation benchmarks.</p>
     <h3>Ablations and evaluation setup</h3>
-    <p>To be able to compare the impact of a given processing
-        step, we typically train two models on two versions of the dataset, one version processed with the extra –ablated– step and another version with this step
-        ablated (cut/removed). Apart from the data, these two models are otherwise identical: same number of parameters, architecture hyper-parameters, and are trained
         on an equal number of randomly sampled tokens from each version of the data, for a single epoch — the only difference being thus the
-        training data. We then evaluate each model on the same set of tasks and compare average
         scores.</p>
-    <p>Our ablation models are trained using <a
             href="https://github.com/huggingface/nanotron"><code>nanotron</code></a> with this config [<strong>TODO:
-        INSERT SIMPLIFIED NANOTRON CONFIG HERE</strong>]. Ablation models have 1.82B parameters (including embeddings), used the Llama
-        architecture with a 2048 sequence length, a global batch size of ~2 million tokens and GPT2 tokenizer as mentioned above. For most
         ablations we trained on ~28B tokens (roughly the Chinchilla<d-cite bibtex-key="hoffmann2022training"></d-cite> optimal training size for this
-        model size). To confirm relative performance after several steps of filtering we conducted longer training runs in 350 billion tokens as mentioned further below.</p>
     <p>We evaluated the models using <a
             href="https://github.com/huggingface/lighteval/"><code>lighteval</code></a>. We carefully selected a set of benchmark for ablations by selecting
         benchmarks that would provide good signal at a relatively small scale ("small" models trained on only "a few
@@ -266,17 +161,17 @@
     <ul>
         <li>small variance between runs trained on different samplings of the same
             dataset: we want our runs on a subset of the data to be representative of the whole dataset, and the
-            resulting scores to have as little evaluation noise as possible and sensitive to exact data choice (apart from larger ablation that we are concerned with)
         </li>
     </ul>
     <ul>
         <li>performance increasing monotonically (or close) over a training run:
-            ideally, as the number of seen tokens increases, the performance of a high-signal benchmark should not decrease
             (which would be indicative of unreliable results at a small scale)
         </li>
     </ul>
     <ul>
-        <li>performance above the random noise level with a few standard deviations at least. Given our small ablation models and trainings we usually don't reach extremely high scores on any benchmark, but we want to make sure that the scores we get are above random noise.
         </li>
     </ul>
     <p>After consideration, we selected the following list of benchmarks:</p>
@@ -291,7 +186,7 @@
         <li>MMLU<d-cite bibtex-key="hendrycks2021measuring"></d-cite></li>
     </ul>
     <p>To
-        compute our checkpoint evaluation in a constrained time, we capped the longer benchmarks at 1000 samples (wall-clock evaluation taking less than 5
         min on a single node of 8 GPUs - done in parallel to the training).</p>
     <aside>You can find the full list of tasks and prompts we used <a
             href="https://huggingface.co/datasets/HuggingFaceFW/fineweb/blob/main/lighteval_tasks.py">here</a>.</aside>

 <!doctype html>
 <head>
     <script src="distill.bundle.js" type="module" fetchpriority="high" blocking></script>
     <script src="main.bundle.js" type="module" fetchpriority="low" defer></script>
     <meta name="viewport" content="width=device-width, initial-scale=1">
     <meta charset="utf8">
     <base target="_blank">
     <title>FineWeb: decanting the web for the finest text data at scale</title>
+    <link rel="stylesheet" href="style.css">
 </head>
 <body>
 <d-front-matter>
     <script id='distill-front-matter' type="text/json">{
     "title": "🍷 FineWeb: decanting the web for the finest text data at scale",
+    "description": "This blog covers a discussion on processing and evaluating data quality at scale, the 🍷 FineWeb recipe (listing and explaining all of our design choices), and the process followed to create its 📚 FineWeb-Edu subset.",
     "published": "May 28, 2024",
     "affiliation": {"name": "HuggingFace"},
     "authors": [
     <p>Both datasets are released under the permissive <a href="https://opendatacommons.org/licenses/by/1-0/">ODC-By 1.0 license</a></p>
     <p>As 🍷 FineWeb has gathered a lot of interest from the
+        community, we decided to explain in full detail the steps involved in creating it as well as our processing decisions and
+        many lessons learned along the way. Hence, the present (lengthy) technical report. Read on for all the juicy details on large text dataset creation!</p>
+    <aside>For the best possible reading experience, we recommend not using a mobile phone.</aside>
     <p><strong>TLDR:</strong> This blog covers a discussion on processing and evaluating data quality at scale, the 🍷 FineWeb
+        recipe (listing and explaining all of our design choices), and the process followed to create its 📚 FineWeb-Edu subset.</p>
     <h2>General considerations on web data</h2>
     <h3>Sourcing the data</h3>
     <p>A common question often asked regarding web datasets used
         to train LLMs is “where do they even get all that data?”. There are generally two options:</p>
     <ul>
+        <li>you either crawl it yourself, like companies such as OpenAI or Anthropic (among others) do (see <a
                 href="https://platform.openai.com/docs/gptbot">here</a> and <a
                 href="https://darkvisitors.com/agents/claudebot">here</a>)
         </li>
         <li>you use a public repository of crawled webpages, like the one maintained by
             the non-profit <a href="https://commoncrawl.org/">CommonCrawl</a></li>
     </ul>
+    <p>To build 🍷 FineWeb, following what has been done in the past by a number of LLM training teams,
       we used <a href="https://commoncrawl.org/">CommonCrawl</a> (CC) as a starting point.
         The Common Crawl non–profit organization has been crawling the web since 2007 and
+        releases a new crawl containing 200 to 400 TiB of textual content obtained via automatic web crawling usually
         every 1 or 2 months. </p>
+    <p>As an example, the latest CC crawl (April 2024) contains 2.7
+        billion web pages, totaling 386 TiB of uncompressed HTML text content<d-footnote>Note that the size changes from crawl to crawl</d-footnote>.
+      Ninety-six crawls have been released since 2013 and 3 crawls from 2008 to 2012, which are in a different (older) format.
+      <d-footnote>We have not processed these 3 older crawls.</d-footnote> </p>
     <h3>Processing at scale</h3>
     <p>Given the sheer size of the data involved, one of the main
             href="https://github.com/huggingface/datatrove"><code>datatrove</code></a><d-cite bibtex-key="penedo2024datatrove"></d-cite>, an open-source data
         processing library that allowed us to seamlessly scale our filtering and deduplication setup to thousands of
         CPU cores. All the data processing steps involved in the creation of 🍷 FineWeb used this <a
+                href="https://github.com/huggingface/datatrove">library</a>. You will find the exact scripts we used in the
+        <a href="https://github.com/huggingface/datatrove/blob/main/examples/fineweb.py"><code>datatrove</code> repository</a>.</p>
     <h3>What is good data?</h3>
     <p>This is probably the main question to keep in mind when
+        creating a dataset. In most contexts and, in particular, in the context of large language model pretraining <d-footnote>Note that this report is focused on the special field of web-scale datasets ("web-scale" typically meaning >100 billion tokens obtained from the web) used to pretrain a Large Language Model (by pretraining we mean the very first step in the training of a model, starting from random weights). We don't pretend to cover any other field of dataset creation nor that the lessons or hypothesis we develop in this document can extend to any field besides this specific field.</d-footnote>, "high quality" is not a very well defined term<d-cite bibtex-key="albalak2024survey"></d-cite>, and not even a property of documents that can always be clearly perceived through direct human observation alone.<d-cite bibtex-key="longpre2023pretrainers"></d-cite></p>
     <p>It is still common to train a model on a given corpus considered "clean"
         (typically wikipedia<d-footnote>Even though as we mentioned above the notion of "clean" is so ill-defined that it should probably not been seen as equivalent to wikipedia-type of text</d-footnote>) and use it to check the perplexity on the dataset
         that we were trying to curate<d-cite bibtex-key="wenzek2019ccnet"></d-cite>. Unfortunately this does not always correlate with improved performance on a set of downstream
+        tasks of interest<d-cite bibtex-key="soldaini2024dolma"></d-cite>, and as a result another often used approach is to train small models<d-footnote>"Small" in comparison to standard sizes of today's LLMs, i.e. small in comparison to 7-70 billion parameters. In this work "small" means about 1-2 billion parameters</d-footnote> on a representative subset of our dataset and evaluate them on
+        a set of evaluation tasks. Small models are used because training costs and time are a function of model size. In this second approach, it is important to
         choose a diverse and representative set of dataset-evaluation tasks and try not to overfit to any one individual benchmark as it would risk hurting the generality of the obtained LLM after pretraining.</p>
     <p>Yet another way to compare different datasets would be to
         train a model on each dataset and have humans rate and compare the generations of the models (like on the <a
                 href="https://chat.lmsys.org/">LMSYS Chatbot Arena</a>)<d-cite bibtex-key="chiang2024chatbot"></d-cite>. This would arguably provide the most
         reliable results in terms of representing real model usage, but getting ablation results this way is unfortunately
+        expensive and slow. It also often requires for the models to have undergone an instruction finetuning stage to acquire conversational capabilities, as pretrained models are not directly designed to follow instructions and are thus much more sensitive to prompt details.<d-cite bibtex-key="ouyang2022training"></d-cite></p>
     <p>In this work, we went with the approach of training small
         models and evaluating them on a set of "early-signal" benchmark tasks. We believe this is a reasonable proxy for the quality
+        of the data used to train these models, when keeping in mind the above-mentioned caveat around overfitting on the evaluation benchmarks.</p>
     <h3>Ablations and evaluation setup</h3>
+    <p>To compare the impact of a given processing
+        step, we trained two models on two versions of the dataset, one version processed with the extra step (the one we wish to evaluate) and another version with this step
+        ablated (cut/removed). Apart from the data, these two models would be otherwise identical: the same number of parameters, architecture hyper-parameters, and trained
         on an equal number of randomly sampled tokens from each version of the data, for a single epoch — the only difference being thus the
+        training data. We then evaluated each model on the same set of tasks and compared average
         scores.</p>
+    <p>Our ablation models were trained using <a
             href="https://github.com/huggingface/nanotron"><code>nanotron</code></a> with this config [<strong>TODO:
+        INSERT SIMPLIFIED NANOTRON CONFIG HERE</strong>]. Our ablation models have 1.82B parameters (including embeddings), used the Llama
+        architecture with a 2048 sequence length, a global batch size of ~2 million tokens, and the GPT2 tokenizer. For most
         ablations we trained on ~28B tokens (roughly the Chinchilla<d-cite bibtex-key="hoffmann2022training"></d-cite> optimal training size for this
+        model size). To confirm relative performance improvements after each step of filtering we conducted longer training runs on 350 billion tokens as mentioned further below.</p>
     <p>We evaluated the models using <a
             href="https://github.com/huggingface/lighteval/"><code>lighteval</code></a>. We carefully selected a set of benchmark for ablations by selecting
         benchmarks that would provide good signal at a relatively small scale ("small" models trained on only "a few
     <ul>
         <li>small variance between runs trained on different samplings of the same
             dataset: we want our runs on a subset of the data to be representative of the whole dataset, and the
+            resulting scores to have as little sensitivity to exact data choice as possible (apart from larger ablations that we are concerned with)
         </li>
     </ul>
     <ul>
         <li>performance increasing monotonically (or close) over a training run:
+            ideally, as the number of seen tokens increases, the performance on a high-signal benchmark should not decrease
             (which would be indicative of unreliable results at a small scale)
         </li>
     </ul>
     <ul>
+        <li>performance above random baseline for this task by at least a few standard deviations: given our small ablation models and trainings we usually don't reach extremely high scores on any benchmark, but we want to make sure that the scores we get are above random noise.
         </li>
     </ul>
     <p>After consideration, we selected the following list of benchmarks:</p>
         <li>MMLU<d-cite bibtex-key="hendrycks2021measuring"></d-cite></li>
     </ul>
     <p>To
+        ensure our checkpoint evaluation stayed within a limited timeframe, we capped the longer benchmarks at 1000 samples (wall-clock evaluation taking less than 5
         min on a single node of 8 GPUs - done in parallel to the training).</p>
     <aside>You can find the full list of tasks and prompts we used <a
             href="https://huggingface.co/datasets/HuggingFaceFW/fineweb/blob/main/lighteval_tasks.py">here</a>.</aside>

src/style.css CHANGED Viewed

@@ -141,4 +141,103 @@ d-byline .byline {
 d-contents > nav a.active {
     text-decoration: underline;
 }

 d-contents > nav a.active {
     text-decoration: underline;
+}
+@media (max-width: 1199px) {
+    d-contents {
+        display: none;
+        justify-self: start;
+        align-self: start;
+        padding-bottom: 0.5em;
+        margin-bottom: 1em;
+        padding-left: 0.25em;
+        border-bottom: 1px solid rgba(0, 0, 0, 0.1);
+        border-bottom-width: 1px;
+        border-bottom-style: solid;
+        border-bottom-color: rgba(0, 0, 0, 0.1);
+    }
+}
+d-contents a:hover {
+    border-bottom: none;
+}
+@media (min-width: 1200px) {
+    d-article {
+        /* Ensure d-article does not prevent sticky positioning */
+        overflow: visible;
+    }
+    d-contents {
+        align-self: start;
+        grid-column-start: 1 !important;
+        grid-column-end: 4 !important;
+        grid-row: auto / span 6;
+        justify-self: end;
+        margin-top: 0em;
+        padding-right: 3em;
+        padding-left: 2em;
+        border-right: 1px solid rgba(0, 0, 0, 0.1);
+        border-right-width: 1px;
+        border-right-style: solid;
+        border-right-color: rgba(0, 0, 0, 0.1);
+        position: -webkit-sticky; /* For Safari */
+        position: sticky;
+        top: 10px; /* Adjust this value if needed */
+    }
+}
+d-contents nav h3 {
+    margin-top: 0;
+    margin-bottom: 1em;
+}
+d-contents nav div {
+    color: rgba(0, 0, 0, 0.8);
+    font-weight: bold;
+}
+d-contents nav a {
+    color: rgba(0, 0, 0, 0.8);
+    border-bottom: none;
+    text-decoration: none;
+}
+d-contents li {
+    list-style-type: none;
+}
+d-contents ul, d-article d-contents ul {
+    padding-left: 1em;
+}
+d-contents nav ul li {
+    margin-bottom: .25em;
+}
+d-contents nav a:hover {
+    text-decoration: underline solid rgba(0, 0, 0, 0.6);
+}
+d-contents nav ul {
+    margin-top: 0;
+    margin-bottom: 6px;
+}
+d-contents nav > div {
+    display: block;
+    outline: none;
+    margin-bottom: 0.5em;
+}
+d-contents nav > div > a {
+    font-size: 13px;
+    font-weight: 600;
+}
+d-contents nav > div > a:hover,
+d-contents nav > ul > li > a:hover {
+    text-decoration: none;
 }