🍷 FineWeb: decanting the web for the finest text data at scale
We have recently released 🍷 FineWeb, our new large scale (15T gpt2 tokens, 44TB disk space) dataset of clean text sourced from the web for LLM pretraining. You can download it here.
The performance of a large language model (LLM) depends heavily on the quality and size of its pretraining dataset. However, the pretraining datasets for state-of-the-art open LLMs like Llama 3
🍷 FineWeb, a 15-trillion token dataset derived from 96 CommonCrawl snapshots, produces better-performing LLMs than other open pretraining datasets. To advance the understanding of how best to curate high-quality pretraining datasets, we carefully document and ablate all of the design choices used in FineWeb, including in-depth investigations of deduplication and filtering strategies.
We are also excited to announce the release of 📚 FineWeb-Edu, a version of 🍷 FineWeb that was filtered for educational content, available in two sizes: 1.3 trillion (very high quality) and 5.4 trillion (high quality) tokens. 📚 FineWeb-Edu outperforms all existing public web datasets, with models pretrained on it showing notable improvements on knowledge- and reasoning-intensive benchmarks like MMLU, ARC, and OpenBookQA. You can download it here.
Both datasets are released under the permissive ODC-By 1.0 license
As 🍷 FineWeb has gathered a lot of interest from the community, we decided to further explain the steps involved in creating it, our processing decisions and some lessons learned along the way. Read on for all the juicy details on large text dataset creation!
TLDR: This blog covers a discussion on processing and evaluating data quality at scale, the 🍷 FineWeb recipe (listing and explaining all of our design choices), and the process followed to create 📚 FineWeb-Edu.
A common question we see asked regarding web datasets used to train LLMs is “where do they even get all that data?” There are generally two options:
For 🍷 FineWeb, similarly to what was done for a large number of other public datasets, we used CommonCrawl as a starting point. They have been crawling the web since 2007 (long before LLMs became widespread) and release a new dump usually every 1 or 2 months, which can be freely downloaded.
As an example, their latest crawl (2024-18) contains 2.7
billion web pages, totaling 386 TiB of uncompressed HTML text content (the size changes from dump to dump). There
are 96 dumps since 2013 and 3 dumps from 2008 to 2012, which are in a different (older) format.
Given the sheer size of the data involved, one of the main challenges we had to overcome was having a modular, scalable codebase that would allow us to quickly iterate on our processing decisions and easily try out new ideas, while appropriately parallelizing our workloads and providing clear insights into the data.
For this purpose, we developed datatrove
This is probably the main question to keep in mind when
creating a dataset. In the context of large language model pretraining, "high quality" is not a very well defined term
It is still common to train a model on a given corpus
(wikipedia, or some other web dataset considered clean) and use it to check the perplexity on the dataset
that we were trying to curate
Another way to evaluate different datasets would be to
train a model on each one and have humans rate and compare their outputs (like on the LMSYS Chatbot Arena)
The approach we ultimately went with was to train small models and evaluate them on a set of benchmark tasks. We believe this is a reasonable proxy for the quality of the data used to train these models.
To be able to compare the impact of a given processing step, we would train 2 models, one where the data included the extra step and another where this step was ablated (cut/removed). These 2 models would have the same number of parameters, architecture, and be trained on an equal number of randomly sampled tokens from each step's data, for a single epoch, and with the same hyperparameters — the only difference would be in the training data. We would then evaluate each model on the same set of tasks and compare the average scores.
Our ablation models were trained using nanotron with this config [TODO:
INSERT SIMPLIFIED NANOTRON CONFIG HERE]. The models had 1.82B parameters, used the Llama
architecture with a 2048 sequence length, and a global batch size of ~2 million tokens. For filtering
ablations we mostly trained on ~28B tokens (which is roughly the Chinchilla
We evaluated the models using lighteval. We tried selecting
benchmarks that would provide good signal at a relatively small scale (small models trained on only a few
billion tokens). Furthermore, we also used the following criteria when selecting benchmarks:
We selected the following list of benchmarks:
To have results quickly we capped longer benchmarks at 1000 samples (wall-clock evaluation taking less than 5 min on a single node of 8 GPUs - done in parallel to the training).
In the next subsections we will explain each of the steps taken to produce the FineWeb dataset.
CommonCrawl data is available in two main formats: WARC and WET. WARC (Web ARChive format) files contain the raw data from the crawl, including the full page HTML and request metadata. WET (WARC Encapsulated Text) files provide a text only version of those websites.
A large number of datasets take the WET files as their
starting point. In our experience the default text extraction (extracting the main text of a webpage from
its HTML) used to create these WET files is suboptimal and there are a variety of open-source libraries that
provide better text extraction (by, namely, keeping less boilerplate content/navigation menus). We extracted
the text content from the WARC files using the trafilatura library
To validate this decision, we processed the 2019-18 dump
directly using the WET files and with text extracted from WARC files using trafilaturafavour_precision=True.
It is important to note, however, that text extraction is one of the most costly steps of our processing, so we believe that using the readily available WET data could be a reasonable trade-off for lower budget teams.
Filtering is an important part of the curation process. It removes part of the data (be it words, lines, or full documents) that would harm performance and is thus deemed to be “lower quality”.
As a basis for our filtering we used part of the setup
from RefinedWeb
After applying this filtering to each of the text
extracted dumps (there are currently 96 dumps) we obtained roughly 36 trillion tokens of data (when
tokenized with the gpt2 tokenizer).
Deduplication is one of the most important steps when creating large web datasets for LLM pretraining. Methods to deduplicate datasets attempt to identify and remove redundant/repeated data from the dataset.
The web has many aggregators, mirrors, templated pages or just otherwise repeated content spread over different domains and webpages. Often, these duplicated pages can be introduced by the crawler itself, when different links point to the same page.
Removing these duplicates (deduplicating) has been linked to an improvement in model performance
There are different ways to identify and even define duplicated data. Common approaches rely on hashing techniques to speed up the process, or on building efficient data structures to index the data (like suffix arrays). Methods can also be “fuzzy”, by using some similarity metric to mark documents as duplicates, or “exact” by checking for exact matches between two documents (or lines, paragraphs, or whatever other granularity level being used).
Similarly to RefinedWeb
This would mean that for two documents with a similarity ($$s$$) of 0.7, 0.75, 0.8 and 0.85, the probability that they would be identified as duplicates would be 56%, 77%, 92% and 98.8% respectively ($$1-(1-s^8)^{14}$$). See the plot below for a match probability comparison between our setup with 112 hashes and the one from RefinedWeb, with 9000 hashes, divided into 450 buckets of 20 hashes (that requires a substantially larger amount of compute resources, as each individual hash must be computed, stored and then compared with hashes from other documents):
While the high number of hash functions in RefinedWeb allows for a steeper, more well defined cut off (documents with real similarity near the threshold are more likely to be correctly identified), we believe the compute and storage savings are a reasonable trade off.
It should also be noted that intra-document deduplication is already handled by our repetition filter, which removes documents with many repeated lines and paragraphs.
Our initial approach was to take the entire dataset (all 90+ dumps) and deduplicate them together as one big dataset using MinHash.
We did this in an iterative manner: starting with the most recent dump (which at the time was 2023-50) and proceeding chronologically until the oldest one, we would deduplicate each dump not only within itself, but we would also remove any matches with documents from the previously processed (more recent) dumps.
For instance, for the second most recent dump (2023-40 at the time), we deduplicated it against the most recent one in addition to within itself. In particular, the oldest dump was deduplicated against all other dumps. As a result, more data was removed from the oldest dumps (last to be deduplicated) than from the most recent ones.
Deduplicating the dataset in this manner resulted in 4 trillion tokens of data, but, quite surprisingly for us, when training on a randomly sampled 350 billion tokens subset, the model showed no improvement over one trained on the non deduplicated data (see orange and green curves below), scoring far below its predecessor RefinedWeb on our aggregate of tasks.
This was quite puzzling as our intuition regarding web data was that more deduplication would always result in improved performance. We decided to take a closer look at one of the oldest dumps, dump 2013-48:
As an experiment, we tried training two models on 28 billion tokens sampled from the following data from 2013-48:
These results show that, for this older dump from which we had removed over 90% of the original data, the data that was kept was actually worse than the data removed (considered independently of all the other dumps). This is also confirmed by visual inspection: originally kept data contains far more ads, lists of keywords and generally badly formatted text than originally removed data.
We then tried an alternative approach: we deduplicated each dump with MinHash individually (without considering the other dumps). This resulted in 20 trillion tokens of data.
When training on a random sample from this dataset we see that it now matches RefinedWeb’s performance (blue and red curves below):
We hypothesize that the main improvement gained from deduplication is the removal of very large clusters that are present in every single dump (you will find some examples of these clusters on the RefinedWeb paper, each containing hundreds of thousands of documents) and that further deduplication for clusters with a low number of duplicates (less than ~100 i.e. the number of dumps) actually harms performance: data that does not find a duplicate match in any other dump might actually be worse quality/more out of distribution (as evidenced by the results on the 2013-48 data).
While you might see some performance improvement when deduplicating a few dumps together, at the scale of the entire dataset (all the dumps), the effect from this upsampling of lower quality data side effect seems to be more impactful.
One possibility to consider is that as filtering quality improves, this effect may not be as prevalent, since the filtering might be able to remove some of this lower quality data. We also experimented with applying different, and often “lighter”, deduplication approaches on top of the individually deduplicated dumps. You can read about them further below.
Given the nature of deduplication, its effect is not always very visible in a smaller slice of the dataset (such as 28B tokens, the size we used for our filtering ablations). Furthermore, one must consider the fact that there are specific effects at play when deduplicating across all CommonCrawl dumps, as some URLs/pages are recrawled from one dump to the next.
To visualize the effect of scaling the number of training tokens on measuring deduplication impact, we considered the following (very extreme and unrealistic regarding the degree of duplication observed) theoretical scenario:
We then simulated uniformly sampling documents from this entire dataset of 20 trillion tokens, to obtain subsets of 1B, 10B, 100B, 350B and 1T tokens. In the image below you can see how often each document would be repeated.
For 1B almost all documents would be unique (#duplicates=1), despite the fact that in the entire dataset each document is repeated 100 times (once per dump). We start seeing some changes at the 100B scale (0.5% of the total dataset), with a large number of documents being repeated twice, and a few even 4-8 times. At the larger scale of 1T (5% of the total dataset), the majority of the documents are repeated up to 8 times, with some being repeated up to 16 times.
We ran our performance evaluations for the deduplicated data at the 350B scale, which would, under this theoretical scenario, be made up of a significant portion of documents duplicated up to 8 times. This simulation illustrates the inherent difficulties associated with measuring deduplication impact on the training of LLMs, once the biggest duplicate clusters have been removed.
We attempted to improve the performance of the independently minhash deduped 20 trillion tokens of data by further deduplicating it (globally, over all dumps) with the following methods:
The performance of the models trained on each of these was consistently worse (even if to different degrees) than that of the original independently deduplicated data:
By this point we had reached the same performance as
RefinedWeb with base filtering + independent MinHash, but on our aggregate of tasks, another heavily filtered dataset, the C4 dataset
We therefore set out to find new filtering steps that would, at first, allow us to match the performance of C4 and, at a second stage, surpass it. A natural starting point was to look into the processing of C4 itself.
The C4
dataset was first released in 2019. It was obtained from the 2019-18 CommonCrawl dump by
removing non english data, applying some heuristic filters on both the line and document level,
deduplicating on the line level, and removing documents containing words from a word blocklist.
Despite its age and limited size for current standards (around 175B gpt2 tokens), this dataset is, to this day, a common sub-set of typical LLM training, being used in models such as the relatively recent Llama1
{) allows us to match C4’s HellaSwag performance (purple versus
pink curves).
We decided to apply all C4 filters mentioned above except the terminal punctuation one. We validated these results with a longer run, which you will find in a plot in the next section.
To develop new heuristic filters and select their thresholds we devised a systematic process:
Due to our assumption that global MinHash greatly upsamples lower quality data in the oldest dumps, we computed metrics on both the independently MinHashed and the (worse quality) global MinHashed versions of the 2013-48 and 2015-22 crawls (two older crawls). We then compared the statistics at a macro level, by looking at the distribution of these metrics for each one.
Perhaps not too surprisingly given our findings for deduplication, we found significant
disparities in most of the metrics for the two deduplication methods. For instance, the line-char-duplicates
metric (nb. of characters in duplicated lines / nb. characters), roughly doubled from the independent dedup
(0.0053 for 2015-22 and 0.0058 for 2013-48), to the global dedup (0.011 for 2015-22 and 0.01 for 2013-48),
indicating that the latter had higher inter-document repetition.
Following the process listed above for these datasets yielded 17 candidate metric-threshold pairs. In the image below, you can see 3 of these histograms:
As an example, we inspected the histograms of "fraction of lines ending with punctuation" (see the image above) and observed an increased document density of global MinHash at around 0.12. We then filtered with this threshold and found that the removed data had a higher amount of short lists or consisted of only document layout text ("Home", "Sign up", etc).
We then assessed the effectiveness of these 17 newly created filters, by conducting 28B tokens ablation runs on the 2019-18 crawl. Out of all those runs, we identified three filters (the ones based on the histograms above) that demonstrated the most significant improvements on the aggregate score:
These filters allowed us to further improve performance and to, notably, surpass the C4 dataset performance.
The final 🍷 FineWeb dataset comprises 15T tokens and includes the following previously mentioned steps, in order, each providing a performance boost on our group of benchmark tasks:
We compared 🍷 FineWeb with the following datasets:
You will find these models on this collection. We have uploaded checkpoints at every 1000 training steps. You will also find our full evaluation results here.
Large language models pretrained on 🍷 FineWeb, the largest publicly available clean LLM pretraining dataset, are better-performing than other open pretraining datasets.
A new approach has recently emerged for filtering LLM training datasets: using synthetic data to develop classifiers for identifying educational content. This technique was used in the trainings of Llama 3
The popular Phi3 models were trained on 3.3 and 4.8 trillion tokens, with the paper
Our training data consists of heavily filtered publicly available web data (according to the 'educational level') from various open internet sources, as well as synthetic LLM-generated data.
Similarly, Llama 3 blog post
We found that previous generations of Llama are good at identifying high-quality data, so we used Llama 2 to help build the text-quality classifiers that are powering Llama 3.
However, these classifiers and filtered datasets are not publicly available. To further enhance 🍷 FineWeb's quality, we developed an educational quality classifier using annotations generated by Llama-3-70B-Instruct to create 📚 FineWeb-Edu.
We used Llama-3-70B-Instruct to annotate 500k samples from 🍷 FineWeb, scoring each for their educational quality on a scale from 0 to 5.
We explored various prompts and found that the additive scale by Yuan et al.
We also experimented with Mixtral-8x-7B-Instruct and Mixtral-8x22B-Instruct and a jury of all three models
We added a classification head with a single regression output to Snowflake-arctic-embed and trained it on 450,000 Llama 3 annotations for 20 epochs with a learning rate of 3e-4, freezing the embedding and encoder layers. We saved the checkpoint with the highest F1 score on our held-out validation set of 45k samples, treating Llama 3 annotations as ground-truth. After training, we rounded the scores to integers from 0 to 5.
We then converted the problem to a binary classification task by using a fixed threshold to determine if a file is educational. With a threshold of 3, the model achieved an F1 score of 82% on the validation set, indicating strong performance in distinguishing high-quality educational content.
The classifier is available at: HuggingFaceFW/fineweb-edu-classifier. The training and inference code is available on GitHub.
We applied the classifier to the 15T tokens of 🍷 FineWeb, a process that required 6,000 H100 GPU hours. We investigated the impact of using different thresholds for the filtering and found that threshold 3 gave the best overall results. Although using a threshold higher than 3 improves performance on knowledge and reasoning intensive benchmarks, it significantly degrades performance on HellaSwag and PIQA. The plot below shows the performance of each threshold compared to FineWeb on six different benchmarks; it uses a 1.82B model trained on 8B tokens.
We then built 📚 FineWeb-Edu by filtering out samples with scores lower than 3. This removed 92% of the dataset, leaving us with 1.3 trillion educational tokens. To evaluate the effectiveness of this filtering at a larger scale, we conducted an ablation using a 1.82B model trained on 350 billion tokens, similar to the FineWeb filtering ablation mentioned above:
Here are the key highlights of the ablation results above:
Given that a threshold of 2 also demonstrated strong performance while retaining more data, we are releasing an additional dataset filtered with this threshold, containing 5.4 trillion tokens under HuggingFaceFW/fineweb-edu-score-2.
You can find the two datasets along with the classifier used for the filtering in this collection.
Through our open data efforts we hope to give every model trainer the ability to create state-of-the-art large language models. As part of this process, we plan to continue iterating on FineWeb and to release more specialised filtered subsets of web data, in a fully open and reproducible manner.
While English currently dominates the large language model landscape, we believe that making high quality training data for other languages more easily accessible would allow millions of non english speakers to benefit from these technologies and, as such, will also strive to adapt the FineWeb Recipe to a multilingual version.