@inproceedings{van-noord-etal-2024-language,
    title = "Do Language Models Care about Text Quality? Evaluating Web-Crawled Corpora across 11 Languages",
    author = "van Noord, Rik  and
      Kuzman, Taja  and
      Rupnik, Peter  and
      Ljube{\v{s}}i{\'c}, Nikola  and
      Espl{\`a}-Gomis, Miquel  and
      Ram{\'\i}rez-S{\'a}nchez, Gema  and
      Toral, Antonio",
    editor = "Calzolari, Nicoletta  and
      Kan, Min-Yen  and
      Hoste, Veronique  and
      Lenci, Alessandro  and
      Sakti, Sakriani  and
      Xue, Nianwen",
    booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.lrec-main.465",
    pages = "5221--5234",
    abstract = "Large, curated, web-crawled corpora play a vital role in training language models (LMs). They form the lion{'}s share of the training data in virtually all recent LMs, such as the well-known GPT, LLaMA and XLM-RoBERTa models. However, despite this importance, relatively little attention has been given to the quality of these corpora. In this paper, we compare four of the currently most relevant large, web-crawled corpora (CC100, MaCoCu, mC4 and OSCAR) across eleven lower-resourced European languages. Our approach is two-fold: first, we perform an intrinsic evaluation by performing a human evaluation of the quality of samples taken from different corpora; then, we assess the practical impact of the qualitative differences by training specific LMs on each of the corpora and evaluating their performance on downstream tasks. We find that there are clear differences in quality of the corpora, with MaCoCu and OSCAR obtaining the best results. However, during the extrinsic evaluation, we actually find that the CC100 corpus achieves the highest scores. We conclude that, in our experiments, the quality of the web-crawled corpora does not seem to play a significant role when training LMs.",
}

Large, curated, web-crawled corpora play a vital role in training language models (LMs). They form the lion{'}s share of the training data in virtually all recent LMs, such as the well-known GPT, LLaMA and XLM-RoBERTa models. However, despite this importance, relatively little attention has been given to the quality of these corpora. In this paper, we compare four of the currently most relevant large, web-crawled corpora (CC100, MaCoCu, mC4 and OSCAR) across eleven lower-resourced European languages. Our approach is two-fold: first, we perform an intrinsic evaluation by performing a human evaluation of the quality of samples taken from different corpora; then, we assess the practical impact of the qualitative differences by training specific LMs on each of the corpora and evaluating their performance on downstream tasks. We find that there are clear differences in quality of the corpora, with MaCoCu and OSCAR obtaining the best results. However, during the extrinsic evaluation, we actually find that the CC100 corpus achieves the highest scores. We conclude that, in our experiments, the quality of the web-crawled corpora does not seem to play a significant role when training LMs.
van Noord, Rik Kuzman, Taja Rupnik, Peter Ljube{\v{s}}i{\'c}, Nikola Espl{\`a}-Gomis, Miquel Ram{\'\i}rez-S{\'a}nchez, Gema Toral, Antonio
Do Language Models Care about Text Quality? Evaluating Web-Crawled Corpora across 11 Languages