Quality at a Glance:
An Audit of Web-Crawled Multilingual Datasets
Julia Kreutzera;b, Isaac Caswella, Lisa Wanga, Ahsan Wahabc, 
Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets
With the success of large-scale pre-training and multilingual modeling in
Natural Language Processing (NLP), recent years have seen a proliferation of
large, web-mined text datasets covering hundreds of languages. We manually
audit the quality of 205 language-specific corpora released with five major
public datasets (CCAligned, ParaCrawl, WikiMatrix, OSCAR, mC4). Lower-resource
corpora have systematic issues: At least 15 corpora have no usable text, and a
significant fraction contains less than 50% sentences of acceptable quality. In
addition, many are mislabeled or use nonstandard/ambiguous language codes. We
demonstrate that these issues are easy to detect even for non-proficient
speakers, and supplement the human audit with automatic analyses. Finally, we
recommend techniques to evaluate and improve multilingual corpora and discuss
potential risks that come with low-quality data releases.
Julia Kreutzer Isaac Caswell Lisa Wang Ahsan Wahab Daan van Esch Nasanbayar Ulzii-Orshikh Allahsera Tapo Nishant Subramani Artem Sokolov Claytone Sikasote Monang Setyawan Supheakmungkol Sarin Sokhar Samb Benoît Sagot Clara Rivera Annette Rios Isabel Papadimitriou Salomey Osei Pedro Ortiz Suarez Iroro Orife Kelechi Ogueji Andre Niyongabo Rubungo Toan Q. Nguyen Mathias Müller André Müller Shamsuddeen Hassan Muhammad Nanda Muhammad Ayanda Mnyakeni Jamshidbek Mirzakhalov Tapiwanashe Matangira Colin Leong Nze Lawson Sneha Kudugunta Yacine Jernite Mathias Jenny Orhan Firat Bonaventure F. P. Dossou Sakhile Dlamini Nisansa de Silva Sakine Çabuk Ballı Stella Biderman Alessia Battisti Ahmed Baruwa Ankur Bapna Pallavi Baljekar Israel Abebe Azime Ayodele Awokoya Duygu Ataman Orevaoghene Ahia Oghenefego Ahia Sweta Agrawal Mofetoluwa Adeyemi