Gilles Adda, Annelies Braffort, Ioana Vasilescu, François Yvon &ndash; Université Paris-Saclay, CNRS, LISN, Paris, France
The CommonCrawl project³⁷ [³⁷https://commoncrawl.org/] aggregates Web crawled data that is orders or magnitude larger than these resources for many languages; furthermore this corpus is being updated on a regular basis. By using parts of the French subset of CommonCrawl, possibly conjoined with the more curated corpora alluded to above has enabled to train large-scale BERT-style Language Models (LMs) – FlauBERT (Le et al., 2020) is built with a corpus containing about 12B running words, CamemBERT (Martin et al., 2020) uses the 22B words OSCAR, and these numbers continue to grow, albeit at a much slower pace than the corresponding English cor- pora.