An effort similar to this work has been made for the English language with the introduction of Pile in 2020 [1]}. The Pile is an 825 GB large and diverse corpus that is composed of 22 smaller and high-quality sources. The diversity in data-sources improves general cross-domain knowledge of both the LM and the downstream generalization capability. Pile covers the diverse knowledge and reasoning ability in a number of different domains which makes it a robust benchmark for general text modeling ability for large LM.
