[1]} makes the case that researchers building language models should be purposeful in curating training datasets, as curation choices are effectively world design choices. Corpora built on top of news scrapes snapshotted at a specific point in time will capture all of the inherent social bias and structural issues related to news reporting at a given point in time. [2]} demonstrate that models with fixed sizes will be capacity-limited, further highlighting the need for careful data curation: in practice, most deployed language models are fixed-size, and care must be taken to ensure that they are learning from the highest-quality data possible to make the best use of their capacity.
