Leo Gao &ndash; EleutherAI
While conventional wisdom suggests that more aggressively filtering data from low-quality sources like Common Crawl always monotonically improves the quality of training data, we find that aggressive filtering can in fact lead to a decrease in model quality on a wide array of downstream tasks for a GPT-like language model. We speculate that this is because optimizing sufficiently strongly for a proxy metric harms performance on the true objective, suggesting a need for more robust filtering objectives when attempting to filter more aggressively. We hope this work leads to detailed analysis of the effects of dataset filtering design choices on downstream model performance in future work.
The recent proliferation of ever larger language models has led to increasing demands on training data (Radford et al., 2018, 2019; Gokaslan and Cohen, 2019; Rosset, 2019; Shoeybi et al., 2019; Devlin et al., 2019; Liu et al., 2019; Raffel et al., 2020; Brown et al., 2020; Zeng et al., 2021). This data is increasingly derived from internet corpora like Common Crawl (Radford et al., 2019; Ortiz Suárez et al., 2019; Wenzek et al., 2020; Conneau et al., 2020; Brown et al., 2020; Gao et al., 2020; Raffel et al., 2020). However, the quality of raw Common Crawl data is often insufficient to be directly used. To combat this, many existing works use some kind of proxy for quality, like a classifier between known high quality data and low quality data (Brown et al., 2020; Gao et al., 2020; Zeng et al., 2021), handcrafted heuristics (Yang et al., 2020; Raffel et al., 2020), or keeping only documents with perplexity scores that fall in some middle quantile of an existing language model (Wenzek et al., 2020). Brown et al. (2020) in particular filter extremely aggres- sively using their classifier, discarding about 98.7% of their data. Previous work has shown that models trained on heuristic-filtered datasets perform better on downstream tasks (Raffel et al., 2020). However, Gao et al. (2020) show that a perplexity-filtered CC- derived dataset actually performs worse than unfiltered CC on certain tasks. [...] We hypothesize that this decline in performance is because of misalignment between the classifier objective, intended to be a proxy for quality, and actual document quality. For instance, a classifier to distinguish WebText2 from Common Crawl, as in GPT-3, would also exclude domains of text data not found as often in WebText2.