Safety in PretrainingAligning with standard pretraining data safety practices [5; 58; 77], we build a set of filters based on heuristic rules, keyword matching, and learned classifiers to remove text containing personal identifiers and private data, and reduce sexual, violent, and extremist content.