The ever-growing diversity of pre-training text corpora has equipped language models with generalization capabilities across various downstream tasks. However, such diverse datasets are often too large for academic budgets; hence, most research on Transformer architectures, training procedures, optimizers, etc. gets conducted on smaller, homogeneous datasets. To this end, we presentThe MiniPile Challenge, where one pre-trains a language model on a diverse text corpus containing at most 1M documents.MiniPileis a 6GB subset of the deduplicated 825GBThe Pilecorpus. To curateMiniPile, we perform a simple, three-step data filtering process: we (1)  infer embeddings for all documents of the Pile, (2)  cluster the embedding space using-means, and (3)  filter out low-quality clusters. To verifyMiniPile’s suitability for language model pre-training, we use it to pre-train a BERT and T5 model, yielding a performance drop of only/on the GLUE and SNI benchmarks compared to the original pre-trained checkpoints trained on 2.6x/745x the amount of data.MiniPileis available athuggingface.co/datasets/jeankaddour/minipile.