Motivated by the intuition that these duplicate-driven clusters need tshould be pruned, as well as the recent success of pruning methods in vision and vision-language models [1, 47], we focus our efforts on data selection strategies that manipulate data points based on their position in an embedding space. We embed each document by feeding it into a 125M OPT model and use the last-layer embedding of the last token (we experiment with different embedding spaces in Section A.7).