In recent years, data quality has emerged as an important factor for training massive models. Analytical theories suggest that higher-quality data can lead to lower test errors in models trained on a fixed data budget. Moreover, a model can be trained on a lower compute budget without compromising performance if a dataset can be stripped of its redundancies. Coreset selection (or data pruning)  seeks to select a subset of the training data so as to maximize the performance of models trained on this subset, also referred to as coreset. There are two dominant approaches: (1)  geometry-based data selection for maximizingdata diversityin the coreset, and (2)  functions that assigndifficulty scoresto samples based on training dynamics. Optimizing for data diversity leads to a coreset that is biased towards easier samples, whereas, selection by difficulty ranking omits easy samples that are necessary for the training of deep learning models. This demonstrates that data diversity and importance scores are two complementary factors that need to be jointly considered during coreset selection. In this work, we represent a dataset as an undirected graph and propose a novel pruning algorithm,Pruning, that uses forward and reverse message passing over this dataset graph for coreset selection.Pruningupdates the difficulty scores of each example by incorporating the difficulty of its neighboring examples in the dataset graph. Then, these updated difficulty scores direct a graph-based sampling method to select a coreset that encapsulates both diverse and difficult regions of the dataset space. We evaluate supervised and self-supervised versions of our method on various vision and language datasets. Results show thatPruningimproves coreset selection over previous state-of-the-art methods for up to 70% pruning rates. Additionally, we find that usingPruningfor filtering large multimodal datasets leads to increased diversity in the dataset and improved generalization of pretrained models. Our work shows thatPruningis a versatile framework for understanding and processing datasets.111Our code is available athttps://github.com/adymaharana/d2pruning