As described in [1]}, it is crucial to generate datasets for training language models that are free of duplicity. This makes it possible to train models that do not memorise sentences due to their high degree of duplicity, which artificially results in fewer training steps and higher accuracy. Using deduplication, the overlap between training, validation and test sets is reduced, improving the training procedure as well as the confidence on the model proficiency.
