Broadly speaking, ExactSubstr-Cut might remove text mid-sentence resulting in disconnected text; ExactSubstr-Mask does not have this issue, but might be less efficient as a significant portion of the training tokens will not directly contribute to updating the model's weights; ExactSubstr-Drop might still keep considerable duplicated sections in its Partial version, especially on larger documents, while the Any version might be overly aggressive. Following ablations in Appendix E.1, we choose to stick with the vanilla approach, ExactSubstr-Cut.