* ExactSubstr-Cut: we remove the duplicated spans, and discard documents where there are fewer than 20 non-duplicated characters left-this is the vanilla setting used by Lee et al. (2022);
* ExactSubstr-Mask: we loss-mask the duplicated spans, preventing a loss from being computed on the duplicated text during pretraining, and discard documents where there are fewer than 20 non-masked characters left. * ExactSubstr-DropPartial: if more than 20% of the document is duplicated, we remove the entire document;
* ExactSubstr-DropPary: we drop any document with a duplicated span in it.