The simplest technique to find duplicate examples would be to perform exact string matching between all example pairs, but as we will show, this is insufficient. We introduce two complementary methods for performing deduplication. First, using a suffix array Manber and Myers (1993), we remove duplicate substrings from the dataset if they occur verbatim in more than one example. Second, we use MinHash Broder (1997), an efficient algorithm for estimating the \(n\)-gram similarity between all pairs of examples in a corpus, to remove entire examples from the dataset if they have high \(n\)-gram overlap with any other example.