Perplexity filteringWe trained a KenLM model Heafield (2011) on a small subset of carefully inspected legal data, and used it to filter any high perplexity paragraph. This removed non-English text as well as most of the "weird" unicode sequences present in the data. We show some of the most common \(10\)-grams in the filtered data on Table 2.