Finding all repeated sequences is thus a matter of linearly scanning the suffix array from beginning to end and looking for sequences \(\mathcal{A}_{i},\mathcal{A}_{i+1}\) that share a common prefix of at least some threshold length. Any satisfying sequences are recorded. This algorithm is embarrassingly parallel, and so we can efficiently process the dataset. Based on experimentation (Appendix B), we choose a threshold length of 50 BPE tokens for all experiments.