\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Minhash** & **Exact substring** & pile-bpb \(\downarrow\) & agg-dev-1 \(\uparrow\) \\ \hline \multicolumn{3}{c}{RefinedWeb-Filtered} & 1.11 & 43.51 \\ \hline  & Mask & 1.08 & 45.84 \\ ✓ & Mask & 1.07 & 46.28 \\ ✓ & & 1.07 & 46.57 \\ ✓ & Cut & **1.05** & 47.11 \\  & Cut & 1.06 & 47.24 \\ ✓ & Drop partial & **1.05** & 47.25 \\  & Drop any & 1.07 & 47.77 \\ ✓ & Drop any & 1.07 & 47.86 \\  & Drop partial & 1.06 & **47.97** \\ \hline  & Pile & 0.88 & 43.70 \\ \hline \hline \end{tabular}
\end{table}
Table 8: **MinHash alone is insufficient to match the performance of exact substring deduplication, and combining the two does not significantly improve performance. Of all of the exact substring approaches, masking duplicated spans underperform, but all others exhibit similar performance. ✓ Minhash + Exact substring-Cut corresponds to our final deduplication setup. Perplexity in bits-per-bytes on The Pile (pile-bpb, lower is better), zero-shot performance aggregated over LAMBADA, PIQA, and HellaSwag (agg-dev).