\begin{table}
\begin{tabular}{l l l l|l l} \(S\) & \(T_{total}\) & \(T_{selected}\) & Epochs & Non-Web Snapshot PPL & Instruction + Answers PPL \\ \hline Random & 40B & 40B & \(1\) & \(16.27\pm 0.012\) & \(14.19\pm 0.003\) \\  & 40B & 20B & \(2\) & \(16.39\pm 0.011\) (+\(0.12\)) & \(14.37\pm 0.015\) (+\(0.18\)) \\ \hline D4 & 40B & 20B & 2 & \(\mathbf{16.10\pm 0.024}\) (-\(0.17\)) & \(\mathbf{13.85\pm 0.016}\) (\(-0.34\)) \\ \end{tabular}
\end{table}
Table 1: For fixed data selection method and source dataset size, we compare the effects of choosing new tokens or repeating token. All models are 1.3B OPT models trained on 40B tokens. \(T_{selected}\) denotes the number of tokens selected from the source dataset. The top row denotes baseline training. Mean and standard error across 3 seeds are shown.