To demonstrate our method's practicality, we must ensure the cost of selecting data is significantly less than this. As described in Section 3.4, selecting data via D4 involves: first, embedding documents via a 125M OPT model; second, computing K-Means indices + distance to indices. The first step is completed on a single machine with 96 CPU cores in approximately one day. Given the two orders of magnitude difference between the prices of CPU and GPU cores 1, we consider this cost negligible. For the second step, embedding 400B tokens with a 125M parameter model takes approximately 888 GPU hours, using the same A100 GPUs. Subtracting this from the _naive_ efficiency gain of 4300 GPU hours, we arrive at an _overall_ efficiency gain of 3412 GPU hours. This is how much compute D4 saved us in practice when training our single 6.7B parameter model. In Fig.