2. Despite the advantages of a larger dataset, the improvements saturate. For example, RefinedWeb (320B) model outperforms the RefinedWeb (160B) model by only 0.2%. Similarly, the TinyLlama model (1T tokens) performs comparably to the **WRAP** model, which only had 85B tokens of raw web data.