Fig. 5 shows our experiments where we train language models on the 5 coarse-grained domains of Pile7 and evaluate a validation set mixed with these 5 domains. We compare the errors obtained by implicit domain aggregation with different numbers of implicit domains to those obtained by explicit domain aggregation. We find that applying implicit domain aggregation and setting the number of implicit domains no smaller than the actual one (5 in the experimented case) results in lower errors than explicit domain aggregation. Moreover, the error remains low as we set the number of implicit domains much larger. This verifies the prediction accuracy of our implicit domain aggregation strategy for data mixing law and the number of implicit domains \(K\) can be a large number without careful tuning8.