Figure 9: The result of scaling experiments with WSD Scheduler (above) and the fitted scaling curve (below). The x-axis is the computation Flops \(C=6ND\), each color of line represents the same model with different computation Flops. We can see that smaller models are better than larger models when the Flops is small, and worse when the Flops is large. Thus models of difference sizes will intersect with each other in the plot around the compute optimal regime.