Figure 18: We extract the training loss data from Llama2 paper (left part) and estimate the compute optimal \(\frac{D_{opt}}{N_{opt}}\) in their paper using the right part. The straight lines are plotted to estimate the optimal loss envelope assuming using WSD Scheduler.