Scaling LawsScaling laws guide us in discovering the impact of factors such as parameter count, data size, and compute on language model performance and behavior. These studies usually focus on predicable scaling though power law Kaplan et al. (2020), Hernandez et al. (2021), optimal resource allocation Hoffmann et al. (2022), downstream tasks Wei et al. (2022); Isik et al. (2024); Gadre et al. (2024), architectures Tay et al. (2022), memorization Tirumala et al. (2022); Carlini et al. (2022); Henighan et al. (2023); Biderman et al. (2024), and repeating data Hernandez et al. (2022); Muennighoff et al. (2024); Xue et al. (2024). Most scaling laws on model performance study cross-entropy loss on all training tokens, while we focus on the tokens loss of desired distributions.