Theoretical Models for Scaling Laws:From a theoretical angle, scaling laws have been shown analytically even before the emergence of large foundation models. For instance, Caponnetto and de Vito (2007) characterize the power-law generalization error of regularized least-squares kernel algorithms. The role of optimization can also be taken into account in this setting ((Nitanda and Suzuki, 2021)). In the nonparametric literature, for example (Schmidt-Hieber, 2017) and (Suzuki, 2019) derived the test error scaling of deep neural network in fitting certain target functions and (Bordelon et al., 2020) analyze spectral dependence.