Scaling Laws:Neural scaling laws have been ubiquitously observed in vision, language and speech. Early large scale empirical studies are performed in (Hestness et al., 2017; Rosenfeld et al., 2020), demonstrating power law scaling across a range of learning scenarios. This is followed by well-known large-scale studies from OpenAI (Kaplan et al., 2020) and DeepMind (Hoffmann et al., 2022), which empirically demonstrate power-law scaling in LLMs across a wide set of scales.