GPT2 was introduced in [26]. Due to its limitations from the absolute positional embedding [2], we adopt its modern variant, _rotary positional embedding_[7, 31], which we still refer to as GPT2 for convenience. Additionally, we disable dropout, which has been shown to improve performance in language models [33]. We explore a wide range of model sizes while using a fixed dimension-per-head of 64. The notation GPT2-\(\ell\)-\(h\) represents \(\ell\) layers, \(h\) heads, and \(64h\) dimensions; for example, GPT2-small corresponds to GPT2-12-12. The default GPT2Tokenizer is used, converting people's names and most attributes into tokens of variable lengths. In examining the impact of model architectures on scaling laws in Section 7, we will also use LLaMA/Mistral architectures [19, 32].