GPT2 model.As elaborated in Section 2.3, we refer to the original GPT2 model [26] as GPT2, _after_ substituting its positional embedding with _rotary embedding_[7, 31] and removing its dropout layer [33]. These modifications are widely recognized for enhancing performance in language modeling tasks (see also [2] for a controlled experiment comparing that). We explore various GPT2 model sizes, maintaining a dimension-per-head of 64. The notation GPT2-\(\ell\)-\(h\) represents the (modified) GPT2 architecture with \(\ell\) layers, \(h\) heads, and \(64h\) dimensions. The context length is set to 512.