From Figure 3, it is evident that, except for tiny models, LLaMA, Mistral, GPT2\({}_{0}\), and GPT2\({}_{1/4}\) architectures closely follow GPT2's scaling law over 1000 exposures. For tiny models with \(\leq 10M\) parameters, tying model weights increases their capacity (refer to Figure 3(c)). This indicates that the _2bit/param capacity ratio is a relatively universal law_ among most typical (decoder-only) language model architectures.