_Remark 1.6_.: Since 2bit/param is obtained after sufficient training, training longer _may not_ further improve model capacity, _but quantization can_. While not covered in this paper, our framework also provides a principled playground to compare different quantization methods. * Section 9: How sparsity (MoE) affects model capacity. Mixture-of-experts (MoE) models offer faster inference than dense models but often underperform dense models with the same total parameter count (not effective parameters). We show that this performance drop is likely not due to a lack of knowledge storage capability. * Result 9: MoE models, even with 32 experts, only reduce 1.3x in capacity compared to the base scaling laws, despite using just 8.8% of the total parameters during inference. * Section 10: How junk knowledge affects model capacity. Not all pretrain data are equally useful. Much of the internet data lacks valuable knowledge for training language models [24], while knowledge-rich sources like Wikipedia represent only a small fraction of the training tokens. We explore the impact on model capacity by conducting a controlled experiment with both useful and "junk" data. * Result 10+11: Junk data significantly reduces model capacity. As an example, with a 1:7 ratio of "useful to junk" training tokens, capacity for useful knowledge _loses by a factor of 20_x, even when useful knowledge is exposed 100 times.8 Footnote 8: The loss factor improves to 3x/1.5x/1.3x with 300/600/1000 exposures of useful knowledge, compared to Result 4 which involves training without junk for only 100 exposures. Result 12]An _effective mitigation_ is to prepend a special token to all useful knowledge. This is akin to adding a domain name like wikipedia.org at the start of every Wikipedia paragraph; the model _autonomously_ identifies high-quality data without prior knowledge of valuable domains. In the example above, the loss factor improves from 20x to 2x.