We have seen that LLMs can efficiently compress knowledge into their parameter space, achieving 2bit/param even with 8-bit parameters. This raises the question: how and where is such knowledge stored? Our preliminary answer is that knowledge can be compactly stored within the model in a not-so-redundant manner. It is unlikely that the MLP layers alone store knowledge, as Attention layers, being of comparable sizes, also contribute to knowledge storage (c.f. Result 5). Moreover, particularly in models near the capacity boundary, removing the last transformer layer of an \(L\)-layer model to "probe" for remaining knowledge reveals that the "leftover knowledge" can be significantly less than \(1-\frac{1}{L}\) of the total.19 This suggests knowledge is stored not in individual layers but in a complex manner, akin to a safe with combination locks, where removing one layer may eliminate much more than \(\frac{1}{L}\) of the total knowledge.