* LLaMA architecture performs comparably to GPT2, albeit slightly inferior for the tiny model (i.e., \(<\) 10M). This discrepancy can be mitigated by also requiring LLaMA architecture to tie weights, as shown in Figure 3(c) compared to Figure 3(b). * A similar observation applies to Mistral architecture (see Figure 3(d)). * Reducing the MLP size of GPT2 architecture by 1/4 or even eliminating all MLP layers _does not affect_ its capacity ratio, see Figure 3(e) and Figure 3(f). This suggests, contrary to conventional beliefs, the Attention layers are also capable of storing knowledge.