Note that the hidden dimension size is much smaller than the size of the token dictionary, i.e., \(h\ll l\). For example, LLaMA (Touvron et al., 2023) chooses \(h\in\{4096,5120,6656,8192\}\) and \(l=32{,}000\), and there is a recent trend towards increasingly large token sizes; GPT-4, for example, has a \(\approx\)100,000 token vocabulary.