**Decoding Efficiency Issues.** In this part, we briefly analyze the decoding efficiency issues of LLMs. Overall, the decoding process of LLMs can be divided into two stages for overhead analysis: (1) the _prefill_ stage, which computes the hidden states of the input sequence, and (2) the _incremental decoding_ stage, which generates a token and updates hidden states in an auto-regressive manner [321]. As shown in the above _memory wall_ box, the arithmetic intensity of the incremental decoding stage is only \(1.97\), which is far from the expected value of 156 (calculated according to the standard configuration of A100 80GB GPU). In contrast, the arithmetic intensity of the prefill stage achieves \(113.78\) for LLaMA-13B.