* _Mixed-precision decomposition_. As observed in [413], extreme large values occur in hidden activations (called _the emergence of outliers_) when the model size reaches 6.7B parameters or above. Interestingly, these outliers are mainly distributed in some specific feature dimensions at Transformer layers. Based on this finding, a vector-wise quantization approach, called _LLM.int8(0)_, has been proposed in [413], which separates the feature dimensions with outliers and the rest dimensions in matrix multiplication. Then, the calculations for the two parts are performed with _16-bit floating numbers_ and _8-bit integers_, respectively, so as to recover these outliers in a high precision.