* _Fine-grained quantization_. For Transformer models, weights and activations are usually represented in the form of tensors. A straightforward approach is to use coarse-grained quantization parameters for the whole tensor (_i.e.,_ per-tensor quantization) [414]. However, it usually leads to inaccurate reconstruction results. Thus, fine-grained methods are proposed to reduce the quantization error. ZeroQuant [415] adopts a token-wise quantization approach with dynamic calibration for compressing activations. Whereas for weights (easier to be quantized), it uses a group-wise quantization. In practice, a group size of 128 [415, 416] is commonly used for model quantization. * _Balancing the quantization difficulty_. Considering that weights are easier to be quantized than activations, SmoothQuant [414] proposes to migrate the difficulty from activations to weights. Specially, they incorporate a scaling transformation to balance the difficulty between weights and activations in a linear layer: \(\mathbf{Y}=(\mathbf{X}\text{diag}(\mathbf{s})^{-1})\cdot(\text{diag}(\mathbf{ s})\mathbf{W})\). By introducing an mathematically equivalent transformation, this formula controls the quantization difficulty through the scaling factor \(\mathbf{s}\). To set \(\mathbf{s}\), it incorporates a migration strength parameter \(\alpha\) to balance the difficulties, where each entry \(s_{j}=\max(\mathbf{x}_{j})^{\alpha}/\max(\mathbf{w}_{j})^{(1-\alpha)}\) is determined by the migration strength. * _Layerwise quantization_. This approach finds optimal quantized weights that minimize a layerwise reconstruction loss: \(\arg\min_{\widetilde{\mathbf{W}}}\parallel\mathbf{W}\mathbf{X}-\widetilde{ \mathbf{W}}\mathbf{X}\parallel_{2}^{2}\). To efficiently optimize this objective, GPTQ [417] improves the original optimal brain quantization (OBO) [418] method by fixing the quantization order of weights for all rows. Further, with specially designed methods (_i.e.,_ lazy batch-updates and Cholesky reformulation), GPTQ is feasible to quantize very large models (_e.g.,_ 175B OPT) in 3 or 4 bit precision. More recently, AWQ [416] further simplifies the optimization form by incorporating activation-aware scaling for weights, which resembles the idea of SmoothQuant [414]: weights corresponding to outlier activations are more important to be precisely quantized. It does not directly optimize the reconstruction loss, but instead performs simple hyper-parameter search to achieve the minimal loss on calibration data.