We conduct 4-bit quantization on our model. We do not quantize the parameters of embedding and layer normalization, since the performance of the model is sensitive to these parameters. Therefore, we only need to quantify each weight matrix. Denote the weight matrix as \(\mathbf{W}\in\mathbb{R}^{d_{out}\times d_{in}}\). We group every \(G\) consecutive parameter at the \(d_{in}\) dimension and form \(d_{in}/G\) group. Then we quantize each group of the parameters separately.