nuQmm: Quantized MatMul for Efficient Inference of Large-Scale Generative Language Models

The recent advance of self-supervised learning associated with the Transformer architecture enables natural language processing (NLP) to exhibit extremely low perplexity. Such powerful models demand ever-increasing model size and, thus, large amounts of computations and memory footprints. In this paper, we propose an efficient inference framework for large-scale generative language models. As the key to reducing model size, we quantize weights by a non-uniform quantization method. Then, quantized matrix multiplications are accelerated by our proposed kernel, called nuQmm, which allows a wide trade-off between compression ratio and accuracy. Our proposed nuQmm reduces the latency of not only each GPU but also the entire inference of large LMs because a high compression ratio (by low-bit quantization) mitigates the minimum required number of GPUs. Assuming 2-bit quantization, we demonstrate that nuQmm can reduce latency to generate each token for OPT-175B (that requires 8 GPUs without nuQmm) by 47.3% using 8 GPUs or by 23.2% using only 2 GPUs.


INTRODUCTION
Recent years have observed large-scale language models presenting state-of-the-art performance on various natural language process (NLP) tasks. Such rapid progress in NLP performance has been highly facilitated by the self-supervised learning method. Since pre-training dominates the entire training process without an expensive labeling process (Baevski et al., 2020;Chen et al., 2020;Devlin et al., 2019;Hu et al., 2020), the size of the training dataset can substantially increase. Combined with efficient sequence-to-sequence model architectures, such as the Transformers (Vaswani et al., 2017), the number of model parameters also significantly increases.
To address such concerns, model parallelism has been suggested to distribute computations over multiple GPUs through GPU-to-GPU communication (Narayanan et al., 2021;Shoeybi et al., 2019). As shown in Fig. 1, model parallelism splits the parameters of a large LM model into numerous GPUs, and information during training/inference can be shared among GPUs through dedicated channels.
Model parallelism can be divided into tensor parallelism and pipeline parallelism to improve latency and throughput, respectively. Such parallelism schemes, however, require various communication primitives, such as AllReduce, Reduce, Broadcast, and AllGather, to synchronize partial outputs produced by GPUs (Awan et al., 2018). Even though GPU-specific external communication protocols (e.g., NVLink (Li et al., 2019)) can reduce communication latency, an inherent variance of GPU performance (caused by various random factors such as fabrication process variation and operating system conditions) is another performance bottleneck (Roy et al., 2022). In addition, since a large matrix is separated into submatrices, each GPU faces tall-and-skinny matrix multiplications with low utilization of resources (Bell & Garland, 2008;Jeon et al., 2020). As a result, performance gain by model parallelism becomes a sub-linear function of the number of GPUs.
To alleviate the challenges of model parallelism, parameter quantization (Choi et al., 2017;McDonnell, 2018;Xu et al., 2018) is a practical solution to reduce the model size such that the number of GPUs to serve inference can be lower, as described in Fig. 1. Among various quantization schemes, uniform quantization is a popular choice to exploit integer-based arithmetic units (Jacob et al., 2018;Lin et al., 2016;Wu et al., 2018b;Zhao et al., 2019). Uniform quantization, however, is practically limited to 8 bits while non-linear operations (e.g., softmax and normalization) can be imprecise (Bhandare et al., 2019;Kim et al., 2021b). Moreover, activation quantization/dequantization should be implemented on-the-fly on top of the requirement of accurate estimation of the distribution of activations in advance (Dettmers et al., 2022;Yao et al., 2022). Thus, we consider non-uniform quantization to achieve a high compression ratio.
Specifically, we utilize binary-coding quantization (BCQ) scheme (Rastegari et al., 2016) to gain benefits of simple arithmetic operations. Note that non-uniform quantization relies on customized hardware to support bit-level operations. To enable an efficient BCQ scheme for inference on GPUs, we propose a new BCQ-dedicated matrix multiplication kernel, called nuQmm, to obtain low inference latency while avoiding the necessity of activation quantization.
Our major contributions in this work include the following: • Compared to the previously proposed matrix multiplication kernels with quantized parameters, nuQmm provides a wide search space of trade-offs between compression ratio and latency (based on GPU-specific efficient hardware utilization method to implement various BCQ configurations).
• Under such new search space, we quantize wellknown generative LMs, such as GPT-2 and OPT, to investigate the range of compression ratio achievable by nuQmm. Figure 2. An illustration of the generative language model. Given an input context, a single token is generated in an autoregressive manner. The entire generation procedure can be categorized into the summarization stage (along with a large batch size using an input context) and the generation stage executing single-batch operations using the previously generated token.
• For large LMs, we show that nuQmm can considerably accelerate matrix multiplications with small quantization bits while power consumption is saved a lot by reducing the number of GPUs. Consequently, nuQmm leads to low energy consumption.
• Assuming a 2-bit BCQ format for weights of OPT-175B, our experimental results show that nuQmm can reduce latency to generate each token for OPT-175B (that require 8 GPUs without nuQmm) by 47.3% using 8 GPUs or by 23.2% using only 2 GPUs.

Generative Language Models
Representative large-scale generative LMs, such as GPT-3, are autoregressive models that predict future tokens using the previous tokens in a feed-forward fashion. For example, as shown in Fig. 2, assuming an input context is given (for in-context learning ), a new token can be predicted by using the previously generated tokens. Correspondingly, autoregressive modeling employs both largebatch operations (for the summarization stage conducting in-context learning using a given input context) and singlebatch operations (for the generation stage generating a single token at a time index).
Since the Transformer introduced a self-attention mechanism and a parallelized training algorithm for autoregressive models using a teacher forcing technique (Vaswani et al., 2017), virtually all large-scale generative LMs follow the decoder structure of the Transformer. In addition to the architectural advantages of the Transformer to scale the model size, generative LMs are increasing the number of parameters (as depicted in Table 6 and 7 in Ap- Number of input tokens OPT-6.7B (1 GPU) OPT-13B (1 GPU) OPT-30B (1 GPU) OPT-66B (2 GPU) OPT-175B (8 GPU) 2 6 2 8 2 10 2 6 2 8 2 10 2 6 2 8 2 10 2 6 2 8 2 10 Others MatMul Communication  pendix) because: 1) self-supervised learning alleviates the burden of the expensive labeling process and 2) scalinglaw Kaplan et al., 2020) provides a predictable performance on the cross-entropy loss as the model size increases. Surprising qualitative evaluation results (e.g., human-like writing) of extreme-scale LMs also fueled the competition in model size (Kim et al., 2021a;Rae et al., 2021).

GPU-Accelerated Generative LMs
For large LMs, the processing time of matrix multiplications dominates the entire inference latency because of higher time complexity compared to activation functions, normalization layers, and so on (Bhandare et al., 2019;Chung et al., 2020;Dettmers et al., 2022;Kim et al., 2021b). To validate such a claim, Fig. 3 shows the latency breakdown of various large-scale generative LMs (OPT 6.7B, 13B, 30B, 66B, and 175B) with a different number of input tokens. Fig. 3 is obtained by A100 (80GB) and FasterTransformer, inference framework of Nvidia 1 . We can observe that matrix multiplications dominate the entire processing time (at least 75%) for various LM sizes and input token lengths (note that since GPU has limited memory capacity, large LMs may need multiple GPUs and result in high communication latency between GPUs). Hence, GPUs are widely adopted to accelerate inference because GPUs embed lots of arithmetic units and support multiple threads that are critical to expediting matrix multiplications (Migacz, 2017;Narayanan et al., 2021). Note that extracting high performance from GPUs depends on arithmetic intensity. In other words, batch size should be large enough to ensure a high reuse ratio of data once retrieved from main memory (Markidis et al., 2018).
Tensor Parallelism For highly parallel computing systems, high memory bandwidth is essential to feed lots of arithmetic units so as to maintain high resource utilization. The main memory system of GPUs, hence, is inclined to focus on high bandwidth instead of large capacity. Correspondingly, even though new innovative memory architectures (e.g., HBM (Rajbhandari et al., 2021)) are proposed, the maximum memory capacity for a single GPU is still limited up to a few tens of gigabytes (Li et al., 2019). Such limited GPU's memory capacity derived various parallelism ideas to partition a large-scale LM over multiple GPUs (Narayanan et al., 2021;Shoeybi et al., 2019). Tensor parallelism can split matrix multiplications over multiple GPUs so as to generate smaller sub-tasks executed simultaneously. Note that such parallelism induces additional synchronization and GPU-to-GPU communication overhead.
For a comparison between nuQmm (that we describe in detail later) and cuBLAS, we measure the latency of completing a large matrix multiplication as shown in Fig. 4. We assume that we multiply an (m × m) matrix and an (m × 1) vector (as a single-batch operation to run inference of generation steps in Fig. 2) when m can be 4096, 8192, or 16384. We also assume that nuQmm supports 2 or 4 bits to represent weights. For FP16 cuBLAS (Migacz, 2017), one to eight GPUs serve tensor parallelism through NCCL library (Li et al., 2019) to communicate via NVLink 3.0. From Fig. 4, we observe the following: 1) for cuBLAS, more GPUs with tensor parallelism can reduce the overall computation latency while communication latency increases, 2) tensor parallelism is more effective with larger m, and 3) for all three m configurations, nuQmm using one GPU is faster than cuBLAS employing even 8 GPUs. Note that communication latency (including GPU-to-GPU communication and synchronization across GPUs) cannot be ig-nored. As such, latency improvement by tensor parallelism is a sub-linear function of the number of GPUs (for any m) such that certain configurations of tensor parallelism may even harm the overall inference latency (while the cost of inference system increases due to more GPUs). In the case of nuQmm, the requirement for tensor parallelism can be eliminated (or alleviated) due to reduced model size. As a result, nuQmm can reduce both latency and system design cost of inference.
Tensor Cores Nvidia Tensor Cores are gaining popularity to accelerate matrix multiplications in GPUs (Markidis et al., 2018). In the extreme case of a single batch, however, matrix multiplication becomes memorybound (in the form of a matrix-vector multiplication) such that utilization of internal arithmetic units is forced to be low. As such, for the generation stage (shown in Fig. 2) requiring single-batch operations, Tensor Cores are not effective (in fact, not available because four batches are required as the minimum). Thus, we can expect the speed of GPUs running generative LM inference to be far from peak performance.

Binary-Coding Quantization
Quantization reduces the number of bits to represent model parameters. Shrinking memory footprint by quantization for generative LMs is effective in addressing the concerns on GPU performance, namely, 1) memorybound single-batch operations for generating tokens and 2) the usage of multiple GPUs and associated communication overhead. Note that various uniform quantization methods are being intensively studied because uniform quantization can be implemented by simple fixedpoint operations (Bhandare et al., 2019;Kim et al., 2021b;Nagel et al., 2017;Zhao et al., 2019). Recently, variants of uniform quantization for GPT-like LMs (Dettmers et al., 2022;Yao et al., 2022) have also been invented based on a fine-grained assignment of scaling factors for weights and token-wise dynamic quantization for activations, which can be considerable overheads to the inferences of large LMs. On the other hand, non-uniform quantization usually demands complicated operations of low parallelism while supporting hardware instructions may not be available (Guo et al., 2017;Xu et al., 2018).
In this paper, we choose binary-coding quantization (BCQ), first introduced in (Rastegari et al., 2016), which is one of the non-uniform quantization schemes. When a weight vector w (of size n) is quantized by BCQ and q is the number of quantization bits, w is approximated to be q i=1 α i b i where α i ∈ R + is a scaling factor and b i ∈ {−1, +1} n is a binary vector. Note that a scaling factor α can be shared by many weights (n can be any number) such that larger n value results in a relatively smaller memory footprint for scaling factors. The quantization process broadly involves finding scaling factors and binary vectors to minimize the quantization error as follows: (1) which does not have analytical solutions except when q = 1. Thus, scaling factors and binary vectors are obtained by iterative search methods (Guo et al., 2017;Xu et al., 2018) or by quantization-aware training (Chung et al., 2020). Recently, a parameter-efficient adaptation (Kwon et al., 2022) method with BCQ scheme is proposed to accelerate finetuned models on downstream tasks.

DESIGN METHODOLOGY OF NUQMM
Our overall goal is to explore high-performance and lowenergy inference systems for large-scale generative language models such as GPT-3 175B. To this end, an efficient quantization method is supposed to compress models with a high compression ratio while reducing quantization error (for given quantization bits) in consideration of the characteristics of large-scale LMs. In addition, a new kernel that efficiently supports such an efficient quantization technique is also necessary so that latency improvement through quantization can be maximized. In this section, we propose new advanced BCQ formats that can be supported by our proposed kernel nuQmm. The proposed nuQmm is designed to directly utilize binary weights in a compressed format without additional overhead, such as dequantization. Consequently, we demonstrate that nuQmm can reduce the latency and/or the number of GPUs to run large-scale LM inference.

Group-wise Binary-Coding Quantization
In practice, conventional BCQ methods assume that a scaling factor is assigned to each row of a weight matrix (or even to the entire matrix) so as to support vector instructions of CPUs or GPUs (Rastegari et al., 2016;Xu et al., 2018). As large-scale LMs introduce deeper and wider model structures along with ever-increasing parameter size (Rae et al., 2021;Shoeybi et al., 2019), however, we argue that such conventional row-wise BCQ format encounters various challenges. Suppose that a relatively small hidden size (e.g., d model = 1024 in Table 6 in Appendix) is selected along with small weight matrices correspondingly, row-wise assignment of scaling factors might be reasonable to obtain low quantization error. On the other hand, if the hidden size increases rapidly (e.g., d model = 12288 for GPT-3 175B in Table 6 in Appendix) according to the advent of large-scale LMs, it would be more difficult to compute a proper scaling factor shared by a larger number of weights.
In order to enable low-bit quantization schemes, it would Figure 5. Group-wise BCQ example with two different (g, q) configurations to quantize an (8 × 8) matrix. Assuming a scaling factor of 16 bits, smaller q can yield a larger memory footprint if g is also small.
be necessary to investigate different ways of assigning scaling factors as long as a new assignment can be backed by practical implementation.
Group-wise α Assignment As an alternative to row-wise quantization, we propose group-wise quantization in which a scaling factor can be shared by an arbitrary number of weights. Our proposed new BCQ format introduces a new hyper-parameter g as a group size that represents the number of weights to be shared by a scaling factor. In this paper, g is a fixed number with a range of 8 (as the minimum) to the column width of a matrix (equivalent to row-wise quantization). Since g is a constant number, the hidden size does not affect our group-wise BCQ formats. In Section 3.3, we discuss how to implement the group-wise BCQ of small group size g with negligible latency overhead on GPU.
Impact on Compression Ratio For a given q (i.e., the number of quantization bits), a smaller group size g can lower quantization error at the expense of an increased memory footprint for scaling factors. For a target quantization error, thus, a compression ratio is a compromise between q and g. In other words, due to the introduction of g, we can control the amount of scaling factors and binary vectors as a trade-off process. Note that the memory footprint of conventional row-wise quantization techniques is dominated by the size of binary vectors because the size of scaling factors can usually be ignored if the column width of a matrix is large enough. Compared to the conventional scheme, our proposed group-wise BCQ provides a new wide search space for quantization formats to meet a target compression ratio. Fig. 5 shows an example with two (g, q) configurations to quantize an (8 × 8) matrix. Indeed, even if the number of quantization bits is smaller, Figure 6. Accuracy drop and compression ratio with the various number of quantization bits (q) and group size (g). Three pre-trained models are quantized (by post-training quantization method) and then evaluated on the LAMBADA dataset.
the memory footprint can be large when the group size g is small (i.e., more scaling factors are employed).

Exploration of Compression Ratio
To study the capability of group-wise BCQ to enlarge search space for compression, we conduct experiments using pre-trained OPT 2 models, which are publicly available. Specifically, we apply post-training quantization (with an iterative solver introduced in (Xu et al., 2018)) to three pre-trained OPT models while g and q vary. Then, each quantized model is evaluated on the LAMBADA (Paperno et al., 2016) dataset to obtain the accuracy to find the relationship between compression ratio and accuracy. Fig. 6 shows accuracy degradation and compression ratio 3 when we try various q values (from 3 to 4) and g values (from 32 to matrix column width (i.e., row-wise)). From Fig. 6, we observe that the relationship between accuracy loss and g are vastly different depending on q. As a result, compared to the conventional row-wise quantization, group-wise BCQ offers new optimal configurations. For example, in the case of the OPT-6.7B model in Fig. 6, when a target accuracy degradation is around 1.0, then 4-bit quantization is superior to 5-bit quantization in terms of compression ratio (even though g values are different). Thus, for a target accuracy degradation (or a target compression ratio), we need to explore various q and g values simultaneously to achieve the best compression ratio (or the minimum accuracy degradation). Note that for OPT-13B and OPT-30B, as we discussed the limits of row-wise quantization for large-scale LMs, a small g value is critical to achieving low accuracy degradation (while latency is not heavily affected by a small g as we discuss in Section 3.3). All in all, the effects of q and g on accuracy differ with each model such that q and g are hyper-parameters to

Quantized Matrix Multiplication based on Lookup Tables
Under our quantization scheme that quantizes weights by using BCQ format and maintains activations to be of full precision, naive matrix multiplications result in duplicate and redundant partial computations. To illustrate, assume that a binary matrix B ∈ {−1, +1} 4×4 and an activation vector x ∈ R 4 are given as Then, computing Bx ⊤ (that is to be multiplied by scaling factors) would repeat (1.2 − (−0.7)) three times and (−0.3 + 0.6) two times. Such redundant computations are caused by digitized elements of B, and thus, we expect more duplicated computations as the size of matrices increases according to the growth of model size. Moreover, loading each element of B requires bi-level memory accesses that can be slow for commercial CPUs and GPUs.
To avoid bit-level memory accesses and perform Bx ⊤ efficiently, we can pre-compute all possible combinations of full-precision activations and binary patterns. Note that a lookup table (LUT) has been widely used to save processing time when numerous computations yield outputs within a restricted set (de Queiroz & Stein, 2004;Meher, 2010;Xu et al., 2021). LUT-based computation is justified especially when retrieving a value from a LUT is much faster than carrying out the original calculations. BCQ format (without quantizing activations that require heavy modifications in training codes and model structure (Jacob et al., 2018;Wu et al., 2018b)) is also useful to be implemented by LUT-based approaches. For example, with every 3 elements {x 1 , x 2 , x 3 } in x, we can pre-compute 8 (=2 3 ) possible values as shown in Table 1 and store those values in a LUT. Let µ be the length of a sub-vector of x to construct a LUT (hence, µ is 3 in Table 1). Once 2 µ values of a LUT are generated by using a sub-vector of x, arithmetic operations to obtain partial dot products (of Bx ⊤ ) are replaced with LUT retrieval operations while a key is given by concatenating µ binary elements of B. To complete Bx ⊤ computation, as the final step, those partial products are summed and then multiplied by scaling factors. When the row dimension of B is enlarged (as generative LMs get larger), the utilization of a LUT increases due to more occurrences of redundant computations.
Let us briefly explain how to optimize µ, which is also discussed in . If µ increases, LUT construction cost increases as well by 2 µ . Such increased µ, however, can enhance computational parallelism because we can replace the µ number of FP16 additions with one LUT retrieval operation. Thus, as reported in , there exists an optimal µ to maximize the speed-up. Note that optimizing µ also needs to consider aligned memory accesses. In our work, µ = 8 is used as a practical choice.

nuQmm for Group-Wise BCQ Format
In addition to the LUT-based scheme (eliminating redundant computations and bit-level memory accesses), our proposed nuQmm needs to achieve high performance with group-wise quantization in order to enhance accuracy for a given q. Targeting single-batch operations on GPUs, we propose the following as our strategy: • To improve parallelism, we create as many threads as possible while each thread is allowed to perform independent LUT accesses.
• Binary weights accessed by a thread can share a common scaling factor such that operations related to scaling factors do not degrade the performance of a thread.
• If we allocate too small resources to a thread, then LUT utilization can be low, and synchronization overhead can increase. As such, we need to optimize thread configurations empirically.
For the sake of simplicity, we formulate the proposed group-wise quantized matrix multiplication as y = where A is an (m × n) FP16 scaling matrix, B is an (m × n) FP16 binary matrix, x is an FP16 input vector of size n, and the operator • indicates element-wise multiplication. Note that in real situations, the memory footprint of A is reduced by g since every g weights share a scaling factor.
Overall Architecture For nuQmm, we assign l number of LUTs to a thread block (TB) of GPU. Then, the size of submatrix of A and B allocated to each TB becomes (t h × t w ) when t w = l × µ. Small t h can increase the nuQmm: Quantized MatMul for Efficient Inference of Large-Scale Generative Language Models  Figure 7. The overview of nuQmm implementation on GPUs. In this example, we assume m = 12, n = 96, µ = 8, t h = 4, l = 4, tw = 32, q = 1, and g = 32. "•" denotes element-wise multiplication and "·" indicates a tensor product.
number of available threads while large t h enhances LUT utilization inside a TB. Thus, t h is empirically determined (2048 is a practical number for large-scale LMs). Note that the amount of resources allocated to each TB is small enough such that multiple TBs can share a scaling factor as long as g is larger than l × µ. The overall nuQmm implementation scheme on GPUs is presented in Fig. 7 when we assume µ = 8, l = 4, t w = 32, t h = 4, q = 1, and g = 32.
For q > 1, the entire process of Fig. 7 can be iterated q times while intermediate results are accumulated.
Detailed Implementation Each TB first conducts precomputation using partial x values assigned in order to fill up the l number of LUTs. Then l LUTs can be shared by all threads inside a TB (so as to mitigate costly global memory accesses) and multiple rows of a submatrix of B can be processed by multiple threads (so as to improve throughput). When threads finish retrieving and summing LUT values, scaling factors are fetched (only once for each thread) and multiplied to produce partial outputs. Finally, n l×µ partial outputs are accumulated across TBs (through atomicAdd operations, as illustrated in Fig. 7) to generate the final outputs. LUTs are stored in shared memory inside GPU and the shared memory presents high bandwidth (e.g., 19TB/s for A100). Thus, high memory accesses for LUTs (while multiple FLOPs can be replaced with one LUT access) enable fast matrix computations. As for the memory size of LUTs, only 1KB is required for every 8 hidden dimensions and the shared memory size is more than a few megabytes (e.g., 20MB for A100 with 192KB per SM and 108 SMs available). Thus, the whole LUTs can be safely stored in shared memory. To illustrate, the hidden dimension can be up to 324,000 for A100 while 12,288 is the hidden dimension for GPT-3 175B. our proposed nuQmm. For experiments, we multiply an (m × n) matrix (that can be quantized by q bits) and an (n × 1) matrix using a single Nvidia A100 40GB GPU with CUDA 11.3. We can observe that the memory size and execution time required for both cuBLAS and nuQmm increase with larger m (and q for nuQmm). It is clear that the relative reduction of memory footprint and latency by nuQmm increases when a larger weight matrix is employed. Such observation can be partly explained by the fact that kernel launch overhead appears as a performance bottleneck for small matrices (such as m = 4096). Thus, the merits of nuQmm would be outstanding as larger-scale generative LMs are introduced.
To examine latency variance of nuQmm on group size g, we perform matrix multiplications (using an (m × n) matrix and an (n × 1) matrix) when g values vary. In Fig. 8, for each m (= n) selection, matrix multiplication latency of different g is compared with that of row-wise (i.e., g = n) BCQ format. Interestingly, when g > 32, group-wise nuQmm is as fast as row-wise nuQmm regardless of m in Fig 8. In other words, a reasonably large g (such as 256 and 512) can result in fast nuQmm while accuracy improvement by group-wise is substantial (as can be seen in Fig. 6).
To understand the underlying mechanisms in Fig. 8, we analyze the memory footprint of nuQmm because single-batch operations are basically memory-bound such that latency is proportional to memory footprint. Let S b and S α represent the space complexity of binary weights and scaling factors, respectively. Then the overall space complexity S can be described as = O m · n · q 1 + 16 g . (3) As a consequence, if g ≫ 16, S can be independent of g and approximated to be O (m · n · q). To verify our claim that latency of nuQmm is proportional to memory footprint (when running single-batch operations), we explore various (q, g) pairs and compression ratios correspondingly and measure matrix multiplication latency when m = 12288 as shown in Fig. 9. It can be noticed that the additional search parameter g allows a fine-grained search space of compression ratio that is not available by q alone. Across all available compression ratios in Fig. 9, latency is a function of compression ratio. For instance, if two different pairs (q 1 , g 1 ) and (q 2 , g 2 ) exhibit a similar memory footprint, then we can expect similar latency by nuQmm. Table 3 is a summary of profiling results of matrix multiplications performed by using cuBLAS (with tensor parallelism) or nuQmm (with one GPU). GPU power and other metrics are collected by using nvidia-smi utility (Ali et al.;Tiwari et al., 2015). Again, an (m × m) matrix is multiplied by an (m × 1) matrix while we select m to be 8192, 12288 (used for GPT-3 175B), or 16384. For Table 3, we include the case of q = 2 for nuQmm (with g = m) as 2bit quantization for the Transformer is reported to be feasible by quantization-aware training along with BCQ format (Chung et al., 2020). We notice that throughout all m configurations, increasing GPUs for cuBLAS with tensor parallelism brings about a higher reduction in GPU utilization, memory utilization, and latency ratio of computations. As evidenced by the increase in the latency ratio of communication, such reductions in utilization indicate that some GPUs can be temporarily idle until all GPUs are synchronized. Accordingly, the amount of speed-up that can be obtained by tensor parallelism is a lot smaller than the number of GPUs. As a result, cuBLAS with more GPUs causes increased energy consumption for matrix multiplications.

Comparison with FP16 Tensor Parallelism
On the other hand, nuQmm (with one GPU) can offer high speed-up (that cannot be achieved by tensor parallelism) while retaining high GPU/memory utilization. Combining low latency and a reduced number of GPUs, thus, nuQmm saves energy consumption for matrix multiplications significantly. For example, when m = 12288, nuQmm (with q = 2) achieves 4.8× energy reduction and 6.2× speed-up compared to cuBLAS with one GPU.

EXPERIMENTAL RESULTS ON GPT-3 175B AND OPT MODELS
In this section, we apply nuQmm to major matrix multiplications in GPT-3 175GB ) (chosen as a representative large-scale LM) and estimate speed-up and energy reduction. GPT-3 follows the structure of the Transformer (Vaswani et al., 2017) that consists of identical layers. Assuming that m is the hidden size (i.e., d model ), each layer has multi-head attention and feed-forward network as shown in Fig. 10 and includes 4 major linear computations of higher time complexity (than the other non-linear operations) for which we multiply a (m × 1) activation matrix and the following 4 matrices: 1) (m × m) matrix for attention output, 2) (3m×m) matrix for key, query, and value of  Table 4. Speed-up and energy reduction by nuQmm on GPT-3 175B for which m is set to be hidden dimension size 12288. For nuQmm, row-wise scale factor assignment is assumed (g = m). attention, 3) (m×4m) matrix for the first feed-forward sublayer, and 4) (4m × m) matrix for the second feed-forward sub-layer. Note that all layers of the Transformers have identical structures and matrix multiplications dominate the entire inference latency (Chung et al., 2020).

Layer
By setting m to be hidden dimension size 12288 of GPT-3 175B, Table 4 compares speed-up and energy reduction between cuBLAS (1 or 8 GPUs) and nuQmm (q is 2 or 4). It should be noted that even though we include the result of FP16 cuBLAS with one GPU, the entire parameters of GPT-3 175B cannot be stored in the main memory of one GPU. Thus, multiple GPUs would be mandatory if we do not quantize GPT-3 175B. We can observe that for all matrix multiplications in Table 4, nuQmm presents high performance with less energy consumption. Table 5 provides end-to-end latency to generate a token using various OPT models based on the FasterTransformer framework. Targeting only 4 matrix multiplications, nuQmm can reduce the number of GPUs to run inference while latency decreases as q decreases or the number of GPUs increases. In the case of OPT-175B model, originally 8 GPUs are required as the minimum to run inference assuming FP16 weight representation. After quantization using BCQ format, nuQmm can run inference even using one GPU only while the overall latency is comparable. If multiple GPUs are allowed (for OPT-175B model), nuQmm (with q = 2) can reduce end-to-end latency to generate each token by 23.2% (using 2 GPUs) or 47.3% (using 8 GPUs).
x Figure 10. Transformer layer incorporating multi-head attention and feed-forward network performing 4 major matrix multiplications.

Generation
(1 Batch) Figure 11. Our proposed strategy to support both context process and generation process using quantized weight with BCQ format.

Context Processing using Quantized Weights
As we discussed, the inference of generative LMs can be separated into the summarization (using input context) and the generation as shown in Fig. 2. Even though the same weights can be applied to both summarization and generation, those two processes require different batch sizes. As for summarization, since tokens for input context are already provided, multiple tokens can be fed into the Transformer to improve explicit parallelism. On the other hand, the generation stage is fundamentally processed by singlebatch operations because of autoregressive properties. In other words, inference of generative LMs is basically supposed to support two potentially conflicting operations, namely, compute-bound context process with high parallelism and memory-bound generation process with low parallelism.
We propose our inference strategy as shown in Fig. 11. Quantized weights are stored in memory in the form of BCQ and then 1) dequantized to conduct full-precision cuBLAS matrix multiplications for the context process or 2) used as inputs of nuQmm for the generation process. The rationale behind our approach is as follows: 1) it is necessary to quantize weights even for the context process (otherwise, the effort to save memory by nuQmm would be worthless), 2) dequantization overhead (in latency) becomes ignorable as more tokens are utilized for context process and/or generation process, 3) if input context gets longer, then the latency of cuBLAS matrix multiplications is a lot higher than that of dequantization (due to higher time complexity), and 4) as target tasks require more tokens to be generated, the generation stage would dominate inference latency (and then, the latency of the entire context process including dequantization can be negligible).

DISCUSSION
For our experiments, we measured the accuracy of quantized models by performing a post-training quantization method for fast design exploration of nuQmm. There are several ways to further improve accuracy and/or compression ratio. For example, quantization-aware training for Transformers (Chung et al., 2020) is reported to save additional 1 or 2 quantization bits at the cost of training time with hyper-parameter tuning. In particular to selfsupervised language models, fine-tuning techniques (e.g., LoRA (Hu et al., 2022)) using a small dataset can be a potentially economic choice to improve accuracy after posttraining quantization. Among those tuning techniques, a recently published parameter-efficient adaptation method, AlphaTuning (Kwon et al., 2022), considers BCQ formats to enable quantization as well. Thus, AlphaTuning can utilize nuQmm directly to enhance inference performance. Investigating nuQmm with various quantization algorithms would be interesting.
Pre-trained extreme-scale language models (e.g., GPT-3 (175B) , HyperCLOVA (204B) (Kim et al., 2021a), and Megatron Turing NLG (530B) (Smith et al., 2022)) are usually not publicly available. Thus, in this work, our detailed analysis of group-wise quantization and nuQmm is limited to relatively smaller models (such as GPT Neo). In the case of GPT-3 175B, since it is not feasible to find q for reasonable model accuracy without a pre-trained model, we assumed q = 2 or q = 4 to estimate speed-up and energy consumptions of major matrix multiplications based on the reports that larger neural networks can be quantized by a smaller number of quantization bits (Choi et al., 2017;Stock et al., 2019).
The parameter pruning technique is a promising complement to parameter quantization in order to further compress neural networks (Han et al., 2016;Zhu et al., 2017). When parameters identified as unimportant are removed first, then quantization can be performed with a reduced number of parameters. Then, the number of quantization bits can also be reduced along with a smaller quantization error . Both fine-grained pruning and structured pruning are investigated because of trade-offs between compression ratio and the regularity of memory ac-cess patterns Yu et al., 2017). Exploring pruning techniques to integrate nuQmm with proper sparse representation (to be practical with GPUs or CPUs) would be interesting.

CONCLUSION
Generative language models, such as GPT-3 175B, are attracting attention due to their generation capability on various complicated tasks. The inference speed, however, is a serious concern not only because of parameter size increase but also because of autoregressive operations associated with single-batch operation. To address such concerns, in this paper, we proposed a new group-wise binary-coding quantization format and a dedicated matrix multiplication kernel nuQmm. nuQmm is especially superior to tensor parallelism, which is required if a model size is too big to be accommodated in a single GPU. Combining low latency and a reduced number of GPUs, inference of large language models can be performed with significantly reduced energy consumption.   12 describes PPL (of four GPT-2 models with different sizes and one GPT Neo model) when diversified model sizes for each model are available by exploring q and g values. GPT model sizes and compression ratios are expressed under FP32 precision. Even though we adopt posttraining quantization instead of quantization-aware training (that might be too expensive for recent large LMs), the sizes of all 5 models in Fig. 12 are reduced a lot by our groupwise quantization. Suppose that we can tolerate some reasonable PPL degradation, we notice that larger models tend to be compressed by a higher compression ratio, which is consistent with a report in (Li et al., 2020). In the next subsections, we study a matrix multiplication kernel to facilitate the potential benefits of group-wise BCQ formats.

A.1 Mixed Precision using nuQmm
It is well known that different layers of neural networks present different sensitivity to model compression (Chen et al., 2021;Han et al., 2016). As such, mixed precision techniques for quantization are eligible for obtaining higher compression ratio practical techniques by assigning different quantization bits to layers while measuring accurate sensitivity of layers is complicated and computationally demanding (Chen et al., 2021;Wu et al., 2018a). Note that since we introduce an additional parameter g (i.e., u v x g y g g g Figure 13. Mixed precision quantization results using GPT Neo 1.3B. All matrices of the same sub-layer type are quantized by the same (q, g) configuration. Available sets for exploring q and g are {3, 4, 5} and {128, 256, 512, 2048}, respectively. group size for scaling factors), nuQmm can offer wider search space exploration for mixed precision compared to the conventional methods allowing the number of quantization bits as the only search parameter.
For experiments, we consider two sub-layers (multi-head attention and feed-forward networks) in Fig. 10 to be quantized by different quantization schemes with nuQmm. In order to facilitate efficient exploration of search space for mixed precision, we set the following constraints: 1) all matrices of each sub-layer type (i.e., multi-head attention or feed-forward network) across all layers are quantized by the same (q, g) configuration, 2) q is selected to be one of {3, 4, 5}, and 3) g is selected to be one of {128, 256, 512, 2048} that lead to less than 3% latency overhead compared to row-wise quantization as depicted in Fig. 8. Fig. 13 shows PPL degradation of GPT Neo 1.3B quantized by mixed precision (using two (q, g) configurations as described above) in addition to the previous quantization results of Fig. 14 that allow only one (q, g) configuration across all layers. Notice that mixed precision combined with additional parameter g produces extensive trade-offs between PPL degradation and compression ratio.  Figure 14. PPL degradation and compression ratio with the various number of quantization bits (q) and group size (g). Three pre-trained models are quantized (by post-training quantization method) and then evaluated on the WikiText-2 dataset.