The emergence of accurate open large language models (LLMs)  has led to a race towards performant
quantization techniques which can enable their execution on end-user devices.
In this paper, we revisit the problem of “extreme” LLM compression—defined as targeting extremely low bit counts, such as 2 to 3 bits per parameter—from the point of view of classic methods in Multi-Codebook Quantization (MCQ) .
Our algorithm, called AQLM, generalizes the classicAdditive Quantization (AQ) approach for information retrieval to advance the state-of-the-art in LLM compression, via two innovations: 1)  learned additive quantization of weight matrices in input-adaptive fashion, and 2)  joint optimization of codebook parameters across entire layer blocks.
Broadly, AQLM is the first scheme that is Pareto optimal in terms of accuracy-vs-model-size when compressing to less than 3 bits per parameter, and significantly improves upon all known schemes in the extreme compression (2bit)  regime.
In addition, AQLM is practical: we provide fast GPU and CPU implementations of AQLM for token generation, which enable us to match or outperform optimized FP16 implementations for speed, while executing in a much smaller memory footprint.