Transformers have emerged as the backbone of large language models (LLMs) . However, generation remains inefficient due to the need to store in memory a cache of key–value representations for past tokens, whose size scales linearly with the input sequence length and batch size.
As a solution, we propose Dynamic Memory Compression (DMC) , a method for on-line key–value cache compression at inference time.
Most importantly, the model learns to apply different compression rates in different heads and layers.
We retrofit pre-trained LLMs such as Llama 2 (7B, 13B and 70B)  into DMC Transformers,
achieving up to ~3.7throughput increase during auto-regressive inference on an NVIDIA H100 GPU.
DMC is applied via continued pre-training on a negligible percentage of the original data without adding any extra parameters.
We find that DMC preserves the original downstream performance with up to 4cache compression, outperforming up-trained grouped-query attention (GQA) . GQA and DMC can be even combined to obtain compounded gains. As a result DMC fits longer contexts and larger batches within any given memory budget.