Multi-query attention (MQA) , which only uses a single key-value head, drastically speeds up decoder inference. However,MQAcan lead to quality degradation, and moreover it may not be desirable to train a separate model just for faster inference. We (1)  propose a recipe for uptraining existing multi-head language model checkpoints into models withMQAusing 5% of original pre-training compute, and (2)  introduce grouped-query attention (GQA) , a generalization of multi-query attention which uses an intermediate (more than one, less than number of query heads)  number of key-value heads. We show that uptrainedGQAachieves quality close to multi-head attention with comparable speed toMQA.