The inference process in Large Language Models (LLMs)  is often limited due to the absence of parallelism in the auto-regressive decoding process, resulting in most operations being restricted by the memory bandwidth of accelerators. While methods such as speculative decoding have been suggested to address this issue, their implementation is impeded by the challenges associated with acquiring and maintaining a separate draft model. In this paper, we presentMedusa, an efficient method that augments LLM inference by adding extra decoding heads to predict multiple subsequent tokens in parallel. Using atree-based attention mechanism,Medusaconstructs multiple candidate continuations and verifies them simultaneously in each decoding step. By leveraging parallel processing,Medusaintroduces only minimal overhead in terms of single-step latency while substantially reducing the number of decoding steps required. We present two levels of fine-tuning procedures forMedusato meet the needs of different use cases: Medusa-1:Medusais directly fine-tuned on top of afrozenbackbone LLM, enabling lossless inference acceleration. Medusa-2:Medusais fine-tuned together with the backbone LLM, enabling better prediction accuracy ofMedusaheads and higher speedup but needing a special training recipe that preserves the backbone model’s capabilities. Moreover, we propose several extensions that improve or expand the utility ofMedusa, including aself-distillationto handle situations where no training data is available and atypical acceptance schemeto boost the acceptance rate while maintaining generation quality. We evaluateMedusaon models of various sizes and training procedures. Our experiments demonstrate thatMedusa-1 can achieve over 2.2speedup without compromising generation quality, whileMedusa-2 further improves the speedup to 2.3-3.6. The code for this implementation is available athttps://github.com/FasterDecoding/Medusa.