As the usage of large language models (LLMs)  grows, performing efficient inference with these models becomes increasingly important.
While speculative decoding has recently emerged as a promising direction for speeding up inference, existing methods are limited in their ability to scale to larger speculation budgets, and adapt to different hyperparameters and hardware.
This paper introducesSequoia, a scalable, robust, and hardware-aware algorithm for speculative decoding.
To attain better scalability,Sequoiaintroduces a dynamic programming algorithm to find the optimal tree structure for the speculated tokens.
To achieve robust speculative performance,Sequoiauses a novel sampling and verification method that outperforms prior work across different decoding temperatures.
Finally,Sequoiaintroduces a hardware-aware tree optimizer that maximizes speculative performance by automatically selecting the token tree size and depth for a given hardware platform.Sequoiaimproves the decoding speed of Llama2-7B, Llama2-13B, and Vicuna-33B on an A100 GPU by up to,, and. For offloading setting on L40,Sequoiaachieves as low as 0.56 s/token for exact Llama2-70B inference latency, which ison our optimized offloading system (5.6 s/token) ,than DeepSpeed-Zero-Inference,than Huggingface Accelerate. The code is available athttps://github.com/Infini-AI-Lab/Sequoia.