Large Language Model (LLM)  inference consists of two distinct phases –prefillphase which processes the input prompt anddecodephase which generates output tokens autoregressively. While the prefill phase effectively saturates GPU compute at small batch sizes, the decode phase results in low compute utilization as it generates one token at a time per request.
The varying prefill and decode times also lead to imbalance across micro-batches when using pipeline-parallelism, resulting in further inefficiency due to bubbles. We presentSarathito address these challenges.Sarathiemployschunked-prefills,
which splits a prefill request into equal sized chunks, anddecode-maximal batching, which constructs a batch using a single prefill chunk and populates the remaining slots with decodes. During inference, the prefill chunk saturates GPU compute, while the decode requests ‘piggyback’ and cost up to an order of magnitude less compared to a decode-only batch.Chunked-prefillsallows constructing multipledecode-maximal batchesfrom a single prefill request, maximizing coverage of decodes that can piggyback. Furthermore, the uniform compute design of these batches ameliorates the imbalance between micro-batches, significantly reducing pipeline bubbles. Our techniques yield significant improvements in inference performance across models and hardware. For the LLaMA-13B model on A6000 GPU,Sarathiimproves decode throughput by up to 10, and accelerates end-to-end throughput by up to 1.33. For LLaMa-33B on A100 GPU, we achieve 1.25higher end-to-end-throughput and up to 4.25higher decode throughput. When used with pipeline parallelism on GPT-3,Sarathireduces bubbles by 6.29, resulting in an end-to-end throughput improvement of 1.91.