Large language models (LLMs)  with hundreds of billions of parameters have
sparked a new wave of exciting AI applications. However, they are computationally expensive at inference time.
Sparsity is a natural approach to reduce this cost, but existing methods
either require costly retraining, have to forgo LLM’s in-context learning
ability, or do not yield wall-clock time speedup on modern hardware.
We hypothesize thatcontextual sparsity, which are small, input-dependent sets
of attention heads and MLP parameters that yield approximately the same output
as the dense model for a given input, can address these issues.
We show that contextual sparsity exists, that it can be accurately predicted,
and that we can exploit it to speed up LLM inference in wall-clock time
without compromising LLM’s quality or in-context learning ability.
Based on these insights, we proposedejavu, a system that uses a low-cost
algorithm to predict contextual sparsity on the fly given inputs to each layer, along
with an asynchronous and hardware-aware implementation that speeds up LLM
inference. We validate thatdejavucan reduce the inference latency of
OPT-175B by over 2compared to the state-of-the-art FasterTransformer, and
over 6compared to the widely used Hugging Face implementation, without
compromising model quality. The code is available athttps://github.com/FMInference/DejaVu.