Large language models(LLMs)  have sparked a new wave of exciting AI applications. Hosting these models at scale requires significant memory resources. One crucial memory bottleneck for the deployment stems from the context window. It is commonly recognized that model weights are memory hungry; however, the size of key-value embedding stored during the generation process (KV cache)  can easily surpass the model size. The enormous size of the KV cache puts constraints on the inference batch size, which is crucial for high throughput inference workload. Inspired by an interesting observation of the attention scores, we hypothesizethe persistence of importance: only pivotal tokens, which had a substantial influence at one step, will significantly influence future generations. Based on our empirical verification and theoretical analysis around this hypothesis, we proposeScissorhands, a system that maintains the memory usage of KV cache under a fixed budget without finetuning the model. We validate thatScissorhandsreduces the inference memory usage of the KV cache by up to 5without compromising model quality. We further demonstrate thatScissorhandscan be combined with 4-bit quantization for further compression