Fast and Efficient InferenceWe primarily use quantization, dynamic batching, and Paged Attention for improving decoding speed and memory usage. We use quantization to decrease both the memory footprint and computation demand. By 4-bit model quantization [81] and 8-bit KV cache quantization [18], we are able to achieve significant GPU memory saving with near-zero performance degradation (e.g., less than \(1\%\) accuracy drop in MMLU/CMMLU benchmark). We use dynamic batching [86] to minimize the response time and improve batching efficiency. We use PagedAttention[41] to improve memory utilization and improve decoding.