We introduce,Q-Sparse, a simple yet effective approach to training sparsely-activated large language models (LLMs) .Q-Sparseenablesfull sparsity of activationsin LLMs which can bring significant efficiency gains in inference. This is achieved by applying top-sparsification to the activations and the straight-through-estimator to the training. We also introduceBlockQ-Sparsefor batch training and inference. The key results from this work are, (1) Q-Sparsecan achieve results comparable to those of baseline LLMs while being much more efficient at inference time;
(2)  We present an inference-optimal scaling law for sparsely-activated LLMs; (3) Q-Sparseis effective in different settings, including training-from-scratch, continue-training of off-the-shelf LLMs, and finetuning; (4) Q-Sparseworks for both full-precision and 1-bit LLMs (e.g.,BitNet b1.58) . Particularly, the synergy of BitNet b1.58 andQ-Sparse(can be equipped with MoE)  provides the cornerstone and a clear path to revolutionize the efficiency, including cost and energy consumption, of future LLMs.