Parameter-efficient fine-tuning (PEFT)  techniques make it possible to efficiently adapt a language model to create “expert” models that specialize to new tasks or domains. Recent techniques in model merging and compositional generalization leverage these expert models by dynamically composing modules to improve zero/few-shot generalization. Despite the efficiency of PEFT methods, the size of expert models can make it onerous to retrieve expert models per query over high-latency networks like the Internet or serve multiple experts on a single GPU. To address these issues, we present, a novel method for compressing fine-tuning residuals (task vectors)  of PEFT-based models.employs sparsification and ternary quantization to reduce the size of the PEFT module without performing any additional retraining while preserving or enhancing model performance. In extensive evaluation across,, and-based models withparameters,achieves compression ratios of. In particular, we show thatimproves with scale – stronger models exhibit higher compressibility and better performance. For example, we showapplied to LLaMA outperforms QLoRA byon MMLU with a storage size reduction of up to. In addition, we show that the compressed experts produced bymaintain few-shot compositional generalization capabilities, facilitate efficient communication and computation, and exhibit enhanced performance when merged. Lastly, we provide an analysis of different method components, compare it with other PEFT methods, and test’s efficacy for compressing the residual of full-finetuning.111Code is available athttps://github.com/prateeky2806/compeft.