@inproceedings{
gao2024displlm,
title={{DISP}-{LLM}: Dimension-Independent Structural Pruning for Large Language Models},
author={Shangqian Gao and Chi-Heng Lin and Ting Hua and Zheng Tang and Yilin Shen and Hongxia Jin and Yen-Chang Hsu},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=YxaY6tHgg0}
}
Large Language Models (LLMs) have achieved remarkable success in various natural language processing tasks, including language modeling, understanding, and generation. However, the increased memory and computational costs associated with these models pose significant challenges for deployment on resource-limited devices. Structural pruning has emerged as a promising solution to reduce the costs of LLMs without requiring post-processing steps. Prior structural pruning methods either follow the dependence of structures at the cost of limiting flexibility, or introduce non-trivial additional parameters by incorporating different projection matrices. In this work, we propose a novel approach that relaxes the constraint imposed by regular structural pruning methods and eliminates the structural dependence along the embedding dimension. Our dimension-independent structural pruning method offers several benefits. Firstly, our method enables different blocks to utilize different subsets of the feature maps. Secondly, by removing structural dependence, we facilitate each block to possess varying widths along its input and output dimensions, thereby significantly enhancing the flexibility of structural pruning. We evaluate our method on various LLMs, including OPT, LLaMA, LLaMA-2, Phi-1.5, and Phi-2. Experimental results demonstrate that our approach outperforms other state-of-the-art methods, showing for the first time that structural pruning can achieve an accuracy similar to semi-structural pruning.
DISP-LLM: Dimension-Independent Structural Pruning for Large Language Models
Shangqian Gao Chi-Heng Lin Ting Hua Zheng Tang Yilin Shen Hongxia Jin Yen-Chang Hsu