@inproceedings{
shen2024search,
title={Search for Efficient Large Language Models},
author={Xuan Shen and Pu Zhao and Yifan Gong and Zhenglun Kong and Zheng Zhan and Yushu Wu and Ming Lin and Chao Wu and Xue Lin and Yanzhi Wang},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=lxSmLxlVks}
}
Large Language Models (LLMs) have long held sway in the realms of artificial intelligence research.
Numerous efficient techniques, including weight pruning, quantization, and distillation, have been embraced to compress LLMs, targeting memory reduction and inference acceleration, which underscore the redundancy  in LLMs.
However, most model compression techniques concentrate on weight optimization, overlooking the exploration of optimal architectures.
Besides, traditional architecture search methods, limited by the elevated complexity with extensive parameters, struggle to demonstrate their effectiveness on LLMs.
In this paper, we propose a training-free architecture search framework to identify optimal subnets that preserve the fundamental strengths of the original LLMs while achieving inference acceleration.
Furthermore, after generating subnets that inherit specific weights from the original LLMs, we introduce a reformation algorithm that utilizes the omitted weights to rectify the inherited weights  with a small amount of calibration data.
Compared with SOTA training-free structured pruning works that can generate smaller networks, our method demonstrates superior performance across standard benchmarks.
Furthermore, our generated subnets can directly reduce the usage of GPU memory and achieve inference acceleration.
Search for Efficient Large Language Models
Xuan Shen Pu Zhao Yifan Gong Zhenglun Kong Zheng Zhan Yushu Wu Ming Lin Chao Wu Xue Lin Yanzhi Wang