@inproceedings{sun-etal-2024-decoding,
    title = "Decoding at the Speed of Thought: Harnessing Parallel Decoding of Lexical Units for {LLM}s",
    author = "Sun, Chenxi  and
      Zhang, Hongzhi  and
      Lin, Zijia  and
      Zhang, Jingyuan  and
      Zhang, Fuzheng  and
      Wang, Zhongyuan  and
      Chen, Bin  and
      Song, Chengru  and
      Zhang, Di  and
      Gai, Kun  and
      Xiong, Deyi",
    editor = "Calzolari, Nicoletta  and
      Kan, Min-Yen  and
      Hoste, Veronique  and
      Lenci, Alessandro  and
      Sakti, Sakriani  and
      Xue, Nianwen",
    booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.lrec-main.401",
    pages = "4476--4487",
    abstract = "Large language models have demonstrated exceptional capability in natural language understanding and generation. However, their generation speed is limited by the inherently sequential nature of their decoding process, posing challenges for real-time applications. This paper introduces Lexical Unit Decoding (LUD), a novel decoding methodology implemented in a data-driven manner, accelerating the decoding process without sacrificing output quality. The core of our approach is the observation that a pre-trained language model can confidently predict multiple contiguous tokens, forming the basis for a lexical unit, in which these contiguous tokens could be decoded in parallel. Extensive experiments validate that our method substantially reduces decoding time while maintaining generation quality, i.e., 33{\%} speed up on natural language generation with no quality loss, and 30{\%} speed up on code generation with a negligible quality loss of 3{\%}. Distinctively, LUD requires no auxiliary models and does not require changes to existing architectures. It can also be integrated with other decoding acceleration methods, thus achieving an even more pronounced inference efficiency boost. We posit that the foundational principles of LUD could define a new decoding paradigm for future language models, enhancing their applicability for a broader spectrum of applications. All codes are be publicly available at https://github.com/tjunlp-lab/Lexical-Unit-Decoding-LUD-.",
}

Large language models have demonstrated exceptional capability in natural language understanding and generation. However, their generation speed is limited by the inherently sequential nature of their decoding process, posing challenges for real-time applications. This paper introduces Lexical Unit Decoding (LUD), a novel decoding methodology implemented in a data-driven manner, accelerating the decoding process without sacrificing output quality. The core of our approach is the observation that a pre-trained language model can confidently predict multiple contiguous tokens, forming the basis for a lexical unit, in which these contiguous tokens could be decoded in parallel. Extensive experiments validate that our method substantially reduces decoding time while maintaining generation quality, i.e., 33{\%} speed up on natural language generation with no quality loss, and 30{\%} speed up on code generation with a negligible quality loss of 3{\%}. Distinctively, LUD requires no auxiliary models and does not require changes to existing architectures. It can also be integrated with other decoding acceleration methods, thus achieving an even more pronounced inference efficiency boost. We posit that the foundational principles of LUD could define a new decoding paradigm for future language models, enhancing their applicability for a broader spectrum of applications. All codes are be publicly available at https://github.com/tjunlp-lab/Lexical-Unit-Decoding-LUD-.
Sun, Chenxi Zhang, Hongzhi Lin, Zijia Zhang, Jingyuan Zhang, Fuzheng Wang, Zhongyuan Chen, Bin Song, Chengru Zhang, Di Gai, Kun Xiong, Deyi
Decoding at the Speed of Thought: Harnessing Parallel Decoding of Lexical Units for LLMs