On the Transformations across Reward Model, Parameter Update, and In-Context Prompt

Despite the general capabilities of pre-trained large language models (LLMs), they still need further adaptation to better serve practical applications. In this paper, we demonstrate the interchangeability of three popular and distinct adaptation tools: parameter updating, reward modeling, and in-context prompting. This interchangeability establishes a triangular framework with six transformation directions, each of which facilitates a variety of applications. The primary contribution of this paper is to offer a holistic view about the triangular framework depicted in the following figure, which encompasses six distinct transformation directions in total. We systematically analyze each transformation by first formally defining its objectives, then investigating the transformation methods, and reviewing pertinent existing works that utilize these transformations for various purposes. Our work spans a substantial breadth of the current frontier in LLM research and establishes insightful connections among diverse prior studies that may initially seem unrelated, which contribute to advancing the understanding of the current landscape in LLM research. In addition to our extensive survey of existing applications, we delineate several promising future research avenues within each transformation direction.