**Alignment Data Collection.** The construction of alignment data is important to effectively align the behaviors of LLMs with human preferences. To collect high-quality alignment data, some work tries to reuse existing reward models to select high-rated responses, and others explore to leverage powerful LLMs (_e.g._, ChatGPT) or build a simulated environment to generate synthetic alignment examples. Next, we will discuss these three lines of research.