**Conditional Text Generation.** As an important topic in language generation, conditional text generation [48] focuses on generating texts satisfying specific task demands based on the given conditions, typically including machine translation [624], text summarization [548], and question answering [557]. To measure the quality of the generated text, automatic metrics (_e.g._, Accuracy, BLEU [625] and ROUGE [626]) and human ratings have been typically used for evaluating the performance. Due to the powerful language generation capabilities, LLMs have achieved remarkable performance on existing datasets and benchmarks. For instance, GPT-4 exhibits comparable performance as commercial translation products, even for the translation task of languages that are with significant linguistic distance [627]. On news summarization tasks (_i.e._, CNN/DM and XSUM), LLMs also demonstrate comparable performance with human freelance writers [628]. Despite the rapid progress on model capacity, there are increasing concerns on the feasibility of existing automatic metrics to faithfully assess the performance of LLMs in conditional text generation tasks [628, 629, 630]. As the alternatives to automatic metrics, recent studies also propose to incorporate LLMs as generation evaluators to examine the quality of the generated content [631, 632, 138]. Moreover, researchers also explore more challenging language generation tasks for LLMs, such as structured data generation [458] and long text generation [633, 46, 46].