\(\bullet\)_Primary training objective._ Since the alignment data typically consists of an input instruction and an output response, the primary training loss is still the traditional cross-entropy loss for sequence-to-sequence learning. Based on this loss, many studies propose a number of improvement variants for enhancing the supervised alignment tuning. For example, CoH [388] constructs the training data by prepending _"A helpful answer:"_ and _"An unhelpful answer:"_ to the annotated good and bad responses, respectively, and only compute losses for those response tokens with special masking. Quark [384] sorts model responses into different quantiles with varying alignment quality, it prepends a special reward token to each model response to represent the reward level of the response. Further, to enable the preference modeling via the maximum likelihood objective, DPO [389] first reparameterizes the response rewards using the policy model (_i.e._, the language model being optimized), and then the original reward modelling objective can be reformulated only based on the policy model. In this way, DPO removes the explicit reward modeling step, and optimizing the new learning objective only involving the policy model is equivalent to optimizing the rewards. Furthermore, FIGA [386] designs a fine-grained contrastive loss that aims to encourage desirable tokens, penalize undesirable ones, and disregard trivial tokens.