**Reward Modelling Phase**: In the second phase the SFT model is prompted with prompts \(x\) to produce pairs of answers \((y_{1},y_{2})\sim\pi^{\text{SFT}}(y\mid x)\). These are then presented to human labelers who express preferences for one answer, denoted as \(y_{w}\succ y_{l}\mid x\) where \(y_{w}\) and \(y_{l}\) denotes the preferred and dispreferred completion amongst \((y_{1},y_{2})\) respectively. The preferences are assumed to be generated by some latent reward model \(r^{*}(y,x)\), which we do not have access to. There are a number of approaches used to model preferences, the Bradley-Terry (BT) [5] model being a popular choice (although more general Plackett-Luce ranking models [30; 21] are also compatible with the framework if we have access to several ranked answers).