where \(\sigma\) is the logistic function. In the context of LMs, the network \(r_{\phi}(x,y)\) is often initialized from the SFT model \(\pi^{\text{SFT}}(y\mid x)\) with the addition of a linear layer on top of the final transformer layer that produces a single scalar prediction for the reward value [49]. To ensure a reward function with lower variance, prior works normalize the rewards, such that \(\mathbb{E}_{x,y\sim\mathcal{D}}\left[r_{\phi}(x,y)\right]=0\text{ for all }x\).