The operator \(f\) simply normalizes the reward function with the logarithm of the partition function of \(\pi_{r}\). Since the added normalization term is only a function of the prefix \(x\), \(f(r;\pi_{\text{ref}},\beta)(x,y)\) is a reward function in the equivalence class of \(r(x,y)\). Finally, replacing \(r\) with the RHS of Eq. 5 (which holds for any reward function), we have \(f(r;\pi_{\text{ref}},\beta)(x,y)=\beta\log\frac{\pi_{r}(y|x)}{\pi_{\text{ref}} (y|x)}\). That is, the projection \(f\) produces a member of the equivalence class of \(r\) with the desired form, and we do not lose any generality in our reward model from the proposed reparameterization.