**Proposition 1**.: _Assume, we have a reference model, such that \(\pi_{\text{ref}}(y|x)>0\) for all pairs of prompts \(x\) and answers \(y\) and a parameter \(\beta>0\). Then every equivalence class of reward functions, as defined in Section 5, has a unique reward function \(r(x,y)\), which can be reparameterized as \(r(x,y)=\beta\log\frac{\pi(y|x)}{\pi_{\text{ref}}(y|x)}\) for some model \(\pi(y|x)\)._Proof.: We will proceed using proof by contradiction. Assume we have two reward functions from the same class, such that \(r^{\prime}(x,y)=r(x,y)+f(x)\). Moreover, assume that \(r^{\prime}(x,y)=\beta\log\frac{\pi^{\prime}(y|x)}{\pi_{\text{ref}}(y|x)}\) for some model \(\pi^{\prime}(y|x)\) and \(r(x,y)=\beta\log\frac{\pi(y|x)}{\pi_{\text{ref}}(y|x)}\) for some model \(\pi(y|x)\), such that \(\pi\neq\pi^{\prime}\).