where \(Z(x)=\sum_{y}\pi_{\text{ref}}(y\mid x)\exp\left(\frac{1}{\beta}r(x,y)\right)\) is the partition function. See Appendix A.1 for a complete derivation. Even if we use the MLE estimate \(r_{\phi}\) of the ground-truth reward function \(r^{*}\), it is still expensive to estimate the partition function \(Z(x)\)[17; 15], which makes this representation hard to utilize in practice. However, we can rearrange Eq. 4 to express the reward function in terms of its corresponding optimal policy \(\pi_{r}\), the reference policy \(\pi_{\text{ref}}\), and the unknown partition function \(Z(\cdot)\).