Offline/Off-\(\mathbf {Q}\) . Via the Performance Difference Lemma [1]}, we can write that
\(\frac{1}{T}(J(\pi ^E) - J(\pi )) &= \frac{1}{T} \sum _{t=1}^T \mathbb {E}_{\tau \sim \pi ^E}[Q^{\pi }(s_t, a_t, c) - \mathbb {E}_{a \sim \pi ^E}[Q^{\pi }(s_t, a, c)]] \\&\le \sup _{(f, \tilde{f}) \in \mathcal {F}_{Q} \times \tilde{\mathcal {F}}_{Q}} \frac{1}{T} \sum _{t=1}^T \mathbb {E}_{\tau \sim \pi ^E}[f_t - \tilde{f}_t + \tilde{f}_t - \mathbb {E}_{a \sim \pi }[f_t - \tilde{f}_t + \tilde{f}_t]] \\&\le \frac{T}{T} \sum _{t=1}^T \epsilon _{\textrm {off}}(t) + \sup _{(f, \tilde{f}) \in \mathcal {F}_{\textrm {off}} \times \tilde{\mathcal {F}}_{\textrm {off}}} \frac{T}{T} \sum _{t=1}^T \mathbb {E}_{\tau \sim \pi ^E}[f_t - \tilde{f}_t - \mathbb {E}_{a\sim \pi }[f_t - \tilde{f}_t]] \nonumber \\&= \sum _{t=1}^T \epsilon _{\textrm {off}}(t) + \delta _{\textrm {off}}(t).\) 
