In practice, the episodes in the offline dataset have finite horizon.
Thus, special treatment is needed for those terminal states in calculating the Bellman update target.
We combine common practice [1]}, [2]} with our critic update target Eq. (REF ) as
\(\widetilde{Q}\left(s, a\right) = {\left\lbrace \begin{array}{ll}r(s,a) + \gamma \cdot \widetilde{Q}\left(s, a\right) \cdot \left(\left|\widetilde{Q}\left(s, a\right)\right| < 2000\right) & \text{ if $s^{\prime }$ is a non-terminal state} \\r(s,a) & \text{ if $s^{\prime }$ is a terminal state}\end{array}\right.},\) 
