This work uses the standard MDP definition and variables [1]}, with state \(\mathbf {s}_t\in \mathbf {S}\) , action \(\mathbf {a}_t\in \mathbf {A}\) , reward \(r_t(\mathbf {s}_t, \mathbf {a}_t, \mathbf {s}_{t+1}) \in \mathbb {R}\)  and environment state transition distribution \(\rho _\text{env}\) .
During training, transition tuples of \(\lbrace \mathbf {s}_t, \mathbf {a}_t, r_t, \mathbf {s}_{t+1}\rbrace \)  are stored in a replay buffer \(\mathcal {D}\) , and the agent's policy is \(\pi (\mathbf {a}_t| \mathbf {s}_t)\) .
