We adopt a Markov Decision Processes (MDPs) formulation defined as the tuple \(M=(\mathcal {S}, \mathcal {A}, P, R, \gamma )\)   [1]}. Here, \(\mathcal {S}\)  is a set of states, \(\mathcal {A}\)  is a set of actions the agent can take, \(P\)  specifies the transition function of the environment, and \(R\)  is the reward function. At timestep \(t\) , the agent observes state \(s_t \in \mathcal {S}\) , selects an action \(a_t \in \mathcal {A}\) , after which the environment returns a next state \(s_{t+1} \sim P(\cdot |s_t,a_t)\)  and associated reward \(r_t = R(s_t,a_t,s_{t+1})\) . We act in the MDP according to a policy \(\pi (a|s)\) , which maps a state to a distribution over actions. When we unroll the policy and environment for \(T\)  steps, define the cumulative reward (return) of the resulting trace as \(\sum ^{T}_{k=0} \gamma ^k \cdot r_{t+k+1}\) , where \(\gamma \in [0,1]\)  denotes a discount factor. The goal in RL is to learn a policy which can maximize the expected return.
